Windows Anti-Debug Reference Nicolas Falliere 2007-09-12
This paper classifies and presents several anti-debugging techniques used on Windows NT-based operating systems. Anti-debugging techniques are ways for a program to detect if it runs under control of a debugger. They are used by commercial executable protectors, packers and malicious software, to prevent or slow-down the process of reverse-engineering. We'll suppose the program is analyzed under a ring3 debugger, such as OllyDbg on Windows platforms. The paper is aimed towards reverse-engineers and malware analysts. Note that we will talk purely about generic anti-debugging and anti-tracing techniques. Specific debugger detection, such as window or processes enumeration, registry scanning, etc. will not be addressed here.
Some packed files have their entry point RVA set to 0, which means they will start executing 'MZ...' which corresponds to 'dec ebx / pop edx ...'.
This is not an anti-debug trick in itself, but can be annoying if you want to break on the entry-point by using a software breakpoint.
If you create a suspended process, then set an INT3 at RVA 0, you will erase part of the magic MZ value ('M'). The magic was checked when the process was created, but it will get checked again by ntdll when the process is resumed (in the hope of reaching the entry-point). In that case, an INVALID_IMAGE_FORMAT exception will be raised.
If you create your own tracing or debugging tool, you will want to use hardware breakpoint to avoid this problem.