As I myself freely state, virtually every serious cracker/reverser will use a debugger of sorts to break an application, of those out there I'll wager a substantial amount that SoftICE is the "debugger of choice". Modern day programmers have still not really caught up with "crackers", maybe they just consider it a wasted cause?, or perhaps their desire to roll out their latest software overrides any desire to protect. At any rate, with the progression of commercial wrapper schemes e.g. VBox, I think a list of the most common tricks encountered by reversers would be useful.
Most of these tricks require implementation in Assembly language, evidently that isn't a problem if you are familiar with in-line ASM coding in your compiler environment. Most of these examples are designed to thwart SoftICE users (virtually all of the existing documents on the web describe tricks which would have been useful in the days of DOS Debug/Turbo Debug but are happily traced by SoftICE).
1. - Using the API function CreateFileA to check for the presence of the SoftICE vxd/sys device (sice, siwdebug, ntice.sys in Windows NT).
Bypass by bpx CreateFileA (change API result) or HEX edit the strings from the detection. (Seen in Advanced Disk Catalog v1.16, MeltICE, new HASP 7 envelope).
2. - INT 41h Debugger Notification.
PUSH EDI
PUSH 0000004F
PUSH 002A002A
CALL Kernel32_1 <-- ORD_1 or VxdCall.
SUB AX,F386
POP EDI
JZ SoftICE_is_running
This detection pushes 002A002A as a parameter, the high 2A means we are
calling VWIN_32 vxd, the low 2A the INT 41 dispatch service. Bypass
with bpint 30 if ax==0xF386 and clear AX, described in Matt
Pietrek's legendary book.
3. - Issue commands to SoftICE. This was one detection I uncovered whilst running an installation program protected by a Hardlock dongle (HASP wrappers also use it). I had inadvertently enabled INT 3's with SoftICE's I3HERE toggle.
:? 4647
00004647 0000017991 "FG" <-- Magic Val. 1.
:? 4A4D
00004A4D 0000019021 "JM" <-- Magic Val. 2.
28F1:0092 MOV SI,4647
28F1:0095 MOV DI,4A4D
28F1:0098 PUSH CS
28F1:0099 POP DS
28F1:009A MOV AX,0911h <-- Function 0911.
28F1:009D MOV DX,000Eh <-- Points at null-terminated command (in this case HBOOT).
28F1:00A0 INT 3 <-- Call Interrupt.
Evidently there are several easy ways to beat this, changing SoftICE's magic values is one option (as described by The_Owl), another would be editing DI/SI or modifying the string at DX, you could also just NOP the INT 3 altogether, this detection routine works with varying levels of success. Other subfunctions exist (0912h, 0913h & 0914h) which can be used to manipulate SoftICE breakpoints.
Another set of magic values are also known ('BCHK'), this is the documented BoundsChecker interface, if you place BCHK into EBP and set EAX=4 calling INT 3 will return AL=0 in the presence of Winice (works in Windows).
MOV EBP, 04243484Bh <-- 'BCHK'.
MOV AX, 4h
INT 3 <-- Trap debugger.
CMP AL, 4
JNZ SoftICE_is_here
4. - Using NuMega's own functions. This requires the use of nmtrans.dll but may well appeal to HLL programmers who aren't keen on using in-line ASM. The idea is to call DevIO_ConnectToSoftICE and verify the functions return value before taking appropriate action. Sadly this method is based heavily on CreateFileA (although a small amount of work could change that), hence in its current form it could easily be found. Looking through some of the other exports from nmtrans.dll I reckon a sneaky programmer could use some of the other exports.
5. - ICECream detection (Windows 95).
Get the Interrupt Descriptor Table (IDT) with the SIDT command.
Get the address of Interrupt gate 1.
Move 16 bytes back.
Check if byte is 1Eh - if so SoftICE is running.
SIDT FWORD PTR opIDT <-- Store IDT.
MOV EAX, DWORD PTR [opIDT+2] <-- EAX=IDT.
ADD EAX, 8h <-- EAX has INT 1 vector.
MOV EBX, [EAX] <-- EBX = INT 1 vector.
ADD EAX, 16h <-- EAX points at INT 3 vector.
MOV EAX, [EAX] <-- Get EAX = INT 3 vector.
AND EAX, 0FFFFh
AND EBX, 0FFFFh <-- Remove selectors.
SUB EAX, EBX <-- Find displacement.
CMP EAX, 01Eh
JZ SoftICE_3.0_is_running
6. - Detect SoftICE VxD or SoftICE GFX VxD (obviously ineffective under NT).
XOR DI,DI <-- Clear DI.
MOV ES,DI
MOV AX, 1684h
MOV BX, 0202h <-- VxD ID for SoftICE.
INT 2Fh
MOV AX, ES <-- VxD Entry Point.
ADD AX, DI
TEST AX,AX
JNZ SoftICE_is_here
The GFX id code is identical to that shown above except AX=1684h & BX=7A5Fh.
7. - Detect WinICE handler using INT 68h (V86).
MOV AH, 43h
INT 68h
CMP AX, 0F386h <-- Will be set by all system debuggers.
JZ SoftICE_is_here
8. - Detect and crash SoftICE with an illegal form of the instruction CMPXCHG8B (LOCK prefix) - opcode: F0 0F C7 C8.
9. - Searching for names installed by SoftICE in low-memory, "WINICE.BR", "SOFTICE1" + others (described in Stone's code below).
10. - Use the debug registers - I remember reading somewhere (perhaps in a fairly old Anti-debugging FAQ) that SoftICE doesn't handle the debug registers very well or not at all - specifically dr4 and above, a little something for you to play with me thinks :).
11. - Dongle protection (used by dongle SSI Win32 Aegis).
xxxx: xxxx
EB01: JMP $+1
E8: DB E8h
xxxx: xxxx
SoftICE View:
xxxx: xxxx
EB01: JMP $+1
E8xxxxxxxx: CALL bad
TRW (not affected):
xxxx: xxxx
EB01E8: NOP
xxxx: xxxx
12. Calling the Windows function IsDebuggerPresent() (exported from kernel32.dll), returns non-zero in the presence of a debugger, implemented only under NT so not a great trick on its own. A few reversers have already patched their kernel32's against this one.
BPX Detection & Tricking Series - Detecting breakpoints by duelist - (55k).
FrogsICE - by Frogs Print, anti-SoftICE detection via vxd - (13k).
SoftICE Detection - by Stone (includes ASM source code) - (5k).
I'd really like to add more tricks here so if you encounter any others please post me a small text file (I'd certainly be grateful).