Stack-Based Exploitation (User-Mode)

Last updated: 2026-04-10
Related: Rop, Mitigations, Heap Internals
Tags: user-mode, stack-overflow, rop, dep-bypass

Summary

Stack-based buffer overflows are the classic memory corruption bug. While heap exploitation dominates modern Windows research (due to mitigations making stack exploitation harder), stack bugs still appear in kernel drivers, network parsers, and legacy code. Stack exploitation on modern Windows requires defeating /GS cookies, bypassing SEHOP, and chaining ROP to defeat DEP/NX.

Stack Layout (x64 Windows)

High addresses
+------------------+
| Function args    | (if >4 args, 5th+ on stack)
+------------------+
| Return address   | ← overflow target
+------------------+
| Saved RBP        | (if used)
+------------------+
| /GS cookie       | ← must overwrite correctly
+------------------+
| Local variables  | ← overflow starts here
+------------------+
Low addresses

Note: x64 uses registers RCX, RDX, R8, R9 for first 4 args — stack layout cleaner than x86.

/GS Stack Cookie

Location: between local variables and return address
Value: random per-process, per-image-load (with ASLR)
Validation: function epilogue checks saved cookie against __security_cookie global

Defeating /GS

1. Info leak the cookie: find a read vulnerability that exposes the stack
   • Format string: %n$p to read stack values including cookie
   • OOB read: read beyond array into cookie location
2. Overwrite entire record correctly: need exact cookie value
3. Avoid the stack altogether: target heap objects, vtables instead of return address
4. Exception-based bypass: trigger exception before /GS check runs (indirect path through SEH)

/GS Cookie Location

// From TEB:
ULONG64 cookie = *(ULONG64*)(ImageBase + __security_cookie_offset);
// Cookie XOR'd with current RSP for each function:
stored_cookie = cookie ^ (ULONG64)current_rsp;

SEH-Based Exploitation (x86 / Legacy 32-bit)

In 32-bit Windows, exception handler records on the stack:

[SEH record]: NextHandler → DispatchHandler
              ExceptionHandler → function pointer

Overflow into SEH record → trigger exception → controlled RIP.

SafeSEH (Vista+)

• Module’s valid handler list in PE header
• Dispatcher validates handler address is in the list
• Bypass: use handler address from non-SafeSEH DLL

SEHOP (Vista SP1+)

• Validates SEH chain terminates at ntdll!FinalExceptionHandler
• Bypass: not meaningful on 64-bit (64-bit uses table-based exception handling, no inline SEH)

64-bit Exception Handling

x64 Windows uses table-based exception handling (.pdata section):

• No SEH chain on stack
• Exception dispatch looks up handler in PE’s .pdata table
• Cannot overwrite exception handlers via stack overflow
• Stack overflow → return address overwrite or local pointer corruption only

Return Address Overwrite + ROP Chain

Standard modern stack exploitation flow:

1. Overflow into return address
2. Overwrite with stack pivot gadget (if needed) or first ROP gadget
3. ROP chain on stack (or in heap spray if stack not fully controlled)
4. Chain calls VirtualProtect / VirtualAlloc → make shellcode region executable
5. Jump to shellcode

If /GS present: need cookie value (see above).
If DEP present: use ROP (see Rop).
If ASLR present: need info leak for module base.

Kernel Driver Stack Exploitation

Driver IOCTL handlers often use stack buffers for temporary storage. Overflow from IOCTL input:

// Typical vulnerable driver IOCTL handler:
NTSTATUS DeviceControl(PDEVICE_OBJECT DevObj, PIRP Irp) {
    PIO_STACK_LOCATION stack = IoGetCurrentIrpStackLocation(Irp);
    ULONG inputLen = stack->Parameters.DeviceIoControl.InputBufferLength;
    PVOID inputBuf = Irp->AssociatedIrp.SystemBuffer;
    
    CHAR localBuf[0x100];
    // BUG: inputLen not checked:
    RtlCopyMemory(localBuf, inputBuf, inputLen);  // kernel stack overflow!

Kernel stack overflow → overwrite kernel return address → kernel ROP.

Kernel stack notes:

• Kernel stack is 12-16KB by default, fixed size
• No SMEP bypass needed for kernel stack ROP (code is already in kernel space)
• HVCI still prevents placing new executable code → must be kernel-ROP or data-only

Practical Stack Bug Hunting Checklist

For IOCTL handlers:
[ ] Fixed-size local buffers used with user-controlled size?
[ ] strcmp/strcpy/sprintf/memcpy with user input?
[ ] RtlCopyMemory without prior size validation?
[ ] Recursive functions with user-controlled depth?

For network parsers:
[ ] Length fields from packet used as copy size?
[ ] String parsing without null-termination check?
[ ] Multi-byte character handling in fixed buffer?

Exploit Relevance

Stack exploitation in user-mode is mostly encountered in:

• Legacy code (pre-Vista 32-bit targets)
• Embedded/IoT Windows targets (old builds)
• Kernel drivers with IOCTL input handling bugs

For modern 64-bit user-mode targets: heap and vtable-based exploitation dominates. Stack bugs in kernel drivers remain a relevant and underexplored surface.

References

• “Exploit Writing Tutorial” series — Corelan Team (corelan.be)
• “Stack Overflow Exploitation” — SANS GIAC resources
• “Windows x64 Exception Handling” — Ken Johnson / Matt Miller