7/27/2019 11 Windows[1]
http://slidepdf.com/reader/full/11-windows1 1/28
Crash Dump AnalysisDebugging in Windows
Jakub Jermář Martin Děcký
7/27/2019 11 Windows[1]
http://slidepdf.com/reader/full/11-windows1 2/28
Crash Dump Analysis – MFF UK – Debugging in Windows 2
Windows vs. Unixes
● Many things very similar (in principle)
● Many things slightly different
▪
Terminology▪ Tools and file formats
– Visual C++, PE, PDB
▪ Technology and techniques
– In-kernel graphics, blending of kernel and user space
library functions (heap, resources, etc.)
▪ Conventions and habbits
7/27/2019 11 Windows[1]
http://slidepdf.com/reader/full/11-windows1 3/28
Crash Dump Analysis – MFF UK – Debugging in Windows 3
Windows calling conventions
● cdecl (C calling convention)
▪ Almost identical to System V ABI
▪ Arguments passed on stack in reverse order
– Support for variadic functions
▪ Usual prologue, epilogue and stack frames
push %ebp leave
movl %esp, %ebp ret
sub $imm, %esp
▪ Caller cleans the stack (pops the arguments)
7/27/2019 11 Windows[1]
http://slidepdf.com/reader/full/11-windows1 4/28
Crash Dump Analysis – MFF UK – Debugging in Windows 4
Windows calling conventions
● stdcall (standard calling convention)
▪ Used for all Win32 API calls (WIN32API macro)
▪ Arguments passed on stack in reverse order
▪ No support for variadic functions
– Callee cleans the stack (using ret $imm )
● Slightly shorter code
– Arity encoded in function name
▪ Variants of prologue and epilogue
enter $imm, 0 leave
ret $imm
7/27/2019 11 Windows[1]
http://slidepdf.com/reader/full/11-windows1 5/28
Crash Dump Analysis – MFF UK – Debugging in Windows 5
Windows calling conventions
● fastcall
▪ Almost identical to stdcall
– Arguments passed on stack in reverse order
– First two integer arguments passed in ECX , EDX
– No support for variadic functions
● Callee cleans the stack (using ret $imm )
●
thiscall (C++)▪ Almost identical to stdcall
– Arguments passed on stack in reverse order, callee
cleans, implicit argument (*this) passed in ECX
7/27/2019 11 Windows[1]
http://slidepdf.com/reader/full/11-windows1 6/28
Crash Dump Analysis – MFF UK – Debugging in Windows 6
Windows calling conventions
● AMD64 cdecl
▪ Used universally on AMD64, similar to System V
▪ First four arguments passed in RCX , RDX , R8 , R9
– Space on stack is reserved for possible spill
▪ Other arguments passed on stack in reverse order
– Caller cleans the arguments
●
Support for variadic functions
– 16B alignment
– 16B red zone
– Scratch: RAX , RCX , RDX , R8 , R9 , R10 , R11
7/27/2019 11 Windows[1]
http://slidepdf.com/reader/full/11-windows1 7/28
Crash Dump Analysis – MFF UK – Debugging in Windows 7
Debugging overview
● Windows debugging possibilities
▪ User space debuggers
– Common debugging API (dbghelp.dll)
● Visual Studio Debugger
● CDB, NTSD
● Others
▪ Windows kernel debugger
– Part of Windows NT kernel
● KD
● Remote debugging (serial, FireWire, USB 2.0, VMWare
extension)
7/27/2019 11 Windows[1]
http://slidepdf.com/reader/full/11-windows1 8/28
Crash Dump Analysis – MFF UK – Debugging in Windows 8
Debugging overview (2)
▪ WinDbg
– GUI front-end for CDB, NTSD and KD
– Both intruction level and source code level debugging
– Extensible via DLL plugins (somehow similar to mdb)● Support for debugging .NET binaries, etc.
▪ 3rd party kernel debuggers
– SoftICE, Syser, Rasta Ring 0 Debugger
● Kernel-only instruction level debugging
● Various run-time kernel patching techniques to gain control over
the Windows NT kernel
● Can make some use of virtualization environments
7/27/2019 11 Windows[1]
http://slidepdf.com/reader/full/11-windows1 9/28
Crash Dump Analysis – MFF UK – Debugging in Windows 9
Resources
● Debugging Tools for Windows
▪ WinDbg and friends freely downloadable
http://www.microsoft.com/whdc/devtools/debugging/
▪ Documentation in MSDN
http://msdn.microsoft.com/en-us/library/cc267445.aspx
▪ Good tutorials
http://www.codeproject.com/KB/debug/windbg_part1.asp
http://windbg.info/ (WinDbg from A to Z)
7/27/2019 11 Windows[1]
http://slidepdf.com/reader/full/11-windows1 10/28
Crash Dump Analysis – MFF UK – Debugging in Windows 10
Windows debugging API
● Common methods for writing debuggers
▪ Parsing binaries (ImageNtHeader)
▪ Dumping core (MiniDumpWriteDump)
▪ Generating stack trace (StackWalk)
▪ Symbol handling (SymFromAddr)
– Original symbol information format: COFF
– Current symbol information format: PDB file
7/27/2019 11 Windows[1]
http://slidepdf.com/reader/full/11-windows1 11/28
Crash Dump Analysis – MFF UK – Debugging in Windows 11
Symbols
● Symbol location
▪ _NT_SYMBOL_ATH environment variable
– Binaries and symbols matched according to compilation
timestamp and/or GUID– Symbols for Windows components (all public builds)
● Downloadable from Microsoft public symbol server
_NT_SYMBOL_PATH=srv*c:\sym_cache*http://msdl.microsoft.com/download/symbols
●
Can be also downloaded by hand (~ 200 MB)http://www.microsoft.com/whdc/devtools/debugging/symbolpkg.mspx
– Own symbol server and source server
● For debugging of release binaries at customer
7/27/2019 11 Windows[1]
http://slidepdf.com/reader/full/11-windows1 12/28
Crash Dump Analysis – MFF UK – Debugging in Windows 12
CDB
● Command line user space debugger
▪ NTSD is almost identical, but it is not a console
application
▪ Two debugging modes
– Invasive debugging
● A break-in thread is created in target process
● Full-featured debugging (but only one debugging session)
● Prior to XP: no detach was possible
– Non-invasive debugging
● Only frozen threads
● Memory analysis possible, but no flow control (breakpoints, etc.)
7/27/2019 11 Windows[1]
http://slidepdf.com/reader/full/11-windows1 13/28
Crash Dump Analysis – MFF UK – Debugging in Windows 13
KD
● Command line kernel debugger
▪ Limited possibilities to debug local kernel
▪ Debugging of a remote kernel
– Serial line
● Limited to 115 kbaud
● VMWare virtual serial line can be much faster
– FireWire (IEEE 1394)
● Fast, but the generic FireWire driver has to be deactivated
– USB 2.0
● Fast, but special debugging cable is required
7/27/2019 11 Windows[1]
http://slidepdf.com/reader/full/11-windows1 14/28
Crash Dump Analysis – MFF UK – Debugging in Windows 14
WinDbg
● Universal GUI front-end
▪ Both CDB and KB
– Running processes
– Attaching to existing processes
– Opening core dumps and crash dumps
– Remote debugging
▪
Basically still the same command line interface– But more windows and some special views for easier
navigation
● Watches, breakpoints, disassembly, source code, registers, etc.
7/27/2019 11 Windows[1]
http://slidepdf.com/reader/full/11-windows1 15/28
Crash Dump Analysis – MFF UK – Debugging in Windows 15
Remote debugging
● For user space applications
▪ Kernel needs to be always debugged remotely
using a dedicated connection
▪ Debugging target
dbgsrv.exe -t tcp:port=1025
▪ Debugging client
windbg.exe -premote tcp:server=hostname ,port=1025 – Usefull command
!tli"t
– List processes running on target
7/27/2019 11 Windows[1]
http://slidepdf.com/reader/full/11-windows1 16/28
Crash Dump Analysis – MFF UK – Debugging in Windows 16
WinDbg commands
● Regular commands
▪ No prefix, but different suffixes (variants)
▪ Controlling the debugging session
# %cmd&
● Help on commands
g
● Continue execution
p● Step over (instruction or source line)
t
● Step into
7/27/2019 11 Windows[1]
http://slidepdf.com/reader/full/11-windows1 17/28
Crash Dump Analysis – MFF UK – Debugging in Windows 17
WinDbg commands (2)
pt● Step over until next return
tt
● Step into until next return (skipping nested returns)
pc● Step over until next call
● If the current instruction is call then it is ignored
tc
●
Step into until next callpa %addr&
● Step over until address is reached
7/27/2019 11 Windows[1]
http://slidepdf.com/reader/full/11-windows1 18/28
Crash Dump Analysis – MFF UK – Debugging in Windows 18
WinDbg commands (3)
r● Dump all registers
u 'addr(
● Disassemble
lm● List loaded modules (DLLs)
)
● Get information from all threads
)!● Get current thread information
)'num(
● Get information from thread num
7/27/2019 11 Windows[1]
http://slidepdf.com/reader/full/11-windows1 19/28
Crash Dump Analysis – MFF UK – Debugging in Windows 19
WinDbg commands (4)
k● Stack trace (of current thread)
)* k
● Stack trace of all threads
k● Stack trace with full function arguments
k+
● Stack trace with the information about calling conventions
dd %addr& da %addr& du %addr&● Display doubleword, ASCII, Unicode
, %addr& %+alue& !!!
● Fill memory with doubleword values
7/27/2019 11 Windows[1]
http://slidepdf.com/reader/full/11-windows1 20/28
Crash Dump Analysis – MFF UK – Debugging in Windows 20
WinDbg commands (5)
-l● List breakpoints
-p %addr&
● Set (execution) breakpoint
-u %"ym-ol&● Set (execution) breakpoint with lazy symbol resolving (waiting for
given module to be loaded)
-a %addr&
● Memory access breakpoint
-c %addr&
● Clear breakpoint
-e %addr& -d %addr&
● Enable, disable breakpoint
7/27/2019 11 Windows[1]
http://slidepdf.com/reader/full/11-windows1 21/28
Crash Dump Analysis – MFF UK – Debugging in Windows 21
WinDbg expressions
## ..c//0 !!! 1
– Returns the value of any C++ expression which does not
have any side effects (you cannot call functions)
● Understands compound types, arrays, pointer arithmetics, etc.
– Implicitly used in watch and locals windows for watches
and displaying local variables
● Display value of integer variable
## local_+ar
n $23
● Display memory location of an integer variable
## 4local_+ar
n $*$565573829:
7/27/2019 11 Windows[1]
http://slidepdf.com/reader/full/11-windows1 22/28
Crash Dump Analysis – MFF UK – Debugging in Windows 22
Advanced breakpoints
▪ Setting breakpoints on multiple functions
– Wildcards
-p module;my_,unc_*
▪ Setting breakpoints on a member function of all
instances of a class
– C++ expression evaluation
-p ..c//0 My<la""=MyMet>od 1
▪
Setting breakpoint on a function which hits only in agiven thread
– Lazy resolving
)7 -u kernel83;LoadLi-rary?6W
7/27/2019 11 Windows[1]
http://slidepdf.com/reader/full/11-windows1 23/28
Crash Dump Analysis – MFF UK – Debugging in Windows 23
WinDbg commands (6)
● Dot commands
▪ Slightly advanced
!>elp %cmd&
●
Help on dot commands
!la"te+ent
● Display information about last event/exception
!dump
●
Create core dump (also Dr. Watson utility)!attac> %pid&
● Attach to a process
!detac>
7/27/2019 11 Windows[1]
http://slidepdf.com/reader/full/11-windows1 24/28
Crash Dump Analysis – MFF UK – Debugging in Windows 24
WinDbg commands (7)
!re"tart● Restart process
!i, !el"e !el"ei,
● Optional command execution
●
C++ expressions as conditions● Multiple commands can be enclosed in {} blocks
!,or !@>ile !Break !continue
● Advanced scripting
!,oreac> %cmd& %-lock&
● The output of a command is fed to a block of commands
● Usually line-by-line
● The semantics of the input differs command by command
7/27/2019 11 Windows[1]
http://slidepdf.com/reader/full/11-windows1 25/28
Crash Dump Analysis – MFF UK – Debugging in Windows 25
WinDbg commands (8)
● Extension commands
▪ Supplied by add-on modules (DLLs)
▪ Usually very advanced functionality
;runa@ay
● Display timing information of all threads
● Can be used for detecting hangs (livelocks)
;lock"
● Display information about all locked critical sections
;addre"" %addr&
● Display information (protection status, owner) of the given virtual
memory localtion
7/27/2019 11 Windows[1]
http://slidepdf.com/reader/full/11-windows1 26/28
Crash Dump Analysis – MFF UK – Debugging in Windows 26
WinDbg commands (9)
;analye
;analye >ang
● Various heuristics for analyzing the root cause of the previous
event or exception
●
Understands kernel internal structures and runs variousconsistency checks
● Stack analysis
● Heap analysis
● Corrupted code stream analysis (bad RAM)
●
Invalid call sequences (bad CPU)
7/27/2019 11 Windows[1]
http://slidepdf.com/reader/full/11-windows1 27/28
Crash Dump Analysis – MFF UK – Debugging in Windows 27
WinDbg pseudoregisters
● Contains various values useful for debugging
▪ Can be used in expressions or directly as command
arguments
Cra● Current stack return address
Cip
● Current instruction (EIP , RIP )
Cretreg● Current value of the return register (EAX , RAX )
Cc"p
● Current stack pointer (ESP , RSP )