Post on 13-Oct-2015
description
transcript
5/22/2018 07-memory
1/35
Crash Dump Analysis
Memory management issues
Jakub Jerm
Martin Dck
5/22/2018 07-memory
2/35
Crash Dump Analysis MFF UK Memory management issues 2
Overview
Slab allocator
Kernel memory allocator
Internal debugging features
Support in mdb
Memory management errors
Library libumem
5/22/2018 07-memory
3/35
Crash Dump Analysis MFF UK Memory management issues 3
References
Jeff Bonwick: The Slab Allocator: An Object-
Caching Kernel Memory Allocator
http://www.usenix.org/publications/library/proceedings/bos94/bonwick.html
Jeff Bonwick, Jonathan Adams: Magazines andVmem: Extending the Slab Allocator to Many
CPUs and Arbitrary Resources
http://www.usenix.org/event/usenix01/bonwick.html
http://www.usenix.org/publications/library/proceedings/bos94/bonwick.htmlhttp://www.usenix.org/event/usenix01/bonwick.htmlhttp://www.usenix.org/event/usenix01/bonwick.htmlhttp://www.usenix.org/publications/library/proceedings/bos94/bonwick.html5/22/2018 07-memory
4/35
Crash Dump Analysis MFF UK Memory management issues 4
References (2)
Man pages
libumem(3LIB), umem_debug(3MALLOC)
Identifying Memory Management Bugs ...
http://developers.sun.com/solaris/articles/libumem_library.html
http://developers.sun.com/solaris/articles/libumem_library.htmlhttp://developers.sun.com/solaris/articles/libumem_library.html5/22/2018 07-memory
5/35
Crash Dump Analysis MFF UK Memory management issues 5
Slab allocator
Object caching allocator
API for dealing with objectswhich are frequently
allocated / deallocated
Avoid full initialization after each allocation
Also for non-caching allocations
Kernel version of malloc/free
kmem_alloc(size) allocate a buffer kmem_free(bufp, size) release it
5/22/2018 07-memory
6/35
Crash Dump Analysis MFF UK Memory management issues 6
Slab allocator (2)
Cache
Contains objectsof same type/size
Several caches for different purposes
Object caches: process_cache, thread_cache, ...
Used for types such as proc_t, kthread_t, ...
Non-caching allocations use caches internally:
kmem_alloc_8, kmem_alloc_16, ...
Suffix _8 or _16 is maximum allocation size
Best fit: kmem_alloc(14)allocates a bufferfrom thekmem_alloc_16cache (2 bytes wasted)
Cache consists of slabs
5/22/2018 07-memory
7/35
Crash Dump Analysis MFF UK Memory management issues 7
Slab allocator (3)
Slab
Allocation unit for a cache(internally)
One or more pages
Buffer (buf)
Raw memory area
Non-cached data
Object
Bufctl
Buffer's metadata
kmem_slab
kmem_
bufctl
kmem_
bufctl
kmem_
bufctl
bufbuf buf unused
slab
Metadata
about slab
5/22/2018 07-memory
8/35
Crash Dump Analysis MFF UK Memory management issues 8
Slab allocator (4)
Small objects
Size
5/22/2018 07-memory
9/35
Crash Dump Analysis MFF UK Memory management issues 9
Buffer layout
Buftag is appended to each buffer depending
on value of kmem_flagsvariable
Redzone pattern
Metadata (bufctl back pointer, stats)
Redzone byte marks end of allocated user area
It's used in case of non-caching allocations
dataredzone
byteunused
redzone
pattern
general buffer
metadata
buftag
5/22/2018 07-memory
10/35
Crash Dump Analysis MFF UK Memory management issues 10
Vmem allocator
Backing store for Slab allocator
General-purpose large allocations
One or multiple pages
5/22/2018 07-memory
11/35
Crash Dump Analysis MFF UK Memory management issues 11
Using kmem_flags
Debugging features
Kernel variable
Add set kmem_flags = 0x0f to /etc/system
0x0f is used by DEBUG kernels
Bits for particular features
See defines in /usr/include/sys/kmem_impl.h
defi!e "#$_%&' 0x00000001 *+ tra!sactio! auditi!g +*
defi!e "#$_'%'-$ 0x0000000. *+ deadbeef checki!g +*
defi!e "#$_/'2 0x00000004 *+ redzo!e checki!g +*
defi!e "#$_322 0x00000008 *+ freed5buffer co!te!t loggi!g +*
defi!e "#$_2#%%2 0x000000.0 *+ disable per5cpu magazi!es +*
defi!e "#$_$/7% 0x00000040 *+ put all bufs before u!mapped pages +*
5/22/2018 07-memory
12/35
Crash Dump Analysis MFF UK Memory management issues 12
Debugging patterns
Memory patterns are human-readable strings
Free pattern deadbeef
Uninitialized pattern baddcafe
Redzone pattern feedface
Defined in /usr/include/sys/kmem_impl.h
defi!e "##_$/_9%/2 0xdeadbeefdeadbeef&
defi!e "##_&22%'_9%/2 0xbaddcafebaddcafe&defi!e "##_/'2_9%/2 0xfeedfacefeedface&
defi!e "##_/'2_-: 0xbb
5/22/2018 07-memory
13/35
Crash Dump Analysis MFF UK Memory management issues 13
Memory management errors
Multiple free
Buffer overrun
Use of uninitialized memory Use after free
Memory leaks
Memory allocator specific
Freeing buffer to wrong cache
Freeing invalid pointer
5/22/2018 07-memory
14/35
Crash Dump Analysis MFF UK Memory management issues 14
Multiple free
Impact
May corrupt heap
May free someone else's bufferker!el memor; allocator10 cache< kmem_alloc_.
pre?ious tra!sactio! o! buffer ffffff008ada088 slab=ffffff008ace88A8 cache< kmem_alloc_.
kmem_cache_free_debugB1.f
kmem_cache_freeBC
kmem_freeB1f>
barB.A
fooBe
taskD_threadB16
thread_startB8
pa!icEcpu0F*thread=ffffff0001.fac808 cache< kmem_alloc_8
pre?ious tra!sactio! o! buffer ffffff00ba0fa660
taskD_threadB16
thread_startB8
pa!icEcpu0F*thread=ffffff00014.fc80c80