Sun Microsystems, Inc.
spacerspacer
spacer www.sun.com docs.sun.com |
spacer
black dot
 
 
  Previous   Contents   Next 
   
 
Chapter 7

Kernel Debugging Modules

This chapter describes the debugger modules, dcmds, and walkers provided to debug the Solaris kernel. Each kernel debugger module is named after the corresponding Solaris kernel module, so that it will be loaded automatically by MDB. The facilities described here reflect the current Solaris kernel implementation and are subject to change in the future; writing shell scripts that depend on the output of these commands is not recommended. In general, the kernel debugging facilities described in this chapter are meaningful only in the context of the corresponding kernel subsystem implementation. See "Related Books and Papers" for a list of references that provide more information about the Solaris kernel implementation.


Note - This guide reflects the Solaris 9 operating environment implementation; these modules, dcmds, and walkers may not be relevant, correct, or applicable to past or future releases, since they reflect the current kernel implementation. They do not define a permanent public interface of any kind. All of the information provided about modules, dcmds, walkers, and their output formats and arguments is subject to change in future releases of the Solaris operating environment.


Generic Kernel Debugging Support (genunix)

Kernel Memory Allocator

This section discusses the dcmds and walkers used to debug problems identified by the Solaris kernel memory allocator and to examine memory and memory usage. The dcmds and walkers described here are discussed in more detail in Chapter 8, Debugging With the Kernel Memory Allocator.

dcmds

thread ::allocdby

Given the address of a kernel thread, print a list of memory allocations it has performed in reverse chronological order.

bufctl ::bufctl [-a address] [-c caller] [-e earliest] [-l latest] [-t thread]

Print a summary of the bufctl information for the specified bufctl address. If one or more options are present, the bufctl information is printed only if it matches the criteria defined by the option arguments; in this way, the dcmd can be used as a filter for input from a pipeline. The -a option indicates that the bufctl's corresponding buffer address must equal the specified address. The -c option indicates that a program counter value from the specified caller must be present in the bufctl's saved stack trace. The -e option indicates that the bufctl's timestamp must be greater than or equal to the specified earliest timestamp. The -l option indicates that the bufctl's timestamp must be less than or equal to the specified latest timestamp. The -t option indicates that the bufctl's thread pointer must be equal to the specified thread address.

[ address ] ::findleaks [-v]

The ::findleaks dcmd provides powerful and efficient detection of memory leaks in kernel crash dumps where the full set of kmem debug features has been enabled. The first execution of ::findleaks processes the dump for memory leaks (this can take a few minutes), then coalesces the leaks by the allocation stack trace. The findleaks report shows a bufctl address and the topmost stack frame for each memory leak that was identified.

If the -v option is specified, the dcmd prints more verbose messages as it executes. If an explicit address is specified prior to the dcmd, the report is filtered and only leaks whose allocation stack traces contain the specified function address are displayed.

thread ::freedby

Given the address of a kernel thread, print a list of memory frees it has performed, in reverse chronological order.

value ::kgrep

Search the kernel address space for pointer-aligned addresses that contain the specified pointer-sized value. The list of addresses that contain matching values is then printed. Unlike MDB's built-in search operators, ::kgrep searches every segment of the kernel's address space and searches across discontiguous segment boundaries. On large kernels, ::kgrep can take a considerable amount of time to execute.

::kmalog [ slab | fail ]

Display events in a kernel memory allocator transaction log. Events are displayed in time-reverse order, with the most recent event displayed first. For each event, ::kmalog displays the time relative to the most recent event in T-minus notation (for example, T-0.000151879), the bufctl, the buffer address, the kmem cache name, and the stack trace at the time of the event. Without arguments, ::kmalog displays the kmem transaction log, which is present only if KMF_AUDIT is set in kmem_flags. ::kmalog fail displays the allocation failure log, which is always present; this can be useful in debugging drivers that don't cope with allocation failure correctly. ::kmalog slab displays the slab create log, which is always present. ::kmalog slab can be useful when searching for memory leaks.

::kmastat

Display the list of kernel memory allocator caches and virtual memory arenas, along with corresponding statistics.

::kmausers [-ef] [cache ...]

Print information about the medium and large users of the kernel memory allocator that have current memory allocations. The output consists of one entry for each unique stack trace specifying the total amount of memory and number of allocations that was made with that stack trace. This dcmd requires that the KMF_AUDIT flag is set in kmem_flags.

If one or more cache names (for example, kmem_alloc_256) are specified, the scan of memory usage is restricted to those caches. By default all caches are included. If the -e option is used, the small users of the allocator are included. The small users are allocations that total less than 1024 bytes of memory or for which there are less than 10 allocations with the same stack trace. If the -f option is used, the stack traces are printed for each individual allocation.

[ address ] ::kmem_cache

Format and display the kmem_cache structure stored at the specified address, or the complete set of active kmem_cache structures.

::kmem_log

Display the complete set of kmem transaction logs, sorted in reverse chronological order. This dcmd uses a more concise tabular output format than ::kmalog.

[ address ] ::kmem_verify

Verify the integrity of the kmem_cache structure stored at the specified address, or the complete set of active kmem_cache structures. If an explicit cache address is specified, the dcmd displays more verbose information regarding errors; otherwise, a summary report is displayed. The ::kmem_verify dcmd is discussed in more detail in "Kernel Memory Caches".

[ address ] ::vmem

Format and display the vmem structure stored at the specified address, or the complete set of active vmem structures. This structure is defined in <sys/vmem_impl.h>.

address ::vmem_seg

Format and display the vmem_seg structure stored at the specified address. This structure is defined in <sys/vmem_impl.h>.

address ::whatis [-abv]

Report information about the specified address. In particular, ::whatis will attempt to determine if the address is a pointer to a kmem-managed buffer or another type of special memory region, such as a thread stack, and report its findings. If the -a option is present, the dcmd reports all matches instead of just the first match to its queries. If the -b option is present, the dcmd also attempts to determine if the address is referred to by a known kmem bufctl. If the -v option is present, the dcmd reports its progress as it searches various kernel data structures.

Walkers

allocdby

Given the address of a kthread_t structure as a starting point, iterate over the set of bufctl structures corresponding to memory allocations performed by this kernel thread.

bufctl

Given the address of a kmem_cache_t structure as a starting point, iterate over the set of allocated bufctls associated with this cache.

freectl

Given the address of a kmem_cache_t structure as a starting point, iterate over the set of free bufctls associated with this cache.

freedby

Given the address of a kthread_t structure as a starting point, iterate over the set of bufctl structures corresponding to memory deallocations performed by this kernel thread.

freemem

Given the address of a kmem_cache_t structure as a starting point, iterate over the set of free buffers associated with this cache.

kmem

Given the address of a kmem_cache_t structure as a starting point, iterate over the set of allocated buffers associated with this cache.

kmem_cache

Iterate over the active set of kmem_cache_t structures. This structure is defined in <sys/kmem_impl.h>.

kmem_cpu_cache

Given the address of a kmem_cache_t structure as a starting point, iterate over the per-CPU kmem_cpu_cache_t structures associated with this cache. This structure is defined in <sys/kmem_impl.h>.

kmem_slab

Given the address of a kmem_cache_t structure as a starting point, iterate over the set of associated kmem_slab_t structures. This structure is defined in <sys/kmem_impl.h>.

kmem_log

Iterate over the set of bufctls stored in the kmem allocator transaction log.

leak

Given the address of a bufctl structure, iterate over the set of bufctl structures corresponding to leaked memory buffers with similar allocation stack traces. The ::findleaks dcmd must be applied to locate memory leaks before the leak walker can be used

leakbuf

Given the address of a bufctl structure, iterate over the set of buffer addresses corresponding to leaked memory buffers with similar allocation stack traces. The ::findleaks dcmd must be applied to locate memory leaks before the leakbuf walker can be used.

File Systems

The MDB file systems debugging support includes a built-in facility to convert vnode pointers to the corresponding file system path name. This conversion is performed using the Directory Name Lookup Cache (DNLC); because the cache does not hold all active vnodes, some vnodes might not be able to be converted to path names and "??" is displayed instead of a name.

dcmds

::fsinfo

Display a table of mounted file systems, including the vfs_t address, ops vector, and mount point of each file system.

::lminfo

Display a table of vnodes with active network locks registered with the lock manager. The pathname corresponding to each vnode is shown.

address ::vnode2path [-v]

Display the pathname corresponding to the given vnode address. If the -v option is specified, the dcmd prints a more verbose display, including the vnode pointer of each intermediate path component.

Walkers

buf

Iterate over the set of active block I/O transfer structures (buf_t structures). The buf structure is defined in <sys/buf.h> and is described in more detail in buf(9S).

Virtual Memory

This section describes the debugging support for the kernel virtual memory subsystem.

dcmds

address ::addr2smap [offset]

Print the smap structure address that corresponds to the given address in the kernel's segmap address space segment.

as ::as2proc

Display the proc_t address for the process corresponding to the as_t address as.

[ address ] ::memlist [-aiv]

Display the specified memlist structure or one of the well-known memlist structures. If no memlist address and options are present or if the -i option is present, the memlist representing physically installed memory is displayed. If the -a option is present, the memlist representing available physical memory is displayed. If the -v option is present, the memlist representing available virtual memory is displayed.

::memstat

Display a system-wide memory usage summary. The amount and percentage of system memory consumed by different classes of pages (kernel, anonymous memory, executables and libraries, page cache, and free lists) are displayed, along with the total amount of system memory.

[ address ] ::page

Display the properties of the specified page_t. If no page_t address is specified, the dcmd displays the properties of all system pages.

seg ::seg

Format and display the specified address space segment (seg_t address).

[ address ] ::swapinfo

Display information on all active swapinfo structures or about the specified struct swapinfo. The vnode, filename, and statistics for each structure are displayed.

vnode ::vnode2smap [offset]

Print the smap structure address that corresponds to the given vnode_t address and offset.

Walkers

anon

Given the address of an anon_map structure as a starting point, iterate over the set of related anon structures. The anon map implementation is defined in <vm/anon.h>.

memlist

Iterate over the spans of the specified memlist structure. This walker can be used in conjuction with the ::memlist dcmd to display each span.

page

Iterate over all system page structures. If an explicit address is specified for the walk, this is taken to be the address of a vnode and the walker iterates over only those pages associated with the vnode.

seg

Given the address of an as_t structure as a starting point, iterate over the set of address space segments (seg structures) associated with the specified address space. The seg structure is defined in <vm/seg.h>.

swapinfo

Iterate over the list of active swapinfo structures. This walker may be used in conjunction with the ::swapinfo dcmd.

 
 
 
  Previous   Contents   Next