Disclaimer : All the postings on this site are my own and don’t necessarily represent IBM’s positions, strategies or opinions.


Some time back, I had written this blog AIX ‘bosdebug’ to debug Kernel extensions on the usefulness of pretty printing a data-structure in AIX Kernel Debugger “KDB” by providing it the symbol information using the ‘bosdebug‘ command. That pretty much serves the purpose on a live system during kernel_extension / device_driver development. That though, is not very useful while working on system dumps where you need to use “command kdb“. Also for several reasons even during development it would be preferable to observe data-structures from “command kdb” rather than by halting a system and using KDB(System Kernel Debugger).

This article covers the details on enabling pretty printing of symbols in ‘command kdb‘. For that, you would need to use “KDBSYM environment variable” and i stumbled across this information while reading through the help of ‘pr command’ in ‘command kdb‘.
The help message of ‘pr‘ command pretty much cover it all :

(0)> pr -?
print <type> <address>
    Formatted dump of memory at <address> as if it were of type <type>.
    <type> must be a type recognized by the debug object file kdb
    draws its symbols from.  This file can either be generated
    automatically when crash is run via -i flags, or by setting
    the KDBSYM environment variable to be the name of a file
    containing debug symbols with the structure types you want to
    print. “address” can be an address or a kernel global variable.

    For example, to print the struct vnode at 12345,
    kdb -i /usr/include/sys/vnode.h
    (0)> print vnode 012345

    To create a symbols file ahead of time for faster invocation
    $echo ‘#include <sys/vnode.h>’ > symbols.c
    echo ‘main() { ; }’ >> symbols.c
    $ cc -g -o symbols symbols.c -qdbxextra /* for 32 bit kernel */
    $ cc -g -q64 -o symbols symbols.c -qdbxextra /* for 64 bit kernel */
    $ KDBSYM=/bin/pwd/symbols ; export KDBSYM
    $ kdb dump unix
    (0)> print vnode 012345

    Kernel global variable can be used instead of absolute address.
    For example,
    (0)> print Simple_lock suspending_q_lock


Using a system dump of a sample kernel extension as an example i’ll demonstrate this functionality.
Below is the ‘stat‘ command output from this sample simulated system crash :


This stack above is because of an “Illegal Trap Instruction Interrupt in Kernel” because of a failed assert() in function : read_contents(); seen on top of the above stack.

From the dummy kernel extension code of read_contents(), i know that the input to this function was a pointer of type ‘struct info‘ and the crash was caused by a failed assert ( as seen in code below ) :

struct prim
    unsigned short type;
    unsigned int   id;
    unsigned long  priority;

struct info
    struct prim primary;
    unsigned int flags;
    unsigned int state;
    unsigned long size;
    int error;
    char buffer[1024];

int read_contents(struct info *infop)
    assert(infop->primary.type == 0x99);
    .. ..

To investigate further, i need to look at the values of members in ‘struct info‘ and deduce the possible code flow.
The stock method is to use the  “display double word data”  command dd and dump the contents of the structure in terminal and mark out the structure member values based on individual data-types sizes and considering space of structure padding. phew !


This method of inferring structure member values though might be sufficient in some cases, is a very tedious and many times error-prone for complex structure types.
This is where the wonderful command in kdb called “pr comes in handy 🙂

pr       print             print a formatted structure at an address

As was mentioned in the ‘pr‘ command usage message, it requires KDBSYM environment variable to point to the name of the object file which contains the debug symbols with the structure types you want to do a formatted print.

In my case, I set KDBSYM to point the debug symbol object file i had generated :
export KDBSYM=/home/sangeek/sKE/symbols
and initiated command kdb on the system dump file.

And this is what i see using the ‘pr’ command in kdb :

The above formatted structure output makes things pretty easy to view and understand.
I now save a lot of time debugging kernel extension issues and really spend my time on where it should be spent 😉

A very useful functionality, lacking adoption; attributing some of the blame to lack of sufficient documentation.

In short, command kdb, pr command and KDBSYM rocks \o/


What do you think ?

Set your Twitter account name in your settings to use the TwitterBar Section.
%d bloggers like this: