Disclaimer : All the postings on this site are my own and don’t necessarily represent IBM’s positions, strategies or opinions.
 
Most applications use POSIX threads, usually referred to as Pthreads for creating and managing multi-threaded processes. Threads are managed using the Pthread library.

A multithreaded application could have multiple user-threads and each thread would be having a unique Pthread-ID associated with it. In AIX each user-thread is mapped to one kernel thread, which means there would be same number of kernel threads of a process.

In this post I’ll highlight some of the AIX user-space and KDB commands which would be very handy for debugging these kind of multi-threaded applications.

Command to get the number of kernel threads of a process :

# ps -o thcount -p <pid>

example :
# ps -o thcount -p 8061078
THCNT
10               <<<<<<<<< This example process has 10 kernel threads

List all the kernel threads of a process :

# ps -m -o THREAD -p <pid>       <<<<<<<< "-m" option displays the associated kernel threads.  "-o" flag with  the THREAD field specifier is used to display extra thread-related columns.

example :
# ps -m -o THREAD -p 8061078
USER      PID     PPID       TID ST  CP PRI SC    WCHAN        F     TT BND COMMAND
root  8061078  3342536         - A  120  60 44        * 10240011      -   - /usr/bin/multiThApp
-        -        -  14221513    S    0  60  1 f1000f0a1000d940  8410400      -   - -
-        -        -  17301579    R  120 120  1        -        0      -   - -
-        -        -  17563743    Z    0  60  1        -   c00001      -   - -
-        -        -  17694791    S    0  60  1 f1000a00e08b3fb0   810400      -   - -
-        -        -  18874439    S    0  60  1 f1000f0a10012040  8430400      -   - -
-        -        -  20119743    Z    0  60  1        -   c00001      -   - -
-        -        -  20250875    Z    0  60  1        -   c00001      -   - -
-        -        -  21954723    S    0  60  1 f1000e0002d3da08   410400      -   - -
-        -        -  22020267    S    0  60  1 f1000f0a10015040  8410400      -   - -
-        -        -  22085801    S    0  60  1        -   418400      -   - -

The above output lists the Kernel threads of the process along with their thread-ids(TID) in decimal, State(ST) of the threads designated by S(Suspended), Z(Zombie), R(Runnable), Priority(PRI) of the thread, Suspend Count(SC) of the thread etc.

Map Pthread ID with Kernel thread-id (using procstack) :

A very important aspect of debugging a multi-threaded application is the ability to map the Pthread-id of a thread to it’s Kernel thread. Pthread-id is the thread-id seen by the process and usually got by pthread_self() function. Procstack command comes in handy for this :

# procstack <pid>

example :
# procstack 8061078
8061078: /usr/bin/multiThApp
---------- tid# 14221513 (pthread ID:   7711) ---------- <<<<< 14221513 is the kernel thread id and 7711 is the pthread_id
0x090000000050a584  _event_sleep(??, ??, ??, ??, ??, ??) + 0x5a4
0x090000000050b018  _event_wait(??, ??) + 0x2b8
0x09000000005192c4  _cond_wait_local(??, ??, ??) + 0x4e4
0x090000000051989c  _cond_wait(??, ??, ??) + 0xbc
0x090000000051a508  pthread_cond_wait(??, ??) + 0x1a8
0x000000010001f1e8  mth_count_thread(0x11004aa30) + 0x268
0x09000000004f7d30  _pthread_body(??) + 0xf0
---------- tid# 17301579 (pthread ID:   8998) ----------
0x090000000002b364  write(??, ??, ??) + 0x284
0x08000000002b47b8  pondWrite(??, ??, ??) + 0xb8
0x08000000002b4c60  pondCntl(??, ??, ??) + 0x240
0x000000010003ce40  lookoutEvent(0x110052fb0, 0xd0000000d, 0x1100363f0) + 0x920
0x000000010003dfdc  mth_lookout_thread(0x110052fb0) + 0x1c
0x09000000004f7d30  _pthread_body(??) + 0xf0
.. .. .. .. .. .. <o/p trunctated> .. .. .. .. .. .. ..

Procstack o/p provided above gives the Pthread-id and kernel thread-id in decimal. It also provides the complete point-in-time user-space stack of the threads of the process-id provided as input. Of course, you can get similar information using DBX, more on that in a separate article <TODO>.

Stack of the thread in Kernel space :

In some cases it might be relevant to understand the kernel stack corresponding to a particular erring thread. That can be easily got on “command KDB” using the kernel thread-id.

>> 1. Get’s thread-slot from the kernel thread_id <<

usage: ttid [[-x | -d] <tid>]

example :

(0)> ttid -d 14221513
SLOT NAME     STATE    TID PRI   RQ CPUID  CL  WCHAN

pvthread+042100 1057 vio_daem SLEEP 4210083 03C  512         0  pvthread+042140   <<<<<  thread slot for the thread is pvthread+042100

NAME……………. vio_daemon
WTYPE…………… WEVENT
……………..tid :0000000004210083  ……tsleep :0000000004210083
.. .. .. .. .. .. <o/p trunctated> .. .. .. .. .. .. ..

>> 2. Use the thread-slot from the previous result to get kernel stack of the thread <<

(0)> stack pvthread+042100
pvthread+042100 STACK:
[000AF110]ep_block_thread+000290 (00000000000F1658 [??])
[000AD97C]_thread_tsleep+00073C (??, ??, ??, ??, ??)
[00003888]mfspurr_sc_flih01+0000E4 ()
[90000000050A580]_event_sleep+0005A0 (??, ??, ??, ??, ??, ??)
[90000000050B018]_event_wait+0002B8 (??, ??)
[9000000005192C4]_cond_wait_local+0004E4 (??, ??, ??)
[90000000051989C]_cond_wait+0000BC (??, ??, ??)
[90000000051A508]pthread_cond_wait+0001A8 (??, ??)
[10001F1E8]mth_count_thread(0x11004aa30) +000268 (000000011004AFC8)
[9000000004F7D30]_pthread_body+0000F0 (??)

The above stack from “command KDB” gives the complete stack showing the kernel/Kernel Extension calls made and would provide useful pointers to understand the complete code-flow.

Hope this article was useful.

 

What do you think ?

Set your Twitter account name in your settings to use the TwitterBar Section.
%d bloggers like this: