150 lines
5.0 KiB
Plaintext
150 lines
5.0 KiB
Plaintext
Demonstrations of drsnoop, the Linux eBPF/bcc version.
|
||
|
||
|
||
drsnoop traces the direct reclaim system-wide, and prints various details.
|
||
Example output:
|
||
|
||
# ./drsnoop
|
||
COMM PID LAT(ms) PAGES
|
||
summond 17678 0.19 143
|
||
summond 17669 0.55 313
|
||
summond 17669 0.15 145
|
||
summond 17669 0.27 237
|
||
summond 17669 0.48 111
|
||
summond 17669 0.16 75
|
||
head 17821 0.29 339
|
||
head 17825 0.17 109
|
||
summond 17669 0.14 73
|
||
summond 17496 104.84 40
|
||
summond 17678 0.32 167
|
||
summond 17678 0.14 106
|
||
summond 17678 0.16 67
|
||
summond 17678 0.29 267
|
||
summond 17678 0.27 69
|
||
summond 17678 0.32 46
|
||
base64 17816 0.16 85
|
||
summond 17678 0.43 283
|
||
summond 17678 0.14 182
|
||
head 17736 0.57 135
|
||
^C
|
||
|
||
While tracing, the processes alloc pages，due to insufficient memory available
|
||
in the system, direct reclaim events happened, which will increase the waiting
|
||
delay of the processes.
|
||
|
||
drsnoop can be useful for discovering when allocstall(/proc/vmstat) continues to increase,
|
||
whether it is caused by some critical processes or not.
|
||
|
||
The -p option can be used to filter on a PID, which is filtered in-kernel. Here
|
||
I've used it with -T to print timestamps:
|
||
|
||
# ./drsnoop -Tp 17491
|
||
TIME(s) COMM PID LAT(ms) PAGES
|
||
107.364115000 summond 17491 0.24 50
|
||
107.364550000 summond 17491 0.26 38
|
||
107.365266000 summond 17491 0.36 72
|
||
107.365753000 summond 17491 0.22 49
|
||
^C
|
||
|
||
This shows the summond process allocs pages, and direct reclaim events happening,
|
||
and the delays are not affected much.
|
||
|
||
The -U option include UID on output:
|
||
|
||
# ./drsnoop -U
|
||
UID COMM PID LAT(ms) PAGES
|
||
1000 summond 17678 0.32 46
|
||
0 base64 17816 0.16 85
|
||
1000 summond 17678 0.43 283
|
||
1000 summond 17678 0.14 182
|
||
0 head 17821 0.29 339
|
||
0 head 17825 0.17 109
|
||
^C
|
||
|
||
The -u option filtering UID:
|
||
|
||
# ./drsnoop -Uu 1000
|
||
UID COMM PID LAT(ms) PAGES
|
||
1000 summond 17678 0.19 143
|
||
1000 summond 17669 0.55 313
|
||
1000 summond 17669 0.15 145
|
||
1000 summond 17669 0.27 237
|
||
1000 summond 17669 0.48 111
|
||
1000 summond 17669 0.16 75
|
||
1000 summond 17669 0.14 73
|
||
1000 summond 17678 0.32 167
|
||
^C
|
||
|
||
A maximum tracing duration can be set with the -d option. For example, to trace
|
||
for 2 seconds:
|
||
|
||
# ./drsnoop -d 2
|
||
COMM PID LAT(ms) PAGES
|
||
head 21715 0.15 195
|
||
|
||
The -n option can be used to filter on process name using partial matches:
|
||
|
||
# ./drsnoop -n mond
|
||
COMM PID LAT(ms) PAGES
|
||
summond 10271 0.03 51
|
||
summond 10271 0.03 51
|
||
summond 10259 0.05 51
|
||
summond 10269 319.41 37
|
||
summond 10270 111.73 35
|
||
summond 10270 0.11 78
|
||
summond 10270 0.12 71
|
||
summond 10270 0.03 35
|
||
summond 10277 111.62 41
|
||
summond 10277 0.08 45
|
||
summond 10277 0.06 32
|
||
^C
|
||
|
||
This caught the 'summond' command because it partially matches 'mond' that's passed
|
||
to the '-n' option.
|
||
|
||
|
||
The -v option can be used to show system memory state (now only free mem) at
|
||
the beginning of direct reclaiming:
|
||
|
||
# ./drsnoop.py -v
|
||
COMM PID LAT(ms) PAGES FREE(KB)
|
||
base64 34924 0.23 151 86260
|
||
base64 34962 0.26 149 86260
|
||
head 34931 0.24 150 86260
|
||
base64 34902 0.19 148 86260
|
||
head 34963 0.19 151 86228
|
||
base64 34959 0.17 151 86228
|
||
head 34965 0.29 190 86228
|
||
base64 34957 0.24 152 86228
|
||
summond 34870 0.15 151 86080
|
||
summond 34870 0.12 115 86184
|
||
|
||
USAGE message:
|
||
|
||
# ./drsnoop -h
|
||
usage: drsnoop.py [-h] [-T] [-U] [-p PID] [-t TID] [-u UID] [-d DURATION]
|
||
[-n NAME]
|
||
|
||
Trace direct reclaim
|
||
|
||
optional arguments:
|
||
-h, --help show this help message and exit
|
||
-T, --timestamp include timestamp on output
|
||
-U, --print-uid print UID column
|
||
-p PID, --pid PID trace this PID only
|
||
-t TID, --tid TID trace this TID only
|
||
-u UID, --uid UID trace this UID only
|
||
-d DURATION, --duration DURATION
|
||
total duration of trace in seconds
|
||
-n NAME, --name NAME only print process names containing this name
|
||
|
||
examples:
|
||
./drsnoop # trace all direct reclaim
|
||
./drsnoop -T # include timestamps
|
||
./drsnoop -U # include UID
|
||
./drsnoop -p 181 # only trace PID 181
|
||
./drsnoop -t 123 # only trace TID 123
|
||
./drsnoop -u 1000 # only trace UID 1000
|
||
./drsnoop -d 10 # trace for 10 seconds only
|
||
./drsnoop -n main # only print process names containing "main"
|