109 lines
5.0 KiB
Plaintext
109 lines
5.0 KiB
Plaintext
Demonstrations of dirtop, the Linux eBPF/bcc version.
|
|
|
|
|
|
dirtop shows reads and writes by directory. For example:
|
|
|
|
# ./dirtop.py -d '/hdfs/uuid/*/yarn'
|
|
Tracing... Output every 1 secs. Hit Ctrl-C to end
|
|
|
|
14:28:12 loadavg: 25.00 22.85 21.22 31/2921 66450
|
|
|
|
READS WRITES R_Kb W_Kb PATH
|
|
1030 2852 8 147341 /hdfs/uuid/c11da291-28de-4a77-873e-44bb452d238b/yarn
|
|
3308 2459 10980 24893 /hdfs/uuid/bf829d08-1455-45b8-81fa-05c3303e8c45/yarn
|
|
2227 7165 6484 11157 /hdfs/uuid/76dc0b77-e2fd-4476-818f-2b5c3c452396/yarn
|
|
1985 9576 6431 6616 /hdfs/uuid/99c178d5-a209-4af2-8467-7382c7f03c1b/yarn
|
|
1986 398 6474 6486 /hdfs/uuid/7d512fe7-b20d-464c-a75a-dbf8b687ee1c/yarn
|
|
764 3685 5 7069 /hdfs/uuid/250b21c8-1714-45fe-8c08-d45d0271c6bd/yarn
|
|
432 1603 259 6402 /hdfs/uuid/4a833770-767e-43b3-b696-dc98901bce26/yarn
|
|
993 5856 320 129 /hdfs/uuid/b94cbf3f-76b1-4ced-9043-02d450b9887c/yarn
|
|
612 5645 4 249 /hdfs/uuid/8138a53b-b942-44d3-82df-51575f1a3901/yarn
|
|
818 21 6 166 /hdfs/uuid/fada8004-53ff-48df-9396-165d8e42925b/yarn
|
|
174 23 1 171 /hdfs/uuid/d04fccd8-bc72-4ed9-bda4-c5b6893f1405/yarn
|
|
376 6281 2 97 /hdfs/uuid/0cc3683f-4800-4c73-8075-8d77dc7cf116/yarn
|
|
370 4588 2 96 /hdfs/uuid/a78f846a-58c4-4d10-a9f5-42f16a6134a0/yarn
|
|
190 6420 1 86 /hdfs/uuid/2c6a7223-cb18-4916-a1b6-8cd02bda1d31/yarn
|
|
178 123 1 17 /hdfs/uuid/b3b2a2ed-f6c1-4641-86bf-2989dd932411/yarn
|
|
[...]
|
|
|
|
This shows various directories read and written when hadoop runs.
|
|
By default the output is sorted by the total read size in Kbytes (R_Kb).
|
|
Sorting order can be changed via -s option.
|
|
This is instrumenting at the VFS interface, so this is reads and writes that
|
|
may return entirely from the file system cache (page cache).
|
|
|
|
While not printed, the average read and write size can be calculated by
|
|
dividing R_Kb by READS, and the same for writes.
|
|
|
|
This script works by tracing the vfs_read() and vfs_write() functions using
|
|
kernel dynamic tracing, which instruments explicit read and write calls. If
|
|
files are read or written using another means (eg, via mmap()), then they
|
|
will not be visible using this tool.
|
|
|
|
This should be useful for file system workload characterization when analyzing
|
|
the performance of applications.
|
|
|
|
Note that tracing VFS level reads and writes can be a frequent activity, and
|
|
this tool can begin to cost measurable overhead at high I/O rates.
|
|
|
|
|
|
A -C option will stop clearing the screen, and -r with a number will restrict
|
|
the output to that many rows (20 by default). For example, not clearing
|
|
the screen and showing the top 5 only:
|
|
|
|
# ./dirtop -d '/hdfs/uuid/*/yarn' -Cr 5
|
|
Tracing... Output every 1 secs. Hit Ctrl-C to end
|
|
|
|
14:29:08 loadavg: 25.66 23.42 21.51 17/2850 67167
|
|
|
|
READS WRITES R_Kb W_Kb PATH
|
|
100 8429 0 48243 /hdfs/uuid/b94cbf3f-76b1-4ced-9043-02d450b9887c/yarn
|
|
2066 4091 8176 26457 /hdfs/uuid/d04fccd8-bc72-4ed9-bda4-c5b6893f1405/yarn
|
|
10 2043 0 8172 /hdfs/uuid/b3b2a2ed-f6c1-4641-86bf-2989dd932411/yarn
|
|
38 1368 0 2652 /hdfs/uuid/a78f846a-58c4-4d10-a9f5-42f16a6134a0/yarn
|
|
86 19 0 123 /hdfs/uuid/c11da291-28de-4a77-873e-44bb452d238b/yarn
|
|
|
|
14:29:09 loadavg: 25.66 23.42 21.51 15/2849 67170
|
|
|
|
READS WRITES R_Kb W_Kb PATH
|
|
1204 5619 4388 33767 /hdfs/uuid/b94cbf3f-76b1-4ced-9043-02d450b9887c/yarn
|
|
2208 3511 8744 22992 /hdfs/uuid/d04fccd8-bc72-4ed9-bda4-c5b6893f1405/yarn
|
|
62 4010 0 21181 /hdfs/uuid/8138a53b-b942-44d3-82df-51575f1a3901/yarn
|
|
22 2187 0 8748 /hdfs/uuid/b3b2a2ed-f6c1-4641-86bf-2989dd932411/yarn
|
|
74 1097 0 4388 /hdfs/uuid/4a833770-767e-43b3-b696-dc98901bce26/yarn
|
|
|
|
[..]
|
|
|
|
|
|
|
|
USAGE message:
|
|
|
|
# ./dirtop.py -h
|
|
usage: dirtop.py [-h] [-C] [-r MAXROWS] [-s {all,reads,writes,rbytes,wbytes}]
|
|
[-p PID] -d ROOTDIRS
|
|
[interval] [count]
|
|
|
|
File reads and writes by process
|
|
|
|
positional arguments:
|
|
interval output interval, in seconds
|
|
count number of outputs
|
|
|
|
optional arguments:
|
|
-h, --help show this help message and exit
|
|
-C, --noclear don't clear the screen
|
|
-r MAXROWS, --maxrows MAXROWS
|
|
maximum rows to print, default 20
|
|
-s {all,reads,writes,rbytes,wbytes}, --sort {all,reads,writes,rbytes,wbytes}
|
|
sort column, default all
|
|
-p PID, --pid PID trace this PID only
|
|
-d ROOTDIRS, --root-directories ROOTDIRS
|
|
select the directories to observe, separated by commas
|
|
|
|
examples:
|
|
./dirtop -d '/hdfs/uuid/*/yarn' # directory I/O top, 1 second refresh
|
|
./dirtop -d '/hdfs/uuid/*/yarn' -C # don't clear the screen
|
|
./dirtop -d '/hdfs/uuid/*/yarn' 5 # 5 second summaries
|
|
./dirtop -d '/hdfs/uuid/*/yarn' 5 10 # 5 second summaries, 10 times only
|
|
./dirtop -d '/hdfs/uuid/*/yarn,/hdfs/uuid/*/data' # Running dirtop on two set of directories
|