226 lines
9.5 KiB
Plaintext
226 lines
9.5 KiB
Plaintext
What is it ?
|
|
----------
|
|
ioblame is a IO profiler. It gives a detailed listing of the list of
|
|
files that running pids are doing IO to (with the aggregate amount of
|
|
read/write to each file). The goal is that using this tool, app
|
|
developers can identify IO heavy paths, and tune them, then
|
|
iteratively run this tool again to measure improvements/IO reduction.
|
|
|
|
ioblame is implemented as a collection of kernel filesystem
|
|
tracepoints, and a script.
|
|
|
|
What does the output from ioblame look like ?
|
|
-------------------------------------------
|
|
ioblame gives 2 different views.
|
|
|
|
File view is the default.
|
|
|
|
1) File view : For each file, it gives a list of pids that are doing
|
|
IO on that file (it also gives the amount of IO each pid does and
|
|
the aggregate amount of IO done to the file by all pids).
|
|
|
|
File: /app/Chrome/Chrome.apk
|
|
Chrome_ChildIOT Reads: 96 KB
|
|
Chrome_DBThread Reads: 652 KB
|
|
Chrome_FileUser Reads: 72 KB
|
|
Chrome_InProcGp Reads: 732 KB
|
|
Chrome_IOThread Reads: 1396 KB
|
|
Compositor Reads: 652 KB
|
|
CookieMonsterBa Reads: 284 KB
|
|
CrRendererMain Reads: 17012 KB
|
|
dboxed_process0 Reads: 512 KB
|
|
JavaBridge Reads: 128 KB
|
|
ogle.android.gm Reads: 14996 KB
|
|
WorkerPool/8556 Reads: 408 KB
|
|
Total Reads: 36940 KB i_size: 71354 KB
|
|
File: /app/Gmail2Light/Gmail2Light.apk
|
|
AsyncTask_#2 Reads: 24 KB
|
|
Chrome_IOThread Reads: 68 KB
|
|
ogle.android.gm Reads: 1440 KB
|
|
Thread-6 Reads: 8 KB
|
|
Total Reads: 1540 KB i_size: 9736 KB
|
|
|
|
And ditto for writes.
|
|
|
|
To enable PID view, use -v
|
|
|
|
PID: CrRendererMain
|
|
/app/Chrome/Chrome.apk Reads: 17012 KB i_size: 71354 KB
|
|
/etc/fonts.xml Reads: 24 KB i_size: 22 KB
|
|
/fonts/CarroisGothicSC-Regular.ttf Reads: 40 KB i_size: 39 KB
|
|
/fonts/ComingSoon.ttf Reads: 60 KB i_size: 57 KB
|
|
/fonts/CutiveMono.ttf Reads: 68 KB i_size: 67 KB
|
|
/fonts/DancingScript-Bold.ttf Reads: 116 KB i_size: 112 KB
|
|
/fonts/DancingScript-Regular.ttf Reads: 116 KB i_size: 113 KB
|
|
/fonts/DroidSansMono.ttf Reads: 108 KB i_size: 105 KB
|
|
/fonts/NotoColorEmoji.ttf Reads: 204 KB i_size: 7108 KB
|
|
/fonts/NotoNaskhArabic-Bold.ttf Reads: 116 KB i_size: 113 KB
|
|
,,,
|
|
/fonts/NotoSerif-BoldItalic.ttf Reads: 248 KB i_size: 256 KB
|
|
/fonts/NotoSerif-Bold.ttf Reads: 244 KB i_size: 242 KB
|
|
/fonts/NotoSerif-Italic.ttf Reads: 244 KB i_size: 243 KB
|
|
/fonts/NotoSerif-Regular.ttf Reads: 244 KB i_size: 240 KB
|
|
/fonts/Roboto-BlackItalic.ttf Reads: 256 KB i_size: 322 KB
|
|
/fonts/Roboto-Black.ttf Reads: 256 KB i_size: 299 KB
|
|
/fonts/Roboto-BoldItalic.ttf Reads: 256 KB i_size: 324 KB
|
|
/fonts/RobotoCondensed-BoldItalic.ttf Reads: 256 KB i_size: 322 KB
|
|
/fonts/RobotoCondensed-Bold.ttf Reads: 256 KB i_size: 296 KB
|
|
/fonts/RobotoCondensed-Italic.ttf Reads: 256 KB i_size: 321 KB
|
|
/fonts/RobotoCondensed-LightItalic.ttf Reads: 256 KB i_size: 324 KB
|
|
/fonts/RobotoCondensed-Light.ttf Reads: 256 KB i_size: 295 KB
|
|
/fonts/RobotoCondensed-MediumItalic.ttf Reads: 256 KB i_size: 322 KB
|
|
/fonts/RobotoCondensed-Medium.ttf Reads: 256 KB i_size: 296 KB
|
|
/fonts/Roboto-LightItalic.ttf Reads: 256 KB i_size: 324 KB
|
|
/fonts/Roboto-MediumItalic.ttf Reads: 256 KB i_size: 323 KB
|
|
/fonts/Roboto-Regular.ttf Reads: 88 KB i_size: 298 KB
|
|
/fonts/Roboto-ThinItalic.ttf Reads: 256 KB i_size: 321 KB
|
|
/fonts/Roboto-Thin.ttf Reads: 256 KB i_size: 300 KB
|
|
/lib/libft2.so Reads: 56 KB i_size: 479 KB
|
|
/lib/libicuuc.so Reads: 88 KB i_size: 1182 KB
|
|
Total Reads: 32760 KB
|
|
|
|
And ditto for writes.
|
|
|
|
For the -p, writepages option, ioblame does not (and cannot) give you
|
|
the app/pid that did the original write (since the write most likely
|
|
happens from a flush thread or VM reclaim). In this mode, we only get
|
|
the pathname of the file and the amount of data written out.
|
|
|
|
Finally, it reports the total amount of file data IO done by pids and
|
|
the total IO done to the block device. So we can look at IO overheads
|
|
(eg block level prefetching, filesystem metadata overhead etc).
|
|
|
|
For detailed examples, look at ioblame-gmail-launch.example and
|
|
ioblame-gmail-run.example.
|
|
|
|
How do I run ioblame ?
|
|
--------------------
|
|
ioblame -r [ I am only interested in reads ]
|
|
ioblame -w [ I am only interested in writes ]
|
|
ioblame -p [ I am only interested in writepage(s) - mmap'ed writes ]
|
|
ioblame -r -w -p [ I am only interested in reads, writes and writepages ]
|
|
and finally, -v adds the PID view
|
|
|
|
1) The most common way of running ioblame is to start ioblame, then go
|
|
off and launch the app(s) of interest, do interesting stuff with the
|
|
app(s), and when finished with the app, hit ^C ONCE on
|
|
ioblame. Hitting ^C once will cause ioblame to catch the signal, break
|
|
out of the sleep, terminate its tracing, and process the ftraces and
|
|
dump out IO stats for the app.
|
|
|
|
example :
|
|
|
|
srmohan0.mtv.corp.google.com> sh ioblame.sh -r -w -p
|
|
Found aosp_gobo Device
|
|
OK to kill sleep when test is done
|
|
^Csignal INT received, killing streaming trace capture
|
|
|
|
(at this point, run your apps, when done type ^C ONCE) and output will
|
|
appear on stdout.
|
|
|
|
2) Sometimes, we want to do IO profiling in a different way (instead of
|
|
running something and then hitting ^C on ioblame). For instance, we
|
|
might want to do IO profiling related to the launch of an App. This
|
|
requires you to modify the script slightly, as described in the next
|
|
paragraph.
|
|
|
|
If you want to do IO profiling related to App launch, you need to modify
|
|
2 functions - prep_to_do_something and do_something. Again examples of
|
|
changes to these functions clarifies things better :
|
|
|
|
prep_to_do_something() {
|
|
adb shell "am force-stop com.google.android.gm" # This line I added for example
|
|
# stops the running app (gmail)
|
|
adb shell 'echo 3 > /proc/sys/vm/drop_caches'
|
|
sleep 1
|
|
}
|
|
|
|
do_something() {
|
|
# Arrange things so that the first SIGINT will kill the
|
|
# child process (sleep), but will return to the parent.
|
|
# trap 'catch_sigint' INT
|
|
# echo "OK to kill sleep when test is done"
|
|
# sleep 1d
|
|
# For the purpose of this example, I commented out the above 3 lines that
|
|
# make ioblame sleep forever after catching the SIGINT and instead it launches
|
|
# gmail, waiting for launch to complete. When launch completes, ioblame will
|
|
# stop tracing, and process the traces
|
|
adb shell "am start -W -n com.google.android.gm/.ConversationListActivityGmail"
|
|
}
|
|
|
|
Limitations :
|
|
-----------
|
|
The ioblame kernel tracepoints currently only cover the getpage(s)
|
|
path (so all filesystem reads) and filesystem write() syscalls. There
|
|
are no tracepoints in the putpage(s) paths. This is because when
|
|
putpage(s) is called, we most often cannot tell which thread/pid has
|
|
dirtied the page that is being written out, because the majority of
|
|
putpage(s) happen from the flush threads or from the pageout
|
|
threads. The upshot of this is the mmap'ed writes are not reported by
|
|
ioblame. In practice, while mmap'ed reads are very common, mmap'ed
|
|
writes are infrequent.
|
|
|
|
Kernel Patches Required :
|
|
-----------------------
|
|
ioblame needs the kernel to be patched with these 3 kernel patches.
|
|
Without these patches, ioblame won't work at all.
|
|
|
|
commit ae2f6765db98e2fcb99082e336dd8b24e7644620
|
|
Author: Mohan Srinivasan <srmohan@google.com>
|
|
Date: Fri Mar 10 16:08:30 2017 -0800
|
|
|
|
ANDROID: Replace spaces by '_' for some android filesystem tracepoints.
|
|
|
|
Andoid files frequently have spaces in them, as do cmdline strings.
|
|
Replace these spaces with '_', so tools that parse these tracepoints
|
|
don't get terribly confused.
|
|
|
|
Change-Id: I1cbbedf5c803aa6a58d9b8b7836e9125683c49d1
|
|
Signed-off-by: Mohan Srinivasan <srmohan@google.com>
|
|
(cherry picked from commit 5035d5f0933758dd515327d038e5bef7e40dbaa7)
|
|
|
|
commit a2a04ea83d2960867324fa059ba1eedc2fc7784e (HEAD -> android-4.4)
|
|
Author: Mohan Srinivasan <srmohan@google.com>
|
|
Date: Fri Feb 3 15:48:03 2017 -0800
|
|
|
|
ANDROID: Refactor fs readpage/write tracepoints.
|
|
|
|
Refactor the fs readpage/write tracepoints to move the
|
|
inode->path lookup outside the tracepoint code, and pass a pointer
|
|
to the path into the tracepoint code instead. This is necessary
|
|
because the tracepoint code runs non-preemptible. Thanks to
|
|
Trilok Soni for catching this in 4.4.
|
|
|
|
Change-Id: I7486c5947918d155a30c61d6b9cd5027cf8fbe15
|
|
Signed-off-by: Mohan Srinivasan <srmohan@google.com>
|
|
|
|
commit 32cbbe59538d2dd0b77822cc27ce32ca487c467d
|
|
Author: Mohan Srinivasan <srmohan@google.com>
|
|
Date: Mon Sep 19 17:33:50 2016 -0700
|
|
|
|
ANDROID: fs: FS tracepoints to track IO.
|
|
|
|
Adds tracepoints in ext4/f2fs/mpage to track readpages/buffered
|
|
write()s. This allows us to track files that are being read/written
|
|
to PIDs.
|
|
|
|
Change-Id: I26bd36f933108927d6903da04d8cb42fd9c3ef3d
|
|
Signed-off-by: Mohan Srinivasan <srmohan@google.com>
|
|
|
|
The -w (writepages) option requires this additional patch and
|
|
currently only with f2fs.
|
|
|
|
commit c60bc59c6af4fbdeaf7bbeaebee6b55d9e488324 (HEAD ->
|
|
android-mtk-gobo-3.18)
|
|
Author: Mohan Srinivasan <srmohan@google.com>
|
|
Date: Fri Sep 8 13:53:01 2017 -0700
|
|
|
|
Tracepoints in f2fs data writepage(s).
|
|
|
|
Tracepoints f2fs writepage(s). This is experimental (for now).
|
|
Allowing ioblame to track <pathname, amount of data written>
|
|
for files.
|
|
|
|
Signed-off-by: Mohan Srinivasan <srmohan@google.com>
|
|
Change-Id: I4c76c6f442e0a2c5872225f8113935f9f368cc64
|