251 lines
12 KiB
Plaintext
251 lines
12 KiB
Plaintext
Demonstrations of kvm exit reasons, the Linux eBPF/bcc version.
|
|
|
|
|
|
Considering virtual machines' frequent exits can cause performance problems,
|
|
this tool aims to locate the frequent exited reasons and then find solutions
|
|
to reduce or even avoid the exit, by displaying the detail exit reasons and
|
|
the counts of each vm exit for all vms running on one physical machine.
|
|
|
|
|
|
Features of this tool
|
|
=====================
|
|
|
|
- Although there is a patch: [KVM: x86: add full vm-exit reason debug entries]
|
|
(https://patchwork.kernel.org/project/kvm/patch/1555939499-30854-1-git-send-email-pizhenwei@bytedance.com/)
|
|
trying to fill more vm-exit reason debug entries, just as the comments said,
|
|
the code allocates lots of memory that may never be consumed, misses some
|
|
arch-specific kvm causes, and can not do kernel aggregation. Instead bcc, as
|
|
a user space tool, can implement all these functions more easily and flexibly.
|
|
- The bcc python logic could provide nice kernel aggregation and custom output,
|
|
like collpasing all tids for one pid (e.i. one vm's qemu process id) with exit
|
|
reasons sorted in descending order. For more information, see the following
|
|
#USAGE message.
|
|
- The bpf in-kernel percpu_array and percpu_cache further improves performance.
|
|
For more information, see the following #Help to understand.
|
|
|
|
|
|
Limited
|
|
=======
|
|
|
|
In view of the hardware-assisted virtualization technology of
|
|
different architectures, currently we only adapt on vmx in intel.
|
|
And the amd feature is on the road..
|
|
|
|
|
|
Example output:
|
|
===============
|
|
|
|
# ./kvmexit.py
|
|
Display kvm exit reasons and statistics for all threads... Hit Ctrl-C to end.
|
|
PID TID KVM_EXIT_REASON COUNT
|
|
^C1273551 1273568 EXIT_REASON_HLT 12
|
|
1273551 1273568 EXIT_REASON_MSR_WRITE 6
|
|
1274253 1274261 EXIT_REASON_EXTERNAL_INTERRUPT 1
|
|
1274253 1274261 EXIT_REASON_HLT 12
|
|
1274253 1274261 EXIT_REASON_MSR_WRITE 4
|
|
|
|
# ./kvmexit.py 6
|
|
Display kvm exit reasons and statistics for all threads after sleeping 6 secs.
|
|
PID TID KVM_EXIT_REASON COUNT
|
|
1273903 1273922 EXIT_REASON_EXTERNAL_INTERRUPT 175
|
|
1273903 1273922 EXIT_REASON_CPUID 10
|
|
1273903 1273922 EXIT_REASON_HLT 6043
|
|
1273903 1273922 EXIT_REASON_IO_INSTRUCTION 24
|
|
1273903 1273922 EXIT_REASON_MSR_WRITE 15025
|
|
1273903 1273922 EXIT_REASON_PAUSE_INSTRUCTION 11
|
|
1273903 1273922 EXIT_REASON_EOI_INDUCED 12
|
|
1273903 1273922 EXIT_REASON_EPT_VIOLATION 6
|
|
1273903 1273922 EXIT_REASON_EPT_MISCONFIG 380
|
|
1273903 1273922 EXIT_REASON_PREEMPTION_TIMER 194
|
|
1273551 1273568 EXIT_REASON_EXTERNAL_INTERRUPT 18
|
|
1273551 1273568 EXIT_REASON_HLT 989
|
|
1273551 1273568 EXIT_REASON_IO_INSTRUCTION 10
|
|
1273551 1273568 EXIT_REASON_MSR_WRITE 2205
|
|
1273551 1273568 EXIT_REASON_PAUSE_INSTRUCTION 1
|
|
1273551 1273568 EXIT_REASON_EOI_INDUCED 5
|
|
1273551 1273568 EXIT_REASON_EPT_MISCONFIG 61
|
|
1273551 1273568 EXIT_REASON_PREEMPTION_TIMER 14
|
|
|
|
# ./kvmexit.py -p 1273795 5
|
|
Display kvm exit reasons and statistics for PID 1273795 after sleeping 5 secs.
|
|
KVM_EXIT_REASON COUNT
|
|
MSR_WRITE 13467
|
|
HLT 5060
|
|
PREEMPTION_TIMER 345
|
|
EPT_MISCONFIG 264
|
|
EXTERNAL_INTERRUPT 169
|
|
EPT_VIOLATION 18
|
|
PAUSE_INSTRUCTION 6
|
|
IO_INSTRUCTION 4
|
|
EOI_INDUCED 2
|
|
|
|
# ./kvmexit.py -p 1273795 5 -a
|
|
Display kvm exit reasons and statistics for PID 1273795 and its all threads after sleeping 5 secs.
|
|
TID KVM_EXIT_REASON COUNT
|
|
1273819 EXTERNAL_INTERRUPT 64
|
|
1273819 HLT 2802
|
|
1273819 IO_INSTRUCTION 4
|
|
1273819 MSR_WRITE 7196
|
|
1273819 PAUSE_INSTRUCTION 2
|
|
1273819 EOI_INDUCED 2
|
|
1273819 EPT_VIOLATION 6
|
|
1273819 EPT_MISCONFIG 162
|
|
1273819 PREEMPTION_TIMER 194
|
|
1273820 EXTERNAL_INTERRUPT 78
|
|
1273820 HLT 2054
|
|
1273820 MSR_WRITE 5199
|
|
1273820 EPT_VIOLATION 2
|
|
1273820 EPT_MISCONFIG 77
|
|
1273820 PREEMPTION_TIMER 102
|
|
|
|
# ./kvmexit.py -p 1273795 -v 0
|
|
Display kvm exit reasons and statistics for PID 1273795 VCPU 0... Hit Ctrl-C to end.
|
|
KVM_EXIT_REASON COUNT
|
|
^CMSR_WRITE 2076
|
|
HLT 795
|
|
PREEMPTION_TIMER 86
|
|
EXTERNAL_INTERRUPT 20
|
|
EPT_MISCONFIG 10
|
|
PAUSE_INSTRUCTION 2
|
|
IO_INSTRUCTION 2
|
|
EPT_VIOLATION 1
|
|
EOI_INDUCED 1
|
|
|
|
# ./kvmexit.py -p 1273795 -v 0 4
|
|
Display kvm exit reasons and statistics for PID 1273795 VCPU 0 after sleeping 4 secs.
|
|
KVM_EXIT_REASON COUNT
|
|
MSR_WRITE 4726
|
|
HLT 1827
|
|
PREEMPTION_TIMER 78
|
|
EPT_MISCONFIG 67
|
|
EXTERNAL_INTERRUPT 28
|
|
IO_INSTRUCTION 4
|
|
EOI_INDUCED 2
|
|
PAUSE_INSTRUCTION 2
|
|
|
|
# ./kvmexit.py -p 1273795 -v 4 4
|
|
Traceback (most recent call last):
|
|
File "tools/kvmexit.py", line 306, in <module>
|
|
raise Exception("There's no v%s for PID %d." % (tgt_vcpu, args.pid))
|
|
Exception: There's no vCPU 4 for PID 1273795.
|
|
|
|
# ./kvmexit.py -t 1273819 10
|
|
Display kvm exit reasons and statistics for TID 1273819 after sleeping 10 secs.
|
|
KVM_EXIT_REASON COUNT
|
|
MSR_WRITE 13318
|
|
HLT 5274
|
|
EPT_MISCONFIG 263
|
|
PREEMPTION_TIMER 171
|
|
EXTERNAL_INTERRUPT 109
|
|
IO_INSTRUCTION 8
|
|
PAUSE_INSTRUCTION 5
|
|
EOI_INDUCED 4
|
|
EPT_VIOLATION 2
|
|
|
|
# ./kvmexit.py -T '1273820,1273819'
|
|
Display kvm exit reasons and statistics for TIDS ['1273820', '1273819']... Hit Ctrl-C to end.
|
|
TIDS KVM_EXIT_REASON COUNT
|
|
^C1273819 EXTERNAL_INTERRUPT 300
|
|
1273819 HLT 13718
|
|
1273819 IO_INSTRUCTION 26
|
|
1273819 MSR_WRITE 37457
|
|
1273819 PAUSE_INSTRUCTION 13
|
|
1273819 EOI_INDUCED 13
|
|
1273819 EPT_VIOLATION 53
|
|
1273819 EPT_MISCONFIG 654
|
|
1273819 PREEMPTION_TIMER 958
|
|
1273820 EXTERNAL_INTERRUPT 212
|
|
1273820 HLT 9002
|
|
1273820 MSR_WRITE 25495
|
|
1273820 PAUSE_INSTRUCTION 2
|
|
1273820 EPT_VIOLATION 64
|
|
1273820 EPT_MISCONFIG 396
|
|
1273820 PREEMPTION_TIMER 268
|
|
|
|
|
|
Help to understand
|
|
==================
|
|
|
|
We use a PERCPU_ARRAY: pcpuArrayA and a percpu_hash: hashA to collaboratively
|
|
store each kvm exit reason and its count. The reason is there exists a rule when
|
|
one vcpu exits and re-enters, it tends to continue to run on the same physical
|
|
cpu (pcpu as follows) as the last cycle, which is also called 'cache hit'. Thus
|
|
we turn to use a PERCPU_ARRAY to record the 'cache hit' situation to speed
|
|
things up; and for other cases, then use a percpu_hash.
|
|
|
|
BTW, we originally use a common hash to do this, with a u64(exit_reason)
|
|
key and a struct exit_info {tgid_pid, exit_reason} value. But due to
|
|
the big lock in bpf_hash, each updating is quite performance consuming.
|
|
|
|
Now imagine here is a pid_tgidA (vcpu A) exits and is going to run on
|
|
pcpuArrayA, the BPF code flow is as follows:
|
|
|
|
pid_tgidA keeps running on the same pcpu
|
|
// \\
|
|
// \\
|
|
// Y N \\
|
|
// \\
|
|
a. cache_hit b. cache_miss
|
|
(cacheA's pid_tgid matches pid_tgidA) ||
|
|
| ||
|
|
| ||
|
|
"increase percpu exit_ct and return" ||
|
|
[*Note*] ||
|
|
pid_tgidA ever been exited on pcpuArrayA?
|
|
// \\
|
|
// \\
|
|
// \\
|
|
// Y N \\
|
|
// \\
|
|
b.a load_last_hashA b.b initialize_hashA_with_zero
|
|
\ /
|
|
\ /
|
|
\ /
|
|
"increase percpu exit_ct"
|
|
||
|
|
||
|
|
is another pid_tgid been running on pcpuArrayA?
|
|
// \\
|
|
// Y N \\
|
|
// \\
|
|
b.*.a save_theLastHit_hashB do_nothing
|
|
\\ //
|
|
\\ //
|
|
\\ //
|
|
b.* save_to_pcpuArrayA
|
|
|
|
|
|
[*Note*] we do not update the table in above "a.", in case the vcpu hit the same
|
|
pcpu again when exits next time, instead we only update until this pcpu is not
|
|
hitted by the same tgidpid(vcpu) again, which is in "b.*.a" and "b.*".
|
|
|
|
|
|
USAGE message:
|
|
==============
|
|
|
|
# ./kvmexit.py -h
|
|
usage: kvmexit.py [-h] [-p PID [-v VCPU | -a] ] [-t TID | -T 'TID1,TID2'] [duration]
|
|
|
|
Display kvm_exit_reason and its statistics at a timed interval
|
|
|
|
optional arguments:
|
|
-h, --help show this help message and exit
|
|
-p PID, --pid PID display process with this PID only, collpase all tids with exit reasons sorted in descending order
|
|
-v VCPU, --v VCPU display this VCPU only for this PID
|
|
-a, --alltids display all TIDS for this PID
|
|
-t TID, --tid TID display thread with this TID only with exit reasons sorted in descending order
|
|
-T 'TID1,TID2', --tids 'TID1,TID2'
|
|
display threads for a union like {395490, 395491}
|
|
duration duration of display, after sleeping several seconds
|
|
|
|
examples:
|
|
./kvmexit # Display kvm_exit_reason and its statistics in real-time until Ctrl-C
|
|
./kvmexit 5 # Display in real-time after sleeping 5s
|
|
./kvmexit -p 3195281 # Collpase all tids for pid 3195281 with exit reasons sorted in descending order
|
|
./kvmexit -p 3195281 20 # Collpase all tids for pid 3195281 with exit reasons sorted in descending order, and display after sleeping 20s
|
|
./kvmexit -p 3195281 -v 0 # Display only vcpu0 for pid 3195281, descending sort by default
|
|
./kvmexit -p 3195281 -a # Display all tids for pid 3195281
|
|
./kvmexit -t 395490 # Display only for tid 395490 with exit reasons sorted in descending order
|
|
./kvmexit -t 395490 20 # Display only for tid 395490 with exit reasons sorted in descending order after sleeping 20s
|
|
./kvmexit -T '395490,395491' # Display for a union like {395490, 395491}
|