91 lines
3.8 KiB
Plaintext
91 lines
3.8 KiB
Plaintext
Demonstrations of tcpretrans, the Linux eBPF/bcc version.
|
|
|
|
|
|
This tool traces the kernel TCP retransmit function to show details of these
|
|
retransmits. For example:
|
|
|
|
# ./tcpretrans
|
|
TIME PID IP LADDR:LPORT T> RADDR:RPORT STATE
|
|
01:55:05 0 4 10.153.223.157:22 R> 69.53.245.40:34619 ESTABLISHED
|
|
01:55:05 0 4 10.153.223.157:22 R> 69.53.245.40:34619 ESTABLISHED
|
|
01:55:17 0 4 10.153.223.157:22 R> 69.53.245.40:22957 ESTABLISHED
|
|
[...]
|
|
|
|
This output shows three TCP retransmits, the first two were for an IPv4
|
|
connection from 10.153.223.157 port 22 to 69.53.245.40 port 34619. The TCP
|
|
state was "ESTABLISHED" at the time of the retransmit. The on-CPU PID at the
|
|
time of the retransmit is printed, in this case 0 (the kernel, which will
|
|
be the case most of the time).
|
|
|
|
Retransmits are usually a sign of poor network health, and this tool is
|
|
useful for their investigation. Unlike using tcpdump, this tool has very
|
|
low overhead, as it only traces the retransmit function. It also prints
|
|
additional kernel details: the state of the TCP session at the time of the
|
|
retransmit.
|
|
|
|
|
|
A -l option will include TCP tail loss probe attempts:
|
|
|
|
# ./tcpretrans -l
|
|
TIME PID IP LADDR:LPORT T> RADDR:RPORT STATE
|
|
01:55:45 0 4 10.153.223.157:22 R> 69.53.245.40:51601 ESTABLISHED
|
|
01:55:46 0 4 10.153.223.157:22 R> 69.53.245.40:51601 ESTABLISHED
|
|
01:55:46 0 4 10.153.223.157:22 R> 69.53.245.40:51601 ESTABLISHED
|
|
01:55:53 0 4 10.153.223.157:22 L> 69.53.245.40:46444 ESTABLISHED
|
|
01:56:06 0 4 10.153.223.157:22 R> 69.53.245.40:46444 ESTABLISHED
|
|
01:56:06 0 4 10.153.223.157:22 R> 69.53.245.40:46444 ESTABLISHED
|
|
01:56:08 0 4 10.153.223.157:22 R> 69.53.245.40:46444 ESTABLISHED
|
|
01:56:08 0 4 10.153.223.157:22 R> 69.53.245.40:46444 ESTABLISHED
|
|
01:56:08 1938 4 10.153.223.157:22 R> 69.53.245.40:46444 ESTABLISHED
|
|
01:56:08 0 4 10.153.223.157:22 R> 69.53.245.40:46444 ESTABLISHED
|
|
01:56:08 0 4 10.153.223.157:22 R> 69.53.245.40:46444 ESTABLISHED
|
|
[...]
|
|
|
|
See the "L>" in the "T>" column. These are attempts: the kernel probably
|
|
sent a TLP, but in some cases it might not have been ultimately sent.
|
|
|
|
To spot heavily retransmitting flows quickly one can use the -c flag. It will
|
|
count occurring retransmits per flow.
|
|
|
|
# ./tcpretrans.py -c
|
|
Tracing retransmits ... Hit Ctrl-C to end
|
|
^C
|
|
LADDR:LPORT RADDR:RPORT RETRANSMITS
|
|
192.168.10.50:60366 <-> 172.217.21.194:443 700
|
|
192.168.10.50:666 <-> 172.213.11.195:443 345
|
|
192.168.10.50:366 <-> 172.212.22.194:443 211
|
|
[...]
|
|
|
|
This can ease to quickly isolate congested or otherwise awry network paths
|
|
responsible for clamping tcp performance.
|
|
|
|
TCP sequence numbers can be included via -s, except in count mode. These numbers
|
|
are useful for identifying specific retransmissions in large packet caputes.
|
|
Note, lossprobe -l output will display 0 for the sequence number for L type.
|
|
|
|
# ./tcpretrans.py -s
|
|
TIME PID IP LADDR:LPORT T> RADDR:RPORT STATE SEQ
|
|
18:03:46 0 4 192.168.10.50:41976 R> 172.217.21.194:443 SYN_SENT 2879306108
|
|
18:03:49 0 4 192.168.10.50:41976 R> 172.217.21.194:443 SYN_SENT 2879306108
|
|
|
|
USAGE message:
|
|
|
|
# ./tcpretrans -h
|
|
usage: tcpretrans.py [-h] [-s] [-l] [-c] [-4 | -6]
|
|
|
|
Trace TCP retransmits
|
|
|
|
optional arguments:
|
|
-h, --help show this help message and exit
|
|
-s, --sequence display TCP sequence numbers
|
|
-l, --lossprobe include tail loss probe attempts
|
|
-c, --count count occurred retransmits per flow
|
|
-4, --ipv4 trace IPv4 family only
|
|
-6, --ipv6 trace IPv6 family only
|
|
|
|
examples:
|
|
./tcpretrans # trace TCP retransmits
|
|
./tcpretrans -l # include TLP attempts
|
|
./tcpretrans -4 # trace IPv4 family only
|
|
./tcpretrans -6 # trace IPv6 family only
|