用于Linux性能监视,联网等的BCC动态跟踪工具

分享于 

21分钟阅读

Linux

  繁體

BCC (BPF编译器集合)是一组强大的工具和示例文件,用于创建资源丰富的内核跟踪和操作程序,它使用扩展的BPF (Berkeley数据包筛选器),最初称为eBPF,它是Linux 3.15中的新特性之一。

如何在Linux系统中安装BCC

请记住,BCC使用Linux内核版本4.1或更高版本中添加的功能,并且作为一项要求,应使用下面设置的标志编译内核:

CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
# [optional, for tc filters]
CONFIG_NET_CLS_BPF=m
# [optional, for tc actions]
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_JIT=y
CONFIG_HAVE_BPF_JIT=y
# [optional, for kprobes]
CONFIG_BPF_EVENTS=y

要检查你的内核标志,请查看文件/proc/config.gz或运行以下示例中的命令:

[helplib] ~ $ grep CONFIG_BPF= /boot/config-`uname -r`
CONFIG_BPF=y[helplib] ~ $ grep CONFIG_BPF_SYSCALL= /boot/config-`uname -r`
CONFIG_BPF_SYSCALL=y[helplib] ~ $ grep CONFIG_NET_CLS_BPF= /boot/config-`uname -r`
CONFIG_NET_CLS_BPF=m[helplib] ~ $ grep CONFIG_NET_ACT_BPF= /boot/config-`uname -r`
CONFIG_NET_ACT_BPF=m[helplib] ~ $ grep CONFIG_BPF_JIT= /boot/config-`uname -r`
CONFIG_BPF_JIT=y[helplib] ~ $ grep CONFIG_HAVE_BPF_JIT= /boot/config-`uname -r`
CONFIG_HAVE_BPF_JIT=y[helplib] ~ $ grep CONFIG_BPF_EVENTS= /boot/config-`uname -r`
CONFIG_BPF_EVENTS=y

验证内核标志之后,是时候在Linux系统中安装BCC工具了。

在ubuntu 16.04上

只为Ubuntu 16.04创建夜间包,但是安装说明非常简单,不需要内核升级或从源代码编译。

$ echo"deb [trusted=yes] https://repo.iovisor.org/apt/xenial xenial-nightly main" | sudo tee /etc/apt/sources.list.d/iovisor.list
$ sudo apt-get update
$ sudo apt-get install bcc-tools

在Ubuntu 14.04上

从安装4.3+Linux内核开始,http://kernel.ubuntu.com/~kernel-ppa/mainline

举个例子,用下面的内容编写一个小的shell script"bcc-install.sh"

注意:将PREFIX值更新为最新日期,并浏览提供的PREFIX URL中的文件以获取实际的REL值,然后将它们替换为Shell脚本。

#!/bin/bash
VER=4.5.1-040501
PREFIX=http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.5.1-wily/
REL=201604121331
wget ${PREFIX}/linux-headers-${VER}-generic_${VER}.${REL}_amd64.deb
wget ${PREFIX}/linux-headers-${VER}_${VER}.${REL}_all.deb
wget ${PREFIX}/linux-image-${VER}-generic_${VER}.${REL}_amd64.deb
sudo dpkg -i linux-*${VER}.${REL}*.deb

保存文件然后退出,使它可执行,然后运行它,如下所示:

$ chmod +x bcc-install.sh
$ sh bcc-install.sh

然后,重新启动系统。

$ reboot

接下来,运行下面的命令安装签名的BCC软件包:

$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys D4284CDD
$ echo"deb https://repo.iovisor.org/apt trusty main" | sudo tee /etc/apt/sources.list.d/iovisor.list
$ sudo apt-get update
$ sudo apt-get install binutils bcc bcc-tools libbcc-examples python-bcc

在Fedora 24-23

如果你的系统版本低于所需版本,请从安装4.2+内核,以下是如何执行此操作的示例:

$ sudo dnf config-manager --add-repo=http://alt.fedoraproject.org/pub/alt/rawhide-kernel-nodebug/fedora-rawhide-kernel-nodebug.repo
$ sudo dnf update
$ reboot

然后,添加BBC tools repository,更新系统并执行下一系列命令来安装工具:

$ echo -e '[iovisor]nbaseurl=https://repo.iovisor.org/yum/nightly/f23/$basearchnenabled=1ngpgcheck=0' | sudo tee /etc/yum.repos.d/iovisor.repo
$ sudo dnf update
$ sudo dnf install bcc-tools

在Arch Linux-AUR上

你应该首先将内核升级到至少4.3.1-1版本,然后使用Arch包管理器(如pacaur 、 yaourt 、 cower等)安装下面的包。

bcc bcc-tools python-bcc python2-bcc

在Linux系统中,如何使用BCC工具

所有BCC工具都安装在/usr/share/bcc/tools目录下,然而,你也可以在/tools下从BCC Github存储库运行它们,它们以.py扩展名结束。

$ ls /usr/share/bcc/tools 
argdist capable filetop offwaketime stackcount vfscount
bashreadline cpudist funccount old stacksnoop vfsstat
biolatency dcsnoop funclatency oomkill statsnoop wakeuptime
biosnoop dcstat gethostlatency opensnoop syncsnoop xfsdist
biotop doc hardirqs pidpersec tcpaccept xfsslower
bitesize execsnoop killsnoop profile tcpconnect zfsdist
btrfsdist ext4dist mdflush runqlat tcpconnlat zfsslower
btrfsslower ext4slower memleak softirqs tcpretrans
cachestat filelife mysqld_qslower solisten tplist
cachetop fileslower offcputime sslsniff trace

跟踪open()系统调用

首先使用open()跟踪所有opensnoop系统调用,这使我们能够通过识别数据文件,配置文件和更多内容来告诉我们各种应用程序的工作方式:

$ cd /usr/share/bcc/tools 
$ sudo ./opensnoop
PID COMM FD ERR PATH
1 systemd 35 0 /proc/self/mountinfo
2797 udisksd 13 0 /proc/self/mountinfo
1 systemd 35 0 /sys/devices/pci0000:00/0000:00:0d.0/ata3/host2/target2:0:0/2:0:0:0/block/sda/sda1/uevent
1 systemd 35 0 /run/udev/data/b8:1
1 systemd -1 2 /etc/systemd/system/sys-kernel-debug-tracing.mount
1 systemd -1 2 /run/systemd/system/sys-kernel-debug-tracing.mount
1 systemd -1 2 /run/systemd/generator/sys-kernel-debug-tracing.mount
1 systemd -1 2 /usr/local/lib/systemd/system/sys-kernel-debug-tracing.mount
2247 systemd 15 0 /proc/self/mountinfo
1 systemd -1 2 /lib/systemd/system/sys-kernel-debug-tracing.mount
1 systemd -1 2 /usr/lib/systemd/system/sys-kernel-debug-tracing.mount
1 systemd -1 2 /run/systemd/generator.late/sys-kernel-debug-tracing.mount
1 systemd -1 2 /etc/systemd/system/sys-kernel-debug-tracing.mount.wants
1 systemd -1 2 /etc/systemd/system/sys-kernel-debug-tracing.mount.requires
1 systemd -1 2 /run/systemd/system/sys-kernel-debug-tracing.mount.wants
1 systemd -1 2 /run/systemd/system/sys-kernel-debug-tracing.mount.requires
1 systemd -1 2 /run/systemd/generator/sys-kernel-debug-tracing.mount.wants
1 systemd -1 2 /run/systemd/generator/sys-kernel-debug-tracing.mount.requires
1 systemd -1 2 /usr/local/lib/systemd/system/sys-kernel-debug-tracing.mount.wants
1 systemd -1 2 /usr/local/lib/systemd/system/sys-kernel-debug-tracing.mount.requires
1 systemd -1 2 /lib/systemd/system/sys-kernel-debug-tracing.mount.wants
1 systemd -1 2 /lib/systemd/system/sys-kernel-debug-tracing.mount.requires
1 systemd -1 2 /usr/lib/systemd/system/sys-kernel-debug-tracing.mount.wants
1 systemd -1 2 /usr/lib/systemd/system/sys-kernel-debug-tracing.mount.requires
1 systemd -1 2 /run/systemd/generator.late/sys-kernel-debug-tracing.mount.wants
1 systemd -1 2 /run/systemd/generator.late/sys-kernel-debug-tracing.mount.requires
1 systemd -1 2 /etc/systemd/system/sys-kernel-debug-tracing.mount.d
1 systemd -1 2 /run/systemd/system/sys-kernel-debug-tracing.mount.d
1 systemd -1 2 /run/systemd/generator/sys-kernel-debug-tracing.mount.d
....

块设备I/O延迟摘要

在本例中,它显示了使用bioloteccy的磁盘I/O延迟的汇总分布,执行命令后,等待几分钟并按ctrl c结束它并查看输出。

$ sudo ./biolatecncy
Tracing block device I/O... Hit Ctrl-C to end.
^C
 usecs : count distribution
 0 -> 1 : 0 | |
 2 -> 3 : 0 | |
 4 -> 7 : 0 | |
 8 -> 15 : 0 | |
 16 -> 31 : 0 | |
 32 -> 63 : 0 | |
 64 -> 127 : 0 | |
 128 -> 255 : 3 |****************************************|
 256 -> 511 : 3 |****************************************|
 512 -> 1023 : 1 |************* |

通过exec()Syscalls跟踪新进程

在本节中,使用execsnoop工具跟踪执行中的新进程,每当一个进程被fork()exec() syscalls派生时,它就会显示在输出中,但是,并非所有进程都被捕获。

$ sudo ./execsnoop
PCOMM PID PPID RET ARGS
gnome-screensho 14882 14881 0 /usr/bin/gnome-screenshot --gapplication-service
systemd-hostnam 14892 1 0 /lib/systemd/systemd-hostnamed
nautilus 14897 2767 -2 /home/tecmint/bin/net usershare info
nautilus 14897 2767 -2 /home/tecmint/.local/bin/net usershare info
nautilus 14897 2767 -2 /usr/local/sbin/net usershare info
nautilus 14897 2767 -2 /usr/local/bin/net usershare info
nautilus 14897 2767 -2 /usr/sbin/net usershare info
nautilus 14897 2767 -2 /usr/bin/net usershare info
nautilus 14897 2767 -2 /sbin/net usershare info
nautilus 14897 2767 -2 /bin/net usershare info
nautilus 14897 2767 -2 /usr/games/net usershare info
nautilus 14897 2767 -2 /usr/local/games/net usershare info
nautilus 14897 2767 -2 /snap/bin/net usershare info
compiz 14899 14898 -2 /home/tecmint/bin/libreoffice --calc
compiz 14899 14898 -2 /home/tecmint/.local/bin/libreoffice --calc
compiz 14899 14898 -2 /usr/local/sbin/libreoffice --calc
compiz 14899 14898 -2 /usr/local/bin/libreoffice --calc
compiz 14899 14898 -2 /usr/sbin/libreoffice --calc
libreoffice 14899 2252 0 /usr/bin/libreoffice --calc
dirname 14902 14899 0 /usr/bin/dirname /usr/bin/libreoffice
basename 14903 14899 0 /usr/bin/basename /usr/bin/libreoffice
...

跟踪慢速ext4操作

使用ext4 slower来跟踪ext4文件系统的常见操作,这些操作的速度小于10ms,以帮助我们通过文件系统独立地识别速度较慢的磁盘I/O。

它只输出那些超过阈值的操作:

$ sudo ./execslower
Tracing ext4 operations slower than 10 ms
TIME COMM PID T BYTES OFF_KB LAT(ms) FILENAME
11:59:13 upstart 2252 W 48 1 10.76 dbus.log
11:59:13 gnome-screensh 14993 R 144 0 10.96 settings.ini
11:59:13 gnome-screensh 14993 R 28 0 16.02 gtk.css
11:59:13 gnome-screensh 14993 R 3389 0 18.32 gtk-main.css
11:59:25 rs:main Q:Reg 1826 W 156 60 31.85 syslog
11:59:25 pool 15002 R 208 0 14.98 .xsession-errors
11:59:25 pool 15002 R 644 0 12.28 .ICEauthority
11:59:25 pool 15002 R 220 0 13.38 .bash_logout
11:59:27 dconf-service 2599 S 0 0 22.75 user.BHDKOY
11:59:33 compiz 2548 R 4096 0 19.03 firefox.desktop
11:59:34 compiz 15008 R 128 0 27.52 firefox.sh
11:59:34 firefox 15008 R 128 0 36.48 firefox
11:59:34 zeitgeist-daem 2988 S 0 0 62.23 activity.sqlite-wal
11:59:34 zeitgeist-fts 2996 R 8192 40 15.67 postlist.DB
11:59:34 firefox 15008 R 140 0 18.05 dependentlibs.list
11:59:34 zeitgeist-fts 2996 S 0 0 25.96 position.tmp
11:59:34 firefox 15008 R 4096 0 10.67 libplc4.so
11:59:34 zeitgeist-fts 2996 S 0 0 11.29 termlist.tmp
...

有PID和延迟的跟踪块设备I/O

接下来,深入到每秒打印每个磁盘I/O的行,其中包括使用biosnoop的进程ID 、扇区、字节、延迟等详细信息:

$ sudo ./biosnoop
TIME(s) COMM PID DISK T SECTOR BYTES LAT(ms)
0.000000000 ? 0 R -1 8 0.26
2.047897000 ? 0 R -1 8 0.21
3.280028000 kworker/u4:0 14871 sda W 30552896 4096 0.24
3.280271000 jbd2/sda1-8 545 sda W 29757720 12288 0.40
3.298318000 jbd2/sda1-8 545 sda W 29757744 4096 0.14
4.096084000 ? 0 R -1 8 0.27
6.143977000 ? 0 R -1 8 0.27
8.192006000 ? 0 R -1 8 0.26
8.303938000 kworker/u4:2 15084 sda W 12586584 4096 0.14
8.303965000 kworker/u4:2 15084 sda W 25174736 4096 0.14
10.239961000 ? 0 R -1 8 0.26
12.292057000 ? 0 R -1 8 0.20
14.335990000 ? 0 R -1 8 0.26
16.383798000 ? 0 R -1 8 0.17
...

跟踪页缓存命中/未命中比率

然后,我们继续使用cachestat每秒从系统缓存显示一行汇总统计信息,这可以通过指出低缓存命中率和高未命中率来启用系统调优操作:

$ sudo ./cachestat
 HITS MISSES DIRTIES READ_HIT% WRITE_HIT% BUFFERS_MB CACHED_MB
 0 0 0 0.0% 0.0% 19 544
 4 4 2 25.0% 25.0% 19 544
 1321 33 4 97.3% 2.3% 19 545
 7476 0 2 100.0% 0.0% 19 545
 6228 15 2 99.7% 0.2% 19 545
 0 0 0 0.0% 0.0% 19 545
 7391 253 108 95.3% 2.7% 19 545
 33608 5382 28 86.1% 13.8% 19 567
 25098 37 36 99.7% 0.0% 19 566
 17624 239 416 96.3% 0.5% 19 520
...

跟踪TCP活动连接

每秒使用tcpconnect监视TCP连接,它输出包括源地址和目标地址以及端口号,工具对于跟踪意外的TCP连接非常有用,因此有助于我们识别应用程序配置或攻击者中的低效。

$ sudo ./tcpconnect
PID COMM IP SADDR DADDR DPORT
15272 Socket Threa 4 10.0.2.15 91.189.89.240 80 
15272 Socket Threa 4 10.0.2.15 216.58.199.142 443 
15272 Socket Threa 4 10.0.2.15 216.58.199.142 80 
15272 Socket Threa 4 10.0.2.15 216.58.199.174 443 
15272 Socket Threa 4 10.0.2.15 54.200.62.216 443 
15272 Socket Threa 4 10.0.2.15 54.200.62.216 443 
15272 Socket Threa 4 10.0.2.15 117.18.237.29 80 
15272 Socket Threa 4 10.0.2.15 216.58.199.142 80 
15272 Socket Threa 4 10.0.2.15 216.58.199.131 80 
15272 Socket Threa 4 10.0.2.15 216.58.199.131 443 
15272 Socket Threa 4 10.0.2.15 52.222.135.52 443 
15272 Socket Threa 4 10.0.2.15 216.58.199.131 443 
15272 Socket Threa 4 10.0.2.15 54.200.62.216 443 
15272 Socket Threa 4 10.0.2.15 54.200.62.216 443 
15272 Socket Threa 4 10.0.2.15 216.58.199.132 443 
15272 Socket Threa 4 10.0.2.15 216.58.199.131 443 
15272 Socket Threa 4 10.0.2.15 216.58.199.142 443 
15272 Socket Threa 4 10.0.2.15 54.69.17.198 443 
15272 Socket Threa 4 10.0.2.15 54.69.17.198 443 
...

上面的所有工具还可以与各种选项一起使用,以启用给定工具的帮助页面,使用-h选项,例如:

$ sudo ./tcpconnect -h
usage: tcpconnect [-h] [-t] [-p PID] [-P PORT]
Trace TCP connects
optional arguments:
 -h, --help show this help message and exit
 -t, --timestamp include timestamp on output
 -p PID, --pid PID trace this PID only
 -P PORT, --port PORT comma-separated list of destination ports to trace.
examples:
 ./tcpconnect # trace all TCP connect()s
 ./tcpconnect -t # include timestamps
 ./tcpconnect -p 181 # only trace PID 181
 ./tcpconnect -P 80 # only trace port 80
 ./tcpconnect -P 80,81 # only trace port 80 and 81

跟踪失败exec ()s系统调用

要跟踪fail exec ()的syscalls,请使用-x选项,如下所示:

$ sudo ./opensnoop -x
PID COMM FD ERR PATH
15414 pool -1 2 /home/.hidden
15415 (ostnamed) -1 2 /sys/fs/cgroup/cpu/system.slice/systemd-hostnamed.service/cgroup.procs
15415 (ostnamed) -1 2 /sys/fs/cgroup/cpu/system.slice/cgroup.procs
15415 (ostnamed) -1 2 /sys/fs/cgroup/cpuacct/system.slice/systemd-hostnamed.service/cgroup.procs
15415 (ostnamed) -1 2 /sys/fs/cgroup/cpuacct/system.slice/cgroup.procs
15415 (ostnamed) -1 2 /sys/fs/cgroup/blkio/system.slice/systemd-hostnamed.service/cgroup.procs
15415 (ostnamed) -1 2 /sys/fs/cgroup/blkio/system.slice/cgroup.procs
15415 (ostnamed) -1 2 /sys/fs/cgroup/memory/system.slice/systemd-hostnamed.service/cgroup.procs
15415 (ostnamed) -1 2 /sys/fs/cgroup/memory/system.slice/cgroup.procs
15415 (ostnamed) -1 2 /sys/fs/cgroup/pids/system.slice/systemd-hostnamed.service/cgroup.procs
2548 compiz -1 2 
15416 systemd-cgroups -1 2 /run/systemd/container
15416 systemd-cgroups -1 2 /sys/fs/kdbus/0-system/bus
15415 systemd-hostnam -1 2 /run/systemd/container
15415 systemd-hostnam -1 13 /proc/1/environ
15415 systemd-hostnam -1 2 /sys/fs/kdbus/0-system/bus
1695 dbus-daemon -1 2 /run/systemd/users/0
15415 systemd-hostnam -1 2 /etc/machine-info
15414 pool -1 2 /home/tecmint/.hidden
15414 pool -1 2 /home/tecmint/Binary/.hidden
2599 dconf-service -1 2 /run/user/1000/dconf/user
...

跟踪特定的进程函数

下面的最后一个示例演示如何执行自定义跟踪操作,我们用它的PID跟踪一个特定的进程。

首先确定进程ID:

$ pidof firefox
15437

稍后,运行自定义跟踪命令,在下面的命令中:-p指定进程ID,do_sys_open()是一个内核函数,它动态地将第二个参数包括为字符串。

$ sudo ./trace -p 4095 'do_sys_open"%s", arg2'
TIME PID COMM FUNC -
12:17:14 15437 firefox do_sys_open /run/user/1000/dconf/user
12:17:14 15437 firefox do_sys_open /home/tecmint/.config/dconf/user
12:18:07 15437 firefox do_sys_open /run/user/1000/dconf/user
12:18:07 15437 firefox do_sys_open /home/tecmint/.config/dconf/user
12:18:13 15437 firefox do_sys_open /sys/devices/system/cpu/present
12:18:13 15437 firefox do_sys_open /dev/urandom
12:18:13 15437 firefox do_sys_open /dev/urandom
12:18:14 15437 firefox do_sys_open /usr/share/fonts/truetype/liberation/LiberationSans-Italic.ttf
12:18:14 15437 firefox do_sys_open /usr/share/fonts/truetype/liberation/LiberationSans-Italic.ttf
12:18:14 15437 firefox do_sys_open /usr/share/fonts/truetype/liberation/LiberationSans-Italic.ttf
12:18:14 15437 firefox do_sys_open /sys/devices/system/cpu/present
12:18:14 15437 firefox do_sys_open /dev/urandom
12:18:14 15437 firefox do_sys_open /dev/urandom
12:18:14 15437 firefox do_sys_open /dev/urandom
12:18:14 15437 firefox do_sys_open /dev/urandom
12:18:15 15437 firefox do_sys_open /sys/devices/system/cpu/present
12:18:15 15437 firefox do_sys_open /dev/urandom
12:18:15 15437 firefox do_sys_open /dev/urandom
12:18:15 15437 firefox do_sys_open /sys/devices/system/cpu/present
12:18:15 15437 firefox do_sys_open /dev/urandom
12:18:15 15437 firefox do_sys_open /dev/urandom
....

摘要

BCC是一个功能强大且易于使用的工具包,用于执行各种系统管理任务,如跟踪系统性能监视、跟踪块设备I/O 、 TCP函数、文件系统操作、系统调用,Node.js探测器等等。


tool  network  PERF  TRAC  动态  监视器  
相关文章