如何使用bcc的memleak分析内存泄露问题

问题

[root@ip-10-130-20-14 ~]# top
top - 17:29:54 up 125 days, 21:11,  1 user,  load average: 73.07, 70.28, 68.11
Tasks: 472 total,   1 running, 209 sleeping,   0 stopped,   0 zombie
%Cpu(s): 80.8 us,  5.5 sy,  0.0 ni, 10.2 id,  0.0 wa,  0.0 hi,  3.5 si,  0.0 st
KiB Mem : 52795657+total, 10496171+free, 41021644+used, 12778420 buff/cache
KiB Swap: 13421772+total, 13421772+free,        0 used. 11037457+avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 1157 presto    20   0   16.5t  28.6g  31020 S  4156  5.7  27961:51 presto-server
 2267 root      20   0   16624   5380   1648 S  18.8  0.0  13569:51 cgrulesengd
27866 root      20   0  160636   4784   3760 R  12.5  0.0   0:00.07 top

怀疑 presto-server 存在内存泄露

troubleshooting

[root@ip-10-130-20-14 ~]# /usr/share/bcc/tools/memleak -p 1157
Attaching to pid 1157, Ctrl+C to quit.
Traceback (most recent call last):
  File "/usr/share/bcc/tools/memleak", line 449, in <module>
    attach_probes("malloc")
  File "/usr/share/bcc/tools/memleak", line 439, in attach_probes
    pid=pid)
  File "/usr/lib/python2.7/site-packages/bcc/__init__.py", line 1303, in attach_uprobe
    (path, addr) = BPF._check_path_symbol(name, sym, addr, pid, sym_off)
  File "/usr/lib/python2.7/site-packages/bcc/__init__.py", line 908, in _check_path_symbol
    raise Exception("could not determine address of symbol %s" % symname)
Exception: could not determine address of symbol malloc

直接指定pid，未必可以获取到malloc的symbal，需要，指定 glibc的动态库。

[root@ip-10-130-20-14 ~]# /usr/share/bcc/tools/memleak -p 1157 -O /lib64/libc-2.17.so
Attaching to pid 1157, Ctrl+C to quit.
[17:30:05] Top 10 stacks with outstanding allocations:
[17:30:10] Top 10 stacks with outstanding allocations:
[17:30:15] Top 10 stacks with outstanding allocations:
[17:30:20] Top 10 stacks with outstanding allocations:
[17:30:25] Top 10 stacks with outstanding allocations:

muahao@aliyun.com