在使用rk3588开发云手机过程中,有部分核心板随机出现hostserver进程占用cpu过高问题

因为golang编译时候去除了调试信息,没有加上pprof,无法看到goroutine的工作状态,同时hostserver进程为服务进程
无法使用上面命令打印堆栈
同时环境不容易复现,无法通过反复修改代码来验证问题,只能在不破坏环境的情况下,尽可能多收集信息,然后定位问题。
打开ida,导入goparser



太清晰了,简直就是明文[呲牙],以后用golang写程序要上加壳了

如果出现以上错误执行以下命令:
运行查看结果
现在我们已经知道消耗cpu的代码地址,0x0000000000086ba4 0x00000000004309f0 0x00000000003ab5ac
0x0000000000086ba4 代码:
0x00000000004309f0 代码:
0x00000000003ab5ac 代码:
从以上片段可以得出以下结论
runtime_goexit1 是 Go runtime 在销毁 goroutine 时的内部函数,说明有大量的goroutine创建和销毁
问题应该出在github_com_vishvananda_netlink__Handle_LinkByIndex这个函数
找到我们自己的代码

应该是上面这行代码导致的
用strace工具再次验证下
满屏的输出
基本可以肯定是上面这行代码导致的
case addrUpdate := <-addrCh:
link, err = netlink.LinkByIndex(addrUpdate.LinkIndex)
addrCh应该在什么场景下被close了,golang select 一个close的channel会导致立即返回,导致netlink.LinkByIndex(addrUpdate.LinkIndex)被无限调用,知道原因就好办了。
kill -QUIT $(pidof host_server)
kill -QUIT $(pidof host_server)
perf record -F 99 -p $(pidof host_server) -g -- sleep 30
perf record -F 99 -p $(pidof host_server) -g -- sleep 30
echo 1 > /proc/sys/kernel/kptr_restrict
echo 1 > /proc/sys/kernel/kptr_restrict
perf report
Samples: 1K of event 'cycles:P', Event count (approx.): 24575912110
Children Self Command Shared Object Symbol
+ 50.49% 0.00% host_server host_server [.] 0x0000000000086ba4
+ 42.66% 0.00% host_server host_server [.] 0x00000000004309f0
+ 42.25% 0.00% host_server host_server [.] 0x00000000003ab5ac
+ 42.07% 0.00% host_server host_server [.] 0x00000000003ab5f4
+ 41.87% 0.00% host_server host_server [.] 0x00000000003a5f2c
+ 37.75% 0.00% host_server [kernel.kallsyms] [k] el0t_64_sync
+ 37.75% 0.00% host_server [kernel.kallsyms] [k] el0t_64_sync_handler
+ 36.42% 0.00% host_server host_server [.] 0x000000000008485c
+ 30.67% 0.00% host_server [kernel.kallsyms] [k] el0_svc
+ 27.78% 0.00% host_server [kernel.kallsyms] [k] do_el0_svc
+ 27.72% 1.95% host_server [kernel.kallsyms] [k] el0_svc_common.constprop.0
+ 25.63% 0.06% host_server [kernel.kallsyms] [k] invoke_syscall
+ 25.23% 0.00% host_server host_server [.] 0x00000000000190a0
+ 17.42% 0.00% host_server host_server [.] 0x00000000000ca99b
+ 15.47% 0.00% host_server host_server [.] 0x000000000002d914
+ 15.28% 0.00% host_server host_server [.] 0x00000000003a63c8
+ 14.77% 0.00% host_server host_server [.] 0x0000000000031884
+ 11.29% 0.00% host_server host_server [.] 0x00000000000548c4
+ 11.11% 0.00% host_server host_server [.] 0x00000000003a6248
+ 11.01% 0.00% host_server host_server [.] 0x000000000002445c
+ 10.55% 0.00% host_server host_server [.] 0x00000000000847c4
+ 9.54% 0.00% host_server host_server [.] 0x00000000003a765c
+ 9.54% 0.00% host_server host_server [.] 0x000000000007ce70
+ 9.46% 0.00% host_server host_server [.] 0x0000000000054dc0
+ 8.59% 0.00% host_server host_server [.] 0x0000000000024018
+ 8.53% 0.00% host_server host_server [.] 0x00000000000243d3
+ 8.50% 0.00% host_server host_server [.] 0x00000000003a62fc
+ 8.23% 0.00% host_server host_server [.] 0x00000000000cb4fc
+ 7.97% 0.00% host_server host_server [.] 0x000000000002d8e8
+ 7.70% 0.00% host_server host_server [.] 0x00000000000caa7c
+ 7.52% 0.00% host_server host_server [.] 0x0000000000087430
+ 7.01% 0.18% host_server [kernel.kallsyms] [k] el0_da
+ 6.89% 0.00% host_server host_server [.] 0x0000000000031ae4
+ 6.73% 0.07% host_server [kernel.kallsyms] [k] do_mem_abort
+ 6.72% 0.00% host_server host_server [.] 0x00000000003a741c
+ 6.72% 0.00% host_server host_server [.] 0x00000000000ee794
+ 6.71% 0.00% host_server host_server [.] 0x000000000002aff8
+ 6.66% 0.10% host_server [kernel.kallsyms] [k] do_translation_fault
+ 6.50% 0.00% host_server host_server [.] 0x00000000000e54d4
+ 6.50% 0.00% host_server host_server [.] 0x00000000003a752c
+ 6.46% 0.08% host_server [kernel.kallsyms] [k] do_page_fault
+ 6.44% 0.00% host_server host_server [.] 0x0000000000163d94
+ 6.44% 0.00% host_server host_server [.] 0x000000000016522c
+ 6.07% 0.07% host_server [kernel.kallsyms] [k] __arm64_sys_sendto
+ 6.00% 0.00% host_server [kernel.kallsyms] [k] __sys_sendto
Samples: 1K of event 'cycles:P', Event count (approx.): 24575912110
Children Self Command Shared Object Symbol
+ 50.49% 0.00% host_server host_server [.] 0x0000000000086ba4
+ 42.66% 0.00% host_server host_server [.] 0x00000000004309f0
+ 42.25% 0.00% host_server host_server [.] 0x00000000003ab5ac
+ 42.07% 0.00% host_server host_server [.] 0x00000000003ab5f4
+ 41.87% 0.00% host_server host_server [.] 0x00000000003a5f2c
+ 37.75% 0.00% host_server [kernel.kallsyms] [k] el0t_64_sync
+ 37.75% 0.00% host_server [kernel.kallsyms] [k] el0t_64_sync_handler
+ 36.42% 0.00% host_server host_server [.] 0x000000000008485c
+ 30.67% 0.00% host_server [kernel.kallsyms] [k] el0_svc
+ 27.78% 0.00% host_server [kernel.kallsyms] [k] do_el0_svc
+ 27.72% 1.95% host_server [kernel.kallsyms] [k] el0_svc_common.constprop.0
+ 25.63% 0.06% host_server [kernel.kallsyms] [k] invoke_syscall
+ 25.23% 0.00% host_server host_server [.] 0x00000000000190a0
+ 17.42% 0.00% host_server host_server [.] 0x00000000000ca99b
+ 15.47% 0.00% host_server host_server [.] 0x000000000002d914
+ 15.28% 0.00% host_server host_server [.] 0x00000000003a63c8
+ 14.77% 0.00% host_server host_server [.] 0x0000000000031884
+ 11.29% 0.00% host_server host_server [.] 0x00000000000548c4
+ 11.11% 0.00% host_server host_server [.] 0x00000000003a6248
+ 11.01% 0.00% host_server host_server [.] 0x000000000002445c
+ 10.55% 0.00% host_server host_server [.] 0x00000000000847c4
+ 9.54% 0.00% host_server host_server [.] 0x00000000003a765c
+ 9.54% 0.00% host_server host_server [.] 0x000000000007ce70
+ 9.46% 0.00% host_server host_server [.] 0x0000000000054dc0
+ 8.59% 0.00% host_server host_server [.] 0x0000000000024018
+ 8.53% 0.00% host_server host_server [.] 0x00000000000243d3
+ 8.50% 0.00% host_server host_server [.] 0x00000000003a62fc
+ 8.23% 0.00% host_server host_server [.] 0x00000000000cb4fc
+ 7.97% 0.00% host_server host_server [.] 0x000000000002d8e8
+ 7.70% 0.00% host_server host_server [.] 0x00000000000caa7c
+ 7.52% 0.00% host_server host_server [.] 0x0000000000087430
+ 7.01% 0.18% host_server [kernel.kallsyms] [k] el0_da
+ 6.89% 0.00% host_server host_server [.] 0x0000000000031ae4
+ 6.73% 0.07% host_server [kernel.kallsyms] [k] do_mem_abort
+ 6.72% 0.00% host_server host_server [.] 0x00000000003a741c
+ 6.72% 0.00% host_server host_server [.] 0x00000000000ee794
+ 6.71% 0.00% host_server host_server [.] 0x000000000002aff8
+ 6.66% 0.10% host_server [kernel.kallsyms] [k] do_translation_fault
+ 6.50% 0.00% host_server host_server [.] 0x00000000000e54d4
+ 6.50% 0.00% host_server host_server [.] 0x00000000003a752c
+ 6.46% 0.08% host_server [kernel.kallsyms] [k] do_page_fault
+ 6.44% 0.00% host_server host_server [.] 0x0000000000163d94
+ 6.44% 0.00% host_server host_server [.] 0x000000000016522c
+ 6.07% 0.07% host_server [kernel.kallsyms] [k] __arm64_sys_sendto
+ 6.00% 0.00% host_server [kernel.kallsyms] [k] __sys_sendto
.text:0000000000086BA0 ; =============== S U B R O U T I N E =======================================
.text:0000000000086BA0
.text:0000000000086BA0 ; Attributes: noreturn
.text:0000000000086BA0
.text:0000000000086BA0 runtime_goexit ; DATA XREF: runtime_oneNewExtraM+38↑o
.text:0000000000086BA0 ; runtime_newproc1+E0↑o
.text:0000000000086BA0 MOV X0, X0
.text:0000000000086BA4
.text:0000000000086BA4 loc_86BA4
.text:0000000000086BA4 BL runtime_goexit1_0
.text:0000000000086BA0 ; =============== S U B R O U T I N E =======================================
.text:0000000000086BA0
.text:0000000000086BA0 ; Attributes: noreturn
.text:0000000000086BA0
.text:0000000000086BA0 runtime_goexit ; DATA XREF: runtime_oneNewExtraM+38↑o
.text:0000000000086BA0 ; runtime_newproc1+E0↑o
.text:0000000000086BA0 MOV X0, X0
.text:0000000000086BA4
.text:0000000000086BA4 loc_86BA4
.text:0000000000086BA4 BL runtime_goexit1_0
.text:00000000004309D4 SUB X29, SP,
.text:00000000004309D8 BL sub_87260
.text:00000000004309DC
.text:00000000004309DC loc_4309DC ; DATA XREF: host_server_netlink__Netlink_Init_func1+10C↑o
.text:00000000004309DC SUB X29, SP,
.text:00000000004309E0 LDR X1, [SP,
.text:00000000004309E4 ADRP X27,
.text:00000000004309E8 LDR X0, [X27,
.text:00000000004309EC BL github_com_vishvananda_netlink__Handle_LinkByIndex
.text:00000000004309F0 CBNZ X2, loc_4309FC
.text:00000000004309F4 MOV X4, XZR
.text:00000000004309D4 SUB X29, SP,
.text:00000000004309D8 BL sub_87260
.text:00000000004309DC
.text:00000000004309DC loc_4309DC ; DATA XREF: host_server_netlink__Netlink_Init_func1+10C↑o
.text:00000000004309DC SUB X29, SP,
.text:00000000004309E0 LDR X1, [SP,
.text:00000000004309E4 ADRP X27,
.text:00000000004309E8 LDR X0, [X27,
.text:00000000004309EC BL github_com_vishvananda_netlink__Handle_LinkByIndex
.text:00000000004309F0 CBNZ X2, loc_4309FC
.text:00000000004309F4 MOV X4, XZR
.text:00000000003AB59C STR X2, [X25,
.text:00000000003AB5A0
.text:00000000003AB5A0 loc_3AB5A0 ; CODE XREF: github_com_vishvananda_netlink__Handle_LinkByIndex+29C↑j
.text:00000000003AB5A0 STR X0, [X1,
.text:00000000003AB5A4 MOV X0, X5
.text:00000000003AB5A8 BL github_com_vishvananda_netlink_execGetLink
.text:00000000003AB5AC LDP X29, X30, [SP,
.text:00000000003AB5B0 ADD SP, SP,
.text:00000000003AB59C STR X2, [X25,
.text:00000000003AB5A0
.text:00000000003AB5A0 loc_3AB5A0 ; CODE XREF: github_com_vishvananda_netlink__Handle_LinkByIndex+29C↑j
.text:00000000003AB5A0 STR X0, [X1,
.text:00000000003AB5A4 MOV X0, X5
[培训]Windows内核深度攻防:从Hook技术到Rootkit实战!