Kernel Panic Calltrace 분석

Kernel Panic이 발생한 경우 하기와 같이 Calltrace log를 출력하기 때문에 해당 정보를 기반으로 디버깅 하는 방법 정리

 

example) xen porting 과정 중 kernel 부팅 시 발생한 error

[    0.500423] Internal error: synchronous external abort: 96000010 [#1] PREEMPT SMP

[    0.501475] Modules linked in:

[    0.502306] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 4.20.11 #6

[    0.503174] Hardware name: linux,dummy-virt (DT)

[    0.507173] pstate: 40000005 (nZcv daif -PAN -UAO)

[    0.507904] pc : amba_device_try_add+0x1b8/0x290

[    0.508512] lr : amba_device_try_add+0x1a4/0x290

[    0.509163] sp : ffff00000806bc30

[    0.509647] x29: ffff00000806bc30 x28: 0000000000000000

[    0.510329] x27: 0000000000000001 x26: ffff00000910efb8

[    0.511391] x25: ffff80003dff3700 x24: ffff00000800d000

[    0.512127] x23: 0000000000001000 x22: 0000000000000000

[    0.512728] x21: ffff80003cb13af0 x20: 0000000000000000

[    0.513335] x19: ffff80003cb13800 x18: ffff0000090496c8

[    0.514501] x17: 0000000000000000 x16: 0000000000000000

[    0.514830] x15: 0000000000000010 x14: ffffffffffffffff

[    0.515021] x13: 0000000000000000 x12: 0101010101010101

[    0.515358] x11: 0000000000000020 x10: 0101010101010101

[    0.515619] x9 : 0000000000000000 x8 : ffff80003cb62280

[    0.515868] x7 : 0000000000000000 x6 : 0000000000000004

[    0.516027] x5 : 0000000000000001 x4 : ffff80003c910000

[    0.516192] x3 : 0000000000000000 x2 : 0000000000000000

[    0.516454] x1 : ffff80003c910000 x0 : ffff00000800dfe0

[    0.516792] Process swapper/0 (pid: 1, stack limit = 0x(____ptrval____))

[    0.517376] Call trace:

[    0.517767]  amba_device_try_add+0x1b8/0x290

[    0.518314]  amba_device_add+0x18/0xd0

[    0.518705]  of_platform_bus_create+0x2e8/0x370

[    0.519157]  of_platform_populate+0x50/0xc8

[    0.519563]  of_platform_default_populate_init+0xac/0xc8

[    0.520207]  do_one_initcall+0x54/0x158

[    0.520356]  kernel_init_freeable+0x188/0x224

[    0.520540]  kernel_init+0x10/0x100

[    0.520920]  ret_from_fork+0x10/0x18

[    0.521412] Code: 35000580 d10082e0 52800002 8b000300 (b9400001)

[    0.522357] ---[ end trace 75a863087b1fff96 ]---

[    0.523900] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b

[    0.525260] SMP: stopping secondary CPUs

[    0.527740] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]---

 

Panic 발생 위치 찾기

PC 값이 amba_device_try_add+0x1b8/0x290를 가리키므로 objdump를 통해 function name에 해당하는 base address를 찾는다.

 

aarch64-linux-gnu-objdump -d vmlinux | grep amba_device_try_add

ffff00000851f5e8 <amba_device_try_add>:

ffff00000851f608:       540005e0        b.eq    ffff00000851f6c4 <amba_device_try_add+0xdc>  // b.none

ffff00000851f614:       54000620        b.eq    ffff00000851f6d8 <amba_device_try_add+0xf0>  // b.none

ffff00000851f628:       35000160        cbnz    w0, ffff00000851f654 <amba_device_try_add+0x6c>

 

amba_device_try_add의 주소가 ffff00000851f5e8인 것을 알았으며 offset 값을 더해 panic를 발생시킨 코드의 주소를 알수 있다.

 

amba_device_try_add(0xffff00000851f5e8) + 0x1b8 = 0xffff00000851f7a0

 

addr2line을 활용하여 panic를 발생시킨 코드 위치를 알 수 있다.

 

root@4beceef56ea9:/user/winxp4333/WD/Virt_ENV/linux-4.20.11# aarch64-linux-gnu-addr2line -e vmlinux 0xffff00000851f7a0 -i

/user/winxp4333/WD/Virt_ENV/linux-4.20.11/./arch/arm64/include/asm/io.h:88 (discriminator 3)

/user/winxp4333/WD/Virt_ENV/linux-4.20.11/drivers/amba/bus.c:397 (discriminator 3)