Debug Symbols...

[hmn]

I'm currently trying to debug a program that locks up after a while (see here and here). I have a basic working debugging setup, but I'm missing a lot of symbols and also I get corrupted backtraces at a certain point.

My question is how to get symbols for the system libraries (libc, syscalls etc.)... is it even possible without rebuilding all the libs and putting them in the Wiz fs?

Here is my setup so far: I basically took the USBSerial package and replaced the part where getty is started with gdbserver (instead of gdb from the official SDK I'm currently using a self-compiled gdb 7.1):

/path/to/gdbserver --multi /dev/ttyG0

After launching the script on the Wiz, I start gdb on the PC:

$ arm-linux-gdb
(gdb) target extended-remote /dev/ttyUSB0
(gdb) set solib-search-path /path/to/<copy of /lib from Wiz fs>:/path/to/<Toolchain libs from GPH-SDK>
(gdb) set remote exec-file program.bin
(gdb) symbol-file program_unstripped.bin
(gdb) handle SIG32 nostop noprint
(gdb) handle SIG33 nostop
(gdb) run

 

[hmn]

Ok, so I think I got this now.

So far I could only get something like this:

(gdb) thread apply all bt
[New Thread 970.973]
[New Thread 970.974]

Thread 3 (Thread 970.974):
#0  0x40511af4 in ?? () from /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libc.so.6
#1  0x4052abf4 in ?? () from /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 2 (Thread 970.973):
#0  0x4051600c in ?? () from /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libc.so.6
#1  0x4052abf4 in ?? () from /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 1 (Thread 970.970):
#0  main (argc=<value optimized out>, argv=<value optimized out>) at main.cpp:143

(gdb) info sharedlibrary 
From        To          Syms Read   Shared Object Library
0x40022438  0x400341a4  No          /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libiconv_plug.so
0x40107674  0x40140560  No          /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libSDL-1.2.so.0
0x40174b38  0x401755d0  No          /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libts-0.0.so.0
0x4017f8c0  0x40186cc0  No          /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libSDL_image-1.2.so.0
0x401a90b4  0x4020d3e0  No          /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libSDL_mixer-1.2.so.0
0x40243fb8  0x40273c58  No          /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libsmpeg-0.4.so.0
0x402a7e7c  0x402b7084  No          /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libSDL_gfx.so.0
0x402fbe90  0x4038dad0  No          /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libstdc++.so.6
0x403a934c  0x404145ac  No          /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libm.so.6
0x40456c84  0x4045f270  No          /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libgcc_s.so.1
0x4047e8b0  0x405583a0  No          /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libc.so.6
0x40577c78  0x40578988  No          /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libdl.so.2
0x405860a0  0x405902b0  No          /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libpthread.so.0
0x40000830  0x40016a1f  No          /lib/ld-linux.so.2

So while gdb knows where to find all the libraries, it has not loaded the symbols. It turns out that I have to load them myself:

(gdb) sharedlibrary libc
Reading symbols from /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libc.so.6...done.
Loaded symbols for /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libc.so.6
(gdb) sharedlibrary libpthread
Reading symbols from /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libpthread.so.0...done.
Loaded symbols for /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libpthread.so.0
(gdb) sharedlibrary libSDL
Reading symbols from /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libSDL-1.2.so.0...done.
Loaded symbols for /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libSDL-1.2.so.0
Reading symbols from /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libSDL_image-1.2.so.0...done.
Loaded symbols for /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libSDL_image-1.2.so.0
Reading symbols from /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libSDL_mixer-1.2.so.0...done.
Loaded symbols for /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libSDL_mixer-1.2.so.0
Reading symbols from /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libSDL_gfx.so.0...done.
Loaded symbols for /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libSDL_gfx.so.0
(gdb) info sharedlibrary 
From        To          Syms Read   Shared Object Library
0x40022438  0x400341a4  No          /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libiconv_plug.so
0x40107674  0x40140560  Yes         /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libSDL-1.2.so.0
0x40174b38  0x401755d0  No          /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libts-0.0.so.0
0x4017f8c0  0x40186cc0  Yes         /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libSDL_image-1.2.so.0
0x401a90b4  0x4020d3e0  Yes         /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libSDL_mixer-1.2.so.0
0x40243fb8  0x40273c58  No          /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libsmpeg-0.4.so.0
0x402a7e7c  0x402b7084  Yes         /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libSDL_gfx.so.0
0x402fbe90  0x4038dad0  No          /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libstdc++.so.6
0x403a934c  0x404145ac  No          /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libm.so.6
0x40456c84  0x4045f270  No          /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libgcc_s.so.1
0x4047e8b0  0x405583a0  Yes         /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libc.so.6
0x40577c78  0x40578988  No          /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libdl.so.2
0x405860a0  0x405902b0  Yes         /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libpthread.so.0
0x40000830  0x40016a1f  No          /lib/ld-linux.so.2

And now the backtrace just became a lot more meaningful:

(gdb) thread apply all bt
Thread 3 (Thread 970.974):
#0  0x40511af4 in write () from /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libc.so.6
#1  0x401330d8 in DSP_PlayAudio (this=0xa9378) at ./src/audio/dsp/SDL_dspaudio.c:135
#2  0x40108270 in SDL_RunAudio (audiop=<value optimized out>) at ./src/audio/SDL_audio.c:215
#3  0x40111208 in SDL_RunThread (data=0xa9470) at ./src/thread/SDL_thread.c:202
#4  0x40139fb4 in RunThread (data=0xfffffe00) at ./src/thread/pthread/SDL_systhread.c:47
#5  0x40588418 in pthread_start_thread () from /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libpthread.so.0
#6  0x4051f2e8 in clone () from /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libc.so.6
#7  0x4051f2e8 in clone () from /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 2 (Thread 970.973):
#0  0x4051600c in poll () from /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libc.so.6
#1  0x40588114 in __pthread_manager () from /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libpthread.so.0
#2  0x4051f2e8 in clone () from /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libc.so.6
#3  0x4051f2e8 in clone () from /home/chenz/workspace/wiz/ports/zelda/roth_us/src/lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 1 (Thread 970.970):
#0  main (argc=<value optimized out>, argv=<value optimized out>) at main.cpp:143