Debugging

From Maemo Leste Wiki
Revision as of 18:58, 5 May 2021 by Wizzup (talk | contribs)
Jump to navigationJump to search

Things break. How do you debug the problems?

You will want to take the following steps:

  • Add the devuan/debian debug repositories.
  • Identify the program that crashes (you can use `dpkg -S /usr/bin/program-name`) and install the -dbg or -dbgsym packages
  • Install the -dbg and -dbgsym dependencies of the packages
  • Attach to the program in gdb (or start it from gdb)


Add the debug repositories

Maemo Leste ships its debug packages in the normal repository, but for the Devuan/Debian packages, you need to add an additional source list file: 

# cat /etc/apt/sources.list.d/debug.list
deb http://deb.debian.org/debian-debug/ buster-debug main
deb http://deb.debian.org/debian-debug/ buster-backports-debug main
deb https://pkgmaster.devuan.org/devuan beowulf main contrib non-free
deb https://pkgmaster.devuan.org/devuan beowulf-updates main contrib non-free
deb https://pkgmaster.devuan.org/devuan beowulf-security main contrib non-free


Identify the program that crashes

Example: location-status (the location status applet) crashes.

Find out which main program or library is likely involved in the crash. If clicking the location status applet button causes a crash, the problem is likely in that package, and we can use dpkg -S to figure out what package the file is part of, and then install the debug symbols: 

# dpkg -S /usr/lib/arm-linux-gnueabihf/hildon-desktop/location-status.so
location-status: /usr/lib/arm-linux-gnueabihf/hildon-desktop/location-status.so
# apt-cache search location-status | grep dbg
location-status-dbgsym - debug symbols for location-status
# apt install location-status-dbgsym
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
  libmicrodns0
Use 'apt autoremove' to remove it.
The following NEW packages will be installed:
  location-status-dbgsym
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 27,1 kB of archives.
After this operation, 41,0 kB of additional disk space will be used.
0% [Working]
Get:1 https://maedevu.maemo.org/leste beowulf/main armhf location-status-dbgsym armhf 0.111-1+2m7 [27,1 kB]
Fetched 27,1 kB in 0s (55,8 kB/s)
Selecting previously unselected package location-status-dbgsym.
(Reading database ... 96891 files and directories currently installed.)
Preparing to unpack .../location-status-dbgsym_0.111-1+2m7_armhf.deb ...
Unpacking location-status-dbgsym (0.111-1+2m7) ...
Setting up location-status-dbgsym (0.111-1+2m7) ...

Install additional debug files

To get useful stack traces, gdb typically requires the debug symbols of other libraries to be installed as well. Following our example location-status, we can get a list of all the libraries involved using ldd: 

# ldd /usr/lib/arm-linux-gnueabihf/hildon-desktop/location-status.so

This lists a lot of libraries, so we will start with the basic ones, e.g. glib-2.0: 

root@maindroid:/etc/apt# dpkg -S /usr/lib/arm-linux-gnueabihf/libglib-2.0.so.0.5800.3
libglib2.0-0:armhf: /usr/lib/arm-linux-gnueabihf/libglib-2.0.so.0.5800.3
root@maindroid:/etc/apt# apt-cache search libglib2.0-0 | grep dbg
libglib2.0-0-dbgsym - debug symbols for libglib2.0-0
# apt install libglib2.0-0-dbgsym
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
  libmicrodns0
Use 'apt autoremove' to remove it.
The following NEW packages will be installed:
  libglib2.0-0-dbgsym
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 4240 kB of archives.
After this operation, 4984 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian-debug buster-debug/main armhf libglib2.0-0-dbgsym armhf 2.58.3-2+deb10u2 [4240 kB]
0% [1 libglib2.0-0-dbgsym 0 B/4240 kB 0%]
Fetched 4240 kB in 3s (1278 kB/s)
Selecting previously unselected package libglib2.0-0-dbgsym:armhf.
(Reading database ... 96893 files and directories currently installed.)
Preparing to unpack .../libglib2.0-0-dbgsym_2.58.3-2+deb10u2_armhf.deb ...
Unpacking libglib2.0-0-dbgsym:armhf (2.58.3-2+deb10u2) ...
Setting up libglib2.0-0-dbgsym:armhf (2.58.3-2+deb10u2) ...

We can repeat this process until we get satisfactory backtraces in gdb. For example: 

# gdb attach <pid>
[...]
Thread 1 "hildon-status-m" received signal SIGSEGV, Segmentation fault.
0xb6625f72 in g_atomic_ref_count_compare (arc=arc@entry=0x38, val=val@entry=0) at ../../../glib/grefcount.c:284
284	../../../glib/grefcount.c: No such file or directory.
(gdb) bt
#0  0xb6625f72 in g_atomic_ref_count_compare (arc=arc@entry=0x38, val=val@entry=0) at ../../../glib/grefcount.c:284
#1  0xb660cf0a in g_hash_table_lookup_node
    (hash_return=<synthetic pointer>, key=0xbed97c68, hash_table=<error reading variable: Cannot access memory at address 0x30>0x10) at ../../../glib/ghash.c:1153
#2  0xb660cf0a in g_hash_table_lookup (hash_table=<error reading variable: Cannot access memory at address 0x30>0x10, key=0xbed97c68) at ../../../glib/ghash.c:1153
#3  0xb64b03fe in  () at /usr/lib/arm-linux-gnueabihf/libosso.so.1

This means we should probably install debug symbols for libosso.so.1, so we add it and repeat: 

root@maindroid:/etc/apt# dpkg -S /usr/lib/arm-linux-gnueabihf/libosso.so.1
libosso1: /usr/lib/arm-linux-gnueabihf/libosso.so.1
root@maindroid:/etc/apt# apt-cache search libosso1 | grep dbg
libosso1-dbgsym - debug symbols for libosso1
root@maindroid:/etc/apt# apt install libosso1-dbgsym
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
  libmicrodns0
Use 'apt autoremove' to remove it.
The following NEW packages will be installed:
  libosso1-dbgsym
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 111 kB of archives.
After this operation, 137 kB of additional disk space will be used.
Get:1 https://maedevu.maemo.org/leste beowulf/main armhf libosso1-dbgsym armhf 2.35+2m7 [111 kB]
Fetched 111 kB in 1s (197 kB/s)
Selecting previously unselected package libosso1-dbgsym.
(Reading database ... 96903 files and directories currently installed.)
Preparing to unpack .../libosso1-dbgsym_2.35+2m7_armhf.deb ...
Unpacking libosso1-dbgsym (2.35+2m7) ...
Setting up libosso1-dbgsym (2.35+2m7) ...

Try again in gdb: 

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
__libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
46	../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S: No such file or directory.
(gdb) c
Continuing.

Thread 1 "hildon-status-m" received signal SIGSEGV, Segmentation fault.
0x10460f44 in ?? ()
(gdb) bt
#0  0x10460f44 in  ()
#1  0xb660cf16 in g_hash_table_lookup_node
    (hash_return=<synthetic pointer>, key=0xbed97c60, hash_table=0xb66cb765<error reading variable: Cannot access memory at address 0x2dbf004b>)
    at ../../../glib/ghash.c:379
#2  0xb660cf16 in g_hash_table_lookup (hash_table=0xb66cb765<error reading variable: Cannot access memory at address 0x2dbf004b>, key=key@entry=0xbed97c60)
    at ../../../glib/ghash.c:1153
#3  0xb64b03fe in try_plugin (dir=<optimized out>, file=file@entry=0xb474d75c "liblocation_applet.so", osso=<optimized out>) at osso-cp-plugin.c:103
#4  0xb64b0466 in osso_cp_plugin_execute (osso=0x63a4b8, filename=0xb474d75c "liblocation_applet.so", data=data@entry=0x669850, user_activated=user_activated@entry=1)
    at osso-cp-plugin.c:148
#5  0xb474d418 in execute_cp_plugin (obj=0x669850) at location-status.c:84
#9  0xb66e10b2 in <emit signal ??? on instance 0x669850 [HildonButton]> (instance=<optimized out>, signal_id=<optimized out>, detail=0)
    at ../../../gobject/gsignal.c:3447
    #6  0xb66ccb7e in g_closure_invoke (closure=0x664af8, return_value=0x0, n_param_values=1, param_values=0xbed98df0, invocation_hint=0xbed98d8c)
    at ../../../gobject/gclosure.c:810
    #7  0xb66dab0a in signal_emit_unlocked_R
    (node=node@entry=0x551458, detail=0, instance=0x669850, emission_return=emission_return@entry=0x0, instance_and_params=0xbed98df0)
    at ../../../gobject/gsignal.c:3635
    #8  0xb66e0c0e in g_signal_emit_valist (instance=<optimized out>, signal_id=<optimized out>, detail=detail@entry=1, var_args=..., var_args@entry=...)
    at ../../../gobject/gsignal.c:3391
#10 0xb6a9cd6c in  () at /usr/lib/arm-linux-gnueabihf/libgtk-x11-2.0.so.0

Do the same for libgtk-x11-2.0.so.0, etc. Finally, we get the full trace: 

root@maindroid:/etc/apt# gdb attach 8990
GNU gdb (Debian 8.2.1-2+b3) 8.2.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
attach: No such file or directory.
Attaching to process 8990
[New LWP 8991]
[New LWP 8992]
[New LWP 8993]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
__libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
46	../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S: No such file or directory.
(gdb) c
Continuing.

Thread 1 "hildon-status-m" received signal SIGSEGV, Segmentation fault.
g_type_class_meta_marshalv (closure=0xbed97c60, return_value=0x0, instance=0x955d3700, args=..., marshal_data=0x63a090, n_params=-1093038708, param_types=0x0)
    at ../../../gobject/gclosure.c:1028
1028	../../../gobject/gclosure.c: No such file or directory.
(gdb) bt
#0  0xb66cb768 in g_type_class_meta_marshalv
    (closure=0xbed97c60, return_value=0x0, instance=0x955d3700, args=..., marshal_data=0x63a090, n_params=-1093038708, param_types=0x0)
    at ../../../gobject/gclosure.c:1028
#1  0xb660cf16 in g_hash_table_lookup_node (hash_return=<synthetic pointer>, key=0xbed97c60, hash_table=0x544200 = {...}) at ../../../glib/ghash.c:379
#2  0xb660cf16 in g_hash_table_lookup (hash_table=0x544200 = {...}, key=key@entry=0xbed97c60) at ../../../glib/ghash.c:1153
#3  0xb64b03fe in try_plugin (dir=<optimized out>, file=file@entry=0xb474d75c "liblocation_applet.so", osso=<optimized out>) at osso-cp-plugin.c:103
#4  0xb64b0466 in osso_cp_plugin_execute (osso=0x639cb8, filename=0xb474d75c "liblocation_applet.so", data=data@entry=0x669850, user_activated=user_activated@entry=1)
    at osso-cp-plugin.c:148
#5  0xb474d418 in execute_cp_plugin (obj=0x669850) at location-status.c:84
#9  0xb66e10b2 in <emit signal ??? on instance 0x669850 [HildonButton]> (instance=<optimized out>, signal_id=<optimized out>, detail=0)
    at ../../../gobject/gsignal.c:3447
    #6  0xb66ccb7e in g_closure_invoke (closure=0x64f0e0, return_value=0x0, n_param_values=1, param_values=0xbed98df0, invocation_hint=0xbed98d8c)
    at ../../../gobject/gclosure.c:810
    #7  0xb66dab0a in signal_emit_unlocked_R
    (node=node@entry=0x551458, detail=0, instance=0x669850, emission_return=emission_return@entry=0x0, instance_and_params=0xbed98df0)
    at ../../../gobject/gsignal.c:3635
    #8  0xb66e0c0e in g_signal_emit_valist (instance=<optimized out>, signal_id=<optimized out>, detail=detail@entry=1, var_args=..., var_args@entry=...)
    at ../../../gobject/gsignal.c:3391
#10 0xb6a9cd6c in gtk_real_button_released (button=0x669850 [HildonButton]) at /build/gtk+2.0-2.24.32/gtk/gtkbutton.c:1712
#14 0xb66e10b2 in <emit signal ??? on instance 0x669850 [HildonButton]> (instance=<optimized out>, signal_id=<optimized out>, detail=0)
    at ../../../gobject/gsignal.c:3447
    #11 0xb66ccb7e in g_closure_invoke (closure=0x551240, return_value=0x0, n_param_values=1, param_values=0xbed99068, invocation_hint=0xbed99004)
    at ../../../gobject/gclosure.c:810
    #12 0xb66dab7c in signal_emit_unlocked_R
    (node=node@entry=0x551258, detail=0, instance=0x669850, emission_return=emission_return@entry=0x0, instance_and_params=0xbed99068)
    at ../../../gobject/gsignal.c:3565
    #13 0xb66e0c0e in g_signal_emit_valist (instance=<optimized out>, signal_id=<optimized out>, detail=detail@entry=3201929896, var_args=..., var_args@entry=...)
    at ../../../gobject/gsignal.c:3391
#15 0xb6a9c1e4 in gtk_button_button_release (widget=<optimized out>, event=<optimized out>) at /build/gtk+2.0-2.24.32/gtk/gtkbutton.c:1604
#20 0xb66e10b2 in <emit signal ??? on instance 0x669850 [HildonButton]> (instance=instance@entry=0x669850, signal_id=<optimized out>, detail=0)
    at ../../../gobject/gsignal.c:3447
    #16 0xb6b0de34 in _gtk_marshal_BOOLEAN__BOXED
    (closure=0x542e98, return_value=0xbed992a8, n_param_values=<optimized out>, param_values=0xbed992f8, invocation_hint=0xbed99294, marshal_data=0xb6a9c1d5 <gtk_button_button_release>) at gtkmarshalers.c:84
    #17 0xb66ccb7e in g_closure_invoke (closure=0x542e98, return_value=0xbed992a8, n_param_values=2, param_values=0xbed992f8, invocation_hint=0xbed99294)
    at ../../../gobject/gclosure.c:810
    #18 0xb66da2e2 in signal_emit_unlocked_R
    (node=node@entry=0x542ec0, detail=0, instance=0x669850, emission_return=emission_return@entry=0xbed993a8, instance_and_params=0xbed992f8)
    at ../../../gobject/gsignal.c:3673
    #19 0xb66e08d2 in g_signal_emit_valist (instance=instance@entry=0x669850, signal_id=<optimized out>, detail=detail@entry=3201930332, var_args=...,
    var_args@entry=...) at ../../../gobject/gsignal.c:3401
#21 0xb6bccb52 in gtk_widget_event_internal (widget=widget@entry=0x669850 [HildonButton], event=event@entry=0x6e0570) at /build/gtk+2.0-2.24.32/gtk/gtkwidget.c:5211
#22 0xb6bcccd0 in IA__gtk_widget_event (widget=widget@entry=0x669850 [HildonButton], event=event@entry=0x6e0570) at /build/gtk+2.0-2.24.32/gtk/gtkwidget.c:5008
#23 0xb6b0ccaa in IA__gtk_propagate_event (widget=0x669850 [HildonButton], event=0x6e0570) at /build/gtk+2.0-2.24.32/gtk/gtkmain.c:2503
#24 0xb6b0cf9a in IA__gtk_main_do_event (event=0x6e0570) at /build/gtk+2.0-2.24.32/gtk/gtkmain.c:1698
#25 0xb6b0cf9a in IA__gtk_main_do_event (event=<optimized out>) at /build/gtk+2.0-2.24.32/gtk/gtkmain.c:1503
#26 0xb69ed6f2 in gdk_event_dispatch (source=<optimized out>, callback=<optimized out>, user_data=<optimized out>)
    at /build/gtk+2.0-2.24.32/gdk/x11/gdkevents-x11.c:2425
#27 0xb661987e in g_main_dispatch (context=0x571b68) at ../../../glib/gmain.c:3182
#28 0xb661987e in g_main_context_dispatch (context=context@entry=0x571b68) at ../../../glib/gmain.c:3847
#29 0xb6619a2a in g_main_context_iterate (context=0x571b68, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../../../glib/gmain.c:3920
#30 0xb6619ca2 in g_main_loop_run (loop=0x641090) at ../../../glib/gmain.c:4116
#31 0xb6b0c4b0 in IA__gtk_main () at /build/gtk+2.0-2.24.32/gtk/gtkmain.c:1270
#32 0xb51231f0 in main () at /usr/bin/hildon-status-menu.launch
#33 0x004d208a in main ()

Debugging applications launched via maemo-launcher

When programs are launched via maemo-launcher (many are), like the status area applet, then to debug problems, you can attach with gdb to the running processes. However, it is more likely that you will need to recompile and debug, in which case you should stop the 'service' or program with dsmetool (if required), and invoke the program using maemo-summoner, like so: 

gdb --args maemo-summoner hildon-status-menu.launch

TODO

  • valgrind
  • gdb

Example debugging location-status bug:

Retrieved from "https://leste.maemo.org/index.php?title=Debugging&oldid=998"