Remote Debugging with Eclipse

I am not familiar with Eclipse, as I prefer to use CLion. However, for projects that do not use CMake build system, I will have to use Eclipse.

In this post, I will discuss how to remote-debug with Eclipse. The setup is as follows:

target (local): running the application from, say terminal
host (Eclipse): debug the program as it is running

First, open up the project with Eclipse. Make sure that Eclipse can build the project.

Next, setup gdbserver on the target:
$ gdbserver :7777 EXECUTABLE ARG1 ARG2 ...

Here, 7777 is the port we will use for remote-debugging, EXECUTABLE is the binary file we are going to debug as it is running, and ARG1, ARG2, ... are appropriate arguments for this program.

Next, we setup Eclipse debugging.
From the menu, select Run --> Debug Configurations... --> C/C++ Remote Application (double click) --> Using GDB (DSF) Auto Remote Debugging Launcher (Select other) --> GDB (DSF) Manual Remote Debugging Launcher --> OK. Basically, we have selected "manual" remote debugging configuration here.

Make sure Project and C/C++ Application fields are properly filled, i.e., you should be able to select the project from the drop down menu if the project import/build is successful, and choose the EXECUTABLE for C/C++ Application.

In the Debugger tab --> Connection tab, change Port Number to 7777.

Finally, click Debug button. You now should be able to remote debug with Eclipse.

Happy hacking!

GDB on Mac OS X (Safe Method)

The officially supported debugging package for Mac OS X is lldb, which is a fine debugger. However, many people including me still prefer to use gdb, the GNU debugger. In fact, I use cgdb, which provides nice color interface to gdb, and that is why I must install gdb on my Mac. Unfortunately, it is not so easy to install and enable gdb to debug on Mac. In the previous post, I presented a method to do this, but I figure now that it was quite unsafe way to do so. For more details, please take a look at the official documentation.

Here is a better way of enabling gdb on your Mac. First, if you haven't installed gdb, do so.
$ brew install gdb

If you are using Sierra or above, run the following as well:
$ echo "set startup-with-shell off" >> ~/.gdbinit

Next, this is the part where we give gdb debug other processes. Open up Keychain Access application and on the menu select Keychain Access -> Certificate Assistant -> Create a certificate.

Enter gdb-cert for the name, select Self Signed Root for Identity Type, select Code Signing for Certificate Type, and check the box Let me override defaults.

Click Continue several times until you see Specify a Location For the Certificate window. Select System for the Keychain and proceed.

Once the certificate is created, double click on it, and in the Trust section locate Code Signing item. Select Always Trust and exit. You will be prompted with your admin password to make the change.

Finally, you can close Keychain Access app and type in the following in terminal:
$ sudo killall taskgated
$ codesign -fs gdb-cert $(which gdb)

That's it! You will be able to use gdb on your Mac!

How to Debug a Running Process Using GDB

Let's say you want to debug vim when run with sudo command, for example:
$ sudo vim /etc/fstab

It is easy to debug vim directly, but how would you do it when run with sudo? Well, here is one way to do it.

First, I will assume that you have vim executable file debug symbols. To find out how to compile vim with debug symbol, please refer to here. Say the file path is /usr/local/bin/vim.

Next, run the command that you want in one terminal:
$ sudo /usr/local/bin/vim /etc/fstab

Now, open up another terminal, and search for your vim process:
$ ps aux | grep vim
root     24687  0.0  0.0  63144  4212 pts/6    S+   19:04   0:00 sudo /usr/local/bin/vim /etc/fstab
root     24688  0.0  0.0  34180  5684 pts/6    S+   19:04   0:00 /usr/local/bin/vim /etc/fstab
linuxnme 24692  0.0  0.0  22572   988 pts/0    S+   19:04   0:00 grep --color=auto vim

We see two processes by root, one is sudo and the other is /usr/local/bin/vim, which is precisely what we want to debug, and its pid is 24668. To attach gdb to this process,

$ sudo gdb /usr/local/bin/vim 24688 -q

Reading symbols from /usr/local/bin/vim...done.

Attaching to program: /usr/local/bin/vim, process 24688

...

(gdb) 

Note that we are issuing sudo command before gdb here because we need superuser privilege to debug the process run by root.

Happy hacking!

Conditional Break with GDB

Say you want to setup a break point for given condition. To do this in gdb, simply enter the following command:
(gdb) break main.c:80 if x == 0

This will break only when the variable x is equal to 0 in the context. This will be very handy when debugging for certain conditions.

Quick gdb Tip: Enhance Debugging Experience with TUI or CGDB

I have been using gdb for quite some time, more than years in fact. Let's see... I first learned gdb in my sophomore class in Computer Systems Engineering's course, back in 2007. Wow, it's been almost 10 years now.

And yet, here I am, learning new features every day. In fact, I learned one feature in gdb today that will change my development completely: text user interface.

Try it yourself, if you don't already know.

$ gdb a.out -tui -q

I wonder why my class in 2007 never taught this feature. Maybe this feature was non-existent then?

Another option is to use cgdb. You could download the latest from git directly and compile:
$ git clone git://github.com/cgdb/cgdb.git
$ cd cgdb
$ ./autogen.sh
$ ./configure --prefix=/usr/local
$ make
$ sudo make install
$ cgdb -q a.out

Some Useful Commands in GDB

I would like to list common useful commands in gdb:

(gdb) b main
set up a break point at main function after function prologue

(gdb) b *main
set up a break point right at the address of main, so that when $pc points to *main, it will break

(gdb) delete 1
delete break point 1

(gdb) p/d i
print the content of variable i as a signed decimal

(gdb) p/u j
print the content of variable j as an unsigned decimal

(gdb) p/x k
print the content of variable k in hex

(gdb) x/5i $pc
print next 5 instructions in assembly

(gdb) x/s buffer
print string at address pointed by variable buffer

(gdb) x/xg 0x123456789abcdef0
print 64-bit value in hex at address 0x123456789abcdef0

(gdb) x/10wx {void*}$rbp
print 10 consecutive 32-bit values in hex starting from the address pointed by $rbp register

(gdb) info reg
examine all the register values

(gdb) r arg1 arg2
run the program with command-line argument arg1 and arg2

(gdb) si
execute one machine instruction; if it is a function call, step into the subroutine

(gdb) ni
execute one machine instruction; if it is a function call, do not step into the subroutine

(gdb) step
execute one line of C/C++ code; step into function

(gdb) next
execute one line of C/C++ code; step over function

(gdb) p func(arg1, arg2)
print return value by calling function func with arguments arg1 and arg2

(gdb) finish
continue execution until the end of current frame (i.e., subroutine)

(gdb) disass main
show assembly instructions of main function

(gdb) list main.c:37
display source code of main.c file at around line 37

(gdb) display/i $pc
keep displaying next machine instruction

(gdb) until 37
continue execution until the specified line

How to Enable GDB on Mac OS X El Capitan (NOT RECOMMENDED)

*** I do not recommend the instructions below ***
*** Please take a look at this post for safer method ***

I have a macbook air (MBA) which I carry around to use at various locations other than my desk at home. Of course I could install Ubuntu on my MBA, but I really like how easy it is to do anything on OS X,  so I am keeping it. I also do a lot of software development on my MBA. That being said, it bothers me that by default I cannot run gdb on OS X El Capitan. In this post, I will show how to enable gdb on OS X El Capitan. The credit goes to here.

First, you will need to install gdb. I would use brew. In case you don't have brew installed on the system, follow the instructions here.
$ brew install gdb

When you try to run a program on gdb, you will encounter error similar to below:
(gdb) run 
Starting program: a.out
Unable to find Mach task port for process-id 627: (os/kern) failure (0x5).
(please check gdb is codesigned - see taskgated(8))

The solution is as follows:
1. Restart OS X. Enter recovery mode by pressing and holding [command + R] until you see Apple logo. See here for more detail.

2. In the recovery mode, choose utilities menu and open up terminal

3. In the terminal, disable system integrity protection (SIP)
$ csrutil disable && reboot

4. Add -p option to /System/Library/LaunchDaemons/com.apple.taskgated.plist file. After your edit, it should read something like (line 22)
<array>
    <string>/usr/libexec/taskgated</string>
    <string>-sp</string>
</array>

5. (Optional) Re-enable SIP by repeating steps 1~3 with the command and reboot.

$ csrutil enable && reboot

6. Add your username to procmod group
$ sudo dseditgroup -o edit -a $USER -t user procmod

7. Locate gdb executable file and run

$ sudo chgrp procmod $(which gdb)

$ sudo chmod g+s $(which gdb)

Please be advised that you will need to reboot your system for the change to take effect.

Now, you should be able to use gdb on Mac OS X!

How to Record Function Calls Using GDB Automated Script

Probably one of the best ways to analyze new program source files is by looking at the function calls. In this post, I will discuss how to record the function flow from the main function using gdb. The credit goes to Juan M. Bello Rivas, and I found this from the link here.

From the link above, download the callgraph.tar.gz file and extract it..
$ wget http://web.archive.org/web/20090317091725/http://superadditive.com/software/callgraph.tar.gz
$ tar xfz callgraph.tar.gz
$ cd callgraph

You will need gawk to be able to successfully run it, so let's install if not already installed on your system.
$ sudo apt-get install -y gawk

Sometimes, it may be the case that your system has the original version of awk, which won't work with the callgraph script. You may need to replace awk with gawk in the file.
$ sed -i 's/awk/gawk/g' callgraph

OK, you are now ready to run the script. Let's first test it with the given test.c code.
$ gcc -g test.c

You should have a.out executable file. Let's run the script file. Before that, by the way, don't forget to enable execution flag.
$ chmod u+x callgraph
$ ./callgraph a.out

You should now see the following output
main foo (n=23)
foo bar (n=23)

This is very cool! Note that this script is probably suitable to work with only a small or medium size program, but this shall be still very useful!

In case the link will become dead in the future, I will copy the callgraph script file and test.c file below:

#!/bin/sh

  # Copyright (c) 2004-2007 Juan M. Bello Rivas <jmbr@superadditive.com>

#

# Permission is hereby granted, free of charge, to any person

# obtaining a copy of this software and associated documentation files

# (the "Software"), to deal in the Software without restriction,

# including without limitation the rights to use, copy, modify, merge,

# publish, distribute, sublicense, and/or sell copies of the Software,

# and to permit persons to whom the Software is furnished to do so,

# subject to the following conditions:

#

# The above copyright notice and this permission notice shall be

# included in all copies or substantial portions of the Software.

#

# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,

# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF

# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND

# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS

# BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN

# ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN

# CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE

# SOFTWARE.

prog_name="callgraph"

if [ $# -lt 1 ]; then

    echo "Usage: $prog_name EXECUTABLE [ARGS...]"

    echo

    echo "Example: $prog_name ~/bin/test-program foo 23"

    exit 1

fi

# Sanity checks.

FILE=$1

if [ ! -x $FILE ]; then

    echo "$prog_name: Unable to find executable '$FILE'"

    exit 1

fi

LANG="" gdb --eval-command=quit $FILE 2>&1 \

    | grep -E '(no\ debugging\ symbols\ found|not\ in\ executable\ format)' 2>&1 > /dev/null

if [ $? -eq 0 ]; then

    echo -n "$prog_name: Can't print call graph for '$FILE' because it's not a "

    echo "binary executable compiled with debugging symbols."

    exit 1;

fi

shift

# Set up temporary files.

TRACE="`mktemp -t $prog_name.XXXXXXXXXX`" || exit

GETFUNCS="`mktemp -t $prog_name.XXXXXXXXXX`" || exit

trap 'rm -f -- "$TRACE" "$GETFUNCS"' EXIT

trap 'trap - EXIT; rm -f -- "$TRACE" "$GETFUNCS"; exit 1' HUP INT QUIT TERM

# Take control of GDB and print call graph.

cat > $GETFUNCS <<EOF

set height 0

info functions

EOF

gdb --batch --command=$GETFUNCS $FILE 2>/dev/null | awk '

function get_func_name(str)

{

  split(str, part, "(");

  len = split(part[1], part, " ");

  len = split(part[len], part, "*");

  return part[len];

}

BEGIN {

  total = 0;

  print "set width 0";

  print "set height 0";

  print "set verbose off";

}

/[a-zA-Z_][a-zA-Z0-9_]*\(/ {

  fn = get_func_name($0);

  printf("break %s\n", fn);

  ++total;

}

END {

  for (i = 1; i <= total; i++) {

    print "commands", i;

    /* print "info args"; */

    print "backtrace 2";

    print "continue";

    print "end";

  }

  print "run"

}

' > $TRACE

gdb --batch --command=$TRACE --tty=/dev/null --args $FILE $@ 2>/dev/null | awk '

function get_callee(s)

{

  split(s, info, ",");

  split(info[2], fn, " ");

  callee = fn[1];

  return callee;

}

function get_params(s, n)

{

  split(s, par, n);

  split(par[2], par, " at ");

  sub(/ \(/, "(", par[1]);

  return par[1];

}

BEGIN {

  isrecord = 0;

  callee = "";

  caller = "*INITIAL*";

  params = "";

}

/^Breakpoint [0-9]+,/ {

  isrecord = 1;

  callee = get_callee($0);

  params = get_params($0, callee);

}

/^#1[ \t]+/ {

  if (isrecord)

    caller = $4;

}

/^$/ {

  if (isrecord && (caller != "*INITIAL*")) {

    printf("%s %s %s\n", caller, callee, params);

    callee = caller = params = "";

  }

}

'

Here is test.c file

#include <stdio.h>

#include <stdlib.h>

static void foo(int n);

static void bar(int n);

static void baz(int n);

int

main(int argc, char *argv[])

{

foo(23);

exit(EXIT_SUCCESS);

}

void 

foo(int n)

{

bar(n);

}

void

bar(int n)

{

baz(n);

}

void

baz(int n)

{

return;

}