-
Notifications
You must be signed in to change notification settings - Fork 69
Blog posts
(By Antti Kervinen on Oct 03 2019)
In this blog post I'll show by example how you can do the following.
-
Set a "breakpoint" to any of the following:
- Python program
- Shell script (including Dockerfile)
- Makefile.
-
Get interactive access to the program - even if it runs on a remote host or in a container.
The idea is to make the script call home when you want to stop it for debugging.
Furthermore, you can make your Python program call home in case of a crash (an unhandled exception). This enables remote post-mortem debugging.
First, start waiting for a home call: launch fmbt-debug at the HOME host. This works for debugging both Python 2 and 3. Launch fmbt-debug with -p PORT_NUMBER to listen to.
HOME$ fmbt-debug -p 33720
Example 1: A Python program that always calls home on line 2. Replace HOME with a hostname, or localhost when running the Python program and fmbt-debug on the same host.
import fmbt
fmbt.debug("HOME:33720") # this is a "breakpoint" that calls to fmbt-debug at HOME
my_var = 3
for i in range(1, my_var / 2):
print("loop...")
In case you do not want to include full utils/fmbt.py (for Python 2) or utils3/fmbt.py (for Python 3) file to your project, you can copy only the debug() function from the library. It's self-contained: it has no external dependencies even inside fmbt.py and it includes the imports that it needs.
Example 2: A Python program that calls home in case of an unhandled exception (on line 4):
import fmbt
fmbt.debug("HOME:33720", post_mortem=True) # call HOME in case of an unhandled exception, not now.
my_var = 3
for i in range(1, my_var / 2):
print("loop...")
When the program calls home, you will get Python debugger (pdb) prompt through fmbt-debug. There you can print local variables (p VARNAME), walk up (u) and down (d) in the stack, list code where the execution is going (l), execute Python code, and finally either quit (q) or continue (c) the execution, for instance. See Python Debugger Commands for more information.
First, start waiting for a home call with netcat:
HOME$ nc -k -l -p 33720
Example 3: A shell script that always calls home at the breakpoint called before-loop. If you want to debug the script remotely, replace localhost with the hostname/address from which you want to do the debugging.
#!/bin/sh
BP_HOME=localhost BP_PORT=33720
BP_CALLHOME='BP_FIFO=/tmp/$BP.$BP_HOME.$BP_PORT; (rm -f $BP_FIFO; mkfifo $BP_FIFO) && (echo "\"c\" continues"; echo -n "($BP) "; tail -f $BP_FIFO) | nc $BP_HOME $BP_PORT | while read cmd; do if test "$cmd" = "c" ; then echo -n "" >$BP_FIFO; sleep 0.1; fuser -k $BP_FIFO >/dev/null 2>&1; break; else eval $cmd >$BP_FIFO 2>&1; echo -n "($BP) " >$BP_FIFO; fi; done'
my_var=3
BP=before-loop eval $BP_CALLHOME
for i in $(seq 1 $my_var); do
echo loop...
done
When you run the shell script and it enters a "breakpoint" (evaluates BP_CALLHOME), you will get a breakpoint prompt to the netcat:
HOME$ nc -k -l -p 33720
"c" continues
(before-loop) echo $my_var
3
(before-loop) c
The idea for a "shell breakpoint" is basically an eval version of the elegant unix trick on interactive shell prompt:
mkfifo /tmp/f; cat /tmp/f | sh -i 2>&1 | nc HOME PORT > /tmp/f
Using eval instead of new shell process gives access to local variables.
Shell script debugging enables adding breakpoints to Makefile rules, too. That is, you can get a shell prompt between any commands in a Makefile to see what exactly is happening. And of course do it remotely, if needed:
Introduce BP_CALLHOME variable early in the Makefile. (Replace "localhost" by the name of the remote host in case you want to debug the Makefile from that host.)
BP_CALLHOME := 'BP_HOME=localhost; BP_PORT=33720; BP_FIFO=/tmp/$$BP.$$BP_HOME.$$BP_PORT; (rm -f $$BP_FIFO; mkfifo $$BP_FIFO) && (echo "\"c\" continues"; echo -n "($$BP) "; tail -f $$BP_FIFO) | nc $$BP_HOME $$BP_PORT | while read cmd; do if test "$$cmd" = "c" ; then echo -n "" >$$BP_FIFO; sleep 0.1; fuser -k $$BP_FIFO >/dev/null 2>&1; break; else eval $$cmd >$$BP_FIFO 2>&1; echo -n "($$BP) " >$$BP_FIFO; fi; done'
Add the following command wherever you want to stop the Makefile for debugging. Using a descriptive my-breakpoint-name will help following where you have been stopped in case you have many breakpoints. You can use Makefile variables in the breakpoint name, too.
BP=my-breakpoint-name sh -c $(BP_CALLHOME)
Just like with the shell script above, first start listening to a connection from the Makefile ($ nc -k -l -p 33720) and then let the make run.
Update 2019-10-31:
Examples works with netcat (nc) flavors: OpenBSD, Ncat, Busybox.
However, the "classic" version of nc (netcat-traditional package in Debian/Ubuntu) does not have an option for "keep listening" (-k) after disconnection. In case of using that, instead of nc -k -l -p 33720 use
socat STDIN TCP-LISTEN:33720,reuseaddr,fork
Thanks to Jarkko Sakkinen for mentioning potential issues with netcat flavors and preferring socat!
Update 2020-08-19: Makefile example added.
(By Antti Kervinen on Nov 09, 2018)
If a program crashes, Valgrind reports an error, or AddressSanitizer instrumentation has found an issue, it's time for debugging.
Typically debugging starts by reading source code around reported lines, possibly adding breakpoints and rerunning the program to see variable values. This is time consuming task.
fMBT/autodebug is a tool that makes the first step in debugging very simple: just give it the command that crashed or which Valgrind or AddressSanitizer complained about. In all these cases autodebug prints you similar report. Read the report, and you probably don't need to take further steps.
See three different errors (invalid pointer, stack underflow and uninitialized value in a condition) being detected and debugged in this example.