An Investigative Debugging Toolbox

Last week I ran an experimental workshop on Investigative Debugging. It was a lot of fun and the attendees seemed to get a lot out of it. Afterwards I sent out some extra reference material, including this Linux-oriented list of investigative debugging tools!

The focus here is on “surveillance tools” – utilities that give you observability into exactly what programs are really doing at runtime. It’s far from complete, but it represents a ton of research and I hope you find it helpful!

Tool	Description
BCC tools	A collection of over 70 specialized tracing and performance profiling tools built on BPF.
`bpftrace`	A Linux tool analogous to DTrace, built on BPF. `bpftrace` enables users to compose “questions” at the command-line, using a concise AWK-inspired syntax, for which they would once have had to use (or create) a dedicated tracing tool. It is often possible to recreate BCC tools using bpftrace commands. But when a dedicated BCC tool exists for a given query, using it may be easier than hand-rolling a `bpftrace` command.
Docker	A technology for isolating Linux-based services into containers, each with their own isolated filesystem, process table, and virtual network. Containers are useful from an investigative debugging standpoint because they enable us to isolate and observe how a process interacts with its environment.
`docker diff`	Get a comprehensive list of all files changed within a Docker container.
`ftrace`	A low-level, “kind of janky” (Julia Evans’ words) interface to Linux kprobe and uprobe tracing.
`HTTP_PROXY`, `http_proxy`, `HTTPS_PROXY`	These environment variables are sometimes respected by programs as a way to configure a SOCKS proxy.
`LD_PRELOAD`	An environment variable that can be used to force a program to link to arbitrary libraries before running. This can be used to substitute user-provided functions that override default behavior with e.g. tracing instrumentation.
`ldd`	List all the libraries that an ELF executable links to.
`lsof`	List open filehandles, including network connections.
LTTng	A linux tracing framework. I believe this is now one of the inputs to BPF?
`ltrace`	Trace library calls.
`mitmproxy`	An interactive HTTP proxy server that can log and inspect requests and responses. Can be configured as either a SOCKS proxy or a transparent proxy.
`nm`	List all the names (e.g. function names) in an ELF executable/library.
Postman	A workspace for experimenting with and documenting HTTP calls, which can also act as a `mitmproxy`-like request inspector for other processes.
`readelf`	A general tool for inspecting and dumping information about ELF executables and libraries.
`strace`	Trace system calls.
`strings`	Dump all the static strings found in an ELF executable or library.
`tplist`	A BCC tool for listing all USDT tracepoints available in a process.
`uflow`	A BCC tool for tracing method calls and returns in high-level languages like Python, Ruby, and Java. May be installed somewhere weird and not in the path, like `/usr/sbin/lib/uflow`
WireShark	The preeminent tool for tapping into and dissecting IP network traffic at the packet level.

Blog

An Investigative Debugging Toolbox

Avdi Grimm

Join Our Newsletter. No Spam, Only the good stuff.

Related