A Balanced Stance on Debugging Containers

Putting your code in a container shouldn't make it harder to debug. In this post I'll share some ideas about how you can spend less time debugging containerized code.

We'll start with the quick 'n dirty approach and work towards the full IDE experience.

Each section below describes a debug workflow that answers these questions differently:

• Which tools are hostside?
• Which tools are containerside?
• Is the program copied or mounted into the container?

Debugging by Braille

In this method:

• program is copied into image
• other tools are hostside
• who needs a debugger?

I call it "braille" because unless you've planned ahead, you're blind to what's going on inside the container.

The strategy here is to alter the program so that it gives hints about what's going on. The simplest way is to add print statements.

for target in targets:
  if not dexsave(target):
      print(target.dict())      damage(target, 8 * d(6))
      ignite(target)

With the added print statement we can see that Megalo-centipede is missing the hp attribute:

{'name': 'Maggris', 'x': 150, 'y': 295, 'hp': 36}
Maggris takes damage
Maggris stops looking angry and starts looking scared
{'name': 'Megalo-centipede', 'x': 150, 'y': 285}
Traceback (most recent call last):
  ...
  File "cast_fireball.py", line 52, in damage
    target.hp -= amount
AttributeError: 'Mob' object has no attribute 'hp'

Pros

• familiar workflow
• low toolchain complexity

Cons

• waiting for a new image to build for each code change
• cleaning up the braille once it's no longer needed

Tips

• git commit before you start adding braille
• fix the bug, but also improve the error message for next time
• print braille to stderr to avoid confusing callers that parse your stdout
• use a logging library to en/disable braille as needed
• trace functionality can generate braille automatically
• instead of redownloading packages, --mount=type=cache in your image build

Debugging Live Code

Code is "live" if it's available hostside--ready to edit, and also conainerside--ready to run. To make this work, mount the code into the container.

It's different from the previous strategy because the mounted files are not copies. Writing to them from one filesystem will affect how they're seen by both filesystems.

The mounted files will hide whatever was copied during the image build. This is handy because the copies stale as soon as you start making edits.

(Note that this is only true for a bind mount, which is the type recommended here--volume mounts play by different rules.)

Pros

• save file -> try changes, less waiting
• no need to migrate changes from container back to codebase

Cons

• files created containerside might seem out of place hostside
• ...and they'll have containerside ownership/permissions
• filesystem mappings can get complex if you have many containers

Tips

• files are mounted during container creation (not at image build time)
• docker-compose can automate your host/container folder mappings
• use .gitignore to prevent container-created files from showing up in your commits

Shell-Style Debugging...

In this mode your debugger is entirely containerside. There are two ways to start such a debugger.

...Attached to App Process

Some debuggers require extra runtime steps before they'll attach to a process:

This means either providing a process ID, or having the debugger launch the app:

$ node inspect hello.js
< Debugger listening on ws://127.0.0.1:...
Break on start in file:///containerside/path/to/hello.js:1
> 1 console.log('hello world')
debug> ▉ <-- debugger commands go here

...Called by App Code

Several other debuggers expect your program's code to call them.

This means calling a function to start the debugger:

for target in targets:
  if not dexsave(target):
      breakpoint()      damage(target, 8 * d(6))
      ignite(target)

Either Way

In either case, certain conditions will cause the debugger to pause.

> /containerside/path/to/cast_fireball.py(10)fireball()
-> damage(target, 8 * d(6))
(Pdb) ▉ <-- debugger commands go here

Then you can ask for information or advance the state of your program.

(Pdb) target.hp
36
(Pdb) continue
Maggris takes damage
Maggris stops looking angry and starts looking scared
> /containerside/path/to/cast_fireball.py(9)fireball()
-> breakpoint()
(Pdb) target.hp
*** AttributeError: 'Mob' object has no attribute 'hp'

Unlike with braille, you can also test ideas about how to fix the bug without restarting the program. We'll do this by setting an attribute on the fly:

(Pdb) target.hp = 1
(Pdb) continue
Megalo-centipede dies

The exception wasn't raised, so we know we're on the right track.

Pros

• requires only stdin, stdout, and stderr access to use
• debuggers are powerful tools
• minimal hostside setup

Cons

• limited to shell-only debuggers
• more containerside setup

Tips

• check out Use ipdb to Debug in Python for Python-specific details about this strategy

Graphically Debug an Instrumented Container

Some debuggers come in two parts, a debug server and an IDE. It's the server's job to be near the bug, and it's the up to the IDE (integrated development environment) to provide a human-friendly interface.

In this mode, the debugger is partly containerside, and partly hostside.

Some IDE's have container-specific debugging features. These will place the debug server in the container and handle the networking for you. Others provide "remote debugging" without addressing containers specifically. You can make these work too, but you'll need to place the pieces by hand.

Once the IDE and the debug-server connect, we can use them to debug the process in the container.

In this case, the debug server saw that line 40 raised an exception, so the IDE focused the edtor on that line.

Pros

• feature-rich GUI
• familiar workflow

Cons

• more hostside setup
• extra network requirements

Tips

• check out Debug Containers in VS Code for more about how to do this with VS Code

Summary

We started with a simple (but slow) container debug workflow: Treat the container like a black box and reason based on its inputs and outputs.

This works, but you have to wait for the image to build every time you make a code change. To speed things up, we mounted the code into our debug container.

The tighter debug loop let us iterate faster, but line-level execution control was still lacking, so we added a debugger to the container. This reclaimed much of the functionality that often gets lost in the transition to containers.

Finally, we added a hostside user interface which matched the containerside debugger. Now we can control the debugger with a GUI.

Different situations will call for different configurations, so it's worth getting comfortable with more than one debug workflow. That way you can balance your hostside and containerside tooling in a way that fits the task.

We hope these ideas save you as much time as they've saved us.

P.S.

The tricks in this post are baked into Conducto, which makes it nice to debug with. We're always looking for ways to improve it, so if you know a trick that deserves a mention here, please stop into our slack and let us know about it.

Happy Debugging.