Debugging docker containers with the host's tools

Marcos Dione

2024-01-06 13:29

Update:

cntr seems to be less hacky and much better maintained, so got use that instead.

At $NEW_JOB we heavily rely on Docker containers. The two biggest reasons I don't like them are that I think they're a nightmare to keep up to date security wise without also getting new versions with potential changes in behavior; and because they're usually built with as less tools inside the image as possible. I understand that the reasons might be also double: smaller image size and maybe intentionally reducing the attack surface and/or tools available to any hacker that might break into your shell from your service. I would rather have better defenses than having no tools to help me debug a break in.

For a while I've been thinking that it should be possible to mount the hosts's filesystem inside the container and use the host's tools to debug. I was attacking the problem kind of the wrong way, reading about how filesystem namespaces are implemented and how containers use them, until I finally changed my search query and found how to "Mount volumes into a running container" by Kynan Rilee.

The idea is really simple: find out the host's device that has the filesystem for the root partition (what? you have separate /usr?), create the device in the container, and then use nsenter without the --user namespace to mount it on /opt/host inside the container (otherwise you get a 'permission denied' error).

But that's still not enough. We have a few envvars to set before we can use the tools in /opt/host. First one is obviously PATH:

export PATH="$PATH":/opt/host/bin:/opt/host/sbin:/opt/host/usr/bin:/opt/host/usr/sbin

Still not enough, you need to also be able to load libraries from the new tree:

root@3e282deec242:/# mtr
mtr: error while loading shared libraries: libgtk-3.so.0: cannot open shared object file: No such file or directory

Here we have a dychotomy. We have to prioritize one of the two trees, either the container or the host. I think it's best to use the container's, but YMMV:

export LD_LIBRARY_PATH=/lib:/usr/lib:/opt/host/lib:/opt/host/usr/lib:/opt/host/usr/lib/x86_64-linux-gnu

Perl tools will also complain:

root@3e282deec242:/# ack
Can't locate File/Next.pm in @INC (you may need to install the File::Next module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.36.0 /usr/local/share/perl/5.36.0 /usr/lib/x86_64-linux-gnu/perl5/5.36 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl-base /usr/lib/x86_64-linux-gnu/perl/5.36 /usr/share/perl/5.36 /usr/local/lib/site_perl) at /opt/host/bin/ack line 11.
BEGIN failed--compilation aborted at /opt/host/bin/ack line 11.

So you need another one:

export PERL5LIB=/opt/host/etc/perl:/opt/host/usr/local/lib/x86_64-linux-gnu/perl/5.36.0:/opt/host/usr/local/share/perl/5.36.0:/opt/host/usr/lib/x86_64-linux-gnu/perl5/5.36:/opt/host/usr/share/perl5:/opt/host/usr/lib/x86_64-linux-gnu/perl-base:/opt/host/usr/lib/x86_64-linux-gnu/perl/5.36:/opt/host/usr/share/perl/5.36:/opt/host/usr/local/lib/site_perl

Incredibly python3 works OOTB.

I think that's all. I'll update this post if I find more envvars to set.

Here's a scripted version, except for all the exports up there. This omission has two or three reasons:

bash does not has a way to accept commands to run before showing the prompt. More below.
Some of those values are hard to guess; you will have to adapt them to your particular host's system.
I guess that's all :)

You can put them in your container's .bashrc and it will be read when bash starts.

Finally, the promised script:

#! /bin/bash

set -eu pipefail

container=$1

root_device=$(findmnt --noheadings --mountpoint / | awk '{ print $2 }')
container_pid=$(docker inspect --format {{.State.Pid}} "$container")

# create device and mount point
# the lack of double quotes around this -v----------------------------------v is intentional
docker exec "$container" mknod --mode 0660 "$root_device" b $(stat --format '%Hr %Lr' "$root_device")
docker exec "$container" mkdir -p /opt/host

# mount with host's root perms; that's why --user is not there
nsenter --target "$container_pid" --mount --uts --ipc --net --pid -- mount "$root_device" /opt/host

echo "go debug; don't forget to set envvars!"
docker exec "$container" /bin/bash

# cleanup
nsenter --target "$container_pid" --mount --uts --ipc --net --pid -- umount /opt/host
docker exec "$container" rm "$root_device"

You will probably need to run this as root, even if you can run docker naked, only because of nsenter.

Maybe I should also use nsenter for the debus session; that way I would be full root there too. I'll update this post if I ever find out situations where I needed that.