# Update:

> [cntr](https://github.com/Mic92/cntr) seems to be less hacky and much better maintained, so go use that instead.

At $NEW_JOB we heavily rely on Docker containers. The two biggest reasons I don't like them are that I think they're a
nightmare to keep up to date security wise without also getting new versions with potential changes in behavior; and
because they're usually built with as less tools inside the image as possible. I understand that the reasons might be
also double: smaller image size and maybe intentionally reducing the attack surface and/or tools available to any hacker
that might break into your shell from your service. I would rather have better defenses than having no tools to help me
debug a break in.

For a while I've been thinking that it should be possible to mount the hosts's filesystem inside the container and use
the host's tools to debug. I was attacking the problem kind of the wrong way, reading about how filesystem namespaces
are implemented and how containers use them, until I finally changed my search query and found how to
["Mount volumes into a running container" by Kynan Rilee](https://medium.com/kokster/mount-volumes-into-a-running-container-65a967bee3b5).

The idea is really simple: find out the host's device that has the filesystem for the root partition (what? you have
separate `/usr`?), create the device in the container, and then use `nsenter` _without_ the `--user` namespace to mount
it on `/opt/host` inside the container (otherwise you get a 'permission denied' error).

But that's still not enough. We have a few envvars to set before we can use the tools in `/opt/host`. First one is
obviously `PATH`:

```bash
export PATH="$PATH":/opt/host/bin:/opt/host/sbin:/opt/host/usr/bin:/opt/host/usr/sbin
```

Still not enough, you need to also be able to load libraries from the new tree:

```
root@3e282deec242:/# mtr
mtr: error while loading shared libraries: libgtk-3.so.0: cannot open shared object file: No such file or directory
```

Here we have a dychotomy. We have to prioritize one of the two trees, either the container or the host. I think it's
best to use the container's, but YMMV:


```bash
export LD_LIBRARY_PATH=/lib:/usr/lib:/opt/host/lib:/opt/host/usr/lib:/opt/host/usr/lib/x86_64-linux-gnu
```

Perl tools will also complain:

```
root@3e282deec242:/# ack
Can't locate File/Next.pm in @INC (you may need to install the File::Next module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.36.0 /usr/local/share/perl/5.36.0 /usr/lib/x86_64-linux-gnu/perl5/5.36 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl-base /usr/lib/x86_64-linux-gnu/perl/5.36 /usr/share/perl/5.36 /usr/local/lib/site_perl) at /opt/host/bin/ack line 11.
BEGIN failed--compilation aborted at /opt/host/bin/ack line 11.
```

So you need another one:

```bash
export PERL5LIB=/opt/host/etc/perl:/opt/host/usr/local/lib/x86_64-linux-gnu/perl/5.36.0:/opt/host/usr/local/share/perl/5.36.0:/opt/host/usr/lib/x86_64-linux-gnu/perl5/5.36:/opt/host/usr/share/perl5:/opt/host/usr/lib/x86_64-linux-gnu/perl-base:/opt/host/usr/lib/x86_64-linux-gnu/perl/5.36:/opt/host/usr/share/perl/5.36:/opt/host/usr/local/lib/site_perl
```

Incredibly `python3` works OOTB.

I think that's all. I'll update this post if I find more envvars to set.

Here's a scripted version, except for all the `export`s up there. This omission has two or three reasons:

* `bash` does not has a way to accept commands to run before showing the prompt. More below.
* Some of those values are hard to guess; you will have to adapt them to your particular host's system.
* I guess that's all :)

You can put them in your container's `.bashrc` and it will be read when `bash` starts.

Finally, the promised script:

```
#! /bin/bash

set -eu pipefail

container=$1

root_device=$(findmnt --noheadings --mountpoint / | awk '{ print $2 }')
container_pid=$(docker inspect --format {{.State.Pid}} "$container")

# create device and mount point
# the lack of double quotes around this -v----------------------------------v is intentional
docker exec "$container" mknod --mode 0660 "$root_device" b $(stat --format '%Hr %Lr' "$root_device")
docker exec "$container" mkdir -p /opt/host

# mount with host's root perms; that's why --user is not there
nsenter --target "$container_pid" --mount --uts --ipc --net --pid -- mount "$root_device" /opt/host

echo "go debug; don't forget to set envvars!"
docker exec "$container" /bin/bash

# cleanup
nsenter --target "$container_pid" --mount --uts --ipc --net --pid -- umount /opt/host
docker exec "$container" rm "$root_device"
```

You will probably need to run this as `root`, even if you can run `docker` naked, only because of `nsenter`.

Maybe I should also use `nsenter` for the debus session; that way I would be full root there too. I'll update this post
if I ever find out situations where I needed that.