LXD certificate verification failure

Yesterday I upgraded my work machine to Ubuntu 21.10. Wrong move.

I run test systems on LXD. For compat reasons, we stopped at version 4.9. But this version stopped working with the new 5.13 kernel. The symptom was that few commands really finished exeuting. I tried using the old 5.11 kernel, but the system didn't even finish booting (it stayed waiting for a wireguard service that would never succeed because the underlaying network was not connected yet).

So next I tried upgrading the LXD conector (because we really use terraform), then upgrading LXD itself (and hoping our issues were fixed), but still no cigar.

So I went full nuclear, purged LXD and reinstalled. Not my preferred way to solve things, but I was already wasting a full day on this. This seemed to show some progress, but then I started getting this error in terraform (which actually comes from the provider):

lxd_network.singlenode: Refreshing state... [id=singlenode]
Error: Unable to create client for remote [localhost]: 
  Get "https://127.0.0.1:5555/1.0": 
      x509: certificate signed by unknown authority (possibly because of 
            "x509: ECDSA verification failure" while trying to verify candidate 
                   authority certificate "root@nimbus")

(I wrapped the line a little so it's more readable).

Ok, certificate issue, even when accept_remote_certificate = true was configured on the provider. The weird part was that f.i. lxd list was working fine. I run lxd remote and it was not showing localhost, so I tried lxc remote add nimbus 127.0.0.1:5555 --accept-certificate --password=xxxxx but I was still getting the same error!

Then a coworker reminded me: you have to remove ~/.config/lxc/servercerts/remote.crt so it forgets about it and gets a new one.

Now I have issues that might be related to the new LXD provider for terraform, but at least LXD is working fine again.