Remotely upgrading CentOS6 to CentOS7

At $WORK we're in the middle of a massive OS upgrade in all of our client's appliances. Unfortunately, the OS is CentOS, which has no official method to do it except for 'reinstall'. This of course is unacceptable for us, we need an automatic system that can do it remotely. We have developed such a thing, mostly based on RedHat's official tool, redhat-upgrade-tool (rut). There are plenty of tutorials on how to do it on the web.

All was fine until we hit clients using UEFI instead of BIOS¹. Unluckily rut does not seem to handle the grub to grub2 transition wery well in this case. I had to create a dummy RPM package that, if it detects that the machine boots via EFI, generates a grub2 config file in /boot/efi/EFI/centos/grub.cfg and runs efibootmgr to tell the EFI system to use the EFI partition and file to boot. Here's the whole %posttrans section:

(
    set -x

    efi_parition=$(mount | awk '$3 == "/boot/efi" { print $1 }' | cut -d / -f 3)

    if [ -n "${efi_parition}" ]; then
        # create grub2 config file for the EFI module
        mkdir -pv /boot/efi/EFI/centos
        grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg

        # it's a shame we have to this here and then efibootmgr it's going to do the inverse
        # convert /boot/efi -> /dev/fooX -> foo + X
        efi_disk=$(find -H /sys/block/* -name ${efi_parition} | cut -d / -f 4)
        # we can't just cut the last char (part number could be beyond 9)
        # and we can't assume it's just after the disk name (disk: nvme0n1, partition: nvme0n1p1, part_num: 1)
        efi_parition_number=$(echo ${efi_parition} | egrep -o '[0-9]+$')

        # create an entry in the EFI boot manager and make it the only one to boot
        efibootmgr --create --bootnum 0123 --label grub2 --disk "/dev/${efi_disk}" \
            --part ${efi_parition_number} --loader \\EFI\\centos\\grubx64.efi --bootorder 0123
    fi
) &> /tmp/upgrade-centos-6.9-7.2.log

Another part of the automatic upgrade script detects EFI, installs grub2-efi and that other RPM, so the code is executed during the actual process of upgrading to CentOS7.

But the gist of this post is about how did I manage to test such a thing. Luckily we had 3 Lenovo servers laying around, but I could only reach them via ssh and IPMI. If you don't know, BMC/IPMI is a 'second' computer in your computer with which, among many, many other things, you can remotely turn on/off your computer, access the console remotely and even upload an ISO image and mount it in the 'main' machine as if it was a USB CD-ROM disk. This last one will come handy later. Notice that this is the system in the middle of Bloomberg's article about China infiltrating into companies, so you better get acquainted to it.

These Lenovo machines already had CentOS7 installed, so the first step was to install CentOS6. This should be possible by copying a CentOS6's installer and booting it via grub2. But before that I also had to enable UEFI boot. This broke booting, because the existing system had grbu2 installed in a disk's MBR instead on a EFI partition, and the disk had DOS partitions instead of GPT ones.

So the steps I took to fix this were:

On The client machine:

• Install the icedtea-plugin (the remote console is a java applet).
• Download Super Grub2 Disk.
• Connect via IPMI to the server, request the remote console.

On the remote server (all this through the IMPI remote console):

• Login, download CentOS6 installer iso.
• Tell the virtual device manager to add an ISO, using the SG2D ISO.
• Reboot the machine.
• Get into the Setup (another EFI module), enable EFI boot mode, reboot.
• Start the Boot Selection Menu, boot from the virtual CD.

On SG2D/Grub2's interface:

• Enable GRUB2's RAID and LVM support (the CentOS6 install ISO was in a LVM part).
• Selected Print the devices/partitions to find my LVM part.
• Loosely followed these instructions to boot from the ISO image:
• Open the Grub2 prompt (Press c).
• loopback loop (lvm/vg0-root)/root/CentOS-6.9-x86_64-minimal.iso
• linux (loop)/images/pxeboot/vmlinuz
• linux (loop)/images/pxeboot/initrd.img
• boot

This boots the installer, but it's not able to find the source media, so install from a URL. After setting up TCP/IP (fixed in my case), I used the URL http://mirrors.sonic.net/centos/6/os/x86_64/.

Freebie: More than I ever wanted to know about UEFI.