Debootstrap a standalone Debian system across multiple partitions

Tuesday, July 07, 2015

Having finally finished my MSci Computer Science course at Loughborough University (hoorah!) I am now exploring plausible migration options to move my Debian Wheezy 7.8 root Debian 7.8 installation from my Samsung 840 250GB to my new Samsung 850 512GB.

Intro

The substantial (and controversial*) transition of the core Debian GNU/Linux init system from the System V (found in "Wheezy" ~ old stable) to systemd (found in "Jessie" ~ stable) would potentially favour a complete reinstallation of the system so as to avoid any unforeseen/edge case upgrade issues down the line.

This post covers how the flexible debootstrap utility can be used to install a complete (albeit minimal) installation of a Debian Jessie environment on a typical partition scheme that would normally be done through the traditional Debian installer via CD/USB/Network.
Unlike an external media installation debootstrap enables the creation of a Debian system while the host is running.

Currently official Debian guides covering the debootstrap utility are either outdated (https://www.debian.org/releases/lenny//amd64/apds03.html.en ~ Lenny!) or provide very little in terms of reasoning for the operations performed (https://wiki.debian.org/Debootstrap). This post aims to alleviate these two issues in addition to serving as a reference for myself if I decide on the debootstrap solution.

Again, If you'd like to skip the explanations and just want the commands then see the TL;DR section at the bottom of this page.

Packages Used

gdisk: 0.8.5-1
debootstrap: 1.0.48+deb7u2

1. Prerequisites & Assumptions

This post pertains to the installation of a comfortably configured (sane and minimal!) standalone Debian 8.1 "Jessie" installation on a separate disk to that of the host machine's root installation.
Nevertheless, installation of the Debian 8.1 "Jessie" on the external disk can be performed through either a Live Debian CD/USB or an existing Debian 7.x "Wheezy" installation. (Most probably older versions of Debian or any derivatives would work too but I cannot confirm this for certain!)

To avoid any accidental misconfiguration adversely affecting my server (i.e. messing up the GRUB2 bootloader, incorrect partition letter, etc.) I made the decision to perform all operations in a Debian 7.8 VM that had 2 disks:

virt-install   
  --name debootstrap_partitions 
  --ram 1024 
  --virt-type kvm 
  --cpu host 
  --vcpus 1 
  --disk path=/media/VM/Self Study/debootstrap partitions/debian-7.2.qcow2,format=qcow2 
  --import 
  --network=bridge:br1 
  --disk path=/media/VM/Self Study/mdadm debootstrap/debootstrapped_install_debian-8.1.qcow2,format=qcow2

Where the debootstrapped_install_debian-8.1.qcow2 is a 20GiB QCOW2 virtual disk image:
qemu-img create -f qcow2 /media/VM/Self Study/debootstrap partitions/debootstrapped_install_debian-8.1.qcow2 20G

2. Partition & Format

In most cases the final system's intended usage will determine its underlying partitioning scheme. To best accommodate for this I have created a GPT partition scheme with several entries that can be understood in such a way that it would be easy to either add or remove partitions (based on the user's needs) which ultimately corresponded to the final system's mount points.

The layout for my example 20GiB external disk (/dev/sdb) is as follows:

bios_boot: 1MiB
'/boot': 75MiB
'/': 3GiB
'/home': 3GiB
'/usr': 5GiB
'/var': 5GiB
'swap': ~2GiB (remaining space)

1. Update the APT package cache and install any outstanding upgrades:
sudo aptitude update && sudo aptitude safe-upgrade -y

2. Install the gdisk utility suite so we can create GPT disk partition structures:
sudo aptitude install gdisk

3. Clear any pre-existing GPT (or MBR) structures on the external disk (WARNING! - ensure no desired data is on this external disk beforehand):
sudo sgdisk --zap-all /dev/sdb

4. Create a blank GPT structure on the external disk:
sudo sgdisk /dev/sdb

5. Create the partitions that will correspond to the final system's mount points. Partitions are aligned on the gdisk default recommended 2,048-sector boundaries; this means that the start sector for every partition must be a multiple of 2048. For the example 7 partitions listed above the following invocations of sgdisk (non-interactive) were used:
sudo sgdisk --new=1:2048:4097 --typecode=1:EF02 --change-name=1:bios_boot --print /dev/sdb
sudo sgdisk --new=2:6144:159743 --typecode=2:8300 --change-name=2:boot --print /dev/sdb
sudo sgdisk --new=3:159744:6451199 --typecode=3:8300 --change-name=3:root --print /dev/sdb
sudo sgdisk --new=4:6451200:16936959 --typecode=4:8300 --change-name=4:home --print /dev/sdb
sudo sgdisk --new=5:16936960:27422719 --typecode=5:8300 --change-name=5:usr --print /dev/sdb
sudo sgdisk --new=6:27422720:37908479 --typecode=6:8300 --change-name=6:var --print /dev/sdb
sudo sgdisk --largest-new=7 --typecode=7:8200 --change-name=7:swap --print /dev/sdb
I used Google's calculator function to figure out the size of the partitions needed. Make sure to use the KiB/MiB/GiB (*ibibyte) scale NOT the kB/MB/GB scale when calculating the the sector range for the partition to span across.
I adopted a simple formula for figuring out the last sector of the partition for a desired partition size: (part_size_in_ibibyte / 512B) + offset - 1. Where offset is the start of the partition that aligns with the 2,048-sector boundary.
The --typecode assists (I believe) the OS with determining the underlying partition format. The typecode 8300 corresponds to "Linux filesystem" which in turn corresponds to the partition GUID of: 0FC63DAF-8483-4772-8E79-3D69D8477DE4. A summarised list of GPT typecodes can be listed with: sudo sgdisk --list-types. Typecode partition GUID's can be found on Wikipedia here.

6. Refresh the kernel's view of the GPT table structure on the external disk:
echo 1 > /sys/block/sdb/device/rescan

7. (Optional) Backup the newly created GPT partition table:
sudo sgdisk --backup=/path/to/backup/debootstrapped_gpt_partitions.bin /dev/sdb

8. Now we have achieved the desired partition scheme we can begin formatting partitions. I started by formatting the dedicated '/boot' partition with an ext2 filesystem:
sudo mke2fs /dev/sdb2

9. Format the other system partitions with your favoured filesystem. For my environment I stuck with the mature ext4 filesystem:

for i in {3..6};  
do  
  sudo mkfs.ext4 -q /dev/sdb$i && echo "Partition /dev/sdb$i formatted"
done

10. Finally format the 'swap' partition to prepare it for use:
sudo mkswap /dev/sdb7

3. Deboostrap & Configure

At this stage we have a finalised GPT partitioning scheme for the external disk with each partition being appropriately formatted in preparation for the Debian 8.1's installation.

1. We need to mount the designated root ('/') partition of the external disk (/dev/sdb2) and create the mount points (directories) that link to the other dedicated partitions:
sudo mkdir /mnt/debootstrap
sudo mount /dev/sdb3 /mnt/debootstrap
sudo mkdir -v /mnt/debootstrap/{boot,home,usr,var}

2. Mount each of the external disk's partitions to their corresponding directory that was just created. Double check the partitions line up to their desired mount point:
sudo mount /dev/sdb2 /mnt/debootstrap/boot
sudo mount /dev/sdb4 /mnt/debootstrap/home
sudo mount /dev/sdb5 /mnt/debootstrap/usr
sudo mount /dev/sdb6 /mnt/debootstrap/var

3. Install the debootstrap utility:
sudo aptitude install debootstrap -y

4. Invoke the debootstrap and include some core packages that enable the bootstrapped installation to operate on a standalone basis. Depending on your internet connection, host disk performance, and overall host load this process may take over 5 minutes to complete.

sudo debootstrap   
  --include=sudo,openssh-server,locales,
            linux-image-amd64,grub-pc 
  --arch amd64 stable 
  /mnt/debootstrap 
  http://http.debian.net/debian/

5. Mount the host's virtual filesystem's ('/proc', '/sys') and bind mount the host's '/dev' directory within the newly installed Debian 8.1 system. These special mounts are required by GRUB2 for device identification when installing the bootloader to /dev/sdb.
sudo mount -t proc proc /mnt/debootstrap/proc
sudo mount -t sysfs sysfs /mnt/debootstrap/sys
sudo mount -B /dev /mnt/debootstrap/dev

6. While a very core Debian 8.1 system has now been installed across the targeted external disk's partitions several basic configurations that would commonly be automated by the Debian installer are missing (i.e. /etc/fstab, /etc/network/interfaces). To resolve these issues we need to chroot into the new Debian 8.1. installation:
LANG=C sudo chroot /mnt/debootstrap
The environment variable LANG is set to eliminate locale warnings when certain applications (i.e. apt-get) are ran within the chrooted environment.
NOTE: From this point onward I will assume you are in the chroot environment. Double check you are before making the following changes otherwise you will cause booting issues with your host installation.

7. To ensure all system partitions are mounted at boot we need to add them in the /etc/fstab file. UUID's of partitions can be found by using the blkid binary. Feel free to add any other entries to the /etc/fstab file for other mounts desired at startup. Any text editor can be used here, for my environment I used vim.

#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system>                                 <mount point>   <type>  <options>       <dump>  <pass>

# /boot (/dev/sdb2)
UUID=368ba92a-4b4e-446c-b5d1-b132131dc286       /boot           ext2    defaults        0       2

# /     (/dev/sdb3)
UUID=8639b78b-49d8-40c8-9bb1-62b013c478ea       /               ext4    defaults,errors=remount-ro      0       1

# /home  (/dev/sdb4)
UUID=51ba6674-0b3d-46f9-84d7-670564c5f264       /home           ext4    defaults        0       2

# /usr  (/dev/sdb5)
UUID=b8b4b210-888e-4862-a2fd-e3c69e5aea92       /usr            ext4    defaults        0       2

# /var  (/dev/sdb6)
UUID=02331787-02b3-4df1-82fe-2ee90f72a200       /var           ext4    defaults        0       2

# swap  (/dev/sdb7)
UUID=61b10995-44a6-4e13-8ff0-55fbbe279643       none            swap    defaults        0       0

These UUID values were discovered using the blkid as follows:
sudo blkid /dev/sdb{2..7}

14. To configure network interfaces at boot we need to edit the /etc/network/interfaces file. For my environment I simply copied the contents of the host's /etc/network/interfaces file into the chrooted Debian 8.1's. Any text editor can be used here, for my environment I used vim.

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo  
iface lo inet loopback

# The primary network interface
allow-hotplug eth0  
iface eth0 inet dhcp

15. As of current the root account for the Debian 8.1 system has no password, lets address this now:
passwd root

16. The Debian installer requires (I believe) the creation of a standard user account. While this isn't 100% necessary for the operation of our bootstrapped Debian 8.1 system, it is good practice to access the system as a non-privileged user.
adduser grindon

17. Under the assumption the standard user created in step 16. is required to execute certain binaries in a privileged state we can add them to the sudo group. This step can be skipped if the user is not permitted any form of privilege escalation.
usermod -a -G sudo grindon

18. We should set the bootstrapped Debian 8.1's hostname so it appears as a separate machine on the network when booted.
echo "debootstrap-debian" > /etc/hostname

19. Configure the default APT sources to enable the non-free and contrib repositories. Update the APT package cache if you decide to alter the /etc/apt/sources.list file in order to accommodate the changes. Any text editor can be used here, for my environment I used vim.
deb http://http.debian.net/debian stable main non-free contrib
apt-get update

20. Install the GRUB2 bootloader to the external disk the bootstrapped Debian 8.1 system resides upon:
grub-install /dev/sdb

21. Generate the /boot/grub/grub.cfg configuration so as to create the boot menu entries for the Debian 8.1 system:
grub-mkconfig -o /boot/grub/grub.cfg

22. Exit the chroot environment. If you wish further changes you may make them now, finish up by running the following command to get us back in the context of the host system:
exit

23. Now we need to unmount all virtual and physical filesystems required by the chroot environment. As partitions are mounted on top of the external disk root ('/') partition (/dev/sdb3) we need to unmount those first:
sudo umount /mnt/debootstrap/{proc,sys,dev}
sudo umount /dev/sdb2
sudo umount /dev/sdb{4..6}
sudo umount /dev/sdb3

Boot Options

The external disk now contains a bootable, standalone Debian 8.1 installation that has system files spread across numerous partitions. At this stage I offer three options for confirming the successful installation of the bootstrapped Debian 8.1 system. These options in order of user involvement are as follows:

A. Update the /boot/grub/grub.cfg on the host allowing GRUB2's os-prober to detect the Debian installation on /dev/sdb3.
sudo grub-mkconfig -o /boot/grub/grub.cfg
This method doesn't explicitly confirm that GRUB2 and the bootloader have been correctly installed on the external disk. I personally recommend trying out method B. or C. to ensure the system boots without the assistance of another GRUB2 setup.

B. Alter the system's BIOS/UEFI boot order and select the external disk the Debian 8.1 system was bootstrapped as the first disk to boot from.

C. When accessing the original host's GRUB2 menu (upon boot) enter the GRUB2 console environment by hitting the 'c' key before the first OS option is booted.
Once in the GRUB2 console environment enter the following commands to chainload the external disk's GRUB2 setup. In my environment the second disk was labelled by GRUB2 as (hd1):

set root='(hd1)'  
chainloader (hd1)+1  
boot

TL;DR

Partitioning & Formatting

# Update system
grindon@wheezy:~$ sudo aptitude update && sudo aptitude safe-upgrade -y

# GPT Partitioning
grindon@wheezy:~$ sudo sgdisk --zap-all /dev/sdb  
grindon@wheezy:~$ sudo sgdisk /dev/sdb  
grindon@wheezy:~$ sudo sgdisk --new=1:2048:4097 --typecode=1:EF02 --change-name=1:bios_boot --print /dev/sdb  
grindon@wheezy:~$ sudo sgdisk --new=2:6144:159743 --typecode=2:8300 --change-name=2:boot --print /dev/sdb  
grindon@wheezy:~$ sudo sgdisk --new=3:159744:6451199 --typecode=3:8300 --change-name=3:root --print /dev/sdb  
grindon@wheezy:~$ sudo sgdisk --new=4:6451200:16936959 --typecode=4:8300 --change-name=4:home --print /dev/sdb  
grindon@wheezy:~$ sudo sgdisk --new=5:16936960:27422719 --typecode=5:8300 --change-name=5:usr --print /dev/sdb  
grindon@wheezy:~$ sudo sgdisk --new=6:27422720:37908479 --typecode=6:8300 --change-name=6:var --print /dev/sdb  
grindon@wheezy:~$ sudo sgdisk --largest-new=7 --typecode=7:8200 --change-name=7:swap --print /dev/sdb

# Update kernel about GPT table
root@wheezy:/home/grindon$ echo 1 > /sys/block/sdb/device/rescan

# Backup partition table
grindon@wheezy:~$ sudo sgdisk --backup=/path/to/backup/debootstrapped_gpt_partitions.bin /dev/sdb

# Filesystem Formatting
grindon@wheezy:~$ sudo mkfs.ext2 /dev/sdb2  
grindon@wheezy:~$ for i in {3..6}; do sudo mkfs.ext4 -q /dev/sdb$i done  
grindon@wheezy:~$ sudo mkswap /dev/sdb7

Debootstrap & Configure

# Mount physical filesystems
grindon@wheezy:~$ sudo mkdir /mnt/debootstrap  
grindon@wheezy:~$ sudo mount /dev/sdb3 /mnt/debootstrap  
grindon@wheezy:~$ sudo mkdir -v /mnt/debootstrap/{boot,home,usr,var}  
grindon@wheezy:~$ sudo mount /dev/sdb2 /mnt/debootstrap/boot  
grindon@wheezy:~$ sudo mount /dev/sdb4 /mnt/debootstrap/home  
grindon@wheezy:~$ sudo mount /dev/sdb5 /mnt/debootstrap/usr  
grindon@wheezy:~$ sudo mount /dev/sdb6 /mnt/debootstrap/var

# Debootstrap Debian Stable ("Jessie") 
grindon@wheezy:~$ sudo debootstrap   
  --include=sudo,openssh-server,locales,
            linux-image-amd64,grub-pc 
  --arch amd64 stable 
  /mnt/debootstrap 
  http://http.debian.net/debian/

# Mount virtual/other filesystems
grindon@wheezy:~$ sudo mount -t proc proc /mnt/debootstrap/proc  
grindon@wheezy:~$ sudo mount -t sysfs sysfs /mnt/debootstrap/sys  
grindon@wheezy:~$ sudo mount -B /dev /mnt/debootstrap/dev

# Chroot into the new Debian 8.1 installation
grindon@wheezy:~$ LANG=C sudo chroot /mnt/debootstrap

# Configure the system
root@wheezy:/# vi /etc/fstab  
** SEE GUIDE ABOVE FOR CONFIG **
root@wheezy:/# vi /etc/network/interfaces  
** SEE GUIDE ABOVE FOR CONFIG **
root@wheezy:/# passwd root  
root@wheezy:/# usermod -a -G sudo grindon  
root@wheezy:/# echo "debootstrap-debian" > /etc/hostname  
root@wheezy:/# vi /etc/apt/sources.list  
** SEE GUIDE ABOVE FOR CONFIG **
root@wheezy:/# grub-install /dev/sdb  
root@wheezy:/# grub-mkconfig -o /boot/grub/grub.cfg  
root@wheezy:/# exit

# Unmount virtual & physical filesystems
grindon@wheezy:~$ sudo umount /mnt/debootstrap/{proc,sys,dev}  
grindon@wheezy:~$ sudo umount /dev/sdb2  
grindon@wheezy:~$ sudo umount /dev/sdb{4..6}  
grindon@wheezy:~$ sudo umount /dev/sdb3

Booting Options

grindon@wheezy:~$ sudo grub-mkconfig -o /boot/grub/grub.cfg  
--- OR ---
grub> set root='(hd1)'  
grub> chainloader (hd1)+1  
grub> boot

Final Words

Hopefully this post helps illuminate one of the several possibilities the versatile debootstrap utility offers when combined with other mounting techniques. Although arguably an edge case scenario for promoting debootstrap, I relied upon it for installing a Debian "Wheezy" system (from a Live Debian system via USB) on my Dell Studio 1558 laptop whose screen would no longer output any video of the kind. Thankfully USB boot was set as first priority in its BIOS to boot from USB by allowing me to blindly hit enter once the Live system had booted and then proceed to SSH'ing into the laptop and commence the debootstrap process.

If there is user interest I can write a guide for handling more complex filesystem configurations that may reside on a combination of mdadm, lvm, and dm-crypt utilities. I've recently (successfully) tried out a mdadm setup through debootstrap so I see no reason why a combination usage of low-level disk manipulation utilities would not work.

* I reserve the right to comment my feelings/opinions on systemd until I examine and understand its features/capabilities to a greater extent.