Being able to auto-build VMs is important for a Lab to be able to rapidly experiment. Both Debian and Red Hat (and derivatives) have a mechanism for this. In this case Red Hat has Kickstart (can't help thinking Jumpstart every time I hear it) and that's what I'm doing here, but unusually using my Debian Jessie KVM host. The basic idea should be very similar on other distros.

Strictly speaking I'm kicking CentOS instead of official Red Hat, but otherwise everything is the same. With this config I'm also leaving scope to run Preseed (Debian) on the same network/config.

It can be a bit of a pain to get this stuff working since it does require a load of rebooting and the installer is very fussy - failure to retrieve one file and the installer exits and reboots.

PXE (Network) Boot

When starting with an empty VM, this is the first step - essentially it uses DHCP to find out where to get boot files, then TFTP to fetch the files, which will be the PXELINUX bootloader, config, kernels etc. Packages that you will need installed:

• isc-dhcp-server
• tftpd-hpa
• pxelinux (and recommended syslinux-common)

This config is loosely derived from the Debian PXEBootInstall wiki article.

DHCP

You can configure your DHCP any way that works for handing out addresses (a dynamically allocated range or fixed addresses), except hosts that are network booting need the following options:

next-server ip.of.TFTP.server;
filename "pxelinux.0";

After this you should get KVM VMs fail to get pxelinux.0 with an error "Connection timed out" if no TFTP server is running or "No such file or directory" if you have a working TFTP server.

TFTP

From the default install this should be serving /srv/tftp/ but the one thing we may want to do is to force it to only listen on our build network only by editing /etc/default/tftpd-hpa and changing the line:

TFTP_ADDRESS="ip.of.build.net:69"

Then restart tftpd-hpa:

# systemctl restart tftpd-hpa.service

PXELINUX

The Debian pxelinux and syslinux-common packages puts the bootloader files in /usr/lib/PXELINUX/pxelinux.0 and /usr/lib/syslinux/modules/bios/ldlinux.c32 (assuming you don't want an EFI flavour) which we can copy to the root of our TFTP server:

# cp /usr/lib/PXELINUX/pxelinux.0 /usr/lib/syslinux/modules/bios/ldlinux.c32 /srv/tftp/

Note that you can't symlink this as tftpd-hpa will not read the symlink. Because of this you may also want to put some sort of check script (eg. with CRON) to report if the source file is newer than the one served up on TFTP.

After this KVM VMs will get further and error with "Unable to locate configuration file"

Configuration

Depending on how you intend to use your setup, you could have a common config for everything (eg. we are only kicking one version and configuration of Red Hat/CentOS) or provide scope to for multiple distros, versions, and configurations as I am. For this we need to look at how PXELINUX find's it's config. It searches a number of paths below pxelinux.cfg/ in order:

• pxelinux.cfg/<UUID>
• pxelinux.cfg/<01-MAC-address-lower-case-hyphen-separated>
• pxelinux.cfg/<Hex IP of VM capitalised>
• pxelinux.cfg/<Hex IP of VM capitalised, removing a nybble at a time>
• pxelinux.cfg/default

So, if you want to boot one config, then just use "default" else you can customise your config for the particular kick.

The exact configuration details to put in the file depends on your image (see next).

Boot Images

There isn't a lot of tidy documentation on this, but there is a decent writeup  by Brian Keefer on PXE Kickstart which is a good place to start.

In order to boot, we need the image from the CentOS ISO. In this case I mounted the image and copied out images/pxeboot/ to a separate directory below srv/tftp/

# mount -o loop /var/lib/libvirt/images/CentOS-7-x86_64-NetInstall-1503.iso <mountpoint>/
# cp -r <mountpoint>/images/pxeboot/ /srv/tftp/CentOS-7-x86_64-NetInstall-1503

This way I can select the particular boot image for the distro I want and always can trace it back to the the original ISO it came from.

IMPORTANT: After a while of use you may start to get errors about non-existent filesystems that breaks the install - this is a very simple but can waste a lot of time. When a new point release comes out the kernel modules that get downloaded automatically might not be compatible with the initial boot images you copied above. The answer is simply to update them by repeating the above with the newer images.

Configuration

For this our PXELINUX config will be:

# some comment - we like comments!
default linux
label linux
        kernel CentOS-7-x86_64-NetInstall-1503/vmlinuz
        append sdevice=eth0 load_ramdisk=1 initrd=CentOS-7-x86_64-NetInstall-1503/initrd.img network ks=nfs:ip.of.nfs.server:/path/to/kickstart/file

There are lots of options for the ks= argument available in the CentOS docs. In my case I'm using NFS since I have this setup already and can deliver ISO images and Kickstart files easily via NFS.

Kickstart File

There are various approaches to generating this including:

• Take the /root/anaconda-ks.cfg from an existing install, tweak it as needed
• Manually build a file with a text editor
• GUI tool system-config-kickstart

It's not unusual for some combination of these and other tweaks over time. There is comprehensive CentOS docs on optinos for the Kickstart file.

Probably the main thing to focus on for getting this working is the "install" option which we need to set to our installation media. If you have a full ISO (not the Netinstall) "nfs":

install
nfs --server ip.of.nfs.server --dir /path/to/kickstart/installfiles/

In this case it would assume again mounting ISOs on named sub-directories so different distros can be used for different kicks.

For network installs like I use something like:

install
url --url http://mirror.example.com/centos/7/os/x86_64 --proxy http://proxy.example.com:3128

Detailed creation the Kickstart file is beyond the scope of this article, but here is a minimal sanitised version for installing an 1024MiB / 8GiB CentOS 7 Server with GUI I run on KVM:

# Install OS instead of upgrade
install

# Use network installation from a nearby mirror via a proxy
url --url="http://mirror.of.your.choice/centos/7/os/x86_64" --proxy="http://address.of.your.proxy:3128"

# For ugly but practical use this instead of graphical (below)
#text
# Use graphical install
graphical
firstboot --disable
ignoredisk --only-use=vda
# Keyboard layouts
# old format: keyboard uk
# new format:
keyboard --vckeymap=uk --xlayouts='gb'
# System language
lang en_GB.UTF-8

# Network information
network  --bootproto=dhcp --device=eth0 --ipv6=auto --activate
# TODO grumbles about this line
network  --hostname=localhost.localdomain
# Root password
rootpw --iscrypted HASHEDPASSWORD
# System services
services --enabled="chronyd"
# System timezone
timezone Europe/London --isUtc --ntpservers=address.of.ntp.server
# X Window System configuration information
xconfig  --startxonboot
# System bootloader configuration
bootloader --append=" crashkernel=auto" --location=mbr --boot-drive=vda
#autopart --type=lvm
# Partition clearing information
clearpart --none --initlabel
part /boot --fstype=xfs --size=256 --label=/boot
part pv.01 --grow
volgroup vg00 pv.01
logvol / --vgname=vg00 --name=root --fstype=xfs --size=4096 --label=root
logvol swap --vgname=vg00 --name=swap --size=1024

# reboot after completion
reboot

%packages
@base
@core
@desktop-debugging
@fonts
@gnome-desktop
@guest-agents
@guest-desktop-agents
@input-methods
@internet-browser
@x11
chrony
kexec-tools

%end

%addon com_redhat_kdump --enable --reserve-mb='auto'

%end

%post
# do some stuff ... like maybe you want to put in some ssh keys or something for Chef
# Note that this stuff is run chrooted so "/" is actually the root of the new installation
%end

Obviously if you want a minimal (non-GUI) server for automation rather than Lab experimentation then just leave (or comment) out the GUI parts.

If you don't specify necessary information then you will find yourself thrown into the CentOS installer to complete the config:

NFS Hitches

NFSv4

All the above assumes legacy NFS (ie. v3) but v4 introduces a few changes since Kickstart seems to assume all NFS is NFSv3.

In the PXELINUX config changes for NFSv4 (CentOS/RHEL 7 only) will be:

# some comment - we like comments!
default linux
label linux
        kernel CentOS-7-x86_64-NetInstall-1503/vmlinuz
        append sdevice=eth0 load_ramdisk=1 initrd=CentOS-7-x86_64-NetInstall-1503/initrd.img network ks=nfs4:ip.of.nfs.server:/path/to/kickstart/file

In the Kickstart (.ks) file changes for NFSv4 will be:

install
nfs --server ip.of.nfs.server --opts=nfsvers=4 --dir /path/to/kickstart/installfiles/

With those it's quite happy installing from NFSv4.

NFS with sub-mounts

When you configure an export for NFS, it only applies to the filesystem the export is within. If you mount an ISO below the export point then that directory will be empty on the client. There are options that can be given like "nohide" but that only applies with single host exports.

The robust way of doing this is to have an additional export for each of your ISO mounts.