=======================
PXE provisioning server
=======================

Preboot eXecution Environment (PXE) is used to image bonders over a
network. It replaces CD/USB disk boot methods and allows many devices to
be provisioned at once.

The following instructions can be used to set up a new PXE boot server on any
Linux host.


System requirements
-------------------

The host should meet the following requirements:

* Two Ethernet interfaces or a single interface with VLANs. One interface will
  be used for Internet access and the other will be used to serve DHCP and PXE
* 10 GB hard disk
* 256 MB memory


Install packages
----------------

The following packages are required. Please install them using the appropriate
package manager:

* dnsmasq
* nftables
* nginx
* rsync
* whois


Configure services
------------------

Assign a static IP address to the second interface attached to the boot
network. This guide will assume this IP address is ``10.9.9.1/24`` and the
interface name is ``eth1``.

Set up forwarding and NAT::

    echo "net.ipv4.ip_forward=1" > /etc/sysctl.d/ip-forward.conf
    systemctl restart systemd-sysctl

    public_interface=$(ip -o r get to 1.1.1.1 | cut -d " " -f 5)
    cat << EOF > /etc/nftables.conf
    flush ruleset

    table ip nat_ipv4 {
            chain postrouting {
                    type nat hook postrouting priority 100; policy accept;
                    oif "$public_interface" masquerade
            }
    }
    EOF

    cat << EOF > /etc/systemd/system/nftables.service
    [Unit]
    Description=nftables
    Wants=network-pre.target
    Before=network-pre.target shutdown.target
    Conflicts=shutdown.target
    DefaultDependencies=no

    [Service]
    Type=oneshot
    RemainAfterExit=yes
    StandardInput=null
    ProtectSystem=full
    ProtectHome=true
    ExecStart=/usr/sbin/nft -f /etc/nftables.conf
    ExecReload=/usr/sbin/nft -f /etc/nftables.conf
    ExecStop=/usr/sbin/nft flush ruleset

    [Install]
    WantedBy=sysinit.target
    EOF

    systemctl enable --now nftables

Next set up dnsmasq and nginx::

    pxeinterface=eth1
    mkdir -p /srv/tftpboot
    cat << EOF > /etc/dnsmasq.conf
    interface=$pxeinterface
    dhcp-range=10.9.9.10,10.9.9.254,12h
    dhcp-boot=undionly.kpxe
    enable-tftp
    tftp-root=/srv/tftpboot
    EOF

    cat << EOF > /etc/nginx/nginx.conf
    user www-data;
    worker_processes auto;
    pid /run/nginx.pid;
    include /etc/nginx/modules-enabled/*.conf;

    events {
        worker_connections 768;
    }

    http {
        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        gzip on;

        server {
            listen 80 default_server;
            listen [::]:80 default_server;


            server_name _;

            location / {
                root /srv/tftpboot;
            }
        }
    }
    EOF

Finally set up a service to update the images nightly::

    cat << 'EOF' > /usr/local/sbin/update-bonding-pxe-images
    #!/bin/bash -e
    /usr/bin/rsync -av rsync://download.multapplied.net/oem-pxe/stable/ /srv/tftpboot/bonding-stable/
    for tarball in $(find /srv/tftpboot/bonding-stable/ -name '*.install.tar') ; do
        tar -C $(dirname $tarball) -xf $tarball
    done
    EOF
    chmod +x /usr/local/sbin/update-bonding-pxe-images

    cat << EOF > /etc/systemd/system/update-bonding-pxe-images.service
    [Unit]
    Description=Update Bonding PXE images

    [Service]
    Type=oneshot
    TimeoutStartSec=1200
    ExecStart=/usr/local/sbin/update-bonding-pxe-images
    EOF

    cat << EOF > /etc/systemd/system/update-bonding-pxe-images.timer
    [Unit]
    Description=Update Bonding PXE images daily

    [Timer]
    OnCalendar=daily
    RandomizedDelaySec=300

    [Install]
    WantedBy=timers.target
    EOF

    systemctl enable --now update-bonding-pxe-images.timer

Run the service now to get the current images without waiting for the timer to
fire::

    systemctl start update-bonding-pxe-images.service


Set up TFTP Directory
----------------------

Download the iPXE software to the TFTP boot directory::

    curl http://boot.ipxe.org/undionly.kpxe -o /srv/tftpboot/undionly.kpxe

At this point there are two ways to set up the installation:

1. Use a boot menu to allow individual settings for each installation
2. Automatically install each device booted on the PXE network as a deafult
   bonder


Use a boot menu
+++++++++++++++

A boot menu allows for setting up devices with specific versions and preloaded
configurations. This will require console access to each device as it is
installed in order to choose the options.

As an example, run this to set up a menu that can install bonding with various
options, and can also boot a rescue image and memory tester::

    cat << 'EOF' > /srv/tftpboot/menu.ipxe
    #!ipxe

    # Options
    set bonding_version 6.7
    set bonding_hostname bondingadmin.mydomain
    set bonding_password $6$7D1qZNfWj$GS0cDliKNDkyY3fDgve7R/04bAOuAHqzxZeO6wwv8ct5/00tFRNx6iBKFEB6j7hbkRWtj3yEQJiWyNEv.LklM1
    set bonding_timezone America/Vancouver
    set pxe_server_ip 10.9.9.1

    # Some menu defaults
    set menu-timeout 5000
    set submenu-timeout ${menu-timeout}
    isset ${menu-default} || set menu-default exit

    # Figure out if client is 64-bit capable
    cpuid --ext 29 && set arch x64 || set arch x86
    cpuid --ext 29 && set archl amd64 || set archl i386

    :start
    menu iPXE Extravaganza
    item --key x exit         Exit iPXE and continue BIOS boot
    item --gap --             ------------------------- Operating systems ------------------------------
    item --key b bonding      Install Bonding
    item --gap --             ------------------------------- Tools ------------------------------------
    item --key r rescue       Boot rescue image
    item --key m memtest      Run memtest86+
    item --gap --             ------------------------- Advanced options -------------------------------
    item --key c config       Configure settings
    item shell                Drop to iPXE shell
    item reboot               Reboot computer
    choose --timeout ${menu-timeout} --default ${menu-default} selected || goto cancel
    set menu-timeout 0
    goto ${selected}

    :cancel
    echo You cancelled the menu, dropping you to a shell

    :shell
    echo Type 'exit' to get the back to the menu
    shell
    set menu-timeout 0
    set submenu-timeout 0
    goto start

    :failed
    echo Booting failed, dropping to shell
    goto shell

    :reboot
    reboot

    :exit
    exit

    :config
    config
    goto start

    :back
    set submenu-timeout 0
    clear submenu-default
    goto start

    :rescue
    kernel http://download.opensuse.org/tumbleweed/repo/oss/boot/x86_64/loader/linux vga=normal install=http://download.opensuse.org/tumbleweed/repo/oss/ rescue=1
    initrd http://download.opensuse.org/tumbleweed/repo/oss/boot/x86_64/loader/initrd
    boot
    goto start

    :memtest
    chain https://boot.netboot.xyz/utils/memtest86-5.01.0 && goto main_menu ||
    goto start

    :bonding
    # Set initial variables
    set bonding_setup_type With node key
    set bonding_console Graphical
    set bonding_kernel bonding/${bonding_version}/pxeboot.Bonding.x86_64-${bonding_version}.kernel
    set bonding_initramfs bonding/${bonding_version}/pxeboot.Bonding.x86_64-${bonding_version}.initrd.xz
    set bonding_base_params rd.neednet=1 bonding.network=dhcp rd.kiwi.install.pxe rd.kiwi.install.image=http://boot.h.funktronics.ca/bonding/${bonding_version}/Bonding.x86_64-${bonding_version}.xz rd.shell
    set bonding_defaults password='${bonding_password}' timezone=${bonding_timezone} management=${bonding_hostname} valid_mgmt_cert=True mirror=httpredir.debian.org nameservers=1.1.1.1
    set bonding_kernel_params ${bonding_base_params} ${bonding_defaults}
    goto bonding-menu

    :bonding-menu
    menu Install bonding
    item --key s bonding-start Start install
    item --gap -- Set options
    item --key t bonding-setup-type Setup type: ${bonding_setup_type}
    item --key h bonding-hostname Bondingadmin: ${bonding_hostname}
    item --key c bonding-console Console: ${bonding_console}
    item --key e bonding-extra-params Extra parameters: ${bonding_extra_params}
    item --gap --
    item --key 0x08 back Back
    choose --timeout ${menu-timeout} --default ${menu-default} selected || goto cancel
    set menu-timeout 0
    goto ${selected}

    :bonding-setup-type
    menu
    item --key k bonding-setup-type-node-key With node key
    item --key d bonding-setup-type-default-bonder Default bonder
    item --gap --
    item --key 0x08 bonding Back
    choose --timeout ${menu-timeout} --default ${menu-default} selected || goto cancel
    set menu-timeout 0
    goto ${selected}

    :bonding-setup-type-node-key
    set bonding_setup_type With node key
    clear bonding_default_bonder
    goto bonding-update-kernel-params

    :bonding-setup-type-default-bonder
    set bonding_setup_type Default bonder
    set bonding_default_bonder bonding.default_bonder=1
    goto bonding-update-kernel-params

    :bonding-hostname
    echo -n Bondingadmin hostname: && read bonding_hostname
    goto bonding-update-kernel-params

    :bonding-console
    menu
    item --key g bonding-console-graphical Graphical
    item --key 1 bonding-console-serial1 Serial 1 (ttyS0)
    item --key 2 bonding-console-serial2 Serial 2 (ttyS1)
    item --gap --
    item --key 0x08 bonding Back
    choose --timeout ${menu-timeout} --default ${menu-default} selected || goto cancel
    set menu-timeout 0
    goto ${selected}

    :bonding-console-graphical
    set bonding_console Graphical
    clear bonding_console_params
    goto bonding-update-kernel-params

    :bonding-console-serial1
    set bonding_console Serial 1 (ttyS0)
    set bonding_console_params console=ttyS0,115200n8
    goto bonding-update-kernel-params

    :bonding-console-serial2
    set bonding_console Serial 2 (ttyS1)
    set bonding_console_params console=ttyS1,115200n8
    goto bonding-update-kernel-params

    :bonding-extra-params
    echo -n Extra parameters: && read bonding_extra_params
    goto bonding-menu

    :bonding-update-kernel-params
    set bonding_base_params rd.neednet=1 bonding.network=dhcp rd.kiwi.install.pxe rd.kiwi.install.image=http://${pxe_server_ip}/bonding/${bonding_version}/Bonding.x86_64-${bonding_version}.xz rd.shell
    set bonding_defaults password='${bonding_password}' timezone=${bonding_timezone} management=${bonding_hostname} valid_mgmt_cert=True mirror=httpredir.debian.org nameservers=1.1.1.1
    set bonding_kernel_params ${bonding_base_params} ${bonding_default_bonder} ${bonding_defaults} ${bonding_extra_params}
    goto bonding-menu

    :bonding-start
    kernel ${bonding_kernel} ${bonding_kernel_params}
    initrd ${bonding_initramfs}
    boot
    goto start
    EOF

You will want to edit this file and change at least 2 of the options defined
at the top:

``bonding_hostname``
    This should be set to the hostname of your bondingadmin server.

``bonding_password``
    The value here is for the password ``bonder69237``. It is recommended to
    change this. The following command will let you generate an alternative
    hash that can used for that option::

        mkpasswd -m sha-512


Automatic default bonder
++++++++++++++++++++++++

This simpler iPXE script will force install the image configured as a default
bonder. It is important to never boot anything on the PXE network that is not
intended to be overwritten with the bonding image.

The advantage of this method is that no console is needed during the setup. A
series of devices can be plugged into the network and booted. after a
sufficient period of time, they can be assumed to be installed and deployed.

Run this to set up for automatic installation::

    cat << 'EOF' > /srv/tftpboot/menu.ipxe
    #!ipxe

    # Options
    set bonding_version 6.7
    set bonding_hostname bondingadmin.mydomain
    set bonding_password $6$7D1qZNfWj$GS0cDliKNDkyY3fDgve7R/04bAOuAHqzxZeO6wwv8ct5/00tFRNx6iBKFEB6j7hbkRWtj3yEQJiWyNEv.LklM1
    set bonding_timezone America/Vancouver
    set pxe_server_ip 10.9.9.1

    # Set initial variables
    set bonding_kernel bonding/${bonding_version}/pxeboot.Bonding.x86_64-${bonding_version}.kernel
    set bonding_initramfs bonding/${bonding_version}/pxeboot.Bonding.x86_64-${bonding_version}.initrd.xz
    set bonding_base_params rd.neednet=1 bonding.network=dhcp rd.kiwi.install.pxe rd.kiwi.install.image=http://boot.h.funktronics.ca/bonding/${bonding_version}/Bonding.x86_64-${bonding_version}.xz rd.shell
    set bonding_defaults password='${bonding_password}' timezone=${bonding_timezone} management=${bonding_hostname} valid_mgmt_cert=True mirror=httpredir.debian.org nameservers=1.1.1.1
    set bonding_extra_params bonding.default_bonder=1 console=ttyS0,115200n8
    set bonding_kernel_params ${bonding_base_params} ${bonding_defaults} ${bonding_extra_params}

    kernel ${bonding_kernel} ${bonding_kernel_params}
    initrd ${bonding_initramfs}
    boot
    EOF

As with the menu version you will want to edit this file and change at least 2
of the options defined at the top:

``bonding_hostname``
    This should be set to the hostname of your bondingadmin server.

``bonding_password``
    The value here is for the password ``bonder69237``. It is recommended to
    change this. The following command will let you generate an alternative
    hash that can used for that option::

        mkpasswd -m sha-512

