# Operating system deployment

This section includes,

  • Deploying ESXI on multiple machines in an automated fashion

  • Deploying Red Hat CoreOS using iPXE server

  • Deploying RHEL using PXE server

What is PXE boot?

Preboot Execution Environment (PXE) is a client-server interface that allows computers in a network to be booted from the server before deploying the obtained OS image in local and remote offices, for PXE enabled clients. PXE network boot is performed using client-server protocols like DHCP (Dynamic Host Configuration Protocol) and TFTP (Trivial File Transfer Protocol).

# ESXi deployment

This section outlines the steps to programmatically deploy ESXi on all the bare-metal nodes.

Prerequisites

  1. ESXi ISO image is present in the HTTP file path within the installer machine.

  2. iLO account with administrative privileges required on servers for configuring ESXi via automation.

# Installation

  1. Enable Python 3 and Ansible environment as mentioned in Installer machine section of deployment guide.

  2. Execute the following command on the installer VM to point to the ESXi deployment directory.

    > $BASE_DIR/os_deployment/deploy_esxi
    

    NOTE

    BASE_DIR is defined in Installer machine section in deployment guide.

  3. Installing requirements

    > sudo sh setup.sh 
    
  4. Update the input_files/config.yml file with web server details for ESXi installation.

  5. Use the following command to edit input files/config.yml file.

    > sudo ansible-vault edit input_files/config.yml
    
  6. Example values for the input configuration is as follows:

     config:
       HTTP_server_base_url: http://10.0.x.x/
       HTTP_file_path: /usr/share/nginx/html/
       OS_type: esxi67
       OS_image_name: <ISO_image_name>.iso
       base_kickstart_filepath: kickstart_files/ks_esxi67.cfg
    

    NOTE

    The default password for the Ansible vault file is changeme.

    Acceptable values for "OS_type" variable is "esxi67" for ESXi 6.7.

  7. Update the input_files/server_details.yml file with server details for ESXi installation.

  8. Use the following command to edit input files/server_details.yml file.

    > sudo ansible-vault edit input_files/server_details.yml
    
  9. Example values for the input configuration for deploying ESXi 6.7 is as follows:

     servers:
        -  Server_serial_number: MXxxxxxDP
           ILO_Address: 10.0.x.x
           ILO_Username: username
           ILO_Password: password
           Hostname: vsphere01.twentynet.local
           Host_IP: 20.x.x.x
           Host_Username: root
           Host_Password: Password
           Host_Netmask: 255.x.x.x
           Host_Gateway: 20.x.x.x
           Host_DNS: 20.x.x.x
        - Server_serial_number: MXxxxxxDQ
           ILO_Address: 10.0.x.x
           ILO_Username: username
           ILO_Password: password
           Hostname: vsphere02.twentynet.local
           Host_IP: 20.0.x.x
           Host_Username: root
           Host_Password: Password
           Host_Netmask: 255.x.x.x
           Host_Gateway: 20.x.x.x
           Host_DNS: 20.x.x.x
    

    NOTE

    The default password for the Ansible vault file is changeme.

    It is recommended to provide a complex password for the "Host_Password" variable.

    Provide administrative privileged iLO account username and password.

  10. Run playbook to deploy ESXi.

    > ansible-playbook deploy.yml --ask-vault-pass
    

    NOTE

    In the process of ESXi deployment, ISO image contents will be forcefully moved to BASE_DIR/os_deployment/deploy_esxi/files folder and it needs to be deleted in case of space issues.

    BASE_DIR is defined in Installer machine section in deployment guide.

    NOTE

    Generic settings done as part of kickstart file for ESXi are as follows. It is recommended that the user reviews and modifies the kickstart file (kickstart_files/ks_esxi67.cfg) to suit their requirements.

    • Accept End User License Agreement (EULA).

    • clearpart --alldrives --overwritevmfs

    • install --firstdisk --overwritevmfs

    • %firstboot --interpreter=busybox

    • One standard switch vswitch0 is created with uplinks vmnic0 and vmnic1. it is assigned with the Host_IP defined in the input_files/server_details.yml input file.

    • NIC teaming is performed with vmnic0 being the active uplink and vmnic1 being the standby uplink.

    • NIC failover policy is set to --failback yes --failure-detection link --load-balancing mac --notify-switches yes.

# Red Hat CoreOS deployment using iPXE server

In this solution, an iPXE Server setup is used for Red Hat Enterprise Linux CoreOS (RHCOS) deployment. This section explains in detail the playbooks required to configure iPXE server on an RHEL 7.6 VM to perform an unattended installation of Red Hat Enterprise Linux CoreOS (RHCOS) for non-ESXi VMs and bare metal servers.

Prerequisites

  1. RHEL Ansible engine VM with Ansible 2.9.x and Python 3.6.x.

  2. Ansible Engine is configured as mentioned in the Installer machine section of this deployment guide.

  3. If the user wishes to setup the iPXE server on a separate RHEL 7.6 VM (other than the Ansible engine VM) preferably or could be bare metal server, they need to have the following minimum configuration:

    • At least 200 GB disk space

    • Two (2) CPU cores

    • 8 GB RAM

    • /var has at least 15 GB disk space allocated while partitioning

    • Static IP on the same network as the RHCOS server

    • Internet access

  4. Disable Selinux using following steps:

    • Check the status of SELINUX using command /etc/selinux/config
    > $ getenforce
    
    • Disable SELINUX
    > sudo sed -i 's/permissive/disabled/g' /etc/selinux/config /etc/selinux/config
    

    NOTE

    If the current state of "selinux" is "enforcing," replace the keyword "permissive" in the above command with "enforcing" and then run it.

    • Reboot the RHEL machine.

    • Check the status of SELINUX using command /etc/selinux/config

    > getenforce
    

# Automated Installation and Configuration of iPXE Server

Perform the steps listed below to execute the playbooks for installing and configuring iPXE server

  1. Browse to the following directory on the Ansible engine.

    > cd $BASE_DIR/os_deployment/deploy_rhcos/ 
    

    NOTE

    Refer to installer machine section of this deployment guide to know the "BASE_DIR" path.

  2. Copy Red Hat OpenShift 4.x above install files to directory, preferably under /tmp/ Example: /tmp/image/

  3. Update the inventory file "hosts" with the RHEL machine IP on which the user is trying to setup the iPXE server.

    [ansible_host]
    20.x.x.x
    
  4. Update the secrets.yml file located in BASE_DIR/os_deployment/deploy_rhcos/secrets.yml.

  5. Use the following command to edit secrets.yml file.

    > sudo ansible-vault edit $BASE_DIR/os_deployment/deploy_rhcos/secrets.yml 
    

    NOTE

    Refer to Installer machine section of this deployment guide to know the "BASE_DIR" .

    The default password for the Ansible vault file is changeme.

     # Details of Ansible User Credentials required to set-up the iPXE
    
     #Path to hpe-solutions-openshift Directory; Example: /home/openshift_admin (no trailing forwardslash required)
    
     base_path: '/home/openshift_admin'
    
     #Is Environment Airgapped; ('yes' or 'no'):
     is_environment_airgap: 'yes'
    
     ###Common Networking
     # Interface Name Example: interface_name: enp0s10f0
     interface_name: <interface_name>
    
     # Ansible Host IP Example: ansible_engine_ip: 20.0.15.115
     ansible_engine_ip: <ansible_engine_ip>
    
     # base DNS domain, Example: base_domain: ocp42.local
     base_domain: <base_domain>
    
     # Openshift sub domain, Example:app_domain: apps.ocp42.local
     app_domain: <app_domain>
    
     # Network router IP Example: gateway: 192.168.42.254
     gateway: <gateway>
    
     # Network DNS server Example: dns_server: 192.168.42.252
     dns_server: <dns_server>
    
     # DHCP range that is used for OCP nodes Example: dhcp_range: 192.168.42.204,192.168.42.210,24h
     dhcp_range: <dhcp_range>
    
     # Subnet Mask Example: net_mask: 255.255.240.0
     net_mask: <net_mask>
    
     ## OpenShift node network
     # mac address of master node 1 Example: master1_mac: 08:00:27:36:0A:01
     master1_mac: <master1_mac>
    
     # master node 1 IP address Example: master1_ip: 192.168.42.204
     master1_ip: <master1_ip>
    
     # master node 2 mac address Example: master2_mac: 08:00:27:36:0A:02
     master2_mac: <master2_mac>
    
     # master node 2 IP address Example: master2_ip: 192.168.42.205
     master2_ip: <master2_ip>
    
     # master node 3 mac address Example: master3_mac: 08:00:27:36:0A:03
     master3_mac: <master3_mac>
    
     # master node 3 IP address Example: master3_ip: 192.168.42.206
     master3_ip: <master3_ip>
    
     # worker node 1 mac address Example: worker1_mac: 08:00:27:36:0A:04
     worker1_mac: <worker1_mac>
    
     # worker node 1 IP address Example: worker1_ip: 192.168.42.207
     worker1_ip: <worker1_ip>
    
     # worker node 2 mac address Example: worker2_mac: 08:00:27:36:0A:05
     worker2_mac: <worker2_mac>
    
     # worker node 1 IP address Example: worker2_ip: 192.168.42.208
     worker2_ip: <worker2_ip>
    
     # worker node 3 mac address Example: worker3_mac: 08:00:27:36:0A:06
     worker3_mac: <worker3_mac>
    
     # worker node 3 IP address Example: worker3_ip: 192.168.42.209
     worker3_ip: <worker3_ip>
    
     # bootstrap mac address Example: bootstrap_mac: 08:00:27:36:0A:08
     bootstrap_mac: <bootstrap_mac>
    
     # bootstrap IP address Example: bootstrap_ip: 192.168.42.210
     bootstrap_ip: <bootstrap_ip>
    
     # install media details Example: image_location: "/tmp/image/"
     image_location: <image_location>
    
     # Example: rhcos-4.9.1-x86_64-live-initramfs.x86_64.img
     live_initramfs_name: <live_initramfs_name>
    
     # Example: rhcos-4.9.1-x86_64-live-kernel-x86_64
     live_kernel_name:<live_kernel_name>
    
     # Example: rhcos-4.9.1-x86_64-live-rootfs.x86_64.img
     live_rootfs_name: <live_rootfs_name>
    
  6. Run the following Ansible playbook to configure the iPXE server.

    > ansible-playbook -i hosts master.yml --ask-vault-pass
    
  7. Test the setup with MAC address by running the following command

    > curl http://localhost:8080/ipxe?mac=08:00:27:36:0A:xx 
    
  8. Copy the ignition files in " /var/lib/matchbox/ignition " and verify the status of ignition file by using MAC address of all nodes.

    > curl http://192.168.42.200:8080/ignition?mac=08:00:27:36:0A:xx 
    
  9. Boot the VM/Bare Metal using iPXE..

# RHEL deployment using PXE Server

In this solution, a PXE Server is used for RHEL 7.x OS deployment and is configured on CentOS (version: CentOS Linux release 7.6.1810 (Core)). The PXE server uses the FTP service for file distribution but can be altered to support HTTP or NFS. This section highlights the steps to configure a PXE server.

# Procedure for setting up PXE server and deploying RHEL OS

  1. Login to the CentOS server to be configured as a PXE server as a user that can run commands as root via sudo.

  2. Install packages such as DHCP, TFTP-server, vSFTPD (FTP server), and xinetd using the following command.

    > sudo yum install dhcp tftp tftp-server syslinux vsftpd xinetd
    
  3. Update the DHCP configuration file at /etc/dhcp/dhcpd.conf with the MAC addresses, IP addresses, DNS, and routing details of the installation environment. Domain search is optional. A sample DHCP

    configuration file is shown as follows.

        ddns-update-style interim;
            ignore client-updates;
            authoritative;
            allow booting;
            allow bootp;
    
            # internal subnet for my DHCP Server
            subnet 20.0.x.x netmask 255.0.0.0 {
            range 20.0.x.x 20.0.x.x;
            deny unknown-clients;
            option domain-name-servers 20.x.x.x;
            option domain-name "twentynet.local";
            option routers 20.x.x.x;
            option broadcast-address 20.255.255.255;
            default-lease-time 600;
            max-lease-time 7200;
            next-server 20.x.x.x;
            filename "pxelinux.0";
            }
    
            #######################################
            host bootstrap {
            hardware ethernet 00:50:56:xx:98:df;
            fixed-address 20.0.x.x;
            }
            host master01 {
            hardware ethernet 00:50:56:95:xx:82;
            fixed-address 20.0.x.x;
            }
            host worker01 {
            hardware ethernet 00:50:56:xx:ab:82;
            fixed-address 20.0.x.x;
            }
    
  4. Trivial File Transfer Protocol (TFTP) is used to transfer files from data server to clients without any kind of authentication. TFTP is used for ignition file loading in PXE-based environment. To configure the TFTP server, edit the configuration file /etc/xinetd.d/tftp. Change the parameter 'disable = yes' to 'disable = no' and leave the other parameters as is. To edit the /etc/xinetd.d/tftp file, execute the following command.

    > sudo vi  /etc/xinetd.d/tftp
    

    The TFTP configuration file is shown below.

        service tftp
           {
    
                socket_type = dgram
                protocol = udp
                wait = yes
                user = root
                server = /usr/sbin/in.tftpd
                server_args = -s /var/lib/tftpboot
                disable = no
                per_source = 11
                cps = 100 2
                flags = IPv4
            }
    

    Network boot related files must be placed in the tftp root directory /var/lib/tftpboot. Run the following commands to copy the required network boot files to /var/lib/tftpboot/.

    > sudo cp –v /usr/share/syslinux/pxelinux.0 /var/lib/tftpboot
    
    > sudo cp –v /usr/share/syslinux/menu.c32 /var/lib/tftpboot
    
    > sudo cp –v /usr/share/syslinux/memdisk /var/lib/tftpboot
    
    > sudo cp –v /usr/share/syslinux/mboot.c32 /var/lib/tftpboot
    
    > sudo cp –v /usr/share/syslinux/chain.c32 /var/lib/tftpboot
    
    > sudo mkdir /var/lib/tftpboot/pxelinux.cfg
    
    > sudo mkdir /var/lib/tftpboot/networkboot
    
  5. Copy the RHEL 7.6 (or above) ISO files to the PXE server. Mount it to the /mnt/ directory and then copy the contents of the ISO to the local FTP server using the following commands.

    > sudo mount –o loop OS file name /mnt/
    
    > cd /mnt/
    
    > sudo cp –av * /var/ftp/pub/
    
  6. Copy the kernel file (vmlinuz) and initrd file from /mnt to /var/lib/tftpboot/networkboot/ using the following commands.

    > sudo cp /mnt/images/pxeboot/vmlinuz /var/lib/tftpboot/networkboot/
    
    > sudo cp /mnt/images/pxeboot/initrd.img /var/lib/tftpboot/networkboot
    
  7. Unmount the ISO files using the following command.

    > sudo unmount /mnt/
    
  8. For RHEL nodes, create and utilize a new kickstart file under the folder /var/ftp/pub with the name "rhel7.cfg" using the following command.

    > sudo vi /var/ftp/pub/rhel7.cfg
    

    A sample kickstart file is shown as follows. The installation user should create a kickstart file to meet the requirements of their installation environment.

    
    firewall --disabled
    # Install OS instead of upgrade
    install
    # Use FTP installation media
    url --url="ftp://FTP_server_IP_address/pub/rhel76/"
    # Root password
    # root password can be plaintext as shown below
    # rootpw –plaintext password
    # root password is encrypted using the command “openssl passwd -1 password” and resultant output is provided for rootpw as shown below
    rootpw --iscrypted $6$uiq8l/7xEWsYXhrvaEgan4N21yhLa8K.U7UA12Th3PD11GOXvEcI40gp
    # System authorization information
    auth useshadow passalgo=sha512
    # Use graphical install
    graphical
    firstboot disable
    # System keyboard, timezone, language
    keyboard us
    timezone Europe/Amsterdam
    lang en_US
    # SELinux configuration
    selinux disabled
    # Installation logging level
    logging level=info
    # System bootloader configuration
    bootloader location=mbr
    clearpart --all --initlabel
    part swap --asprimary --fstype="swap" --size=1
    part /boot --fstype xfs --size=300
    part pv.01 --size=1 --grow
    volgroup root_vg01 pv.01
    logvol / --fstype xfs --name=lv_01 --vgname=root_vg01 --size=1 --grow
    %packages
    @^minimal
    @core
    %end
    %addon com_redhat_kdump --disable --reserve-mb='auto'
    %end
    
  9. Create a PXE menu:

    • Create a PXE menu file at the location /var/lib/tftpboot/pxelinux.cfg/default using the command.

      > sudo vi /var/lib/tftpboot/pxelinux.cfg/default
      
    • For each of the OS boot options, provide the following details:

      • MENU LABEL -- Custom name of the respective menu label.

      • KERNEL -- Kernel details of the operating system.

      • APPEND - Path of bootloader file along with path of cfg or configuration file (in case of RHEL).

    • A sample PXE menu is shown as follows.

      default menu.c32
      
      prompt 0
      
      timeout 30
      
      MENU TITLE LinuxTechi.com PXE Menu
      
      LABEL rhel76
      
      MENU LABEL RHEL76-Buedata
      
      KERNEL /rhel76/vmlinuz
      
      APPEND initrd=/rhel76/initrd.img inst.repo=ftp://FTP_server_IP_address/pub/rhel76 ks=ftp://FTP_server_IP_address/pub/rhel76-hcp.cfg
      
      LABEL rhcos-bootstrap
      
      MENU LABEL Install RHCOS4.3 sec-Bootstrap
      
      KERNEL /networkboot/rhcos-4.3.0-x86_64-installer-kernel
      
      APPEND ip=dhcp rd.neednet=1 initrd=/networkboot/rhcos-4.3.0-x86_64-installer-initramfs.img console=tty0 console=ttyS0 coreos.inst=yes coreos.inst.install_dev=sda coreos.inst.image_url= ftp://FTP_server_IP_address/pub/rhcos-4.3.0-x86_64-metal-bios.raw.gz coreos.inst.ignition_url= ftp://FTP_server_IP_address/pub/sec/bootstrap.ign
      
      LABEL rhcos-master
      
      MENU LABEL Install RHCOS4.2 sec-Master
      
      KERNEL /networkboot/rhcos-4.3.0-x86_64-installer-kernel
      
      APPEND ip=dhcp rd.neednet=1 initrd=/networkboot/rhcos-4.3.0-x86_64-installer-initramfs.img console=tty0 console=ttyS0 coreos.inst=yes coreos.inst.install_dev=sda coreos.inst.image_url= ftp://FTP_server_IP_address/pub/rhcos-4.3.0-x86_64-metal-bios.raw.gz coreos.inst.ignition_url=ftp://FTP_server_IP_address/pub/sec/master.ign
      
      LABEL rhcos-worker
      
      MENU LABEL Install RHCOS4.2 sec-Worker
      
      KERNEL /networkboot/rhcos-4.3.0-x86_64-installer-kernel
      
      APPEND ip=dhcp rd.neednet=1 initrd=/networkboot/rhcos-4.3.0-x86_64-installer-initramfs.img console=tty0 console=ttyS0 coreos.inst=yes coreos.inst.install_dev=sda coreos.inst.image_url= ftp://FTP_server_IP_address/pub/rhcos-4.3.0-x86_64-metal-bios.raw.gz coreos.inst.ignition_url=ftp://FTP_server_IP_address/pub/sec/worker.ign
      
  10. Start and enable xinetd, dhcpd and vsftpd using the following commands.

    > sudo systemctl start xinetd
    
    > sudo systemctl enable xinetd
    
    > sudo systemctl start dhcpd.service
    
    > sudo systemctl enable dhcpd.service
    
    > sudo systemctl start vsftpd
    
    > sudo systemctl enable vsftpd
    
  11. Configure SELinux for FTP.

    > sudo setsebool –P allow_ftpd_full_access 1
    
  12. Open ports in the firewall using the following firewall-cmd commands.

    > sudo firewall-cmd --add-service-ftp --permanent
    
    > sudo firewall-cmd --add-service-dhcp --permanent
    
    > sudo firewall-cmd –reload
    

NOTE

It is crucial to generate ignition files, copy them to the TFTP server, and update the path in the PXE default file. For more information about generating the ignition files, refer to the section Kubernetes manifests and ignition files in this document.