OpenShift Container Platform 4.9 on HPE DL380 and DL360 Gen10 Servers

# Operating system deployment

This section includes,

  • Deploying ESXI on multiple machines in an automated fashion

  • Deploying Red Hat CoreOS using iPXE server

  • Deploying RHEL using PXE server

What is PXE boot?

Preboot Execution Environment (PXE) is a client-server interface that allows computers in a network to be booted from the server before deploying the obtained OS image in local and remote offices, for PXE enabled clients. PXE network boot is performed using client-server protocols like DHCP (Dynamic Host Configuration Protocol) and TFTP (Trivial File Transfer Protocol).

# ESXi deployment

This section outlines the steps to programmatically deploy ESXi on all the bare-metal nodes.

Prerequisites

  1. ESXi ISO image is present in the HTTP file path within the installer machine.

  2. iLO account with administrative privileges required on servers for configuring ESXi via automation.

# Installation

  1. Enable Python 3 and Ansible environment as mentioned in Installer machine section of deployment guide.

  2. Execute the following command on the installer VM to point to the ESXi deployment directory.

    > $BASE_DIR/os_deployment/deploy_esxi

    NOTE

    BASE_DIR is defined in Installer machine section in deployment guide.

  3. Installing requirements

    > sudo sh setup.sh

  4. Update the input_files/config.yml file with web server details for ESXi installation.

  5. Use the following command to edit input files/config.yml file.

    > sudo ansible-vault edit input_files/config.yml

  6. Example values for the input configuration is as follows:

     config:
   HTTP_server_base_url: http://10.0.x.x/
   HTTP_file_path: /usr/share/nginx/html/
   OS_type: esxi67
   OS_image_name: <ISO_image_name>.iso
   base_kickstart_filepath: kickstart_files/ks_esxi67.cfg

    NOTE

    The default password for the Ansible vault file is changeme.

    Acceptable values for "OS_type" variable is "esxi67" for ESXi 6.7.

  7. Update the input_files/server_details.yml file with server details for ESXi installation.

  8. Use the following command to edit input files/server_details.yml file.

    > sudo ansible-vault edit input_files/server_details.yml

  9. Example values for the input configuration for deploying ESXi 6.7 is as follows:

     servers:
    -  Server_serial_number: MXxxxxxDP
       ILO_Address: 10.0.x.x
       ILO_Username: username
       ILO_Password: password
       Hostname: vsphere01.twentynet.local
       Host_IP: 20.x.x.x
       Host_Username: root
       Host_Password: Password
       Host_Netmask: 255.x.x.x
       Host_Gateway: 20.x.x.x
       Host_DNS: 20.x.x.x
    - Server_serial_number: MXxxxxxDQ
       ILO_Address: 10.0.x.x
       ILO_Username: username
       ILO_Password: password
       Hostname: vsphere02.twentynet.local
       Host_IP: 20.0.x.x
       Host_Username: root
       Host_Password: Password
       Host_Netmask: 255.x.x.x
       Host_Gateway: 20.x.x.x
       Host_DNS: 20.x.x.x

    NOTE

    The default password for the Ansible vault file is changeme.

    It is recommended to provide a complex password for the "Host_Password" variable.

    Provide administrative privileged iLO account username and password.

  10. Run playbook to deploy ESXi.

    > ansible-playbook deploy.yml --ask-vault-pass

    NOTE

    In the process of ESXi deployment, ISO image contents will be forcefully moved to BASE_DIR/os_deployment/deploy_esxi/files folder and it needs to be deleted in case of space issues.

    BASE_DIR is defined in Installer machine section in deployment guide.

    NOTE

    Generic settings done as part of kickstart file for ESXi are as follows. It is recommended that the user reviews and modifies the kickstart file (kickstart_files/ks_esxi67.cfg) to suit their requirements.

    • Accept End User License Agreement (EULA).

    • clearpart --alldrives --overwritevmfs

    • install --firstdisk --overwritevmfs

    • %firstboot --interpreter=busybox

    • One standard switch vswitch0 is created with uplinks vmnic0 and vmnic1. it is assigned with the Host_IP defined in the input_files/server_details.yml input file.

    • NIC teaming is performed with vmnic0 being the active uplink and vmnic1 being the standby uplink.

    • NIC failover policy is set to --failback yes --failure-detection link --load-balancing mac --notify-switches yes.

# Red Hat CoreOS deployment using iPXE server

In this solution, an iPXE Server setup is used for Red Hat Enterprise Linux CoreOS (RHCOS) deployment. This section explains in detail the playbooks required to configure iPXE server on an RHEL 7.6 VM to perform an unattended installation of Red Hat Enterprise Linux CoreOS (RHCOS) for non-ESXi VMs and bare metal servers.

Prerequisites

  1. RHEL Ansible engine VM with Ansible 2.9.x and Python 3.6.x.

  2. Ansible Engine is configured as mentioned in the Installer machine section of this deployment guide.

  3. If the user wishes to setup the iPXE server on a separate RHEL 7.6 VM (other than the Ansible engine VM) preferably or could be bare metal server, they need to have the following minimum configuration:

    • At least 200 GB disk space

    • Two (2) CPU cores

    • 8 GB RAM

    • /var has at least 15 GB disk space allocated while partitioning

    • Static IP on the same network as the RHCOS server

    • Internet access

  4. Disable Selinux using following steps:

    • Check the status of SELINUX using command /etc/selinux/config
    > $ getenforce
    • Disable SELINUX
    ​> sudo sed -i 's/permissive/disabled/g' /etc/selinux/config /etc/selinux/config

    NOTE

    If the current state of "selinux" is "enforcing," replace the keyword "permissive" in the above command with "enforcing" and then run it.

    • Reboot the RHEL machine.

    • Check the status of SELINUX using command /etc/selinux/config

    > getenforce

# Automated Installation and Configuration of iPXE Server

Perform the steps listed below to execute the playbooks for installing and configuring iPXE server

  1. Browse to the following directory on the Ansible engine.

    > cd $BASE_DIR/os_deployment/deploy_rhcos/

    NOTE

    Refer to installer machine section of this deployment guide to know the "BASE_DIR" path.

  2. Copy Red Hat OpenShift 4.x above install files to directory, preferably under /tmp/ Example: /tmp/image/

  3. Update the inventory file "hosts" with the RHEL machine IP on which the user is trying to setup the iPXE server.

    [ansible_host]
20.x.x.x

  4. Update the secrets.yml file located in BASE_DIR/os_deployment/deploy_rhcos/secrets.yml.

  5. Use the following command to edit secrets.yml file.

    > sudo ansible-vault edit $BASE_DIR/os_deployment/deploy_rhcos/secrets.yml

    NOTE

    Refer to Installer machine section of this deployment guide to know the "BASE_DIR" .

    The default password for the Ansible vault file is changeme.

     # Details of Ansible User Credentials required to set-up the iPXE

 #Path to hpe-solutions-openshift Directory; Example: /home/openshift_admin (no trailing forwardslash required)

 base_path: '/home/openshift_admin'

 #Is Environment Airgapped; ('yes' or 'no'):
 is_environment_airgap: 'yes'

 ###Common Networking
 # Interface Name Example: interface_name: enp0s10f0
 interface_name: <interface_name>

 # Ansible Host IP Example: ansible_engine_ip: 20.0.15.115
 ansible_engine_ip: <ansible_engine_ip>

 # base DNS domain, Example: base_domain: ocp42.local
 base_domain: <base_domain>

 # Openshift sub domain, Example:app_domain: apps.ocp42.local
 app_domain: <app_domain>

 # Network router IP Example: gateway: 192.168.42.254
 gateway: <gateway>

 # Network DNS server Example: dns_server: 192.168.42.252
 dns_server: <dns_server>

 # DHCP range that is used for OCP nodes Example: dhcp_range: 192.168.42.204,192.168.42.210,24h
 dhcp_range: <dhcp_range>

 # Subnet Mask Example: net_mask: 255.255.240.0
 net_mask: <net_mask>

 ## OpenShift node network
 # mac address of master node 1 Example: master1_mac: 08:00:27:36:0A:01
 master1_mac: <master1_mac>

 # master node 1 IP address Example: master1_ip: 192.168.42.204
 master1_ip: <master1_ip>

 # master node 2 mac address Example: master2_mac: 08:00:27:36:0A:02
 master2_mac: <master2_mac>

 # master node 2 IP address Example: master2_ip: 192.168.42.205
 master2_ip: <master2_ip>

 # master node 3 mac address Example: master3_mac: 08:00:27:36:0A:03
 master3_mac: <master3_mac>

 # master node 3 IP address Example: master3_ip: 192.168.42.206
 master3_ip: <master3_ip>

 # worker node 1 mac address Example: worker1_mac: 08:00:27:36:0A:04
 worker1_mac: <worker1_mac>

 # worker node 1 IP address Example: worker1_ip: 192.168.42.207
 worker1_ip: <worker1_ip>

 # worker node 2 mac address Example: worker2_mac: 08:00:27:36:0A:05
 worker2_mac: <worker2_mac>

 # worker node 1 IP address Example: worker2_ip: 192.168.42.208
 worker2_ip: <worker2_ip>

 # worker node 3 mac address Example: worker3_mac: 08:00:27:36:0A:06
 worker3_mac: <worker3_mac>

 # worker node 3 IP address Example: worker3_ip: 192.168.42.209
 worker3_ip: <worker3_ip>

 # bootstrap mac address Example: bootstrap_mac: 08:00:27:36:0A:08
 bootstrap_mac: <bootstrap_mac>

 # bootstrap IP address Example: bootstrap_ip: 192.168.42.210
 bootstrap_ip: <bootstrap_ip>

 # install media details Example: image_location: "/tmp/image/"
 image_location: <image_location>

 # Example: rhcos-4.9.1-x86_64-live-initramfs.x86_64.img
 live_initramfs_name: <live_initramfs_name>

 # Example: rhcos-4.9.1-x86_64-live-kernel-x86_64
 live_kernel_name:<live_kernel_name>

 # Example: rhcos-4.9.1-x86_64-live-rootfs.x86_64.img
 live_rootfs_name: <live_rootfs_name>

  6. Run the following Ansible playbook to configure the iPXE server.

    > ansible-playbook -i hosts master.yml --ask-vault-pass

  7. Test the setup with MAC address by running the following command

    > curl http://localhost:8080/ipxe?mac=08:00:27:36:0A:xx

  8. Copy the ignition files in " /var/lib/matchbox/ignition " and verify the status of ignition file by using MAC address of all nodes.

    > curl http://192.168.42.200:8080/ignition?mac=08:00:27:36:0A:xx

  9. Boot the VM/Bare Metal using iPXE..

# RHEL deployment using PXE Server

In this solution, a PXE Server is used for RHEL 7.x OS deployment and is configured on CentOS (version: CentOS Linux release 7.6.1810 (Core)). The PXE server uses the FTP service for file distribution but can be altered to support HTTP or NFS. This section highlights the steps to configure a PXE server.

# Procedure for setting up PXE server and deploying RHEL OS

  1. Login to the CentOS server to be configured as a PXE server as a user that can run commands as root via sudo.

  2. Install packages such as DHCP, TFTP-server, vSFTPD (FTP server), and xinetd using the following command.

    > sudo yum install dhcp tftp tftp-server syslinux vsftpd xinetd

  3. Update the DHCP configuration file at /etc/dhcp/dhcpd.conf with the MAC addresses, IP addresses, DNS, and routing details of the installation environment. Domain search is optional. A sample DHCP

    configuration file is shown as follows.

        ddns-update-style interim;
        ignore client-updates;
        authoritative;
        allow booting;
        allow bootp;

        # internal subnet for my DHCP Server
        subnet 20.0.x.x netmask 255.0.0.0 {
        range 20.0.x.x 20.0.x.x;
        deny unknown-clients;
        option domain-name-servers 20.x.x.x;
        option domain-name "twentynet.local";
        option routers 20.x.x.x;
        option broadcast-address 20.255.255.255;
        default-lease-time 600;
        max-lease-time 7200;
        next-server 20.x.x.x;
        filename "pxelinux.0";
        }

        #######################################
        host bootstrap {
        hardware ethernet 00:50:56:xx:98:df;
        fixed-address 20.0.x.x;
        }
        host master01 {
        hardware ethernet 00:50:56:95:xx:82;
        fixed-address 20.0.x.x;
        }
        host worker01 {
        hardware ethernet 00:50:56:xx:ab:82;
        fixed-address 20.0.x.x;
        }

  4. Trivial File Transfer Protocol (TFTP) is used to transfer files from data server to clients without any kind of authentication. TFTP is used for ignition file loading in PXE-based environment. To configure the TFTP server, edit the configuration file /etc/xinetd.d/tftp. Change the parameter 'disable = yes' to 'disable = no' and leave the other parameters as is. To edit the /etc/xinetd.d/tftp file, execute the following command.

    > sudo vi  /etc/xinetd.d/tftp

    The TFTP configuration file is shown below.

        service tftp
       {

            socket_type = dgram
            protocol = udp
            wait = yes
            user = root
            server = /usr/sbin/in.tftpd
            server_args = -s /var/lib/tftpboot
            disable = no
            per_source = 11
            cps = 100 2
            flags = IPv4
        }

    Network boot related files must be placed in the tftp root directory /var/lib/tftpboot. Run the following commands to copy the required network boot files to /var/lib/tftpboot/.

    > sudo cp –v /usr/share/syslinux/pxelinux.0 /var/lib/tftpboot

    > sudo cp –v /usr/share/syslinux/menu.c32 /var/lib/tftpboot

    > sudo cp –v /usr/share/syslinux/memdisk /var/lib/tftpboot

    > sudo cp –v /usr/share/syslinux/mboot.c32 /var/lib/tftpboot

    > sudo cp –v /usr/share/syslinux/chain.c32 /var/lib/tftpboot

    > sudo mkdir /var/lib/tftpboot/pxelinux.cfg

    > sudo mkdir /var/lib/tftpboot/networkboot

  5. Copy the RHEL 7.6 (or above) ISO files to the PXE server. Mount it to the /mnt/ directory and then copy the contents of the ISO to the local FTP server using the following commands.

    > sudo mount –o loop OS file name /mnt/

    > cd /mnt/

    > sudo cp –av * /var/ftp/pub/

  6. Copy the kernel file (vmlinuz) and initrd file from /mnt to /var/lib/tftpboot/networkboot/ using the following commands.

    > sudo cp /mnt/images/pxeboot/vmlinuz /var/lib/tftpboot/networkboot/

    > sudo cp /mnt/images/pxeboot/initrd.img /var/lib/tftpboot/networkboot

  7. Unmount the ISO files using the following command.

    > sudo unmount /mnt/

  8. For RHEL nodes, create and utilize a new kickstart file under the folder /var/ftp/pub with the name "rhel7.cfg" using the following command.

    > sudo vi /var/ftp/pub/rhel7.cfg

    A sample kickstart file is shown as follows. The installation user should create a kickstart file to meet the requirements of their installation environment.

    
firewall --disabled
# Install OS instead of upgrade
install
# Use FTP installation media
url --url="ftp://FTP_server_IP_address/pub/rhel76/"
# Root password
# root password can be plaintext as shown below
# rootpw –plaintext password
# root password is encrypted using the command “openssl passwd -1 password” and resultant output is provided for rootpw as shown below
rootpw --iscrypted $6$uiq8l/7xEWsYXhrvaEgan4N21yhLa8K.U7UA12Th3PD11GOXvEcI40gp
# System authorization information
auth useshadow passalgo=sha512
# Use graphical install
graphical
firstboot disable
# System keyboard, timezone, language
keyboard us
timezone Europe/Amsterdam
lang en_US
# SELinux configuration
selinux disabled
# Installation logging level
logging level=info
# System bootloader configuration
bootloader location=mbr
clearpart --all --initlabel
part swap --asprimary --fstype="swap" --size=1
part /boot --fstype xfs --size=300
part pv.01 --size=1 --grow
volgroup root_vg01 pv.01
logvol / --fstype xfs --name=lv_01 --vgname=root_vg01 --size=1 --grow
%packages
@^minimal
@core
%end
%addon com_redhat_kdump --disable --reserve-mb='auto'
%end

  9. Create a PXE menu:

    • Create a PXE menu file at the location /var/lib/tftpboot/pxelinux.cfg/default using the command.

      > sudo vi /var/lib/tftpboot/pxelinux.cfg/default

    • For each of the OS boot options, provide the following details:

      • MENU LABEL -- Custom name of the respective menu label.

      • KERNEL -- Kernel details of the operating system.

      • APPEND - Path of bootloader file along with path of cfg or configuration file (in case of RHEL).

    • A sample PXE menu is shown as follows.

      default menu.c32

prompt 0

timeout 30

MENU TITLE LinuxTechi.com PXE Menu

LABEL rhel76

MENU LABEL RHEL76-Buedata

KERNEL /rhel76/vmlinuz

APPEND initrd=/rhel76/initrd.img inst.repo=ftp://FTP_server_IP_address/pub/rhel76 ks=ftp://FTP_server_IP_address/pub/rhel76-hcp.cfg

LABEL rhcos-bootstrap

MENU LABEL Install RHCOS4.3 sec-Bootstrap

KERNEL /networkboot/rhcos-4.3.0-x86_64-installer-kernel

APPEND ip=dhcp rd.neednet=1 initrd=/networkboot/rhcos-4.3.0-x86_64-installer-initramfs.img console=tty0 console=ttyS0 coreos.inst=yes coreos.inst.install_dev=sda coreos.inst.image_url= ftp://FTP_server_IP_address/pub/rhcos-4.3.0-x86_64-metal-bios.raw.gz coreos.inst.ignition_url= ftp://FTP_server_IP_address/pub/sec/bootstrap.ign

LABEL rhcos-master

MENU LABEL Install RHCOS4.2 sec-Master

KERNEL /networkboot/rhcos-4.3.0-x86_64-installer-kernel

APPEND ip=dhcp rd.neednet=1 initrd=/networkboot/rhcos-4.3.0-x86_64-installer-initramfs.img console=tty0 console=ttyS0 coreos.inst=yes coreos.inst.install_dev=sda coreos.inst.image_url= ftp://FTP_server_IP_address/pub/rhcos-4.3.0-x86_64-metal-bios.raw.gz coreos.inst.ignition_url=ftp://FTP_server_IP_address/pub/sec/master.ign

LABEL rhcos-worker

MENU LABEL Install RHCOS4.2 sec-Worker

KERNEL /networkboot/rhcos-4.3.0-x86_64-installer-kernel

APPEND ip=dhcp rd.neednet=1 initrd=/networkboot/rhcos-4.3.0-x86_64-installer-initramfs.img console=tty0 console=ttyS0 coreos.inst=yes coreos.inst.install_dev=sda coreos.inst.image_url= ftp://FTP_server_IP_address/pub/rhcos-4.3.0-x86_64-metal-bios.raw.gz coreos.inst.ignition_url=ftp://FTP_server_IP_address/pub/sec/worker.ign

  10. Start and enable xinetd, dhcpd and vsftpd using the following commands.

    > sudo systemctl start xinetd

    > sudo systemctl enable xinetd

    > sudo systemctl start dhcpd.service

    > sudo systemctl enable dhcpd.service

    > sudo systemctl start vsftpd

    > sudo systemctl enable vsftpd

  11. Configure SELinux for FTP.

    > sudo setsebool –P allow_ftpd_full_access 1

  12. Open ports in the firewall using the following firewall-cmd commands.

    > sudo firewall-cmd --add-service-ftp --permanent

    > sudo firewall-cmd --add-service-dhcp --permanent

    > sudo firewall-cmd –reload

NOTE

It is crucial to generate ignition files, copy them to the TFTP server, and update the path in the PXE default file. For more information about generating the ignition files, refer to the section Kubernetes manifests and ignition files in this document.

← User-provisioned DNS requirements →