Additional Features and Functionality

OpenShift API Data Protection

Introduction

OpenShift API for Data Protection (OADP) product safeguards customer applications on OpenShift Container Platform. It offers comprehensive disaster recovery protection, covering OpenShift Container Platform applications, application-related cluster resources, persistent volumes, and internal images. OADP is also capable of backing up both containerized applications and virtual machines (VMs).

However, OADP does not serve as a disaster recovery solution for etcd or OpenShift Operators.

Installation

  • In the OpenShift Container Platform web console, click Ecosystem → Software Catalog.

  • Select an existing project name or create new project.

  • Use the Filter by keyword field to find the OADP Operator.

  • Select the OADP Operator and click Install.

  • Click Install to install the Operator in the openshift-adp project.

  • Click Ecosystem → Installed Operators to verify the installation.

Creating secret:

Create a Secret with the default name:

$ oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero

The Secret is referenced in the spec.backupLocations.credential block of the DataProtectionApplication CR when you install the Data Protection Application.

Configuring the Data Protection Application

Configure the client-burst and the client-qps fields in the DPA as shown in the following example:

apiVersion: oadp.openshift.io/v1alpha1 
kind: DataProtectionApplication 
metadata: 
  name: test-dpa 
  namespace: openshift-adp 
spec: 
  backupLocations: 
    - name: default 
      velero: 
        config: 
          insecureSkipTLSVerify: "true" 
          profile: "default" 
          region: <bucket_region> 
          s3ForcePathStyle: "true" 
          s3Url: <bucket_url> 
        credential: 
          key: cloud 
          name: cloud-credentials 
        default: true 
        objectStorage: 
          bucket: <bucket_name> 
          prefix: velero 
        provider: aws 
  configuration: 
    nodeAgent: 
      enable: true 
      uploaderType: restic 
    velero: 
      client-burst: 500  
      client-qps: 300  
      defaultPlugins: 
        - openshift 
        - aws 
        - kubevirt

Setting Velero CPU and memory resource allocations

You can set the CPU and memory resource allocations for the Velero pod by editing the DataProtectionApplication custom resource (CR) manifest.

Prerequisites

  • You must have the OpenShift API for Data Protection (OADP) Operator installed.

Procedure

  • Edit the values in the spec.configuration.velero.podConfig.ResourceAllocations block of the DataProtectionApplication CR manifest, as in the following example:
apiVersion: oadp.openshift.io/v1alpha1 
kind: DataProtectionApplication 
metadata: 
  name: <dpa_sample> 
spec: 
# ... 
  configuration: 
    velero: 
      podConfig: 
        nodeSelector: <node_selector>  
        resourceAllocations:  
          limits: 
            cpu: "1" 
            memory: 1024Mi 
          requests: 
            cpu: 200m 
            memory: 256Mi

Specify the node selector to be supplied to Velero podSpec.

The resourceAllocations listed are for average usage.

Backup process:

  • Goto Installed Operators -> OADP Operator -> backup -> create backup

  • Provide name and required details:

FIGURE 81. Creating a backup using OADP

  • Click create

AirGap Deployment

Deploy OpenShift Container Platform using Airgap Method

This sections of documents describes about deploying the OpenShift Container Platform through disconnected environment.

  1. Create YUM repo server

  2. Create Mirror registry

  3. OpenShift Deployment

  • Prerequisites:

we will be using one server for all the below services and which will be having internet access to download the images.

  • Download server

  • YUM server

  • Mirror Registry

Download/YUM/Mirror registry server requirements

a) Recommended Operating system RHEL 9.6

b) At least 500 GB disk space (especially in the "/" partition), 4 CPU cores and 16GB RAM.

c) OS disk: 2x 1.6 TB ; Data disk: ~2 TB

d) Provide the required values in input.yaml file (vi /opt/hpe-solutions-openshift/DL-LTI-Openshift/input.yaml )

e) Setup the Download server to configure the nginx, development tools and other python packages required for LTI installation.

Navigate to the directory, cd /opt/hpe-solutions-openshift/DL-LTI-Openshift/ and run the below command.

  'sh setup.sh'

As part of setup.sh script it will create nginx service, so user must download and copy rhel 9.6 DVD ISO to /usr/share/nginx/html/

  1. Create Yum Repo server

a) Navigate to /opt/hpe-solutions-openshift/DL-LTI-Openshift/ folder and update the hosts file with the yumrepo server details.

b) Navigate to yum folder

cd /opt/hpe-solutions-openshift/DL-LTI-Openshift/airgap/yum

c) Run the below command to create yum repo server

ansible-playbook -i /opt/hpe-solutions-openshift/DL-LTI-Openshift/hosts playbooks/create_local_yum_repo.yaml
  1. Mirror Registry

a) Navigate to folder "/opt/hpe-solutions-openshift/DL-LTI-Openshift/".

For airgap deployment provide the below values in input.yaml file

vi /opt/hpe-solutions-openshift/DL-LTI-Openshift/input.yaml

# fill the below values for the airgap deployment
is_environment_airgap: 'yes'
mirror_registry_ip:
mirror_registry_fqdn:
LOCAL_REGISTRY:
LOCAL_REPOSITORY:
ARCHITECTURE:

b) Navigate to mirror_registry folder

cd /opt/hpe-solutions-openshift/DL-LTI-Openshift/airgap/mirror_registry

c) Download and install the mirror registry

ansible-playbook playbooks/download_mirror_registry_package.yaml
ansible-playbook playbooks/install_mirror_registry.yaml

generate ssl certificates

  ansible-playbook playbooks/generate_ssl_certs.yaml

run the below commands to copy the above generated ssl certs

  cp certs/ssl.key quay-install/quay-config/
  cp certs/ssl.cert quay-install/quay-config/
  cat certs/rootCA.pem >> quay-install/quay-config/ssl.cert
  mkdir -p /etc/containers/certs.d/<mirror_registry_fqdn>        # Here provide you mirror registryg fqdn
  cp certs/rootCA.pem  /etc/containers/certs.d/<mirror_registry_fqdn>/ca.crt
  cp certs/rootCA.pem /etc/pki/ca-trust/source/anchors/
  sudo update-ca-trust extract
  systemctl restart quay-app

d) execute the playbook site.yaml

site.yaml file contains the following playbooks

    - import_playbook: playbooks/download_openshift_components.yaml  # it will download ocp related images, client and installer
    - import_playbook: playbooks/create_json_pull_secret.yaml
    - import_playbook: playbooks/update_json_pull_secret.yaml
    - import_playbook: playbooks/mirroring_ocp_image_repository.yaml
  1. For OpenShift Solution Deployment follow the existing process listed for OpenShift Deployment

Red Hat Advanced Cluster Management for Kubernetes

Introduction

Red Hat Advanced Cluster Management for Kubernetes provides end-to-end management visibility and control to manage your Kubernetes environment. Take control of your application modernization program with management capabilities for cluster creation, application lifecycle, and provide security and compliance for all of them across data centers and hybrid cloud environments. Clusters and applications are all visible and managed from a single console, with built-in security policies. Run your operations from anywhere that Red Hat OpenShift Container Platform runs, and manage your Kubernetes clusters.

With Red Hat Advanced Cluster Management for Kubernetes:

  • Work across a range of environments, including multiple data centers, private clouds and public clouds that run Kubernetes clusters.

  • Easily create OpenShift Container Platform Kubernetes clusters and manage cluster lifecycle in a single console.

  • Enforce policies at the target clusters using Kubernetes-supported custom resource definitions.

  • Deploy and maintain day two operations of business applications distributed across your cluster landscape.

Figure 19 shows Architecture of Advanced Cluster Management for Kubernetes

FIGURE 82. Multi-Cluster Management with Red Hat Advanced Cluster Management

Installing Red Hat Advanced Cluster Management from the console

Prerequisites

  • Hub Cluster

    OpenShift Container Platform 4.x successfully deployed.

    Operator Hub availability.

  • Managed Clusters

    OpenShift 4.x on from public cloud providers (Amazon Web Services, Google Cloud, IBM and Microsoft Azure) or Private clouds (Openstack, OpenShift).

Flow Diagram

FIGURE 83. RedHat Advanced Cluster Management Solution Flow Diagram

Installation

Following steps needs to perform prior to installing Advanced Cluster Management from the OpenShift Console.

  • Create Namespace

  • Create Pull Secrets

Follow below steps to creating namespace.

  1. Create a hub cluster namespace for the operator requirements:

    • In the OpenShift Container Platform console navigation, select Administration -> Namespaces.

    • Select create Namespace.

  2. Provide a name for your namespace. This is the namespace that you use throughout the installation process.

    NOTE

    The value for namespace might be referred to as Project in the OpenShift Container Platform environment.

Follow below steps to create secrets.

  1. Switch your project namespace to the one that you created in step 1. This ensures that the steps are completed in the correct namespace. Some resources are namespace specific.

    I. In the OpenShift Container Platform console navigation, select Administration -> Namespaces.

    II. Select the namespace that you created in step 1 from the list.

  2. Create a pull secret that provides the entitlement to the downloads.

    I. Copy your OpenShift Container Platform pull secret from cloud.redhat.com

    II. In the OpenShift Container Platform console navigation, select Workloads -> Secrets.

    III. Select Create -> Image Pull Secret.

    IV. Enter a name for your secret.

    V. Select Upload Configuration File as the authentication type.

    VI. In the Configuration file field, paste the pull secret that you copied from cloud.redhat.com.

    VII. Select Create to create the pull secret.

Installing Advanced Cluster Management Operator

  1. In the OpenShift Container Platform console navigation, select  Ecosystem -> Software Catalog.

  2. Select an existing project name or create new project.

  3. Select Red Hat Advanced Cluster Management.

  4. Select Install.

    FIGURE 84. RedHat Advanced Cluster Management Operator deployment

  5. Update the values, if necessary.

    FIGURE 85. RedHat Advanced Cluster Management Operator

  6. Select specific namespace on the cluster for the Installation Mode option.

    I. Select open-cluster-management namespace from the drop down menu.

  7. selected by default for the Update Channel option.

  8. Select an Approval Strategy:

    I. Automatic specifies that you want OpenShift Container Platform to upgrade Advanced Cluster Management for kubernetes Operator.

    II. Manual specifies that you want to have control to upgrade Advanced Cluster Management for kubernetes Operator manually

  9. Select Install.

    FIGURE 86. Deployed RedHat ACM Operator

  10. The Installed Operators page is displayed with the status of the operator.

Create the MultiClusterHub custom resource

  1. In the OpenShift Container Platform console navigation, select Installed Operators -> Advanced Cluster Management for Kubernetes.

  2. Select the MultiClusterHub tab.

  3. Select Create MultiClusterHub.

  4. Update the values, according to your needs.

  5. Tip: You can edit the values in the YAML file by selecting YAML View. Some of the values are only available in the YAML view. The following example shows some sample data in the YAML view:

    apiVersion: operator.open-cluster-management.io/v1

kind: MultiClusterHub

metadata:

namespace: << newly created namespace>>

name: multiclusterhub

spec:

imagePullSecret: <secret>

    Add the pull secret that you created to the imagePullSecret field on the console. In t he YAML View, confirm that the namespace is your project namespace.

  6. Select Create to initialize the custom resource. It can take up to 10 minutes for the hub to build and start.

    After the hub is created, the status for the operator is Running on the Installed Operators page.

Access the Advanced Cluster Management console for the hub

  1. Red Hat Advanced Cluster Management for Kubernetes web console is integrated with the Red Hat OpenShift Container Platform web console as a console plug-in. You can access Red Hat Advanced Cluster Management within the OpenShift Container Platform console from the top navigation drop-down menu, which now provides two options: Administrator and Fleet Management. The menu initially displays Administrator.

    FIGURE 87. ACM console

  2. To create/ import an existing cluster from Advanced Cluster Management console, click on Clusters.

Import an existing Cluster using Advanced Cluster Management console

  1. In the Cluster list tab, Click Import cluster.

  2. It displays a window as below, there you can provide Cluster name, Cloud (public or on-premise), Environment and labels are optional.

FIGURE 88. Importing existing cluster to ACM

  1. Once you provide all the details, Click on Generate Command.

FIGURE 89. Generated command from ACM

  1. Copy the command generated, Run it on imported cluster.

  2. After running the command, we should navigate to Advanced Cluster Management console there we can find the status of imported cluster details in cluster option as per below snap.

FIGURE 90. Imported cluster details to ACM Console

Red Hat Advanced Cluster security for Kubernetes

Red Hat Advanced Cluster Security (RHACS) for Kubernetes is the pioneering Kubernetes-native security platform, equipping organizations to build, deploy, and run cloud-native applications more securely. The solution helps protect containerized Kubernetes workloads in all major clouds and hybrid platforms. Central services include the user interface (UI), data storage, RHACS application programming interface (API), and image scanning capabilities. RHACS Cloud Service allows you to secure self-managed clusters that communicate with a Central instance. The clusters you secure, called Secured Clusters.

Prerequisites

  1. Access to a Red Hat OpenShift Container Platform cluster using an account with Operator installation permissions.
  2. Red Hat OpenShift Container Platform 4.21 or later. For more information, see Red Hat Advanced Cluster Security for Kubernetes Support Policy

Install the Red Hat Advanced Cluster Security Operator

  1. Navigate in the web console to the Ecosystem -> Software Catalog page.

  2. Select an existing project name or create new project.

  3. If Red Hat Advanced Cluster Security for Kubernetes is not displayed, enter Advanced Cluster Security into the Filter by keyword box to find the Red Hat Advanced Cluster Security for Kubernetes Operator.

  4. Select the Red Hat Advanced Cluster Security for Kubernetes Operator to view the details page.

  5. Read the information about the Operator, and then click Install.

  6. On the Install Operator page:

    a. Keep the default value for Installation mode as All namespaces on the cluster.

    b. Choose a specific namespace in which to install the Operator for the Installed namespace field. Install the Red Hat Advanced Cluster Security for Kubernetes Operator in the rhacs-operator namespace.

    c. Select automatic or manual updates for Update approval.

  7. Click Install.

Verification: After the installation completes, navigate to Ecosystem > Installed Operators to verify that the Red Hat Advanced Cluster Security for Kubernetes Operator is listed with the status of Succeeded.

FIGURE 91. Red Hat ACS Operator Installation Install the Central Red Hat Advanced Cluster Security

  1. On the Red Hat OpenShift Container Platform web console, navigate to the Ecosystem > Installed Operators page.
  2. Select the Red Hat Advanced Cluster Security for Kubernetes Operator from the list of installed Operators.
  3. If you have installed the Operator in the recommended namespace, OpenShift Container Platform lists the project as rhacs-operator, Select Project: rhacs-operator → Click Create project.Operator
  4. Enter the new project name (stackrox) and click Create. Red Hat recommends that you use stackrox as the project name.
  5. Under the Provided APIs section, select Central. Click Create Central.

FIGURE 92. Central Red Hat Advanced Cluster Security (ACS) Installation

  1. Configure via yaml view, keep default values, and click create.
  2. In stackrox-central-services details we will find Admin credentials information.

FIGURE 93. Stackrox-central-services details

  1. Click central-htpasswd below Admin Password Secret Reference to find the central login credentials and note the htpasswd username(admin) and password.
  2. Navigate to Networking > Routes to find the central url.

FIGURE 94. Stackrox central url

  1. Log in to the central console using htpasswd credentials.

FIGURE 95. Central console dashboard

Create init bundle using RHACS portal

  1. On the RHACS portal, navigate to Platform Configuration > Integrations.
  2. Navigate to the Authentication Tokens section and click Cluster Init Bundle.

FIGURE 96. Create init bundle

  1. Click Create bundle.
  2. Enter a name for the cluster init bundle and click Download.

Apply the init bundle by creating a resource on the secured cluster

Before installing secured clusters, we must use the init bundle to create the required resources on the cluster that will allow the services on the secured clusters to communicate with Central. Use any one of the following methods.

  1. Method 1: Using the CLI

    a. Run the following commands to create the resources:

    oc create namespace stackrox

    b. (Specify the file name of the init bundle containing the secrets)

    oc create -f <init_bundle.yaml> -n <stackrox>

    c. Specify the name of the project where secured cluster services will be installed

  2. Method 2 : using the web console

    a. In the OpenShift Container Platform web console on the cluster that you are securing, in the top menu, click + to open the Import YAML page. You can drag the init bundle file or copy and paste its contents into the editor, and then click Create.

    FIGURE 97. Resource creation on secured cluster

Install secured cluster services within same cluster

Prerequisites

  1. If using OpenShift Container Platform, ensure the installation of version 4.21 or later.
  2. Installation of the RHACS Operator and generation of an init bundle that is applied to the cluster.

Procedure

  1. On the OpenShift Container Platform web console, navigate to the Ecosystem > Installed Operators page and select the RHACS Operator.
  2. Click Secured Cluster from the central navigation menu in the Operator details page.
  3. Click Create Secured Cluster.
  4. Select one of the following options in the Configure via field: either Form view or Yaml view.
  5. Enter the new project name by accepting or editing the default name. The default value is stackrox-secured-cluster-services.
  6. Keep all values default for stackrox-secured-cluster-services for same cluster.
  7. Now the stackrox-secured-cluster-services are ready to perform activity.

Red Hat Quay

Red Hat Quay is an enterprise-quality container registry. Use Red Hat Quay to build and store container images, then make them available to deploy across your enterprise. The Red Hat Quay Operator provides a simple method to deploy and manage Red Hat Quay on an OpenShift cluster.

Note

This section is required to executed only if user has selected Red Hat OpenShift Data Foundation (ODF) as storage option. Skip this section if HPE Alletra is the storage option.

This section explains configuring Red Hat Quay v3.16 container registry on existing OpenShift Container Platform 4.21 using the Red Hat Quay Operator.

Installation of Red Hat Quay Operator

  1. Log in to Red Hat OpenShift Container Platform console, select Ecosystem → Software Catalog.

  2. Select an existing project name or create new project.

  3. In the search box, type Red Hat Quay and select the official Red Hat Quay Operator provided by Red Hat. This directs you to the Installation page, which outlines the features, prerequisites, and deployment information.

  4. Select Install. This directs you to the Operator Installation page.


FIGURE 98. Red Hat Quay operator installation

  1. Update the details for Update channel, Installation mode, Installed Namespace, and update approval as shown in the following figure.


FIGURE 99. Red Hat Quay operator installation details

  1. Select Install.

  2. After installation you can view the operator in the Installed Operators tab.


FIGURE 100. Red Hat Quay operator installed successfully

Create the Red Hat Quay Registry

  1. Click the Quay Registry tab → Create QuayRegistry.


FIGURE 101. Red Hat Quay registry creation

  1. Provide details like name and click create.

FIGURE 102. Red Hat Quay registry creation yaml details

  1. Quay registry is created and appears as shown in the following figure.

FIGURE 103. Red Hat Quay registry created successfully

  1. Click Create Quay Registry to see registry information.

FIGURE 104. Red Hat Quay registry created details

  1. Click Registry Endpoint to open registry dashboard.

If dashboard is not accessible add the entry of registry url to /etc/hosts as follows.

haproxy-ip  <Registry Endpoint URL>
haproxy-ip example-registry-quay-openshift-operators.apps.ocp.ocpdiv.local
  1. Quay registry dashboard login page appears as shown in the following figure.

FIGURE 105. Red Hat Quay account login page

  1. Enter details as shown and click Create Account.

  2. Click sign in and enter details provided while creating the account.

Veeam Kasten

Kasten K10 by Veeam is a cloud-native data management platform purpose-built for Kubernetes applications. It provides reliable backup, restore, disaster recovery, and application mobility for workloads running on Kubernetes distributions, including Red Hat OpenShift.

Kasten K10 takes an application-centric approach to backup. Instead of only protecting storage volumes, it captures all the resources that define an application, including Kubernetes objects, configurations, secrets, and persistent volumes. This ensures consistent application recovery whether restoring within the same cluster or migrating to another OpenShift cluster.

Prerequisites

  1. A running Red Hat OpenShift cluster with sufficient CPU, memory, and storage.
  2. Cluster admin privileges for installation.
  3. A configured CSI (Container Storage Interface) driver (Internal ODF, External Ceph, or HPE Alletra Storage MP).
  4. A default StorageClass with snapshot support.
  5. An object storage bucket (e.g., MinIO) configured as the backup target.
  6. Ingress or Route access for the Kasten K10 dashboard.

NOTE

The VolumeSnapshotClass associated with the storage class used for Kasten backups must include the annotation: k10.kasten.io/is-snapshot-class: "true"

Deploying MinIO for Kasten Backup Storage

  1. Prepare an Ubuntu machine with internet access.

  2. Download and install the MinIO server binary:

    wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
sudo mkdir -p /minioap
sudo chown $(whoami):$(whoami) /minioap
./minio server /minioap

    The MinIO service runs at http://<Ubuntu-IP>:9000 with default credentials minioadmin:minioadmin.

  3. Configure MinIO for Kasten:

    mc alias set myminio http://<Ubuntu-IP>:9000 minioadmin minioadmin
mc admin user add myminio <username> <password>
mc admin policy attach myminio readwrite --user <username>

  4. Create a bucket for Kasten backups:

    1. Log in to the MinIO console using the new credentials.
    2. Click Create Bucket in the MinIO console.
    3. Name the bucket, e.g., kasten-backup.
    4. The bucket should now appear in the console.

    The MinIO server is now ready as an object store backend for Kasten K10 backups.

Installing Kasten K10 Operator on OpenShift

  1. Log in to the OpenShift web console as a cluster admin.
  2. Navigate to Ecosystem -> Software Catalog.
  3. Select an existing project name or create new project.
  4. Search for Veeam Kasten and click Install.

FIGURE 106. Veeam Kasten Operator Installation

  1. Choose installation settings:
    • Installation Mode: All namespaces on the cluster
    • Installed Namespace: kasten-io (auto-created)
  2. Wait until the operator status shows Succeeded.

FIGURE 107. Veeam Kasten Operator successfully installed

Creating the Kasten K10 Instance

  1. Go to Ecosystem ->Installed Operators -> select Veeam Kasten.
  2. Under the K10 tab, click Create K10.
  3. Keep default values and click Create.

FIGURE 108. Kasten instance created

  1. Verify pod status:
    oc get pods -n kasten-io

Exposing the K10 Dashboard

  1. Get K10 services:
    oc get svc -n kasten-io
  2. Expose the gateway service using an OpenShift route:
    oc expose svc gateway -n kasten-io
  3. Retrieve the route URL:
    oc get route -n kasten-io
  4. Access the dashboard at http://gateway-kasten-io.apps.<cluster-domain>/k10/.

FIGURE 109. Kasten Dashboard

Backup and Restore Using Kasten K10

Creating a Location Profile (MinIO)
  1. Log in to the Kasten K10 dashboard.
  2. Navigate to Profiles -> Location -> New Profile.
  3. Fill details as below:
    • Access Key / Secret Key: MinIO credentials
    • Endpoint URL: http://<MinIO-IP>:9000
    • Bucket Name: kasten-backup
    • Region: us-east-1
    • Enable Skip TLS Verification if using HTTP.

FIGURE 110. Creating Location Profile

FIGURE 111. Location Profile created

Creating a Backup Policy
  1. Go to Policies -> Create New Policy.
  2. Set policy name.
  3. Select Snapshot under Actions.
  4. Set Backup Frequency as per your requirement.
  5. Enable Backup via Snapshot Export and choose the MinIO Location Profile.
  6. Select target namespace.
  7. Save and manually trigger the policy for initial backup.

FIGURE 112. Backup Policy

  1. Monitor the progress of the backup policy directly from the Kasten dashboard.

FIGURE 113. Running backup policy

Grant Required Permissions Before Restore

Before restoring an application or namespace, ensure that the target namespace has the appropriate Security Context Constraints (SCC) assigned to allow restored pods to run successfully.

Run the following command to grant the anyuid SCC to the default service account in the target namespace. Replace <namespace> with the namespace where the application will be restored:

oc adm policy add-scc-to-user anyuid -z default -n <namespace>
Restoring a Namespace
  1. Navigate to Restore Points.
  2. Select a restore point.
  3. Choose the target namespace (existing or new).
  4. Confirm Restore and wait until the pods are recreated.

FIGURE 114. Restoring the namespace

FIGURE 115. Restore successfully completed

Cross-Cluster Restore
  1. Ensure Kasten K10 is deployed in the target cluster and has access to the same MinIO bucket.
  2. In the target cluster, create the same Location Profile pointing to the MinIO bucket.
  3. Go to Policies -> Create New Policy.
  4. Enter a policy name.
  5. Under Actions, select Import.
  6. Set the Import Frequency.
  7. Under Config Data for Import, paste the configuration text from the backup policy on the source cluster.
  8. Select the Profile for Import.
  9. Save the policy and run it manually for the first import.
  10. Navigate to Restore Points.
  11. Choose the desired Restore Point.
  12. Select a Namespace in the target cluster where the application will be restored.
  13. Confirm Restore. The application will be recreated with its persistent data and cluster resources.

FIGURE 116. Cross cluster restore

Backup and Restore using Velero

Introduction

Velero is an open-source tool used for backup and disaster recovery of Kubernetes cluster resources and persistent volumes. It enables users to safely back up, restore, and migrate workloads across clusters. Velero supports multiple storage backends such as AWS S3, and integrates with Container Storage Interface (CSI) for volume snapshots.

This section covers the installation and configuration of Velero, validation through backup and restore scenarios, and an alternative YAML-based backup approach for critical OpenShift resources.

Install and Configure Velero

Velero is used for backup and disaster recovery of Kubernetes resources.

  1. Download the Velero binary:
wget https://github.com/vmware-tanzu/velero/releases/download/v1.17.2/velero-v1.17.2-linux-amd64.tar.gz
  1. Extract the package:
tar -xvf velero-v1.17.2-linux-amd64.tar.gz
  1. Move the binary to system path:
sudo mv velero-v1.17.2-linux-amd64/velero /usr/local/bin/
  1. Verify Velero installation:
velero version
  1. Create AWS credentials file:
vi credentials-velero

Add the following content:

[default]
aws_access_key_id=<Access Key ID>
aws_secret_access_key=<Secret Access Key>
  1. Install Velero with S3 backend:
velero install \
--provider aws \
--plugins velero/velero-plugin-for-aws:v1.8.0 \
--bucket vel-backups \
--backup-location-config region=ap-southeast-2 \
--secret-file ./credentials-velero \
--features=EnableCSI \
--use-volume-snapshots=true

NOTE: Update bucket name and region as per your environment.

  1. Verify installation:
oc get pods -n velero

Expected output:

NAME                      READY   STATUS    RESTARTS   AGE
velero-xxxxxxxxxx-xxxxx   1/1     Running   0          XXs

Test scenarios of Backup and restore using Velero

VM Backup and Restore using Velero

  1. Stop the VM before backup for consistency:
oc patch vm rhel9-vm -n default \
--type merge -p '{"spec":{"runStrategy":"Halted"}}'
  1. Create backup:
velero backup create rhel9-vm-backup \
--include-namespaces default \
--snapshot-volumes=true \
--wait
  1. Verify backup:
velero backup get
velero backup describe rhel9-vm-backup --details
  1. Delete VM and PVC:
oc delete vm rhel9-vm -n default
oc delete pvc rhel9-vm -n default
  1. Restore VM:
velero restore create rhel9-vm-restore \
--from-backup rhel9-vm-backup \
--wait
  1. Verify restored resources:
velero restore get
oc get vm
oc get pvc
  1. Start restored VM:
oc patch vm rhel9-vm -n default \
--type merge -p '{"spec":{"running":true}}'
  1. Validation:
  • VM is running
  • PVC is restored
  • Network connectivity is working

Application Backup and Restore using Velero with NFS

NFS Server Setup
sudo dnf install -y nfs-utils --transient
sudo systemctl enable --now nfs-server
sudo mkdir -p /var/nfs/velero
sudo chmod 777 /var/nfs/velero

Edit exports:

vi /etc/exports

Add:

/var/nfs/velero *(rw,sync,no_root_squash,no_subtree_check)

Apply:

sudo exportfs -v
showmount -e <NFS-SERVER-IP>

Application Deployment

  1. Create namespace:
oc create namespace test-app

  1. Create PV and PVC (NFS आधारित storage)

  2. Deploy Apache application and expose via Service and Route

  3. Write sample data:

echo "Apache application running with NFS storage" > /usr/local/apache2/htdocs/test.txt
  1. Verify:
curl http://<route-url>/test.txt

Backup Application

velero backup create apache-backup \
--include-namespaces test-app \
--snapshot-volumes=false

Verify:

velero backup get

FIGURE 117. AWS Bucket for Backup

Restore Application after Cluster Reinstallation

  1. Restore NFS data:
tar -xzvf apache-nfs-backup.tar.gz -C /var/nfs/velero

  1. Install Velero again (same steps as above)

  2. Restore application:

velero restore create --from-backup apache-backup
  1. Verify:
oc get all -n test-app
oc get pvc -n test-app
oc get pv
  1. Validate application:
curl http://<route-url>/test.txt

Expected output:

Apache application running with NFS storage

OpenShift Backup and Restore Using YAML (Operators, CRDs, and Workloads)

This approach is used for backing up critical OpenShift configurations using YAML manifests.

Backup Steps

Operator Metadata
oc get subscription --all-namespaces -o yaml > subscriptions.yaml
oc get operatorgroup --all-namespaces -o yaml > operatorgroups.yaml
oc get csv --all-namespaces -o yaml > csvs.yaml
oc get crd -o yaml > crds.yaml
Operator Custom Resources
for crd in $(oc get crd -o name); do
  oc get $crd -A -o yaml >> crs.yaml
done
Workloads and Namespaces
for ns in $(oc get ns -o jsonpath='{.items[*].metadata.name}'); do
  oc get all -n $ns -o yaml > ${ns}-resources.yaml
done
Persistent Storage
oc get pvc --all-namespaces -o yaml > pvcs.yaml
oc get pv -o yaml > pvs.yaml

Restore Steps

Apply resources in the correct order:

  1. CRDs
oc apply -f crds.yaml
  1. OperatorGroups
oc apply -f operatorgroups.yaml
  1. Subscriptions
oc apply -f subscriptions.yaml

Verify operators:

oc get csv --all-namespaces
  1. Custom Resources
oc apply -f crs.yaml
  1. Workloads
for file in *-resources.yaml; do
  oc apply -f $file
done

oc apply -f pvcs.yaml
oc apply -f pvs.yaml

Observability and Monitoring for VM Fleet

Introduction

Observability for virtual machine fleets is critical in multi-cluster environments to monitor performance, resource utilization, and overall health. Red Hat Advanced Cluster Management (ACM) provides centralized observability across clusters, enabling administrators to visualize metrics for virtual machines, nodes, and workloads from a single interface.

This section covers enabling observability using ACM, configuring object storage using MinIO, and visualizing VM fleet metrics using Grafana dashboards.

Prerequisites

Before enabling observability, ensure the following:

  • Red Hat Advanced Cluster Management (ACM) is installed
  • Hub cluster is configured and ready
  • Managed clusters are imported into the hub
  • Object storage (S3-compatible such as MinIO) is available for storing metrics data

Enable Observability

Step 1 — Deploy MinIO on Hub Cluster

  1. Create namespace:
oc create namespace minio
  1. Deploy MinIO:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: minio
  namespace: minio
spec:
  replicas: 1
  selector:
    matchLabels:
      app: minio
  template:
    metadata:
      labels:
        app: minio
    spec:
      containers:
      - name: minio
        image: quay.io/minio/minio:latest
        args:
        - server
        - /data
        - --console-address
        - ":9001"
        env:
        - name: MINIO_ROOT_USER
          value: admin
        - name: MINIO_ROOT_PASSWORD
          value: password123
        ports:
        - containerPort: 9000
          name: api
        - containerPort: 9001
          name: console
        volumeMounts:
        - name: data
          mountPath: /data
      volumes:
      - name: data
        emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: minio
  namespace: minio
spec:
  selector:
    app: minio
  ports:
  - name: api
    port: 9000
    targetPort: 9000
  - name: console
    port: 9001
    targetPort: 9001
  type: ClusterIP

Apply:

oc apply -f minio.yaml

Verify:

oc rollout status deployment minio -n minio
oc get pods -n minio

Step 2 — Expose MinIO Console (Optional)

oc create route edge minio-console \
--service=minio \
--port=console \
-n minio

Access the console using the route:

oc get route -n minio

Login credentials:

  • Username: admin
  • Password: password123

Step 3 — Create Bucket for ACM

Create a bucket named:

acm-observability

Configure access:

mc alias set local http://<minio-ip>:9000 admin password123
mc ls local
mc admin user add local user1 Password1234
mc admin policy attach local readwrite --user user1

Step 4 — Create Observability Namespace

oc create namespace open-cluster-management-observability --dry-run=client -o yaml | oc apply -f -

Step 5 — Register Observability Addon

cat <<EOF | oc apply -f -
apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ClusterManagementAddOn
metadata:
  name: observability-controller
spec:
  addOnMeta:
    displayName: Observability Controller
    description: "Manages Observability components"
  installStrategy:
    type: Manual
  addOnConfiguration:
    crdName: observabilityaddons.observability.open-cluster-management.io
EOF

Step 6 — Create Object Storage Secret

cat <<EOF | oc apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: thanos-object-storage
  namespace: open-cluster-management-observability
type: Opaque
stringData:
  thanos.yaml: |
    type: s3
    config:
      bucket: acm-observability
      endpoint: minio.minio.svc.cluster.local:9000
      access_key: user1
      secret_key: Password1234
      insecure: true
EOF

Step 7 — Create MultiClusterObservability Resource

cat <<EOF | oc apply -f -
apiVersion: observability.open-cluster-management.io/v1beta2
kind: MultiClusterObservability
metadata:
  name: observability
  namespace: open-cluster-management-observability
spec:
  enableDownsampling: true
  observabilityAddonSpec:
    enableMetrics: true
    interval: 300
    scrapeSizeLimitBytes: 1073741824
    workers: 1
  storageConfig:
    metricObjectStorage:
      name: thanos-object-storage
      key: thanos.yaml
    storageClass: lvms-vg1
    alertmanagerStorageSize: 1Gi
    compactStorageSize: 100Gi
    receiveStorageSize: 100Gi
    ruleStorageSize: 1Gi
    storeStorageSize: 10Gi
EOF

Step 8 — Verify Deployment

watch oc get pods -n open-cluster-management-observability

On managed cluster:

oc get pods -n open-cluster-management-addon-observability

Expected components:

  • Thanos (query, store, compact, receive)
  • Grafana
  • Alertmanager
  • Metrics collector pods

Accessing Grafana Dashboards

  1. Navigate to Networking → Routes

  2. Open Grafana route and log in using hub cluster credentials

  3. Go to Dashboards → ACM / OpenShift Virtualization

After navigating to the dashboards, you will find multiple pre-built dashboards available for monitoring the VM fleet. Some of the key dashboards include:

  • Cluster Overview- Provides a consolidated view across all clusters including total nodes, total virtual machines, running vs stopped VMs, and overall CPU, memory, network, and storage utilization.

  • Single Cluster View- Provides detailed insights for an individual cluster including node count, VM count and status, resource utilization, and alerts related to virtual machines.

  • Single Virtual Machine View- Provides detailed metrics for a specific virtual machine including CPU and memory usage, network metrics, storage consumption, and overall VM health status.

FIGURE 118. ACM / OpenShift Virtualization pre-built dashboards in Grafana

Creating Custom Dashboard

If additional metrics are required, custom dashboards can be created.

Step 1 — Create JSON Dashboard

vi vm-fleet-dashboard.json

{
  "uid": "vm-fleet-monitoring",
  "title": "OpenShift Virtualization VM Fleet Monitoring",
  "schemaVersion": 38,
  "version": 1,
  "refresh": "30s",
  "timezone": "browser",
  "editable": true,
  "time": {
    "from": "now-6h",
    "to": "now"
  },
  "tags": [
    "kubevirt",
    "openshift-virtualization",
    "acm",
    "vm-fleet"
  ],
  "panels": [
    {
      "id": 1,
      "type": "stat",
      "title": "Total VMs",
      "datasource": "Observatorium",
      "targets": [
        {
          "expr": "count(kubevirt_vm_info)"
        }
      ],
      "gridPos": {
        "h": 4,
        "w": 12,
        "x": 0,
        "y": 0
      }
    },
    {
      "id": 2,
      "type": "stat",
      "title": "Running VMs",
      "datasource": "Observatorium",
      "targets": [
        {
          "expr": "sum by (cluster)(kubevirt_vmi_phase_count{phase=\"running\"})"
        }
      ],
      "gridPos": {
        "h": 4,
        "w": 12,
        "x": 12,
        "y": 0
      }
    },
    {
      "id": 4,
      "type": "timeseries",
      "title": "VM CPU Usage",
      "datasource": "Observatorium",
      "targets": [
        {
          "expr": "rate(kubevirt_vmi_cpu_usage_seconds_total[15m])"
        }
      ],
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 6
      }
    },
    {
      "id": 5,
      "type": "timeseries",
      "title": "VM Memory Usage",
      "datasource": "Observatorium",
      "targets": [
        {
          "expr": "sum by (cluster)(kubevirt_vmi_memory_used_bytes)"
        }
      ],
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 12,
        "y": 6
      }
    },
    {
      "id": 6,
      "type": "stat",
      "title": "VMs per Cluster",
      "datasource": "Observatorium",
      "targets": [
        {
          "expr": "count by (cluster) (kubevirt_vmi_info)"
        }
      ],
      "gridPos": {
        "h": 6,
        "w": 12,
        "x": 0,
        "y": 16
      }
    },
    {
      "id": 8,
      "type": "stat",
      "title": "Successful Cross-Cluster VM Migrations",
      "datasource": "Observatorium",
      "targets": [
        {
          "expr": "count(count by (name) (max_over_time(label_replace(kubevirt_vmi_info, \"vm\", \"$1\", \"name\", \"(.*)\")[1h:])) > 1)"
        }
      ],
      "gridPos": {
        "h": 6,
        "w": 12,
        "x": 12,
        "y": 16
      }
    }
  ],
  "templating": {
    "list": []
  },
  "annotations": {
    "list": []
  }
}

Step 2 — Create Deployment Script

vi deploy-json.sh

#!/bin/bash

set -e

DASHBOARD_JSON="vm-fleet-dashboard.json"
CONFIGMAP_NAME="grafana-dashboard-vm-fleet"
NAMESPACE="open-cluster-management-observability"

oc create configmap "$CONFIGMAP_NAME" \
  --from-file="$DASHBOARD_JSON" \
  --namespace="$NAMESPACE" \
  --dry-run=client -o yaml | \
  oc label -f - grafana-custom-dashboard=true --local --dry-run=client -o yaml | \
  oc apply -f -

oc get configmap "$CONFIGMAP_NAME" -n "$NAMESPACE"

Step 3 — Deploy Dashboard

sh deploy-json.sh

You will see the following output:

FIGURE 119. Custom Dashboard creation

Step 4 — Verify in Grafana

Navigate to Grafana → Dashboards → Custom

You will see:

OpenShift Virtualization VM Fleet Monitoring

FIGURE 120. Custom OpenShift Virtualization VM Fleet Monitoring dashboard in Grafana

Notes

  • Dashboard updates reflect within 30–60 seconds
  • Restart Grafana if required:
oc rollout restart deployment/observability-grafana -n open-cluster-management-observability
  • Modify JSON as per requirement and re-run script to update dashboard