cmf
Installation & Setup Guide¶
Overview¶
This guide provides step-by-step instructions for installing, configuring, and using CMF (Common Metadata Framework) for ML pipeline metadata tracking.
The installation process consists of following components:
- cmflib: exposes APIs to track the pipeline metadata. It also provides APIs to query the stored metadata.
- cmf-server with GUI: enables users to store, retrieve, and view ML training metadata through an intuitive UI.
Prerequisites¶
Before installing CMF, ensure you have the following prerequisites:
- Linux/Ubuntu/Debian
- Python: Version 3.9 to 3.10 (3.10 recommended)
⚠️ Warning: "Python 3.9 Installation Issue on Ubuntu"
Issue: When creating Python 3.9 virtual environments, you may encounter:
ModuleNotFoundError: No module named 'distutils.cmd'
Root Cause: Python 3.9 may be missing required modules like
distutils
orvenv
when installed on Ubuntu systems.Resolution:
- Add the deadsnakes PPA (provides newer Python versions):
bash sudo add-apt-repository ppa:deadsnakes/ppa sudo apt-get update
- Install Python 3.9 with required modules:
sudo apt install python3.9 python3.9-dev python3.9-distutils
This ensures Python 3.9 and its essential modules are fully installed.
- Git: Latest version for code versioning
- Docker: For containerized deployment
- Storage Backend: S3, MinIO, or local storage for artifacts
Components¶
Install cmf library i.e. cmflib¶
1. Set up Python Virtual Environment
conda create -n cmf python=3.10
conda activate cmf
virtualenv --python=3.10 .cmf
source .cmf/bin/activate
2. Install CMF:¶
pip install git+https://github.com/HewlettPackard/cmf
# pip install cmflib
Install cmf-server¶
Docker Installation
-
Clone the GitHub repository.
git clone https://github.com/HewlettPackard/cmf
-
Install Docker Engine with non-root user privileges.
- Install Docker Compose Plugin.
In earlier versions of Docker Compose,
docker compose
was independent of Docker. Hence,docker-compose
was the command. However, after the introduction of Docker Compose Desktop V2, the compose command became part of Docker Engine. The recommended way to install Docker Compose is by installing a Docker Compose plugin on Docker Engine. For more information - Docker Compose Reference. - Docker Proxy Settings are needed for some of the server packages. Refer to the official Docker documentation for comprehensive instructions: Configure the Docker Client for Proxy.
Using docker compose
file
This is the recommended way as docker compose starts cmf-server, postgres db and ui-server in one go. It is neccessary to start postgres db before cmf-server.
- Go to root
cmf
directory. - Replace
xxxx
with your user-name in docker-compose-server.yml available in the root cmf directory....... services: server: image: server:latest volumes: - /home/xxxx/cmf-server/data:/cmf-server/data # for example /home/hpe-user/cmf-server/data:/cmf-server/data - /home/xxxx/cmf-server/data/static:/cmf-server/data/static # for example /home/hpe-user/cmf-server/data/static:/cmf-server/data/static container_name: cmf-server build: ....
- Create a
.env
file in the same directory asdocker-compose-server.yml
and add the necessary environment variables.POSTGRES_USER: myuser POSTGRES_PASSWORD: mypassword POSTGRES_PORT: 5470
⚠️Warning: Avoid using
@
character inPOSTGRES_PASSWORD
to prevent connection issues. -
Execute one of the following commands to start both containers.
IP
variable is the IP address andhostname
is the host name of the machine on which you are executing the following command.IP=200.200.200.200 docker compose -f docker-compose-server.yml up OR hostname=host_name docker compose -f docker-compose-server.yml up
Replace
docker compose
withdocker-compose
for older versions. Also, you can adjust$IP
indocker-compose-server.yml
to reflect the server IP and run thedocker compose
command without specifying IP=200.200.200.200........ environment: REACT_APP_MY_IP: ${IP} ......
-
Stop the containers.
docker compose -f docker-compose-server.yml stop
It is necessary to rebuild images for cmf-server and ui-server after
cmf version update
or after pulling the latest cmf code from git.