Getting started with cmf-server¶

cmf-server is a key interface for the user to explore and track their ML training runs, storing the metadata file on the cmf-server. The user can retrieve the saved metadata file and can view the content of the saved metadata file using the UI provided by the cmf-server.

Setup a cmf-server¶

There are two ways to start a cmf server -

Using docker compose file
Using docker run

Prerequisites¶

Clone the GitHub repository.

git clone https://github.com/HewlettPackard/cmf

Install Docker Engine with non-root user privileges.
Install Docker Compose Plugin.

In earlier versions of Docker Compose, docker compose was independent of Docker. Hence, docker-compose was the command. However, after the introduction of Docker Compose Desktop V2, the compose command became part of Docker Engine. The recommended way to install Docker Compose is by installing a Docker Compose plugin on Docker Engine. For more information - Docker Compose Reference.
Docker Proxy Settings are needed for some of the server packages. Refer to the official Docker documentation for comprehensive instructions: Configure the Docker Client for Proxy.

Using `docker compose` file¶

This is the recommended way as docker compose starts cmf-server, postgres db and ui-server in one go. It is neccessary to start postgres db before cmf-server.

Go to root cmf directory.

Replace xxxx with your user-name in docker-compose-server.yml available in the root cmf directory.

......
services:
server:
  image: server:latest
  volumes:
     - /home/xxxx/cmf-server/data:/cmf-server/data                 # for example /home/hpe-user/cmf-server/data:/cmf-server/data
     - /home/xxxx/cmf-server/data/static:/cmf-server/data/static   # for example /home/hpe-user/cmf-server/data/static:/cmf-server/data/static
  container_name: cmf-server
  build:
....

Create a .env file in the same directory as docker-compose-server.yml and add the necessary environment variables.
```
POSTGRES_USER: myuser
POSTGRES_PASSWORD: mypassword
POSTGRES_PORT: 5470
```
⚠️Warning: Avoid using @ character in POSTGRES_PASSWORD to prevent connection issues.
Execute one of the following commands to start both containers. IP variable is the IP address and hostname is the host name of the machine on which you are executing the following command.
```
IP=200.200.200.200 docker compose -f docker-compose-server.yml up
           OR
hostname=host_name docker compose -f docker-compose-server.yml up
```
Replace docker compose with docker-compose for older versions. Also, you can adjust $IP in docker-compose-server.yml to reflect the server IP and run the docker compose command without specifying IP=200.200.200.200.
```
.......
environment:
REACT_APP_MY_IP: ${IP}
......
```

Stop the containers.

  docker compose -f docker-compose-server.yml stop

It is necessary to rebuild images for cmf-server and ui-server after cmf version update or after pulling the latest cmf code from git.

OR

Using `docker run` command¶

Install cmflib on your system.
Go to cmf/server directory.
```
cd server
```
List all docker images.
```
docker images
```
Execute the following command to create a cmf-server docker image.
```
Usage:  docker build -t [image_name] -f ./Dockerfile ../
```
Example:
```
docker build -t server_image -f ./Dockerfile ../
```
Note - '../' represents the Build context for the docker image.

Launch a new docker container using the image with directory /home/user/cmf-server/data mounted. Pre-requisite: mkdir /home/<user>/cmf-server/data/static

Usage: docker run --name [container_name] -p 0.0.0.0:8080:80 -v /home/<user>/cmf-server/data:/cmf-server/data -e MYIP=XX.XX.XX.XX [image_name]

Example:

docker run --name cmf-server -p 0.0.0.0:8080:80 -v /home/user/cmf-server/data:/cmf-server/data -e MYIP=0.0.0.0 server_image

After the cmf-server container is up, start ui-server. Go to cmf/ui folder.
```
cd cmf/ui
```

Execute the below-mentioned command to create a ui-server docker image.

Usage:  docker build -t [image_name] -f ./Dockerfile ./

Example:

docker build -t ui_image -f ./Dockerfile ./

Launch a new docker container for UI.

Usage: docker run --name [container_name] -p 0.0.0.0:3000:3000 -e REACT_APP_MY_IP=XX.XX.XX.XX [image_name]

Example:

docker run --name ui-server -p 0.0.0.0:3000:3000 -e REACT_APP_MY_IP=0.0.0.0 ui_image

Note: If you face issues regarding Libzbar-dev similar to the snapshot, add proxies to '/.docker/config.json'

Screenshot (115)

{
   proxies: {
        "default": {
                     "httpProxy": "http://web-proxy.labs.xxxx.net:8080",
                     "httpsProxy": "http://web-proxy.labs.xxxx.net:8080",
                     "noProxy": ".labs.xxxx.net,127.0.0.0/8"
             }
         }
 }

To stop the docker container.
```
docker stop [container_name]
```
To delete the docker container.
```
docker rm [container_name]
```
To remove the docker image.
```
docker image rm [image_name]
```

API Reference¶

cmf-server APIs are organized around FastAPI. They accept and return JSON-encoded request bodies and responses and return standard HTTP response codes.

List of APIs¶

Method	URL	Description
`POST`	`/mlmd_push`	Pushes JSON-encoded data to the cmf-server.
`GET`	`/mlmd_pull/{pipeline_name}`	Retrieves an MLMD file from the cmf-server.
`GET`	`/executions/{pipeline_name}`	Retrieves all executions from the cmf-server.
`GET`	`/list-of-executions/{pipeline_name}`	Retrieves a list of execution types.
`GET`	`/execution-lineage/tangled-tree/{uuid}/{pipeline_name}`	Retrieves a dictionary of nodes and links for a given execution type.
`GET`	`/artifacts/{pipeline_name}/{type}`	Retrieves all artifacts of the specified type from the cmf-server.
`GET`	`/artifact-lineage/tangled-tree/{pipeline_name}`	Retrieves a nested list of dictionaries with `id` and `parents` keys for artifacts.
`GET`	`/artifact_types`	Retrieves a list of artifact types.
`GET`	`/pipelines`	Retrieves all pipelines present in the MLMD file.
`POST`	`/tensorboard`	Uploads TensorBoard logs to the cmf-server.
`GET`	`/model-card`	Retrieves model data, input/output artifacts, and executions for a model.
`GET`	`/artifact-execution-lineage/tangled-tree/{pipeline_name}`	Retrieves a nested list of dictionaries with `id` and `parents` keys for artifacts and executions.
`POST`	`/python-env`	Pushes Python environment data to the cmf-server.
`GET`	`/python-env`	Retrieves environment data from the `/cmf-server/data/env` folder.

HTTP Response Status codes¶

Code	Title	Description
`200`	`OK`	mlmd is successfully pushed (e.g. when using `GET`, `POST`).
`400`	`Bad request`	When the cmf-server is not available.
`500`	`Internal server error`	When an internal error has happened

Getting started with cmf-server¶

Setup a cmf-server¶

Prerequisites¶

Using docker compose file¶

OR

Using docker run command¶

API Reference¶

List of APIs¶

HTTP Response Status codes¶

Using `docker compose` file¶

Using `docker run` command¶