Getting started with cmf-client commands¶
cmf¶
Usage: cmf [-h] {init, artifact, metadata, execution, pipeline}
cmf command is a comprehensive tool designed to initialize an artifact repository and perform various operations on artifacts, execution, pipeline and metadata.
cmf init¶
Usage: cmf init [-h] {minioS3,amazonS3,local,sshremote,osdfremote,show}
cmf init initializes an artifact repository for cmf. Local directory, Minio S3 bucket, Amazon S3 bucket, SSH Remote and Remote OSDF directory are the options available. Additionally, user can provide cmf-server url.
cmf init show¶
Usage: cmf init show
cmf init show displays current cmf configuration.
cmf init minioS3¶
Usage: cmf init minioS3 [-h] --url [url]
--endpoint-url [endpoint_url]
--access-key-id [access_key_id]
--secret-key [secret_key]
--git-remote-url[git_remote_url]
--cmf-server-url [cmf_server_url]
--neo4j-user [neo4j_user]
--neo4j-password [neo4j_password]
--neo4j-uri [neo4j_uri]
cmf init minioS3 configures Minio S3 bucket as a cmf artifact repository. Refer minio-server.md to set up a minio server.
cmf init minioS3 --url s3://dvc-art --endpoint-url http://x.x.x.x:9000 --access-key-id minioadmin --secret-key minioadmin --git-remote-url https://github.com/user/experiment-repo.git --cmf-server-url http://x.x.x.x:8080 --neo4j-user neo4j --neo4j-password password --neo4j-uri bolt://localhost:7687
Here, "dvc-art" is provided as an example bucket name. However, users can change it as needed, if the user chooses to change it, they will need to update the Dockerfile for MinIOS3 accordingly.
Required Arguments
--url [url] Specify bucket url.
--endpoint-url [endpoint_url] Specify the endpoint url of minio UI.
--access-key-id [access_key_id] Specify Access Key Id.
--secret-key [secret_key] Specify Secret Key.
--git-remote-url [git_remote_url] Specify git repo url.
-h, --help show this help message and exit
--cmf-server-url [cmf_server_url] Specify cmf-server url. (default: http://127.0.0.1:80)
--neo4j-user [neo4j_user] Specify neo4j user. (default: None)
--neo4j-password [neo4j_password] Specify neo4j password. (default: None)
--neo4j-uri [neo4j_uri] Specify neo4j uri. Eg bolt://localhost:7687 (default: None)
cmf init local¶
Usage: cmf init local [-h] --path [path] -
--git-remote-url [git_remote_url]
--cmf-server-url [cmf_server_url]
--neo4j-user [neo4j_user]
--neo4j-password [neo4j_password]
--neo4j-uri [neo4j_uri]
cmf init local initialises local directory as a cmf artifact repository.
cmf init local --path /home/XXXX/local-storage --git-remote-url https://github.com/user/experiment-repo.git --cmf-server-url http://x.x.x.x:8080 --neo4j-user neo4j --neo4j-password password --neo4j-uri bolt://localhost:7687
Replace 'XXXX' with your system username in the following path: /home/XXXX/local-storage
Required Arguments
--path [path] Specify local directory path.
--git-remote-url [git_remote_url] Specify git repo url.
-h, --help show this help message and exit
--cmf-server-url [cmf_server_url] Specify cmf-server url. (default: http://127.0.0.1:80)
--neo4j-user [neo4j_user] Specify neo4j user. (default: None)
--neo4j-password [neo4j_password] Specify neo4j password. (default: None)
--neo4j-uri [neo4j_uri] Specify neo4j uri. Eg bolt://localhost:7687 (default: None)
cmf init amazonS3¶
Before setting up, obtain AWS temporary security credentials using the AWS Security Token Service (STS). These credentials are short-term and can last from minutes to hours. They are dynamically generated and provided to trusted users upon request, and expire after use. Users with appropriate permissions can request new credentials before or upon expiration. For further information, refer to the Temporary security credentials in IAM page.
To retrieve temporary security credentials using multi-factor authentication (MFA) for an IAM user, you can use the below command.
aws sts get-session-token --duration-seconds <duration> --serial-number <MFA_device_serial_number> --token-code <MFA_token_code>
--serial-number Specifies the serial number of the MFA device associated with the IAM user.
--token-code Specifies the one-time code generated by the MFA device.
Optional Arguments
--duration-seconds Specifies the duration for which the temporary credentials will be valid, in seconds.
Example
aws sts get-session-token --duration-seconds 3600 --serial-number arn:aws:iam::123456789012:mfa/user --token-code 123456
This will return output like
{
"Credentials": {
"AccessKeyId": "ABCDEFGHIJKLMNO123456",
"SecretAccessKey": "PQRSTUVWXYZ789101112131415",
"SessionToken": "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijlmnopqrstuvwxyz12345678910ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijlmnopqrstuvwxyz12345678910ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijlmnopqrstuvwxyz12345678910",
"Expiration": "2021-05-10T15:31:08+00:00"
}
}
Usage: cmf init amazonS3 [-h] --url [url]
--access-key-id [access_key_id]
--secret-key [secret_key]
--session-token [session_token]
--git-remote-url [git_remote_url]
--cmf-server-url [cmf_server_url]
--neo4j-user [neo4j_user]
--neo4j-password neo4j_password]
--neo4j-uri [neo4j_uri]
cmf init amazonS3 initialises Amazon S3 bucket as a CMF artifact repository.
cmf init amazonS3 --url s3://bucket-name --access-key-id XXXXXXXXXXXXX --secret-key XXXXXXXXXXXXX --session-token XXXXXXXXXXXXX --git-remote-url https://github.com/user/experiment-repo.git --cmf-server-url http://x.x.x.x:8080 --neo4j-user neo4j --neo4j-password password --neo4j-uri bolt://localhost:7687
Here, use the --access-key-id, --secret-key and --session-token generated from the
aws stscommand which is mentioned above.The bucket-name must exist within Amazon S3 before executing the
cmf artifact pushcommand.
Required Arguments
--url [url] Specify bucket url.
--access-key-id [access_key_id] Specify Access Key Id.
--secret-key [secret_key] Specify Secret Key.
--session-token Specify session token. (default: )
--git-remote-url [git_remote_url] Specify git repo url.
-h, --help show this help message and exit
--cmf-server-url [cmf_server_url] Specify cmf-server url. (default: http://127.0.0.1:80)
--neo4j-user [neo4j_user] Specify neo4j user. (default: None)
--neo4j-password [neo4j_password] Specify neo4j password. (default: None)
--neo4j-uri [neo4j_uri] Specify neo4j uri. Eg bolt://localhost:7687 (default: None)
cmf init sshremote¶
Usage: cmf init sshremote [-h] --path [path]
--user [user]
--port [port]
--password [password]
--git-remote-url [git_remote_url]
--cmf-server-url [cmf_server_url]
--neo4j-user [neo4j_user]
--neo4j-password neo4j_password]
--neo4j-uri [neo4j_uri]
cmf init sshremote command initialises remote ssh directory as a cmf artifact repository.
cmf init sshremote --path ssh://127.0.0.1/home/user/ssh-storage --user XXXXX --port 22 --password example@123 --git-remote-url https://github.com/user/experiment-repo.git --cmf-server-url http://x.x.x.x:8080 --neo4j-user neo4j --neo4j-password password --neo4j-uri bolt://localhost:7687
--path [path] Specify ssh directory path.
--user [user] Specify user.
--port [port] Specify Port.
--password [password] Specify password.
--git-remote-url [git_remote_url] Specify git repo url.
-h, --help show this help message and exit
--cmf-server-url [cmf_server_url] Specify cmf-server url. (default: http://127.0.0.1:80)
--neo4j-user [neo4j_user] Specify neo4j user. (default: None)
--neo4j-password [neo4j_password] Specify neo4j password. (default: None)
--neo4j-uri [neo4j_uri] Specify neo4j uri. Eg bolt://localhost:7687 (default: None)
cmf init osdfremote¶
Usage: cmf init osdfremote [-h] --path [path]
--cache [cache]
--key-id [key_id]
--key-path [key_path]
--key-issuer [key_issuer]
--git-remote-url[git_remote_url]
--cmf-server-url [cmf_server_url]
--neo4j-user [neo4j_user]
--neo4j-password [neo4j_password]
--neo4j-uri [neo4j_uri]
cmf init osdfremote configures a OSDF Origin as a cmf artifact repository.
cmf init osdfremote --path https://[Some Origin]:8443/nrp/fdp/ --cache http://[Some Redirector]/nrp/fdp --key-id c2a5 --key-path ~/.ssh/fdp.pem --key-issuer https://[Token Issuer]] --git-remote-url https://github.com/user/experiment-repo.git --git-remote-url https://github.com/user/experiment-repo.git --cmf-server-url http://127.0.0.1:80 --neo4j-user neo4j --neo4j-password password --neo4j-uri bolt://localhost:7687
--path [path] Specify FQDN for OSDF origin including including port and directory path if any
--key-id [key_id] Specify key_id for provided private key. eg. b2d3
--key-path [key_path] Specify path for private key on local filesystem. eg. ~/.ssh/XXX.pem
--key-issuer [key_issuer] Specify URL for Key Issuer. eg. https://t.nationalresearchplatform.org/XXX
--git-remote-url [git_remote_url] Specify git repo url. eg: https://github.com/XXX/example.git
--cache [cache] Specify FQDN for OSDF cache including including port and directory path if any
-h, --help show this help message and exit
--cmf-server-url [cmf_server_url] Specify cmf-server url. (default: http://127.0.0.1:80)
--neo4j-user [neo4j_user] Specify neo4j user. (default: None)
--neo4j-password [neo4j_password] Specify neo4j password. (default: None)
--neo4j-uri [neo4j_uri] Specify neo4j uri. Eg bolt://localhost:7687 (default: None)
cmf artifact¶
Usage: cmf artifact [-h] {pull,push,list}
cmf artifact pull, push or list artifacts from or to the user configured artifact repository, respectively.
cmf artifact pull¶
Usage: cmf artifact pull [-h] -p [pipeline_name] -f [file_name] -a [artifact_name]
cmf artifact pull command pull artifacts from the user configured repository to the user's local machine.
cmf artifact pull -p 'pipeline-name' -f '/path/to/mlmd-file-name' -a 'artifact-name'
-p [pipeline_name], --pipeline-name [pipeline_name] Specify Pipeline name.
-h, --help show this help message and exit.
-a [artifact_name], --artifact_name [artifact_name] Specify artifact name only; don't use folder name or absolute path.
-f [file_name], --file-name [file_name] Specify mlmd file name.
cmf artifact push¶
Usage: cmf artifact push [-h] -p [pipeline_name] -f [file_name]
cmf artifact push command push artifacts from the user's local machine to the user configured artifact repository.
cmf artifact push -p 'pipeline_name' -f '/path/to/mlmd-file-name'
-p [pipeline_name], --pipeline-name [pipeline_name] Specify Pipeline name.
-h, --help show this help message and exit.
-f [file_name], --file-name [file_name] Specify mlmd file name.
cmf artifact list¶
Usage: cmf artifact list [-h] -p [pipeline_name] -f [file_name] -a [artifact_name]
cmf artifact list command displays artifacts from the MLMD file with a few properties in a 7-column table, limited to 20 records per page.
cmf artifact list -p 'pipeline_name' -f '/path/to/mlmd-file-name' -a 'artifact_name'
-p [pipeline_name], --pipeline-name [pipeline_name] Specify Pipeline name.
-h, --help show this help message and exit.
-f [file_name], --file-name [file_name] Specify the absolute or relative path for the input MLMD file.
-a [artifact_name], --artifact_name [artifact_name] Specify the artifact name to display detailed information about the given artifact name.
cmf metadata¶
Usage: cmf metadata [-h] {pull,push,export}
cmf metadata push, pull or export the metadata file to and from the cmf-server, respectively.
cmf metadata pull¶
Usage: cmf metadata pull [-h] -p [pipeline_name] -f [file_name] -e [exec_id]
cmf metadata pull command pulls the metadata file from the cmf-server to the user's local machine.
cmf metadata pull -p 'pipeline-name' -f '/path/to/mlmd-file-name' -e 'execution_id'
-p [pipeline_name], --pipeline_name [pipeline_name] Specify Pipeline name.
-h, --help show this help message and exit.
-e [exec_id], --execution [exec_id] Specify execution id.
-f [file_name], --file_name [file_name] Specify mlmd file name with full path(either relative or absolute).
cmf metadata push¶
Usage: cmf metadata push [-h] -p [pipeline_name] -f [file_name] -e [exec_id] -t [tensorboard]
cmf metadata push command pushes the metadata file from the local machine to the cmf-server.
cmf metadata push -p 'pipeline-name' -f '/path/to/mlmd-file-name' -e 'execution_id' -t '/path/to/tensorboard-log'
-p [pipeline_name], --pipeline_name [pipeline_name] Specify Pipeline name.
-h, --help show this help message and exit.
-f [file_name], --file_name [file_name] Specify mlmd file name.
-e [exec_id], --execution [exec_id] Specify execution id.
-t [tensorboard], --tensorboard [tensorboard] Specify path to tensorboard logs for the pipeline.
cmf metadata export¶
Usage: cmf metadata export [-h] -p [pipeline_name] -j [json_file_name] -f [file_name]
cmf metadata export export local mlmd's metadata in json format to a json file.
cmf metadata export -p 'pipeline-name' -j '/path/to/json-file-name' -f '/path/to/mlmd-file-name'
-p [pipeline_name], --pipeline_name [pipeline_name] Specify Pipeline name.
-h, --help show this help message and exit.
-f [file_name], --file_name [file_name] Specify the absolute or relative path for the input MLMD file.
-j [json_file_name], --json_file_name [json_file_name] Specify output json file name with full path.
cmf execution¶
Usage: cmf execution [-h] {list}
cmf execution command to displays executions from the MLMD file.
cmf executions list¶
Usage: cmf execution list [-h] -p [pipeline_name] -f [file_name] -e [execution_id]
cmf execution list command to displays executions from the MLMD file with a few properties in a 7-column table, limited to 20 records per page.
cmf execution list -p 'pipeline_name' -f '/path/to/mlmd-file-name' -e 'execution_id'
-p [pipeline_name], --pipeline-name [pipeline_name] Specify Pipeline name.
-h, --help show this help message and exit.
--f [file_name], --file-name [file_name] Specify the absolute or relative path for the input MLMD file.
-e [exe_id], --execution_id [exe_id] Specify the execution id to retrieve execution.
cmf pipeline¶
Usage: cmf pipeline [-h] {list}
cmf pipeline command displays a list of pipeline name(s) from the available mlmd file.
cmf pipeline list¶
Usage: cmf pipeline list [-h] -f [file_name]
cmf pipeline list command displays a list of pipeline name(s) from the available mlmd file.
cmf pipeline list -f '/path/to/mlmd-file-name'
-h, --help show this help message and exit.
--f [file_name], --file-name [file_name] Specify the absolute or relative path for the input MLMD file.