Pipeline Root

Starting from Kubeflow Pipelines SDK v2 and Kubeflow Pipelines 1.7.0, Kubeflow Pipelines supports a new intermediate artifact repository feature: pipeline root in both standalone deployment and AI Platform Pipelines.

Before you start

This guide tells you the basic concepts of Kubeflow Pipelines pipeline root and how to use it. This guide assumes that you already have Kubeflow Pipelines installed, or want to use standalone or AI Platform Pipelines options in the Kubeflow Pipelines deployment guide to deploy Kubeflow Pipelines.

What is pipeline root?

Pipeline root represents an artifact repository where Kubeflow Pipelines stores a pipeline’s artifacts. This feature supports MinIO, S3, GCS natively using Go CDK. Artifacts can be more accessible in S3 and GCS when integrating Kubeflow Pipelines with other systems.

Note: For MinIO, you can’t change the MinIO instance. Kubeflow Pipelines can only use the Minio instance deployed with itself. (Please thumb up this GitHub Issue if you need to specify the Minio instance.)

How to configure pipeline root authentication

MinIO

You don’t need to pass the authentication for MinIO. Kubeflow Pipelines is configured with the authentication of the MinIO instance deployed with itself.

GCS

If you want to specify the pipeline root to GCS :

check authentication-pipelines

S3

If you want to specify the pipeline root to S3, please choose one of the following options:

• Via AWS IRSA:

• Via kfp sdk: dsl.get_pipeline_conf().add_op_transformer(aws.use_aws_secret('xxx', ‘xxx’, ‘xxx’))

references:

• add-op-transformer
• use-aws-secret

How to configure pipeline root

Via ConfigMaps

You can configure default pipeline root for Kubeflow Pipelines via changing the defaultPipelineRoot entry of ConfigMaps kfp-launcher in Kubernetes namespace.

kubectl edit configMap kfp-launcher -n ${namespace}

This pipeline root will be the default pipeline root for all pipelines running in the Kubernetes namespace unless you override it using one of the following options:

Via Building Pipelines

You can configure a pipeline root through the kfp.dsl.pipeline annotation when building pipelines

Via Submitting a Pipeline through SDK

You can configure pipeline root via pipeline_root argument when you submit a Pipeline using one of the following:

• create_run_from_pipeline_func
• create_run_from_pipeline_package
• run_pipeline.

Via Submitting a Pipeline Run through UI

You can configure a pipeline root via the pipeline_root run parameters when you submit a pipeline run in the UI