Getting Started with Multi-user Isolation

Usage overview

After Kubeflow is installed and configured, you will by default be accessing your primary profile. A profile owns a Kubernetes namespace of the same name along with a collection of Kubernetes resources. Users have view and modify access to their primary profiles. You can share access to your profile with another user in the system. When sharing the access to a profile with another user, you can choose to whether to provide only read access or read/modify access. For all practical purposes when working through the Kubeflow central dashboard, the active namespace is directly tied with the active profile.

Example of usage

You can select your active profile from the top bar on the Kubeflow central dashboard. Note that you can only see the profiles to which you have view or modify access.

This guide illustrates the user isolation functionality using the Jupyter notebooks service which is the first service in the system to have full integration with the multi-user isolation functionality.

After you select an active profile, the Notebooks Servers UI displays only the active notebook servers in the currently selected profile. All other notebook servers remain hidden from you. If you switch the active profile, the view switches the list of active notebooks appropriately. You can connect to any of the listed notebook servers and view and modify the existing Jupyter notebooks available in the server.

For example, the following image shows the list of notebook servers available in a user’s primary profile:

When an unauthorized user accesses the notebooks in this profile, they see an error:

When you create Jupyter notebook servers from the Notebooks Servers UI, the notebook pods are created in your active profile. If you don’t have modify access to the active profile, you can only browse currently active notebook servers and access the existing notebooks but cannot create new notebook servers in that profile. You can create notebook servers in your primary profile which you have view and modify access to.

Onboarding a new user

An administrator can manually create a profile for any user in the Kubeflow cluster. Here an administrator is a person who has cluster-admin role binding in the Kubernetes cluster. This person has permissions to create and modify Kubernetes resources in the cluster. For example, the person who deployed Kubeflow will have administration privileges in the cluster.

We recommend this approach, since it encourages the adoption of GitOps processes for handling the Profile creation.

Kubeflow v1.5.0 optionally provides automatic Profile creation workflow for authenticated users on first login.

Pre-requisites: grant user minimal Kubernetes cluster access

You must grant each user the minimal permission scope that allows them to connect to the Kubernetes cluster.

For example, for Google Cloud users, you should grant the following Cloud Identity and Access Management (IAM) roles. In the following commands, replace [PROJECT] with your Google Cloud project and replace [EMAIL] with the user’s email address:

• To access the Kubernetes cluster, the user needs the Kubernetes Engine Cluster Viewer role:

  gcloud projects add-iam-policy-binding [PROJECT] --member=user:[EMAIL] --role=roles/container.clusterViewer
  

• To access the Kubeflow UI through IAP, the user needs the IAP-secured Web App User role:

  gcloud projects add-iam-policy-binding [PROJECT] --member=user:[EMAIL] --role=roles/iap.httpsResourceAccessor
  

  Note: you need to grant the user IAP-secured Web App User role even if the user is already an owner or editor of the project. IAP-secured Web App User role is not implied by just Project Owner or Project Editor roles.

• To be able to run gcloud get credentials and see logs in Cloud Logging (formerly Stackdriver), the user needs viewer access on the project:

  gcloud projects add-iam-policy-binding [PROJECT] --member=user:[EMAIL] --role=roles/viewer
  

Manual profile creation

An administrator can manually create profiles for users as described below.

Create a profile.yaml file with the following content on your local machine:

apiVersion: kubeflow.org/v1beta1
kind: Profile
metadata:
  name: profileName   # replace with the name of profile you want, this will be user's namespace name
spec:
  owner:
    kind: User
    name: userid@email.com   # replace with the email of the user

  resourceQuotaSpec:    # resource quota can be set optionally
   hard:
     cpu: "2"
     memory: 2Gi
     requests.nvidia.com/gpu: "1"
     persistentvolumeclaims: "1"
     requests.storage: "5Gi"

Run the following command to create the corresponding profile resource:

kubectl create -f profile.yaml

kubectl apply -f profile.yaml  #if you are modifying the profile

The above command creates a profile named profileName. The profile owner is userid@email.com and has view and modify access to that profile. The following resources are created as part of the profile creation:

• A Kubernetes namespace that shares the same name with the corresponding profile.
• Kubernetes RBAC (Role-based access control) role binding role binding for the namespace: Admin. This makes the profile owner the namespace administrator, thus giving them access to the namespace using kubectl (via the Kubernetes API).
• Istio namespace-scoped AuthorizationPolicy: user-userid-email-com-clusterrole-edit. This allows the user to access data belonging to the namespace the AuthorizationPolicy was created in
• Namespace-scoped service-accounts default-editor and default-viewer to be used by user-created pods in the namespace.
• Namespace scoped resource quota limits will be placed.

Note: Due to a one-to-one correspondence of profiles with Kubernetes namespaces, the terms profile and namespace are sometimes used interchangeably in the documentation.

Batch creation of user profiles

Administrators might want to create profiles for multiple users as a batch. You can do this by creating a profile.yaml on the local machine with multiple sections of profile descriptions as shown below:

apiVersion: kubeflow.org/v1beta1
kind: Profile
metadata:
  name: profileName1   # replace with the name of profile you want
spec:
  owner:
    kind: User
    name: userid1@email.com   # replace with the email of the user
---
apiVersion: kubeflow.org/v1beta1
kind: Profile
metadata:
  name: profileName2   # replace with the name of profile you want
spec:
  owner:
    kind: User
    name: userid2@email.com   # replace with the email of the user

Run the following command to apply the namespaces to the Kubernetes cluster:

kubectl create -f profile.yaml

kubectl apply -f profile.yaml  #if you are modifying the profiles

This will create multiple profiles, one for each individual listed in the sections in profile.yaml.

Automatic profile creation

Kubeflow v1.5.0 provides automatic profile creation:

• Automatic profile creation is not activated by default, and needs to be explicitly included as part of deployment. After turning on automatic user profile creation during deployment, a new user profile is created for authenticated users on their first login. Users will be able to see their new profile in the dropdown list of the Kubeflow central dashboard.
• The automatic profile creation can be enabled as part of the deployment by setting the CD_REGISTRATION_FLOW env variable to true. Modify the <manifests-path>/apps/centraldashboard/upstream/base/params.env to set the registration variable to true

CD_CLUSTER_DOMAIN=cluster.local
CD_USERID_HEADER=kubeflow-userid
CD_USERID_PREFIX=
CD_REGISTRATION_FLOW=true

• When an authenticated user logs into the system and visits the central dashboard for the first time, they trigger a profile creation automatically.
  • A brief message introduces profiles:
  • The user can name their profile and click Finish:
  • This redirects the user to the dashboard where they can view and select their profile in the dropdown list.

Listing and describing profiles

An administrator can list the existing profiles in the system:

$ kubectl get profiles

and describe a specific profile using:

$ kubectl describe profile profileName

Deleting an existing profile

An administrator can delete an existing profile using:

$ kubectl delete profile profileName

This will delete the profile, the corresponding namespace and any Kubernetes resources associated with the profile. The profile’s owner or other users with access to the profile will no longer have access to the profile and will not see it in the dropdown list on the central dashboard.

Managing contributors through the Kubeflow UI

Kubeflow v1.5.0 allows sharing of profiles with other users in the system. An owner of a profile can share access to their profile using the Manage Contributors tab available through the dashboard.

Here is an example of the Manage Contributors tab view:

Notice that in the above view the account associated with the profile is a cluster administrator (Cluster Admin) as this account was used to deploy Kubeflow. The view lists the profiles accessible to the user along with the role associated with that profile.

To add or remove a contributor, add/remove the email address or the user identifier in the Contributors to your namespace field.

The Manage Contributors tab shows the contributors that the namespace owner has added. Note that the cluster administrator can view all the profiles in the system along with their contributors.

The contributors have access to all the Kubernetes resources in the namespace and can create notebook servers as well as access existing notebooks.

Managing contributors manually

An administrator can manually add contributors to an existing profile as described below.

Create a rolebinding.yaml file with the following content on your local machine:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  annotations:
    role: edit
    user: userid@email.com   # replace with the email of the user from your Active Directory case sensitive
  name: user-userid-email-com-clusterrole-edit
  # Ex: if the user email is lalith.vaka@kp.org the name should be user-lalith-vaka-kp-org-clusterrole-edit
  # Note: if the user email is Lalith.Vaka@kp.org from your Active Directory, the name should be user-lalith-vaka-kp-org-clusterrole-edit
  namespace: profileName # replace with the namespace/profile name that you are adding contributors to
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubeflow-edit
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: userid@email.com   # replace with the email of the user from your Active Directory case sensitive

Create an authorizationpolicy.yaml file with the following content on your local machine:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  annotations:
    role: edit
    user: userid@email.com # replace with the email of the user from your Active Directory case sensitive
  name: user-userid-email-com-clusterrole-edit
  namespace: profileName # replace with the namespace/profile name that you are adding contributors to
spec:
  action: ALLOW
  rules:
  - when:
    - key: request.headers[kubeflow-userid] # for GCP, use x-goog-authenticated-user-email instead of kubeflow-userid for authentication purpose
      values:
      - accounts.google.com:userid@email.com   # replace with the email of the user from your Active Directory case sensitive

Run the following command to create the corresponding contributor resources:

kubectl create -f rolebinding.yaml
kubectl create -f authorizationpolicy.yaml

The above command adds a contributor userid@email.com to the profile named profileName. The contributor userid@email.com has view and modify access to that profile.