How to Resolve Jupyter Notebook Spawning Issues in Red Hat OpenShift Data Science, couldn't parse image reference ":py3.8-v1"

#openshift #jupyter #rhods #rhoai

Are you experiencing issues with spawning a Jupyter Notebook in your Red Hat OpenShift Data Science environment? Does the server get stuck at 96% and the notebook pod return a warning message? If you're facing this issue, I recently encountered the same problem in a newly created OpenShift cluster, and in this blog post, I will explain the process I used to resolve it.

In my installation, I encountered a problem where the progress was halted at 96%. The pods were stuck, as seen in the following output:

127.0.0.1 $ oc get pods -n rhods-notebooks
NAME                        READY   STATUS             RESTARTS   AGE
jupyter-nb-kube-3aadmin-0   1/2     InvalidImageName   0          22h         5s

When I described the pod, it returned an error message similar to the one below:

Warning  InspectFailed           0s (x2 over 2s)  kubelet                  Failed to apply default image tag ":py3.8-v1": couldn't parse image reference ":py3.8-v1": invalid reference format
  Warning  Failed                  0s (x2 over 2s)  kubelet                  Error: InvalidImageName
  Normal   Created                 0s               kubelet                  Created container

The warning message indicates that the default image tag could not be parsed correctly.

Solution:

RHODS uses pre-built notebooks that are pulled to the internal registry. If the registry is not running, it can cause issues with the spawning of Jupyter Notebooks. To resolve this issue, ensure the internal registry runs in the cluster before installing RHODS.
To check if the cluster has an image registry pod, run the following command in the terminal:

oc get pods -n openshift-image-registry

You should receive output similar to the following:

The image-registry pod is the one you're interested in. If it's not running, you need to deploy it. For a test cluster, you can use emptyDir, which is not recommended for a production cluster. To patch the config and deploy an image registry pod, run the following commands in the terminal:

oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"storage":{"emptyDir":{}}}}'
oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"managementState":"Managed"}}'