It was working well before adding the following change in the spark driver pod yaml specification to make driver pod use a PVC backed by externally provisioned storage for spilling files to a custom path:

spec:
  containers:
    env:
      - name: SPARK_LOCAL_DIRS
        value: /ephemeral
    volumeMounts:
      - name: spark-local-dir-1
        mountPath: /ephemeral
  volumes:
    - name: spark-local-dir-1
      persistentVolumeClaim:
        claimName: shared-persistence

Then the pod gets stuck at ContainerCreating state and there aren't any events or errors in kubectl describe pod. PVC shared-persistence is bound. One potential reason I can think of is that we also have ephemeral-storage definitions in the resources section:

containers:
    - resources:
        limits:
          cpu: '1'
          ephemeral-storage: 4Gi
          memory: 2457Mi
        requests:
          cpu: '1'
          ephemeral-storage: 500Mi
          memory: 2457Mi

Are they conflicting? It is difficult to remove the ephemeral-storage definitions due to some issues in our codebase so I would like to confirm first. Thanks.