Title: MySQL on GKE Autopilot with GCSFuse: Permission denied on /var/lib/mysql/

Body:

I am deploying MySQL 8.0 on Google Kubernetes Engine (GKE) Autopilot using Google Cloud Storage (GCS) with GCSFuse as a PersistentVolume. The database files are stored in a GCS bucket, and the volume is mounted to /var/lib/mysql.

However, MySQL fails to start due to permission issues, with the following error:

Defaulted container "mysql" out of: mysql, gke-gcsfuse-sidecar (init), fix-permissions (init)
2025-02-04 16:58:35+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.41-1.el9 started.
2025-02-04 16:58:38+00:00 [Note] [Entrypoint]: Initializing database files
mysqld: Can't create/write to file '/var/lib/mysql/is_writable' (OS errno 13 - Permission denied)
2025-02-04T16:58:38.828764Z 0 [ERROR] [MY-010460] [Server] --initialize specified but the data directory exists and is not writable. Aborting.
2025-02-04T16:58:38.828780Z 0 [ERROR] [MY-013236] [Server] The designated data directory /var/lib/mysql/ is unusable.
2025-02-04T16:58:38.829179Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.41)  MySQL Community Server - GPL

Setup Overview

I have the following setup:

1. PersistentVolume (PV) and PersistentVolumeClaim (PVC) using gcsfuse-csi:

   • Bucket: www_1
   • Mount Path: /var/lib/mysql
   • Access Mode: ReadWriteMany (RWX)

2. Deployment (MySQL) includes:

   • An initContainer (fix-permissions) that sets correct ownership (chown -R 999:999 /var/lib/mysql)
   • gke-gcsfuse-sidecar for mounting the GCS bucket
   • MySQL 8.0 running with volume mount at /var/lib/mysql

3. Service Account Configuration:

   • The GKE service account is linked to a custom service account.
   • The service account has all required GCS permissions (roles/storage.admin, roles/storage.objectAdmin).
   • The required directories (databases/api) exist in the GCS bucket.

What I Have Tried

• Checked PVC status:

  kubectl get pvc shared-storage-pvc
  

  • Output: Bound** with RWX**, so Kubernetes sees it as writable.

• Checked PersistentVolume (PV) status:

  kubectl describe pv shared-storage-pv
  

  • Output shows the volume is correctly assigned to the PVC.

• Checked MySQL user permissions:

  kubectl exec -it deployment/api-mysql-deployment -- ls -lah /var/lib/mysql
  

  • Cannot execute because the pod fails to start.

• Checked if GCSFuse is mounted read-only:

  kubectl exec -it deployment/api-mysql-deployment -- mount | grep /var/lib/mysql
  

  • Cannot execute because the pod fails to start.

• Checked logs of failed pod:

  kubectl logs deployment/api-mysql-deployment
  

  • Shows permission errors on ****/var/lib/mysql/.

• Tried adding MySQL flags for GCSFuse compatibility:

  args:
    - "--skip-log-bin"
    - "--skip-ssl"
    - "--ssl=0"
    - "--innodb-flush-method=fsync"
    - "--innodb-use-native-aio=0"
  

• Attempted to manually remove /var/lib/mysql/* and restart

  kubectl delete pod api-mysql-deployment
  kubectl rollout restart deployment api-mysql-deployment
  

  • Still fails with permission errors.

Deployment YAML Configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-mysql-deployment
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
      annotations:
        gke-gcsfuse/volumes: "true"
    spec:
      serviceAccountName: spock
      initContainers:
        - name: fix-permissions
          image: busybox
          command: ["sh", "-c", "chown -R 999:999 /var/lib/mysql && chmod -R 777 /var/lib/mysql"]
          securityContext:
            runAsUser: 0
          volumeMounts:
            - name: shared-storage
              mountPath: "/var/lib/mysql"
              subPath: "databases/api"
      containers:
        - name: mysql
          image: mysql:8.0
          args:
            - "--skip-log-bin"
            - "--skip-ssl"
            - "--ssl=0"
            - "--innodb-flush-method=fsync"
            - "--innodb-use-native-aio=0"
          env:
            - name: MYSQL_ROOT_PASSWORD
              value: "password"
            - name: MYSQL_DATABASE
              value: "api"
            - name: MYSQL_USER
              value: "user"
            - name: MYSQL_PASSWORD
              value: "password"
          volumeMounts:
            - name: shared-storage
              mountPath: "/var/lib/mysql"
              subPath: "databases/api"
      volumes:
        - name: shared-storage
          persistentVolumeClaim:
            claimName: shared-storage-pvc

Questions

1. Why does MySQL still see /var/lib/mysql as read-only despite setting correct PVC and volume attributes?
2. How can I force GCSFuse to mount as read-write in GKE Autopilot?
3. Is there a known workaround for MySQL and GCSFuse permission conflicts in Kubernetes?

Would appreciate any insights from those who have successfully deployed MySQL with GCSFuse on GKE Autopilot!

Thanks in advance!

Title: MySQL on GKE Autopilot with GCSFuse: Permission denied on /var/lib/mysql/

Body:

I am deploying MySQL 8.0 on Google Kubernetes Engine (GKE) Autopilot using Google Cloud Storage (GCS) with GCSFuse as a PersistentVolume. The database files are stored in a GCS bucket, and the volume is mounted to /var/lib/mysql.

However, MySQL fails to start due to permission issues, with the following error:

Defaulted container "mysql" out of: mysql, gke-gcsfuse-sidecar (init), fix-permissions (init)
2025-02-04 16:58:35+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.41-1.el9 started.
2025-02-04 16:58:38+00:00 [Note] [Entrypoint]: Initializing database files
mysqld: Can't create/write to file '/var/lib/mysql/is_writable' (OS errno 13 - Permission denied)
2025-02-04T16:58:38.828764Z 0 [ERROR] [MY-010460] [Server] --initialize specified but the data directory exists and is not writable. Aborting.
2025-02-04T16:58:38.828780Z 0 [ERROR] [MY-013236] [Server] The designated data directory /var/lib/mysql/ is unusable.
2025-02-04T16:58:38.829179Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.41)  MySQL Community Server - GPL

Setup Overview

I have the following setup:

1. PersistentVolume (PV) and PersistentVolumeClaim (PVC) using gcsfuse-csi:

   • Bucket: www_1
   • Mount Path: /var/lib/mysql
   • Access Mode: ReadWriteMany (RWX)

2. Deployment (MySQL) includes:

   • An initContainer (fix-permissions) that sets correct ownership (chown -R 999:999 /var/lib/mysql)
   • gke-gcsfuse-sidecar for mounting the GCS bucket
   • MySQL 8.0 running with volume mount at /var/lib/mysql

3. Service Account Configuration:

   • The GKE service account is linked to a custom service account.
   • The service account has all required GCS permissions (roles/storage.admin, roles/storage.objectAdmin).
   • The required directories (databases/api) exist in the GCS bucket.

What I Have Tried

• Checked PVC status:

  kubectl get pvc shared-storage-pvc
  

  • Output: Bound** with RWX**, so Kubernetes sees it as writable.

• Checked PersistentVolume (PV) status:

  kubectl describe pv shared-storage-pv
  

  • Output shows the volume is correctly assigned to the PVC.

• Checked MySQL user permissions:

  kubectl exec -it deployment/api-mysql-deployment -- ls -lah /var/lib/mysql
  

  • Cannot execute because the pod fails to start.

• Checked if GCSFuse is mounted read-only:

  kubectl exec -it deployment/api-mysql-deployment -- mount | grep /var/lib/mysql
  

  • Cannot execute because the pod fails to start.

• Checked logs of failed pod:

  kubectl logs deployment/api-mysql-deployment
  

  • Shows permission errors on ****/var/lib/mysql/.

• Tried adding MySQL flags for GCSFuse compatibility:

  args:
    - "--skip-log-bin"
    - "--skip-ssl"
    - "--ssl=0"
    - "--innodb-flush-method=fsync"
    - "--innodb-use-native-aio=0"
  

• Attempted to manually remove /var/lib/mysql/* and restart

  kubectl delete pod api-mysql-deployment
  kubectl rollout restart deployment api-mysql-deployment
  

  • Still fails with permission errors.

Deployment YAML Configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-mysql-deployment
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
      annotations:
        gke-gcsfuse/volumes: "true"
    spec:
      serviceAccountName: spock
      initContainers:
        - name: fix-permissions
          image: busybox
          command: ["sh", "-c", "chown -R 999:999 /var/lib/mysql && chmod -R 777 /var/lib/mysql"]
          securityContext:
            runAsUser: 0
          volumeMounts:
            - name: shared-storage
              mountPath: "/var/lib/mysql"
              subPath: "databases/api"
      containers:
        - name: mysql
          image: mysql:8.0
          args:
            - "--skip-log-bin"
            - "--skip-ssl"
            - "--ssl=0"
            - "--innodb-flush-method=fsync"
            - "--innodb-use-native-aio=0"
          env:
            - name: MYSQL_ROOT_PASSWORD
              value: "password"
            - name: MYSQL_DATABASE
              value: "api"
            - name: MYSQL_USER
              value: "user"
            - name: MYSQL_PASSWORD
              value: "password"
          volumeMounts:
            - name: shared-storage
              mountPath: "/var/lib/mysql"
              subPath: "databases/api"
      volumes:
        - name: shared-storage
          persistentVolumeClaim:
            claimName: shared-storage-pvc

Questions

1. Why does MySQL still see /var/lib/mysql as read-only despite setting correct PVC and volume attributes?
2. How can I force GCSFuse to mount as read-write in GKE Autopilot?
3. Is there a known workaround for MySQL and GCSFuse permission conflicts in Kubernetes?

Would appreciate any insights from those who have successfully deployed MySQL with GCSFuse on GKE Autopilot!

Thanks in advance!

• mysql
• google-cloud-platform
• google-cloud-storage
• google-kubernetes-engine

1 Answer 1

I used these as steps in replicating your deployment:

1. Set up the Cloud Storage FUSE CSI driver for GKE: starts with Workload Identity Federation for GKE so that you can set fine grained permissions on how your GKE Pods can access data stored in Cloud Storage.

2. Mount Cloud Storage buckets as persistent volumes: I use this for my reference in proper PersistentVolume and PersistentVolumeClaim which gives correct access modes to your volume mounts.

3. Use your deployment YAML configuration without the added MySQL flags for GCSFuse compatibility and use namespace from step1 as spec.serviceAccountName.

4. Deploying these YAML configurations gives us running status for deployment:

For the answers to your questions:

1. It's due to spec.accessModes of your PVC and PV it should have matched field as ReadWriteMany to have proper attributes.

2. You can force GCSFuse to mount in GKE using the first step.

3. Don't skip the first step to grant IAM roles to the Kubernetes ServiceAccount.