I have a kubernetes cluster and a PostgreSQL Database running on Google Cloud.
The pod that has the problem is a cronjob with the following configuration:
apiVersion: batch/v1
kind: CronJob
metadata:
name: taxiq-cronjob-reminder
annotations:
cloud.google/neg: '{"ingress": true}'
cloud.google/backend-config: '{"default": "taxiq-healthconfig"}'
spec:
schedule: "31 4 * * *"
timeZone: "Europe/Berlin"
jobTemplate:
spec:
template:
spec:
serviceAccountName: taxiq-stage-gke
initContainers:
- image: gcr.io/google/cloudsdktool/cloud-sdk:326.0.0-alpine
name: workload-identity-initcontainer
command:
- '/bin/bash'
- '-c'
- "curl -s -H 'Metadata-Flavor: Google' 'http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token' --retry 30 --retry-connrefused --retry-max-time 30 > /dev/null || exit 1"
containers:
- name: cloud-sql-proxy
image: gcr.io/cloudsql-docker/gce-proxy:1.33.5
command:
- "/cloud_sql_proxy"
- "-instances=taxiq-stage-app:europe-west3:taxiq-stage=tcp:5432"
securityContext:
runAsNonRoot: true
- name: taxiq-cronjob-reminder
image: europe-west3-docker.pkg.dev/brantpoint-artifacts/taxiq/cronjob-reminder:beta09.7
env:
- name: PGURL
valueFrom:
secretKeyRef:
name: secrets
key: PGURL
restartPolicy: Never
I expect the proxy to start as well as the cronjob to run without errors.
But I get the following error:
failed to connect to `host=127.0.0.1 user=masterchief database=tasks_buildingblock`: dial error (dial tcp 127.0.0.1:5432: connect: connection refused)
I know this error is related to the cloud-sql-proxy not running (yet). I had removed it with restartPolicy: OnFailure.
But I can not use this restartPolicy value for the following reason:
The cronjob is supposed to send mails -> if I get a Failure for any other reason after some mails have already been sent, the cronjob will run again and send the mails multiple times, which might make customers/users unhappy
How can I ensure the cloud-sql-proxy is listening before the cronjob starts?
I have a kubernetes cluster and a PostgreSQL Database running on Google Cloud.
The pod that has the problem is a cronjob with the following configuration:
apiVersion: batch/v1
kind: CronJob
metadata:
name: taxiq-cronjob-reminder
annotations:
cloud.google/neg: '{"ingress": true}'
cloud.google/backend-config: '{"default": "taxiq-healthconfig"}'
spec:
schedule: "31 4 * * *"
timeZone: "Europe/Berlin"
jobTemplate:
spec:
template:
spec:
serviceAccountName: taxiq-stage-gke
initContainers:
- image: gcr.io/google/cloudsdktool/cloud-sdk:326.0.0-alpine
name: workload-identity-initcontainer
command:
- '/bin/bash'
- '-c'
- "curl -s -H 'Metadata-Flavor: Google' 'http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token' --retry 30 --retry-connrefused --retry-max-time 30 > /dev/null || exit 1"
containers:
- name: cloud-sql-proxy
image: gcr.io/cloudsql-docker/gce-proxy:1.33.5
command:
- "/cloud_sql_proxy"
- "-instances=taxiq-stage-app:europe-west3:taxiq-stage=tcp:5432"
securityContext:
runAsNonRoot: true
- name: taxiq-cronjob-reminder
image: europe-west3-docker.pkg.dev/brantpoint-artifacts/taxiq/cronjob-reminder:beta09.7
env:
- name: PGURL
valueFrom:
secretKeyRef:
name: secrets
key: PGURL
restartPolicy: Never
I expect the proxy to start as well as the cronjob to run without errors.
But I get the following error:
failed to connect to `host=127.0.0.1 user=masterchief database=tasks_buildingblock`: dial error (dial tcp 127.0.0.1:5432: connect: connection refused)
I know this error is related to the cloud-sql-proxy not running (yet). I had removed it with restartPolicy: OnFailure.
But I can not use this restartPolicy value for the following reason:
The cronjob is supposed to send mails -> if I get a Failure for any other reason after some mails have already been sent, the cronjob will run again and send the mails multiple times, which might make customers/users unhappy
How can I ensure the cloud-sql-proxy is listening before the cronjob starts?
Share Improve this question asked Mar 23 at 4:17 code4Godcode4God 135 bronze badges 2 |1 Answer
Reset to default 1A mentioned in one of the comments, deploying the Cloud SQL Proxy using initContainers is the solution to start the Proxy as a native sidecar.
Also worth pointing out that you are using the old v1 Cloud SQL Proxy. It is recommended to migrate to the new v2 Cloud SQL Proxy to leverage both performance and reliability benefits.
There are examples of using the v2 Cloud SQL Proxy as a sidecar here and example health check usage here.
Your sample updated would look like the following:
apiVersion: batch/v1
kind: CronJob
metadata:
name: taxiq-cronjob-reminder
annotations:
cloud.google/neg: '{"ingress": true}'
cloud.google/backend-config: '{"default": "taxiq-healthconfig"}'
spec:
schedule: "31 4 * * *"
timeZone: "Europe/Berlin"
jobTemplate:
spec:
template:
spec:
serviceAccountName: taxiq-stage-gke
initContainers:
- image: gcr.io/google/cloudsdktool/cloud-sdk:326.0.0-alpine
name: workload-identity-initcontainer
command:
- '/bin/bash'
- '-c'
- "curl -s -H 'Metadata-Flavor: Google' 'http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token' --retry 30 --retry-connrefused --retry-max-time 30 > /dev/null || exit 1"
- name: cloud-sql-proxy
# v2 Cloud SQL Proxy image
image: gcr.io/cloud-sql-connectors/cloud-sql-proxy:2.15.2
restartPolicy: Always
env:
- name: CSQL_PROXY_HEALTH_CHECK
value: "true"
- name: CSQL_PROXY_HTTP_PORT
value: "9801"
- name: CSQL_PROXY_HTTP_ADDRESS
value: 0.0.0.0
startupProbe:
failureThreshold: 60
httpGet:
path: /startup
port: 9801
scheme: HTTP
periodSeconds: 1
successThreshold: 1
timeoutSeconds: 10
args:
- "--port=5432"
- "taxiq-stage-app:europe-west3:taxiq-stage"
securityContext:
runAsNonRoot: true
containers:
- name: taxiq-cronjob-reminder
image: europe-west3-docker.pkg.dev/brantpoint-artifacts/taxiq/cronjob-reminder:beta09.7
env:
- name: PGURL
valueFrom:
secretKeyRef:
name: secrets
key: PGURL
restartPolicy: Never
initContainersand give it the "Always" restart policy so it starts before your main container and keeps running. – Botje Commented Mar 23 at 8:31