I want to run Dataflow jobs with a per job dedicated custom service account.

Upon creation, the Dataflow job wants to create a new Pub/Sub subscription, on deployment, to use as the watermark tracking subscription. It has the form of <SOURCE_SUB_NAME>__df_internal<HASH> where SOURCE_SUB_NAME is the actual subscription that the Dataflow pipeline has been configured to pull data from.

My question is how this can be done under the Principle of Least Privilege using a custom service account for this specific Dataflow job. Since the job needs to create a copy of the source Pub/Sub subscription, it needs to make a new subscription on the Pub/Sub topic which feeds source subscription. However, even if I grant the job service account the roles/pubsub.subscriber OR roles/pubsub.editor on the topic in question, I still get 403 errors in the pipeline, trying to call the Subscriber.CreateSubscription API endpoint. Empirically, I found I could only get Dataflow to make the new tracking subscription if I granted roles/pubsub.editor against the entire GCP project.

Given that, how can you use PLP without making your Dataflow job a Pub/Sub Editor on the entire GCP project? Being a project wide Pub/Sub editor means your job could read from any other topic, thus giving it more potential data access than necessary for a given job.

I want to run Dataflow jobs with a per job dedicated custom service account.

Upon creation, the Dataflow job wants to create a new Pub/Sub subscription, on deployment, to use as the watermark tracking subscription. It has the form of <SOURCE_SUB_NAME>__df_internal<HASH> where SOURCE_SUB_NAME is the actual subscription that the Dataflow pipeline has been configured to pull data from.

My question is how this can be done under the Principle of Least Privilege using a custom service account for this specific Dataflow job. Since the job needs to create a copy of the source Pub/Sub subscription, it needs to make a new subscription on the Pub/Sub topic which feeds source subscription. However, even if I grant the job service account the roles/pubsub.subscriber OR roles/pubsub.editor on the topic in question, I still get 403 errors in the pipeline, trying to call the Subscriber.CreateSubscription API endpoint. Empirically, I found I could only get Dataflow to make the new tracking subscription if I granted roles/pubsub.editor against the entire GCP project.

Given that, how can you use PLP without making your Dataflow job a Pub/Sub Editor on the entire GCP project? Being a project wide Pub/Sub editor means your job could read from any other topic, thus giving it more potential data access than necessary for a given job.

• google-cloud-platform
• google-cloud-dataflow
• google-iam
• cloud-security

• Hey Joseph. Does your subscription (created by Dataflow) have a deterministic name? – guillaume blaquiere Commented Jan 29 at 20:18
• Hi, @guillaumeblaquiere, unfortunately not. Every time you drain and redeploy the pipeline, you get a new internal Dataflow subscription, because the 16 hex char suffix changes. – Joseph Lust Commented Jan 31 at 1:42

2 Answers 2

The error 403 refers to the incorrect IAM permission and as for your project, my insight is make a custom role with the permission only necessary to create and manage subscription (not the roles/pubsub.editor). After that, assign that custom role at the topic level (roles/pubsub.subscriber) and this will follow the PLP and avoid granting unnecessary permission.

You have to use 2 custom roles.:

• ROLE_CREATION: Create subscription, with the permission pubsub.subscriptions.create, the pubsub.subscriptions.get, pubsub.subscriptions.list, pubsub.subscriptions.update could be required, I don't know exact how work your dataflow pipeline
• ROLE_ATTACHMENT: attach the subscription to the topic, permission pubsub.topics.attachSubscription

With this 2 created, you have to grant your Dataflow Service Account like this:

• At the project level, grant the ROLE_CREATION custom role. Like this, Dataflow will be able to create a subscription.
• At the topic level, grant the ROLE_ATTACHMENT custom role. Like this, Dataflow will be able to use your topic.

By doing this, your dataflow can create multiple subscription, but can attach it only on the authorized topic, not on other. No data leakage like this.