So I have two services: Server App + Embedding Generator App, deployed via Cloud Run V2. Server App is publicly accessible and Embedding Generator App is only meant to be contacted by Server App. I setup a subnet and VPC connector to enable that connectivity. I'm including the Terraform files I used to setup the services and VPC connector.

Now the problem when Server App tries to contact Embedding Generator I get a 404 error, nothing even show up in the Cloud Run logs for that service. However when I create a VM and attach it to the Horcrux subnet, I'm able to successfully call Embedding Generator. This makes me think there's an issues with the connectivity between Server App + Embedding Generator. Can anyone take a look at my TF files to see if they see any issues. Here are my Terraform files:

embedding_generator_app.tf

resource "google_service_account" "embedding_generator_app" {
  account_id = "embedding-generator-app"
}

resource "google_cloud_run_v2_service" "embedding_generator_app" {
  name     = "embedding-generator-app"
  location = "us-central1"
  ingress  = "INGRESS_TRAFFIC_INTERNAL_ONLY"

  template {
    service_account = google_service_account.embedding_generator_app.email

    scaling {
      max_instance_count = 10
    }

    vpc_access {
      connector = google_vpc_access_connector.horcrux.id
      egress    = "PRIVATE_RANGES_ONLY"
    }

    containers {
      image = "project-registry/embedding-generator-app"

      ports {
        container_port = 1010
      }

      resources {
        startup_cpu_boost = true
        limits = {
          cpu    = "4000m"
          memory = "2Gi"
        }
      }
    }
  }
}

resource "google_cloud_run_v2_service_iam_binding" "embedding_generator_app_run_invoker" {
  name     = google_cloud_run_v2_service.embedding_generator_app.name
  project  = google_cloud_run_v2_service.embedding_generator_app.project
  location = google_cloud_run_v2_service.embedding_generator_app.location
  role     = "roles/run.invoker"
  members = [
    "allUsers",
  ]
}

server_app.tf

resource "google_service_account" "server_app" {
  account_id   = "server-app"
}

resource "google_service_account_key" "server_app" {
  service_account_id = google_service_account.server_app.name
}

resource "google_cloud_run_v2_service" "server_app" {
  name     = "server-app"
  location = "us-central1"
  ingress  = "INGRESS_TRAFFIC_ALL"

  template {
    service_account = google_service_account.server_app.email

    scaling {
      max_instance_count = 10
    }

    vpc_access {
      connector = google_vpc_access_connector.horcrux.id
      egress    = "PRIVATE_RANGES_ONLY"
    }

    containers {
      image = "project-registry/serverapp"

      ports {
        container_port = 9090
      }

      resources {
        startup_cpu_boost = true
        cpu_idle          = true
        limits = {
          cpu    = "2000m"
          memory = "1Gi"
        }
      }

      env {
        name  = "EMBEDDING_GENERATOR_APP_URL"
        value = google_cloud_run_v2_service.embedding_generator_app.uri
      }
    }
  }
}

resource "google_cloud_run_v2_service_iam_binding" "server_app_run_invoker" {
  name     = google_cloud_run_v2_service.server_app.name
  project  = google_cloud_run_v2_service.server_app.project
  location = google_cloud_run_v2_service.server_app.location
  role     = "roles/run.invoker"
  members = [
    "allUsers",
  ]
}

vpc.tf

resource "google_project_service" "vpc_access_api" {
  project    = "project-id"
  service = "vpcaccess.googleapis"
}

resource "google_compute_subnetwork" "horcrux" {
  name          = "horcrux"
  ip_cidr_range = "10.2.0.0/28"
  region        = "us-central1"
  network       = "default"
  private_ip_google_access = true
  depends_on = [google_project_service.vpc_access_api]
}

resource "google_vpc_access_connector" "horcrux" {
  name          = "horcrux"
  machine_type  = "e2-micro"
  min_instances = 2
  max_instances = 3
  subnet {
    name = google_compute_subnetwork.horcrux.name
  }
}

So I have two services: Server App + Embedding Generator App, deployed via Cloud Run V2. Server App is publicly accessible and Embedding Generator App is only meant to be contacted by Server App. I setup a subnet and VPC connector to enable that connectivity. I'm including the Terraform files I used to setup the services and VPC connector.

Now the problem when Server App tries to contact Embedding Generator I get a 404 error, nothing even show up in the Cloud Run logs for that service. However when I create a VM and attach it to the Horcrux subnet, I'm able to successfully call Embedding Generator. This makes me think there's an issues with the connectivity between Server App + Embedding Generator. Can anyone take a look at my TF files to see if they see any issues. Here are my Terraform files:

embedding_generator_app.tf

resource "google_service_account" "embedding_generator_app" {
  account_id = "embedding-generator-app"
}

resource "google_cloud_run_v2_service" "embedding_generator_app" {
  name     = "embedding-generator-app"
  location = "us-central1"
  ingress  = "INGRESS_TRAFFIC_INTERNAL_ONLY"

  template {
    service_account = google_service_account.embedding_generator_app.email

    scaling {
      max_instance_count = 10
    }

    vpc_access {
      connector = google_vpc_access_connector.horcrux.id
      egress    = "PRIVATE_RANGES_ONLY"
    }

    containers {
      image = "project-registry/embedding-generator-app"

      ports {
        container_port = 1010
      }

      resources {
        startup_cpu_boost = true
        limits = {
          cpu    = "4000m"
          memory = "2Gi"
        }
      }
    }
  }
}

resource "google_cloud_run_v2_service_iam_binding" "embedding_generator_app_run_invoker" {
  name     = google_cloud_run_v2_service.embedding_generator_app.name
  project  = google_cloud_run_v2_service.embedding_generator_app.project
  location = google_cloud_run_v2_service.embedding_generator_app.location
  role     = "roles/run.invoker"
  members = [
    "allUsers",
  ]
}

server_app.tf

resource "google_service_account" "server_app" {
  account_id   = "server-app"
}

resource "google_service_account_key" "server_app" {
  service_account_id = google_service_account.server_app.name
}

resource "google_cloud_run_v2_service" "server_app" {
  name     = "server-app"
  location = "us-central1"
  ingress  = "INGRESS_TRAFFIC_ALL"

  template {
    service_account = google_service_account.server_app.email

    scaling {
      max_instance_count = 10
    }

    vpc_access {
      connector = google_vpc_access_connector.horcrux.id
      egress    = "PRIVATE_RANGES_ONLY"
    }

    containers {
      image = "project-registry/serverapp"

      ports {
        container_port = 9090
      }

      resources {
        startup_cpu_boost = true
        cpu_idle          = true
        limits = {
          cpu    = "2000m"
          memory = "1Gi"
        }
      }

      env {
        name  = "EMBEDDING_GENERATOR_APP_URL"
        value = google_cloud_run_v2_service.embedding_generator_app.uri
      }
    }
  }
}

resource "google_cloud_run_v2_service_iam_binding" "server_app_run_invoker" {
  name     = google_cloud_run_v2_service.server_app.name
  project  = google_cloud_run_v2_service.server_app.project
  location = google_cloud_run_v2_service.server_app.location
  role     = "roles/run.invoker"
  members = [
    "allUsers",
  ]
}

vpc.tf

resource "google_project_service" "vpc_access_api" {
  project    = "project-id"
  service = "vpcaccess.googleapis"
}

resource "google_compute_subnetwork" "horcrux" {
  name          = "horcrux"
  ip_cidr_range = "10.2.0.0/28"
  region        = "us-central1"
  network       = "default"
  private_ip_google_access = true
  depends_on = [google_project_service.vpc_access_api]
}

resource "google_vpc_access_connector" "horcrux" {
  name          = "horcrux"
  machine_type  = "e2-micro"
  min_instances = 2
  max_instances = 3
  subnet {
    name = google_compute_subnetwork.horcrux.name
  }
}

• google-cloud-platform
• terraform
• google-cloud-run
• vpc
• google-vpc

• Hmmm, there is many possible case for your issue in your deployment. Let me start by a more generic question first: Do you perform "client side" call to the embedding generator or "server side"? I mean the request to the embedding generator is sent from the user browser or the Server App Cloud Run service – guillaume blaquiere Commented Feb 2 at 22:52
• Request to Server App is from the browser. Request to Embedding Generator is from Server App. – mdornfe1 Commented Feb 2 at 23:03
• Ok, Can you share how the request is made from the server app to the embedding generator? – guillaume blaquiere Commented Feb 3 at 12:01

1 Answer 1

For internal communication between 2 cloud run you should configure egress = "ALL_TRAFFIC" in the source cloud run.

And you have other solutions:

according to this documentation: https://cloud.google/run/docs/securing/private-networking#from-other-services

To receive requests from other Cloud Run services or App Engine, perform the following steps:

1. Configure the source service to use either Direct VPC egress or a connector.

2. Make sure traffic to Cloud Run routes through the VPC network by using one of the following options:

• Configure the source service to send all traffic through the VPC network and enable Private Google Access on the subnet associated with Direct VPC egress or the connector.
• Set up Private Service Connect or an internal Application Load Balancer to front your destination Cloud Run service. With this configuration, you access Cloud Run by using internal IP addresses, so requests are routed through the VPC network.
• Enable Private Google Access on the subnet associated with the source service and configure DNS to resolve run.app URLs to the private.googleapis (199.36.153.8/30) or restricted.googleapis (199.36.153.4/30) ranges. Requests to these ranges are routed through the VPC network.