My beam python on flinkrunner(PortableRunner) has the following code.
from apache_beam.io.external.gcp.pubsub import ReadFromPubSub
from dependency_injector import providers
...
pubsub_reader = providers.Singleton(
ReadFromPubSub,
subscription=config.gcp_pubsub_subscription_info,
expansion_service="beam-expansion-service.flink:8097",
)
And I ran the beam application on K8S with the java expansion service like the below.
(in Dockerfile)
FROM --platform=linux/amd64 openjdk:11
WORKDIR /app
RUN curl -o beam-sdks-java-io-google-cloud-platform-expansion-service.jar .61.0/beam-sdks-java-io-google-cloud-platform-expansion-service-2.61.0.jar
(in Deployment/Service yaml)
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: beam-expansion-service
spec:
replicas: 1
selector:
matchLabels:
app: beam-expansion-service
template:
metadata:
labels:
app: beam-expansion-service
spec:
containers:
- name: expansion-service
image: beam-expansion-service:1
command: ["java", "-jar", "/app/beam-sdks-java-io-google-cloud-platform-expansion-service.jar", "8097", "--javaClassLookupAllowlistFile=*"]
ports:
- containerPort: 8097
---
apiVersion: v1
kind: Service
metadata:
name: beam-expansion-service
spec:
clusterIP: None
selector:
app: beam-expansion-service
ports:
- protocol: TCP
port: 8097
targetPort: 8097
---
I use apache_beam.io.external.gcp.pubsub.ReadFromPubSub
so that I make my beam application get messages from GCP PubSub on FlinkRunner, not Dataflow runner. And they ("java expansion service pod" and "beam on flinkrunner" pod) are running on the same namespace "flink".
And I got an error like the below.
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:10.232.23.14:8097: Endpoint is neither UDS or TCP loopback address."
debug_error_string = "UNKNOWN:Error received from peer {created_time:"2025-03-17T16:02:35.189471737+00:00", grpc_status:14, grpc_message:"failed to connect to all addresses; last error: UNKNOWN: ipv4:10.232.23.14:8097: Endpoint is neither UDS or TCP loopback address."}"
Why can't my beam connect service to java expansion service(grpc server)? When I executed "nc -v 10.232.23.14 8097" or "nc -v beam-expansion-service.flink 8097", the address was opened.
Thanks in advance.