kubernetes - Istio: X-Forwarded-For and X-Real-IP Headers Show Internal IP Instead of Client's Real IP

I'm encountering an issue where the X-Forwarded-For and X-Real-IP headers are showing internal IP addresses instead of the client's real IP when traffic is routed through Istio.

Problem: In the logs, the X-Forwarded-For and X-Real-IP headers contain internal IP addresses (e.g., 10.244.8.14 or 10.116.0.105), while in a similar setup with NGINX ingress, these headers correctly show the client's real IP (e.g., 15.186.175.412).

Context: I'm using Istio as my service mesh, with the default sidecar-based proxy mode.

The traffic goes through Istio Ingress Gateway before reaching the backend services.

The issue is observed when traffic passes through Istio, but the X-Forwarded-For and X-Real-IP headers show the internal IP addresses of the proxies in the mesh.

What I have tried: Increased xff_num_trusted_hops in the Istio EnvoyFilter configuration to account for multiple hops, but the issue persists.

Explicitly set the X-Real-IP and X-Forwarded-For headers in the VirtualService configuration using %DOWNSTREAM_REMOTE_ADDRESS%, but this did not resolve the problem.

I have also ensured that the use_remote_address: true option is enabled in the EnvoyFilter configuration.

Verified that NGINX ingress works correctly and forwards the client IP in the headers, while Istio does not.

Configuration: I have applied the following settings in the EnvoyFilter for Istio:

configPatches:
  - applyTo: NETWORK_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: "envoy.filterswork.http_connection_manager"
    patch:
      operation: MERGE
      value:
        typed_config:
          "@type": "type.googleapis/envoy.extensions.filterswork.http_connection_manager.v3.HttpConnectionManager"
          use_remote_address: true
          xff_num_trusted_hops: 2
          skip_xff_append: false

Additionally, I've tried to explicitly set the X-Forwarded-For and X-Real-IP headers in the VirtualService configuration like so:

yaml

http:
  headers:
    request:
      set:
        X-Real-IP: "%DOWNSTREAM_REMOTE_ADDRESS%"
        X-Forwarded-For: "%DOWNSTREAM_REMOTE_ADDRESS%"

How can I ensure that Istio forwards the real client IP in the X-Forwarded-For and X-Real-IP headers, similar to how NGINX handles this?

Are there any other Istio configurations I may have missed to properly preserve the client IP?

Is there any additional setup needed to propagate the real client IP from the ingress gateway to the application services?

Any help or insights would be greatly appreciated!