Enabling x-forwarded-* headers

[gid-fieldcode]

In our first steps with Envoy Gateway we have installed it with Helm on Kubernetes in AWS, we have set up a few HTTPRoutes successfully over HTTPS and we are now trying to get it to talk to Keycloak for authentication.

As the original target microservice configured in HTTPRoute redirects the client to Keycloak for authentication, Keycloak expects to see the following headers:

- x-forwarded-for
- x-forwarded-proto 
- x-forwarded-host 
- x-forwarded-port

In https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/headers#x-forwarded-for I can see how such functionality can be configured in Envoy. And it looks quite challenging. Still, we're not using Envoy exactly, we are using Envoy Gateway, so what is the recommended way to enable such headers? Is this done with a BackendTrafficPolicy, perhaps? Or with extra settings in the
definition of the EnvoyProxy resource?

Please note that we have enabled the proxy protocol with a ClientTrafficPolicy. Details in #4064 (thanks to Lior)

[stecullum]

We are using Keycloak like this..and it works as expected

( https )NLB ---> (http) Envoy ---> Keycloak

apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: keycloak
  namespace: keycloak
spec:
  parentRefs:
    - name: eg-external
      namespace: envoy-gateway-system
      kind: Gateway
  hostnames:
    - "auth.example.com"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: keycloak-service
          namespace: keycloak
          port: 8080
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /
      filters:
      - type: RequestHeaderModifier
        requestHeaderModifier:
          set:
            - name: X-Forwarded-Proto
              value: "https"

[gid-fieldcode]

Thanks for responding!

I tried the "filters" addition you have in your definition of the Keycloak HTTPRoute hoping that it would do the trick but it didn't. When I execute a curl to a testing app that merely prints the headers, it still returns X-Forwarded-Proto as http as you can see below. And when I invoke the /auth URI for the keycloak route and then click to get into the Administration Console, it redirects me to URLs that have port 80 and thus fail to load.

 "headers": {
  "Accept": [
   "*/*"
  ],
  "User-Agent": [
   "curl/7.68.0"
  ],
  "X-Envoy-External-Address": [
   "87.130.118.226"
  ],
  "X-Forwarded-For": [
   "87.130.118.226"
  ],
  "X-Forwarded-Proto": [
   "http"
  ],
  "X-Request-Id": [
   "34a7c7af-c2de-4f3b-ad6d-3296f63301ad"
  ]
 }

Below is my setup if you'd care to take a look.

Please note that I don't have "Keycloak operator", I have Keycloak as a helm chart, namely the codecentric/keycloak chart, with app version 17.0.1. And I have no resource definition named "Keycloak".

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
  parametersRef:
    group: gateway.envoyproxy.io
    kind: EnvoyProxy
    namespace: envoy-gateway
    name: custom-proxy-config

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: envoy-gateway
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        annotations:          
          service.beta.kubernetes.io/aws-load-balancer-type: external
          service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
          service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
          service.beta.kubernetes.io/aws-load-balancer-ssl-cert: 'arn:aws:acm:eu-west-1:361789914539:certificate/6a3b4415-ad5e-4037-96f9-e46bd894acc3'
          service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
          service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: 'ELBSecurityPolicy-TLS13-1-2-2021-06'          
          service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*' 
          service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https
          service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: TCP          

---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: apps
  namespace: test
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      protocol: HTTP
      port: 80
    - name: https
      protocol: HTTP
      port: 443

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
  name: enable-proxy-protocol-policy-client
  namespace: test
spec:
  clientIPDetection:
    xForwardedFor:
      numTrustedHops: 1
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: apps
  enableProxyProtocol: true

---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: keycloak-ref-grant
  namespace: keycloak
spec:
  from:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    namespace: test
  to:
  - group: ""
    kind: Service

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: keycloak-proxy
  namespace: test
spec:
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: apps
  hostnames:
    - "envoy.fieldcode.club"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: keycloak-proxy
          namespace: keycloak
          port: 4900
          weight: 1          
      matches:
        - path:
            type: PathPrefix
            value: "/auth"
      filters:
      - type: RequestHeaderModifier
        requestHeaderModifier:
          set:
            - name: X-Forwarded-Proto
              value: "https"            

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: echo
  namespace: test
spec:
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: apps
  hostnames:
    - "envoy.fieldcode.club"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: echo
          port: 3000
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /echo

[arkodg]

shouldnt it be responseHeaderModifier ?

[gid-fieldcode]

We figured it out and now it works nicely. Your answer was correct. Thanks!

[stecullum]

Sorry i have been on holiday..

Apologies.. I am on the latest v25.0.* releases and things did work different in the older release

What was the correct solution for you?
What did you do to make it work?

[gid-fieldcode]

Your first answer was the solution. Initially I didn't apply it properly, it was my mistake. We like Envoy, looks like we'll adopt it. You can expect a few more questions from us.... :-)

[arkodg]

there's also a dedicated setting in ClientTrafficPolicy for setting numTrustedHops
https://gateway.envoyproxy.io/docs/api/extension_types/#clientipdetectionsettings

[larivierec]

Just curious, I know the question was answered but is the same thing possible with TCPRoutes? Xff and ProxyProtocol?

[arkodg]

yeah proxyProtocol support for downstream is possible, more in https://gateway.envoyproxy.io/docs/tasks/traffic/client-traffic-policy/#enable-proxy-protocol-for-downstream-client

[larivierec]

That works, thank you, I missed it in the docs 🤦‍♂️