Skip to content

Gateway API: Envoy External Auth with oauth2-proxy per team (Azure AD OIDC)

Introduction

This document describes how to protect multiple AKS-hosted applications using Envoy as a Gateway API implementation with external authentication and a per-team oauth2-proxy instance backed by Azure AD (OIDC).

The typical use case for this setup is multiple teams running applications in the same AKS cluster, with each team using its own oauth2-proxy instance to handle authentication and session management while Envoy handles routing and external authentication. If you're deciding between this setup and a single oauth2-proxy instance for the whole cluster, consider the following:

  • Multiple applications are accessed by the same team.
  • You want to save resources by sharing a single oauth2-proxy instance among multiple applications.
  • It is impractical to create separate Entra ID app registrations for every application.

In the example below we protect two monitoring applications, alertmanager and prometheus, using a single oauth2-proxy instance. Access to these applications is restricted to members of specific Entra ID groups.

Prerequisites

This guide assumes you already have an AKS cluster running and the Envoy proxy Gateway API implementation installed. If not, see the previous guide Envoy proxy as a Gateway API implementation in Azure Kubernetes Service (AKS).

You also need an Entra ID (Azure AD) tenant and the ability to register applications. Ensure your Enterprise Applications are configured with callback URLs for all oauth2-proxy instances.

Finally, Helm must be installed and configured to work with your AKS cluster.

Graphical Overview

sequenceDiagram
    autonumber
    participant User
    participant Envoy Gateway
    participant alertmanager-team-a
    participant service-alertmanager-team-a
    participant monitoring-oauth2-proxy-team-a
    participant EntraId
    User ->> Envoy Gateway: Access Application 
    Envoy Gateway-->> alertmanager-team-a: Match HTTPRoute
    critical Check Access
        alertmanager-team-a ->> monitoring-oauth2-proxy-team-a: Check Access
    option Unautheticated
        monitoring-oauth2-proxy-team-a -->> EntraId: Authenticate
        EntraId ->> User: Credentials Request
        User ->> EntraId: Credentials
        EntraId ->> monitoring-oauth2-proxy-team-a: Redirecting to callback URL
        monitoring-oauth2-proxy-team-a -->> User: Set Cookie and redirect to [alertmanager-team-a]

    option Authenticated
        monitoring-oauth2-proxy-team-a ->> alertmanager-team-a: OK
        alertmanager-team-a ->> service-alertmanager-team-a: Request
        service-alertmanager-team-a ->> alertmanager-team-a: Response
        alertmanager-team-a ->> User: Response
    end

The above diagram shows the simplified flow of authentication and access to the alertmanager application. The detailed OAuth2 flow is omitted for brevity. In the diagram is omitted the flow when OAuth2 Proxy page is shown and the passthrough HTTPRoute is used.

Step 1: Deploy oauth2-proxy instances per team

Deploy a separate oauth2-proxy instance for each team in the AKS cluster. Each instance should be configured with its own Entra ID application registration details (Client ID, Client Secret, Redirect URIs, etc.). If creating multiple Entra ID app registrations is impractical, you can reuse registrations where appropriate.

Prepare a values.yaml file for one team. The example below is for Team A:

  ---
  fullnameOverride: monitoring-oauth2-proxy-team-a
  gatewayApi:
    enabled: true
    # The name of the Gateway resource to attach the HTTPRoute to
    gatewayRef:
      name: main
      namespace: envoy-gateway
    # Hostnames to match in the HTTPRoute
    hostnames:
      - monitoring-oauth2-team-a.<YOUR_DOMAIN>
  resources:
    limits:
      cpu: 50m
      memory: 128Mi
    requests:
      cpu: 10m
      memory: 64Mi
  podAnnotations:
    prometheus.io/scrape: 'true'
    prometheus.io/port: '8080'
  nodeSelector:
    fantastic-team-nodes/purpose: critical-addons
  tolerations:
    - key: CriticalAddonsOnly
      operator: Exists
  redis:
    enabled: true
    auth:
      existingSecret: oauth2-proxy-redis-team-a
      existingSecretPasswordKey: redis-password
    global:
      security:
        allowInsecureImages: true
    image:
      registry: <ACR_NAME>.azurecr.io
      repository: bitnami/redis
      tag: 8.0.3
    master:
      count: 1
      nodeSelector:
        fantastic-team-nodes/purpose: critical-addons
      persistence:
        enabled: true
        size: 1Gi
      tolerations:
      - key: CriticalAddonsOnly
        operator: Exists
    replica:
      nodeSelector:
        fantastic-team-nodes/purpose: critical-addons
      replicaCount: 0
      tolerations:
      - key: CriticalAddonsOnly
        operator: Exists
  sessionStorage:
    type: redis
    redis:
      clientType: standalone
      existingSecret: oauth2-proxy-redis-team-a
      passwordKey: redis-password
  initContainers:
    waitForRedis:
      image:
        repository: <ACR_NAME>.azurecr.io/library/alpine
        tag: 3.22.1
      resources:
        limits:
          cpu: 100m
          memory: 300Mi
        requests:
          cpu: 100m
          memory: 300Mi
  image:
    repository: <ACR_NAME>.azurecr.io/oauth2-proxy/oauth2-proxy
  config:
    configFile: |
      email_domains = [ "*" ]
    existingSecret: oauth2-proxy-team-a
  extraArgs:
    - --provider=azure
    - --provider-display-name=Azure AD
    - --reverse-proxy=true
    - --azure-tenant=<YOUR_TENANT_ID>
    - --oidc-issuer-url=https://login.microsoftonline.com/<YOUR_TENANT_ID>/v2.0
    - --redirect-url=https://monitoring-oauth2-team-a.<YOUR_DOMAIN>/oauth2/callback
    - --cookie-secure=false
    - --insecure-oidc-skip-issuer-verification=true
    - --ssl-insecure-skip-verify=true
    - --ssl-upstream-insecure-skip-verify=true
    - --oidc-email-claim=upn # or "email" depending on your Entra ID app registration configuration
    - --scope=openid email profile
    - --metrics-address=:8080
    - --upstream=static://200
    - --silence-ping-logging=false
    - --allowed-group=<ENTRA_ID_GROUP_ID_FOR_TEAM_A>,<ENTRA_ID_GROUP_ID_FOR_TEAM_A_2>,<ENTRA_ID_GROUP_ID_FOR_ADMINS>
    - --whitelist-domain=*.<YOUR_DOMAIN>
    - --cookie-domain=.<YOUR_DOMAIN>
    - --proxy-prefix=/oauth2
    - --set-xauthrequest=true
    - --show-debug-on-error=false
    - --pass-host-header=true

For reference parameters used in the values.yaml file, see the oauth2-proxy Helm chart documentation.

Because this example uses Redis for session storage, create a Kubernetes Secret to store the Redis password. Below is an example for Team A.

First, generate a random password for Redis. You can use any method you prefer; here is an example using /dev/urandom, tr, and head:

$ cat /dev/urandom | gtr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1 | base64
bkNXeEM5ZEVnZ2k5MWFibXc0SmpuRm5HV1B6TUl2VDcK

Using the generated password, create the Kubernetes Secret with the following manifest:

cat <<EOF | kubectl apply -f -
apiVersion: v1
data:
  redis-password: bkNXeEM5ZEVnZ2k5MWFibXc0SmpuRm5HV1B6TUl2VDcK
kind: Secret
metadata:
  name: oauth2-proxy-redis-team-a
  namespace: monitoring-team-a
type: Opaque
EOF

Create another Kubernetes Secret to store the oauth2-proxy configuration, including the Client ID, Client Secret, and Cookie Secret. Replace the placeholders with your actual values:

cat <<EOF | kubectl apply -f -
apiVersion: v1
data:
  client-id: <BASE_64_ENCODED_CLIENT_ID> # This is Application (client) ID from Entra ID app registration
  client-secret: <BASE_64_ENCODED_CLIENT_SECRET> # This is Client Secret from Entra ID app registration
  cookie-secret: <BASE_64_ENCODED_RANDOM_32_CHAR_STRING> # Random 32 characters base64 encoded
kind: Secret
metadata:
  name: oauth2-proxy-team-a
  namespace: monitoring-team-a
type: Opaque
EOF

Deploy the oauth2-proxy instance for Team A using the values.yaml file:

helm upgrade --install monitoring-oauth2-proxy-team-a oci://ghcr.io/oauth2-proxy/charts -f values.yaml --namespace monitoring-team-a

Ensure that the oauth2-proxy instance is running and the URL is accessible:

curl -vk https://monitoring-oauth2-team-a.<YOUR_DOMAIN>

Step 2: Configure Envoy Gateway External Authentication to protect alertmanager and prometheus

With the oauth2-proxy instance deployed for Team A, configure Envoy Gateway to use it for external authentication when accessing the alertmanager and prometheus applications.

As shown in the graphical overview, Envoy Gateway forwards authentication requests to the oauth2-proxy instance, which handles OIDC authentication with Entra ID. To implement this, create two HTTPRoute resources and one SecurityPolicy per application.

Below is an example passthrough HTTPRoute to the oauth2-proxy for the alertmanager application. If this HTTPRoute is not created, the OAuth2 Proxy page will be shown to the user, but it will appear broken.

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: alertmanager-passthrough-oauth-team-a
  namespace: monitoring-team-a
spec:
  hostnames:
    - alertmanager-team-a.<YOUR_DOMAIN>
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: main
      namespace: gateway-envoy
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: monitoring-oauth2-proxy-team-a # Be careful, this is the oauth2-proxy service name
          port: 80
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /oauth2/start # oauth2-proxy start endpoint
        - path:
            type: Exact
            value: /oauth2/static/css/bulma.min.css # oauth2-proxy static assets

And here is an example of "real" HTTPRoute resource to alertmanager application to which the SecurityPolicy will be applied:

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: alertmanager-team-a
  namespace: monitoring-team-a
spec:
  hostnames:
  - alertmanager-team-a.<YOUR_DOMAIN>
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: main
    namespace: gateway-envoy
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: alertmanager-service-team-a
      port: 9090
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /
status:
  parents: []

Next, create a SecurityPolicy resource that targets the HTTPRoute and configures Envoy Gateway to use the oauth2-proxy instance for external authentication. Here is an example SecurityPolicy:

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: oauth2-alertmanager-team-a
  namespace: monitoring-team-a
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: alertmanager-team-a
  extAuth:
    failOpen: false
    headersToExtAuth:
    - Cookie
    - X-Auth-Request-User
    - X-Auth-Request-Groups
    - X-Auth-Request-Email
    - X-Auth-Request-Preferred-Username
    - X-Forwarded-Proto
    - X-Forwarded-Host
    - X-Forwarded-Uri
    - X-Auth-Request-Access-Token
    - Referer
    - X-Auth-Request-Redirect
    - Host
    http:
      backendRefs:
        - name: monitoring-oauth2-proxy-team-a
          kind: Service
          port: 80
      path: "/?rd=https://alertmanager-team-a.<YOUR_DOMAIN>" # this parameter is very important, it tells oauth2-proxy where to redirect after successful authentication. Without it oauth2-proxy will redirect to itself.
      headersToBackend:
      - "Path"
      - "X-Auth-Request-Access-Token"
      - "X-Auth-Request-User"
      - "X-Auth-Request-Groups"
      - "X-Auth-Request-Email"
      - "X-Auth-Request-Preferred-Username"
      - "X-Forwarded-Host"
      - "X-Auth-Request-Redirect"
    recomputeRoute: true

The above can be repeated for the prometheus application by creating similar HTTPRoute and SecurityPolicy resources, adjusting names and hostnames accordingly.

Conclusion

This guide demonstrated how to set up Envoy Gateway API with external authentication using oauth2-proxy instances per team in an AKS cluster. This setup allows multiple teams to securely access their applications while maintaining isolated authentication sessions and configurations. Adjust the configurations as needed to fit your specific requirements and security policies. The setup in some cases simplifies management of Entra ID app registrations by allowing reuse across multiple applications, saving resources and administrative overhead of managing multiple oauth2-proxy instances.