Application Networking Integration

This guide covers how to expose new applications through the prokube networking stack, including Istio VirtualServices, dedicated Ingress resources, and sidecar injection configuration.

Architecture Overview

prokube uses a layered networking architecture:

External Traffic
      │
      ▼
┌─────────────────────────────────────────────┐
│  NGINX Ingress Controller                   │
│  (TLS termination, cert-manager)            │
└─────────────────────────────────────────────┘
      │
      ▼
┌─────────────────────────────────────────────┐
│  Istio Ingress Gateway                      │
│  (istio-system namespace)                   │
└─────────────────────────────────────────────┘
      │
      ▼
┌─────────────────────────────────────────────┐
│  Istio VirtualServices                      │
│  (path-based routing to services)           │
└─────────────────────────────────────────────┘
      │
      ▼
┌─────────────────────────────────────────────┐
│  Application Service (with Istio sidecar)   │
└─────────────────────────────────────────────┘

Why Istio?

Istio is the service mesh that handles routing and security for applications in prokube. It provides:

• Authentication enforcement: The Istio gateway validates JWT tokens and redirects unauthenticated users to login. This is how prokube ensures only logged-in users can access applications.
• mTLS: Automatic encryption of traffic between services inside the cluster (for services with sidecars).
• Traffic routing: VirtualServices provide flexible path-based routing without needing separate Ingress resources for each application.
• Observability: Metrics and tracing for service-to-service communication.

Is Istio required? There are two separate concerns:

1. Routing via VirtualServices: Required if you want your application accessible through the main prokube URL (e.g., cluster.example.com/myapp). The VirtualService tells Istio's gateway where to route traffic.

2. Sidecar injection: Optional. Your application doesn't need to be part of the service mesh to receive traffic from a VirtualService. The gateway routes to your Service, and that works whether or not your pods have Istio sidecars.

   With sidecars enabled, your application can use mTLS for encrypted traffic between services (enable via PeerAuthentication in strict mode), and gains access to detailed traffic metrics and distributed tracing. However, prokube enforces a deny-all policy by default for mesh traffic—services with sidecars cannot receive traffic unless you create an AuthorizationPolicy that explicitly allows it. See Authorization Policies for details.

Most applications are exposed via VirtualServices that route traffic from the main Ingress through the Istio ingress gateway. A dedicated Ingress is only needed when an application cannot run with a URL prefix (e.g., requires its own subdomain).

Understanding URL Prefixes

A URL (web address) has several parts. The path is everything after the domain name:

https://cluster.example.com/mlflow/experiments/1
       └───────┬──────────┘└───────┬───────────┘
            domain              path

A prefix is the first segment of the path that identifies which application should handle the request:

https://cluster.example.com/mlflow/experiments/1
                            └──┬──┘└─────┬─────┘
                            prefix   rest of path

With prefix-based routing, multiple applications share a single domain:

URL	Routed to
cluster.example.com/mlflow/...	MLflow
cluster.example.com/grafana/...	Grafana
cluster.example.com/minio/...	MinIO

This is the preferred approach because it requires only one DNS record and one TLS certificate. However, the application must be aware it's running under a prefix—when it generates links, they must include /mlflow/ rather than starting from /. Most modern applications support this via configuration or by reading the X-Forwarded-Prefix header.

If an application cannot handle prefixes (e.g., it hardcodes absolute URLs like /api/data instead of respecting the prefix), it needs its own subdomain like mlflow.example.com.

Istio Gateways

prokube provides two Istio gateways for different use cases:

Gateway	Namespace	Authentication	Use Case
kubeflow-gateway	kubeflow	Required (JWT)	User-facing applications
cluster-local-gateway	istio-system	None	Internal services, model serving endpoints

kubeflow-gateway (default)

Use this for most applications. Traffic through this gateway requires authentication—users must be logged in via Keycloak. This is enforced by an AuthorizationPolicy that denies requests without a valid JWT token.

gateways:
  - kubeflow/kubeflow-gateway

cluster-local-gateway

Use this for services that need to be called programmatically without user authentication, such as:

• Model inference endpoints (KServe)
• Internal APIs called by other services
• Webhook receivers

gateways:
  - istio-system/cluster-local-gateway

Traffic to this gateway is routed via the /serving path on the main ingress.

Applications with Their Own Authentication

Some applications handle authentication themselves, typically via SSO with Keycloak (e.g., MLflow, ArgoCD, MinIO). These still use kubeflow-gateway but need to be exempted from the gateway's authentication so they can handle auth independently.

You must add your application's path to both gateway AuthorizationPolicies. If you miss one, users will either get a 403 error or be redirected to the wrong login page.

See Authentication Integration - Exempt from Gateway Authentication for the files to edit and what happens if each is missing.

The application is then responsible for:

1. Redirecting unauthenticated users to Keycloak
2. Validating tokens/sessions
3. Managing its own authorization

Option 1: VirtualService (Preferred)

Use this approach when the application can run under a URL path prefix (e.g., https://cluster.example.com/myapp/).

Prerequisites

Your application still needs a standard Kubernetes Service. The VirtualService doesn't replace the Service—it tells Istio how to route external traffic to your Service:

VirtualService  →  Service  →  Pods
(routing rules)    (internal)   (your app)

Note

If you're integrating an existing application (e.g., from a Helm chart), it likely comes with a Service already. Reference that existing Service in your VirtualService.

Example Service (if you need to create one):

apiVersion: v1
kind: Service
metadata:
  name: myapp-service
  namespace: myapp
spec:
  selector:
    app: myapp
  ports:
    - port: 80
      targetPort: 8080

Basic VirtualService

Create a VirtualService that routes traffic from the Istio gateway to your Service:

# paas/myapp/base/virtualservice.yaml
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: myapp-vs
  namespace: myapp
spec:
  hosts:
    - "*"
  gateways:
    - kubeflow/kubeflow-gateway
  http:
    # Redirect /myapp to /myapp/ for consistency
    - match:
        - uri:
            exact: /myapp
      redirect:
        uri: /myapp/
        redirectCode: 301
    # Route all /myapp/* traffic to the service
    - match:
        - uri:
            prefix: /myapp/
      rewrite:
        uri: /
      route:
        - destination:
            host: myapp-service.myapp.svc.cluster.local
            port:
              number: 80

VirtualService with Prefix Header

Some applications need to know their URL prefix. Use the X-Forwarded-Prefix header:

# paas/myapp/base/virtualservice.yaml
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: myapp-vs
  namespace: myapp
spec:
  hosts:
    - "*"
  gateways:
    - kubeflow/kubeflow-gateway
  http:
    - match:
        - uri:
            exact: /myapp
      redirect:
        uri: /myapp/
        redirectCode: 301
    - headers:
        request:
          set:
            X-Forwarded-Prefix: /myapp
      match:
        - uri:
            prefix: /myapp/
      rewrite:
        uri: /
      route:
        - destination:
            host: myapp-service.myapp.svc.cluster.local
            port:
              number: 80

Reference Example

See the MLflow VirtualService at paas/mlflow/base/virtualservice.yaml for a production example.

Option 2: Dedicated Ingress

Use this approach when the application requires its own subdomain and cannot function with a URL prefix.

When to Use a Dedicated Ingress

• Application hardcodes absolute URLs without respecting prefix headers
• Application requires WebSocket connections that don't work through Istio
• Application needs specific Ingress annotations (e.g., large upload limits)
• Compliance requirements mandate traffic isolation

Basic Dedicated Ingress

# paas/myapp/base/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp-ingress
  namespace: myapp
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  ingressClassName: public
  rules:
    - host: myapp.example.com
      http:
        paths:
          - path: "/"
            pathType: Prefix
            backend:
              service:
                name: myapp-service
                port:
                  number: 80
  tls:
    - hosts:
        - myapp.example.com
      secretName: myapp-tls-cert

Ingress with Custom Annotations

For applications with special requirements:

# paas/myapp/base/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp-ingress
  namespace: myapp
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    # Large file uploads
    nginx.ingress.kubernetes.io/proxy-body-size: "100m"
    # WebSocket support
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    # Increase buffer size for large headers (e.g., JWTs)
    nginx.ingress.kubernetes.io/proxy-buffer-size: "32k"
spec:
  ingressClassName: public
  rules:
    - host: myapp.example.com
      http:
        paths:
          - path: "/"
            pathType: Prefix
            backend:
              service:
                name: myapp-service
                port:
                  number: 80
  tls:
    - hosts:
        - myapp.example.com
      secretName: myapp-tls-cert

Reference Example

See the MinIO Ingress at paas/ingress/base/minio_ingress.yaml for a production example of a dedicated Ingress.

Istio Sidecar Injection

Istio sidecar injection adds an Envoy proxy to each pod, enabling mTLS, traffic management, and observability.

Namespace-Level Injection (Recommended)

Enable injection for all pods in a namespace:

# paas/myapp/base/ns.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: myapp
  labels:
    istio-injection: enabled

Pod-Level Injection

For fine-grained control, use pod labels:

# Enable injection for specific deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: myapp
spec:
  template:
    metadata:
      labels:
        sidecar.istio.io/inject: "true"

Disabling Injection

Some workloads should not have sidecars (e.g., jobs, init containers, gateways):

# Disable injection for specific deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-worker
spec:
  template:
    metadata:
      labels:
        sidecar.istio.io/inject: "false"

Common cases where you might disable injection:

• Admission webhooks (API server isn't part of the mesh and can't connect through mTLS)
• Short-lived Jobs (sidecar prevents pod termination)
• Spark executors
• Training operators
• Databases with their own connection management

Authorization Policies

prokube uses a deny-all by default security model. A global AuthorizationPolicy in istio-system blocks all traffic to services with Istio sidecars unless explicitly allowed. This means:

• If your application has sidecar injection enabled, you must create an AuthorizationPolicy to allow traffic
• Without an allow policy, requests to your service will return 403 Forbidden

Allowing Traffic to Your Application

To get your application working, create an AuthorizationPolicy that allows all traffic to your namespace:

# paas/myapp/base/authorizationpolicy.yaml
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-all-in-namespace
  namespace: myapp
spec:
  action: ALLOW
  rules:
    - {}  # Allow all traffic from any source

This policy allows any traffic (from the gateway, from other services, from anywhere) to reach services in the myapp namespace. It's the simplest way to get things working.

Security consideration

An allow-all policy disables Istio's deny-by-default protection for your namespace. In multi-tenant or internet-exposed clusters, this means any service (or compromised workload) can reach your application. For production, define narrower policies that restrict access to specific sources—see the Istio AuthorizationPolicy documentation.

Start permissive, tighten later

For initial development, an allow-all policy is fine to get things working. Once your application is functional, review and tighten access as needed.

Gateway Path Exceptions

If your application handles its own authentication (e.g., has built-in OIDC support), you need to exempt it from the gateway's JWT requirement. See Authentication Integration - Exempt from Gateway Authentication for details.

Complete Application Example

Here's a complete kustomization structure for a new application:

paas/myapp/
├── base/
│   ├── kustomization.yaml
│   ├── ns.yaml
│   ├── deployment.yaml
│   ├── service.yaml
│   └── virtualservice.yaml
└── overlays/
    └── production/
        └── kustomization.yaml

kustomization.yaml

# paas/myapp/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: myapp
resources:
  - ns.yaml
  - deployment.yaml
  - service.yaml
  - virtualservice.yaml

Namespace

# paas/myapp/base/ns.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: myapp
  labels:
    istio-injection: enabled

Service

# paas/myapp/base/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: myapp-service
  namespace: myapp
spec:
  selector:
    app: myapp
  ports:
    - port: 80
      targetPort: 8080

Decision Flowchart

Does the app support running with a URL prefix?
│
├── YES → Use VirtualService (Option 1)
│         │
│         ├── Does it need to know the prefix?
│         │   └── YES → Add X-Forwarded-Prefix header
│         │
│         └── Does it need public (unauthenticated) access?
│             └── YES → Add path exception to AuthorizationPolicy
│
└── NO → Use Dedicated Ingress (Option 2)
         │
         └── Does it need Istio features (mTLS, observability)?
             ├── YES → Enable sidecar injection in namespace
             └── NO → Disable sidecar injection

Troubleshooting

VirtualService not routing traffic

1. Verify the gateway reference is correct: kubeflow/kubeflow-gateway
2. Check the service FQDN: service-name.namespace.svc.cluster.local
3. Ensure the namespace has the VirtualService applied: kubectl get vs -n myapp

Sidecar injection not working

1. Check namespace labels: kubectl get ns myapp --show-labels
2. Verify istio-injection label: istio-injection=enabled
3. Check pod annotations or labels aren't overriding: kubectl get pod -n myapp -o yaml | grep sidecar

503 errors after enabling Istio

1. Check if the service port names follow Istio conventions (e.g., http, grpc)
2. Verify PeerAuthentication isn't blocking traffic
3. Check destination rule TLS settings match