Istio Proxy(Envoy) with Nginx Ingress

Istio service mesh offers a quick and easy way to secure communication in a Kubernetes cluster. We will discuss setting up MTLS in a Kubernetes cluster that is using the Nginx ingress controller instead of the Istio ingress gateway. Istio proxy will not be enabled for the entire cluster, instead the focus will be on specific namespace only.

Download Istio as defined below, the installation will be minimal so as to focus on service only.

Download:

https://istio.io/downloadIstio | ISTIO_VERSION=1.4.3 sh -

Edit istio/install/Kubernetes/helm/istio/requirements.yaml and remove all but below components.

dependencies:
  - name: sidecarInjectorWebhook
    version: 1.4.7
    condition: sidecarInjectorWebhook.enabled
  - name: security
    version: 1.4.7
    condition: security.enabled
  - name: pilot
    version: 1.4.7
    condition: pilot.enabled
  - name: galley
    version: 1.4.7
    condition: galley.enabled
  - name: kiali
    version: 1.4.7
    condition: kiali.enabled
  - name: tracing
    version: 1.4.7
    condition: tracing.enabled

Edit istio/install/Kubernetes/helm/istio/values.yaml as below.

# Top level istio values file has the following sections.
#
# global: This file is the authoritative and exhaustive source for the global section.
#
sidecarInjectorWebhook:
  enabled: true
  rewriteAppHTTPProbe: true
galley:
  enabled: true
mixer:
  policy:
    # if policy is enabled the global.disablePolicyChecks has affect.
    enabled: true
telemetry:
    enabled: true
pilot:
  enabled: true
security:
  enabled: true
tracing:
  enabled: true
kiali:
  enabled: true
  prometheusAddr: http://prometheus:9090
  dashboard:
    jaegerURL: http://jaeger-query.istio-system:16686 
  external_services:
    tracing:
      in_cluster_url: http://tracing.istio-system/jaeger
# Common settings used among istio subcharts.
global:
  hub: docker.io/istio
# Default tag for Istio images.
  tag: 1.4.7
  logging:
    level: "default:info"
  monitoringPort: 15014
k8sIngress:
    enabled: false
    gatewayName: ingressgateway
    enableHttps: false
proxy:
    # Configuration for the proxy init container
    init:
      resources:
        limits:
          cpu: 100m
          memory: 50Mi
        requests:
          cpu: 10m
          memory: 10Mi
    # use fully qualified image names for alternate path to proxy.
    image: proxyv2
# cluster domain. Default value is "cluster.local".
    #: "cluster.local"
# Resources for the sidecar.
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
      limits:
        cpu: 500m
        memory: 1024Mi
# Controls number of Proxy worker threads.
    # If set to 0, then start worker thread for each CPU thread/core.
    concurrency: 2
# Configures the access log for each sidecar.
    # Options:
    #   "" - disables access log
    #   "/dev/stdout" - enables access log
    accessLogFile: "/dev/stdout"
# Configure how and what fields are displayed in sidecar access log. Setting to
    # empty string will result in default log format
    accessLogFormat: ""
# Configure the access log for sidecar to JSON or TEXT.
    accessLogEncoding: JSON
# Configure envoy gRPC access log service.
    envoyAccessLogService:
      enabled: false
      host: # example: accesslog-service.istio-system
      port: # example: 15000
      tlsSettings:
        mode: DISABLE # DISABLE, SIMPLE, MUTUAL, ISTIO_MUTUAL
        clientCertificate: # example: /etc/istio/als/cert-chain.pem
        privateKey:        # example: /etc/istio/als/key.pem
        caCertificates:    # example: /etc/istio/als/root-cert.pem
        sni:               # example: als.somedomain
        subjectAltNames: []
        # - als.somedomain
      tcpKeepalive:
        probes: 3
        time: 10s
        interval: 10s
logLevel: ""
    componentLogLevel: ""
    dnsRefreshRate: 300s
    protocolDetectionTimeout: 100ms
#If set to true, istio-proxy container will have privileged securityContext
    privileged: false
# If set, newly injected sidecars will have core dumps enabled.
    enableCoreDump: false
# Image used to enable core dumps. This is only used, when "enableCoreDump" is set to true.
    enableCoreDumpImage: ubuntu:xenial
# Default port for Pilot agent health checks. A value of 0 will disable health checking.
    statusPort: 15020
# The initial delay for readiness probes in seconds.
    readinessInitialDelaySeconds: 60
# The period between readiness probes.
    readinessPeriodSeconds: 5
# The number of successive failed probes before indicating readiness failure.
    readinessFailureThreshold: 30
    includeIPRanges: "100.64.0.0/13"
    excludeIPRanges: ""
    excludeOutboundPorts: ""
# pod internal interfaces
    kubevirtInterfaces: ""
    includeInboundPorts: ""
    excludeInboundPorts: ""
# This controls the 'policy' in the sidecar injector.
    autoInject: enabled
    envoyStatsd:
      # If enabled is set to true, host and port must also be provided. Istio no longer provides a statsd collector.
      enabled: false
      host: # example: statsd-svc.istio-system
      port: # example: 9125
envoyMetricsService:
      enabled: false
      host: # example: metrics-service.istio-system
      port: # example: 15000
      tlsSettings:
        mode: DISABLE # DISABLE, SIMPLE, MUTUAL, ISTIO_MUTUAL
        clientCertificate: # example: /etc/istio/ms/cert-chain.pem
        privateKey:        # example: /etc/istio/ms/key.pem
        caCertificates:    # example: /etc/istio/ms/root-cert.pem
        sni:               # example: ms.somedomain
        subjectAltNames: []
        # - ms.somedomain
      tcpKeepalive:
        probes: 3
        time: 10s
        interval: 10s
    tracer: "zipkin"
proxy_init:
    # Base name for the istio-init container, used to configure iptables.
    image: proxyv2
  imagePullPolicy: IfNotPresent
controlPlaneSecurityEnabled: true
disablePolicyChecks: true
policyCheckFailOpen: false
enableTracing: true
tracer:
    lightstep:
      address: ""                # example: lightstep-satellite:443
      accessToken: ""            # example: abcdefg1234567
      secure: true               # example: true|false
      cacertPath: ""             # example: /etc/lightstep/cacert.pem
    zipkin:
      address: ""
    datadog:
      # Host:Port for submitting traces to the Datadog agent.
      address: "$(HOST_IP):8126"
    stackdriver:
      # enables trace output to stdout.
      debug: false
      # The global default max number of attributes per span.
      maxNumberOfAttributes: 200
      # The global default max number of annotation events per span.
      maxNumberOfAnnotations: 200
      # The global default max number of message events per span.
      maxNumberOfMessageEvents: 200
  mtls:
    enabled: true
    auto: false
imagePullSecrets: []
arch:
    amd64: 2
    s390x: 2
    ppc64le: 2
oneNamespace: false
defaultNodeSelector: {}
defaultTolerations: []
configValidation: false
  meshExpansion:
    enabled: false
    useILB: false
multiCluster:
    enabled: false
    clusterName: ""
  defaultResources:
    requests:
      cpu: 10m
      memory: 128Mi
    limits:
      cpu: 100m
      memory: 128Mi
defaultPodDisruptionBudget:
    enabled: true
  priorityClassName: ""
  useMCP: true
  trustDomain: ""
  trustDomainAliases: []
meshID: ""
  outboundTrafficPolicy:
    mode: ALLOW_ANY
sds:
    enabled: false
    udsPath: ""
token:
      aud: istio-ca
meshNetworks: {}
network: ""
localityLbSetting:
enableHelmTest: false
certificates: []
  operatorManageWebhooks: false

Install Istio

helm install istio/install/kubernetes/helm/istio-init --name istio-init --namespace istio-system
$ kubectl get pods -n istio-system
# The pods should be in complete state, before excuting the below command
helm install istio/install/kubernetes/helm/istio --name istio --namespace istio-system

Once istio is installed, setup destination rules and policies for the namespaces where envoy is to enabled to enable sidecar injection, disable communication over envoy proxy to Kubernetes API server. A mesh policy will be needed to be set up to allow communication to non-istio enabled namespaces inside the cluster. The policy and destination rule can be created as below.

The below policy and destination rule needs to be created for all namespaces where envoy will be enabled.

---
apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: "default"
  namespace: "namespacename"
spec:
  peers:
  - mtls:
      mode: PERMISSIVE
---
apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
  name: "default"
  namespace: "namespacename"
spec:
  host: "*.svc."
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
---
apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
  name: "default"
  namespace: "istio-system"
spec:
  host: "*.svc."
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

Create MESH policy to enable communication between istio enabled and not enabled namespace.

apiVersion: authentication.istio.io/v1alpha1
kind: MeshPolicy
metadata:
  name: default
spec:
  peers:
  - mtls:
      mode: PERMISSIVE

Destination rule is to disable calls to Kubernetes API server

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
 name: disable-mtls-to-k8s
 namespace: istio-system
spec:
 host: "kubernetes.default.svc.cluster.local"
 trafficPolicy:
   tls:
     mode: DISABLE

To enable envoy sidecar injection to pods, execute below command to label namespace.

kubectl label namespace namespacename istio-injection=enabled
# To disbale use below command
kubectl label namespace default istio-injection-

Annotate your Nginx deployment as below and application ingress as below

traffic.sidecar.istio.io/includeInboundPorts: ""

Once a namespace is labeled as above, delete one of the pods and check if the pod comes up with istio-proxy containers injected. Check traffic to a pod from different namespace with istio proxy enabled and to a pod with istio proxy disabled. Also check traffic to the outside of the cluster from the pod, like https://www.google.com . If it fails create a service entry like below.

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: google.com
spec:
  hosts:
  - '*.google.com'
  location: MESH_EXTERNAL
  ports:
  - name: https
    number: 443
    protocol: HTTPS
  resolution: NONE

Follow us on Twitter 🐦 and Facebook 👥 and Instagram 📷 and join our Facebook and Linkedin Groups 💬.

To join our community Slack team chat 🗣️ read our weekly Faun topics 🗞️, and connect with the community 📣 click here⬇

If this post was helpful, please click the clap 👏 button below a few times to show your support for the author! ⬇