Skip to main content

K8s Ingress Controllers

Default Ingress​

By default, RKE deploys the NGINX ingress controller on all schedulable nodes.

note

As of v0.1.8, only workers are considered schedulable nodes, but before v0.1.8, worker and controlplane nodes were considered schedulable nodes.

RKE will deploy the ingress controller as a DaemonSet with hostNetwork: true, so ports 80, and 443 will be opened on each node where the controller is deployed.

note

As of v1.1.11, the network options of the ingress controller are configurable. See Configuring network options.

The images used for ingress controller is under the system_images directive. For each Kubernetes version, there are default images associated with the ingress controller, but these can be overridden by changing the image tag in system_images.

Scheduling Ingress Controllers​

If you only wanted ingress controllers to be deployed on specific nodes, you can set a node_selector for the ingress. The label in the node_selector would need to match the label on the nodes for the ingress controller to be deployed.

nodes:
- address: 1.1.1.1
  role: [controlplane,worker,etcd]
  user: root
  labels:
    app: ingress

ingress:
  provider: nginx
  node_selector:
    app: ingress

Ingress Priority Class Name​

Available as of RKE v1.2.6+

The pod priority is set by configuring a priority class name:

ingress:
  provider: nginx
  ingress_priority_class_name: system-cluster-critical

Tolerations​

Available as of v1.2.4

The configured tolerations apply to the default-http-backend Deployment.

ingress:
  tolerations:
  - key: "node.kubernetes.io/unreachable"
    operator: "Exists"
    effect: "NoExecute"
    tolerationseconds: 300
  - key: "node.kubernetes.io/not-ready"
    operator: "Exists"
    effect: "NoExecute"
    tolerationseconds: 300

To check for applied tolerations default-http-backend Deployment, use the following commands:

kubectl -n ingress-nginx get deploy default-http-backend -o jsonpath='{.spec.template.spec.tolerations}'

Disabling the Default Ingress Controller​

You can disable the default controller by specifying none to the ingress provider directive in the cluster configuration.

ingress:
    provider: none

Configuring NGINX Ingress Controller​

For the configuration of NGINX, there are configuration options available in Kubernetes. There are a list of options for the NGINX config map , command line extra_args and annotations.

ingress:
  provider: nginx
  options:
    map-hash-bucket-size: "128"
    ssl-protocols: SSLv2
  extra_args:
    enable-ssl-passthrough: ""

Disabling NGINX Ingress Default Backend​

As of v0.20.0, you can disable the default backend service for the ingress controller. This is possible because ingress-nginx will fall back to a local 404 page, and does not require a backend service. The service can be enabled/disabled with a boolean value.

ingress:
  default_backend: false
What happens if the field is omitted?

The value of default_backend will default to true. This maintains behavior with older versions of rke. However, a future version of rke will change the default value to false.

Configuring network options​

For Kubernetes v1.21 and up, the NGINX ingress controller no longer runs in hostNetwork: true but uses hostPorts for port 80 and port 443. This was done so the admission webhook can be configured to be accessed using ClusterIP so it can only be reached inside the cluster. If you want to change the mode and/or the ports, see the options below.

Here's an example of how to configure the nginx ingress controller using hostPort and override the default ports:

ingress:
  provider: nginx
  network_mode: hostPort
  http_port: 9090
  https_port: 9443
  extra_args:
    http-port: 3080
    https-port: 3443

Here's an example of how to configure the nginx ingress controller using hostNetwork:

ingress:
  provider: nginx
  network_mode: hostNetwork

Here's an example of how to configure the nginx ingress controller with no network mode which will make it run on the overlay network (for example, if you want to expose the nginx ingress controller using a LoadBalancer) and override the default ports:

ingress:
  provider: nginx
  network_mode: none
  extra_args:
    http-port: 8080
    https-port: 8443

Configuring an NGINX Default Certificate​

When configuring an ingress object with TLS termination, you must provide it with a certificate used for encryption/decryption. Instead of explicitly defining a certificate each time you configure an ingress, you can set up a custom certificate that's used by default.

Setting up a default certificate is especially helpful in environments where a wildcard certificate is used, as the certificate can be applied in multiple subdomains.

Prerequisites
  • Access to the cluster.yml used to create the cluster.
  • The PEM encoded certificate you will use as the default certificate.

  1. Obtain or generate your certificate key pair in a PEM encoded form.

  2. Generate a Kubernetes secret from your PEM encoded certificate with the following command, substituting your certificate for mycert.cert and mycert.key.

    kubectl -n ingress-nginx create secret tls ingress-default-cert --cert=mycert.cert --key=mycert.key -o yaml --dry-run=true > ingress-default-cert.yaml

  3. Include the contents of ingress-default-cert.yml inline with your RKE cluster.yml file. For example:

    addons: |-
      ---
      apiVersion: v1
      data:
        tls.crt: [ENCODED CERT]
        tls.key: [ENCODED KEY]
      kind: Secret
      metadata:
        creationTimestamp: null
        name: ingress-default-cert
        namespace: ingress-nginx
      type: kubernetes.io/tls

  4. Define your ingress resource with the following default-ssl-certificate argument, which references the secret we created earlier under extra_args in your cluster.yml:

    ingress:
      provider: "nginx"
      extra_args:
        default-ssl-certificate: "ingress-nginx/ingress-default-cert"

  5. Optional: If you want to apply the default certificate to ingresses in a cluster that already exists, you must delete the NGINX ingress controller pods to have Kubernetes schedule new pods with the newly configured extra_args.

    kubectl delete pod -l app=ingress-nginx -n ingress-nginx