Enrich Network Policy in Kubernetes Cluster with Calico

image src: tigera.io

Control of traffic(ingress/egress) is crucial for a secured Kubernetes cluster. Kubernetes Network Policy provides the offering for so by default but that’s limited to OSI layer 3/4 and is not rich in options. One of the network policy from official doc:

Here, we can limit the traffic from/to a container based on one or many of options: pod label, ip CIDR block, protocol(TCP/UDP/SCTP), source-destination namespace.

Network policy alone is not sufficient if we want even more fine grained control of traffic. For example, we cannot apply the policy on a specific set of node. It’s too binded with namespace and IP level only. If we want to apply policy in global level to all namespace, this has to be applied one-by-one in each namespace. Also, the policy is not aware of L7 traffic.

In this post, we will walkthrough with a networking and network security solution which provides richer network policies and is usable not only for Kubernetes cluster but also in bare metal servers, openshift and openstack too. The solution is Calico by Tigera and feature rich in networking and security.

Installation

The installation step is straight forward with Helm Charts where we need to set the provider name like GKE, EKS.

echo '{ installation: {kubernetesProvider: GKE }}' > custom-values.yaml

The installation enables Custom Resource Definition for managing lifecycle of calico resources.

Hierarchical Policy

For providing effective security policy enforcement, Calico suggest below hierarchical policy which combines both the rules of Calico as well as Kubernetes Network policy and assignment of specific team from application to network level.

Global Network Policy

Enforcing network policy clusterwide is possible by specifying label and the label is applicable irrespective of namespace.

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: deny-transaction
spec:
  selector: tier == 'transaction'
  ingress:
    - action: Deny
      protocol: TCP
      source:
        selector: tier == 'frontend'

In above example, pods with label tier == transaction is not reachable through frontend containers in any namespace.

Network Policy in Order

When applying multiple policies, one may try to overide another. To avoid such cases, Calico enables us to use order value. For example, order:10 will be applied before order:20 in below example.

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: drop-other-ingress
spec:
  order: 20
  #...deny policy rules here...

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: allow-cluster-internal-ingress
spec:
  order: 10
  #...allow policy rules here...

Here, the precedence is from the lowest to highest value.

Generate Logs of Traffic

With Kubernetes Network Policy, after the policy is applied which is unexpected, it’s hard to debug and find out failures. But Calico allows us to generate the logs of successful and failed traffic flow in the cluster.

apiVersion: projectcalico.org/v3
kind: NetworkPolicy
Metadata:
  name: deny-transaction
  namespace: production
Spec:
  selector: tier == 'transaction'
  types:
    - Ingress
    - Egress
  ingress:
    - action: Log
      protocol: TCP
      source:
        selector: tier == 'frontend'
    - action: Deny
      protocol: TCP
      source:
        selector: tier == 'frontend'

Here, the incoming traffic to transaction service from frontend tier is denied as well as logged to syslog. This makes it easier to know if unexpected traffic is trying to get into or any unwanted restriction is applied which might be causing failures.

Policy for Host(Node)

One level below the namespace and container networking is Host(Worker Node). The traffic flow on worker node with label "role == ‘favorite-k8s-worker'" is restricted for ingress from the management CIDR and egress to specific IP and UDP ports.

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: favorite-k8s-worker
spec:
  selector: "role == 'favorite-k8s-worker'"
  order: 0
  ingress:
    - action: Allow
      protocol: TCP
      source:
        nets:
          - '<your management CIDR>'
      destination:
        ports: [22]
    - action: Allow
      protocol: ICMP
    - action: Allow
      protocol: TCP
      destination:
        ports: [10250]
  egress:
    - action: Allow
      protocol: TCP
      destination:
        nets:
          - '<your etcd IP>/32'
        ports: [2379]
    - action: Allow
      protocol: UDP
      destination:
        ports: [53, 67]

This way, we can restrict connections to DB or some specific IP via a set of worker node.

Encrypt Pod Traffic

Calico supports WireGuard encryption for both IPv4 and IPv6 traffic by providing transport-level security for on-the-wire and in-cluster pod traffic.

Firstly, we need to install wireguard on the worker node image(AMI) and enable IPv4 WireGuard encryption across all the nodes using the following command.

kubectl patch felixconfiguration default --type='merge' -p '{"spec":{"wireguardEnabled":true}}'

Defense Against DoS attacks

Calico policy can help mitigate DoS attacks at earliest by creating host endpoint and deny list.

apiVersion: projectcalico.org/v3
kind: HostEndpoint
metadata:
  name: production-host
  labels:
    apply-dos-mitigation: 'true'
spec:
  interfaceName: eth0
  node: secured
  expectedIPs: ['10.0.0.1']

apiVersion: projectcalico.org/v3
kind: GlobalNetworkSet
metadata:
  name: dos-mitigation
  labels:
    dos-deny-list: 'true'
spec:
  nets:
    - '1.2.3.4/32'
    - '5.6.0.0/16'

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: dos-mitigation
spec:
  selector: apply-dos-mitigation == 'true'
  doNotTrack: true
  applyOnForward: true
  types:
    - Ingress
  ingress:
    - action: Deny
      source:
        selector: dos-deny-list == 'true'

The GlobalNetworkPolicy will enforce the packets to be dropped from all the services in all namespaces.

Apart from above mentioned features, there are other interesting options which can be explored with Calico for strengthening the security cluster wide. Like, enforcing policy for Istio in application layer, ICMP/PING rules, allow/deny traffic from/to specific IP set out of calico installed cluster etc.