Falco: A Security Camera For Kubernetes Applications

Security is always a key concern while running application in any type of system may it be in bare metal or in containerized world. With the increase in number of ephemeral containers, it starts getting harder to keep track of what is running inside each container. Specially in a Kubernetes cluster, containers are spun-up and killed in no time which raises the challenge of threat detection.

A malicious process could run in a container, process or send some private data to third party and the container could get killed. If we don’t have a proper threat & anomaly detection mechanism, we may not be aware of what is running inside.

Falco, a cloud native security project developed initially by Sysdig makes it possible for real time monitoring and alerting based on pre-defined and custom rules. It acts as an intelligent security camera which can monitor files, process, networking events happening inside every single pod inside the cluster and alert if a behaviour steps outside specified policy.

Falco Rules

Falco is all based on rules. Example rules could be:

1. Installation of packages, libraries inside any container
2. Creation, deletion, rename and modification of files and folders inside the container after it starts running
3. Execution of binaries like bash, ssh, docker binary, debian binaries, vpn client, mail binaries
4. Unusual outbound traffic
5. Any ssh connection to/from container
6. Change of container files from host machine
7. Detect an attempt to start a pod with a container image outside of a list of allowed images.
8. Detect an attempt to start a pod with a privileged container
9. Attempt to attach/exec to a pod
10. Creation of new namespace etc.

We can define our own set of rule which is very specific for our application. Falco alerts with a real time message if any of the rules is violated.

Falco Output

Falcosidekick, a new addon for Falco which runs as daemon helps in extending the outputs of Falco rules execution. It takes a Falco’s events and forward them to different outputs in a fan-out way. There are a number of outputs ranging from Slack, alertmanager, lambda function, datadog, elasticsearch, grafana, pagerduty, opsgenie, google cloud function etc.

Falco Installation

Falco can run as a daemon inside docker container or as a daemon inside Kubernetes cluster. If we choose to run as daemonset, a new container runs inside each node and process the data of all the container running inside the node.

Each of the Falco daemon consumes 150–300Mi memory and 100m of CPU in general state. Likewise for FalcoSidekick.

There is helm chart available for both which makes the installation very easy. Here is the official helm chart.

It’s better to segregate the falco daemon and other requirements inside a separate namespace. After cloning the repo, install the chart along with sidekick

helm install falco falco/  -n falco --set falcosidekick.enabled=true

This will install both the charts.

If we want to output the events to slack, the following values need to be passed:

slack:
 webhookurl: “https://hooks.slack.com/services/*****"
 footer: “”
 icon: “”
 username: “”
 outputformat: “all”
 minimumpriority: “warning”
 messageformat: “”

This will send a nice message if any rule is triggered. For example, adding any packages inside pod

Check for the values file inside falco and falcosidekick chart. The pre-defined rules are inside falco/ . Feel free to add/update the rules as per need.

Priority for Output

In Falco, a minimum priority can be set to send the output to multiple targets. The order of priority is: emergency|alert|critical|error|warning|notice|informational|debug

For example, we might want our pagerduty alert if a new library is installed inside inside a container as ciritical but send a warning if a log file is created which can be recorded to Elasticsearch output. With this, in case of incident, we can always view the threats on multiple outputs without creating too much alarms. Also, the detected threats could be assigned to appropriate team based on priority.

FalcoSidekick UI

Falcosidekick offers a UI where we can view all the events recorded and filtered based on priority, rule and timerange.

falcosidekick UI

We can see the Rule Timeline as well as the number of events executed along with the priority.

Once the helm chart is installed, above dashboard can be accessed with port-forward:

kubectl port-forward svc/falco-falcosidekick-ui 2802:2802 -n falco

And UI accessible through http://localhost:2802. It doesn’t have inbuilt authentication.