Kubernetes Logging with Fluent Bit, Elasticsearch and Kibana

Whether it be logging cluster activity or debugging problem in application, logging is the first step. On Kubernetes world, containers are temporary entities which looses logs after a restart for any reason. So, I would prefer to have a proper logging setup of the cluster before running real applications. There are many enterprise solutions for that like, logz.io, papertrail, logDNA etc. On Google Kubernetes Engine(GKE), the logs of containers are captured from its own logging service which can be enabled or disabled from cluster settings.

In this post, I am going to walk through logging architecture using open source applications: fluent bit, elasticsearch, kibana and installing them with helm package manager. Means, we will be using helm charts of each of them to avoid manual workout.

image: computingforgeeks.com

If you haven’t used helm for managing applications, take some time to use interactive learning platform katakoda which has simpler learning scenario for helm: https://www.katacoda.com/courses/helm. I have written a blog about the ease helm brings on application management in Kubernetes:

After Using Helm for 2 Years in Production, Now Helm3

Following are the charts we will be using:

1. Elastic stack: https://github.com/elastic/helm-charts
2. Fluent Bit: https://github.com/dwdraju/fluent-bit-chart

Install Elasticsearch

$ cd helm-charts/elasticsearch
$ helm install elasticsearch . --set replicas=1 --set minimumMasterNodes=1 --imageTag=7.13.4

Here, I set only one replica for testing purpose.

Install Kibana

$ cd helm-charts/kibana
$ helm install kibana . --imageTag=7.13.4

Install Fluent-bit

$ cd fluent-bit-chart
$ helm install fluent-bit .

Time to view logs

We use kibana to view the logs and get metrics out of it. You can expose the kibana service to loadbalancer and access it using url. Make sure you don’t install default profile above if using public loadbalancer.

For now, I am port forwarding to my local

kubectl port-forward svc/kibana-kibana 5601:5601

Go to http://localhost:5601

There we have nice UI with newly created index of logs sent from fluent-bit.

Kibana: First view

If you go to “Discover” section, there will be the logs of stdout of all the containers running in the Kubernetes cluster.

Adding Filter

We might not need to send all the logs of all namespaces, for that input filter can be adjusted. Kubernetes stores log files inside /var/log/containers folder in the format: DeploymentName_NAMESPACE_ContainernameHASH.log . So, we add following filter where the deployment with dev prefix and container name auth is sent.

input:
  tail:
    memBufLimit: 5MB
    parser: docker
    path: /var/log/containers/dev-*_default_auth*.log
    ignore_older: ""

That’s for now, I will create a youtube video of this post shortly. Find me on linkedin, twitter.