Kubernetes Logging with Fluent Bit, Elasticsearch and Kibana
Whether it be logging cluster activity or debugging problem in application, logging is the first step. On Kubernetes world, containers are temporary entities which looses logs after a restart for any reason. So, I would prefer to have a proper logging setup of the cluster before running real applications. There are many enterprise solutions for that like, logz.io, papertrail, logDNA etc. On Google Kubernetes Engine(GKE), the logs of containers are captured from its own logging service which can be enabled or disabled from cluster settings.
In this post, I am going to walk through logging architecture using open source applications: fluent bit, elasticsearch, kibana and installing them with helm package manager. Means, we will be using helm charts of each of them to avoid manual workout.
If you haven’t used helm for managing applications, take some time to use interactive learning platform katakoda which has simpler learning scenario for helm: https://www.katacoda.com/courses/helm. I have written a blog about the ease helm brings on application management in Kubernetes:
After Using Helm for 2 Years in Production, Now Helm3
Since the very starting of migrating Pagevamp’s services to Kubernetes on Google Cloud — GKE, we became a fan of Helm…
Following are the charts we will be using:
- Elastic stack: https://github.com/elastic/helm-charts
- Fluent Bit: https://github.com/dwdraju/fluent-bit-chart
$ cd helm-charts/elasticsearch
$ helm install elasticsearch . --set replicas=1 --set minimumMasterNodes=1 --imageTag=7.13.4
Here, I set only one replica for testing purpose.
$ cd helm-charts/kibana
$ helm install kibana . --imageTag=7.13.4
$ cd fluent-bit-chart
$ helm install fluent-bit .
Time to view logs
We use kibana to view the logs and get metrics out of it. You can expose the kibana service to loadbalancer and access it using url. Make sure you don’t install default profile above if using public loadbalancer.
For now, I am port forwarding to my local
kubectl port-forward svc/kibana-kibana 5601:5601
Go to http://localhost:5601
There we have nice UI with newly created index of logs sent from fluent-bit.
If you go to “Discover” section, there will be the logs of stdout of all the containers running in the Kubernetes cluster.
We might not need to send all the logs of all namespaces, for that input filter can be adjusted. Kubernetes stores log files inside
/var/log/containers folder in the format:
DeploymentName_NAMESPACE_ContainernameHASH.log . So, we add following filter where the deployment with
dev prefix and container name
auth is sent.