Kubernetes Logging with Fluent Bit, Elasticsearch and Kibana

image: computingforgeeks.com
  1. Elastic stack: https://github.com/elastic/helm-charts
  2. Fluent Bit: https://github.com/dwdraju/fluent-bit-chart

Install Elasticsearch

$ cd helm-charts/elasticsearch
$ helm install elasticsearch . --set replicas=1 --set minimumMasterNodes=1 --imageTag=7.13.4

Install Kibana

$ cd helm-charts/kibana
$ helm install kibana . --imageTag=7.13.4

Install Fluent-bit

$ cd fluent-bit-chart
$ helm install fluent-bit .

Time to view logs

We use kibana to view the logs and get metrics out of it. You can expose the kibana service to loadbalancer and access it using url. Make sure you don’t install default profile above if using public loadbalancer.

kubectl port-forward svc/kibana-kibana 5601:5601
Kibana: First view

Adding Filter

We might not need to send all the logs of all namespaces, for that input filter can be adjusted. Kubernetes stores log files inside /var/log/containers folder in the format: DeploymentName_NAMESPACE_ContainernameHASH.log . So, we add following filter where the deployment with dev prefix and container name auth is sent.

input:
tail:
memBufLimit: 5MB
parser: docker
path: /var/log/containers/dev-*_default_auth*.log
ignore_older: ""

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store