This post walks through deploying Fluent Bit as a sidecar container alongside an application in Kubernetes. The sidecar pattern keeps your application container focused on its job while Fluent Bit handles log collection, transformation, and distribution to one or more destinations.
Why a sidecar instead of a DaemonSet
The standard Kubernetes logging setup runs Fluent Bit (or Fluentd) as a DaemonSet: one agent per node, reading container logs from the host filesystem at /var/log/containers/. That works fine for basic log collection.
The sidecar pattern is worth considering when:
- Your application writes structured logs to a file rather than stdout
- You need per-application log routing (different apps ship to different destinations)
- You want log parsing or enrichment specific to one application without affecting the cluster-wide agent
- You are in a multi-tenant environment and need strong isolation between log streams
The tradeoff is resource overhead. Each pod gets its own Fluent Bit instance instead of sharing one per node.
How it works
The application container and Fluent Bit container share a volume mounted at a common path. The application writes log files to that path. Fluent Bit tails those files, applies any parsing or filtering you have configured, and ships the output to your chosen destinations.
Pod
├── app-container (writes to /var/log/app/)
├── fluentbit-sidecar (reads /var/log/app/, ships to destinations)
└── shared-volume (emptyDir mounted to both)Step 1: A simple application that writes log files
For this walkthrough, the application is a minimal Python Flask service that writes structured JSON logs to a file. The same pattern applies to any application.
# app.py
import logging
import json
import os
from flask import Flask, request, jsonifyapp = Flask(_name_)
LOGPATH = os.environ.get('LOGPATH', '/var/log/app')
os.makedirs(LOGPATH, existok=True)
# JSON file handler
filehandler = logging.FileHandler(f'{LOGPATH}/app.log')
file_handler.setLevel(logging.INFO)
class JSONFormatter(logging.Formatter):
def format(self, record):
return json.dumps({
'timestamp': self.formatTime(record),
'level': record.levelname,
'service': 'my-api',
'message': record.getMessage(),
'path': getattr(record, 'path', None),
'status': getattr(record, 'status', None),
})
filehandler.setFormatter(JSONFormatter())
logger = logging.getLogger()
logger.addHandler(filehandler)
logger.setLevel(logging.INFO)
@app.route('/health')
def health():
return jsonify({'status': 'ok'})
@app.route('/api/data')
def data():
logger.info('request received', extra={'path': '/api/data', 'status': 200})
return jsonify({'data': 'example'})
if _name == 'main_':
app.run(host='0.0.0.0', port=8080)
```
Step 2: The Fluent Bit configuration
Fluent Bit configuration has three main sections: INPUT (where to read from), FILTER (optional transformations), and OUTPUT (where to send).
Create a ConfigMap with the Fluent Bit config:
apiVersion: v1
kind: ConfigMap
metadata:
name: fluentbit-config
namespace: my-app
data:
fluent-bit.conf: |
[SERVICE]
Flush 5
Daemon off
Log_Level info
Parsers_File parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020[INPUT]
Name tail
Path /var/log/app/.log
Parser json
Tag app.
RefreshInterval 5
MemBufLimit 10MB
SkipLongLines On
DB /var/log/fluentbit/flbkube.db
[FILTER]
Name recordmodifier
Match app.*
Record podname ${HOSTNAME}
Record namespace ${NAMESPACE}
Record container_name my-api
[OUTPUT]
Name es
Match app.*
Host ${ELASTICSEARCHHOST}
Port 9200
Index app-logs
Type doc
LogstashFormat On
LogstashPrefix my-api
Retry_Limit 5
[OUTPUT]
Name s3
Match app.*
bucket ${S3BUCKET}
region ${AWSREGION}
storedir /tmp/fluentbit-s3
totalfilesize 10M
uploadtimeout 10m
s3keyformat /logs/my-api/%Y/%m/%d/%H/%M/%S
parsers.conf: |
[PARSER]
Name json
Format json
TimeKey timestamp
TimeFormat %Y-%m-%d %H:%M:%S,%L
Time_Keep On
```
A few things worth noting in this config:
DB /var/log/fluentbit/flb_kube.db creates a small SQLite database that tracks the read position in each log file. Without this, a Fluent Bit restart reads the entire file from the beginning and you get duplicate log entries. The database path needs to be on a writable volume.
HTTP_Server On exposes a metrics endpoint at port 2020 on the Fluent Bit container. This is useful for health checks and for scraping Fluent Bit's own metrics with Prometheus.
MemBufLimit 10MB caps how much Fluent Bit buffers in memory per input before applying backpressure. Tune this based on your log volume.
Step 3: The Kubernetes deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-api
namespace: my-app
spec:
replicas: 2
selector:
matchLabels:
app: my-api
template:
metadata:
labels:
app: my-api
spec:
volumes:
# Shared log volume between app and Fluent Bit
- name: app-logs
emptyDir: {}
# Fluent Bit position database (survives container restart, not pod restart)
- name: fluentbit-db
emptyDir: {}
# Fluent Bit config from ConfigMap
- name: fluentbit-config
configMap:
name: fluentbit-configcontainers:
# ── Application container ──
- name: my-api
image: registry.example.com/my-api:1.0.0
ports:
- containerPort: 8080
env:
- name: LOG_PATH
value: /var/log/app
volumeMounts:
- name: app-logs
mountPath: /var/log/app
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 256Mi
livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 10
periodSeconds: 15
securityContext:
allowPrivilegeEscalation: false
runAsNonRoot: true
capabilities:
drop: ["ALL"]
# ── Fluent Bit sidecar ──
- name: fluentbit
image: fluent/fluent-bit:3.0
ports:
- containerPort: 2020
name: metrics
env:
- name: ELASTICSEARCHHOST
valueFrom:
secretKeyRef:
name: my-api-secrets
key: elasticsearch-host
- name: S3BUCKET
valueFrom:
secretKeyRef:
name: my-api-secrets
key: s3-bucket
- name: AWS_REGION
value: us-east-1
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: app-logs
mountPath: /var/log/app
readOnly: true
- name: fluentbit-db
mountPath: /var/log/fluentbit
- name: fluentbit-config
mountPath: /fluent-bit/etc
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: 200m
memory: 128Mi
livenessProbe:
httpGet:
path: /api/v1/health
port: 2020
initialDelaySeconds: 10
periodSeconds: 30
securityContext:
allowPrivilegeEscalation: false
runAsNonRoot: true
capabilities:
drop: ["ALL"]
```
The readOnly: true on the app-logs volume mount for Fluent Bit is worth emphasising. Fluent Bit only needs to read the log files, not write them. Mounting read-only is a small but correct security boundary.
Step 4: Deploy and verify
Apply the ConfigMap and Deployment:
kubectl apply -f fluentbit-configmap.yaml
kubectl apply -f deployment.yamlCheck that both containers in the pod are running:
kubectl get pods -n my-app
# NAME READY STATUS RESTARTS AGE
# my-api-7d9b8c6f4-xkp2n 2/2 Running 0 45s
# my-api-7d9b8c6f4-r9m4q 2/2 Running 0 45s# READY shows 2/2 - both app and fluentbit are healthy
```
Check Fluent Bit logs:
kubectl logs -n my-app my-api-7d9b8c6f4-xkp2n -c fluentbitYou should see Fluent Bit initialising and starting to tail your log file. If you see errors like [error] [output:es] could not connect to host, check your Elasticsearch connection details.
Hit the application endpoint a few times to generate some logs:
kubectl port-forward -n my-app deployment/my-api 8080:8080curl http://localhost:8080/api/data
curl http://localhost:8080/api/data
curl http://localhost:8080/api/data
```
Check the Fluent Bit metrics endpoint:
kubectl port-forward -n my-app my-api-7d9b8c6f4-xkp2n 2020:2020curl http://localhost:2020/api/v1/metrics
```
The output.es.proc_records metric tells you how many records have been successfully sent to Elasticsearch. If it is incrementing, logs are flowing.
Common problems
Fluent Bit starts but no logs appear in the output
First check if Fluent Bit is actually reading the file:
curl http://localhost:2020/api/v1/metrics | python3 -m json.tool | grep proc_recordsIf input.tail.proc_records is zero, Fluent Bit is not reading any records at all. Check that the log file exists in the shared volume:
kubectl exec -n my-app my-api-7d9b8c6f4-xkp2n -c fluentbit -- ls -la /var/log/app/If the file is there but proc_records is still zero, check the parser. JSON parsing fails silently if the log lines are not valid JSON. Switch to a dummy parser temporarily to confirm the file tail is working:
[INPUT]
Name tail
Path /var/log/app/*.log
Tag app.*Database file permission errors
If you see [error] could not open database, the Fluent Bit container does not have write permission to the database directory. Since we are running as non-root, the emptyDir volume needs to be writable by the Fluent Bit user.
Fix this with a security context on the volume:
securityContext:
fsGroup: 1000Add this at the pod spec level (not the container level) and Kubernetes will set the group ownership of mounted volumes to 1000.
Log duplication after pod restart
If you are seeing duplicate log entries after a restart, the Fluent Bit database file is not persisting. The emptyDir volume for fluentbit-db is ephemeral. If the Fluent Bit container restarts (not the whole pod), the database survives. If the pod is rescheduled, it is gone.
For persistent position tracking across pod rescheduling, use a PersistentVolumeClaim for the database volume instead of emptyDir. Whether this matters depends on how tolerable duplicate logs are for your use case.
Adding a second output: CloudWatch Logs
One of the advantages of Fluent Bit over simpler solutions is that adding a second output is just adding a config block. To also ship to AWS CloudWatch:
[OUTPUT]
Name cloudwatch_logs
Match app.*
region us-east-1
log_group_name /kubernetes/my-api
log_stream_prefix pod/
auto_create_group trueFluent Bit will now send every log record to both Elasticsearch and CloudWatch simultaneously. The application container has no knowledge of either destination.
---
The sidecar pattern is more resource-intensive than a DaemonSet but gives you significantly more control over per-application log routing. In environments where different applications have different compliance requirements around log retention or destination, that control is worth the overhead.
