Send Kubernetes Cluster logs to Axiom
You can ingest logs from your Kubernetes cluster into Axiom using filebeat.
The following is an example of a DaemonSet configuration to ingest your data logs into Axiom.
Configuration
apiVersion: v1
kind: ServiceAccount
metadata:
name: filebeat
namespace: kube-system
labels:
k8s-app: filebeat
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: filebeat
labels:
k8s-app: filebeat
rules:
- apiGroups: [''] # "" indicates the core API group
resources:
- namespaces
- pods
- nodes
verbs:
- get
- watch
- list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: filebeat
subjects:
- kind: ServiceAccount
name: filebeat
namespace: kube-system
roleRef:
kind: ClusterRole
name: filebeat
apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
data:
filebeat.yml: |-
filebeat.autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
hints.enabled: true
hints.default_config:
type: container
paths:
- /var/log/containers/*${data.kubernetes.container.id}.log
processors:
- add_cloud_metadata:
output.elasticsearch:
hosts: ['${AXIOM_HOST}/api/v1/datasets/${AXIOM_DATASET_NAME}/elastic']
api_key: 'axiom:${AXIOM_API_TOKEN}'
setup.ilm.enabled: false
kind: ConfigMap
metadata:
annotations: {}
labels:
k8s-app: filebeat
name: filebeat-config
namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
k8s-app: filebeat
name: filebeat
namespace: kube-system
spec:
selector:
matchLabels:
k8s-app: filebeat
template:
metadata:
annotations: {}
labels:
k8s-app: filebeat
spec:
containers:
- args:
- -c
- /etc/filebeat.yml
- -e
env:
- name: AXIOM_HOST
value: http://axiom:80
- name: AXIOM_DATASET_NAME
value: dataset
- name: AXIOM_API_TOKEN
value: xaat-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
- name: NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
image: docker.elastic.co/beats/filebeat-oss:7.16.0
imagePullPolicy: IfNotPresent
name: filebeat
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
securityContext:
runAsUser: 0
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/filebeat.yml
name: config
readOnly: true
subPath: filebeat.yml
- mountPath: /usr/share/filebeat/data
name: data
- mountPath: /var/lib/docker/containers
name: varlibdockercontainers
readOnly: true
- mountPath: /var/log
name: varlog
readOnly: true
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: filebeat
serviceAccountName: filebeat
terminationGracePeriodSeconds: 30
volumes:
- configMap:
defaultMode: 416
name: filebeat-config
name: config
- hostPath:
path: /var/lib/docker/containers
type: ''
name: varlibdockercontainers
- hostPath:
path: /var/log
type: ''
name: varlog
- hostPath:
path: /var/lib/filebeat-data
type: ''
name: data
updateStrategy:
rollingUpdate:
maxUnavailable: 1
type: RollingUpdate
Configure env
In the above configuration,
Configure your environment variables
env:
- name: AXIOM_HOST
value: http://axiom:80
- name: AXIOM_DATASET_NAME
value: dataset
- name: AXIOM_API_TOKEN
value: xaat-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Where:
-
AXIOM_HOST is your Axiom deployment URL.
-
Insert the URL and port in the AXIOM_HOST value to match your configuration. For example,
https://api.axiom.co:443
wherehttps://api.axiom.co
is the URL for your Axiom installation. It is important to specify the port number (443
in the URL for Axiom App or self-host with HTTPS, and port80
for self-host with plain HTTP). The usage of port9200
is not supported for Axiom Kubernetes transport. -
AXIOM_DATASET_NAME is your dataset name.
-
AXIOM_API_TOKEN is your API token.
The API token can be retrieved from the settings > API Token page of your Axiom deployment.
- After editing your values, apply the changes to your cluster using
kubectl apply -f daemonset.yaml