Send data from Kubernetes Cluster to Axiom This step-by-step guide helps you ingest logs from your Kubernetes cluster into Axiom using the DaemonSet configuration.
Axiom makes it easy to collect, analyze, and monitor logs from your Kubernetes clusters. Integrate popular tools like Filebeat, Vector, or Fluent Bit with Axiom to send your cluster logs.
Ingest logs from your Kubernetes cluster into Axiom using Filebeat.
The following is an example of a DaemonSet configuration to ingest your data logs into Axiom.
apiVersion : v1 kind : ServiceAccount metadata : name : filebeat namespace : kube-system labels : k8s-app : filebeat --- apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRole metadata : name : filebeat labels : k8s-app : filebeat rules : - apiGroups : [ '' ] # "" indicates the core API group resources : - namespaces - pods - nodes verbs : - get - watch - list --- apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRoleBinding metadata : name : filebeat subjects : - kind : ServiceAccount name : filebeat namespace : kube-system roleRef : kind : ClusterRole name : filebeat apiGroup : rbac.authorization.k8s.io --- apiVersion : v1 data : filebeat.yml : |- filebeat.autodiscover: providers: - type: kubernetes node: ${NODE_NAME} hints.enabled: true hints.default_config: type: container paths: - /var/log/containers/*${data.kubernetes.container.id}.log allow_older_versions: true processors: - add_cloud_metadata: output.elasticsearch: hosts: ['${AXIOM_HOST}/v1/ingest/${AXIOM_DATASET}/elastic'] api_key: 'axiom:${AXIOM_TOKEN}' setup.ilm.enabled: false kind : ConfigMap metadata : annotations : {} labels : k8s-app : filebeat name : filebeat-config namespace : kube-system --- apiVersion : apps/v1 kind : DaemonSet metadata : labels : k8s-app : filebeat name : filebeat namespace : kube-system spec : selector : matchLabels : k8s-app : filebeat template : metadata : annotations : {} labels : k8s-app : filebeat spec : containers : - args : - -c - /etc/filebeat.yml - -e env : - name : AXIOM_HOST value : AXIOM_DOMAIN - name : AXIOM_DATASET value : DATASET_NAME - name : AXIOM_TOKEN value : API_TOKEN - name : NODE_NAME valueFrom : fieldRef : apiVersion : v1 fieldPath : spec.nodeName image : docker.elastic.co/beats/filebeat-oss:8.11.1 imagePullPolicy : IfNotPresent name : filebeat resources : limits : memory : 200Mi requests : cpu : 100m memory : 100Mi securityContext : runAsUser : 0 terminationMessagePath : /dev/termination-log terminationMessagePolicy : File volumeMounts : - mountPath : /etc/filebeat.yml name : config readOnly : true subPath : filebeat.yml - mountPath : /usr/share/filebeat/data name : data - mountPath : /var/lib/docker/containers name : varlibdockercontainers readOnly : true - mountPath : /var/log name : varlog readOnly : true dnsPolicy : ClusterFirst restartPolicy : Always schedulerName : default-scheduler securityContext : {} serviceAccount : filebeat serviceAccountName : filebeat terminationGracePeriodSeconds : 30 volumes : - configMap : defaultMode : 416 name : filebeat-config name : config - hostPath : path : /var/lib/docker/containers type : '' name : varlibdockercontainers - hostPath : path : /var/log type : '' name : varlog - hostPath : path : /var/lib/filebeat-data type : '' name : data updateStrategy : rollingUpdate : maxUnavailable : 1 type : RollingUpdate Replace AXIOM_DOMAIN with the base domain of your edge deployment. For more information, see Edge deployments .
Replace API_TOKEN with the Axiom API token you have generated. For added security, store the API token in an environment variable.
Replace DATASET_NAME with the name of the Axiom dataset where you send your data.
After editing your values, apply the changes to your cluster using kubectl apply -f daemonset.yaml
Collect logs from your Kubernetes cluster and send them directly to Axiom using the Vector daemonset.
apiVersion : v1 kind : ServiceAccount metadata : name : vector namespace : kube-system --- apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRole metadata : name : vector rules : - apiGroups : [ "" ] resources : - pods - nodes - namespaces verbs : - get - list - watch --- apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRoleBinding metadata : name : vector subjects : - kind : ServiceAccount name : vector namespace : kube-system roleRef : kind : ClusterRole name : vector apiGroup : rbac.authorization.k8s.io --- apiVersion : v1 kind : ConfigMap metadata : name : vector-config namespace : kube-system data : vector.yml : |- sources: kubernetes_logs: type: kubernetes_logs self_node_name: ${VECTOR_SELF_NODE_NAME} sinks: axiom: type: axiom inputs: - kubernetes_logs compression: zstd dataset: ${AXIOM_DATASET} token: ${AXIOM_TOKEN} healthcheck: enabled: true log_level: debug logging: level: debug log_level: debug --- apiVersion : apps/v1 kind : DaemonSet metadata : name : vector namespace : kube-system spec : selector : matchLabels : name : vector template : metadata : labels : name : vector spec : serviceAccountName : vector containers : - name : vector image : timberio/vector:0.37.0-debian args : - --config-dir - /etc/vector/ env : - name : AXIOM_HOST value : AXIOM_DOMAIN - name : AXIOM_DATASET value : DATASET_NAME - name : AXIOM_TOKEN value : API_TOKEN - name : VECTOR_SELF_NODE_NAME valueFrom : fieldRef : fieldPath : spec.nodeName volumeMounts : - name : config mountPath : /etc/vector/vector.yml subPath : vector-config.yml - name : data-dir mountPath : /var/lib/vector - name : var-log mountPath : /var/log readOnly : true - name : var-lib mountPath : /var/lib readOnly : true resources : limits : memory : 500Mi requests : cpu : 200m memory : 100Mi securityContext : runAsUser : 0 terminationMessagePath : /dev/termination-log terminationMessagePolicy : File volumes : - name : config configMap : name : vector-config items : - key : vector.yml path : vector-config.yml - name : data-dir hostPath : path : /var/lib/vector type : DirectoryOrCreate - name : var-log hostPath : path : /var/log - name : var-lib hostPath : path : /var/lib dnsPolicy : ClusterFirst restartPolicy : Always schedulerName : default-scheduler securityContext : {} terminationGracePeriodSeconds : 30 updateStrategy : rollingUpdate : maxUnavailable : 1 type : RollingUpdate Replace AXIOM_DOMAIN with the base domain of your edge deployment. For more information, see Edge deployments .
Replace API_TOKEN with the Axiom API token you have generated. For added security, store the API token in an environment variable.
Replace DATASET_NAME with the name of the Axiom dataset where you send your data.
After editing your values, apply the changes to your cluster using kubectl apply -f daemonset.yaml
Collect logs from your Kubernetes cluster and send them directly to Axiom using Fluent Bit.
apiVersion : v1 kind : ServiceAccount metadata : name : fluent-bit namespace : kube-system --- apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRole metadata : name : fluent-bit rules : - apiGroups : [ "" ] resources : - pods - nodes - namespaces verbs : - get - list - watch --- apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRoleBinding metadata : name : fluent-bit subjects : - kind : ServiceAccount name : fluent-bit namespace : kube-system roleRef : kind : ClusterRole name : fluent-bit apiGroup : rbac.authorization.k8s.io --- apiVersion : v1 kind : ConfigMap metadata : name : fluent-bit-config namespace : kube-system data : fluent-bit.conf : |- [SERVICE] Flush 1 Log_Level debug Daemon off Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020 [INPUT] Name tail Tag kube.* Path /var/log/containers/*.log Parser docker DB /var/log/flb_kube.db Mem_Buf_Limit 7MB Skip_Long_Lines On Refresh_Interval 10 [FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc:443 Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token Kube_Tag_Prefix kube.var.log.containers. Merge_Log On Merge_Log_Key log_processed K8S-Logging.Parser On K8S-Logging.Exclude Off [OUTPUT] Name http Match * Host ${AXIOM_HOST} Port 443 URI /v1/ingest/${AXIOM_DATASET} Header Authorization Bearer ${AXIOM_TOKEN} Format json Json_date_key time Json_date_format iso8601 Retry_Limit False Compress gzip tls On tls.verify Off parsers.conf : |- [PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On --- apiVersion : apps/v1 kind : DaemonSet metadata : name : fluent-bit namespace : kube-system spec : selector : matchLabels : name : fluent-bit template : metadata : labels : name : fluent-bit spec : serviceAccountName : fluent-bit containers : - name : fluent-bit image : fluent/fluent-bit:1.9.9 env : - name : AXIOM_HOST value : AXIOM_DOMAIN - name : AXIOM_DATASET value : DATASET_NAME - name : AXIOM_TOKEN value : API_TOKEN volumeMounts : - name : config mountPath : /fluent-bit/etc/fluent-bit.conf subPath : fluent-bit.conf - name : config mountPath : /fluent-bit/etc/parsers.conf subPath : parsers.conf - name : varlog mountPath : /var/log - name : varlibdockercontainers mountPath : /var/lib/docker/containers readOnly : true volumes : - name : config configMap : name : fluent-bit-config - name : varlog hostPath : path : /var/log - name : varlibdockercontainers hostPath : path : /var/lib/docker/containers terminationGracePeriodSeconds : 10 Replace AXIOM_DOMAIN with the base domain of your edge deployment. For more information, see Edge deployments .
Replace API_TOKEN with the Axiom API token you have generated. For added security, store the API token in an environment variable.
Replace DATASET_NAME with the name of the Axiom dataset where you send your data.
After editing your values, apply the changes to your cluster using kubectl apply -f daemonset.yaml
To extract top-level fields from the log body, configure the filelog receiver with operators to parse container log formats and JSON-structured log bodies.
Configure the following settings in your OpenTelemetry Collector configuration:
Setting Description Receiver Use filelog receiver targeting /var/log/pods/*/*/*.log Container operator Add the container operator to parse container log formats (Docker, CRI-O, Containerd) and extract the inner log message as the body JSON parser operator Add the json_parser operator to parse JSON-structured log bodies into structured fields Transform processor Optional: Use the transform processor to promote parsed JSON fields to top-level attributes instead of keeping them in the body
Example configuration:
receivers : filelog/k8s-logs : include : [ /var/log/pods/*/*/*.log ] start_at : end operators : - type : container - type : json_parser if : body matches '^{.*}$' For complete configuration details, including optional timestamp and severity extraction, see the OpenTelemetry Collector documentation .