Send logs from Elastic Bulk API

Axiom is a log management platform that offers an Elasticsearch Bulk API emulation to facilitate migration from Elasticsearch or integration with tools that support the Elasticsearch Bulk API.

Using the Elastic Bulk API and Axiom in your application provides a robust way to store and manage logs.

Send logs to Axiom using the Elasticsearch Bulk API and Go

To send logs to Axiom using the Elasticsearch Bulk API and Go, use the net/http package to create and send the HTTP request.

Prepare your data

The data needs to be formatted as per the Bulk API's requirements. Here's a simple example of how to prepare your data:

data := 
{"index": {"_index": "myindex", "_id": "1"}}
{"timestamp": "2023-06-06T12:00:00Z", "message": "axiom elastic bulk", "severity": "INFO"}
{"index": {"_index": "myindex", "_id": "2"}}
{"timestamp": "2023-06-06T12:00:01Z", "message": "axiom elastic bulk api", "severity": "ERROR"}

Send data to Axiom

Obtain an Axiom API token for the Authorization header, and dataset.

package main

import (
	"bytes"
	"log"
	"net/http"
)

func main() {
	data := []byte(`
	{"index": {"_index": "myindex", "_id": "1"}}
	{"timestamp": "2023-06-06T12:00:00Z", "message": "axiom elastic bulk", "severity": "INFO"}
	{"index": {"_index": "myindex", "_id": "2"}}
	{"timestamp": "2023-06-06T12:00:01Z", "message": "axiom elastic bulk api", "severity": "ERROR"}
	`)

	// Create a new request using http
	req, err := http.NewRequest("POST", "https://api.axiom.co:443/v1/datasets/$DATASET/elastic", bytes.NewBuffer(data))
	if err != nil {
		log.Fatalf("Error creating request: %v", err)
	}

	// add authorization header to the request
	req.Header.Add("Authorization", "Bearer $API_TOKEN")
	req.Header.Add("Content-Type", "application/x-ndjson")

	// send req using http Client
	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		log.Fatalf("Error on response: %v", err)
	}

	defer func() {
		if err := resp.Body.Close(); err != nil {
			log.Fatalf("Error closing response body: %v", err)
		}
	}()
}

Send logs to Axiom using the Elasticsearch Bulk API and Python

To send logs to Axiom using the Elasticsearch Bulk API and Python, use the built-in requests library.

Prepare your data

The data sent needs to be formatted as per the Bulk API's requirements. Here's a simple example of how to prepare the data:

data = """
{"index": {"_index": "myindex", "_id": "1"}}
{"timestamp": "2023-06-06T12:00:00Z", "message": "Log message 1", "severity": "INFO"}
{"index": {"_index": "myindex", "_id": "2"}}
{"timestamp": "2023-06-06T12:00:01Z", "message": "Log message 2", "severity": "ERROR"}
"""

Send data to Axiom

Obtain an Axiom API token for the Authorization header, and dataset.

import requests
import json

data = """
{"index": {"_index": "myindex", "_id": "1"}}
{"timestamp": "2023-06-06T12:00:00Z", "message": "Log message 1", "severity": "INFO"}
{"index": {"_index": "myindex", "_id": "2"}}
{"timestamp": "2023-06-06T12:00:01Z", "message": "Log message 2", "severity": "ERROR"}
"""

try:
    response = requests.post(
        'https://api.axiom.co:443/v1/datasets/$DATASET/elastic', 
        data=data, 
        headers={
            'Content-Type': 'application/x-ndjson', 
            'Authorization': 'Bearer $API_TOKEN'
        }
    )
    response.raise_for_status()  # If the response was successful, no Exception will be raised
except requests.HTTPError as http_err:
    print(f'HTTP error occurred: {http_err}')  # Python 3.6
except Exception as err:
    print(f'Other error occurred: {err}')  # Python 3.6
else:
    print('Success!')

    try:
        print(response.json())  # tries to get the response in JSON format
    except json.JSONDecodeError:
        print(response.text)  # if response is not in JSON format, print as plain text

Send logs to Axiom using the Elasticsearch Bulk API and JavaScript

Use the axios library in JavaScript to send logs to Axiom using the Elasticsearch Bulk API.

Prepare your data

The data sent needs to be formatted as per the Bulk API's requirements. Here's a simple example of how to prepare the data:

let data = `
{"index": {"_index": "myindex", "_id": "1"}}
{"timestamp": "2023-06-06T12:00:00Z", "message": "Log message 1", "severity": "INFO"}
{"index": {"_index": "myindex", "_id": "2"}}
{"timestamp": "2023-06-06T12:00:01Z", "message": "Log message 2", "severity": "ERROR"}
`;

Send data to Axiom

Obtain an Axiom API token for the Authorization header, and dataset.

const axios = require('axios');

// Axiom elastic API URL
const AxiomApiUrl = 'https://api.axiom.co:443/v1/datasets/$DATASET/elastic';

// Your Axiom API token
const AxiomToken = '$API_TOKEN';

// The logs data retrieved from Elasticsearch
// Note: Replace this with your actual code to retrieve logs from Elasticsearch
const logs = [
    {"timestamp": "2023-06-06T12:00:00Z", "message": "axiom logging", "severity": "INFO"},
    {"timestamp": "2023-06-06T12:00:01Z", "message": "axiom log data", "severity": "ERROR"}
];

const events = logs.map((log) => ({
    timestamp: log.timestamp,
    attributes: {
      timestamp: log.timestamp,
      message: log.message,
      severity: log.severity,
    },
  }));
  

// Create the payload for Axiom
const payload = {
    tags: {
        source: 'myapplication',
        host: 'myhost'
    },
    events
};

axios.post(AxiomApiUrl, payload, {
    headers: { 
        'Content-Type': 'application/x-ndjson',
        'Authorization': `Bearer ${AxiomToken}`
    }
})
.then((response) => {
    console.log(response.status, response.statusText);
})
.catch((error) => {
    console.error(error);
});

Send logs to Axiom using the Elasticsearch Bulk API and PHP

To send logs from PHP to Axiom using the Elasticseach Bulk API, make sure you have installed the necessary PHP libraries: Guzzle for making HTTP requests and JsonMachine for handling newline-delimited JSON data.

Prepare your data

The data sent needs to be formatted as per the Bulk API's requirements. Here's a simple example of how to prepare the data:

$data = <<<EOD
{"index": {"_index": "myindex", "_id": "1"}}
{"timestamp": "2023-06-06T12:00:00Z", "message": "Log message 1", "severity": "INFO"}
{"index": {"_index": "myindex", "_id": "2"}}
{"timestamp": "2023-06-06T12:00:01Z", "message": "Log message 2", "severity": "ERROR"}
EOD;

Send data to Axiom

<?php
require 'vendor/autoload.php';

use GuzzleHttp\Client;

$client = new Client([
    'base_uri' => 'https://api.axiom.co:443/v1/datasets/$DATASET/elastic',  // Update with your Axiom host
    'timeout'  => 2.0,
]);

// Your Axiom API token
$AxiomToken = '$API_TOKEN';

// The logs data retrieved from Elasticsearch
// Note: Replace this with your actual code to retrieve logs from Elasticsearch
$logs = [
    ["timestamp" => "2023-06-06T12:00:00Z", "message" => "axiom logger", "severity" => "INFO"],
    ["timestamp" => "2023-06-06T12:00:01Z", "message" => "axiom logging elasticsearch", "severity" => "ERROR"]
];

$events = array_map(function ($log) {
    return [
        'timestamp' => $log['timestamp'],
        'attributes' => $log
    ];
}, $logs);

// Create the payload for Axiom
$payload = [
    'tags' => [
        'source' => 'myapplication',
        'host' => 'myhost'
    ],
    'events' => $events
];

try {
    $response = $client->post('', [
        'headers' => [
            'Authorization' => 'Bearer ' . $AxiomToken,
            'Content-Type' => 'application/x-ndjson',
        ],
        'json' => $payload,
    ]);
    // handle response here
    $statusCode = $response->getStatusCode();
    $content = $response->getBody();
    echo "Status code: $statusCode \nContent: $content";
} catch (\Exception $e) {
    // handle exception here
    echo "Error: " . $e->getMessage();
}

Was this page helpful?

Unable to retrieve cloud status

Copyright © 2023 Axiom, Inc. All rights reserved.