Fluent Bit

Last updated 5 days ago

Send logs to Timber via Fluent Bit

Fluent Bit is a general purpose log forwarder that can forward logs from multiple sources to Timber. You can use Fluent Bit to send logs to your Timber account.

Installation

  1. Install Fluent Bit for your platform. Such as Debian, Ubuntu, CentOS, or building directly.

  2. Install your desired input plugins, such as file tailing, STDIN, Syslog, or TCP.

  3. In /etc/td-agent-bit/td-agent-bit.conf, add the following. Replace TIMBER_API_KEY, TIMBER_SOURCE_ID, and HOSTNAMEaccordingly. Note: Fluent Bit will resolve these values as environment variables if they are set.

    /etc/td-agent-bit/td-agent-bit.conf
    [SERVICE]
    # Reduce the flush interval for better real-time access
    Flush 2
    [OUTPUT]
    Name http
    # Will match all inputs, replace with your match if you want to send a subset
    Match *
    tls On
    Host logs.timber.io
    Port 443
    URI /sources/${TIMBER_SOURCE_ID}/frames
    Header Authorization Bearer ${TIMBER_API_KEY}
    Header Content-Type application/msgpack
    Format msgpack
    Retry_Limit 5
    [FILTER]
    Name record_modifier
    # Will match all inputs, replace with your match if you want to send a subset
    Match *
    Record hostname ${HOSTNAME}
  4. Optionally install any additionally parser and filter plugins to enhance your logs.

  5. Restart your fluent-bit agent:

    sudo service td-agent-bit restart

Configuration

Please see the Fluent Bit configuration documentation. Specifically the parsing and filtering sections.

Automatic Context

Unless you're operating Fluent Bit in a Kubernetes cluster, context is not automatically added to your logs by default. This is why we include the [FILTER] plugin as part of the installation instructions. If possible, we recommend expanding this with any other information you might find helpful.

FAQs

Why do recommend Fluent Bit instead of your own agent?

Fluent Bit is a battle tested, performant, log forwarding utility written in C. It is actively maintained and always improving. It allows Timber to focus on the user experience and not the low level plumbing.

Should I use Fluent Bit or FluentD?

You can read more about both utilities in the FluentD docs. In general, we recommend Fluent Bit when possible due to its improved performance. FluentD should be used in situations where Fluent Bit is not sufficient. For example, if you want to collect data from a source that Fluent Bit does not support, or you want to use a FluentD plugin that is not available in Fluent Bit.

Troubleshooting

To begin, please see our log delivery troubleshooting guide. This covers the most common issues we see with log delivery:

If the above troubleshooting guide does not resolve your issue then we recommend enabling logging for the FluentBit service itself. You can do so by adding the following section to your FluentBit configuration file, typically located at:

[SERVICE]
Log_File /var/log/fluentbit.log
Log_Level DEBUG

More information on these options can be found in the FluentBit configuration documentation.