Elixir

Last updated 6 days ago

Send Elixir logs to Timber

Timber integrates with Elixir through the :timber Hex package, enabling you to send Elixir logs to your Timber account.

Features

Installation

If you're unsure, we recommend installing via the "HTTP" method. To understand why you would choose one over the other, please see the "Sending Logs To Timber" guide.

HTTP
STDOUT

Send logs directly to Timber from within your app over HTTP:

  1. In mix.exs, add the :timber dependency:

    mix.exs
    defp deps do
    [
    {:timber, "~> 3.0"}
    ]
    end
  2. In config.exs, install the Timber logger backend, replace YOUR_API_KEY and YOUR_SOURCE_ID accordingly:

    config/config.ex
    config :logger,
    backends: [Timber.LoggerBackends.HTTP],
    config :timber,
    api_key: "YOUR_API_KEY",
    source_id: "YOUR_SOURCE_ID"
  3. Optionally install integrations for Plug, Phoenix, Ecto, and more!

  4. Optionally test the pipes to verify installation:

    mix timber.test_the_pipes

Write logs to STDOUT and ship them to Timber external from your app:

This method is more advanced and requires a separate step to ship logs to Timber. Basic knowledge of STDOUT and log management is required. If you are unsure, please use the "HTTP" method. For more information on the advantages of this method please see this guide.

  1. In your mix.exs file add the :timber dependency:

    mix.exs
    defp deps do
    [
    {:timber, "~> 3.0"}
    ]
    end
  2. In your config.exs file install the Timber console JSON formatter:

    config/config.exs
    config :logger,
    backends: [:console],
    utc_log: true
    config :logger, :console,
    format: {Timber.Formatter, :format},
    metadata: :all
  3. Optionally install integrations for Plug, Phoenix, Ecto, and more!

  4. At this point your application is writing logs to STDOUT in JSON format. Please choose the appropriate platform, log forwarder, or operating system.

Configuration

Timber Configuration

All configuration options for :timber can be found in the Timber.Config documentation.

Elixir Logger Configuration

All configuration options for the Elixir Logger can be found in the Logger documentation.

Usage

Basic Logging

Timber works with your existing Logger calls, no changes necessary:

Logger.info("Hello world")

Structured Logging

If you haven't already, please see our structured logging best practices guide.

Your Logger calls now take an :event metadata key where you can pass structured data. We recommend logging data with a root level namespace as shown below to avoid type conflicts with other events. You can read more on this in our Event Naming guide.

Logger.info(fn ->
event = %{order_placed: %{order_id: "aarg23dw", total: 100.23}}
message = "Order aarg23dw placed, total: $100.23"
{message, event: event}
end)

Adding Context

Before capturing context, please make sure Timber does not already capture the context you want (see the Automatic Context section).

Local Context (default)

Because Elixir is a highly concurrent language, Timber context, by default, is local to the Elixir process. This makes context management easy since it dies with the process. This fits well with Erlang / Elixir concurrency patterns since processes are typically ephemeral. Adding context in this fashion is simple:

Timber.add_context(user: %{id: "d23f6h7ffx", email: "paul@bunyan.com"})

If you are spawning processes and wish to copy context to that process see the "Copying Context To Other Processes" section.

Global Context

In rare cases you'll want to set global context. Global context applies to all processes and should be used with care. Setting global context follows the same pattern as local context except it must be explicitly specified as the second argument:

Timber.add_context(user: %{id: "d23f6h7ffx", email: "paul@bunyan.com"}, :global)

Deleting Context

As stated above, you should not have to remove context except in rare cases. Context should die with the local process.

Given the above :user context, context can be deleted by passing the key:

Local Context (default)

Timber.delete_context(:user)

Global Context

Timber.delete_context(:user, :global)

Guides

Tying Logs To Users

A very common use-case for context is tying logs to users, you can accomplish this by setting user context immediately after you log the user in. This is typically done in an authentication plug:

auth_plug.ex
defmodule AuthPlug
@behaviour Plug
def call(conn, _default, opts \\ []) do
# ...
Timber.add_context(user: %{id: "abcd1234"})
# ...
end
end

Be careful not to include personally identifiable information for privacy reasons.

Tying Logs To HTTP Requests

Timber offers a Plug and Phoenix integration that does this automatically.

Another common use-case for context is tying logs to HTTP requests through the request ID. You can see an example of this in the :timber_plug 's HTTPContext module.

Timing Code Execution

Timber offers utilities for timing code execution and including those timings in your logs. Timing in Erlang should not be done using wall clock time, instead, it should use monotonic time:

timer = Timber.start_timer()
# ... issue the HTTP request ...
duration_ms = Timber.duration_ms(timer)
Logger.info(
event = %{http_response_received: %{method: "GET", host: "api.payments.com", path: "/payment", duration_ms: duration_ms}}
message = "Received POST https://api.payments.com/payment in 56.7ms",
{message, event: event}
)

Copying Context To Child Processes

As described in the "Setting Context" section, Timber's context is local to each process. This ensures that each process can maintain context specific to it's state without conflicting with other processes, but there are times where you'll want to carry context over to child processes, such as when using the Task module:

current_context = Timber.LocalContext.load()
Task.async fn ->
Timber.LocalContext.save(current_context)
Logger.info("Logs from a separate process")
end

The new process spawned with Task.async/1 will now contain the same Timber context as its parent. Please note, that the runtime context's vm_pid will be overridden and remain true to the local process.

Log to an additional IO Device

Writing to an additional IO device is as simple as installing a Logger backend for your desired device.

Logging to STDOUT

If you have the means to log to STDOUT, we highly recommend that you redirect STDOUT to Timber through one of our platform, log forwarder, or operating system integrations instead of shipping logs from within your app. You can read more about that here.

Logging to :stdout uses the Elixir provided :console backend. You can read more about configuring the :consolebackend here. Simply add it as a Logger backend:

config/config.exs
config :logger, backends: [Timber.LoggerBackends.HTTP, :console]

From here you can redirect STDOUT to a file, Syslog, or any device of your choice.

Logging to a File

We highly recommend that you log to STDOUT and redirect the output to a file when starting your Elixir application. As this follows the 12 Factor methodolgy.

If you cannot redirect STDOUT to a file you can install the logger_file_backend library. Information on installing that can be found here.

Automatic Context

:timber automatically captures context to enrich your logs.

system

The system context captures system level information such as hostname and pid:

{
"context": {
"system": {
"hostname": "ec2-44-125-241-8",
"pid": 20643
}
}
}

Field

Type

Description

context.system.hostname

string

System level hostname, value of :inet.gethostname()

context.system.pid

int

System level process ID, value of System.get_pid()

runtime

The runtime context captures information about the originator of the log line. By default, the Elixir Logger includes runtime metadata that Timber simply takes advantage of.

{
"context": {
"runtime": {
"vm_pid": "<0.9960.261>",
"module_name": "MyModule",
"line": 371,
"function": "my_func/2",
"file": "lib/my_app/my_module.ex",
"application": "my_app"
}
}
}

Field

Type

Description

context.runtime.vm_pid

string

The Elixir VM pid (not the operating system level pid)

context.runtime.module_name

string

The name of the module where the log statement originated.

context.runtime.line

int

The line number where the log statement originated.

context.runtime.file

`string

The file where the log statement originated.

context.runtime.function

The function where the log statement originated.

content.runtime.application

The Elixir component app where the log statement originated.

Integrations

Timber integrates with 3rd party libraries a la carte style, allowing you to pick and choose the integrations you want. Integrations with timber are entirely optional and serve to upgrade your logs with rich context and metadata if you choose to use them:

Performance

Extreme care was taken into the design of :timber to be fast and reliable:

  1. :timber works directly with the Elixir Logger, automatically assuming all of the stability and performance benefits this provides, such as back pressure, load shedding, and defensability around Logger failures.

  2. Log data is buffered and flushed on an interval to optimize performance and delivery.

  3. The :timber HTTP backend uses a controlled multi-buffer design to efficiently ship data to the Timber service.

  4. Connections are re-used and rotated to ensure efficient delivery of log data.

  5. Delivery failures are retried with an exponential backoff, maximizing successful delivery.

  6. Msgpack is used for payload encoding for it's superior performance and memory management.

  7. The Timber service ingest endpoint is a HA servce designed to handle extreme fluctuations of volume, it responds in under 50ms to reduce back pressure.

FAQs

Will installing Timber affect my application's performance?

No. Timber was designed with a keen focus on performance. It leans heavily on Elixir's Logger delegating asynchronous log processing to the Logger processes. Timber is also battle tested on hundreds of Elixir apps. For example, we use Timber internally at Timber in the most performance sensitive areas of our log ingestion pipeline without any issue.

Can Timber crash my app?

No, Timber is installed as an Elixir Logger backend, which are protected from failures, back pressure, and the like. You can read more about the Elixir / Erlang logger and it's design in the docs.

Troubleshooting

To begin, please see our log delivery troubleshooting guide. This covers the most common issues we see with log delivery:

If the above troubleshooting guide does not resolve your issue then we recommend enabling debug logging within the :timber library itself by adding this to your config/config.ex file:

config/config.exs
config :timber,
debug_io_device: :stdio

Because the :timber library operates within the Logger itself it cannot call Log.* statements. The above configuration routes all :timber debug statements to STDOUT , allowing you access generic output statements generated by the :timber library. Once you've done this you'll want to analyze these statements, looking specifically for statements on the error level. This should give you insight into the functioning of the :timber library and any problems it's experiencing.