Archiving

Last updated 9 days ago

Archive logs for cheap long term storage

Timber offers the ability to store your log data in compressed archives on S3. This is a great way to retain your logs long term in a cost efficient manner.

You must have admin privileges to manage archiving.

Getting Started

  1. Navigate to the Settings section using the main navigation.

  2. Click the "Archives" tab.

  3. Click the "Configure" button.

  4. Select "Default Bucket". (Instructions to use your own S3 bucket are here).

  5. Click "Save Archive Location"

Timber Archiving Demo

Usage

Enabling

Use Timber's default S3 bucket for quick easy setup. Use your own S3 bucket for complete control over your archive's life cycle. More details here.

Timber's S3 Bucket
Your Own S3 Bucket
  1. Navigate to the Settings section using the main navigation.

  2. Click the "Archives" tab.

  3. Click the "Configure" button.

  4. Select the "Default" bucket type.

  5. Click "Save Archive Location"

  1. Navigate to the S3 console in the us-east-1 region and create a new bucket with the appropriate settings.

  2. Add a the Timber bucket policy to your newly created bucket, replacing my-bucket-name with the name of your new bucket.

    {
    "Version":"2012-10-17",
    "Statement":[
    {
    "Sid":"AddLogs",
    "Effect":"Allow",
    "Principal": {"AWS": ["arn:aws:iam::754402436383:root"]},
    "Action":["s3:PutObject"],
    "Resource":["arn:aws:s3:::my-bucket-name/*"]
    }
    ]
    }
  3. Navigate to the Settings section using the main navigation.

  4. Click the "Archives" tab.

  5. Click the "Configure" button.

  6. Select the "Custom" bucket type.

  7. Enter your S3 bucket location: s3://my-bucket-name

  8. Click "Save Archive Location"

  9. Timber will validate your location before saving. If your setting successfully save then your bucket is working and active.

Downloading

Timber's S3 Bucket
Your Own S3 Bucket
  1. Navigate to the Settings section using the main navigation.

  2. Click the "Archives" tab.

  3. You'll see a list of archives grouped by date.

  4. Click on the appropriate date.

  5. Click "Download Link" for the app you want to download.

Because you've elected to use your own S3 bucket you'll need to use the S3 console to browse and download your archives. For security and compliance reasons, Timber only has write access to your bucket, not read access.

  1. Open the S3 console.

  2. Navigate to the bucket you configured Timber to use.

  3. Click on the appropriate date.

  4. Download the archive for your chosen source.

Expiring

Timber's S3 Bucket
Your Own S3 Bucket

Timber automatically expires your archives after 1 year. Currently this is not configurable. If you wish to change this we recommend using your S3 own bucket, giving you complete control over your archive lifecycle management.

Because you've elected to use your own S3 bucket, archive expiration is entirely under your control through S3's lifecycle policies. Timber only has write access to your bucket, it does not have read or delete access.

  1. Open the S3 console.

  2. Click on your S3 bucket.

  3. A side panel will slide out.

  4. Click on the "Management" section.

  5. Click "Add lifecycle rule".

  6. Follow the lifecycle rule wizard. Amazon's wizard should make it easy to select the option to suit your expiration preferences.

Disabling

Disabling archives does not delete your archives, it simply stops creating them.

You can disable archiving at any time.

  1. Navigate to the Settings section using the main navigation.

  2. Click the "Archives" tab.

  3. Click the "Configure" button.

  4. Select "Disabled" for Bucket Type.

  5. Click "Save Archive Location"

Deleting

Timber's S3 Bucket
Your Own S3 Bucket

Timber does not offer the ability to delete individual archives. You must delete your entire source to delete the archives. If you want granular control over your archives we recommend using your own S3 bucket.

Because you've elected to use your own S3 bucket you have full control over deleting your archives.

  1. Open the S3 console.

  2. Navigate to the bucket you configured Timber to use.

  3. Delete archives as necessary.

Configuration

Your Own S3 Bucket

Only S3 buckets in the us-east-1 region are supported.

Permissions

In order for Timber to write archives to your S3 bucket you must grant Timber permission through a bucket policy. Replace my-bucket-name with your bucket name!

{
"Version":"2012-10-17",
"Statement":[
{
"Sid":"AddLogs",
"Effect":"Allow",
"Principal": {"AWS": ["arn:aws:iam::754402436383:root"]},
"Action":["s3:PutObject"],
"Resource":["arn:aws:s3:::my-bucket-name/*"]
}
]
}

We recommend configuring your S3 bucket with the following settings:

  1. Enable versioning - Recovery for accidental deletes.

  2. Enable encryption - Extra security.

  3. Add a lifecycle rule to automatically move the files to IFA - Infrequent access is recommended over glacier since it is quicker to access and search when you need it.

  4. Add a lifecycle rule to expire archives after 1 year, adjust as necessary - 1 year should be sufficient for most compliance policies.

How It Works

Interval

Timber's archives are daily. Once a day Timber will roll up all of your log data into an archive and place it in your configured destination. This usually happens around 3am EST but could vary depending on other factors.

Archive Format

Archive files are stored in .tar.gz (tar gzipped) format.

Archive Contents

Archive contents are full dumps of your log data in in ndjson format. Each log line includes every single field in JSON format. To learn more about fields see the Dynamic Schema Management and Reserved Fields sections.

{"dt":"2019-03-12T02:22:12.445Z","level":"info","message":"Log line 1","context":{"system":{"host":"my.host.com"}}}
{"dt":"2019-03-12T02:22:13.426Z","level":"info","message":"Log line 2","context":{"system":{"host":"my.host.com"}}}
{"dt":"2019-03-12T02:22:14.248Z","level":"info","message":"Log line 3","context":{"system":{"host":"my.host.com"}}}
{"dt":"2019-03-12T02:22:15.544Z","level":"info","message":"Log line 4","context":{"system":{"host":"my.host.com"}}}
{"dt":"2019-03-12T02:22:16.521Z","level":"info","message":"Log line 5","context":{"system":{"host":"my.host.com"}}}

Compression

Final archive files are gzipped.

FAQs

When should I use my own S3 bucket?

For convenience and ease of use Timber offers a "default" S3 bucket that we own and maintain. Archives are downloaded directly from the Timber interface. If have specific security or compliance needs then we recommend using your own S3 buckets. This puts you in complete control over durability, security, and compliance settings.

How long are my archives retained?

If you're using Timber's S3 bucket your archives are retained for 1 year. If you're using your own bucket you are entirely in control of when you archives expire. See the configuration section on configuring your own S3 bucket for more details.