# Stream logs to Amazon S3 and S3-compatible storage

An object storage log stream writes the requests of a gateway environment to a bucket, one compressed file per batch. Use it for long retention, a data lake or an import into another system. It works with Amazon S3, DigitalOcean Spaces, Cloudflare R2, Google Cloud Storage and any storage that has an S3 API, such as MinIO, Wasabi or Backblaze B2.

## Before you begin

- A bucket for the logs.
- An access key that can write objects in that bucket. It needs no other permission.

| Storage | Where the key comes from |
| --- | --- |
| Amazon S3 | An IAM user or role key with `s3:PutObject` on the bucket. See [Manage access keys for IAM users](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html). |
| DigitalOcean Spaces | **Spaces Object Storage** > **Access Keys**. A key limited to the Space is enough. See [Manage access to Spaces](https://docs.digitalocean.com/products/spaces/how-to/manage-access/). |
| Cloudflare R2 | **R2** > **Manage API tokens**, an "Object Read & Write" token scoped to the bucket. See [R2 API tokens](https://developers.cloudflare.com/r2/api/tokens/). |
| Google Cloud Storage | **Cloud Storage** > **Settings** > **Interoperability**, an HMAC key for a service account with write access. See [HMAC keys](https://cloud.google.com/storage/docs/authentication/hmackeys). |
| S3-compatible | A key with permission to write objects in the bucket. |

## Add the stream

1. In the portal, go to [**Agent Gateway**](https://auth.faustine.ai/go?to=%2Fintegrations%2Fagent-gateway) and select your gateway.
2. Select the **Logs** tab, then **Log settings**.
3. Under **Log streams**, select **Add a stream**.
4. Under **Destination**, in the **Object storage** group, select your storage.
5. Fill in the fields for your storage.
6. Select **Send test event**. Faustine writes a small test object to the bucket and shows **Delivered**.
7. Select **Continue**.
8. Under **What to send**, choose **All requests** or **Errors and denials**, and turn on **Discovery calls** or **Payloads** if you want them. Select **Continue**.
9. Under **Review and create**, enter a **Name** and select **Create stream**.

**Amazon S3**

| Field | Value |
| --- | --- |
| **Bucket** | The bucket name. |
| **Region** | The bucket's region, such as `us-east-1`. |
| **Folder** | A prefix for the objects, such as `faustine/logs`. Optional. |
| **Access key ID** | The access key ID. |
| **Secret access key** | The secret access key. |

**DigitalOcean Spaces**

| Field | Value |
| --- | --- |
| **Space** | The name of the Space. |
| **Region** | The region of the Space, such as **NYC3** or **FRA1**. |
| **Folder** | A prefix for the objects. Optional. |
| **Access key ID** | The access key ID. |
| **Secret access key** | The secret access key. |

**Cloudflare R2**

| Field | Value |
| --- | --- |
| **Bucket** | The bucket name. |
| **S3 API endpoint** | `https://<account-id>.r2.cloudflarestorage.com`. It is under **R2**, the bucket, **Settings**, **S3 API**. |
| **Folder** | A prefix for the objects. Optional. |
| **Access key ID** | The access key ID of the token. |
| **Secret access key** | The secret access key of the token. |

**Google Cloud Storage**

Faustine writes to `https://storage.googleapis.com` through the S3 interoperability API.

| Field | Value |
| --- | --- |
| **Bucket** | The bucket name. |
| **Folder** | A prefix for the objects. Optional. |
| **Access key ID** | The access ID of the HMAC key. |
| **Secret access key** | The secret of the HMAC key. |

**S3-compatible**

| Field | Value |
| --- | --- |
| **Endpoint** | The `https://` address of the S3 API, such as `https://s3.us-west-1.wasabisys.com`. |
| **Region** | The region the storage expects in signatures. Many accept `us-east-1`. |
| **Bucket** | The bucket name. |
| **Folder** | A prefix for the objects. Optional. |
| **Path-style addressing** | On by default. Addresses the bucket as `endpoint/bucket/key` and not `bucket.endpoint/key`. Most self-hosted storage needs it. |
| **Access key ID** | The access key ID. |
| **Secret access key** | The secret access key. |

The stream's page opens. The first object is written about 30 seconds after the next request ends.

## What the bucket receives

Each batch is one object of up to 500 requests. It is NDJSON compressed with gzip: one request per line, as a JSON document. The content type is `application/x-ndjson`, and the object's `faustine-requests` metadata holds the number of requests in it.

The key of an object is built from the folder, the environment's slug, the date and time in UTC, and the IDs of the first and the last request in the batch:

```text
<folder>/<environment>/YYYY/MM/DD/HHmmss-<first request id>-<last request id>.ndjson.gz
```

The test event is written as `HHmmss-test-event.ndjson.gz` in the folder of the day.

For the fields of a line, see [Log streams](https://docs.faustine.ai/agent-gateway/log-streams/). A batch can be written twice after a failure, so deduplicate by the `id` of each line.

## Troubleshooting

| Symptom | Cause | Fix |
| --- | --- | --- |
| The test shows "The storage rejected the credentials" | The key is wrong, cannot write to the bucket, or the region does not match. | Check the access key, its permission to write objects, and the region. |
| The test shows "The bucket does not exist in that region or endpoint." | The bucket name, the region or the endpoint is wrong. | Check all three against the storage's own page for the bucket. |
| The form shows "Not a bucket name" | The name has capital letters or characters a bucket name cannot have. | Use 3 to 63 lowercase letters, digits, dots and hyphens. |

## Next steps

- [Log streams](https://docs.faustine.ai/agent-gateway/log-streams/): delivery, retries and the fields of a request.
- [Redact sensitive fields](https://docs.faustine.ai/agent-gateway/redact-sensitive-fields/): decide what a stream can carry.
- [Set recording and retention](https://docs.faustine.ai/agent-gateway/retention/): how long Faustine keeps the same requests.