Stream logs to Amazon S3 and S3-compatible storage

An object storage log stream writes the requests of a gateway environment to a bucket, one compressed file per batch. Use it for long retention, a data lake or an import into another system. It works with Amazon S3, DigitalOcean Spaces, Cloudflare R2, Google Cloud Storage and any storage that has an S3 API, such as MinIO, Wasabi or Backblaze B2.

Before you begin

A bucket for the logs.
An access key that can write objects in that bucket. It needs no other permission.

Storage	Where the key comes from
Amazon S3	An IAM user or role key with `s3:PutObject` on the bucket. See Manage access keys for IAM users.
DigitalOcean Spaces	Spaces Object Storage > Access Keys. A key limited to the Space is enough. See Manage access to Spaces.
Cloudflare R2	R2 > Manage API tokens, an "Object Read & Write" token scoped to the bucket. See R2 API tokens.
Google Cloud Storage	Cloud Storage > Settings > Interoperability, an HMAC key for a service account with write access. See HMAC keys.
S3-compatible	A key with permission to write objects in the bucket.

Add the stream

In the portal, go to Agent Gateway and select your gateway.
Select the Logs tab, then Log settings.
Under Log streams, select Add a stream.
Under Destination, in the Object storage group, select your storage.
Fill in the fields for your storage.
Select Send test event. Faustine writes a small test object to the bucket and shows Delivered.
Select Continue.
Under What to send, choose All requests or Errors and denials, and turn on Discovery calls or Payloads if you want them. Select Continue.
Under Review and create, enter a Name and select Create stream.

Field	Value
Bucket	The bucket name.
Region	The bucket's region, such as `us-east-1`.
Folder	A prefix for the objects, such as `faustine/logs`. Optional.
Access key ID	The access key ID.
Secret access key	The secret access key.

Field	Value
Space	The name of the Space.
Region	The region of the Space, such as NYC3 or FRA1.
Folder	A prefix for the objects. Optional.
Access key ID	The access key ID.
Secret access key	The secret access key.

Field	Value
Bucket	The bucket name.
S3 API endpoint	`https://<account-id>.r2.cloudflarestorage.com`. It is under R2, the bucket, Settings, S3 API.
Folder	A prefix for the objects. Optional.
Access key ID	The access key ID of the token.
Secret access key	The secret access key of the token.

Faustine writes to https://storage.googleapis.com through the S3 interoperability API.

Field	Value
Bucket	The bucket name.
Folder	A prefix for the objects. Optional.
Access key ID	The access ID of the HMAC key.
Secret access key	The secret of the HMAC key.

Field	Value
Endpoint	The `https://` address of the S3 API, such as `https://s3.us-west-1.wasabisys.com`.
Region	The region the storage expects in signatures. Many accept `us-east-1`.
Bucket	The bucket name.
Folder	A prefix for the objects. Optional.
Path-style addressing	On by default. Addresses the bucket as `endpoint/bucket/key` and not `bucket.endpoint/key`. Most self-hosted storage needs it.
Access key ID	The access key ID.
Secret access key	The secret access key.

The stream's page opens. The first object is written about 30 seconds after the next request ends.

What the bucket receives

Each batch is one object of up to 500 requests. It is NDJSON compressed with gzip: one request per line, as a JSON document. The content type is application/x-ndjson, and the object's faustine-requests metadata holds the number of requests in it.

The key of an object is built from the folder, the environment's slug, the date and time in UTC, and the IDs of the first and the last request in the batch:

<folder>/<environment>/YYYY/MM/DD/HHmmss-<first request id>-<last request id>.ndjson.gz

The test event is written as HHmmss-test-event.ndjson.gz in the folder of the day.

For the fields of a line, see Log streams. A batch can be written twice after a failure, so deduplicate by the id of each line.

Troubleshooting

Symptom	Cause	Fix
The test shows "The storage rejected the credentials"	The key is wrong, cannot write to the bucket, or the region does not match.	Check the access key, its permission to write objects, and the region.
The test shows "The bucket does not exist in that region or endpoint."	The bucket name, the region or the endpoint is wrong.	Check all three against the storage's own page for the bucket.
The form shows "Not a bucket name"	The name has capital letters or characters a bucket name cannot have.	Use 3 to 63 lowercase letters, digits, dots and hyphens.

Next steps

Log streams: delivery, retries and the fields of a request.
Redact sensitive fields: decide what a stream can carry.
Set recording and retention: how long Faustine keeps the same requests.