Generate and run API tests

An API test suite is a set of cases that call the API's operations through one of its connections and report what passed and what failed. Suites live in the Tests tab of the API. Faustine runs the cases, and every call is a real call to your API.

Before you begin

The API has a connection that is not marked as production. Tests never run against a production connection. See Connections.
Use a staging or development deployment. A suite calls write operations, including the ones that require confirmation.

Generate a suite

To create a suite:

In the portal, go to APIs and select your API.
Select the Tests tab, then New test.
Select how to create the suite:
- Full coverage (recommended) generates one chain per resource and one test for each remaining operation. It takes a few minutes.
- Manual selection lets you choose the resources and operations to cover. Enter a Suite name, select the operations, then select the Generate button, which shows how many tests it creates.
- Custom, with the AI editor opens the Advanced editor, where you describe what to test.

The suite appears in the list while its cases are generated, and can be run once generation finishes.

What a suite leaves behind

Cases are ordered as the life of each resource: create it, read it, update it, delete it, then check that it is gone. Run in that order, a chain removes what it created.

When an operation creates a record and the API has no operation that undoes it, the portal says so before generating. Generate anyway creates the suite, and each run leaves records named Faustine Test in the connection. Such a suite is marked leaves state in the list.

In Manual selection, Also test the cleanup ops adds the operations that clean up after your selection, so the chain is complete.

Run a suite

To run a suite:

In the Tests tab, select the suite.
In Connection, select the connection to run against. Production connections are listed and cannot be selected.
In Run against, select Draft (current) or a published version. The draft is what you are editing. A version is what a gateway environment serves.
If the API signs in with Pass-through, enter a token in Upstream Authorization. It is sent on every call of the run and not stored.
Select Run suite, then confirm.

The run opens with a console that shows the current case. When it finishes, the header shows how many cases passed, failed and were skipped. Failures and skipped cases are listed first. Select a case to see its calls with their input and output, and its logs.

To stop a run in progress, select Cancel.

Fix failures

Fix in AI editor on a run with failures opens the Advanced editor with that run as context. The editor groups the failures by cause and changes the test case or the operation, whichever is at fault.

Regenerate on a suite writes every case again from the current draft with the same scope. Edits made to the cases by hand are lost. Past runs are kept.

Troubleshooting

Symptom	Cause	Fix
No connection can be selected	Every connection of the API is marked as production.	Add a staging or sandbox connection.
`Connection "…" is marked as production: tests never run against it. Pick a staging or sandbox connection.`	The run named a connection marked as production.	Select another connection.
Generation failed	The generation stopped with an error, shown under the heading.	Select Retry generation.
The run reads `The gateway restarted while this run was in progress — run the suite again.`	The run was interrupted.	Run the suite again.

Next steps

The AI editor: edit cases as files and run suites from the conversation.
Operations: fix an operation's schema or kind when a failure points at the contract.
Versions and updates: publish the draft once the suite passes.