Generate and run a test suite

Generate a suite from the agent's actions and run it in a page of your app. The run executes real actions with the session of the user signed in to that page, so use an environment with test data. For what a suite contains, see Tests.

Before you begin

The agent's analysis has finished and the draft has enabled actions.
A page of your app has a debug session open, on an environment that is not Production. See Open a debug session.

Generate a suite

To generate a suite that covers every enabled action:

In the portal, go to Web Agents and select your agent.
Select the Tests tab, then New test.
Select Full coverage (recommended).

Generation takes a few minutes and uses tokens of your model provider. The suite shows generating while it runs and ready when the cases are written.

To cover part of the agent instead:

Select New test, then Manual selection.
Enter a Suite name.
Select the modules and actions to cover.
Select Generate with the number of tests.

Under a module, the portal says when a selected action creates records that the suite cannot clean up, and when a cleanup uses an action that the suite does not test. Select Also test it to add that action.

Review the cases

Select the suite to open it. Cases lists each case in the order it runs, with the chain it belongs to. Select a case to read its code and its cleanup.

To change a case, select the link to the Advanced editor inside the case. To write every case again from the draft, select Regenerate. Regenerating discards the edits made to the cases and keeps the past runs.

Run the suite

Open the suite. Under Live debug session, find the page to run in. The row shows its domain, user and environment.
If the row shows a version followed by reload, reload the page of your app first. It loaded before the version its environment serves.
Select Run suite.

The run opens as soon as the page starts it and shows each case as it finishes. To stop it, select Cancel. The cleanups of the cases that ran are executed.

When the run ends, it shows how many cases passed, failed and were skipped. Cases — what each one did lists every action each case called, with its input and output. Coverage — this run shows the result for each action.

Warning

A run on a page of the Production environment executes real actions with that user's session. The portal asks Run on a Production session? before it starts.

Fix the failures

In a run with failures, select Fix in AI editor. The Advanced editor opens on that run, reads the failures and changes the case or the action. Then run the suite again.

Troubleshooting

Symptom	Cause	Fix
`No live debug sessions`	No page of your app has a debug session open.	See Open a debug session.
Run suite is not available, with `A run is already in progress for this suite.`	The suite has a run in progress.	Wait for it to finish, or open it and select Cancel.
Run suite is not available, with `The suite has no enabled case.`	Every case of the suite is disabled.	Select Regenerate, or change the cases in the Advanced editor.
`The page did not start the run in time — is it still connected?`	The page closed or lost its connection.	Reload the page of your app and run again.
The run ends as `aborted`	The page reloaded, closed or the user's session expired during the run.	Sign in again and run the suite again. Records the run created are still in your app.
Many cases are skipped with `action not exposed by the SDK on this page`	The page loaded a version that does not have those actions.	Deploy the version to the page's environment and reload the page.
`The draft has no enabled modules to test — run an analysis first.`	The agent has no enabled action.	Enable modules in the Actions tab, or run the analysis.

Next steps

Tests: lifecycle chains, cleanup and what a run records.
The Advanced editor: write custom cases and fix failures.
Read the logs: see what the agent did in real conversations.