Compare prompts and models with Prompt tests
Run the same scenarios against selected models and inspect the evidence behind each answer.
In this article
Before you begin
Open AI agents and select the agent you want to edit. Check its name and workspace before saving: other agents have their own settings.
Step by step
- 1
Review the prompt under test
Open Review prompt and check the exact assistant instructions. Editing a test draft is different from saving the production agent.
- 2
Choose answer models
Select one to four offered models. They receive the same prompt and scenarios; the saved agent model evaluates the answers afterwards.
- 3
Write useful scenarios
Enter the customer message and expected behavior. Include a source-backed answer, missing knowledge, a handoff request and a misleading assumption.
- 4
Choose live catalog data deliberately
If enabled, the test uses read-only product/category lookups. Inspect the displayed calls and returned store values.
- 5
Read the results
Compare the answer, evaluation, latency, usage and source trace. Inspect exact retrieved chunks instead of assuming every cited source was relevant.
- 6
Apply only reviewed changes
If the test reveals a better prompt or model, update the agent deliberately and rerun your regression scenarios.
Create a small but meaningful test matrix
Write expected behavior before running a test. “Be helpful” is difficult to evaluate; “explain the policy’s custom-product exception and do not promise a refund” gives you a concrete check.
Use the same prompt, scenarios and source state for each selected model. Changing the prompt and model together makes it harder to identify what improved the result.
- A question with an exact answer in a Ready source.
- The same question phrased differently.
- An ambiguous question needing one clarifying detail.
- A missing fact that must not be invented.
- A request for staff support.
- A policy or account request that requires a boundary check.
Read evidence alongside model evaluations
Review the answer, evaluator findings, source chunks, live catalog calls, latency and usage. A high evaluation score is a useful signal, but an evaluator can share the same mistaken assumption as the answer model.
If one result is wrong, inspect the exact fact and retrieved evidence. Fix the source when the fact is absent or contradictory. Fix instructions when the evidence exists but the behavior is wrong. A model upgrade cannot reliably repair a missing policy.
Test answers before a visitor depends on them.
The Chat tab gives you a direct conversation with the bot. Prompt tests provide a repeatable place to evaluate known questions after changes to the prompt, model, source or tools.
Useful failure test: ask about a policy that is intentionally absent. A good assistant should be transparent and offer a next step, not confidently fill the gap.
- Create a small test set: Include common questions, wording variants, questions that should escalate, and questions where the assistant must say it does not know.
- Check evidence, not just phrasing: Confirm that every policy detail originates from an up-to-date source and that cited knowledge is relevant.
- Test the handoff boundary: Ask for a human, use escalation keywords and try an after-hours scenario to ensure the visitor sees the right next action.
- Re-test after every material change: Model changes, new sources and instruction edits can all change output. Keep the same core questions for comparison.
Good to know
Model evaluations are useful signals, not independent proof that an answer is correct. Compare against your approved facts. Tests can consume model usage.
Check the result
Save, start a fresh test conversation and compare the answer with your approved business facts. Settings do not rewrite messages already sent.
Questions & troubleshooting
The test and live conversation differ
Compare saved prompt, model, sources, tool permissions, conversation history and workflow context.
Rate this guide on this device. Ratings are not sent to Corthex.