> ## Documentation Index
> Fetch the complete documentation index at: https://docs.roark.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Tool Call Testing
> Verify your agent calls the right tools with the right parameters
## Overview
Tool call testing lets you verify that your voice AI agent is invoking the correct tools, with the correct parameters, at the right moments during a conversation. Roark provides built-in metrics for this through the **Tool Invocation Analysis** package.
You can run tool call testing in two contexts:
* **Simulations**: Proactively test tool calling behavior across scenarios before deploying changes
* **Production**: Continuously monitor tool calling quality on live customer calls
Both approaches use the same metrics. You configure the metric once, then apply it wherever you need it.
***
## Prerequisites
Before testing tool calls in either context, you need to set up the tool invocation metrics you want to evaluate.
### 1. Find the Tool Invocation Analysis Package
Navigate to **Metrics > Library** and look for the **Tool Invocation Analysis** package. This package contains five built-in metrics:
| Metric | Type | What It Measures |
| :------------------------------------- | :----- | :--------------------------------------------------------------------------------- |
| **Tool Invocation Correct** | Yes/No | Whether the agent invoked the correct tools at the appropriate times |
| **Tool Invocation Count** | Count | Total number of tool calls made during the conversation |
| **Tool Invocation Order Correct** | Yes/No | Whether tools were called in the correct logical sequence |
| **Tool Invocation Parameters Correct** | Yes/No | Whether the correct parameters were passed to each tool call |
| **Tool Invocation Result Correct** | Yes/No | Whether the agent correctly interpreted and used the results returned by each tool |
### 2. Choose and Configure Your Metrics
Select the metrics relevant to your testing goals. For example, if you want to verify that your agent calls the right tool when a customer asks about the weather:
1. Click on **Tool Invocation Correct** in the library
2. Under **Tool Scoping**, select the specific tool you want to evaluate (e.g., `fetchWeather`)
3. Click **Edit** on the scoped tool to set the evaluation criteria: define when the tool should be called and what result is expected
You can leave tool scoping empty to evaluate all tools, or scope to specific tools for targeted testing. Use the **Additional Instructions** field to add cross-tool rules like "Never call any tool before greeting the customer."
***
## Testing in Simulations
Simulation testing lets you proactively validate tool calling behavior across different scenarios and personas before changes reach production.
Tool call testing in simulations requires **enriched simulations**: your agent must send its call data to Roark so that tool invocation data is available for analysis. See [Enriched Simulations](/documentation/simulation-testing/enriched-simulations) to set this up.
### Setup
Follow the [run plan guide](/documentation/simulation-testing/run-plans#creating-a-run-plan) to set up your simulation. Choose the scenarios that should trigger the tool calls you want to test.
In the **Metrics** section of the run plan, select the tool invocation metrics you configured in the prerequisites.
Make sure your agent sends its call data to Roark via an [integration](/documentation/integrations/overview) or the [API/SDK](/documentation/observability/overview#how-calls-get-into-roark). This is required for tool call data to be available. Without it, Roark only has its simulation agent's recording and cannot see your agent's tool calls.
Execute the run plan. Once your agent's call is matched and merged with the simulation call, the tool invocation metrics will be evaluated automatically.
#### From the CLI
A saved plan is addressable by id, so the same run can be started from a terminal or a CI job:
```bash CLI theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# Find the plan you created above
roark simulation plan list --limit 5
# Start a run of it
roark simulation plan job start
# Follow the job
roark simulation plan job get
```
```bash cURL theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl -X POST https://api.roark.ai/v1/simulation/plan//job \
-H "Authorization: Bearer $ROARK_API_BEARER_TOKEN"
```
The CLI exits non-zero when the API rejects a request, so a CI step that starts a run fails loudly rather than passing on a broken plan. See [exit codes](/documentation/sdks/cli#exit-codes).
***
## Testing in Production
Production testing lets you continuously monitor tool calling quality on live customer calls using metric collectors.
### Setup
Navigate to **Metrics → Collectors** and create a new collector. See the [collectors guide](/documentation/metrics/metric-collectors) for details.
In the collector, select the tool invocation metrics you configured in the prerequisites.
Use collector conditions to filter which calls should be evaluated. For example, you might only want to evaluate tool calls for a specific agent or call direction.
Once active, the collector will automatically evaluate tool calling on every matching call that comes in.
#### From the CLI
Collectors have no direct command: they are a [Config as Code](/documentation/config-as-code/collectors) resource, so the CLI applies them from a file rather than from flags. That is the better fit anyway, since a collector is something you want reviewed and version-controlled rather than typed once into a terminal.
```yaml roark/collectors/tool-calls.yaml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: collector
name: tool-call-quality
modality: call
status: ACTIVE
metrics:
- tool_invocation_correct
- tool_invocation_parameters_correct
filters:
- conditions:
- type: AGENT
key: frontdesk
operator: EQUALS
```
```bash CLI theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# Preview what would change, without writing anything
roark config diff ./roark
# Apply it
roark config apply ./roark
```
Your calls must include [tool invocation data](/documentation/tool-invocations) for these metrics to work. Tool calls are automatically captured for **Vapi** and **Retell** integrations. If you use the API directly or LiveKit, include tool invocations when you create the call, or [attach them afterward](/documentation/tool-invocations#attaching-tool-calls-after-a-call) if your tool data becomes available later.
***
## Next Steps
Learn how to submit tool call data with your calls
Enable call matching for simulation tool testing
Automate metric collection on production calls
Set up simulation test matrices