> ## Documentation Index
> Fetch the complete documentation index at: https://docs.roark.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Studio

> Author, test, and evaluate metrics in one workbench before they run on live traffic

Studio is the metric workbench: build metrics, test them against real calls, and run evaluation batteries, all before anything touches production traffic. Find it in the sidebar under **Measure → Studio**.

<Note>
  Studio replaces the Playground. If you're looking for the old Playground docs, everything it did (and more) now lives here.
</Note>

Studio has three modes:

| Mode         | What it's for                                                                       |
| ------------ | ----------------------------------------------------------------------------------- |
| **Hub**      | The landing page: start authoring or evaluating, and revisit recent evaluation runs |
| **Author**   | Define or edit a single metric, with a test-call rail and version history           |
| **Evaluate** | Run a battery of metrics across a set of calls and compare results side-by-side     |

***

## The hub

Opening Studio lands you on the hub ("Build metrics. Run evaluations.") with two start cards:

* **Define a new metric** opens Author mode with a blank metric.
* **Run a battery of evals** opens Evaluate mode.

Below the cards, **Recent evaluation runs** lists your past batteries (e.g. "3 metrics × 5 calls") with a status chip: In progress, All passing, "N failing", Canceled, or Failed. Reopen any run to see its full result matrix, or fork it into a new evaluation.

***

## Author mode

Author mode is the metric editor. You reach it from the hub, from the **New metric** button on the Metrics page, or by clicking any metric in the [Library](/documentation/metrics/overview).

The editor walks you through four numbered sections:

<Steps>
  <Step title="01 · Engine: what kind of metric is this?">
    Pick how the metric is computed:

    | Engine        | How it works                                                                  |
    | ------------- | ----------------------------------------------------------------------------- |
    | **LLM Judge** | Scores each call against your natural-language prompt                         |
    | **Pattern**   | Composes triggers, outcomes, and time windows. Fires when the pattern matches |
    | **Formula**   | Math over other metrics                                                       |

    <Warning>
      The engine is **locked after the first save**. To measure the same thing with a different engine, create a new metric.
    </Warning>
  </Step>

  <Step title="02 · Output: what does the metric return?">
    Choose the result type. Some types need extra configuration:

    | Output             | Returns                | Extra config             |
    | ------------------ | ---------------------- | ------------------------ |
    | **Boolean**        | Yes / No               | True and False labels    |
    | **Scale**          | A score in a range     | Min/Max plus band labels |
    | **Classification** | One of your categories | Category list            |
    | **Numeric**        | A number               | A unit                   |
    | **Count**          | An occurrence count    | A unit                   |
    | **Text**           | Free-form text         | None                     |
  </Step>

  <Step title="03 · Instructions, Logic, or Formula">
    The third section adapts to your engine. For **LLM Judge**, write instructions tuned to the output type: for a Boolean metric, the yes/no question the model should answer; for a Scale, what each band means. For **Pattern**, define the logic: when should this metric fire? For **Formula**, define the computation over other metrics.
  </Step>

  <Step title="04 · Group: where should this metric live?">
    Assign the metric to a group (analysis package). Groups organize the Library on the Metrics page, so pick one your team will look for it under.
  </Step>
</Steps>

### Test against real calls

The right-hand rail is your test bench. Use **Add test calls** to pick the calls Studio should evaluate the metric against, then run them individually or hit **Run all**. Each run streams through the evaluation pipeline live, so you can watch the result land and tighten your instructions between runs.

<Tip>
  Iterate here until the metric returns what you expect on calls you already know the answer for, before you apply it to live traffic.
</Tip>

### Version history

Every save publishes a new version. The **History** tab in the rail lists them all: view any past version, or restore one, which publishes a new version from the old one (you never lose the intermediate history).

### System metrics and variants

[System metrics](/documentation/metrics/system-metrics) open read-only in Author mode. To change one, click **Customize for your team**. This forks an org-scoped variant you can freely edit and save. The Roark-managed original stays intact, and your instructions go in an "Additional instructions" field layered on top of the system prompt.

### Editing a metric that's already live

If the metric is bound to collectors, a notice appears above the form: **"In use by N collectors: saving publishes a new version every collector will pick up."**

<Warning>
  Saving a metric republishes it everywhere it runs. Every collector using the metric picks up the new version for calls scored from that point on. Test your changes in the rail before saving.
</Warning>

***

## Evaluate mode

Evaluate mode runs a battery of metrics across a set of calls and compares results side-by-side. Use it to validate a metric at scale, compare candidate metrics against each other, or spot-check quality across a sample of production calls.

<Steps>
  <Step title="Pick metrics">
    Use **Add metrics** to choose one or more metrics from your Library. You can attach a pass/fail threshold inline per metric (e.g. "Pass if ≥ 0.7") to turn raw scores into verdicts. See [Thresholds](/documentation/metrics/thresholds).
  </Step>

  <Step title="Pick calls">
    Use **Add calls** to select the conversations to evaluate.
  </Step>

  <Step title="Run battery">
    Hit **Run battery**. Results stream in as a comparison **matrix** (metrics × calls) or grouped **by call**. Switch views to suit the question you're asking. You can cancel a run in progress.
  </Step>
</Steps>

Finished batteries appear under **Recent evaluation runs** on the hub, so you can reopen the matrix later or fork the run into a new evaluation.

<Note>
  Prefer to run evaluations programmatically? [Metric collection jobs](/documentation/metrics/metric-collection-jobs) are the API equivalent: run a set of metrics across a set of calls via the SDK or REST.
</Note>

***

## Apply a metric to live calls

Testing in Studio scores only the calls you pick. To score matching calls automatically as they come in, bind the metric to a collector. The Author mode footer (**"Apply this metric to live calls"**) has an **Add collector** button that opens the collector editor with your metric prefilled and the modality locked to what the metric supports.

Once bound, the footer shows **"Running on N collectors"** instead. See [Collectors](/documentation/metrics/metric-collectors) for how conditions and scoping work.

<Note>
  The REST API and SDKs keep the older name for collectors: endpoints live under `/v1/metric/policies` and the SDK client is `client.metricPolicy.*`.
</Note>

***

## Next steps

<CardGroup cols={2}>
  <Card title="Custom Metrics" icon="gauge" href="/documentation/metrics/custom-metrics">
    Deep-dive on LLM Judge, Pattern, and Formula metric types
  </Card>

  <Card title="Collectors" icon="funnel" href="/documentation/metrics/metric-collectors">
    Run your metrics automatically on matching live calls
  </Card>

  <Card title="Thresholds" icon="ruler" href="/documentation/metrics/thresholds">
    Turn scores into pass/fail verdicts
  </Card>

  <Card title="Metric Collection Jobs" icon="list-checks" href="/documentation/metrics/metric-collection-jobs">
    Run evaluations at scale via the API
  </Card>
</CardGroup>