Ocarina

The reproducible part of an agentic system

MCP tool calls return the same output for the same inputs, regardless of which LLM issued them. Write a playbook by hand or record one from a session, then replay it in CI to check the server still returns what you expect and catch regressions, with no API key.

Install

Requires Go 1.26+.

go install github.com/msradam/ocarina@latest

Or download a binary from releases.

Quick start

See what a server exposes. mcp-server-time needs no credentials:

ocarina docs uvx mcp-server-time

Write a rondo:

server:
  command: uvx
  args: [mcp-server-time]

rondo:
  - name: time in Tokyo
    tool: get_current_time
    args:
      timezone: Asia/Tokyo
    expect:
      contains: "datetime"

Play it. No LLM, no API key:

ocarina play clock.yaml
# ==> time in Tokyo (get_current_time)
# {
#   "datetime": "2026-06-27T09:30:00+09:00",
#   "timezone": "Asia/Tokyo"
# }
#     PASS: contains "datetime"

To capture a live session instead of writing one, see record. For ready-made environments you can clone and run, see Showcases.

Rondo format

A rondo is a YAML file with three parts: keys for variables, a server (or servers) to connect to, and a rondo list of steps. Write one by hand or record it from a live session, then commit it to git.

# Investigate any GitHub repo. Change keys.owner and keys.repo to switch repos.
keys:
  owner: modelcontextprotocol
  repo: go-sdk

server:
  command: npx
  args: [-y, "@modelcontextprotocol/server-github"]

rondo:
  - name: list recent commits
    tool: list_commits
    args:
      owner: "{{owner}}"
      repo: "{{repo}}"
      per_page: 5
    grab: ".0.sha"          # extract the first SHA from the JSON array
    echo: latest_sha        # capture it into keys for later steps

  - name: show latest commit
    tool: get_commit
    args:
      owner: "{{owner}}"
      repo: "{{repo}}"
      sha: "{{latest_sha}}"   # value from the previous step
    expect:
      contains: "author"

  - name: list open issues
    tool: list_issues
    args:
      owner: "{{owner}}"
      repo: "{{repo}}"
      state: open

Step fields

Rondo-level fields

Coming from Ansible? tasks: is accepted in place of rondo:, and register: in place of echo:.

Multiple servers

One rondo can talk to several servers. Declare them under servers and set server: on each step. A step that omits server: uses the first entry. When more than one server is in play, output and diff namespace tool names by server, like time.get_current_time.

servers:
  time:  {command: uvx, args: [mcp-server-time]}
  fetch: {command: uvx, args: [mcp-server-fetch]}

rondo:
  - name: get the time
    server: time
    tool: get_current_time
    args: {timezone: UTC}

  - name: fetch a page
    server: fetch
    tool: fetch
    args: {url: "https://example.com"}

Remote servers

Give a server a url: instead of a command: to reach a hosted MCP server over the Streamable HTTP transport. Headers are sent on every request, so a bearer token works through {{env.X}}. When a tool returns structuredContent, grab and expect run against that typed JSON instead of the text block.

server:
  url: https://api.githubcopilot.com/mcp/
  headers:
    Authorization: "Bearer {{env.GITHUB_TOKEN}}"

rondo:
  - name: who am I
    tool: get_me
    args: {}
    expect:
      contains: "login"

Motifs: reusable fragments

A motif is a reusable, parameterized rondo fragment. Write a login flow or a setup routine once, then pull it into any rondo with motif: and pass parameters with with:. It is the Ansible role, or the pytest fixture, for rondos.

A motif runs in an isolated scope. It sees its own keys: as defaults, overlaid by the with: values the caller passes. It does not inherit the parent's other variables, so thread in anything it needs through with:.

keys:
  zone: UTC          # a default, overridable via with:

rondo:
  - tool: get_current_time
    args:
      timezone: "{{zone}}"
    expect:
      contains: datetime

server:
  command: uvx
  args: [mcp-server-time]

rondo:
  - name: default zone
    motif: motifs/time-probe.yaml

  - name: tokyo
    motif: motifs/time-probe.yaml
    with:
      zone: Asia/Tokyo

A motif runs against the including rondo's servers. See examples/motif.

Error handling: block, rescue, always

A step can hold a block: of steps with a rescue: and an always:, the same construct Ansible uses. The block runs until a step fails. On a failure, rescue: runs; if the rescue is clean, the failure is handled and the run continues with exit 0. always: runs either way, so it is where teardown belongs. Paired with a motif for setup, this is a fixture: stand up state, assert against it, tear it down even on failure.

rondo:
  - name: provision then verify, roll back on failure
    block:
      - tool: read_text_file
        args: {path: "{{dir}}/config.json"}
      - tool: read_text_file
        args: {path: "{{dir}}/missing.json"}   # fails here
      - tool: write_file                         # never runs
        args: {path: "{{dir}}/ok", content: ok}
    rescue:
      - tool: write_file
        args: {path: "{{dir}}/ROLLBACK", content: "recovered"}
    always:
      - tool: list_directory
        args: {path: "{{dir}}"}

examples/block-rescue mirrors the canonical Ansible blocks playbook one to one.

Commands

ocarina docs uvx mcp-server-time
ocarina docs --out docs/github.md npx -y @modelcontextprotocol/server-github

ocarina record out.yaml uvx mcp-server-sqlite --db-path /tmp/db.sqlite
ocarina record out.yaml npx -y @modelcontextprotocol/server-github

ocarina play examples/github-investigation.yaml
ocarina play examples/mcp-smoke-test.yaml   # has assertions
ocarina play examples/time-zones.yaml -e owner=acme --tags smoke

name: provision_workspace
description: Create a workspace and seed it with config files. One call instead of five.
params:
  - name: dir
    type: string
    required: true
return: listing
server:
  command: npx
  args: [-y, "@modelcontextprotocol/server-filesystem", /private/tmp]
rondo:
  - tool: create_directory
    args: {path: "{{dir}}"}
  - tool: write_file
    loop: '["alpha.conf", "beta.conf", "gamma.conf"]'
    args: {path: "{{dir}}/{{item}}", content: "managed by ocarina"}
  - tool: list_directory
    args: {path: "{{dir}}"}
    echo: listing

ocarina serve provision.yaml                  # stdio (default)
ocarina serve provision.yaml --http :8080     # Streamable HTTP

ocarina validate examples/github-investigation.yaml

ocarina diff examples/github-investigation.yaml

ocarina lock audit.yaml           # write audit.yaml.lock
ocarina lock audit.yaml --check   # fail if the live schema drifted

ocarina load smoke.yaml --vus 50 --duration 30s
ocarina load smoke.yaml --vus 20 --iterations 1000 --threshold p95<500ms

ocarina hum uvx mcp-server-time -- get_current_time timezone=UTC
ocarina hum npx -y @modelcontextprotocol/server-github -- list_commits owner=pytorch repo=pytorch per_page=1

Safe mode

play --safe and serve --safe refuse any tool not marked read-only in its MCP annotations (readOnlyHint). A step opts back in with allow_destructive: true. This keeps a read-only rondo from making a write when you point it at a server you do not fully control.

ocarina play audit.yaml --safe

This is a guardrail, not a security boundary. MCP annotations are advisory, and a server is free to misreport them, so treat --safe as protection against mistakes, not against a hostile server.

Output

play prints a per-step run and a final tally. Two open output surfaces let other tools consume a run without Ocarina shipping format-specific reporters.

Examples

Working rondos in examples/, each verified against a live server.

Showcases

Standalone repositories you can clone and run, each a real working environment for a different MCP server.

Use in CI

play exits 0 if all expectations pass, non-zero otherwise. No API keys needed.

name: MCP smoke tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-go@v5
        with:
          go-version: '1.26'
      - run: go install github.com/msradam/ocarina@latest
      - run: ocarina play examples/mcp-smoke-test.yaml

What MCP makes easy, and what it does not yet

Ocarina works because MCP gives every server the same shape: named tools with typed arguments, reachable over one protocol. Learn the grammar once and you can drive a time server, a GitHub server, a Postgres database, and a 3D modeling app with the same YAML. You point Ocarina at a server, fill in the calls, and it runs them the same way on every execution. That part is solid today.

The friction is in what comes back. These servers were built so a model could read the output and decide what to do next, so many of them answer in prose rather than data. mcp-server-fetch returns markdown. mcp-server-sqlite returns the Python repr of a row, single quotes and all. The arguments are typed; the results often are not. Ocarina smooths over the common cases, and will parse that Python repr before a grab: runs, but when a server replies in free text, pulling a clean value out of it stays awkward. That ceiling belongs to the server, not to Ocarina.

Errors are uneven for the same reason. The spec lets a tool flag a failure with isError, but plenty of servers report trouble as ordinary text and never set it. Ocarina catches a misspelled tool name or a missing required argument by checking the live schema before it calls, so those fail loudly. A server that returns table not found as a cheerful string is harder to catch on its own, and that is what an expect: assertion is for.

So the honest scope. Ocarina is an automation framework for the MCP servers that behave like clean RPC, and a deterministic test harness for the ones that do not. Today that means smoke tests in CI and scripts that chain a few well-behaved servers. Recording a session and replaying it without a model works now too. The reach grows as the ecosystem matures and servers return more structure. The Blender demo on the home page is the far end of what is already possible. The rough edges above are the near end you will actually hit.

What Ocarina is not

It does not score LLM outputs, compare models, or track token costs. It connects to the real server on every play run, so a broken server breaks the replay. play runs one linear pass with no scheduler and no state carried between runs. The output is stdout and an exit code, a structured JSON report, or OpenTelemetry traces.

GitHub  ·  Releases  ·  MIT license  ·  Whistle icon by Alessio Capponi / Noun Project (CC BY 3.0)