Ocarina

The reproducible part of an agentic system

MCP tool calls return the same output for the same inputs, regardless of which LLM issued them. Write a playbook by hand or record one from a session, then replay it in CI to check the server still returns what you expect and catch regressions, with no API key.

Install

Requires Go 1.26+.

go install github.com/msradam/ocarina@latest

Or download a binary from releases.

Quick start

See what a server exposes. mcp-server-time needs no credentials:

ocarina docs uvx mcp-server-time

Write a rondo:

server:
  command: uvx
  args: [mcp-server-time]

rondo:
  - name: time in Tokyo
    tool: get_current_time
    args:
      timezone: Asia/Tokyo
    expect:
      contains: "datetime"

Play it. No LLM, no API key:

ocarina play clock.yaml
# ==> time in Tokyo (get_current_time)
# {
#   "datetime": "2026-06-27T09:30:00+09:00",
#   "timezone": "Asia/Tokyo"
# }
#     PASS: contains "datetime"

To capture a live session instead of writing one, see record. For ready-made environments you can clone and run, see Showcases.

Rondo format

A rondo is a YAML file with three parts: keys for variables, a server (or servers) to connect to, and a rondo list of steps. Write one by hand or record it from a live session, then commit it to git.

# Investigate any GitHub repo. Change keys.owner and keys.repo to switch repos.
keys:
  owner: modelcontextprotocol
  repo: go-sdk

server:
  command: npx
  args: [-y, "@modelcontextprotocol/server-github"]

rondo:
  - name: list recent commits
    tool: list_commits
    args:
      owner: "{{owner}}"
      repo: "{{repo}}"
      per_page: 5
    grab: ".0.sha"          # extract the first SHA from the JSON array
    echo: latest_sha        # capture it into keys for later steps

  - name: show latest commit
    tool: get_commit
    args:
      owner: "{{owner}}"
      repo: "{{repo}}"
      sha: "{{latest_sha}}"   # value from the previous step
    expect:
      contains: "author"

  - name: list open issues
    tool: list_issues
    args:
      owner: "{{owner}}"
      repo: "{{repo}}"
      state: open

Step fields

Rondo-level fields

Coming from Ansible? tasks: is accepted in place of rondo:, and register: in place of echo:.

Multiple servers

One rondo can talk to several servers. Declare them under servers and set server: on each step. A step that omits server: uses the first entry. When more than one server is in play, output and diff namespace tool names by server, like time.get_current_time.

servers:
  time:  {command: uvx, args: [mcp-server-time]}
  fetch: {command: uvx, args: [mcp-server-fetch]}

rondo:
  - name: get the time
    server: time
    tool: get_current_time
    args: {timezone: UTC}

  - name: fetch a page
    server: fetch
    tool: fetch
    args: {url: "https://example.com"}

Remote servers

Give a server a url: instead of a command: to reach a hosted MCP server over the Streamable HTTP transport. Headers are sent on every request, so a bearer token works through {{env.X}}. When a tool returns structuredContent, grab and expect run against that typed JSON instead of the text block.

server:
  url: https://api.githubcopilot.com/mcp/
  headers:
    Authorization: "Bearer {{env.GITHUB_TOKEN}}"

rondo:
  - name: who am I
    tool: get_me
    args: {}
    expect:
      contains: "login"

Commands

ocarina docs uvx mcp-server-time
ocarina docs --out docs/github.md npx -y @modelcontextprotocol/server-github

ocarina record out.yaml uvx mcp-server-sqlite --db-path /tmp/db.sqlite
ocarina record out.yaml npx -y @modelcontextprotocol/server-github

ocarina play examples/github-investigation.yaml
ocarina play examples/mcp-smoke-test.yaml   # has assertions
ocarina play examples/time-zones.yaml -e owner=acme --tags smoke

ocarina validate examples/github-investigation.yaml

ocarina diff examples/github-investigation.yaml

ocarina lock audit.yaml           # write audit.yaml.lock
ocarina lock audit.yaml --check   # fail if the live schema drifted

ocarina hum uvx mcp-server-time -- get_current_time timezone=UTC
ocarina hum npx -y @modelcontextprotocol/server-github -- list_commits owner=pytorch repo=pytorch per_page=1

Examples

Working rondos in examples/, each verified against a live server.

Showcases

Standalone repositories you can clone and run, each a real working environment for a different MCP server.

Use in CI

play exits 0 if all expectations pass, non-zero otherwise. No API keys needed.

name: MCP smoke tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-go@v5
        with:
          go-version: '1.26'
      - run: go install github.com/msradam/ocarina@latest
      - run: ocarina play examples/mcp-smoke-test.yaml

What MCP makes easy, and what it does not yet

Ocarina works because MCP gives every server the same shape: named tools with typed arguments, reachable over one protocol. Learn the grammar once and you can drive a time server, a GitHub server, a Postgres database, and a 3D modeling app with the same YAML. You point Ocarina at a server, fill in the calls, and it runs them the same way on every execution. That part is solid today.

The friction is in what comes back. These servers were built so a model could read the output and decide what to do next, so many of them answer in prose rather than data. mcp-server-fetch returns markdown. mcp-server-sqlite returns the Python repr of a row, single quotes and all. The arguments are typed; the results often are not. Ocarina smooths over the common cases, and will parse that Python repr before a grab: runs, but when a server replies in free text, pulling a clean value out of it stays awkward. That ceiling belongs to the server, not to Ocarina.

Errors are uneven for the same reason. The spec lets a tool flag a failure with isError, but plenty of servers report trouble as ordinary text and never set it. Ocarina catches a misspelled tool name or a missing required argument by checking the live schema before it calls, so those fail loudly. A server that returns table not found as a cheerful string is harder to catch on its own, and that is what an expect: assertion is for.

So the honest scope. Ocarina is an automation framework for the MCP servers that behave like clean RPC, and a deterministic test harness for the ones that do not. Today that means smoke tests in CI and scripts that chain a few well-behaved servers. Recording a session and replaying it without a model works now too. The reach grows as the ecosystem matures and servers return more structure. The Blender demo on the home page is the far end of what is already possible. The rough edges above are the near end you will actually hit.

What Ocarina is not

It does not score LLM outputs, compare models, or track token costs. It connects to the real server on every play run, so a broken server breaks the replay. It runs one linear pass with no scheduler and no state carried between runs. The output is stdout and an exit code.

GitHub  ·  Releases  ·  MIT license  ·  Whistle icon by Alessio Capponi / Noun Project (CC BY 3.0)