Your examples, your evidence.

Requirements and availability

Silo Bench needs an Apple silicon Mac running macOS 27 or later. It is coming soon to the Mac App Store and is not on sale yet. The planned price is $39.99 USD, paid once, with no subscription and no Silo Bench account.

Trying it before writing your own cases

On the first screen, choose Open recorded example. It creates a small offline project from handwritten outputs, so you can open Runs, Compare and blind review straight away. In Tests, Add starter pack installs any of the 17 bundled packs, 180 original cases covering every kind of evaluation. They are examples to learn from, not a benchmark.

How an evaluation works

1. Tests. Write cases with a prompt, a reference and the checks that matter, then freeze them. A frozen evaluation never changes; edits create a new version.
2. Runs. Send the cases to a model you connect, or import outputs produced anywhere else. Every request and result is kept, including failures. An interrupted run continues without sending anything twice.
3. Compare. See coverage, outcomes and paired differences between candidates on the same frozen evaluation. Results are labelled exploratory when the sample is small.
4. Review. Rate outputs blind, with the candidate's identity hidden, or review a single result in place.

Silo Bench covers fifteen kinds of work, from writing and structured extraction to transcription, image editing, video, 3D geometry and coding, each with its own checks. Built-in viewers show word and field differences, image overlays and masks, loudness-matched audio, video with captions, 3D models with dimensions, and MIDI as a piano roll.

Do I need a connection?

No. Imported outputs need no connection, and every project starts offline. To have Silo Bench send cases to a model itself, turn on connected mode for that project in Connections, confirm the notice, and add a connection: an OpenAI-compatible endpoint, including a server running on your own Mac, Anthropic, Gemini, or Apple's on-device model where your Mac provides it. Requests are made with your own key, which is kept in the Keychain on this Mac. You can switch a project back to offline at any time.

A connection does one thing: text, text about an image, transcription, speech or image generation. Each must pass a capability check on your endpoint before a run. Transcription, speech and image generation need an endpoint that follows the OpenAI audio and image formats.

What the scores mean

Computed checks are not a verdict on overall quality. They measure what you chose to check, such as word error rate for transcripts, field accuracy for extraction or mask overlap for image edits. Subjective quality needs human review, which Silo Bench records separately from computed scores. When the sample is small, Compare labels the result exploratory.

Your data

Projects start offline, and scoring runs in a separate process that has no network access. Nothing is sent to a model unless you turn on connected mode for a project and add a connection, and then only the inputs you selected go to that service. There is no account, no analytics and no telemetry. The privacy policy has the details.

Troubleshooting

If a connection will not run, check that its capability check passed and that the endpoint supports the kind of request it is set up for. If you still need help, email us with your macOS version, Silo Bench version and what you were trying to do. Do not send API keys, and leave out confidential examples; we will ask if a non-sensitive example would help.