Use case · Research

Run one task on several AI engines and compare the results

To compare AI engines on a real task, create one Brainwrite bot per engine, such as Claude, Codex, and Grok, and give each the same prompt. Each bot runs on your own login with your own billing. Put them in a group to see the answers side by side, check each thread's token use and cost, and pick the engine that did the job best.

Download for Mac See pricing

Compare AI engines on one task

Claude bot, Codex bot, and Grok bot: each of you, read checkout.ts in the shared project and list the three riskiest functions with a reason. Do not change any files. Reviewer: compare the three answers for accuracy and score each one out of five.

Your Chief of Staff put a team on it

Nova· Chief of Staff
Atlas· Engine bots
Sage· Reviewer

3 engines ran the same brief on the same file; none changed any files
All three flagged applyDiscount; only one engine caught the missing idempotency key in chargeCard
Reviewer scores: 5, 4, and 3 out of 5, with one line of reasoning each
Cost and tokens: listed per thread from each engine's usage panel

DoneWhich engine to use for this kind of task from now on waits for you. Nothing was sent.

An illustrated example. Your team works on your own files and apps.

A developer compares three closed notebooks at his desk, laptop and phone beside him, while illustrated Brainwrite AI agents Atlas, Dash and Sage weigh in.

The job

Doing it yourself, and handing it off.

Doing it yourself

Every week someone claims a different model is now the best. Testing it yourself means copying the same prompt into three apps, waiting for each, and pasting the answers into a doc to compare. The apps give different context, so the test is not fair. You rarely see what each answer cost, and the result lives in a doc nobody can rerun next month when the models change again.

What the team does

Run the same brief on each engine with the same files and instructions
Show each engine's answer in one group conversation
Score answers against your criteria and explain each score
Note differences in token use and cost from each thread's usage

What waits for you

Which engine to use for this kind of task from now on
The scoring criteria, and whether a test was fair
Which provider subscriptions are worth paying for

The team

Who's on it.

Your Chief of Staff picks the specialists the job needs. Each has its own role, instructions, model, and app access.

NovaIllustrated example
Chief of Staff
Sends the same brief to each engine bot and writes the comparison.
AtlasIllustrated example
Engine bots
One bot per engine, such as Claude, Codex, Grok, Cursor, or a local model, each with the same instructions and files.
SageIllustrated example
Reviewer
Scores each answer against criteria you set, such as accuracy, completeness, and following the brief.

How it runs

A run, step by step.

01
Install the engines
Install and sign in to each agent CLI you want to test, then check detection in Settings → Engines.
02
Create matching bots
Make one bot per engine with the same instructions, and pick its engine and model in the model picker.
03
Run the same task
Put the bots in a group and send the brief once, or run the same prompt in each bot's own thread.
04
Compare
Read the answers side by side, check token use and cost per thread, and have the reviewer score them.

What you'll connect

Apps this job uses.

FAQ

Questions about Compare AI engines on one task

Which AI engines can I compare in Brainwrite?

Brainwrite starts the agent CLIs you have installed: Claude, Codex, Grok, Cursor, Kimi, Factory Droid, Antigravity, OpenCode, Qwen, Hermes, and Pi. It also supports the Mistral API and local models through OpenCode, Ollama, or LM Studio. You can only test engines you have installed and signed in to.

Who pays for each engine's usage?

Each provider bills your own login or API key, and no Brainwrite plan includes AI usage. Brainwrite shows token use and cost per thread, including uncached and cached input and output, so you can see what each engine's answer cost on the same task.

Is it a fair comparison?

As fair as you make it. Give every bot the same instructions, files, and permissions, and send the brief once in a group so they all start from the same message. Note that the Mistral API connection has no filesystem access, so for file-based tasks, compare engines that can read the working folder.

Built on