Agents and Client Products
A model is not a product. Engineers buy something that fits into their day: it reads the issue, works in their repository, runs their tests and proposes a change they can review. This part designs Maestro's agent and the three products built on it, prices a single agent task from first principles (and shows why caching makes long tasks affordable), compares seat and per-task pricing with a script, and covers integrations, onboarding and the data agreement that keeps the flywheel turning.
task_cost.py and pricing.py were run (real output below) on our assumptions for token counts, prices and usage. tool_server.py uses an external SDK and was not executed for this article. Customer numbers are the scenario. Disclosure: Claude, made by Anthropic, helps write this site.
1. Mission log: months 4โ17
- Maestro-0 runs as an internal agent with three design partners; every session is reviewed weekly.
- Packages the agent as a pull-request bot, a CI-failure fixer and an IDE assistant; writes
task_cost.pyandpricing.py.
- Pilot-to-paid conversions begin; hybrid pricing chosen (section 6).
2. What the agent does
One agent task, step by step
The agent never merges by itself.
- The interface matters as much as the model.
- SWE-agent (Yang et al., 2024) showed that designing the agent's tools for a model (simple file viewing, editing and search commands) changed results substantially on SWE-bench.
- His tools are deliberately few and well-described: read, search, edit, run tests, propose.
- Skills are retrieved first.
- Before exploring, the agent looks up verified skills from the library that match the task (Part 1's H1), which shortens the search.
3. Three products from one agent
| Product | Trigger | Output | Who uses it |
|---|---|---|---|
| PR bot | An issue labelled for the bot | A pull request with tests and explanation | Teams with a backlog of small issues |
| CI fixer | A failing build on the main branch | A proposed fix or a diagnosis | Platform and release engineers |
| IDE assistant | A developer's request in the editor | Edits and explanations inline | Individual engineers |
- Why start with the PR bot.
- Its output is a pull request: easy to review, easy to measure (merged, edited, rejected) and safe by default.
- The CI fixer reuses the same loop with a different trigger; the IDE assistant needs low latency and is the most competitive market, so it comes last.
4. What one task costs to serve
- Agents re-read their own context every step.
- The system prompt, the files read so far and every tool output are sent again with each step.
- Without caching, input tokens grow roughly with the square of the number of steps.
"""task_cost.py โ what one agent task costs, and why context caching matters.
An agent loop re-sends its growing context every step: system prompt,
repository files read so far, tool outputs. Without caching, input tokens
grow roughly quadratically with steps. Prices are Maestro's own serving
costs from Part 15 plus our assumptions for input and cached tokens.
"""
OUT_PRICE = 1.17 / 1e6 # $/output token (serving_capacity.py, FP8 on 2 GPUs)
IN_PRICE = OUT_PRICE / 5 # prefill is cheaper per token than decode (assumption)
CACHED = IN_PRICE / 10 # reusing a cached prefix (assumption)
SYSTEM, PER_STEP_IN, PER_STEP_OUT = 6_000, 2_500, 400 # tokens
def task(steps, cache):
ctx, cost, tin = SYSTEM, 0.0, 0
for _ in range(steps):
new = PER_STEP_IN
if cache:
cost += ctx * CACHED + new * IN_PRICE
else:
cost += (ctx + new) * IN_PRICE
tin += ctx + new
cost += PER_STEP_OUT * OUT_PRICE
ctx += new + PER_STEP_OUT
return tin, cost
def main():
print(f"{'steps':>5s} {'input tokens':>13s} {'no cache':>9s} {'cached':>8s}")
for s in (5, 15, 30, 60):
tin, c0 = task(s, False); _, c1 = task(s, True)
print(f"{s:5d} {tin:13,d} ${c0:8.3f} ${c1:7.3f}")
_, c = task(30, True)
print(f"a typical 30-step task costs ~${c:.3f} to serve; at 60% success, ${c/0.6:.3f} per solved task")
if __name__ == "__main__":
main()Serving cost of one task, with and without prefix caching
Real output of task_cost.py (our token and price assumptions).
- What it means.
- A 30-step task costs about 6.5 cents to serve with caching and 37 cents without; at 60 steps the gap is seven-fold.
- Prefix caching (Part 15) is not an optimisation for this product; it is what makes long tasks affordable.
- Step limits matter too: a task that wanders for 100 steps costs more and rarely succeeds.
5. Connecting to clients' tools
- Each client has its own tracker, CI system and code host.
- Integrations: GitHub and GitLab for code and pull requests; issue trackers; CI systems for build logs.
- A standard protocol saves building every integration twice.
- The Model Context Protocol (MCP), introduced by Anthropic in November 2024 as an open standard, lets an agent call tools exposed by servers in a common format.
- He exposes the sandboxed test runner and pull-request creation as tools; risky ones still go through the guard and confirmation (Part 13). See the site's tool gateways and MCP article.
# tool_server.py โ exposing the client's test runner to the agent as a tool over
# the Model Context Protocol (sketch using the MCP Python SDK's FastMCP helper;
# check names against the SDK version you install).
from mcp.server.fastmcp import FastMCP
from sandbox import run_in_sandbox # micro-VM runner (Part 13)
mcp = FastMCP("maestro-repo-tools")
@mcp.tool()
def run_tests(path: str = "tests/", timeout_s: int = 300) -> str:
"""Run the repository's test suite inside the task sandbox and return a summary."""
res = run_in_sandbox(["pytest", "-q", path], timeout=timeout_s, network=False)
return res.summary(max_chars=4000)
@mcp.tool()
def open_pull_request(title: str, body: str) -> str:
"""Propose the current patch as a pull request. Requires human confirmation (Part 13)."""
return request_confirmation("open_pull_request", title=title, body=body)
if __name__ == "__main__":
mcp.run()6. Pricing: seats or results?
- Two common models.
- Per seat: a monthly fee per engineer, simple to buy and budget.
- Per result: a price per accepted task, aligned with value but harder to forecast.
"""pricing.py โ per-seat vs per-solved-task pricing for the agent product.
Gross margin = (revenue - serving cost) / revenue. Usage per engineer
varies a lot, so seat pricing can lose money on heavy users. All prices,
usage levels and success rates are our assumptions for the story.
"""
COST_PER_TASK = 0.065 # task_cost.py: 30-step task with caching
SUCCESS = 0.6
SEAT = 40.0 # $ per engineer per month
PER_SOLVED = 1.50 # $ per merged/accepted task
def main():
print(f"{'tasks/eng/month':>15s} {'seat margin':>12s} {'per-task margin':>16s}")
for tasks in (20, 100, 300, 600):
cost = tasks * COST_PER_TASK
seat_m = (SEAT - cost) / SEAT
rev = tasks * SUCCESS * PER_SOLVED
task_m = (rev - cost) / rev
print(f"{tasks:15d} {seat_m:12.0%} {task_m:16.0%}")
print("hybrid: seat fee with an included allowance, then per solved task above it")
if __name__ == "__main__":
main()- The finding.
- A $40 seat earns a 97% gross margin from a light user (20 tasks a month) but only about 2% from a heavy one (600 tasks): heavy users consume their seat in serving costs.
- Per-solved-task pricing keeps a steady margin (93% here) whatever the usage.
- His choice: hybrid.
- A seat fee that includes an allowance of tasks, then a price per solved task above it.
- Customers get a predictable base; the company is protected from heavy usage; and pricing per solved task makes the success rate everyone's goal.
7. Try your own pricing
Pricing calculator (our model)
8. Onboarding a client
From signed contract to first merged change
Offline replay on past issues sets expectations before anything touches live work.
- Set expectations with data.
- The offline replay on the client's own past issues gives a success estimate with an interval (Part 12), so promises match reality.
9. How products feed the flywheel
- Every accepted change is a verified skill candidate.
- With the client's agreement, the method behind it is abstracted and added to the library (Part 8).
- Rejected and edited changes are just as valuable: they become new hidden tests and loop tasks (Part 11).
- The data agreement is part of the product.
- Clients choose: contribute abstracted skills and get a price discount, or opt out entirely.
- Either way, raw code never leaves their sandbox for training (Part 14).
10. Where the products stand
- By month 17 the story has nine paying customers (plan: twelve), mostly on the PR bot; the CI fixer is in pilot with three; the IDE assistant is postponed because established tools dominate that market.
- Accepted-change rates in pilots are reported to clients with intervals, not given here as results.
- These are story outcomes.
11. Designing for review
- The agent's output is read by a busy engineer.
- A pull request must be small, focused and explained: what changed, why, how it was tested, what was not done.
- Rules the agent follows.
- One issue, one pull request; no drive-by refactoring.
- Tests added or updated with every behaviour change.
- A short explanation that names the files changed and the test results.
- When unsure, ask in the issue instead of guessing.
- Measured in pilots.
- Acceptance rate, edits before merge and reverts (Part 12) show whether reviews get easier over time.
What a good agent pull request contains
Our product rules for the story.
12. Knowing when to stop
- A failing agent is expensive and annoying.
- Long runs cost the most (task_cost.py) and rarely succeed.
- Stop rules.
- A step limit per task type, set from data on where successes happen.
- Stop early when tests keep failing the same way, and report a diagnosis instead of a patch.
- Hand back to a human with what was tried; a good diagnosis is often worth as much as a fix.
13. Competing with much larger companies
- Frontier labs and large developer-tool companies sell coding agents too.
- They have stronger general models and bigger distribution.
- Where a start-up can win.
- Depth in its clients' own repositories: verified skills learned from their kind of work.
- Trust: gate logs and evaluation reports that show why each update shipped (Part 14).
- Price per solved task, made possible by an efficient 11.7B-active model and caching.
- Where it should not try.
- General chat and the IDE-assistant market, where the leaders are entrenched; hence the postponement.
14. Selling to engineering teams
- The buyer and the user are different people.
- Engineers decide whether the tool is useful; engineering managers and security teams decide whether to buy.
- What each needs.
- Engineers: a trial on their own repository with real pull requests.
- Managers: the offline replay report with an interval and a cost estimate.
- Security: the questionnaire answers, data agreement and sandbox design (Parts 13โ14).
- The sales motion.
- A two-week pilot on labelled issues, a review meeting with numbers, then a contract with the hybrid pricing.
15. Supporting clients
- Support is part of the product for a young company.
- A shared channel with each client; a named engineer for the first month.
- Every complaint is tagged: wrong fix, slow, unclear explanation, integration issue.
- Support feeds the system.
- Wrong fixes become regression tests and loop tasks; unclear explanations become preference data; integration issues become onboarding checks.
Where client feedback goes
Every piece of feedback lands somewhere specific.
16. Product metrics that matter
| Metric | Meaning | Healthy direction |
|---|---|---|
| Accepted-change rate | Share of proposed changes merged | Up |
| Edits before merge | How much reviewers had to change | Down |
| Reverts within 30 days | Merged changes later undone | Down |
| Tasks per active engineer | How much teams rely on it | Up |
| Cost per solved task | Margin | Down |
| Time to first merged change | Onboarding speed | Down |
17. Seven product mistakes
- Shipping a chat window and calling it an agent.
- Letting the agent merge without review.
- Seat pricing with unlimited heavy usage.
- No step limit, so failed tasks cost the most.
- Ignoring caching in the cost model.
- Promising success rates measured on public benchmarks.
- Taking raw client code into training "to improve the product".
18. Delegation box
- Which products to build and in what order
- Pricing
- Data agreements with clients
- Offline replays on client histories
- Writing integration code for review
- Weekly session-review summaries
- Code-host and CI integrations via standard protocols
- An outside pricing adviser for two sessions
- Design partners as early reviewers
19. Budget and risk tracker
| Item | Status after Part 16 | Note |
|---|---|---|
| Spend to date | ~$9.8M (scenario) | Serving, product engineering, integrations |
| Customers | 9 paying (plan 12) (scenario) | PR bot first, CI fixer in pilot |
| Time | Month ~17 | Operations and the scorecard next |
| Market risk | Medium | Behind plan on customers |
| Margin risk | Low | Hybrid pricing, caching |
| Competition risk | High | IDE assistant postponed |
20. The series
The Maestro: 18 parts in five phases
- A product turns Maestro-1 into something engineers use without thinking about models: a pull request to review.
- The cost model showed that caching makes long tasks affordable, and the pricing script showed why heavy users break seat pricing.
- Next, Part 17: running it all every day, and keeping the flywheel turning.
21. Reading the numbers
- Primary / standard
- SWE-agent agent-computer interface (Yang et al., 2024). SWE-bench (Jimenez et al., 2023). Model Context Protocol introduced as an open standard by Anthropic, November 2024.
- Secondary โ ๏ธ
- None in this part.
- Our assumptions
- Token counts per step, prices, usage levels, success rates, customer numbers and product timing.