Momentic Review: QA Agents That Found 269 Bugs in 60 Minutes

Momentic Review: QA Agents That Found 269 Bugs in 60 Minutes
ToolMomentic
Pricing$250 free Mo spend for new signups; paid plans (check official site)
CategoryAI QA and testing
Monetization difficultyMedium

In short6 min read

  • Mo found 269 bugs in a 60-minute benchmark run, versus 86 for Codex.
  • Mo costs $2.67 per bug found; Codex was cheaper at $0.80 but caught far fewer.
  • New signups get $250 in Mo spend, and PR testing works with Vercel and Cloudflare previews.
  • Best fit: solo builders shipping fast with coding agents who lack a QA person.

If you ship software with Claude Code, Codex or Cursor, you already know the uncomfortable part: the code arrives faster than you can check it. Momentic is betting that the fix is not more test scripts but agents that use your app the way a customer would and report what breaks. In this review you will see how its agent, Mo, works, what the founders’ own benchmark showed, what each bug costs, and whether it belongs in a solo builder’s stack.

The timing matters. The founders argue that the change failure rate is rising as coding agent adoption scales, and they just shipped PR testing for Vercel and Cloudflare deploy previews. Every new signup also gets $250 in Mo spend, which is enough to run a real test on your own product before paying anything.

In this article

  1. What Momentic Actually Does
  2. The Benchmark, Read Carefully
  3. Recall vs Cost: A Decision Framework for Solo Builders
  4. Where the Money Is for Freelancers and Small Studios
  5. Caveats Before You Commit
  6. Set up your first QA agent run

What Momentic Actually Does

Momentic was built by Wei-Wei and Jeff, two engineers who, by their account, spent their careers on dev tooling and infrastructure at companies like Robinhood, Retool, WeWork and Qualtrics. They describe testing with Playwright and Selenium as a constant pain, which is why they built agents to do it.

Mo is a QA agent that plans, explores and bug-bashes the controls and user flows of web, iOS and Android apps, with desktop listed as coming soon. Instead of hardcoding XPath selectors, text or aria attributes the way open-source end-to-end frameworks require, you describe what matters in plain English and the agents figure out where to click.

According to the founders, Mo builds on three years of Momentic’s work and millions of test traces from complex, real-world apps.

The parts under the hood

  • A self-organizing agent graph. Mo builds a directed acyclic graph as it explores. Large product areas fork into parallel sub-agents, each with its own browser or device.
  • Specialized agents. Planner agents reason over screenshots and video, while locator agents inspect page structure to resolve plain-English targets. Others handle image analysis, video understanding and audio generation.
  • Resource leases. Parallel agents announce whether they are “using” or “modifying” shared fixtures. Many can use one at the same time, but only one can modify it, which keeps agents from corrupting each other’s test data.
  • Hosted scale. A tunnel exposes private sites and local apps to Momentic’s hosted agents, and one bug bash can spin up hundreds of concurrent browsers and devices.
How a Momentic Bug Bash Runs: Write a plain-English brief, Expose the app, Agents explore in parallel, Bugs get reported, Triage and fix

The Benchmark, Read Carefully

The team tested Mo against general coding agents on four open-source apps: Gitea, Metabase, Twenty and Saleor, with 1,396 human-labeled bugs. Each agent got the same brief, a fresh E2B sandbox and no access to source code. Scoring used an LLM judge plus a human reviewer. Here are the results of the 60-minute run, as published by Momentic:

Agent Bugs found Recall Precision Cost per bug
Mo (Momentic) 269 19.3% 84% $2.67
Codex + Playwright MCP 86 6.2% 86% $0.80
Claude Code + Playwright MCP 59 4.2% 80% $4.09
Devin 27 1.9% 84% $3.89

Two honest observations. First, this is a vendor-run benchmark, so treat it as a strong signal rather than an independent verdict. The team did publish the write-up, dataset and harness, which means anyone can check the method. Second, Mo does not win on every axis. Codex was slightly more precise and much cheaper per bug. Mo’s edge is recall: it simply finds more of the real bugs.

The founders also report that Mo improves with time. Between 30 and 60 minutes it went from 177 to 269 bugs, a 52% gain, while Codex gained 16% over the same window. If you give it a longer run, you should expect more coverage, not a plateau.

Recall vs Cost: A Decision Framework for Solo Builders

The interesting question is not which agent is best. It is what a missed bug costs you compared with what a found bug costs. Here is a simple way to decide.

Your situation What matters most Likely fit
Side project, few users, low stakes Cheapest signal A coding agent with Playwright MCP may be enough
Paying customers, checkout or sign-up flows Catching the bug before users do Mo’s higher recall is worth the higher per-bug cost
You ship many PRs a week with AI help Catching regressions on every preview Mo’s PR testing on Vercel or Cloudflare previews
Client work with a launch deadline Evidence the app was tested A timed Mo bug bash before handoff

A useful rule: if one escaped bug in a payment, onboarding or booking flow would cost you more than a handful of found bugs at $2.67 each, recall is the cheaper choice. If your app is a weekend experiment, it probably is not.

There is also a hidden cost the table cannot show. With script-based tests, someone has to maintain the selectors every time the UI changes. The founders point out that even with coding agents, those scripts take a real amount of time and tokens to build and keep working. A plain-English agent shifts that maintenance burden off your plate.

Where the Money Is for Freelancers and Small Studios

Momentic is a developer tool, but it opens a few practical income angles for people already doing automation or build work. Results depend entirely on your execution and client base.

1. Add a “pre-launch bug bash” to your build packages

If you build apps or internal tools for clients, a documented agent-run test pass before handoff is an easy upsell. You run Mo against the staging site, triage the findings, fix what is yours and deliver a short report. Clients get proof of testing; you get fewer post-launch support calls.

2. Offer QA as a retainer for AI-built products

Plenty of small founders now ship with coding agents and no QA person. A monthly retainer that runs PR testing on their deploy previews and reviews the flagged issues is a natural extension of the work covered in our guide to selling AI automation services.

3. Protect your own product’s revenue paths

If you run a small SaaS, the money is in not losing it. Pointing Mo at sign-up, checkout and upgrade flows is the most direct use of the free $250 credit.

Caveats Before You Commit

  • Pricing beyond the credit is not public in detail. The founders say paid plans include higher infrastructure capacity, included usage and top-up options. Check the official site for current pricing before you quote it in a client proposal.
  • Per-bug cost adds up. $2.67 per bug is cheap for a checkout bug and expensive for cosmetic noise. Write a focused brief so the agents spend time on flows that matter.
  • Precision is not 100%. At 84% precision, some reports will not be real bugs. Budget time to review findings before you forward them to a client.
  • Desktop is not there yet. Web, iOS and Android are supported; desktop is listed as coming soon.
  • Private apps need the tunnel. Local and staging apps reach Momentic’s hosted agents through its tunnel, so check that this fits your or your client’s security rules.

For a broader look at where autonomous agents fit in a one-person business, see our hub on AI agents for solo businesses. If your stack also includes ticket-to-PR agents, our Baloon review covers the other side of the same loop: code written by an agent, then checked by one.

Set up your first QA agent run

  • Sign up for Momentic and confirm the $250 Mo spend is on your account.
  • Pick one revenue-critical flow (sign-up, checkout or booking) and write a short plain-English brief for it.
  • If the app is local or on a private staging site, set up Momentic’s tunnel first.
  • Run a time-boxed bug bash, then compare a 30-minute and a longer run to see how coverage grows.
  • Triage every finding: real bug, false alarm or low priority. Track your own precision.
  • Connect PR testing to your Vercel or Cloudflare deploy previews if you ship often.
  • Work out your own cost per useful bug before you decide on a paid plan or pitch it to clients.

Want an agent to bug-bash your app before users do?

Pick the one flow that makes you money, such as sign-up or checkout, and write a three-sentence brief for it today. Then use the $250 signup credit to run one time-boxed bug bash and see what it finds.

Source: Launch HN: Momentic (YC W24) – Agents that catch bugs by using your app

Get new AI automation playbooks by email

One email when new guides and case studies go up. No spam, unsubscribe anytime.

Response

  1. […] Momentic Review: QA Agents That Found 269 Bugs in 60 Minutes […]

Leave a Reply