A robot clicked every button. It found eight security holes.
We stopped testing pieces and started testing pages. Real browser, real server, real clicks - on every endpoint a customer can touch.
Our code passed type checks, lint and unit tests. It still shipped bugs. Not exotic ones - a reset form that ignored the confirmation password, invoice links that went nowhere, a feed that showed deleted threads.
Every one of those lived in the gap between the pieces. The function was fine. The page around it was not. So we built a test framework that only cares about one thing: what a customer actually sees and sends.
Why test the real page?
Each layer of testing catches a different class of bug. Only the last one sees what your users see.
Checks one function
Fast and cheap. Blind to wiring: wrong URL, wrong field name, a button that never sends.
Checks one request
Proves the server answers. Never proves the page sends that request - or renders the reply.
Checks the whole trip
Opens the page in Chromium, types like a person, clicks submit, reads the response off the wire. If it works here, it works for customers.
How the harness works
Playwright driven from bun test, built on top of next-vibe's endpoint definitions. The definition already knows every field - so the test doesn't need to.
Open the real page
The test opens the customer page - /user/login, /story/framework - not a test stub. It waits until React owns the form before typing.
Fill with typed values
You pass values, not selectors. Field drivers read the definition and operate the visible UI - type, pick an option by its label, attach a file.
Capture the call off the wire
Submit returns the truth: the request body, status and response. The harness asserts the request carried exactly what was typed.
Record once, replay forever
Paid providers - AI models, image, music, video - are recorded per test on the first run and replayed after. No surprise bill, deterministic runs.
Report coverage
A coverage report counts every public endpoint: one success and one rejection each. Gaps are visible, not guessed.
describe("E2E: contact form", () => {
const e2e = useE2E("contact");
e2e.it("sends a message", async ({ open, uniqueEmail }) => {
const form = await open(contactEndpoints.POST, {
path: "/story/framework",
});
await form.fill({
name: "E2E",
email: uniqueEmail("contact"),
subject: ContactSubject.BUG_REPORT,
message: "...",
});
const result = await form.submit();
expect(result.sent).toBe(true);
});
});A full spec. No selectors, no mocks.
The scoreboard
The remaining endpoints need paid provider keys for search, scraping and speech before they can be recorded.
The security holes it found
None of these showed up in type checks or unit tests. All of them showed up the moment a real browser walked the real flow.
Open redirect on /track
SSRF via user-supplied URLs
Server cache shared between requests
System-prompt debug read any thread
Public feed listed hidden threads
Login options leaked accounts
Reset ignored the confirmation
Private files cached publicly
And the things that were just broken
Not dangerous - just quietly failing for real people:
- Chat attachment URLs and client error reports returned 404.
- Root permissions were never fetched - customers couldn't post in Shared.
- The embed widget mount was dead.
- Invoice email links returned 404.
- The models page was unusable.
- Music generation on OpenRouter failed.
- Test fixture scope leaked paid live calls.
- Local PGlite databases returned the wrong rows from INSERTā¦RETURNING.
- Visitor pagination on /tools was broken.
What this means for you
Every page you use to sign up, log in, pay, chat, generate or subscribe now has a test that opens it in a real browser and does what you do. When we change something, the robot walks it again before you do.
The framework ships with next-vibe. If you build on it, your endpoints get the same harness for free.
Read the harness. Use it on your own code.
The E2E framework is open source and part of next-vibe. One spec file per feature, colocated with the definition.