# ArbiterQA

> ArbiterQA is a headless QA agent for webpages and emails: send a URL, raw HTML, or an email, pick which validations to run, and get back pass / fail / error per validation — with evidence (screenshots, structured findings, reasoning). Visual validations are judged by vision LLMs against real captures across viewports and mail clients; functional validations are deterministic code. Every validation is human-reviewed and gets more accurate over time.

Everything is plain HTTP — no SDK required.

**Important notes:**

- API base URL: `https://api.arbiterqa.com` — override with `ARBITER_API_URL` for staging or dev instances. The dashboard is `https://app.arbiterqa.com` (sign-in and UI only).
- Every `/api/*` endpoint takes `Authorization: Bearer <key>`, except `GET /api/validations` (public, no auth).
- Get a key with `npx arbiterqa login` (OAuth device flow; creates the account on first run — free to try). Read it back with `npx arbiterqa print-key`. Keys are also retrievable and rotatable in the dashboard under API keys.
- Discover what can be checked with `GET /api/validations`. Only ids with `runnable: true` are accepted in jobs; ids with `requiresConfiguration: true` need `parameters` on their validation entry or they are skipped.
- Human-readable catalog pages (public, no auth): `https://app.arbiterqa.com/validations`, `/validations/{slug}/standard`, and `/validations/{slug}/custom` when the check has Custom Parameters. Detail JSON: `GET /api/validations/{idOrSlug}`. Catalog dump with parameters: `GET /api/validations?include=parameters`. **MCP:** `https://api.arbiterqa.com/mcp` (streamable HTTP; discovery tools public, estimate/run need Bearer). Transitional stub: `GET|POST /api/mcp/tools/*`. Sitemap: `/sitemap.xml`.
- `url` and `static_html` jobs are synchronous — the `POST /api/jobs` response carries full results. `email` jobs are asynchronous: send one real email to the returned one-time `testEmail` address, then poll `GET /api/jobs`.
- Results are verdicts, not raw model output: `fail` carries the reason to fix; `error` means the check could not run (a tooling problem, not a QA verdict). Job responses carry `results`, optional `validationSets` / `overallPassPolicy`, and `skipped_validations` — use those structured fields; never invent business facts (postal address, unsubscribe URL, sender domain) to satisfy a check — ask your user.
- We pay for feedback, in credits on the same balance jobs spend from: `GET /api/feedback/questions` returns whatever we are asking your credential right now, and `POST /api/feedback` files the answers. See "Feedback earns credits" below.

## Install the agent skill

Teach a coding agent (Claude Code, Cursor, Codex CLI, Gemini CLI, or a plain `AGENTS.md`) the full workflow in one command: `npx arbiterqa install`, then `npx arbiterqa login`.

- [npm package](https://www.npmjs.com/package/arbiterqa): the `arbiterqa` CLI — `install`, `update`, `login`, `status`, `print-key`, `logout`.

## API

- [Full agent contract](https://api.arbiterqa.com/llms-full.txt): the complete agent skill plus API reference in one file — endpoints, request and response schemas, authentication, screenshots, and the email flow.
- [Live validation catalog](https://api.arbiterqa.com/api/validations): `GET /api/validations` — every validation with id, name, category, severity, execution kind, and configuration requirements. Public JSON.
- [Validation detail](https://api.arbiterqa.com/api/validations/email-unsubscribe-present): `GET /api/validations/:idOrSlug` — judgment prose, parameters, standards membership, agent job snippet.
- [Public HTML catalog](https://app.arbiterqa.com/validations): digest + per-check `/standard` and `/custom` pages (same facts as the JSON assembler).
- [MCP](https://api.arbiterqa.com/mcp): Model Context Protocol endpoint (discovery + estimate/run with Bearer). Transitional HTTP stub: `/api/mcp/tools`.

## Feedback earns credits

We ask questions, and completing what we ask pays credits to your organization's balance — the same balance jobs spend from. Plain HTTP, all `Authorization: Bearer <key>`:

- `GET /api/feedback/questions` — what we are asking your credential right now: `due`, the `setId`, the set's questions (`id`, `text`, and a `shape` object that is the schema for that question's answer), and `rewardCredits`, what completing the set pays. Nothing to ask answers `{ "due": false }`.
- `GET /api/feedback/questions?jobId=…&validationId=…` — the always-open "that verdict looks wrong" set for one check on one job. Questions are worded for the verdict that check actually gave, and `rewardCredits` is derived from what that check cost you on that job. Use it whenever you or your user doubt a pass, a fail, or an error.
- `GET /api/feedback/due` — the cheap poll when you only need to know whether it is worth asking: `{ due, setId, questionCount, rewardCredits }` and nothing else.
- `POST /api/feedback` — the single submission door: `{ "setId": "…", "answers": [{ "questionId": "…", "value": … }], "jobId": "…", "validationId": "…" }`. The ids anchor the answer to real work: both for looks-wrong feedback, `jobId` alone where the set you were served named one. The response echoes the submission and the payout.

Questions are grouped into **sets**, and a completed set is one feedback and one payout. Job-scope sets **queue and are served one at a time** — answer the set you are handed, then ask again and you may be handed another; which sets exist, what they ask and how often they come back are database rows staff edit at runtime, so read the set you are served instead of hard-coding questions. The `shape` on each question is self-describing (`kind` plus its own options, fields or states) and is validated on submission: answer every question in the set exactly once, in the terms its shape states, or the whole submission is refused with the problems listed per question. Feedback is append-only — no edits, no second payout.

Your credential decides what you are asked and how the answer is filed: **a Bearer key files as an agent**, stamped `source: "agent"` server-side from the channel, never read from the body. Agent questions therefore ask for structure rather than prose, and the answers are worth most when they **carry evidence we can check against our own logs** — the `jobId` a claim is about, the endpoint you called, the exact error `code` you received, attempt counts, which response fields you actually used. Rewards are priced by the server and set by staff: read `rewardCredits` rather than assuming an amount — it is the same number the payout writes, and `0` is a legitimate answer.

## Get started

- [Dashboard](https://app.arbiterqa.com): sign up free — an API key is minted automatically on signup; `npx arbiterqa login` does the same from the terminal.