AssayCheck
Acceptance testing
for LLM APIsSofia,
Bulgaria

Independent acceptance testing for LLM APIs

Is your API
telling the truth?

ASSAYCHECK / PROVIDER ASSURANCE

We test what it
actually delivers.

A successful response is not proof of a working contract. We check what your route honours, what it silently ignores, and what changed.

Examine a sample report Reviews, monitoring & enterprise deployment

OpenAI-compatible · Anthropic Messages · OpenAI Responses

Inside your network. Evidence in your hands.

A ROUTE UNDER EXAMINATION

Standard run / illustrative replay

ASSAYCHECK / STANDARD RUNIntermediary A · platform-team.v1
resolving endpointPASS
TLS okPASS
no redirectsPASS
availabilityPASS
streamingPASS
tool_loopPASS
json_modePASS
structured_outputWARN
max_tokensFAIL
unknown_parameterFAIL
error_handlingWARN
usage_sanityWARN
exact_replyPASS
json_shapePASS
tool_selectionPASS
refusal_boundaryPASS
long_contextNOT TESTED
REFERENCE CONSISTENCYe-value / log scale
Sequential reference testLogarithmic e-value over 300 samples. Non-monotonic curve peaks and ends at 1.43, below the rejection threshold of 100.0.11101001000reject threshold0100200300paired samples

peak 1.43 — not rejected

BLOCKERS2 required contract checks failed.
Reference consistency: not rejected.
requests321
spend$0.044
elapsed18.7 s
phasecomplete
Illustrative replay · measured results, compressed timeline.
Reference consistency does not override contract failures.
quick run24 requests
standard run321 requests · ≈$0.04
protocolschat · messages · responses
runsinside your network

The gap between a label and a contract

A successful response proves very little.

01 /

The model field is not evidence.

On an aggregator, the name in the response is a line in an alias map. It costs nothing to write and nothing to change. Self-identification is worse — models will confidently state a training cutoff that isn’t theirs.

02 /

Ignored means 200 OK.

A limit that isn’t applied, a parameter accepted and discarded, an unknown field swallowed without complaint. Nothing raises an error. You find out from the invoice, or from a user.

03 /

Benchmarks answer another question.

A leaderboard ranks models in the abstract. You need to know whether this route, today, under your parameters, still meets the requirements you signed off on.

IT DOES

Acceptance testing against your team’s requirements profile. A fixed battery, explicit limitations, and a comparable record of what changed. Where applicable, it names the nearest reference and the distance to it.

IT DOES NOT

Identify the “real model”, rank models on a leaderboard, or certify a provider. AssayCheck reports to the customer who runs the test. It does not publish supplier scores.

The deliverable

One run. One accountable record.

The same instrument.
Three different outcomes.
Uncertainty stays in the report.

route
Intermediary A · chat/completions
model requested
gpt-4o
mode
standard · 321 requests
profile
platform-team.v1
spend
$0.044 of $0.50 cap
report
sha256 signed
BLOCKERS

2 required checks failed

01API contract

availabilityPASSEndpoint reachable, TLS valid, no redirects followed.
streamingPASSReal SSE. A 230-character probe arrived in 14 content deltas — not a buffered response replayed as a stream.
tool_loopPASSTool call issued, result accepted, final message produced.
json_modePASSValid JSON returned on every attempt.
structured_outputWARNSchema honoured, but an optional field was emitted that the schema does not declare.
max_tokensFAILLimit set to 128. Completions returned 681–942 tokens on all five models tested. Reproduced across three runs.
unknown_parameterFAILA deliberately misspelled, wrongly typed parameter was accepted with 200 and silently discarded.
error_handlingWARNUnknown model name returns 503 rather than a 4xx. Marked optional in this profile.
usage_sanityWARNUsage object carries one field outside the documented schema.

02Task cases

exact_replyPASSInstruction followed literally at an 8-token budget.
json_shapePASSRequired keys present, types correct.
tool_selectionPASSCorrect tool chosen over the decoy.
refusal_boundaryPASSRefused as expected; no policy leakage into the response body.
long_contextNOT TESTEDOutside this profile's declared requirements.

03Reference consistency

outcomeNOT REJECTEDThe route was not rejected against the approved reference — but comparison strength is reduced, because the model identifier is self-reported by the provider.
e-value peak1.43Rejection threshold is 100. The sequential test is anytime-valid: it may stop early, and it did not.
aggregate distance0.143Jensen–Shannon divergence over 300 paired samples, zero errors.
referencetier AEnrolled directly from the first-party vendor, generation 2, 411 observations, validity 1.00.

04Performance

latency median2.09 sMeasured from the machine that ran the test — a client-side number, not the provider's.
latency p958.86 sLong tail concentrated in the fingerprint battery, n=300.
time to first chunk3.0 sStreaming path.

05Usage & cost

requests321Every attempt logged, including retries.
cost$0.044Priced from the profile's own rate table, not the provider's invoice.
reported ids2 distinct316 responses carried one snapshot identifier, 3 carried an older one. The route occasionally answers from a different snapshot.
HISTORY / COMPARABLE RUNCompared with the previous run of this profile: reference consistency moved from INCONCLUSIVE to NOT REJECTED under the same conditions. Two contract checks are unchanged failures.

Calibration / first-party references

Distance needs a frame of reference.

Measured bands, not an identity claim.

ComparisonDistanceReading
Same route, repeated against itself0.078Own noise floor
Intermediary vs first-party, same model0.089Within noise — same source
Neighbouring alias of the same model0.093Within noise
Different models, same vendor0.143 – 0.192Distinguishable
Deliberate substitution, sibling modelrejectedCaught at 245 requests, e-value 137.7

Why this matters / independent research

Evidence worth taking to the meeting.

Adoption, reliability and supplier verification are different questions.

Cover of The case for independent API acceptance

Research note 01 / September 2026

The case for independent API acceptance.

A concise review of adoption, reliability and verification research. Sources, practical implications and the limits of each claim.

Download the brief PDF · 4 pages ↓
Cover of Supplier acceptance: two required checks failed

Sample deliverable / Standard run

A report built for the decision file.

The supplied example run, laid out for review and sharing. Contract failures, reference consistency, performance, cost and history.

Download the report PDF · 4 pages ↓
Enterprise adoption / 2025

Agents are entering the workflow.

In McKinsey’s 2025 survey, 23% of respondents said their organisations were scaling an agentic AI system in at least one business function; another 39% were experimenting. This measures reported adoption, not successful deployment.

McKinsey · State of AI 2025 ↗
Reliability research / 2026

Accuracy is only part of reliability.

A study of 15 models across two benchmarks found that recent capability gains brought only small improvements in reliability. The authors distinguish consistency, robustness, predictability and safety from a single success score.

Rabanser et al. · ICML 2026 ↗
API auditing research / 2025

Model identity has verification limits.

Research on model substitution finds that output-based statistical checks can miss subtle substitutions and that log-probability methods face inference nondeterminism. The authors propose hardware-backed verification. This supports keeping identity claims bounded; it is not validation of AssayCheck’s method.

Auditing Model Substitution in LLM APIs ↗

What this means for acceptance: a model benchmark cannot replace checks on the route your system actually calls. AssayCheck tests that supplier boundary; it does not certify the end-to-end safety or reliability of an agent.

Industry perspective: ITPro on the emerging role of managed agent services · 10 September 2026 · contributed opinion.

Independent sources provide context. Their authors have not evaluated or endorsed AssayCheck.

Built for the acceptance workflow

From a first check to a release gate.

Requirements profiles

Encode what your team actually needs as a versioned JSON file. The verdict is measured against that, not a generic checklist.

History and diff

Every run is stored with a comparability hash. The next run is compared against the last comparable one, so “it changed” means something.

CI gates

Export to JSON, HTML, Markdown, JUnit and SARIF. Fail the pipeline on blockers, attach the report to the merge request.

Scheduled monitoring

Re-run on a cadence and fire a webhook when a status changes. Substitution is often per-request or load-dependent, so a single run is a snapshot, not a guarantee.

Hard budget caps

Limits on requests, spend, wall time and concurrency. A run that hits a cap stops and reports partial results rather than silently truncating.

Secrets never travel

Keys come from the environment, are redacted from logs, reports and history. AssayCheck executes inside your network.

Protocol adapters

chat/completions, Anthropic Messages and OpenAI Responses behind one interface.

Reference provenance

Every reference is tiered by how it was obtained, with a manifest, checksum, expiry and generation history. An intermediary is never treated as ground truth.

Who we work with

For the team signing off on the route.

Teams buying through intermediaries

You chose an aggregator for price and coverage. You still need to know what arrives under each label, and when it changes.

Platforms with their own SLA

You resell inference or embed it in a product. Your customers ask what you serve them, and “our supplier says so” is not an answer you can sign.

Procurement and vendor review

You need the same battery run against three candidates and a document you can put in a file.

Reviews / monitoring / enterprise software

You can run the checks.
We help you make the call.

Commission a review.
Keep a route under observation.
Deploy the software with support.

A test run is a starting point. Choose a scoped acceptance review, ongoing monitoring, or a supported deployment inside your own infrastructure.

01 / One-off engagement

Provider acceptance, handled.

For a new supplier, a procurement review, or a route you need to sign off on.

  • Define required checks and blockers with your team.
  • Run candidates against a comparable requirements profile.
  • Review contract failures and ambiguous reference results.
  • Deliver an acceptance report with limitations and questions for the supplier.

You pay for: a scoped review and an evidence-backed recommendation for your team’s decision.

02 / Recurring engagement

Keep the acceptance current.

For routes already in production, where yesterday’s result is not enough.

  • Agree a monitoring cadence and an explicit test budget.
  • Compare each run with the last comparable baseline.
  • Review meaningful changes and notify your team.
  • Maintain requirements, checks and reference provenance as the integration evolves.

You pay for: continued monitoring and interpretation of changes that affect your requirements.

03 / Enterprise deployment

Your infrastructure.
Our software and support.

For organisations that need to own the acceptance workflow and operate AssayCheck inside their network.

  • Deploy AssayCheck in the customer’s environment.
  • Configure requirements profiles and integrate checks into CI and scheduled workflows.
  • Help your team operate the software and interpret results.
  • Provide software updates and technical support under an agreed scope.

You pay for: implementation, a software licence, and ongoing support. Deployment scope, support terms and service levels are agreed with your team.

Why bring us in?

You can build and run your own checks. The continuing work is keeping the battery useful, the references traceable, the runs comparable, and the conclusions within what the evidence supports. AssayCheck brings those pieces into one acceptance workflow — including an explicit “not enough data” when the result cannot support a decision.

Inspect the example deliverable ↓

Enterprise / Agent infrastructure & advisory

Know where the AI budget goes.
Make it work harder.

Consultants and agents.
Working inside your organisation.
With an agreed scope of access.

Our enterprise offering combines deployable agent infrastructure with hands-on consulting. We examine how your organisation uses AI, connect spend to actual workflows, and identify where a different route, configuration or process could deliver better value.

The work starts with your approved data sources and access boundaries. Proposed changes are reviewed with your team before implementation. Savings are assessed against an agreed baseline, not promised in advance.

Configured around your organisation

AssayCheck is customised to your use cases, tools, API routes and acceptance criteria. Requirements profiles, task cases, checklists and integrations are adapted to the workflows your teams actually run. The implementation scope is agreed with you.

Beyond the endpoint test

API acceptance establishes the supplier baseline. The advisory engagement extends the review to usage, cost, resource allocation and key management.

01

Make spend attributable.

Map available usage and billing records to teams, applications and workflows. Identify unowned budgets, costly retries and usage that is difficult to explain.

02

Test the economics of each scenario.

Review model and route selection, token budgets, caching opportunities and repeated work. Compare changes against quality requirements before recommending a cheaper configuration.

03

Review the exposure of API keys.

Work with your security team to review key inventory, ownership, scope and rotation practices. Prioritise suspected exposure and uncontrolled distribution for remediation.

04

Keep decisions measurable.

Deliver a prioritised action plan, named owners and a way to compare before and after. Continue monitoring the agreed cost and acceptance indicators.

Engagement output

Usage and spend map · Supplier acceptance baseline · Prioritised optimisation plan · Access and key-management findings · Follow-up measurement

Selected engagement / Banking sector

Client confidential

A clearer supplier landscape.
A lower monthly bill.

For a banking-sector organisation, our team brought supplier oversight, API optimisation and recurring agent-assisted checks into one operating process.

The starting point
A fragmented supplier landscape and API usage that needed a more structured review of cost and data-exposure risks.
The intervention
We structured the contractor inventory, reviewed usage scenarios, addressed identified risk cases and optimised API workflows. Agents were integrated to support verification and the recurring execution of checklists.
The operating change
The organisation gained a repeatable review process for suppliers and scenarios, with regular checks supported by agents and project consultants.

An anonymised client engagement, published with permission. Figures are reported by the project team. The annual figure is a 12-month extrapolation of approximately $50,000 in monthly consumption savings, not a measured full-year result or a guarantee for other organisations.

Start with your requirements

Bring an endpoint.
Leave with evidence.

Run inside your network. Keep the keys in your environment. Review the report before you connect the route.

Documentation — coming soonRepository — coming soon
DEPLOYMENT / USAGE EXAMPLE
# Install the package supplied for your deployment
assay run --profile platform-team.v1 --provider provider.json --mode quick
provider.json · illustrative adapter config
{
  "protocol": "chat/completions",
  "base_url": "https://your-endpoint.example/v1",
  "api_key_env": "ASSAY_API_KEY",
  "model": "your-model-alias"
}