Announcing LLMJury: statistically defensible A/B testing for LLM products

2026-07-01 · The LLMJury team

Every LLM team we talk to ships prompt changes the same way: try it on a few examples, squint, and merge. "It feels better" is not an eval strategy — and offline evals, however careful, don't tell you what happens with real users, real traffic, and real money.

LLMJury is live A/B testing built specifically for LLM products:

• Deterministic assignment. A frozen MurmurHash3 bucketing hash — identical across our Python, TypeScript, and Java SDKs — maps each user to a variant locally, with no network call on the hot path and no reshuffling when you edit an experiment.
• LLM-as-judge scoring. Built-in quality, safety, and relevance metrics, plus custom metrics you define as a rubric + output schema. Judged samples are cached, versioned, and hard-budgeted.
• Statistics you can defend. Every experiment is SRM-gated (a corrupted split halts analysis at p < 0.001 instead of lying), each metric auto-routes to the correct test, and all p-values are Benjamini–Hochberg FDR-corrected — with the method, window, and permutation count recorded on every result.

Wrap your model call, track an exposure, and your first experiment is live:

import llmjury

client = llmjury.Client("pk_your_key")
variant = client.assign("onboarding_copy", user_id) or "control"
client.track("exposure", {"experiment_id": "onboarding_copy", "user_id": user_id, "variant": variant})

Every plan starts free — no credit card. Start here, or book a demo if you'd like a tour.