Skip to content

Frequently asked questions

Every answer has its own link — hover one and click the to copy it.

Getting started

What is LLMJury?
LLMJury is a statistical experimentation platform for LLM products: it assigns users to prompt or model variants, scores a sample of outputs with an LLM-as-judge, and computes SRM-gated, FDR-corrected results over the live experiment window.
Is LLMJury a jury of models?
No — despite the name, LLMJury scores each sampled output with a single LLM-as-judge against your rubric: one model, one score, no panel and no vote. The name is about the verdict at the end, which is statistically gated rather than a matter of opinion. An LLM jury is a real and separate technique, explained at /llm-as-a-jury along with when a panel is worth its cost and how to feed a panel score you compute yourself back in as a business metric.
What does "beta" mean for LLMJury?
LLMJury is in open beta: the statistical engine and SDKs are production-grade and tested, but the product is young — expect occasional rough edges, fast fixes, and frequent releases. Paid plans work normally during beta, pricing will not change under you mid-subscription, and beta feedback goes straight to the founders and shapes the roadmap, which is published.
Do I need a credit card to start?
No. Every plan starts free with no credit card — the Free plan includes all three SDKs (Python, TypeScript, Java), 50k events per month, and unlimited experiments.
How do I contact you?
Email hello@llmjury.com for sales and general questions or support@llmjury.com for product support — we reply within one business day. You can also book a live demo from the contact page.
How is LLMJury different from Langfuse, LangSmith, Braintrust, or Statsig?
Those tools are strong at tracing, offline evals, or general feature flags. LLMJury runs online LLM A/B tests on live traffic with a statistically defensible verdict (SRM-gated, FDR-corrected), plus prompt versioning wired into the same loop. Langfuse and LangSmith focus on tracing and offline/dataset evals; Braintrust on evals and datasets; Statsig is a general experimentation and feature-flag platform. LLMJury is purpose-built for LLM apps, with an LLM-as-judge and prompt variants and versioning as first-class objects.
I already have months of logged model calls — can I use them?
Yes. POST /v1/events/import takes historical exposures, model calls, and conversions as CSV or JSON, and they flow through the same pipeline as live events: judged, rolled up, SRM-checked, FDR-corrected. Original timestamps are preserved, so the analysis window reflects when things actually happened, and dedup is on your own event_id so re-running an import is safe. The limit worth knowing: an import gives you a baseline and the real distribution of your metrics — it cannot manufacture a randomised comparison out of data that was never split. Import requires a secret key; publishable keys are rejected. See the quickstart.

Integration

Does LLMJury do prompt management and versioning?
Yes. Prompts live in LLMJury, separated from application code, so you edit and deploy new prompt versions without touching the repo or redeploying. Every version has full history and an audit trail of who changed what and when, with side-by-side compare and one-click rollback. The version you edit is the variant you A/B test.
How does LLMJury assign users to variants?
With a deterministic MurmurHash3-based bucketing hash, identical across the Python, TypeScript, and Java SDKs and the backend: the same user always gets the same variant, with no network call on the hot path.
How much latency does the LLMJury SDK add?
Zero added latency on the request path. Variant assignment is pure local computation using deterministic MurmurHash3 bucketing — no network call. LLM-as-judge grading runs after the fact on sampled outputs, off the user request path, so it never slows a response.
Does LLMJury see my prompts and my model’s responses?
That depends on which integration path you use, not on your plan. If you use wrap() (or intercept_model_call), the prompt your application sent and the text your model returned are attached to each model_call event and stored raw for your plan’s retention window — whether or not LLM-as-judge grading is switched on. If you use the key-only path — get_prompt / get_variables / assign to resolve the variant, plus track(…) for your business outcome — no model text ever reaches us; you keep prompt versioning, deterministic assignment, and business metrics, and give up judge-scored quality metrics, which need the text to score. Hash-only storage and per-org redaction are a known gap and not shipped. The full breakdown is on the security page.

Pricing & plans

What can I measure on the Free plan?
Everything the SDK captures without extra model calls: response latency, token cost, and business outcomes such as conversions or revenue events. Automatic quality grading with an LLM-as-judge starts on the Pro plan, because judge grading runs real model inference.
How many teammates can I invite?
Plans include seats for your whole team on one shared organization: Free has 5 seats, Pro has 100, and Business has unlimited seats. You invite teammates by email from the dashboard and everyone works on the same experiments.
Is there an Enterprise plan?
Enterprise — custom limits, SSO, a security review, and a dedicated contact — is coming soon. You can register interest on the pricing page and we will contact you when it opens; early registrants help shape what it includes. Where it sits relative to everything else is on the roadmap.
What happens when I hit the 50k free-tier limit?
When you pass 50k events in a month, the SDK stops pulling new variants and falls back to the in-code default prompt you passed, so your app keeps serving and nothing breaks. Experiments and prompt versions are not deleted, finalized results persist, and the allowance resets at the start of the next month.
Does my experiment stop when my plan retention ends?
No. The analysis window is the experiment’s own duration and is never artificially capped by tier. Retention limits only how long raw events are stored; finalized results are snapshotted and persist.

Statistics

What is an SRM check and why does it matter?
SRM (sample ratio mismatch) means the observed traffic split diverges from the configured allocation, which invalidates results. LLMJury runs a chi-squared SRM check and halts analysis when p < 0.001 instead of showing untrustworthy numbers.
Why are p-values FDR-corrected?
Testing many metrics at once inflates false positives. LLMJury applies Benjamini–Hochberg false-discovery-rate correction across all metric comparisons and reports both the raw and corrected p-values; significance is judged on the corrected one.
Which statistical tests does LLMJury use?
Each metric category auto-routes to the statistically correct default from a deliberately minimal set — permutation tests with bootstrap CIs for continuous, ordinal, percentile, and count metrics (correct for any distribution shape, including heavy-tailed latency and cost), and the closed-form two-proportion z-test for binary rates, with an automatic permutation fallback at small samples. Welch's t-test, Mann–Whitney U, and Fisher's exact are available as advanced per-metric overrides, with a recorded warning.
Can I define my own quality metrics?
Yes. A custom metric is a natural-language rubric plus a structured output schema for the judge. Metrics are versioned, sampled, cached, and protected by a hard judge budget.
Can I trust an LLM to grade my LLM?
The judge grades against your rubric rather than a vibe, scores only a sample under a hard budget, and a verdict is reported only once the numbers clear an SRM check and FDR correction. You can read the graded outputs yourself and define your own metrics.