- What is LLMJury?
- LLMJury is a statistical experimentation platform for LLM products: it assigns users to prompt or model variants, scores a sample of outputs with an LLM-as-judge, and computes SRM-gated, FDR-corrected results over the live experiment window.
- Do I need a credit card to start?
- No. Every plan starts free with no credit card — the Free plan includes all three SDKs (Python, TypeScript, Java), 50k events per month, and unlimited experiments.
- What can I measure on the Free plan?
- Everything the SDK captures without extra model calls: response latency, token cost, and business outcomes such as conversions or revenue events. Automatic quality grading with an LLM-as-judge starts on the Pro plan, because judge grading runs real model inference.
- How many teammates can I invite?
- Plans include seats for your whole team on one shared organization: Free has 1 seat, Pro has 10, and Business has 50. You invite teammates by email from the dashboard and everyone works on the same experiments.
- Is there an Enterprise plan?
- Enterprise — custom limits, SSO, a security review, and a dedicated contact — is coming soon. You can register interest on the pricing page and we will contact you when it opens; early registrants help shape what it includes.
- How do I contact you?
- Email hello@llmjury.com for sales and general questions or support@llmjury.com for product support — we reply within one business day. You can also book a live demo from the contact page.
- How does LLMJury assign users to variants?
- With a deterministic MurmurHash3-based bucketing hash, identical across the Python, TypeScript, and Java SDKs and the backend: the same user always gets the same variant, with no network call on the hot path.
- What is an SRM check and why does it matter?
- SRM (sample ratio mismatch) means the observed traffic split diverges from the configured allocation, which invalidates results. LLMJury runs a chi-squared SRM check and halts analysis when p < 0.001 instead of showing untrustworthy numbers.
- Why are p-values FDR-corrected?
- Testing many metrics at once inflates false positives. LLMJury applies Benjamini–Hochberg false-discovery-rate correction across all metric comparisons and reports both the raw and corrected p-values; significance is judged on the corrected one.
- Can I define my own quality metrics?
- Yes. A custom metric is a natural-language rubric plus a structured output schema for the judge. Metrics are versioned, sampled, cached, and protected by a hard judge budget.
- Which statistical tests does LLMJury use?
- Each metric category auto-routes to the statistically correct default from a deliberately minimal set — permutation tests with bootstrap CIs for continuous, ordinal, percentile, and count metrics (correct for any distribution shape, including heavy-tailed latency and cost), and the closed-form two-proportion z-test for binary rates, with an automatic permutation fallback at small samples. Welch's t-test, Mann–Whitney U, and Fisher's exact are available as advanced per-metric overrides, with a recorded warning.
- Does my experiment stop when my plan retention ends?
- No. The analysis window is the experiment’s own duration and is never artificially capped by tier. Retention limits only how long raw events are stored; finalized results are snapshotted and persist.