Analytics
Monitor test performance, track trends, and identify issues.
Overview
The Analytics dashboard provides visibility into:
- Run Trends: Test executions over time
- Pass/Fail Rates: Success metrics by suite or assistant
- Judge Scores: AI evaluation scores and distributions
- Latency Metrics: Response times (avg, p50, p95, max)
Using Analytics
- Navigate to Analytics in the sidebar
- View the dashboard with default 7-day range
- Use date filters for longer periods
- Click into specific metrics for details
Key Metrics
Test Runs
- Total runs executed
- Success vs. failure breakdown
- Run duration trends
Pass Rate
- Overall pass percentage
- Trends over time
- Breakdown by suite or assistant
Latency
- Average response time
- P50, P95, and P99 percentiles
- Max response times
Judge Scores
- Score distribution
- Threshold breaches
- Comparison across tests
Tips
- Monitor latency trends to catch performance regressions
- Track pass rates to identify flaky tests
- Use date ranges aligned with deploy cycles
- Drill into failures to find root causes
Last updated on