Proposal: Automated Agent Quality Scorecard

[haiyuan-eng-google]

Author: Gayathri Radhakrishnan

Date: April 20, 2026

1. Executive Summary

To evolve the BigQuery Agent Analytics (BQ AA) platform from a reactive diagnostic tool into a proactive fleet-management system. This project implements a systematic "AI Judge" that automatically evaluates and grades every agent interaction, turning raw telemetry into actionable performance KPIs.

2. The "Evaluation Gap"

The current Closed-Loop RCA is an industry-leading tool for deep-diving into why a specific session failed. However, as agent deployments scale, manual root-cause analysis becomes a bottleneck. Organizations need a way to:

• Identify high-performing vs. low-performing agent versions at a glance.
• Monitor global quality trends without manual intervention.
• Flag policy or safety violations in real-time across thousands of logs.

3. Proposed Solution: The Quality Scorecard

I propose building a modular evaluation pipeline that sits on top of the BigQuery event logs. This system will utilize BigQuery’s native AI capabilities (AI.GENERATE) to "grade" sessions across three key pillars:

• Helpfulness Score (1–5): Did the agent resolve the user’s intent effectively?
• Accuracy & Grounding (1–5): Did the agent use the available tools correctly and avoid hallucinations?
• Policy Compliance (Pass/Fail): Did the response adhere to GRC standards (e.g., no PII leakage, authorized tool usage)?

4. Key Features & Flexibility

• Data-Agnostic Design: The evaluation logic is decoupled from specific table names. It can be routed to point at any existing logs table or a fresh "v4" schema, requiring only standard session_id and content fields to function.
• Fleet-Level Benchmarking: Aggregates scores into a "Leaderboard" view, allowing the team to compare performance across different regions, model versions, or system prompts.
• Automated Triage: Automatically flags sessions with a score below a certain threshold for immediate human-in-the-loop (HITL) review.

5. Technical Impact for the Team

• Showcases Platform Capability: Demonstrates the power of using BigQuery as a Governance and Evaluation engine, not just a storage layer.
• Zero Infrastructure Friction: Operates entirely within the BigQuery ecosystem—no external APIs, new IAM permissions, or complex deployments required.
• Modular Architecture: The "Judge" logic can be reused as a template for enterprise customers looking to build their own internal audit trails.

6. Implementation Roadmap

• Phase 1: Develop the SQL-based "AI Judge" prompt and test on a sample dataset.
• Phase 2: Create the aggregated agent_quality_metrics table for reporting.
• Phase 3: Integrate a "Global Agent Health" visualization into the existing dashboard.

[caohy1988]

Read the proposal and audited the current SDK surface (origin/main at 035823a). Quick summary: the intent is sound, and the diagnostic gap this names is real. My concern is that ~70% of the proposed mechanism already exists under different names, and if we ship this without reconciling against the existing surface we'll end up with two overlapping "evaluation" subsystems that users have to choose between.

Below: what already ships, what's genuinely net-new, and suggested reframing so the proposal lands as additive.

---

1. What already ships that this proposal overlaps with

Proposal element	Existing SDK surface	Notes
AI-Judge using AI.GENERATE	src/bigquery_agent_analytics/evaluators.py:728-768 (AI_GENERATE_JUDGE_BATCH_QUERY), LLMAsJudge.correctness/hallucination/sentiment	Already templated; pre-built criteria; batch execution
Pass/fail categorical evaluation	src/bigquery_agent_analytics/categorical_evaluator.py + Client.evaluate_categorical (client.py:~1244)	User-defined categories; strict category validation; AI.CLASSIFY → AI.GENERATE → Gemini-API fallback
Per-session grading of "every agent interaction"	trace_evaluator.BigQueryTraceEvaluator.evaluate_batch + Client.evaluate	Batch orchestration, concurrency control, golden-trajectory matching
Aggregated agent_quality_metrics table	CATEGORICAL_RESULTS_DDL + categorical_views.py with categorical_results_latest dedup view, categorical_daily_counts, categorical_hourly_counts, categorical_operational_metrics	Append-only, dedup by (session_id, metric_name, prompt_version), hourly/daily rollups ready for dashboarding
"Global Agent Health" dashboard	examples/categorical_dashboard.sql + dashboard/ Flask/Jupyter scaffolding	Pre-built queries for daily trend, failure drill-down, alerting thresholds
End-to-end reporting script	scripts/quality_report.py	Runs categorical evaluation + resolves transcripts + per-agent breakdown + Markdown output
Composition of multiple scorers	grader_pipeline.py (Weighted / Binary / Majority strategies)	Already handles "multi-pillar verdict" aggregation
Reproducibility across prompt iterations	prompt_version field on CategoricalEvaluationConfig + SDK telemetry labels (sdk_version, sdk_surface, feature)	Every eval job is already queryable by prompt version in INFORMATION_SCHEMA.JOBS_BY_PROJECT

Concretely: the proposal's Phase 1 ("SQL-based AI Judge prompt on a sample dataset") is already runnable today via python scripts/quality_report.py --app-name my_agent --output-json report.json. Phase 2 ("aggregated quality metrics table") exists as categorical_results_latest. Phase 3 ("Global Agent Health dashboard") exists as the views in categorical_views.py.

---

2. What's genuinely net-new in this proposal

Three pieces that do not have close analogs today:

1. Opinionated three-pillar rubric. Existing categorical evaluation is generic — callers define their own metrics and categories. The proposal nails down a specific, pre-built shape (Helpfulness 1–5, Accuracy/Grounding 1–5, Policy Compliance pass/fail) that users can apply without having to design their own metric schema. That's real product value even if the execution plumbing is reused.
2. HITL triage loop driven by eval scores. Today Client.hitl_metrics() and the HITL event-type plumbing are observational — they count HITL events, they don't flag sessions that should become HITL events. The proposal's "automatically flags sessions with a score below threshold for immediate HITL review" is a new surface: it bridges the eval subsystem and the HITL event stream. Worth building.
3. Cross-dimensional fleet leaderboard. Existing dashboard views aggregate by metric × category × time. They don't rank-order agents by quality across regions, model versions, and prompt versions. The "Leaderboard" framing is the real ask; daily/hourly aggregate views don't deliver it.

Everything else in the proposal is present in the repo, often under a name the proposal doesn't reference.

---

3. Concerns in the current framing

• "Data-agnostic" is aspirational without a schema contract. The proposal says it needs only session_id and content fields. In practice the existing evaluator contracts also require invocation metadata, agent identity, and tool-call events to produce Helpfulness/Grounding scores. The proposal should explicitly cite the minimum schema (ideally as an alias over the existing agent_events contract) rather than hand-wave.
• "Policy Compliance (Pass/Fail)" is under-specified. Which policies? PII regex? Tool-authorization whitelist? LLM-judged hallucination? The proposal's examples (PII, authorized-tool) are three entirely different mechanisms with wildly different cost/latency/false-positive profiles. Shipping Phase 1 without picking one will be a scope-creep trap. I'd recommend defining the v1 policy as just "no PII leakage" via an AI.GENERATE classifier over the final response, with tool-authorization as a deterministic v2 check in CodeEvaluator.
• "Zero infrastructure friction" is understated rather than false. AI.GENERATE usage requires a BigQuery connection, a Vertex AI endpoint, and non-trivial per-call cost. At fleet scale (say 10K sessions/day × 3 pillars × AI.GENERATE invocations) the eval cost becomes a line item. A one-paragraph Fermi estimate in the RFC ("at 10K sessions/day assuming ~$0.001/call, evaluation costs ~$X/month") would protect reviewers from surprise.
• The proposal doesn't reference existing telemetry. Phase 4 of issue Feat: Emit SDK labels on BigQuery jobs (query + load) for usage tracking via INFORMATION_SCHEMA #52 shipped sdk_feature=eval-categorical and sdk_feature=eval-llm-judge labels on every eval job. The proposed "agent version / region / prompt comparison" can lean on these labels + INFORMATION_SCHEMA.JOBS_BY_PROJECT instead of inventing new dimensions — or at least has to explain why those are insufficient. Right now the proposal reads as if that subsystem doesn't exist.
• Python API vs SQL-only. The RFC reads as SQL-first ("SQL-based AI Judge prompt"). The SDK's architectural opinion is Python orchestration + SQL execution: Client.evaluate_categorical(config=...) with Python-native config objects and typed reports. The proposal should either align with that or explicitly argue for a pure-SQL path and accept the incompatibility cost.

---

4. Suggested reframing

I'd be in favor of this shipping if the RFC were rewritten as an additive layer over the existing surface rather than a ground-up subsystem. Concretely:

Phase 1 — Ship the three-pillar rubric as pre-built templates (~200 LOC).

Add src/bigquery_agent_analytics/evaluation_rubrics.py with three CategoricalMetricDefinition factories:

from bigquery_agent_analytics import Client, rubrics

client = Client(project_id=..., dataset_id=...)
report = client.evaluate_categorical(
    config=rubrics.three_pillar(threshold=0.8),  # Helpfulness + Grounding + Policy
    filters=TraceFilter(root_agent_name="my_agent"),
)

Internally this is just a preset over existing CategoricalEvaluationConfig — no new execution plumbing. Ship scripts/quality_report.py --rubric three_pillar in the same PR so the reporting script picks up the template.

Phase 2 — Ship the fleet leaderboard as one view on top of existing aggregates (~50 LOC).

Extend categorical_views.py with a categorical_fleet_leaderboard view that ranks by the scheme/agent dimension combining sdk_feature labels from INFORMATION_SCHEMA.JOBS_BY_PROJECT with categorical_results_latest. No new table; a view keyed on (root_agent_name, sdk_version, prompt_version).

Phase 3 — Ship HITL triage as a small new surface (~150 LOC).

New Client.triage_low_score_sessions(threshold, window) that queries categorical_results_latest, filters by threshold, and writes to a hitl_triage_queue table with a well-defined row contract (session_id, metric, score, flagged_at, reason). This is the genuinely novel bit and deserves its own design conversation — schema, retention, idempotency on re-scoring.

Phase 4 — Docs.

One page tying the above together under docs/quality_scorecard.md that says: if you want the out-of-the-box three-pillar scorecard, use rubrics.three_pillar(); if you want custom metrics, keep defining your own CategoricalMetricDefinition; if you want to ship low-score sessions to human review, use triage_low_score_sessions.

---

5. Verdict

Intent: strong. Moving from reactive RCA to proactive fleet management is the right direction, and the three-pillar rubric + HITL triage + leaderboard are the right three deliverables.

Current RFC: needs a rewrite. As written it reads as if the SDK had no eval subsystem, when ~70% of the proposed plumbing ships today. A reviewer who doesn't already know the repo would approve building a parallel system; a reviewer who does know the repo will ask to merge the work into the existing surface.

Recommended next step: rewrite §3 "Proposed Solution" and §6 "Implementation Roadmap" to reference CategoricalEvaluator, categorical_views.py, quality_report.py, and the Phase-4 telemetry labels explicitly, and scope the new work to (a) pre-built rubric templates, (b) fleet leaderboard view, (c) HITL triage pipeline. Happy to pair on that draft if useful.

[caohy1988]

Follow-up note after re-checking latest origin/main (035823a) against the current issue text and the existing review:

Two nuances worth pinning explicitly:

1. The proposed fleet leaderboard is not quite as cheap as “one more view on the existing aggregates.” The persisted categorical results schema today only carries session_id, metric result fields, endpoint, execution_mode, prompt_version, and created_at (src/bigquery_agent_analytics/categorical_evaluator.py). The stock views only aggregate by metric/category/time/endpoint (src/bigquery_agent_analytics/categorical_views.py). There is no row-level root_agent_name, region, or similar fleet dimension in the results table, so a leaderboard by agent × region × prompt version needs either:

• a join back to traces on session_id, or
• an explicit decision to persist those dimensions during evaluation.

So I still agree the leaderboard is genuine net-new, but I would not frame it as a ~50 LOC view unless the RFC first chooses one of those two dimensionality strategies.

2. The “data-agnostic” claim is even weaker than my first comment suggested. The current evaluator SQL assumes the SDK trace/event contract pretty directly: event_type, timestamp, agent, and specific JSON paths under content are hard-coded into transcript extraction, and app scoping depends on attributes.root_agent_name (categorical_evaluator.py, trace.py). So “Phase 1 is runnable today” is true for logs already normalized to the SDK event schema, but not yet for arbitrary logs with just session_id and content.

Related minor clarification: the Phase-4 telemetry labels (sdk_feature, etc.) are still useful provenance, but they live on BigQuery jobs, not on each scored session row, so they do not by themselves solve the fleet-dimension / leaderboard problem.

Net: I still agree with the main recommendation to reframe this as additive over the existing eval surface. I’d just make the leaderboard phase explicitly include a dimensionality/storage decision, and I’d tighten the schema contract language so reviewers don’t read “data-agnostic” more broadly than the current code supports.

[caohy1988]

One more technical note that complicates my earlier "Phase 1 — just ship rubrics.three_pillar() as a preset CategoricalMetricDefinition" suggestion.

The SDK has two mutually incompatible shapes for per-session LLM judgments today:

1. CategoricalMetricDefinition (src/bigquery_agent_analytics/categorical_evaluator.py:88-99) — strictly unordered named categories. name: str + definition: str. No ordinal relation, no numeric score. Persisted categorical_results table is keyed on (session_id, metric_name, category, prompt_version). Dashboard views count categories, they don't average them.

2. LLMAsJudge + SessionScore (evaluators.py:436-642, evaluators.py:…) — numeric 0–1 scores per criterion, with threshold for pass/fail. No persistence — results are returned in EvaluationReport but not written to a BigQuery table by default.

The proposal's rubric doesn't fit cleanly into either:

Pillar	Proposal shape	Nearest existing shape	Gap
Helpfulness	1–5 Likert	LLMAsJudge.sentiment (0–1 float)	Numeric, but not persisted; no Likert semantics
Accuracy & Grounding	1–5 Likert	LLMAsJudge.correctness / hallucination (0–1 float)	Same gap
Policy Compliance	Pass/Fail	CategoricalMetricDefinition with {pass, fail} categories	Categorical works; Likert ones don't

Consequences for the Phase 1 preset-template path:

• If you represent Likert 1–5 as CategoricalMetricDefinition with five string categories \"1\", \"2\", \"3\", \"4\", \"5\", the SDK treats them as unordered strings. Aggregations like "mean helpfulness per agent-version" require manual CAST(category AS INT64) in every caller SQL. Dashboard views that currently count categories (categorical_daily_counts) would show 1/2/3/4/5 as separate non-ordered buckets. That's not what a "Scorecard" reads like.
• If you use LLMAsJudge for the Likert pillars and CategoricalMetricDefinition for Policy, the proposal's "aggregated agent_quality_metrics table" needs to union two different result shapes, and the fleet-leaderboard phase has to reconcile them. Non-trivial.
• If you shift the proposal to 3-bucket categorical across all pillars (e.g. helpful/partial/unhelpful, grounded/ungrounded/no_tool_needed, compliant/violation), you actually converge with the existing scripts/quality_report.py shape, which already ships those first two. That's the cleanest path — but it drops the "1–5" granularity the proposal explicitly promises.

This is an architectural decision the RFC should call out explicitly, not a drafting nit. Either:

Option A: Accept 3-bucket categorical everywhere, rename the pillars, ship as presets over CategoricalMetricDefinition. Lowest new LOC; converges with existing quality_report.py.

Option B: Add a new OrdinalMetricDefinition (1–N Likert) with an ordered-category schema and aggregations like AVG(score). Higher LOC, cleaner product shape, but now you own a third eval subsystem for N years.

Option C: Keep Likert, but deliver it through LLMAsJudge and pay the cost of persisting judge results to a new table so the leaderboard can aggregate over them. Probably the highest LOC and the most entangled with the existing numeric-eval path.

I'd weakly favor Option A for v1 and defer Option B to a v2 if adopters actually need 1–5 granularity. But the RFC needs to pick.

(Also: my earlier follow-up already noted that the leaderboard needs a dimensionality decision — root_agent_name / region / prompt_version aren't in categorical_results today. Option A doesn't change that; Options B and C would need to decide those dimensions fresh in their new schema.)

[haiyuan-eng-google]

@Gayathri0105RK PTAL

[Gayathri0105RK]

Thanks for the detailed feedback. I’ve reviewed the existing surface and I’m reframing the proposal to be strictly additive. I want to avoid any duplication and lean entirely on the current CategoricalEvaluator and categorical_views.py architecture.

Updated Technical Plan:

Rubric Implementation: I will implement the three-pillar templates (Helpfulness, Accuracy, and Compliance) using Option A (3-bucket categorical). I’ll house these factories in src/bigquery_agent_analytics/evaluation_rubrics.py to keep the execution logic unified with the existing SDK.

Dimensionality: I will update the evaluation logic to persist root_agent_name and region within the categorical_results schema. This handles the 'Leaderboard' requirement without adding join-complexity to the views.

HITL Triage Pipeline: I will focus the new engineering work on Client.triage_low_score_sessions(). This will serve as the bridge to flag specific sessions for the hitl_triage_queue based on the scores generated by the existing evaluators.

I'll start with a small PR for the rubric templates to ensure the schema alignment is correct before moving to the triage pipeline. Let me know if this aligns with your expectations!

[caohy1988]

Thanks for the reframe. This is the right direction: Option A / 3-bucket categorical, additive rubric factories on top of the existing evaluator, and HITL triage treated as the genuinely new workflow. Before the first PR, I would pin the following so the implementation does not split the existing evaluation surface.

1. Reuse the existing metric vocabulary

scripts/quality_report.py already defines two of the proposed pillars as categorical metrics:

Proposal pillar	Existing metric	Existing categories
Helpfulness	response_usefulness	meaningful / partial / unhelpful
Accuracy & Grounding	task_grounding	grounded / ungrounded / no_tool_needed

The evaluation_rubrics.py factories should return these metric names and category strings verbatim. Display labels can say "Helpfulness" and "Grounding", but the persisted values should not become a second vocabulary like helpful/inaccurate; otherwise the existing JSON reports and dashboard views fork into parallel tile sets.

Policy Compliance is the only genuinely new metric. Suggested v1 shape: policy_compliance with compliant / violation. If severity is needed later, add it deliberately rather than overloading the v1 category names.

2. Keep the rubric PR schema-free

root_agent_name and region are useful for the leaderboard, but they are a persistence/schema change, not part of the rubric-template PR. categorical_results currently has only:

session_id, metric_name, category, justification, passed_validation,
parse_error, raw_response, endpoint, execution_mode, prompt_version,
created_at

Adding dimensions means:

• existing tables need ALTER TABLE ADD COLUMN or explicit schema reconciliation before inserts start writing the new fields;
• historical rows will be NULL, so views need an explicit policy such as COALESCE(root_agent_name, 'unknown');
• root_agent_name should be extracted per session from the trace attributes, not only copied from TraceFilter.root_agent_name, because unfiltered fleet evals may include multiple root agents;
• region needs a source of truth before it is persisted. Today it is not a first-class SDK trace attribute. If it means BigQuery dataset/evaluation location, consider naming it evaluation_region; if it means agent deployment region, the SDK needs a way to receive it.

This should be its own PR because it affects every existing evaluate_categorical(..., persist=True) caller and the stock categorical views.

3. Be careful with "low score" under Option A

With Option A, categories are unordered strings; there is no numeric score. So Client.triage_low_score_sessions() needs either:

• a category policy, e.g. flag response_usefulness IN ('partial', 'unhelpful'), task_grounding = 'ungrounded', policy_compliance = 'violation'; or
• a renamed API that makes the categorical behavior explicit, e.g. triage_categorical_sessions(...).

I would not make callers infer severity from category strings ad hoc. Put the default triage mapping in the rubric module and allow overrides.

4. Write a short HITL triage design note before implementation

The queue is the new product surface, so it is worth answering these before code:

• Queue schema: at minimum session_id, metric_name, category, prompt_version, root_agent_name, flag_reason, justification, flagged_at, plus region if PR Revamp README, enhance documentation navigation, and fix CI #2 defines it.
• Idempotency: BigQuery will not enforce a primary key, so decide whether the write uses MERGE keyed by (session_id, metric_name, prompt_version) or remains append-only with a latest-view pattern.
• Lifecycle: define whether rows get resolved_at / resolution_status, move to a resolution table, or remain an append-only audit log.
• HITL event relationship: decide whether this queue is the only signal or whether triage also emits/derives a HITL event compatible with the existing HITL_* event-family views.

5. Suggested PR order

1. Rubric factories only: evaluation_rubrics.py with factories returning CategoricalMetricDefinitions, using the existing response_usefulness and task_grounding definitions. Add tests that assert exact metric/category names. No DDL, no views, no client persistence changes.
2. Persisted dimensions: add root_agent_name and only add region after its definition is pinned. Include migration/schema reconciliation, evaluator insert wiring, and view updates.
3. categorical_fleet_leaderboard view: build on the dimensions from PR Revamp README, enhance documentation navigation, and fix CI #2.
4. HITL triage: implement the queue/API after the design note answers the schema, idempotency, lifecycle, and event-relationship questions.

6. Docs note

The original proposal promised 1-5 Likert for Helpfulness and Accuracy. Option A changes v1 to 3-bucket categorical verdicts. Call that out in the rubric docstrings and docs/quality_scorecard.md, e.g.:

v1 emits 3-bucket categorical verdicts that align with the existing categorical dashboards; 1-5 Likert scoring is deferred pending user demand.

With those pins, the first PR should stay small and reviewable.