Feat: Runtime entity resolution primitives — OntologyRuntime, concept index, EntityResolver protocol (design proposal — feedback wanted)

[caohy1988]

Status: Design proposal. Not yet implemented. Comments welcome — especially on the "Open questions" section at the bottom and on the agent-vs-SDK boundary.

Goal

Today the SDK's ontology pipeline stops at DDL compilation (gm compile emits CREATE PROPERTY GRAPH + table scaffolding). Runtime — the point where an agent receives a user/client input like format_ids: ["display_static"] or geo: ["San Francisco-Stockton-Modesto"] and needs to resolve it against a declared ontology — is left entirely to the application layer.

Feedback from a production user building agentic media buying on top of this SDK quantified the gap: ~85% of brief-validation value for their use case sits at runtime, not schema time. They implemented a 5-layer resolver (notation match → lexical → token-set equality → Jaccard → Levenshtein) on top of ~10K lines of TTL (274 SKOS concepts, 942 synonyms, 210 GAM DMA display names). It works — but every vertical building on the SDK will rewrite some version of this, and today there is no supported runtime surface for them to build against.

This issue proposes a small, opinion-light set of runtime primitives that make resolution implementable in application code without pushing domain-specific matching logic into the SDK core.

Guiding principle: SDK provides, agent decides

The SDK and the agent layer make different kinds of claims:

• SDK: knows what's declared in the ontology. Entities, relationships, synonyms, notations, concept schemes, taxonomy structure. Stable, typed, queryable.
• Agent: knows what's intended. Which matcher to try first, what confidence threshold is safe for this domain, how to phrase a "did you mean" suggestion, whether fuzzy match on a free-text company_name is acceptable or dangerous.

Consequences for the runtime:

1. The SDK exposes read access over loaded ontologies (annotations, synonyms, scheme membership, taxonomy edges). No matching logic.
2. The SDK optionally materializes an ontology-derived concept index into BigQuery so agents can do SQL-native fuzzy match using BQ's existing EDIT_DISTANCE / SOUNDEX / UDFs.
3. The SDK defines an EntityResolver protocol and ships two trivial references (ExactMatchResolver, SynonymResolver). Anything beyond exact-match lives outside core.
4. Domain-specific resolvers (advertising, healthcare, finance) live in contrib/ or user code, never in the runtime's required surface.

The SDK stays general. Verticals get a contract to build against instead of reaching into YAML or reconstructing structure from BQ tables.

Current gaps

1. No runtime accessor over loaded ontologies. load_ontology() returns Pydantic models, but there's no shape-agnostic API like rt.synonyms("DMA") or rt.annotation("DMA", "skos:notation"). Agents parse the model directly, which couples them to schema details the SDK otherwise hides.

2. Annotations are not queryable at runtime. Issue Feat: SKOS import support alongside OWL (design proposal — feedback wanted) #57 proposes persisting SKOS annotations (skos:definition, skos:notation, skos:prefLabel, etc.) through import. Nothing today reads those annotations at runtime. They live in the YAML and die there.

3. No concept index. Synonyms and notations are scattered across per-entity YAML nodes. Agents that want to do SQL-level matching have to flatten this themselves at query time on every request.

4. No resolver interface. Every SDK user writes their own resolution entry point, with their own return type, with their own "did you mean" shape. No convention, no reuse.

Proposed primitives

1. OntologyRuntime — read accessor over loaded ontology + binding

Small, stateless, zero external dependencies at read time. Built on top of existing load_ontology() + load_binding().

from bigquery_agent_analytics import OntologyRuntime

rt = OntologyRuntime.load(
    ontology_path="ontology.yaml",
    binding_path="binding.yaml",
)

rt.entities()                              # list[str]
rt.entity("DMA")                           # Entity with annotations + synonyms
rt.synonyms("DMA")                         # ["Designated Market Area", ...]
rt.annotation("DMA", "skos:notation")      # "807"
rt.in_scheme("NielsenDMA")                 # list[Entity] — all concepts in scheme
rt.broader("RetailBanking")                # list[Entity] — skos:broader targets
rt.narrower("Banking")                     # inverse
rt.related("Account")                      # skos:related abstract-relationship targets

Design notes:

• Only reads. Never mutates ontology or binding.
• Covers both concrete and abstract (SKOS-derived) entities and relationships. Abstract elements are first-class at the runtime layer — they're the whole reason users care about SKOS at runtime.
• Works against the annotations produced by issue Feat: SKOS import support alongside OWL (design proposal — feedback wanted) #57's SKOS import without coupling to SKOS specifically. rt.annotation(name, key) treats skos:definition, owl:equivalentClass, or a user's custom annotation identically.

Identity rules (important after #57 lands):

• Entities are name-addressed. rt.entity(name), rt.synonyms(name), rt.annotation(name, key) are singular lookups — entity names remain globally unique.
• Relationships are traversal-first, not name-addressed. Issue Feat: SKOS import support alongside OWL (design proposal — feedback wanted) #57 relaxes relationship uniqueness to (name, from, to) for abstract relationships, so a single skos_broader can repeat across endpoint pairs. A hypothetical rt.relationship(name) would be unsafe because it has no single answer to return.
• All relationship accessors take an entity and traverse. rt.broader(entity), rt.narrower(entity), rt.related(entity) return the set of entities reachable from the given starting point via the named predicate. That's a well-defined question regardless of how many skos_broader edges exist in the ontology.
• If a relationship-by-name accessor is ever added, its contract must be compound identity (rt.relationship(name, from, to) -> Relationship | None) or list-returning (rt.relationships(name) -> list[Relationship]). Never singular-by-name.

2. Concept index materialization (opt-in)

At gm compile time, optionally emit a BigQuery sidecar table:

CREATE TABLE `{dataset}.ontology_concept_index` (
  entity_name STRING NOT NULL,
  label STRING NOT NULL,         -- for label_kind='notation', this holds the notation value
  label_kind STRING NOT NULL,    -- 'name' | 'pref' | 'alt' | 'hidden' | 'synonym' | 'notation'
  notation STRING,               -- per-entity notation for display; repeats across rows of the same entity
  scheme STRING,                 -- concept scheme this row's entity belongs to;
                                 -- NULL means "entity is not a member of any scheme"
  language STRING,               -- ISO-639 tag; NULL means unspecified or N/A (notation rows)
  is_abstract BOOL NOT NULL,     -- TRUE for SKOS-derived informational entities
  compile_id STRING NOT NULL     -- pair-consistency tag; see "Provenance and compatibility contract"
);

notation is a first-class row kind. For every entity that has a skos:notation, the compiler emits a row with label_kind='notation' and label=<notation value> — so resolvers searching by label naturally catch notation matches without a separate OR notation = @input predicate. The notation column is kept as per-entity metadata that repeats across all rows of the same entity, for display convenience (a caller with a winning match can read the entity's notation directly from the candidate row without a separate lookup).

Row multiplicity contract:

• One row per (entity_name, label, label_kind, language, scheme) membership tuple. A SKOS concept can legally belong to multiple skos:inScheme schemes (a DMA concept may be in both NielsenDMA and CensusMSA, a banking concept may be in both BankingTaxonomy and FinancialProductsTaxonomy). This is denormalized — a concept in 3 schemes × 5 labels produces 15 rows. Intentional; see below.
• Entities that aren't members of any scheme produce rows with scheme IS NULL. They're still in the index; entity= resolution finds them, scheme= resolution skips them.
• notation is per-entity (not per-scheme), so it repeats across membership rows for the same entity. Callers selecting a single notation per entity use DISTINCT notation or aggregate.

Why denormalized rather than ARRAY<STRING> scheme or a separate membership table:

• WHERE scheme = @x stays a trivial clustered lookup — critical for the common scheme=<name> resolver path.
• Predicate push-down into BQ clustering is straightforward; the clustering key (scheme, entity_name) stays usable.
• ARRAY<STRING> forces WHERE @x IN UNNEST(scheme) on every scheme-scoped query, which is less indexable and harder for less-experienced SQL callers to write correctly.
• A separate membership table adds a join to every resolver query, defeats the "one-table SQL lookup" simplicity that motivates the index.
• Row multiplication is bounded: even for pathological multi-scheme ontologies, row count is linear in (concepts × labels × schemes), which stays tractable at BQ scale.

Agents do fuzzy match in SQL:

-- exact, scheme-scoped (the common case)
SELECT DISTINCT entity_name
FROM ontology_concept_index
WHERE scheme = @scheme AND LOWER(label) = LOWER(@input);

-- fuzzy fallback with BQ native functions
SELECT entity_name, MIN(EDIT_DISTANCE(LOWER(label), LOWER(@input))) AS dist
FROM ontology_concept_index
WHERE scheme = @scheme
  AND EDIT_DISTANCE(LOWER(label), LOWER(@input)) <= 3
GROUP BY entity_name
ORDER BY dist ASC
LIMIT 5;

The DISTINCT/GROUP BY on entity_name is how callers collapse the denormalized rows back to one result per matched concept.

Matches the SDK's agent-native ethos: any action an agent takes in SQL is something a user or another tool can also take. No new Python-only runtime, no new service, no new matcher implementation to maintain.

Opt-in. v1 ships with a CLI flag only: gm compile --emit-concept-index. Default off for users who don't need it. A binding-side toggle (index: concept_index block on Binding) was considered but deferred — it requires schema and loader changes in bigquery_ontology.binding_models + binding_loader.py that are worth scoping as their own change once the CLI behavior is settled. If v2 adds it, the explicit precedence rule will be: CLI flag overrides binding setting; binding setting serves as the project default when the CLI flag is absent.

Index population contract

The existing DDL compiler (src/bigquery_ontology/graph_ddl_compiler.py) only emits schema SQL — CREATE TABLE / CREATE PROPERTY GRAPH. A concept index needs rows, which is a new kind of output. This subsection names who writes those rows and when.

Who writes the rows: the ontology compiler itself, in the same gm compile invocation that emits the DDL. The index is a deterministic function of both the ontology YAML and the binding — see "What's in the index" below. Treating it as a separate build step creates two sources of truth and a refresh-skew class of bugs that the SDK shouldn't inherit.

What's in the index (scope relative to binding): compile_concept_index(ontology, binding) takes both inputs because the index respects the binding's subset semantics. Since a binding may legally realize only a subset of the declared ontology (binding_models.py:147), the compiler needs a rule for which entities participate in the index. The rule is:

• All abstract entities from the ontology, regardless of binding — they're informational-only and never bound by construction (Feat: SKOS import support alongside OWL (design proposal — feedback wanted) #57's binding rejection rule). Their value is precisely in being available for runtime resolution even when the agent's BQ tables don't materialize them.
• Only concrete entities that are bound in this binding. Concrete + unbound entities are deliberately excluded from this deployment's runtime surface; including them would let a resolver return matches the agent then can't query. That's worse than a miss.

In short: abstract: always. Concrete: iff bound. This matches the SDK-level invariant from the adapter design ("every element in GraphSpec is bindable and has data") while preserving the taxonomy-browse value that abstract SKOS entities add at runtime.

Consequence: two different bindings over the same ontology produce different indexes. A narrow deployment binding only Account and Customer emits a smaller index than a wide deployment binding all 40 concrete entities, but both share the same abstract skos_Banking / skos_FinancialProduct / etc. nodes. Abstract relationships between abstract entities are always in scope; abstract relationships touching an unbound concrete entity are included (they're informational metadata, not runtime operations).

The is_abstract column in the index row lets resolvers filter at query time: a resolver that wants only runtime-materializable matches does WHERE NOT is_abstract; a resolver producing taxonomy-aware "did you mean" suggestions keeps both.

Table naming contract: because two bindings against the same ontology produce legitimately different indexes, a single global table name is unsafe — the second compile would silently overwrite the first. The output table name is therefore a required parameter, not a fixed convention:

def compile_concept_index(
    ontology: Ontology,
    binding: Binding,
    *,
    output_table: str,   # required — fully-qualified `project.dataset.table`
) -> str: ...

CLI:

gm compile --emit-concept-index \
           --concept-index-table my-project.my_dataset.ontology_concept_index__retail

Both library and CLI error cleanly if the name is missing when --emit-concept-index is set. No silent global default. Users with a single binding per dataset pick any unique name they like (ontology_concept_index is fine); users with multiple bindings per dataset pick distinct names per binding (ontology_concept_index__retail, ontology_concept_index__investment_bank, etc.).

Why required rather than auto-derived:

• Bindings do carry a binding: str identifier (binding_models.py:159), but it isn't a safe or stable source for a BQ table name: it's an identity tag for the binding document, not a deployment-unique BQ-legal identifier. Using it would couple operational naming to a field authors rename for non-operational reasons, and would collide across environments (dev/stage/prod) that share the same binding identity.
• Hash-derived defaults like ontology_concept_index__{sha1(binding)[:8]} are collision-free but unreadable and change on every trivial binding edit — bad ergonomics for a table name that appears in user-written resolver SQL.
• Explicit naming forces the deployment-operator-level decision at compile time, where it belongs.

OntologyRuntime reads the index via the same name the caller passed at compile time — runtime construction takes a matching concept_index_table: str parameter (or reads it from configuration) so lookups target the right table. The name is not stored on the ontology or binding model; it's a runtime/deployment concern.

Provenance and compatibility contract: because the table name is caller-supplied and binding-scoped, nothing in the data columns alone would catch a mismatched wiring like OntologyRuntime.from_models(ontology_A, binding_B, concept_index_table=table_C) where table_C was actually compiled from a different (ontology, binding) pair. Plausible-but-wrong matches are worse than no matches — the agent gets confident answers against stale or unrelated data.

The compiler therefore emits a sibling metadata table named {output_table}__meta, written in the same gm compile invocation. One row per compile:

CREATE OR REPLACE TABLE `{output_table}__meta` AS
SELECT * FROM UNNEST([
  STRUCT(
    'retail' AS ontology_name,                         -- from Ontology.name
    'sha256:abc123...' AS ontology_fingerprint,        -- see "Fingerprint algorithm" below
    'sha256:def456...' AS binding_fingerprint,         -- same algorithm, over Binding model
    'my-project' AS target_project,                    -- from Binding.target.project
    'my_dataset' AS target_dataset,                    -- from Binding.target.dataset
    'gm-1.2.0' AS compiler_version,                    -- version of bigquery_ontology that compiled
    'a1b2c3d4e5f6' AS compile_id                       -- pair-consistency tag; deterministic from inputs
  )
]);

Sibling rather than embedded columns so the bulk of the index (the label/notation rows) stays lean.

Fingerprint algorithm: fingerprints are SHA-256 hashes over a canonical serialization of the validated Ontology / Binding Pydantic models — not over raw YAML text. Concretely:

1. Load YAML → validated model (existing load_ontology() / load_binding() path). Validation normalizes optional fields, default values, and type coercion.
2. Serialize the validated model to a canonical JSON form: keys sorted lexicographically at every nesting level, no extra whitespace, UTF-8, stable encoding of None / booleans / numbers, lists preserved in declaration order (list order is semantically meaningful in the ontology model — e.g., key columns).
3. Hash the resulting bytes with SHA-256, prefix with sha256:.

The same approach is used for both ontology and binding fingerprints, with one runtime difference: ontology fingerprinting covers every field of the Ontology model. Binding fingerprinting covers every field of the Binding model except ephemeral annotations (if any are introduced later) — the binding's identity for the purpose of "does this index correspond to this binding" is its declared structure, not its documentation metadata.

Why model-based and not YAML-text-based:

• Two semantically identical YAML documents with different formatting, comment placement, or emitter behavior must produce the same fingerprint. A strict verification gate that rejects non-semantic edits would be a constant source of false positives and would push operators to disable verification — worse than no verification.
• Pydantic-validated models are already the canonical in-memory form the SDK works with (src/bigquery_agent_analytics/runtime_spec.py:199 and adjacent). Hashing at that layer matches the layer where the rest of the SDK's determinism lives.
• The existing compile contract is already model-based (compile_graph(ontology, binding) -> str takes models, not YAML strings). Keeping fingerprint input at the same layer maintains consistency across compile output and runtime verification.

Two bindings produced from the same source YAML by different emitters (e.g., one with trailing newlines, one without) fingerprint identically. Two bindings that disagree on any declared field — entity names, target dataset, property types — fingerprint differently and correctly fail strict verification.

Canonicalization rules in brief (formal spec in the implementation):

• Keys sorted at every nesting level (stable across Python dict iteration).
• Model fields serialized via Pydantic.model_dump(mode="json", by_alias=False, exclude_none=False) so defaults materialize consistently.
• Enum values serialized as their canonical string form, not member name.
• None / missing-but-defaulted fields serialized as explicit null to distinguish "absent" from "defaulted."
• List order preserved; no reordering of entity/relationship/property lists (order is semantically load-bearing).
• Output encoded as UTF-8 JSON with separators=(",", ":") (no extra whitespace).

OntologyRuntime runtime verification:

• At construction, OntologyRuntime.load(...) / .from_models(...) computes the same fingerprints on the loaded Ontology and Binding models.
• On first access to the concept index (lazy — construction doesn't hit BQ), the runtime reads the __meta sibling and compares fingerprints.
• Mismatch raises ConceptIndexMismatchError with a clear message naming the expected vs actual fingerprints and the table name involved. The runtime refuses to return matches from an index that doesn't correspond to the loaded models.
• Missing __meta sibling (e.g., a manually-created index or one compiled with an older toolchain) raises a distinct ConceptIndexProvenanceMissing — caller can explicitly opt out with OntologyRuntime(..., verify_concept_index="off") for read-only dashboards or interactive exploration.
• Verification re-checks on a configurable TTL, not once-per-lifetime. See "Long-lived runtime verification" below.

Long-lived runtime verification (strict is strict for the whole lifetime, not just the first call). A naive "verify once then cache forever" contract would let a long-lived service sail past an index refresh that swapped in a different (ontology, binding) pair — returning matches against the new index while still believing it was verified. That defeats the "plausible-but-wrong matches are worse than no matches" argument behind the strict default.

The contract:

• After the first successful verification, OntologyRuntime caches the expected compile_id, ontology_fingerprint, and binding_fingerprint on the instance.
• On each resolve / validate call, the runtime checks whether the cached verification is still fresh under a configurable TTL (verify_ttl_seconds, default 60). If the cache is fresh, the call proceeds without a BQ round-trip.
• If the cache is stale, the runtime re-runs the full pair-consistency check plus a full-fingerprint freshness check, not just a single-table sentinel. Concretely:
  1. SELECT DISTINCT compile_id FROM {output_table} LIMIT 2 returns exactly one value.
  2. SELECT compile_id, ontology_fingerprint, binding_fingerprint FROM {output_table}__meta LIMIT 1 — read compile_id and the full fingerprints.
  3. Verify: main.compile_id == meta.compile_id (pair consistency).
  4. Verify: meta.compile_id == cached.compile_id AND meta.ontology_fingerprint == cached.ontology_fingerprint AND meta.binding_fingerprint == cached.binding_fingerprint (full-fingerprint freshness).
• Outcomes:
  • All checks hold → refresh the cache timestamp and proceed.
  • Pair consistent but any cached value differs from meta → raise ConceptIndexRefreshed. Service operator recreates OntologyRuntime with updated models; new instance's full fingerprint verification catches whether the new index matches or not.
  • Main and meta disagree (refresh in progress) → one-shot 2s retry, then raise ConceptIndexInconsistentPair. Same contract as first-load.

Why the sentinel must read both tables, not just meta. An earlier draft checked only meta.compile_id. That has a correctness hole: the inline refresh order is "main first, meta second," so during the swap window a reader could see the old meta compile_id (matches cache, accepted), then query the new main table, and serve data from the refreshed index under stale verification. Reading both tables on TTL re-check closes that window — main's compile_id is authoritative for "which compile does the data belong to," and the meta comparison catches inconsistent pairs.

Why the freshness check compares full fingerprints, not just compile_id. The compile_id column is a 12-hex-char truncation of sha256(ontology_fingerprint || binding_fingerprint || compiler_version) — 48 bits of entropy, chosen to keep the per-row compile_id column short (storage efficiency on a column that repeats across every data row). That's enough for first-pass pair consistency: two tables with different compile_ids definitely belong to different compiles, and the birthday bound on distinct compiles for a single (output_table) over realistic deployment lifetimes is comfortably below collision probability.

But "comfortably below" is not "zero," and a strict verification contract shouldn't rely on it. The meta row carries the full ontology_fingerprint and binding_fingerprint (SHA-256, 256 bits each) — storing those in a single-row meta table costs nothing. The TTL re-check therefore compares all three (compile_id + both full fingerprints) against the cache. A hypothetical 48-bit collision where a legitimately-different (ontology, binding) pair happens to share a 12-char prefix is caught because the full fingerprints won't match.

Pair consistency between the two tables still runs on the short compile_id — it only needs to detect "are these from the same compile or different compiles," and 48 bits is overkill for that single-dataset comparison. The strict freshness check runs on the full 256-bit fingerprints where the safety story demands it.

The three reads are still cheap. Main's SELECT DISTINCT compile_id FROM {output_table} LIMIT 2 reads at most two rows from a clustered column; meta reads exactly one row (and always has, just with more columns than before). Per-TTL-window cost remains negligible even at default 60s.

Configuration surface on OntologyRuntime construction:

• verify_ttl_seconds: int = 60 — default 60. Balance between correctness-staleness window and re-verification cost.
• verify_ttl_seconds=0 — check on every call. Useful for low-QPS services where correctness matters more than cost.
• verify_ttl_seconds=None — snapshot-bound: verify once on first use, never again. Explicit opt-in for services that coordinate refresh out-of-band (e.g., rolling-restart on recompile). Matches the old "verify once" behavior for callers who want it.

Why TTL rather than check-every-call by default: the pair re-check is cheap but not free, and for high-QPS resolver workloads it adds up. A 60s staleness window matches typical service-refresh cadences while keeping per-call cost bounded at O(1) with no BQ hit in the common case.

Pair-consistency contract (the two tables must agree on the same compile). Because {output_table} and {output_table}__meta are written as two separate CREATE OR REPLACE TABLE statements, a reader interleaved with a refresh could otherwise observe:

• new meta + old data → strict verification would pass against stale data (plausible-but-wrong matches).
• new data + old meta → strict verification would raise an incorrect mismatch.

To make the pair coherent without requiring DDL-level transactions (which BigQuery doesn't offer for CREATE OR REPLACE TABLE), both tables carry a compile_id tag that is derived deterministically from compile inputs — not a per-run UUID or a timestamped value:

compile_id = sha256(ontology_fingerprint || binding_fingerprint || compiler_version)[:12]

(First 12 hex chars is enough to make accidental collisions vanishingly unlikely while keeping the column short.)

• compile_id STRING NOT NULL column on the main table — every row of {output_table} shares the same value.
• compile_id field on the single __meta row — same value.
• Write order: main table first, meta second. Readers never see "new meta promising data that doesn't exist yet."

Why deterministic rather than per-run:

• Preserves the byte-identical output contract on compile_concept_index() (see Compiler output contract below). Two compiles of the same ontology + binding + compiler version produce character-identical SQL.
• Pair consistency still works: interleaved compiles with different inputs produce different compile_ids and the runtime check catches the inconsistency. Interleaved compiles with identical inputs produce identical compile_ids and the data is also identical — the worst case is wasted work, not wrong data.
• Callers auditing compile output in code review can diff it against the previous compile and see only the changes caused by ontology/binding edits, not a new UUID every run.

compiled_at is deliberately not in the emitted SQL. An earlier draft included a compiled_at TIMESTAMP field in the meta row; that's been removed to preserve byte-identical output. Operators who want compile timestamp visibility can read it from INFORMATION_SCHEMA.TABLES.creation_time on the __meta table, which BigQuery maintains automatically. The tradeoff is deliberate: runtime correctness (deterministic compile output, reviewable diffs) over embedded operator metadata that BQ already provides.

Runtime pair-consistency check (first concept-index access):

1. SELECT * FROM {output_table}__meta LIMIT 1 → get expected_compile_id, expected fingerprints.
2. SELECT DISTINCT compile_id FROM {output_table} LIMIT 2 → verify exactly one compile_id is present and it equals expected_compile_id.
3. If compile_id mismatches or multiple distinct compile_ids are observed (which would indicate a broken compile), retry once after a short backoff (default 2 seconds) — handles the narrow interleaving window during normal refresh.
4. If the retry also fails, raise ConceptIndexInconsistentPair with both observed compile_ids. This is distinct from ConceptIndexMismatchError (which is a wiring/fingerprint error, not a timing one) so callers can handle them differently.
5. Once pair-consistency is established, fingerprint verification proceeds against the meta row.

The retry is deliberately one-shot and small: a legitimately long refresh window indicates operator misbehavior (concurrent compiles against the same table) and should fail loudly.

Compatibility flag vocabulary:

• verify_concept_index="strict" (default) — fingerprint mismatch, missing meta, or persistent pair inconsistency all raise.
• verify_concept_index="missing_ok" — fingerprint mismatch and pair inconsistency raise, missing meta warns and proceeds.
• verify_concept_index="off" — no verification, purely caller-managed. Intended for explicit "I know what I'm doing" paths.

Rejected alternatives for pair consistency:

• Transactional multi-statement BEGIN TRANSACTION; ... COMMIT;. BigQuery's transaction support doesn't cover CREATE OR REPLACE TABLE — DDL is generally non-transactional. Not a viable primitive.
• Shadow-version both tables + atomic pointer-table swap. Three tables (main, meta, pointer) adds significant operational complexity for a narrow window. The compile_id approach gets the same correctness property with one extra column.
• BQ table OPTIONS(description=...) tagging. Requires INFORMATION_SCHEMA lookups, has its own freshness semantics, more tooling surface. compile_id in a data column is simpler to query from plain SQL.

Rejected alternatives for the provenance storage shape:

• Embed provenance as repeated columns on every index row. Wastes storage proportional to the number of rows; makes diff-based review noisier; still requires runtime verification logic. Sibling table is strictly better. (The compile_id column is a deliberate exception — it's a single short fixed-length tag needed for pair consistency, not full provenance.)
• Encode provenance in BQ table OPTIONS(description=...). BQ-native and elegant, but INFORMATION_SCHEMA queries have their own cost and tooling constraints, and the sibling table approach is easier to inspect from plain SQL (SELECT * FROM concept_index__meta).
• Caller-managed in v1 with no verification. Considered and rejected — shipping a primitive that silently produces wrong results under a plausible operator mistake is a feature bug the SDK shouldn't ship with. v1 ships strict verification on by default, with documented escape hatches.

How the rows reach BQ (atomic-swap semantics for runtime readers): the compiler emits a single CREATE OR REPLACE TABLE ... AS SELECT ... statement. In BigQuery this is atomic — concurrent readers see either the previous table's rows or the new table's rows, never an empty intermediate state. This is the critical difference from a DELETE + INSERT pair, which would expose a window where the index is queryable but empty. For a runtime lookup primitive, that window is a correctness hazard, not just a performance one.

-- gated on --emit-concept-index, name passed via --concept-index-table
-- write order: main table first, __meta second. compile_id ties the pair.
CREATE OR REPLACE TABLE `{output_table}` AS
SELECT * FROM UNNEST([
  STRUCT('DMA' AS entity_name, 'DMA' AS label, 'name' AS label_kind,
         '807' AS notation, 'NielsenDMA' AS scheme,
         CAST(NULL AS STRING) AS language,
         FALSE AS is_abstract,
         'a1b2c3d4e5f6' AS compile_id),
  STRUCT('DMA', 'Designated Market Area', 'synonym',
         '807', 'NielsenDMA', 'en', FALSE, 'a1b2c3d4e5f6'),
  STRUCT('DMA', 'Marché de diffusion désigné', 'pref',
         '807', 'NielsenDMA', 'fr', FALSE, 'a1b2c3d4e5f6'),
  -- first-class notation row: label holds the notation value, label_kind='notation'
  STRUCT('DMA', '807', 'notation',
         '807', 'NielsenDMA', CAST(NULL AS STRING), FALSE, 'a1b2c3d4e5f6'),
  -- multi-scheme example: same entity appearing in two schemes
  STRUCT('BayAreaMetro', 'Bay Area Metro', 'pref',
         CAST(NULL AS STRING), 'NielsenDMA', 'en', FALSE, 'a1b2c3d4e5f6'),
  STRUCT('BayAreaMetro', 'Bay Area Metro', 'pref',
         CAST(NULL AS STRING), 'CensusMSA', 'en', FALSE, 'a1b2c3d4e5f6'),
  -- abstract SKOS concept: informational, no scheme membership
  STRUCT('skos_Banking', 'Banking', 'pref',
         CAST(NULL AS STRING), CAST(NULL AS STRING), 'en', TRUE, 'a1b2c3d4e5f6'),
  ...
]);

The separate CREATE TABLE scaffold is not emitted — CREATE OR REPLACE TABLE creates the table on first run and atomically replaces it on every subsequent run. This collapses "ensure table exists" and "populate rows" into one statement, eliminating the empty-table-existing intermediate state entirely.

For ontologies with tens of thousands of concepts (Yahoo's YAMO example: 274 SKOS concepts × multiple labels ≈ ~1K-10K rows), inline UNNEST(ARRAY<STRUCT<...>>) is well within BigQuery's query-text limits. For ontologies above ~50K rows, the compiler emits a shadow-table swap pattern for both tables in the pair — the pair-consistency contract applies to the shadow path too:

-- both tables get a shadow; suffix is "_shadow" on each production name
CREATE OR REPLACE TABLE `{output_table}_shadow` (...);
INSERT INTO `{output_table}_shadow` VALUES (...);  -- batched, includes compile_id column
CREATE OR REPLACE TABLE `{output_table}__meta_shadow` AS
  SELECT * FROM UNNEST([STRUCT(... 'a1b2c3d4e5f6' AS compile_id)]);

-- swap order: data first, then meta (matches the inline-path write order)
DROP TABLE IF EXISTS `{output_table}`;
ALTER TABLE `{output_table}_shadow` RENAME TO <short name from output_table>;
DROP TABLE IF EXISTS `{output_table}__meta`;
ALTER TABLE `{output_table}__meta_shadow` RENAME TO <short name from output_table>__meta;

Two distinct non-atomicity windows exist on this path:

1. Table-existence window: between DROP and RENAME on each table, that table name does not resolve. Readers get BigQuery's "table not found" error, which they must tolerate as transient.
2. Pair-inconsistency window: between "main renamed" and "meta renamed", the main table carries the new compile_id while the meta row still carries the old one. Readers in this window see compile_id disagreement → ConceptIndexInconsistentPair on the pair-consistency check.

The pair-inconsistency window on the shadow path can exceed the inline path's one-shot 2-second retry budget, because large-ontology rename operations take longer than small-ontology CREATE OR REPLACE TABLE statements. This means strict verification will raise during a legitimate shadow-path refresh if it happens to sample during the swap. That's by-design, not a defect: strict verification correctly rejects an inconsistent pair even when the inconsistency is transient. The alternative (silently serving old data against a new meta, or vice versa) is the failure mode strict verification exists to prevent.

Operational contract for the shadow path:

• Treat shadow-path refreshes as offline/admin operations. Pause reader traffic (or accept ConceptIndexInconsistentPair exceptions) during gm compile runs that hit the shadow path.
• If traffic cannot be paused, the caller has two options, neither of which involves missing_ok (which per the verification-mode contract above still raises on pair inconsistency — transient or not):
  • Increase verify_ttl_seconds so the pair re-check samples less frequently. Reduces the probability of landing inside a swap window at the cost of a longer staleness tolerance.
  • Catch ConceptIndexInconsistentPair at the application layer and retry the call after a short delay. Cleanest at the service-mesh level where transient 5xx handling already exists.
• For services where neither is acceptable: bind the main + meta pair under a higher-level indirection (a separate {output_table}__current pointer table that callers resolve through). Not shipped in v1 — out of scope as a third level of indirection, tracked as follow-up work if real users hit this constraint.

This limitation is specific to the shadow path. The inline-UNNEST path (the default for ontologies under 50K rows, covering the motivating use cases including Yahoo's YAMO) remains fully atomic per-statement and doesn't exhibit either window.

When refresh happens: on every gm compile run with --emit-concept-index. No incremental build. If the user edits ontology YAML, they re-run compile, same as any other DDL change. This matches the compile model users already have for schema changes and avoids adding a second refresh command.

Alternatives considered and rejected:

• Separate gm build-concept-index step. Adds a command users have to remember and introduces drift between "DDL is up to date" and "index is up to date." Two invocations for one conceptual change.
• Runtime lazy-build. Rebuild the index in memory on OntologyRuntime.load() and optionally push to BQ. Surfaces inconsistent state when multiple agent instances load simultaneously and makes the "query the index in SQL" path unreliable until someone has pushed.
• Streaming incremental updates. Possible future work for ontologies with externally-sourced concept rolls. Out of scope for the initial primitive.

Failure modes:

• Inline CREATE OR REPLACE TABLE path: if the statement fails (quota, permissions, query text too large), gm compile errors with a message naming the concept index. Because CREATE OR REPLACE is atomic, there is no half-written state — the previous table (if any) remains queryable, or no table exists at all. The user can re-run compile without cleanup.
• Shadow-table swap path: failure mid-swap can leave the pair in an inconsistent state (main renamed but meta not, or either table dropped but not renamed). gm compile retry detects the orphaned _shadow table(s) and resumes from the swap step. Runtime readers during the orphaned window get either "table not found" or ConceptIndexInconsistentPair — both expected transient conditions on the shadow path, tolerated via the operational contract above (pause traffic during refresh, or accept transient failures).

3. EntityResolver Protocol + two reference implementations

Interface-only in core:

from typing import Protocol
from dataclasses import dataclass

@dataclass
class Candidate:
    entity_name: str         # unique per Candidate in ResolveResult.candidates
    label: str               # the winning label that produced this match
    label_kind: str          # 'name' | 'pref' | 'alt' | 'hidden' | 'synonym' | 'notation'
    scheme: Optional[str]    # scheme the winning match came through (None = entity-scoped or no scheme)
    confidence: float
    reason: str              # 'exact' | 'notation' | 'synonym' | 'fuzzy' | 'none'

@dataclass
class ResolveResult:
    match: Optional[str]           # resolved entity_name (None = no match)
    confidence: float              # 0.0 - 1.0; 1.0 = exact
    candidates: list[Candidate]    # top-k "did you mean" suggestions, one per entity
    reason: str                    # why `match` resolved (or 'none')

class EntityResolver(Protocol):
    def resolve(
        self,
        value: str,
        *,
        scheme: str | None = None,
        entity: str | None = None,
        limit: int = 5,
    ) -> ResolveResult: ...

scheme and entity are mutually exclusive — see "Scope semantics" in the Library API impact section below. Exactly one must be provided.

Candidate dedup contract (important once the index is denormalized per (entity_name, label, label_kind, language, scheme)):

• ResolveResult.candidates contains at most one entry per entity_name. The denormalized index naturally produces multiple matching rows for the same entity (same entity, different label or different scheme). For an agent-facing "did you mean" list, duplicates are noise — the agent wants a list of distinct concepts, each annotated with the best evidence for why it matched.
• limit=N means N distinct entities, not N raw rows. Resolvers do the dedup before truncating.
• Winning-label rule when the same entity matches through multiple rows: pick the row with the highest confidence under the resolver's matching rule. Ties broken by label_kind priority, in this order: name > pref > alt > hidden > synonym > notation. Further ties broken by lexicographic label order for determinism.
• Candidate.label / Candidate.label_kind / Candidate.scheme / Candidate.reason reflect the winning row. The other rows that also matched are discarded — callers wanting the full provenance use the concept index directly via SQL.
• reason values are resolver-defined but drawn from a shared vocabulary so callers can branch on them without ambiguity: exact (name match), notation (notation match), synonym (any label other than name), fuzzy (non-exact match produced by a fuzzy resolver), none (no match found — only valid on the ResolveResult.reason, not on Candidate.reason).

SDK ships two references in core:

• ExactMatchResolver — O(1) lookup against name + skos:notation. Confidence is 1.0 or 0.0. Good for notation-heavy inputs (Nielsen DMA codes, Google Ads Criteria IDs).
• SynonymResolver — extends ExactMatchResolver by also matching against prefLabel / altLabel / hiddenLabel / synonyms. Still exact on each label; still confidence 1.0 or 0.0.

Everything above exact-match — token-set equality, Jaccard, Levenshtein, phonetic, weighted ensembles — lives in user code or contrib/ packages. Verticals pick (or write) a resolver tuned for their domain.

4. validate_against_ontology — small validation helper

Not resolution — just pass/fail against the declared ontology. Return shape is bounded by design so it stays useful on large concept schemes (IAB Taxonomy, Nielsen DMAs, SNOMED excerpts) where the candidate universe is hundreds to tens of thousands of entries:

rt.validate(
    {"format_ids": ["display_static", "display_banner"]},
    scheme="AdFormat",       # see "Scope semantics" below
    sample_limit=10,         # default — cap on known_values_sample
)
# → ValidationResult(
#     valid=["display_banner"],
#     invalid=["display_static"],
#     known_value_count=47,
#     known_values_sample=["display_banner", "display_native", ...],  # up to sample_limit
#     candidates=None,   # populated only when composed with a resolver
# )

Agents combine validate_against_ontology with a resolver to produce "did you mean." The SDK doesn't match; it only knows what exists.

Design notes on the return shape:

• known_value_count is always the full count. Tells the caller whether the sample is representative.
• known_values_sample is capped at sample_limit (default 10). Enough for a "did you mean" hint without bloating every validation miss on a 10K-concept scheme. Callers who genuinely need the full set use rt.in_scheme(...) or rt.entities() — that's what those accessors are for.
• candidates stays None unless the caller composes validation with a resolver. Keeps validate pure set-membership; keeps ranking logic in resolver-land. No double-duty.
• Sample order is not specified by the contract — callers should not rely on alphabetical or any other ordering. If deterministic ordering matters for a specific use, pass a sorted known_values_sample through a resolver that ranks.

Library API impact

This section pins down the parts of the proposal that touch existing public APIs, so they're clear before implementation starts.

Compiler output contract

The existing bigquery_ontology.compile_graph(ontology, binding) -> str is documented to be deterministic — "same inputs → byte-identical text." That contract is preserved. Concept-index emission does not modify compile_graph().

Instead, a new sibling function ships alongside:

# existing, unchanged
def compile_graph(ontology: Ontology, binding: Binding) -> str: ...

# new, additive
def compile_concept_index(
    ontology: Ontology,
    binding: Binding,
    *,
    output_table: str,   # required — see "Table naming contract" above
) -> str: ...

Both return deterministic strings. compile_concept_index() extends the compile_graph() byte-identical contract in the same spirit: same inputs → byte-identical DML text, including row order.

"Same inputs" for compile_concept_index() = (ontology, binding, output_table, compiler_version). Everything in the emitted SQL is derived from those four values:

• compile_id is sha256(ontology_fingerprint || binding_fingerprint || compiler_version)[:12] — deterministic.
• No per-run timestamps, UUIDs, or process identifiers appear in the emitted SQL. Compile timestamps are recoverable from INFORMATION_SCHEMA.TABLES.creation_time on the emitted tables.
• Row order is determined by the sort key below, applied before SQL generation.

Rows are sorted by a stable key before SQL generation:

(scheme, entity_name, label_kind, language, label, notation, is_abstract)

with NULLs ordered last consistently per column. is_abstract is last because it's determined by entity_name — included only for defensive stability if the invariant ever loosens. This sort order guarantees that two invocations of compile_concept_index() on the same ontology + binding emit character-identical SQL — critical for diffing compile output in code review, caching compiled artifacts, and verifying that ontology edits produced only the expected row changes.

Library callers who want only DDL keep calling compile_graph() as today; callers who want the concept index call compile_concept_index() for a separate DML script. The CLI layer composes the two:

# CLI behavior for `gm compile --emit-concept-index`
sql_parts = [compile_graph(ont, binding)]
if args.emit_concept_index:
    sql_parts.append(
        compile_concept_index(ont, binding, output_table=args.concept_index_table)
    )
print("\n\n-- concept index --\n\n".join(sql_parts))

Why a sibling and not a composed option:

• Preserves the byte-identical contract on compile_graph().
• No breaking change to existing callers.
• Each function has one job; easier to test, version, and reason about.
• CLI callers with shell-orchestrated pipelines can write DDL and DML to separate files if they want — composition stays a caller concern.

Rejected alternatives:

• compile_graph(..., emit_concept_index=True) returning concatenated DDL+DML — breaks the byte-identical contract for one config mode and creates a function whose return value depends on a flag.
• Return an object (CompileResult(ddl=..., dml=...)) — breaks every existing caller of compile_graph() that treats the return as a string.
• CLI-only (no library-layer API for the index DML) — forces library users to reimplement concept-index generation themselves, defeats the point of the primitive.

OntologyRuntime construction: paths and models

The example in section 1 shows OntologyRuntime.load(ontology_path=..., binding_path=...). But existing SDK code already carries validated Ontology and Binding models around in memory (e.g., src/bigquery_agent_analytics/runtime_spec.py:199 passes models directly). Forcing callers to reparse YAML or round-trip through disk would be a step backward.

Two classmethods cover both cases:

class OntologyRuntime:
    @classmethod
    def load(
        cls,
        ontology_path: str | Path,
        binding_path: str | Path,
    ) -> "OntologyRuntime":
        """Load from YAML files on disk."""
        ...

    @classmethod
    def from_models(
        cls,
        ontology: Ontology,
        binding: Binding,
    ) -> "OntologyRuntime":
        """Wrap already-validated models."""
        ...

load() is the convenience path for one-off scripts and the CLI. from_models() is the integration path for the SDK's existing flows — runtime_spec, ontology_orchestrator, adapters downstream of load_ontology() can all wrap without touching disk again.

Internal implementation: load() calls load_ontology() + load_binding() then delegates to from_models(). Zero code duplication.

Scope semantics: scheme vs entity

Resolvers and validate() need an explicit target set. Two mutually-exclusive named parameters, no polymorphism:

# Scheme-scoped: resolve/validate against all members of a concept scheme.
# Most common case — this is what the motivating examples (AdFormat, DMA, IAB) want.
rt.validate({"dma": ["Nielsen 807"]}, scheme="NielsenDMA")
resolver.resolve("San Francisco-Oakland", scheme="NielsenDMA")

# Entity-scoped: resolve/validate against a single named entity. Identity check only.
# Rare — used when you want "is this exactly this one entity?" rather than
# "is this a member of a taxonomy?"
rt.validate({"customer_id": ["C-42"]}, entity="Customer")

Rules:

• Exactly one of scheme or entity must be provided. Passing both or neither is an error with a clear message.
• scheme=<name> resolves against the set {e : e.in_scheme(name) or name == e.name and e.is_abstract_scheme_root}. This is the motivating case. Works for both explicit SKOS concept schemes and abstract entities that act as taxonomy roots.
• entity=<name> resolves against the singleton set {e : e.name == name}. Identity check. Returns match iff the input exactly matches the entity's name, notation, or a declared label/synonym.
• Narrower-closure scoping (e.g., "all narrower-than some abstract node") is explicitly deferred. When the need surfaces, it'll come back as scope=Scope.narrower_closure(name) or similar, without changing the meaning of scheme and entity.

Why not polymorphic ("entity= means scheme-scoped if it's a scheme, entity-scoped otherwise"):

• Two implementers following the spec would return different answers for the same call, depending on their interpretation of the ontology's structure.
• Ontology authors who later change an entity from concrete to abstract-scheme-root would silently change the semantics of every entity= call targeting it.
• Callers would need ontology knowledge to predict what a given entity= call does — defeats the point of a stable API.

Explicit parameters keep the contract boring and predictable.

Non-goals

• Ship a general string-matching library. BQ already has EDIT_DISTANCE, SOUNDEX, JACCARD UDFs. If the concept index is materialized, users get these for free. Don't wrap.
• Ship the 5-layer resolver in core. Token-set equality thresholds, Jaccard coefficients, Levenshtein cutoffs — all domain-tuned. Advertising's tuning for DMAs is not the right tuning for SNOMED or legal-entity names. The feedback author's resolver is valuable as a reference for their domain and belongs in contrib/ or a separate package.
• Promise a <50ms SLA. Latency is a function of index size and resolver choice, both of which vary by user. The SDK can guarantee the primitive shapes; it can't guarantee the performance of every application that uses them.
• Provide a concept-scheme browser UI. Out of scope — this is an analytics SDK, not an ontology editor.
• Take a position on "did you mean" phrasing. The SDK returns structured candidates; the agent composes user-facing copy.

How this lands on top of existing code

Using the current SDK's module boundaries:

Piece	Belongs in	Notes
OntologyRuntime class with load() + from_models() classmethods	bigquery_agent_analytics/ontology_runtime.py (new)	Wraps load_ontology + load_binding from bigquery_ontology. Pure Python, no BQ calls. Both construction paths share one implementation.
compile_graph() (existing)	bigquery_ontology/graph_ddl_compiler.py	Unchanged. Preserves byte-identical contract.
compile_concept_index() (new sibling)	bigquery_ontology/graph_ddl_compiler.py (new function)	Separate deterministic DML emitter. CLI composes with compile_graph() when --emit-concept-index is set.
EntityResolver Protocol + references	bigquery_agent_analytics/entity_resolver.py (new)	Core SDK layer. Protocol + two implementations. Both accept scheme= or entity= (mutually exclusive).
validate_against_ontology	Method on OntologyRuntime	Same scheme= / entity= scope parameters.
Domain packs and layered resolvers	bigquery_ontology/contrib/ or external packages	Advertising, healthcare, finance. Never in core.

Changes to existing modules are limited but not zero. Most of the proposal is additive (new files, new functions, new classmethods). Two concrete edits to existing code are needed:

• src/bigquery_ontology/cli.py:299 — the existing compile command gains --emit-concept-index and --concept-index-table <name> flags. When --emit-concept-index is set, the command composes the existing compile_graph() output with compile_concept_index(..., output_table=...). Without the flag, the command's behavior is byte-identical to today.
• src/bigquery_ontology/graph_ddl_compiler.py — adds the new compile_concept_index() function in the same module. compile_graph() itself is not modified.

The runtime accessor (OntologyRuntime) reads the same Ontology/Binding models already loaded today — that path is purely additive in the SDK package.

Ties to issue #57 (SKOS import)

This proposal depends on issue #57 landing first, because the concept index's value comes almost entirely from SKOS annotations (skos:notation, skos:prefLabel, skos:altLabel, skos:broader) being preserved through import. Without #57, the concept index is a thin wrapper over entity names and existing synonyms — useful but not transformative.

Specifically:

• skos:notation in annotations → notation column in concept index → L1 code match becomes trivial
• skos:prefLabel / altLabel / hiddenLabel → rows in concept index with label_kind discriminator → L2 lexical becomes trivial
• skos_broader abstract relationships → rt.broader() traversal → taxonomy-aware "did you mean a parent or sibling"
• Abstract entities with skos_ prefix → rt.in_scheme() enumerates all concepts in a taxonomy → agent can present the scheme to the LLM as context

Open questions — feedback wanted

1. Is OntologyRuntime the right wrapper, or should the accessors live as methods on Ontology / Binding directly? Pro-wrapper: keeps bigquery_ontology pure-data and the runtime layer in bigquery_agent_analytics. Pro-direct: fewer classes to learn. Proposal leans wrapper — the accessor layer is SDK-runtime concern, not ontology-package concern.

2. Should the concept index be opt-in or opt-out? Pro-opt-in: users who don't need it don't pay storage. Pro-opt-out: users discover the primitive because it just exists. Proposal leans opt-in: no silent BQ table creation.

3. Should OntologyRuntime cache the concept index in memory for pure-Python access, or always go to BQ? Pro-memory: fast, no BQ cost, works offline. Pro-BQ-only: always consistent with DDL, scales to ontologies with 100K+ concepts. Proposal: pure-Python by default for ontologies under some size threshold; explicit BQ-backed resolver for large ones.

4. Does EntityResolver need an async variant? Resolution against a BQ-backed index is I/O. Proposal: ship sync; add async later if users ask.

5. Should the SDK ship a richer FuzzyResolver reference (just exact + prefix, not full 5-layer) so users have a middle option? Proposal: no — either exact or bring-your-own. Avoids the "SDK partially solves fuzzy matching" trap where the reference becomes everyone's default despite being domain-unaware.

6. Should the Protocol be typing.Protocol or an ABC? Protocol allows duck typing; ABC forces inheritance. Proposal: Protocol — matches modern typing conventions and doesn't force users to inherit.

7. Should rt.validate() also return a nearest field when values are invalid? Would require calling a resolver inside validate, coupling the two. Proposal: no — keep validate pure set-membership, let callers compose it with a resolver.

8. Concept index: do we need a per-row score or priority for when multiple labels map to the same entity? Some verticals (IAB) prefer one label over another as the "canonical" display form. Proposal: defer — label_kind (name vs pref vs alt) already lets callers prioritize. Add score if needed.

9. Is contrib/ the right home for domain resolvers, or should they be separate packages? Pro-contrib: easy discovery, versioned together. Pro-separate: community can ship without depending on SDK releases. Proposal: contrib for reference implementations (advertising, healthcare); external packages for user-owned domains.

10. Should narrower-closure scoping ship in v1? The current proposal settled on two explicit parameters — scheme= for concept-scheme membership and entity= for single-entity identity. A third mode (narrower-closure: "resolve against all entities narrower-than some abstract node") is deferred. Advertising taxonomies nest (IAB Tier 1 → Tier 2), and a caller may want to resolve against the subtree under a specific abstract node rather than a flat scheme. Proposal: ship scheme= and entity= only in v1; add scope=Scope.narrower_closure(name) in v2 if real callers need it. For most cases, scheme membership plus rt.narrower(entity) traversal covers the need without a new API.

---

Related:

• Issue Feat: SKOS import support alongside OWL (design proposal — feedback wanted) #57 — SKOS import support (prerequisite for the concept index to carry notation/labels/taxonomy edges)
• Feedback gist: https://gist.github.com/haiyuan-eng-google/54c3d3366b3d75b659561ef4e24e9374 (original context from production user)

Please comment if you have opinions, real-world resolver implementations you'd like to see supported, or disagreements about where the SDK/agent boundary should sit.

---

Final design decisions — detailed

After twelve rounds of review the design is frozen. In-repo implementation plan at docs/implementation_plan_concept_index_runtime.md. This section is the design-level recap, split by package.

Ontology package (bigquery_ontology) — changes

New files (all under src/bigquery_ontology/):

• _fingerprint.py (internal — underscore prefix). Single source of truth for model fingerprinting and the compile_id pair-consistency tag. Two functions: fingerprint_model(model) -> "sha256:<64 hex>" and compile_id(ont_fp, bnd_fp, compiler_version) -> "<12 hex>". Contract pinned in docstring (W1): model_dump(mode="json", by_alias=False, exclude_none=False) → json.dumps(sort_keys=True, separators=(",",":"), ensure_ascii=False) → SHA-256. Not re-exported; both packages import via from bigquery_ontology._fingerprint import .... Landed in PR feat(ontology): A1 — internal _fingerprint module for concept-index provenance #71.

• concept_index.py (module importable but not re-exported in v1). Row builder. Function: build_rows(ontology, binding) -> list[ConceptIndexRow]. Applies the "abstract always included, concrete iff bound" rule. Emits one row per (entity_name, label, label_kind, language, scheme) membership tuple, plus one notation row per skos:notation. Sorts deterministically by (scheme, entity_name, label_kind, language, label, notation, is_abstract) with NULLs last. Package-level re-export may be added later; kept out of the root for v1 to avoid growing semver surface ahead of need.

Modified files:

• graph_ddl_compiler.py — gains a new public function compile_concept_index(ontology, binding, *, output_table) -> str alongside the existing compile_graph(). compile_graph() contract is preserved byte-identically; the existing function body is not touched. compile_concept_index() emits two statements by default: CREATE OR REPLACE TABLE {output_table} AS SELECT * FROM UNNEST([STRUCT(...), ...]) for the main index and a matching CREATE OR REPLACE TABLE {output_table}__meta AS SELECT * FROM UNNEST([STRUCT(...)]) for the meta sibling. Shadow-swap fallback activates at > 50K rows. Every row in both tables carries the same compile_id; the meta row additionally carries full ontology_fingerprint and binding_fingerprint.

• cli.py:299 (the compile command) — gains two new flags: --emit-concept-index (boolean) and --concept-index-table <fqn> (required when --emit-concept-index is set — no silent global default). When both flags are absent, command output is byte-identical to today. No other CLI flags change.

• __init__.py — adds from .graph_ddl_compiler import compile_concept_index so the new public function is importable as from bigquery_ontology import compile_concept_index, matching the existing compile_graph re-export. No other exports change. _fingerprint stays unexported.

Unchanged:

• ontology_models.py — model changes for abstract: bool = False landed in feat(owl-import): SKOS support alongside OWL #62 (issue Feat: SKOS import support alongside OWL (design proposal — feedback wanted) #57). No further model changes for concept-index work.
• binding_models.py — no changes in v1. A binding-side index: opt-in block was considered and deferred to v2; precedence rule documented in the plan when/if it lands.
• All other bigquery_ontology/*.py — untouched.

New CLI surface summary:

gm compile \
  --ontology ontology.yaml \
  --binding binding.yaml \
  --emit-concept-index \
  --concept-index-table my-proj.my_ds.ontology_concept_index

Produces: the existing CREATE PROPERTY GRAPH DDL + two concept-index tables (ontology_concept_index and ontology_concept_index__meta). Re-running the same command produces byte-identical SQL — the compile_id is deterministic from inputs (no timestamps, no UUIDs).

Version bump: Minor — new public function (compile_concept_index) and new CLI flags. Existing API byte-identical.

---

SDK package (bigquery_agent_analytics) — changes

New files (all under src/bigquery_agent_analytics/):

• ontology_runtime.py. Hosts OntologyRuntime (the read accessor wrapper), the verification machinery (first-call + TTL re-check), and all four exception classes (ConceptIndexMismatchError, ConceptIndexProvenanceMissing, ConceptIndexInconsistentPair, ConceptIndexRefreshed). OntologyRuntime exposes two constructors — .load(ontology_path, binding_path, ...) and .from_models(ontology, binding, ...) — both routing through one shared implementation.

• entity_resolver.py. Hosts the EntityResolver Protocol (not ABC — duck-typed for modern typing), the Candidate and ResolveResult dataclasses, and two reference implementations: ExactMatchResolver (name + notation) and SynonymResolver (extends exact with label-based match). Candidate dedup: one candidate per entity, winning-label priority (name > pref > alt > hidden > synonym > notation, lexicographic tiebreaker), limit=N returns N distinct entities.

Modified files:

• __init__.py — adds to the existing try/except re-export block (same pattern as Client, CodeEvaluator, etc.):
  • OntologyRuntime — from .ontology_runtime
  • EntityResolver, ExactMatchResolver, SynonymResolver, Candidate, ResolveResult — from .entity_resolver
  • ConceptIndexMismatchError, ConceptIndexProvenanceMissing, ConceptIndexInconsistentPair, ConceptIndexRefreshed — from .ontology_runtime

Unchanged:

• All other SDK modules. The runtime accessor layer is strictly additive.

Read accessors on OntologyRuntime (pure-Python, no BQ round-trip):

Method	Returns	Notes
entities()	list[str]	Names of concrete + abstract entities
entity(name)	Entity	With annotations, synonyms, abstract flag
synonyms(name)	list[str]	Pref + alt + hidden labels
annotation(name, key)	str | None	E.g. skos:notation, skos:definition
in_scheme(scheme_name)	list[Entity]	Concepts in a skos:ConceptScheme
broader(name)	list[Entity]	skos:broader traversal
narrower(name)	list[Entity]	Inverse
related(name)	list[Entity]	skos:related traversal

Identity rules: entities are name-addressed (singular lookup); relationships are traversal-first, not name-addressed — a single skos_broader can repeat across endpoint pairs after #62's relaxed uniqueness, so a hypothetical rt.relationship(name) would have no single answer.

Validation accessor:

Method	Returns	Notes
validate_against_ontology(values, *, scheme=None, entity=None, sample_limit=20)	ValidationResult	scheme= and entity= are mutually exclusive; neither or both = ValueError. Bounded output via known_value_count + known_values_sample. candidates is None unless a resolver is explicitly composed by the caller.

Verification configuration (on construction):

Parameter	Default	Notes
verify_concept_index	"strict"	"strict" (raises on any provenance issue), "missing_ok" (tolerates missing meta), "off" (disables verification entirely — for read-only dashboards)
verify_ttl_seconds	60	0 = every-call check; None = snapshot-bound (verify once, never re-check)

Verification lifecycle:

1. Construction — OntologyRuntime.load(...) / .from_models(...) computes local ontology_fingerprint and binding_fingerprint (both full SHA-256). No BQ round-trip.
2. First concept-index access (lazy — not on construction) — reads the __meta sibling, compares fingerprints. Mismatch → ConceptIndexMismatchError. Missing meta → ConceptIndexProvenanceMissing.
3. TTL re-check (each resolve / validate call past the TTL window) — runs two queries:
   • SELECT DISTINCT compile_id FROM {output_table} LIMIT 2 — asserts exactly one value (pair consistency).
   • SELECT compile_id, ontology_fingerprint, binding_fingerprint FROM {output_table}__meta LIMIT 1 — full-fingerprint freshness.
   • Main/meta disagreement → 2s one-shot retry → persistent disagreement = ConceptIndexInconsistentPair.
   • Fingerprints drift from cache = ConceptIndexRefreshed.

The TTL re-check reading both tables with full fingerprints is a W2 watchpoint in the plan — a single-table sentinel or short-compile-id-only comparison reintroduces either the meta/main race or the 48-bit collision hole.

Resolver surface:

from bigquery_agent_analytics import (
    OntologyRuntime,
    ExactMatchResolver,
    SynonymResolver,
)

rt = OntologyRuntime.load(
    ontology_path="ontology.yaml",
    binding_path="binding.yaml",
    concept_index_table="my-proj.my_ds.ontology_concept_index",
    verify_concept_index="strict",    # default
    verify_ttl_seconds=60,            # default
)

resolver = SynonymResolver(runtime=rt)
result = resolver.resolve(
    input_value="Consumer Banking",
    scheme="BankingTaxonomy",         # scheme= XOR entity=
    limit=5,
)
# result.candidates: list[Candidate] with entity_name, matched_label, label_kind, scheme

Both reference resolvers query the concept index via BigQuery; ExactMatchResolver uses WHERE label = @input and SynonymResolver composes with label_kind preference ordering.

Version bump: Minor — new public API surface (OntologyRuntime, four resolver-related classes, four exception types). No existing behavior changes.

Existing user code: No deprecation. Users with their own resolution layers continue unaffected until they opt into the SDK primitive.

---

Sequencing (from the plan)

PR stack, in merge order:

1. A1 — _fingerprint.py — feat(ontology): A1 — internal _fingerprint module for concept-index provenance #71, open.
2. A2 — concept_index.py row builder.
3. A3-A5 — compile_concept_index + inline-UNNEST SQL emission.
4. A7 — CLI flags.
5. A8 (partial) — docs/ontology/concept-index.md.
6. B1-B7 — SDK read accessors + resolver Protocol + references (verification off as intermediate default).
7. C1-C6 — verification layer (strict default on) + shadow-swap full impl + all four exception types.
8. Phase 4 — integration tests, examples/concept_index_quickstart.py, full docs.
9. Phase 5 — contrib/ scaffolding (Yahoo advertising resolver when contributed).

Each PR leaves main shippable.

[caohy1988]

Updated the issue body based on an internal review. Three changes:

1. Added Index Population Contract (new subsection under Concept index materialization).

The original text said gm compile would "materialize" the index but didn't say who writes rows or when. Fair point — the existing graph_ddl_compiler.py is schema-only (DDL emission), so adding row population is a new kind of output that needs an explicit contract. The new subsection commits to:

• Compiler emits INSERT (or MERGE for large ontologies) alongside DDL in the same gm compile invocation.
• Idempotent re-runs via DELETE + INSERT — ontology YAML is the single source of truth, every compile is a full rewrite.
• No incremental builds, no separate gm build-concept-index command, no runtime lazy-build. One command, one source of truth.
• Inline INSERT up to ~50K rows; staging-table pattern above that. Both stay in pure SQL.
• Failure modes: compile fails loudly if rows can't land; the empty-table-existing state is legal and queries just return no matches.

Alternatives considered and rejected are listed inline.

2. Tightened relationship identity language in OntologyRuntime.

The original draft was implicitly correct — all relationship accessors were already entity-keyed traversal (rt.broader(entity), rt.related(entity)) — but didn't spell out why. After #57's uniqueness relaxation, a singular rt.relationship(name) is unsafe because abstract relationships can repeat across endpoint pairs.

New "Identity rules" block in section 1 says this plainly:

• Entities stay name-addressed.
• Relationships are traversal-first, not name-addressed.
• If a name-based relationship accessor is ever added, it must be (name, from, to) or list-returning. Never singular-by-name.

This also makes #57 and #58 explicit partners: #57 relaxes relationship uniqueness, #58 acknowledges that relaxation at the runtime surface and refuses to leak the name-unique assumption into runtime APIs.

3. Bounded validate() return shape.

Old shape returned full known_values, which is fine for small enums but awful for the motivating use case (hundreds of concepts, ~1K synonyms). Updated to:

• known_value_count — always the full count
• known_values_sample — capped at sample_limit (default 10)
• candidates — stays None unless composed with a resolver

Callers who need the full list call rt.in_scheme(...) or rt.entities() directly. validate is kept pure set-membership — ranking stays in resolver-land.

Thanks for the review — the Index Population Contract was the real gap. The other two were latent but load-bearing after #57.

[caohy1988]

Second round of updates — added a Library API impact section that nails down the three items flagged in review.

1. Compiler output contract.

compile_graph(ontology, binding) -> str stays unchanged. Its byte-identical-text contract is preserved. Concept-index emission ships as a new sibling function in the same module:

def compile_concept_index(ontology: Ontology, binding: Binding) -> str: ...

Separate deterministic string. CLI composes the two when --emit-concept-index is set. No breaking change to existing library callers; no compile_graph(...emit_concept_index=True...) overload that makes return shape depend on a flag; no CompileResult object that breaks string-expecting callers. Rejected alternatives are listed inline.

2. OntologyRuntime construction.

Two classmethods, both share one implementation:

OntologyRuntime.load(ontology_path, binding_path)    # file-based convenience
OntologyRuntime.from_models(ontology, binding)       # in-memory, for existing flows

from_models() is the integration path for the SDK's existing flows — runtime_spec.py:199 and similar can wrap validated models without round-tripping through disk. Matches how in-memory Ontology/Binding already move through the codebase.

3. Scope semantics.

Replaced the ambiguous entity=<name> with two explicit, mutually-exclusive parameters on both validate() and EntityResolver.resolve():

• scheme=<name> — resolve/validate against all members of a concept scheme. Motivating case (AdFormat, DMA, IAB).
• entity=<name> — resolve/validate against a single named entity. Identity check only.

Exactly one must be provided. Narrower-closure scoping (e.g., "all narrower-than some abstract node") is explicitly deferred as a future scope=Scope.narrower_closure(name) extension.

Polymorphism ("entity= means scheme-scoped when the named entity is a scheme, entity-scoped otherwise") was considered and rejected: two implementers following the spec would return different answers for the same call, and ontology changes (concrete → abstract-scheme-root) would silently change API semantics for every entity= call. Explicit parameters keep the contract boring and predictable.

Also updated the examples and the file-placement table to match. The EntityResolver Protocol signature now reflects the scheme= / entity= keyword parameters.

Thanks for the second pass — #58 is now concrete enough that implementers following the issue should land on the same API shape.

[caohy1988]

Third round — contract-hardening updates for the three findings.

1. Concurrency gap closed with CREATE OR REPLACE TABLE ... AS SELECT.

Switched the default emission shape from DELETE + INSERT to a single CREATE OR REPLACE TABLE ... AS SELECT * FROM UNNEST([STRUCT(...)]) statement. In BigQuery this is atomic from the reader's perspective — the previous table stays queryable until the replacement commits, then readers see the new rows. No empty-table-existing window.

The earlier "empty table is legal" wording was correct for an offline artifact but wrong for a runtime lookup primitive. Fixed. Failure-modes block updated accordingly — inline path is fully atomic; users can re-run gm compile without cleanup.

For large ontologies (>~50K rows), where query-text limits push against UNNEST(ARRAY<STRUCT<...>>), the compiler emits a shadow-table swap pattern (CREATE OR REPLACE shadow → DROP + RENAME). That path has a brief non-atomic window, which the issue now calls out explicitly: runtime readers on the shadow path must tolerate transient "table not found" or treat large-ontology refresh as an offline/admin operation. This is a documented limitation of BigQuery's current primitives, not a design choice.

Practical upshot: for the sizes the motivating use case actually has (~1K-10K rows for YAMO's 274 SKOS concepts with multi-label rows), everyone gets the atomic path.

2. Configuration surface narrowed to v1.

Changed from "--emit-concept-index or index: concept_index in binding" to:

• v1 ships CLI flag only. gm compile --emit-concept-index.
• Binding-side toggle deferred — would require schema and loader changes in binding_models + binding_loader.py that are worth scoping as their own change once CLI behavior settles.
• If v2 adds the binding path, precedence is explicit: CLI flag overrides binding setting; binding is the project default when the flag is absent.

Less surface to spec now; clean forward path if needed later.

3. Row-order contract pinned on compile_concept_index().

Added one sentence to extend compile_graph()'s byte-identical contract to the sibling:

Rows are sorted by a stable key before SQL generation: (scheme, entity_name, label_kind, language, label, notation), with NULLs ordered last consistently per column.

This makes diffing compile output, caching artifacts, and verifying that ontology edits produced only the expected row changes all tractable — same properties the existing compile_graph() contract already promises for DDL.

---

These were the last load-bearing contract gaps. Happy to tighten further if anything else surfaces, but at this point the issue should give implementers a concrete enough target to start on the primitives without guessing at the shape.

[caohy1988]

Fourth round — design-level gaps closed.

1. Index scope relative to binding subset.

Added an explicit "What's in the index" rule under the Index Population Contract:

• All abstract entities from the ontology, regardless of binding. They're informational-only, never bound by construction (Feat: SKOS import support alongside OWL (design proposal — feedback wanted) #57's binding rejection rule), and their whole point at runtime is being available for resolution even when the agent doesn't materialize them as tables.
• Only concrete entities that are bound in this binding. Concrete + unbound is deliberately excluded — returning matches the agent can't then query is worse than a miss.

Rule summarized: abstract: always. Concrete: iff bound.

This gives two different bindings over the same ontology different indexes (as expected — they're different deployments), but both share the same abstract taxonomy nodes. Preserves the runtime-materializability invariant on concrete elements while keeping the taxonomy-browse value of SKOS available.

Added an is_abstract BOOL NOT NULL column to the index schema so resolvers can filter at query time: WHERE NOT is_abstract for runtime-materializable matches only, both for taxonomy-aware "did you mean" suggestions.

2. Multi-scheme membership.

Pinned down the row multiplicity contract: one row per (entity_name, label, label_kind, language, scheme) tuple. Denormalized. A SKOS concept in 3 schemes × 5 labels produces 15 rows. Concepts with no scheme membership get scheme IS NULL rows — still indexed for entity= resolution, invisible to scheme= resolution.

Rationale for the denormalized shape (rejected alternatives inline):

• WHERE scheme = @x stays trivial and BQ-clusterable.
• ARRAY<STRING> scheme forces WHERE @x IN UNNEST(scheme) on every scheme-scoped query — harder to index, harder to write.
• Separate membership table forces a join on every resolver call, defeats the one-table-SQL-lookup simplicity.
• Row multiplication is bounded: (concepts × labels × schemes), linear and tractable at BQ scale.

Updated the example SQL queries to show the DISTINCT / GROUP BY entity_name pattern for collapsing denormalized rows. Updated the CREATE OR REPLACE TABLE example with multi-scheme and abstract-entity rows so the shape is concrete.

Also added is_abstract to the deterministic sort key: (scheme, entity_name, label_kind, language, label, notation, is_abstract). Technically redundant since is_abstract is determined by entity_name, but included for defensive stability if the invariant ever loosens.

---

Every design-level question that's come up across the four rounds now has an explicit answer in the issue body. Happy to keep tightening if anything else surfaces, but I agree with your assessment — the issue reads like an implementation target now.

[caohy1988]

Fifth round — both gaps addressed.

1. Table naming contract.

compile_concept_index() and gm compile now require an explicit output table name when --emit-concept-index is set. No silent global default:

def compile_concept_index(
    ontology: Ontology,
    binding: Binding,
    *,
    output_table: str,   # required — fully-qualified `project.dataset.table`
) -> str: ...

gm compile --emit-concept-index \
           --concept-index-table my-project.my_dataset.ontology_concept_index__retail

Rationale for "required" over auto-derived:

• Binding model doesn't currently carry a name field, so safe auto-derivation would require a model change — worth avoiding on the initial primitive.
• Hash-derived defaults (sha1(binding)[:8]) are collision-free but unreadable and change on trivial binding edits. Bad for a table name that appears in user-written resolver SQL.
• Explicit forces the deployment-level decision at compile time, where it belongs. Single-binding-per-dataset users pick any name; multi-binding users pick distinct names (..._retail, ..._investment_bank).

OntologyRuntime reads the index via a matching concept_index_table parameter at construction time. Name isn't stored on ontology or binding models — it's a runtime/deployment concern.

Updated both the inline-UNNEST and shadow-table-swap SQL examples to use {output_table} as a placeholder so the contract is consistent throughout.

2. Candidate dedup contract.

Rewrote the Candidate / ResolveResult dataclasses to make entity-level dedup explicit. New contract:

• ResolveResult.candidates has at most one entry per entity_name. Denormalized-index rows for the same entity collapse to a single candidate.
• limit=N means N distinct entities, not N raw rows.
• Winning-label rule: highest-confidence row under the resolver's rule wins. Ties broken by label_kind priority: name > pref > alt > hidden > synonym > notation. Further ties broken by lexicographic label for determinism.
• Added label_kind: str and scheme: Optional[str] to Candidate so the winning evidence is visible to the caller (agent can say "did you mean X — matched on altLabel from NielsenDMA?").
• reason vocabulary pinned: exact | notation | synonym | fuzzy | none.

This keeps the "agent-facing list of distinct concepts" semantics without forcing every caller to rediscover the dedup rule.

---

Glad the issue is converging. These were both real gaps the denormalization introduced; tightening them now means implementers won't have to invent answers in code review.

[caohy1988]

Sixth round — provenance contract added.

New "Provenance and compatibility contract" subsection under the Concept index materialization section. Summary:

• Compiler emits a sibling metadata table {output_table}__meta alongside the main index, in the same gm compile run. One row per compile, containing: ontology_name, ontology_fingerprint (SHA-256 of canonicalized Ontology YAML), binding_fingerprint (SHA-256 of canonicalized Binding YAML), target_project, target_dataset, compiler_version, compiled_at.
• OntologyRuntime computes the same fingerprints on loaded models at construction, reads the sibling meta lazily on first concept-index access, and raises ConceptIndexMismatchError if they don't match.
• Missing meta table raises distinct ConceptIndexProvenanceMissing — for hand-created indexes or older-toolchain output.
• Verification flag: verify_concept_index="strict" (default) / "missing_ok" / "off". Escape hatch for read-only dashboards and explicit "I know what I'm doing" paths.
• Verification is one-time per runtime instance and cached. Not on every resolve call.

Why sibling table over embedded columns: storage efficiency, cleaner diff review, easier SQL inspection. Rows stay lean.

Why not caller-managed (the reviewer's option 2): the plausible-but-wrong-match scenario is a feature bug the SDK shouldn't ship with. An operator wiring OntologyRuntime.from_models(ontology_A, binding_B, concept_index_table=table_C) where table_C was compiled from different inputs would get confident answers against unrelated data — strictly worse than getting no answers. v1 ships strict verification on by default, with documented escape hatches for callers who explicitly want to skip it.

Canonicalized YAML (keys sorted, whitespace normalized, comments stripped) makes fingerprints deterministic across serialization round-trips — the same (Ontology, Binding) pair always hashes to the same fingerprint regardless of formatting edits.

Rejected alternatives are listed inline: embedded provenance columns (bloats every row), BQ table OPTIONS(description=...) (elegant but higher tooling cost for plain-SQL consumers), caller-managed v1 (ships a known-bad default).

---

That closes the last design-level gap on my side. The issue is now a complete enough implementation target that an engineer working from it should land consistent shape without having to invent missing contracts in code review. Thanks for the iterative reviews — each round surfaced a real gap.

[caohy1988]

Seventh round — all four findings addressed.

1. Fingerprint algorithm, pinned to validated models.

Replaced "SHA-256 of canonicalized YAML" with an explicit model-based fingerprint contract:

• Hash is over a canonical JSON serialization of the validated Ontology / Binding Pydantic models, not raw YAML text.
• Canonicalization rules pinned: keys sorted at every nesting level, model_dump(mode="json", exclude_none=False) so defaults materialize consistently, enum values as canonical strings, None serialized as explicit null, list order preserved (semantically load-bearing for entity/property lists), UTF-8 with separators=(",", ":").
• Binding fingerprint covers declared structure, not ephemeral annotations (if any are added later).

Two YAML documents with different formatting / comment placement / emitter whitespace but identical validated models produce identical fingerprints. Semantic changes to the model produce different fingerprints. This is the layer where the rest of the SDK's determinism already lives (compile_graph(ontology, binding) takes models, not strings).

The strict verification gate no longer rejects non-semantic edits.

2. compile_concept_index() signature consistency.

The "Compiler output contract" section previously showed the old signature without output_table. Updated to match the "Table naming contract" section:

def compile_concept_index(
    ontology: Ontology,
    binding: Binding,
    *,
    output_table: str,   # required — see "Table naming contract" above
) -> str: ...

CLI composition example now calls it with output_table=args.concept_index_table. Every appearance of the function in the issue now uses the same signature.

3. Corrected binding-identifier rationale.

You're right — Binding.binding: str does exist (binding_models.py:159). Fixed the rationale:

• Old (wrong): "bindings don't carry a name field on the model"
• New: "binding: str exists, but it isn't a safe or stable source for a BQ table name — it's an identity tag for the binding document, not a deployment-unique BQ-legal identifier. Using it would couple operational naming to a field authors rename for non-operational reasons, and would collide across environments (dev/stage/prod) that share the same binding identity."

The requirement stays; the reasoning is now accurate.

4. Open Question 10 refreshed.

OQ 10 was written before the explicit scheme= / entity= decision landed and still framed the open question as "start with entity, add scheme-level later." Rewrote to reflect where the design actually ended up:

Should narrower-closure scoping ship in v1? The current proposal settled on two explicit parameters — scheme= for concept-scheme membership and entity= for single-entity identity. A third mode (narrower-closure: "resolve against all entities narrower-than some abstract node") is deferred. Proposal: ship scheme= and entity= only in v1; add scope=Scope.narrower_closure(name) in v2 if real callers need it. For most cases, scheme membership plus rt.narrower(entity) traversal covers the need without a new API.

Readers now see what's actually being proposed for v1 and what was deferred, instead of a question that contradicts the main body.

---

Thanks — these were all internal-consistency catches that would have confused implementers. Cross-cutting gaps like the fingerprint algorithm are exactly the kind of thing that slips through when an issue has been edited across multiple rounds.

[caohy1988]

Eighth round — both gaps closed.

1. Pair-consistency contract via compile_id.

Added a new "Pair-consistency contract" subsection under Provenance. Previously the two tables ({output_table} and {output_table}__meta) were updated as independent CREATE OR REPLACE TABLE statements, which made strict verification unreliable during refresh windows:

• new meta + old data → strict verification passes against stale data (plausible-but-wrong).
• new data + old meta → strict verification raises a false mismatch.

Fix: both tables carry a compile_id tag generated per compile run. One new column on the main table (compile_id STRING NOT NULL, every row shares the same value), one new field on the meta row. Write order is main first, meta second — readers never see "meta promising data that doesn't exist yet."

Runtime check on first concept-index access:

1. Read __meta.compile_id as expected_compile_id.
2. SELECT DISTINCT compile_id FROM {output_table} LIMIT 2 — verify exactly one compile_id, matching expected.
3. On mismatch: one-shot retry after 2s backoff (handles the narrow normal-refresh window).
4. On persistent mismatch: raise ConceptIndexInconsistentPair — distinct from ConceptIndexMismatchError so callers can branch on timing vs wiring.

The one-shot retry is deliberately small: a legitimate long window indicates operator misbehavior (concurrent compiles against the same table pair) and should fail loudly.

Rejected alternatives for pair consistency:

• Transactional multi-statement DDL — BQ doesn't support it for CREATE OR REPLACE TABLE.
• Shadow-version both tables + pointer indirection — three tables is significant operational complexity for a narrow window; the compile_id column gets the same correctness with far less surface.
• OPTIONS(description=...) tagging — INFORMATION_SCHEMA lookups with their own freshness semantics and more tooling surface.

Updated the verification-flag vocabulary so "strict" now covers fingerprint mismatch, missing meta, and persistent pair inconsistency as failure modes.

2. notation promoted to first-class row kind.

You're right that the old contract had notation as a column while Candidate.label_kind='notation' and the priority rule included it — no row-level path to produce that kind. Fixed by making notation a row kind:

• Schema: label_kind STRING NOT NULL now takes 'name' | 'pref' | 'alt' | 'hidden' | 'synonym' | 'notation'.
• For every entity with a skos:notation, the compiler emits one additional row with label_kind='notation' and label=<the notation value>.
• The notation column stays, carrying the entity's notation across all its rows — pure display/lookup convenience, no longer the sole source for notation matching.
• Resolvers searching by label naturally catch notation matches; no OR notation = @input required. label_kind='notation' provenance is visible on the winning Candidate.
• Priority rule is unchanged: name > pref > alt > hidden > synonym > notation. Notation matches rank below label matches in tie-breaking, which matches how humans read "did you mean" suggestions (proper labels over codes).

Added a notation row ('DMA', '807', 'notation', ...) to the example so the shape is concrete. Storage cost: one extra row per entity per notation, minor.

---

The issue should now be both semantically complete and internally consistent. Happy to keep tightening, but I think we're at the point where further changes risk being implementation-shape bikeshedding rather than closing real contract gaps.

[caohy1988]

Ninth round — determinism contradiction resolved and implementation surface honestly named.

1. compile_id made input-deterministic; compiled_at removed from emitted SQL.

Picked option (a): preserve the byte-identical output contract on compile_concept_index().

• compile_id is now sha256(ontology_fingerprint || binding_fingerprint || compiler_version)[:12] — fully derived from inputs. Same ontology + binding + compiler version → same compile_id → identical SQL text.
• compiled_at is removed from the meta row entirely. Operators who want compile timestamp visibility read it from INFORMATION_SCHEMA.TABLES.creation_time on the __meta table. BigQuery maintains that automatically.

Pair consistency still works under this deterministic scheme:

• Interleaved compiles with different inputs → different compile_ids → runtime check catches it.
• Interleaved compiles with identical inputs → identical compile_ids → data is also identical. Worst case is wasted work, not wrong data.

The Compiler output contract section now names inputs precisely: "same inputs for compile_concept_index() = (ontology, binding, output_table, compiler_version). Everything in the emitted SQL is derived from those four values. No per-run timestamps, UUIDs, or process identifiers."

Callers auditing compile output in code review can now diff against the previous compile and see only the changes caused by ontology/binding edits.

2. "Zero changes to existing modules" claim corrected.

You're right — supporting gm compile --emit-concept-index --concept-index-table ... requires edits to the existing compile command in src/bigquery_ontology/cli.py:299, not just additive imports. Rewrote the section:

• src/bigquery_ontology/cli.py:299 — compile command gains --emit-concept-index and --concept-index-table <name> flags. Without the flag, behavior is byte-identical to today.
• src/bigquery_ontology/graph_ddl_compiler.py — adds compile_concept_index(). compile_graph() itself is not modified.

The runtime accessor (OntologyRuntime) path remains purely additive in the SDK package — just new files, new functions, new classmethods. But the ontology-package side does touch existing code, and the issue now says so.

---

Agreed that further iterations risk being bikeshedding. I think #58 is done unless something surfaces during implementation.

[caohy1988]

Tenth round — both gaps addressed.

1. Long-lived runtime verification now has a TTL contract, not a once-per-lifetime cache.

You were right: "verify once, cache forever" defeats the strict-default argument for services that outlive an index refresh. Added a new "Long-lived runtime verification" block:

• After first success, OntologyRuntime caches expected compile_id + fingerprints per-instance.
• On each resolve/validate call, if the cached verification is older than verify_ttl_seconds (default 60), the runtime issues a cheap sentinel read: SELECT compile_id FROM {output_table}__meta LIMIT 1. Single row, clustered, negligible cost.
  • Matching compile_id → refresh cache timestamp, proceed.
  • Differing compile_id → raise new ConceptIndexRefreshed exception. Service operator recreates OntologyRuntime with updated models; the new instance's fingerprint verification catches whether the refresh is compatible.

Configuration:

• verify_ttl_seconds: int = 60 — default, balances staleness window against sentinel cost.
• verify_ttl_seconds=0 — check on every call. For low-QPS / high-correctness services.
• verify_ttl_seconds=None — snapshot-bound (old once-forever behavior). Explicit opt-in for callers coordinating refresh out-of-band.

Why TTL rather than check-every-call by default: sentinel is cheap but not free; 60s balances per-call cost against service-refresh cadences.

2. Shadow-swap path now spells out how the pair moves together.

The shadow path previously only covered {output_table}. Rewrote it to cover both tables in the pair and to document the two distinct non-atomicity windows honestly:

• Both {output_table}_shadow and {output_table}__meta_shadow are built first.
• Swap order: data first, meta second (same as inline-path write order).
• Two non-atomic windows:
  1. Table-existence window between DROP and RENAME on each table — readers get "table not found".
  2. Pair-inconsistency window between "main renamed" and "meta renamed" — readers see compile_id disagreement → ConceptIndexInconsistentPair.
• The pair-inconsistency window on the shadow path can exceed the inline-path's 2-second retry budget because large-ontology renames take longer. Strict verification will raise during legitimate shadow refresh if the sentinel samples during the swap. That's by-design, not a defect — silently serving old data against a new meta is the failure mode strict verification exists to prevent.

Operational contract spelled out:

• Treat shadow-path refresh as an offline/admin operation. Pause reader traffic during gm compile on the shadow path, or tolerate transient ConceptIndexInconsistentPair.
• For services that can't pause: use verify_concept_index="missing_ok" or a longer verify_ttl_seconds to reduce the likelihood of sampling in the window.
• A higher-level indirection ({output_table}__current pointer table) is noted as follow-up work if real users hit this constraint. Not in v1 — three levels of indirection is a lot of surface for a narrow window.

The inline-UNNEST path (default for <50K rows, including the motivating use cases) remains fully atomic per-statement and has neither window.

---

Both gaps were real — the shadow path in particular would have been a silent correctness hazard otherwise. Thanks for catching them.

[caohy1988]

Eleventh round — both gaps closed.

1. TTL re-check now runs the full pair-consistency check, not just a single-table sentinel.

Real catch. The old single-table sentinel (SELECT compile_id FROM __meta LIMIT 1) had a correctness hole: during the inline refresh window (main-written-first, meta-written-second), a reader could see the old meta compile_id (matches cache, accepted), then query the new main table and serve data from the refreshed index under stale verification.

Fix: TTL re-verification reads both tables:

1. SELECT DISTINCT compile_id FROM {output_table} LIMIT 2 — must return exactly one value.
2. SELECT compile_id FROM {output_table}__meta LIMIT 1 — must return that same value.
3. Both must equal the cached compile_id.

Outcomes:

• All three hold → refresh cache timestamp, proceed.
• Main and meta agree with each other but differ from cache → raise ConceptIndexRefreshed.
• Main and meta disagree (refresh in progress) → one-shot 2s retry, then ConceptIndexInconsistentPair.

Cost stays cheap: at most two rows from main's clustered column, exactly one row from meta. Per-TTL-window.

Added an explicit "Why the sentinel must read both tables, not just meta" paragraph that names the exact failure mode — so a future reader won't reintroduce the single-table sentinel as an optimization.

2. Shadow-path operational guidance corrected.

You're right — the earlier "use missing_ok to tolerate transient pair inconsistency" advice contradicted the verification-mode contract that says missing_ok still raises on pair inconsistency. The advice was wrong, the flag vocabulary was right.

Kept the flag vocabulary as-is (clean, minimal), fixed the guidance. Shadow-path services that can't pause traffic now have two honest options listed:

• Increase verify_ttl_seconds to reduce sampling probability inside the swap window. Trades staleness window for lower collision probability.
• Catch ConceptIndexInconsistentPair at the application layer and retry after a short delay. Cleanest at the service-mesh level where transient 5xx handling already exists.

No new verification mode introduced (didn't want to bloat the vocabulary for a narrow case). Kept missing_ok as its documented semantics without broadening it to cover pair-inconsistency transients.

---

These were both the kind of subtle concurrency gap that only shows up under specific interleavings — easy to miss when scanning the contract, critical once hit in production. Thanks for the close read.

[caohy1988]

Twelfth round — TTL freshness check tightened to full fingerprints.

Good catch. 48 bits of entropy on the truncated compile_id is a birthday-bound at ~16M compiles — small probability, but a strict verification contract shouldn't rely on "small probability" for the freshness signal.

Went with option (a): keep short compile_id for pair consistency (where it's only comparing two values on one dataset at one moment), add full-fingerprint freshness check on TTL re-verification.

Updated the TTL re-check contract:

1. SELECT DISTINCT compile_id FROM {output_table} LIMIT 2 — pair consistency.
2. SELECT compile_id, ontology_fingerprint, binding_fingerprint FROM {output_table}__meta LIMIT 1 — read compile_id and the full fingerprints.
3. main.compile_id == meta.compile_id (pair consistency, short id is fine here).
4. meta.compile_id == cached.compile_id AND meta.ontology_fingerprint == cached.ontology_fingerprint AND meta.binding_fingerprint == cached.binding_fingerprint (full-fingerprint freshness, 256-bit safety).

Added an explicit "Why the freshness check compares full fingerprints, not just compile_id" paragraph explaining the split:

• compile_id is a 12-hex truncation chosen for storage efficiency on the per-row column (repeats across every data row).
• Full fingerprints live in the single-row meta table where storage cost is negligible.
• Pair consistency (single-moment, single-dataset comparison) can run on the short id — 48 bits is overkill for that.
• Freshness (comparing cached state to meta) runs on the full 256-bit fingerprints so a hypothetical 48-bit collision gets caught.

Cost stays negligible: meta read now returns three columns instead of one from the same one-row query. No additional round-trips.

Cached state on the runtime now includes all three (compile_id + both fingerprints), which is what matches the fingerprints already computed at first verification — so first-verify and TTL-re-verify share the same cache shape.

---

That's a real contract-correctness fix, not a bikeshed. The 48-bit compile_id was fine for pair consistency but wrong as the sole freshness discriminator under strict semantics.

[caohy1988]

Posting the detailed implementation plan below as a comment for reference. Also lives in-repo at docs/implementation_plan_concept_index_runtime.md.

---

Implementation Plan: Concept Index + Runtime Entity Resolution

Scope

Implement the runtime entity resolution primitives specified in issue #58 on top of the SKOS import work in issue #57.

Two packages touched:

• bigquery_ontology — compiler, CLI, fingerprint infrastructure.
• bigquery_agent_analytics — OntologyRuntime, resolvers, verification.

This plan assumes issue #57 is either merged or lands in parallel. The concept index's value is ~80% from SKOS annotations (skos:notation, skos:prefLabel, skos:altLabel, skos:broader) being preserved through import.

Acceptance criteria

• gm compile --emit-concept-index --concept-index-table <fqn> produces a concept index + sibling meta table for an ontology + binding, with byte-identical SQL across runs.
• OntologyRuntime.load(...) / .from_models(...) wraps a validated (Ontology, Binding) pair and exposes read accessors over annotations, synonyms, notations, concept-scheme membership, and abstract-relationship traversal.
• EntityResolver Protocol + ExactMatchResolver and SynonymResolver references work against the concept index with correct dedup and scope semantics.
• Strict verification defaults on; first-call and TTL re-checks both enforce pair consistency + full-fingerprint freshness.
• All exception types wired: ConceptIndexMismatchError, ConceptIndexProvenanceMissing, ConceptIndexInconsistentPair, ConceptIndexRefreshed.
• Inline-UNNEST path is fully atomic per statement; shadow-swap fallback is documented as offline/admin.
• Test suite covers determinism, pair-consistency, TTL re-check, scope semantics, candidate dedup, bounded validation output, and shadow-path failure handling.

Work breakdown

Bucket A — ontology package (bigquery_ontology/)

Item	File	Additive?	Dependencies
A1	Internal fingerprint module (_fingerprint.py, new) — canonical model serialization + SHA-256. Shared implementation detail, not public API	new	none
A2	Concept-index row builder (concept_index.py, new) — iterates (ontology, binding), applies abstract-always / concrete-iff-bound rule, emits sorted row list	new	A1, #57's abstract field
A3	compile_concept_index() in graph_ddl_compiler.py	additive function in existing module	A1, A2
A4	Meta row emission inside compile_concept_index()	part of A3	A1
A5	Inline-UNNEST path SQL generation (CREATE OR REPLACE TABLE ... AS SELECT UNNEST(...)) with compile_id column	part of A3	A4
A6	Shadow-swap fallback for >50K rows	part of A3	A5
A7	CLI extension: --emit-concept-index + --concept-index-table in cli.py:299	edits existing command	A3
A8	Docs: docs/ontology/concept-index.md (new), update docs/ontology/cli.md and docs/ontology/compilation.md	docs	A3, A7

Bucket B — SDK package (bigquery_agent_analytics/)

Item	File	Additive?	Dependencies
B1	ontology_runtime.py (new) — OntologyRuntime class with load / from_models classmethods	new	A1 (shares fingerprint impl)
B2	Read accessors: entities(), entity(), synonyms(), annotation(), in_scheme(), broader(), narrower(), related()	part of B1	#57 abstract-relationship traversal semantics
B3	validate_against_ontology() with bounded output (known_value_count, known_values_sample, sample_limit, mutually-exclusive scheme= / entity=)	part of B1	B2
B4	entity_resolver.py (new) — EntityResolver Protocol, Candidate, ResolveResult dataclasses	new	none
B5	ExactMatchResolver — name + notation exact match via concept index	part of B4	B4, A3 schema
B6	SynonymResolver — extends B5 with label-based exact match	part of B4	B5
B7	Candidate dedup logic (one per entity, winning-label priority, limit=N distinct)	shared helper	B4

Bucket C — verification layer

Item	File	Notes
C1	Exception classes in ontology_runtime.py — ConceptIndexMismatchError, ConceptIndexProvenanceMissing, ConceptIndexInconsistentPair, ConceptIndexRefreshed	Public API surface
C2	First-call verification — read meta, compute local fingerprints, compare	Lazy (on first concept-index access, not construction)
C3	Pair-consistency check with 2s one-shot retry	Used by C2 and C4
C4	TTL re-check with full pair-consistency + full-fingerprint freshness	Two queries per stale call: SELECT DISTINCT compile_id FROM main LIMIT 2 + SELECT compile_id, ontology_fingerprint, binding_fingerprint FROM meta LIMIT 1
C5	verify_concept_index flag handling: "strict" (default), "missing_ok", "off"
C6	verify_ttl_seconds flag handling: 60 (default), 0 (every-call), None (snapshot-bound)

Bucket D — tests

Item	Scope
D1	Fingerprint determinism (non-semantic YAML edits produce identical fingerprints; semantic edits change them)
D2	Compile output byte-identical across runs for same (ontology, binding, output_table, compiler_version)
D3	Row scope (abstract always included; concrete iff bound; cross-cutting test with mixed ontology)
D4	Multi-scheme denormalization (concept in 2 schemes → 2 rows; DISTINCT entity_name in query returns 1)
D5	Notation as first-class row (notation matches via label = @input AND label_kind = 'notation')
D6	Candidate dedup: same entity via multiple labels/schemes → one candidate; limit=N returns N distinct entities
D7	Winning-label priority rule (name > pref > alt > hidden > synonym > notation, lexicographic tiebreaker)
D8	Scope semantics: scheme= and entity= mutually exclusive; neither = error; both = error
D9	Pair consistency: inconsistent pair triggers retry; persistent inconsistency raises ConceptIndexInconsistentPair
D10	TTL re-check: fresh cache skips BQ; stale cache runs full check; matching values refresh cache; differing values raise ConceptIndexRefreshed
D11	48-bit collision hypothetical: full fingerprints catch the collision that short compile_id doesn't (can simulate by injecting a crafted meta row)
D12	First-call verification: mismatched fingerprints raise ConceptIndexMismatchError; missing meta raises ConceptIndexProvenanceMissing
D13	Validation bounded output: known_values_sample capped at sample_limit; known_value_count correct; candidates=None unless composed
D14	Shadow-path emission correctness (>50K rows → both tables shadow-swap; compile_id present in both)
D15	Abstract-entity filter: resolver with WHERE NOT is_abstract returns only concrete; default returns both

Phase plan

Five phases, each independently mergeable. Each leaves main in a shippable state.

Phase 1 — Ontology compiler foundation (no runtime dependency)

Ships the compile-time half. No SDK code touched. Users can emit the concept index from CLI; nothing reads it yet.

Work: A1, A2, A3, A4, A5, A7, A8 (docs partial).

Tests: D1, D2, D3, D4, D5, D14 (shadow-path skeleton; full shadow test in Phase 3).

Definition of done: gm compile --emit-concept-index --concept-index-table <fqn> produces a valid, byte-identical concept index + meta table for a fixture ontology. Re-running produces identical SQL. SKOS-annotated ontologies produce notation rows.

Out of scope for this phase: shadow-swap (A6 stub only), any SDK consumer code, verification logic.

Phase 2 — SDK read accessors + resolver Protocol (no verification)

Ships OntologyRuntime with lookups but without strict verification. Users can resolve against the concept index if it exists; verification is "off" unless opted in later.

Work: B1, B2, B3, B4, B5, B6, B7, C1 (exception classes defined but not raised yet).

Tests: D6, D7, D8, D13, D15.

Definition of done: OntologyRuntime.load(...) wraps validated models, exposes all read accessors. ExactMatchResolver and SynonymResolver run against a concept index table and return correctly deduped candidates with scope semantics honored. Validation returns bounded output.

Out of scope: verification, TTL re-check, shadow-path.

Phase 3 — Verification layer (strict default on)

The correctness gate. Wires C2-C6 on top of Phase 2. Default changes from "off" to "strict".

Work: C2, C3, C4, C5, C6, A6 (full shadow-swap implementation).

Tests: D9, D10, D11, D12, D14.

Definition of done: Strict verification enforces pair consistency and full-fingerprint freshness on first access and on TTL expiry. All four exception types raise in their documented conditions. Shadow-path emission works for >50K-row fixtures with documented transient-failure behavior.

Special attention: preserve the TTL re-check reading BOTH tables with FULL fingerprints (watchpoint from review). Add a regression test specifically for the single-table-sentinel hole and the 48-bit-collision hypothetical.

Phase 4 — Integration, migration, docs

Work: integration tests across ontology → compile → runtime → resolve; end-to-end example in examples/; migration note for users who had local resolution code; full doc pass.

Definition of done: examples/concept_index_quickstart.py (or similar) runs end-to-end against a real BQ dataset using a fixture ontology. README section added. Migration note published.

Phase 5 — Contrib + polish

Ships reference resolver implementations beyond ExactMatchResolver / SynonymResolver as contrib/ packages. Yahoo's layered (IAB/DMA-tuned) resolver is an early candidate per the feedback gist.

Work: bigquery_ontology/contrib/advertising/ stub with Yahoo's resolver (if contributed). Additional domain packs (healthcare, finance) land later.

File-by-file changes

New files

• src/bigquery_ontology/_fingerprint.py — internal module (underscore prefix) with canonical JSON serialization of Pydantic models + SHA-256. Internal function: fingerprint_model(model: BaseModel) -> str. Short variant for compile_id: compile_id(ontology_fingerprint, binding_fingerprint, compiler_version) -> str. Not re-exported from any __init__.py. Shared implementation between compile_concept_index() (ontology package) and OntologyRuntime (SDK package) via absolute import from bigquery_ontology._fingerprint import .... Underscore prefix makes it clear this isn't semver-stable surface; it's an implementation detail both packages happen to need.
• src/bigquery_ontology/concept_index.py — row builder. Function: build_rows(ontology: Ontology, binding: Binding) -> list[ConceptIndexRow]. Applies "abstract always, concrete iff bound" rule. Emits one row per (entity_name, label, label_kind, language, scheme) tuple plus one notation row per entity per skos:notation. Sorts by (scheme, entity_name, label_kind, language, label, notation, is_abstract) with NULLs last. Not re-exported from bigquery_ontology/__init__.py in v1. Module is importable directly (from bigquery_ontology.concept_index import build_rows, ConceptIndexRow) for users who need pre-SQL row access — same pattern as the existing from bigquery_ontology.graph_ddl_compiler import compile_graph alongside the package-root export. Package-level re-export can be added later if a concrete caller appears; keeping it out of the root for v1 avoids growing semver surface ahead of need.
• src/bigquery_agent_analytics/ontology_runtime.py — OntologyRuntime class with classmethods, read accessors, validation, verification. Exception classes (ConceptIndexMismatchError etc.) live here too.
• src/bigquery_agent_analytics/entity_resolver.py — EntityResolver Protocol, Candidate, ResolveResult, ExactMatchResolver, SynonymResolver.
• docs/ontology/concept-index.md — user-facing documentation for --emit-concept-index, schema, provenance, verification modes.
• Test files mirroring the current repo test layout — SDK tests flat, ontology tests in a subdirectory:
  • tests/bigquery_ontology/test_fingerprint.py (for _fingerprint.py — tests import the underscore module directly)
  • tests/bigquery_ontology/test_concept_index.py
  • tests/bigquery_ontology/test_compile_concept_index.py
  • tests/test_ontology_runtime.py (SDK-level, top-level tests/ per current convention)
  • tests/test_entity_resolver.py
  • tests/test_verification.py

Modified files

• src/bigquery_ontology/graph_ddl_compiler.py — add compile_concept_index(ontology, binding, *, output_table) -> str. Preserve compile_graph() contract byte-identically. No changes to existing function bodies.
• src/bigquery_ontology/cli.py:299 — compile command gains --emit-concept-index and --concept-index-table flags. When absent, behavior is byte-identical to today.
• src/bigquery_ontology/__init__.py — add from .graph_ddl_compiler import compile_concept_index so the new public function is importable as from bigquery_ontology import compile_concept_index, matching the existing pattern for compile_graph (__init__.py:50 today).
• src/bigquery_agent_analytics/__init__.py — add the new public surface to the try/except re-export block (same pattern as Client, CodeEvaluator, etc.):
  • OntologyRuntime from .ontology_runtime
  • EntityResolver, ExactMatchResolver, SynonymResolver, Candidate, ResolveResult from .entity_resolver
  • ConceptIndexMismatchError, ConceptIndexProvenanceMissing, ConceptIndexInconsistentPair, ConceptIndexRefreshed from .ontology_runtime
• docs/ontology/cli.md — document new flags.
• docs/ontology/compilation.md — mention the sibling DML emitter.
• docs/ontology/owl-import.md — note that SKOS skos:notation lands as annotation (for Feat: SKOS import support alongside OWL (design proposal — feedback wanted) #57 compatibility), and will appear as a first-class concept-index row in the resolver surface.

Unchanged files

• src/bigquery_ontology/ontology_models.py — no model changes for this work (issue Feat: SKOS import support alongside OWL (design proposal — feedback wanted) #57 handles the abstract field separately).
• src/bigquery_ontology/binding_models.py — no changes in v1. Binding-side toggle is deferred per the issue.
• All other bigquery_agent_analytics/*.py — runtime accessor is purely additive; no existing file is edited.

Implementation watchpoints

From the final review pass, three watchpoints to preserve across this implementation and any future refactors:

W1 — Canonical serialization rules must match between compiler and runtime

The fingerprint must be computed identically on both sides. src/bigquery_ontology/_fingerprint.py is the single source of truth — both compile_concept_index() (writing meta) and OntologyRuntime (reading and comparing) import from it via absolute import. Regression test: round-trip a model through YAML → load → fingerprint, then edit the YAML's whitespace/comments, re-load, re-fingerprint, assert identical. Also test that semantic edits (rename entity, change target dataset) produce different fingerprints.

Pin specifically in the module docstring: Pydantic.model_dump(mode="json", by_alias=False, exclude_none=False) with keys sorted at every nesting level and json.dumps(..., sort_keys=True, separators=(",", ":"), ensure_ascii=False) for the final encoding. Never use yaml.dump() or str(model) for fingerprint input.

W2 — TTL re-check must read both tables with full fingerprints

A future "optimizer" might see the TTL path and try to reduce it to a single-table sentinel SELECT compile_id FROM meta LIMIT 1. That would silently reintroduce the old meta / new main race. Or might try to reduce to short-compile-id comparison only, reintroducing the 48-bit collision hole.

Guard: a regression test that mocks a race window (meta unchanged, main updated with different compile_id but matching short hash via crafted rows) and asserts the runtime catches it. Also a test that a single-table sentinel implementation fails the test. Comment in _ttl_recheck() citing the specific failure modes, with a link back to issue #58.

W3 — Shadow-path failure handling must match the documented operational contract

When the shadow path's DROP + RENAME pair fails mid-swap, the current contract is: raise cleanly, let the operator's retry detect orphaned shadow tables and resume. A tempting shortcut is to wrap this in background "self-healing" retry logic inside the compiler, which would mask partial-swap states from operators and break the "pause traffic during shadow refresh" guidance.

Guard: keep the compiler's shadow-swap path non-self-healing. The compiler detects orphaned shadow tables on its next invocation and resumes deterministically; it does not spin retry loops on its own. Test: inject a mid-swap failure, verify gm compile errors with a clear message; verify a subsequent gm compile completes the swap without recompiling.

Rollout notes

• Backward compatibility: gm compile without --emit-concept-index is byte-identical to today's output. Existing users see no behavioral change.
• Ontology package version bump: new public API (compile_concept_index, re-exported from bigquery_ontology/__init__.py) warrants a minor version bump. _fingerprint.py is internal and does not factor into semver.
• SDK version bump: new public API (OntologyRuntime, EntityResolver + ExactMatchResolver + SynonymResolver, dataclasses, exception classes, all re-exported from bigquery_agent_analytics/__init__.py) warrants a minor version bump.
• Existing resolution code in user applications: no deprecation. Users continue their existing resolution approach until they opt into the SDK primitive.
• BQ permissions: --emit-concept-index requires bigquery.tables.create on the target dataset (existing compile_graph() requirement, unchanged). Runtime reading requires bigquery.tables.getData on the concept index and meta tables (standard).

Open watchlist (not blocking, track during implementation)

• Behavior when ontology has >100K concepts — current plan emits shadow-swap at >50K; may need a LOAD-job path at the next order of magnitude. Out of scope for v1; track via GitHub issue if real users hit this.
• Pointer-indirection ({output_table}__current) as a future mitigation for shadow-path transient failures. Explicitly deferred per issue Feat: Runtime entity resolution primitives — OntologyRuntime, concept index, EntityResolver protocol (design proposal — feedback wanted) #58; track for v2 if real users request.
• asyncio variants of EntityResolver.resolve() — v1 ships sync only; add async later if adoption signals need.
• Binding-side opt-in (index: block on Binding model) — v1 ships CLI-only; add binding toggle in v2 with explicit precedence rule.

Estimated effort

Rough sizing in engineering-weeks, single developer, including tests and docs:

• Phase 1 (ontology compiler): 2 weeks
• Phase 2 (SDK read + resolver): 2 weeks
• Phase 3 (verification layer): 2 weeks (most subtle; plan for iteration on the retry / TTL semantics)
• Phase 4 (integration + migration + docs): 1 week
• Phase 5 (contrib): 0.5 week for scaffolding + review of Yahoo's contribution when ready

Total: ~7.5 weeks of focused work. Can run Phases 1 and 2 in parallel across two developers for ~4 weeks wall-clock.

References

• Issue Feat: Runtime entity resolution primitives — OntologyRuntime, concept index, EntityResolver protocol (design proposal — feedback wanted) #58: Feat: Runtime entity resolution primitives — OntologyRuntime, concept index, EntityResolver protocol (design proposal — feedback wanted) #58
• Issue Feat: SKOS import support alongside OWL (design proposal — feedback wanted) #57: Feat: SKOS import support alongside OWL (design proposal — feedback wanted) #57
• Feedback gist (original motivating use case): https://gist.github.com/haiyuan-eng-google/54c3d3366b3d75b659561ef4e24e9374
• Migration assessment memory: project_ontology_migration_assessment.md — the SDK→upstream adapter pattern informs where runtime filtering lives