Merge pull request #155 from Entrolution/feat/hybrid-default-entity-extraction

Hybrid retrieval default, entity extraction, temporal misrouting fix

Author: gvonness-apolitical

CHANGELOG.md

@@ -5,6 +5,30 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.10.1] - 2026-03-13
+
+### Added
+
+- **Entity extraction** (`src/ingest/entity-extractor.ts`): Deterministic regex-based extraction of people (@mentions, emails, "X said" patterns), channels (#channels), meetings (standup, retro, 1:1), and URLs from chunk content. Skips code blocks and `[Thinking]` blocks to reduce noise.
+- **Entity store** (`src/storage/entity-store.ts`): CRUD layer for entities, aliases, and chunk mentions. Supports alias resolution, re-ingestion safety (INSERT OR IGNORE), and per-entity chunk lookup capped at 100 most recent.
+- **Entity-aware retrieval**: Entity mentions in queries are matched against stored entities and injected as an RRF source (weight 1.5) in both keyword and hybrid search paths. Project-scoped — gracefully skips when no project filter is provided.
+- **Entity tables** (migration v16): Three new tables (`entities`, `entity_aliases`, `entity_mentions`) with cascade deletes and appropriate indexes.
+- **Entity count in stats**: The `stats` MCP tool now reports entity count.
+
+### Changed
+
+- **Hybrid retrieval default**: `retrieval.primary` changed from `'keyword'` to `'hybrid'`. Vector search is now always active at ~14ms cost (local jina-small), which covers narrative/thematic projects without per-project configuration. Backward compatible — `retrieval.primary: 'keyword'` in config still works.
+- **Temporal misrouting fix**: Updated `search` and `recall` tool descriptions to redirect recent/latest session queries to `reconstruct`. Updated `reconstruct` description to explicitly claim temporal queries.
+- **MCP tool descriptions**: `search` now mentions hybrid retrieval and entity boosting; `recall` and `reconstruct` include temporal routing guidance.
+
+### Fixed
+
+- **MCP integration test timeout**: Increased `beforeAll` hook timeout from 10s to 30s to accommodate heavy module imports (ONNX runtime, tree-sitter, LanceDB).
+
+### Tests
+
+- 2378 tests passing.
+
 ## [0.10.0] - 2026-03-12
 
 ### Added

docs/guides/how-it-works.md

@@ -208,6 +208,19 @@ The pipeline automatically selects the index-based or chunk-based search path at
 
 Both paths converge at cluster expansion, which always operates on chunk IDs and their chunk-level cluster assignments.
 
+### Entity Boosting
+
+During ingestion, Causantic extracts named entities from chunk content using deterministic regex patterns (no LLM required):
+
+- **People**: `@mentions`, email addresses, "X said"/"with X" patterns
+- **Channels**: `#channel` references
+- **Meetings**: Keywords like standup, retro, 1:1, sync
+- **URLs**: Full URL patterns
+
+Entities are resolved to canonical forms with alias tracking (e.g., `@joel` and `Joel` map to the same entity). At query time, if the search query contains recognisable entity references, matching chunks are injected as an additional RRF source with a 1.5x boost weight. This means searching for "@joel" surfaces all chunks mentioning Joel alongside semantically relevant results, without requiring exact keyword matches in every chunk.
+
+Entity extraction skips code blocks and `[Thinking]` blocks to avoid false positives from speculative content.
+
 ### Recall/Predict (episodic)
 
 The `recall` and `predict` tools reconstruct narrative chains:

docs/index.md

@@ -28,7 +28,7 @@ Deep dives into specific topics:
 Technical reference documentation:
 
 - [CLI Commands](reference/cli-commands.md) - Command-line interface reference
-- [MCP Tools](reference/mcp-tools.md) - MCP server tool documentation (9 tools)
+- [MCP Tools](reference/mcp-tools.md) - MCP server tool documentation (10 tools)
 - [Configuration Reference](reference/configuration.md) - All configuration options
 - [Storage API](reference/storage-api.md) - Storage layer internals
 - [Skills Reference](reference/skills.md) - Skill templates for Claude Code

docs/reference/configuration.md

@@ -103,6 +103,12 @@ Controls the semantic index layer, which generates normalised index entries for
 
 When enabled, each chunk gets an LLM-generated description (~130 tokens) at ingestion time. These descriptions are embedded and searched instead of raw chunks, providing uniform information density. See [How It Works](../guides/how-it-works.md#semantic-index) for details.
 
+## Entity Extraction
+
+Entity extraction runs automatically during ingestion with no configuration required. It uses deterministic regex patterns to identify people (`@mentions`, emails, "X said"), channels (`#channel`), meetings (standup, retro, 1:1), and URLs. Extracted entities are stored with alias resolution and used as an RRF boost source (weight 1.5) during search.
+
+Entity extraction skips code blocks and `[Thinking]` blocks to reduce false positives. The feature is always-on with no configuration knobs — it adds zero latency to queries that don't contain entity references.
+
 ## Length Penalty Settings
 
 ### `lengthPenalty`
@@ -138,10 +144,12 @@ Controls time-decay scoring for search results.
 
 Controls the search retrieval pipeline.
 
-| Property    | Type     | Default | Description                                                        |
-| ----------- | -------- | ------- | ------------------------------------------------------------------ |
-| `mmrLambda`      | `number` | `0.7`   | MMR (Maximal Marginal Relevance) lambda parameter (0-1)            |
-| `feedbackWeight` | `number` | `0.1`   | Weight applied to implicit relevance feedback signals (0-1)        |
+| Property           | Type     | Default    | Description                                                        |
+| ------------------ | -------- | ---------- | ------------------------------------------------------------------ |
+| `primary`          | `string` | `"hybrid"` | Primary retrieval method: `"keyword"`, `"vector"`, or `"hybrid"` (BM25 + vector + RRF) |
+| `vectorEnrichment` | `boolean`| `false`    | Use vector search to enrich keyword results when primary is `"keyword"`. No effect in hybrid mode. |
+| `mmrLambda`        | `number` | `0.7`      | MMR (Maximal Marginal Relevance) lambda parameter (0-1)            |
+| `feedbackWeight`   | `number` | `0.1`      | Weight applied to implicit relevance feedback signals (0-1)        |
 
 MMR reranks search results to balance relevance with diversity. After RRF fusion and cluster expansion, candidates are reordered so that semantically redundant chunks yield to novel ones.
 

docs/reference/mcp-tools.md

@@ -16,7 +16,7 @@ All tools return plain text responses via the MCP `content` array with `type: "t
 
 ### search
 
-Search memory semantically to discover relevant past context. Returns ranked results using hybrid BM25 + vector search with RRF fusion, cluster expansion, and MMR diversity reranking.
+Search memory to discover relevant past context. Uses hybrid (BM25 + vector) retrieval with entity boosting. Returns ranked results by relevance. For recent/latest session queries, use `reconstruct` instead.
 
 **Parameters**:
 
@@ -38,7 +38,7 @@ Found 5 relevant memory chunks (1200 tokens):
 
 ### recall
 
-Recall episodic memory by walking backward through causal chains to reconstruct narrative context. Seeds are found by semantic search; the causal graph unfolds them into ordered chains; chains are ranked by aggregate semantic relevance per token. Falls back to search results when no viable chain is found.
+Recall episodic memory by walking backward through causal chains to reconstruct narrative context. Seeds are found by semantic search; the causal graph unfolds them into ordered chains; chains are ranked by aggregate semantic relevance per token. Falls back to search results when no viable chain is found. For recent/latest session queries, use `reconstruct` instead.
 
 **Parameters**:
 
@@ -158,7 +158,7 @@ Returns `"No sessions found for project "[name]"."` if none match.
 
 ### reconstruct
 
-Rebuild session context for a project. Call with just `project` to get the most recent history up to the token budget (timeline mode). Optionally specify a time range with `from`/`to`, `days_back`, `session_id`, or `previous_session`.
+Use this for all recent/latest/last session queries. Rebuild session context for a project. Call with just `project` to get the most recent history up to the token budget (timeline mode). Optionally specify a time range with `from`/`to`, `days_back`, `session_id`, or `previous_session`.
 
 **Parameters**:
 
@@ -200,12 +200,13 @@ Show memory statistics including version, chunk/edge/cluster counts, and per-pro
 **Example**:
 
 ```
-Causantic v0.9.4
+Causantic v0.10.1
 
 Memory Statistics:
 - Chunks: 1234
 - Edges: 5678
 - Clusters: 42
+- Entities: 89
 
 Projects:
 - my-app: 800 chunks (Jan 2025 – Feb 2025)

package-lock.json

@@ -1,6 +1,6 @@
 {
   "name": "causantic",
-  "version": "0.10.0",
+  "version": "0.10.1",
   "description": "Long-term memory for Claude Code — local-first, graph-augmented, self-benchmarking",
   "type": "module",
   "private": false,

package.json

@@ -71,7 +71,9 @@ async function main() {
   console.log(`Starting index backfill (limit=${limit})...\n`);
 
   const status = indexRefresher.getBackfillStatus();
-  console.log(`Current status: ${status.indexed}/${status.total} indexed (${status.remaining} remaining)\n`);
+  console.log(
+    `Current status: ${status.indexed}/${status.total} indexed (${status.remaining} remaining)\n`,
+  );
 
   const result = await indexRefresher.backfill({
     limit,
@@ -91,12 +93,16 @@ async function main() {
   console.log(`  Duration: ${(result.durationMs / 1000).toFixed(1)}s`);
 
   const finalStatus = indexRefresher.getBackfillStatus();
-  console.log(`\nFinal status: ${finalStatus.indexed}/${finalStatus.total} indexed (${finalStatus.remaining} remaining)`);
+  console.log(
+    `\nFinal status: ${finalStatus.indexed}/${finalStatus.total} indexed (${finalStatus.remaining} remaining)`,
+  );
 
   // Show generation method breakdown
   const db = getDb();
   const methods = db
-    .prepare('SELECT generation_method, COUNT(*) as cnt FROM index_entries GROUP BY generation_method')
+    .prepare(
+      'SELECT generation_method, COUNT(*) as cnt FROM index_entries GROUP BY generation_method',
+    )
     .all() as Array<{ generation_method: string; cnt: number }>;
   console.log('\nGeneration methods:');
   for (const m of methods) {

scripts/backfill-index.ts

@@ -101,8 +101,8 @@ Pass these to the \`search\` MCP tool:
 ## Guidelines
 
 - **Always pass the \`project\` parameter** scoped to the current project (derive from the working directory) unless the user explicitly asks for cross-project results
-- By default, search uses **keyword-first (BM25)** retrieval — great for exact matches on function names, error codes, and specific terms
-- Optional vector enrichment can be enabled in config for semantic similarity matching
+- By default, search uses **hybrid (BM25 + vector)** retrieval with entity boosting — combines exact keyword matching with semantic similarity
+- For recent/latest session queries, use \`reconstruct\` instead
 - Use \`search\` for discovery, \`recall\` for narrative reconstruction
 - Combine with \`/causantic-recall\` when you need causal chain context (how things led to outcomes)
 `,

src/cli/skill-templates.ts

@@ -640,9 +640,7 @@ export class ClusterManager {
    * Creates clusters in the `index_entry_clusters` table (parallel to `chunk_clusters`).
    * Each cluster elects a representative (entry closest to centroid) for browsing.
    */
-  async reclusterIndexEntries(
-    options: ClusteringOptions = {},
-  ): Promise<ClusteringResult> {
+  async reclusterIndexEntries(options: ClusteringOptions = {}): Promise<ClusteringResult> {
     const startTime = Date.now();
     const { minClusterSize = this.config.minClusterSize } = options;
 
@@ -657,9 +655,7 @@ export class ClusterManager {
         PRIMARY KEY (index_entry_id, cluster_id)
       )
     `);
-    db.exec(
-      'CREATE INDEX IF NOT EXISTS idx_iec_cluster ON index_entry_clusters(cluster_id)',
-    );
+    db.exec('CREATE INDEX IF NOT EXISTS idx_iec_cluster ON index_entry_clusters(cluster_id)');
 
     // Clear existing index entry cluster assignments
     db.exec('DELETE FROM index_entry_clusters');