Merge branch 'main' into feat/repository-analytics

Author: gaspergrom

.claude/settings.json

@@ -0,0 +1,6 @@
+{
+  "enabledPlugins": {
+    "typescript-lsp@claude-plugins-official": true,
+    "pyright-lsp@claude-plugins-official": true
+  }
+}

.claude/skills/scaffold-snowflake-connector/SKILL.md

@@ -196,6 +196,8 @@ After Step 1 and/or Step 2, build a column registry per table:
 
 **Store this as the canonical column reference. Every column name used in generated code must appear in this registry. Never assume or invent a column name.**
 
+**Flag non-VARCHAR column types** (e.g., `DATE`, `TIME`, `TIMESTAMP_TZ`, `BOOLEAN`, `NUMBER`) — these arrive as native JS types from the Parquet reader, not strings (see touch point 9 rules).
+
 For each JOIN table, check whether any existing transformer in `services/apps/snowflake_connectors/src/integrations/` queries from the same table. If yes, inherit its column mappings; if no, treat every column as unknown and derive it from sample data in the Pre-Analysis step below.
 
 ### Step 3 — Sample data
@@ -342,7 +344,10 @@ After all identity fields are confirmed, summarize how `buildMemberIdentities()`
 
 ### 3b. Organization Mapping
 
-If Pre-Analysis determined there is no org data (no org-related columns found in any table), confirm: "I don't see any organization columns in the schema. Does this source have org/company data?" — if yes, proceed; if no, skip to 3c.
+If Pre-Analysis determined there is no org data (no org-related columns found in any table): before asking the user, first read existing transformers in `services/apps/snowflake_connectors/src/integrations/` to check whether any of them join an org table using a key that also exists in the user's tables. If a match is found, prompt the user:
+> "I don't see org columns in the tables you provided, but [EXISTING_PLATFORM] sources org data from `{ORG_TABLE}` via `{join_key}` — which also appears in your table. Did you mean to include this? (Recommended)"
+
+If no existing pattern is joinable, ask: "I don't see any org columns. Does this source have org/company data?" — if yes, ask for the table; if no, skip to 3c.
 
 If Pre-Analysis identified org columns:
 
@@ -565,6 +570,7 @@ File: `services/apps/snowflake_connectors/src/integrations/{platform}/{source}/b
 **Rules (enforced — do not deviate):**
 - Use explicit column names only. Do not use `table.*` or `table.* EXCLUDE (...)` in new implementations — existing sources (TNC, CVENT) use these patterns but new sources should list columns explicitly to avoid parquet encoding/decoding issues
 - If any TIMESTAMP_TZ columns exist in the schema, exclude and re-cast them as TIMESTAMP_NTZ (see CVENT pattern)
+- Do not concatenate or transform date/time columns in SQL — keep them as separate columns and let the transformer handle type coercion (see touch point 9 rules)
 - Follow the CTE structure:
   1. `org_accounts` CTE (if org data present)
   2. `CDP_MATCHED_SEGMENTS` CTE (always)
@@ -585,6 +591,8 @@ Show the full generated file and ask for confirmation before writing.
 File: `services/apps/snowflake_connectors/src/integrations/{platform}/{source}/transformer.ts`
 
 **Rules (enforced — do not deviate):**
+
+- **Parquet type coercion — never blindly cast `row.COLUMN as string`.** Snowflake types may arrive as native JS types after Parquet decoding (e.g., `DATE` → `Date` object, `TIME` → `number` in ms, `BOOLEAN` → `boolean`). Always check the Snowflake column type from the schema registry and handle the actual JS type the Parquet reader delivers — do not assume every column is a string.
 - All string comparisons must be case-insensitive: use `.toLowerCase()` on both sides of comparison only; preserve the original value in the output
 - No broad `else` statements — every branch must have an explicit condition
 - All column names referenced in code must exactly match the schema registry — never assumed

.gitignore

@@ -196,13 +196,13 @@ services/libs/tinybird/.diff_tmp
 # custom cursor rules
 .cursor/rules/*.local.mdc
 
-# claude code
+# claude
+.claude/settings.local.json
 .claude/cache/
 .claude/tmp/
 .claude/logs/
 .claude/sessions/
 .claude/todos/
-.claude/settings.local.json
 
 # git integration test repositories & output
 services/apps/git_integration/src/test/repos/

CLAUDE.md

@@ -0,0 +1,71 @@
+# CDP — Community Data Platform
+
+CDP is a community data platform by the Linux Foundation. It ingests millions of
+activities and events daily from platforms like GitHub, GitLab, and many others
+(not just code hosting). Open-source projects get onboarded by connecting
+integrations, and data flows continuously at scale.
+
+The ingested data is often messy. A big part of what CDP does is improve data quality: deduplicating member and organization profiles through merge and unmerge operations, enriching data via third-party providers, and resolving identities across sources. The cleaned data powers analytics and insights for LFX products.
+
+The codebase started as crowd.dev, an open-source startup later acquired by the Linux Foundation. Speed was prioritized over standards, but the platform is now stable. The focus has shifted to maintainable patterns, scalability, and good developer experience. Performance matters at this scale, even small inefficiencies compound across millions of data points.
+
+## Tech stack
+
+TypeScript, Node.js, Express, PostgreSQL (pg-promise), Temporal, Kafka, Redis, OpenSearch, Zod, Bunyan, AWS S3.
+
+Vue 3, Vite, Tailwind CSS, Element Plus, Pinia, TanStack Vue Query, Axios.
+
+Package manager is **pnpm**. Monorepo managed via pnpm workspaces.
+
+## Codebase structure
+
+```
+backend/          -> APIs (public endpoints for LFX products + internal for CDP UI)
+frontend/         -> CDP Platform UI
+services/apps/    -> Microservices — Temporal workers, Node.js workers, webhook APIs
+services/libs/    -> Shared libraries used across services
+```
+
+`services/libs/common` holds shared utilities, error classes,
+and helpers. If a piece of logic is reusable (not business logic), it belongs there.
+
+`services/libs/data-access-layer` holds all
+database query functions. Check here before writing new ones — duplicates are
+already a problem.
+
+## Patterns in transition
+
+Old and new patterns coexist. Always use the new pattern.
+
+- **Sequelize -> pg-promise**: Sequelize is legacy (backend only). Use
+  `queryExecutor` from `@crowd/data-access-layer` for all new database code.
+- **Classes -> functions**: Class-based services and repos are legacy. Write
+  plain functions — composable, modular, easy to test.
+- **Multi-tenancy -> single tenant**: Multi-tenancy is being phased out. The
+  tenant table still exists. Code uses `DEFAULT_TENANT_ID` from `@crowd/common`.
+  Don't add new multi-tenant logic.
+- **Legacy auth -> Auth0**: Auth0 is the current auth system. Ignore old JWT
+  patterns.
+- **Zod for validation**: Public API endpoints use Zod schemas with
+  `validateOrThrow`. Follow this pattern for all new endpoints.
+
+## Working with the database
+
+Millions of rows. Every query matters.
+
+- Look up the table schema and indexes before writing any query. Don't select
+  or touch columns blindly.
+- Check existing functions in `data-access-layer` before writing new ones.
+  Weigh the blast radius of modifying a shared function — sometimes a new
+  function is safer.
+- Write queries with performance in mind. Think about what indexes exist, what
+  the query plan looks like, and whether you're scanning more rows than needed.
+
+## Code quality
+
+- Functional and modular. Code should be easy to plug in, pull out, and test
+  independently.
+- Think about performance at scale, even for small changes.
+- Define types properly — extend and reuse existing types. Don't sprinkle `any`.
+- Don't touch working code outside the scope of the current task.
+- Prefer doing less over introducing risk. Weigh trade-offs before acting.