Skip to content

review-pluginQA.md

Latest commit

 

History

History
167 lines (123 loc) · 7.74 KB

review-pluginQA.md

File metadata and controls

167 lines (123 loc) · 7.74 KB

NAAP Plugin Framework — DevX Quality Assessment

Date: 2026-02-18 Branch: feat/architecture-refactoring-simplification-devx Assessor: Architecture Simplification Review Scope: Developer Experience (DevX), Architecture Simplicity, Completeness, Extensibility

Assessment Framework (100 Points)

Category Weight Description
A. Developer First Experience (Day 0) /20 CLI scaffold, first build, first mount — how fast can a new developer ship?
B. Inner Loop DevX (Day 1-30) /20 HMR, debugging, error messages, test loop, standalone dev mode
C. Architecture Simplicity /15 Number of concepts, layers of indirection, canonical patterns
D. Completeness /15 Does the framework cover the full lifecycle without escape hatches?
E. Extensibility & Composability /10 Plugin-to-plugin communication, shared services, integration points
F. Integration Overhead /10 How many packages, configs, and files does a plugin developer touch?
G. Quality & Safety /10 Test coverage, validation, security, runtime error handling

Before / After Comparison

Summary of Changes

Fix Before After
1. Mount Pattern 2 patterns: manual createRoot+ShellProvider (scaffold) vs createPlugin() (13 plugins). New devs copy wrong example. 1 pattern: createPlugin() everywhere. Scaffold, hello-world, todo-list all unified.
2. Vite Configs 3 factories: createPluginConfig (used), createPluginUMDConfig (unused), createPluginViteConfig (unused) 1 active factory: createPluginConfig. Others deleted or deprecated with migration guides.
3. Runtime Validation Cryptic errors: jsx is not a function, Cannot read properties of undefined Human-readable: [NAAP Plugin Error] Plugin "X" failed during mount: ✗ mount() is missing. → Quick fix: use createPlugin()...
4. CDN Discovery 14-entry hardcoded PLUGIN_DIR_MAP. New plugins require shell code changes. Automatic toKebabCase() conversion. Zero shell changes needed for new plugins.
5. Package Model 5+ packages for plugin devs to discover 2 documented packages: @naap/plugin-sdk (runtime) + @naap/plugin-build (build)

Category Scores

A. Developer First Experience (Day 0) — 17/20

Criterion Score Notes
CLI scaffold works out of box 5/5 naap-plugin create generates working plugin with createPlugin()
First build succeeds 4/5 UMD builds pass; standalone dev needs npm run dev setup
First mount in shell works 4/5 Scaffold output matches all 13 production plugins exactly
Documentation guides new devs 4/5 README, quickstart, and guide updated. Could add video walkthrough.

Before: 13/20 — Scaffold generated wrong pattern; confused new developers After: 17/20 (+4) — Single canonical pattern, matching documentation

B. Inner Loop DevX (Day 1-30) — 16/20

Criterion Score Notes
HMR works for plugin development 4/5 enablePluginHMR() available, naap-plugin dev exists
Error messages are actionable 5/5 Runtime validation with fix suggestions, createPlugin() migration hints
Test loop is fast 4/5 naap-plugin test runs vitest; integration tests need running shell
Standalone dev mode available 3/5 main.tsx added to examples; scaffold generates it

Before: 11/20 — Cryptic errors, no runtime validation After: 16/20 (+5) — Human-readable errors with fix suggestions

C. Architecture Simplicity — 13/15

Criterion Score Notes
Single canonical mount pattern 5/5 createPlugin() is the only pattern. Deprecated alternatives marked.
Minimal config factories 4/5 1 active (createPluginConfig). Unused deleted/deprecated.
Low concept count 4/5 Plugin = App + createPlugin + vite config. Clean 3-file pattern.

Before: 8/15 — 2 mount patterns, 3 vite configs, confusing choices After: 13/15 (+5) — One of each, clearly documented

D. Completeness — 12/15

Criterion Score Notes
Full lifecycle coverage 4/5 Create → dev → test → build → package → publish → install/uninstall
CDN serving is automatic 5/5 toKebabCase() replaces hardcoded map; zero config for new plugins
Database schema integration 3/5 Prisma multiSchema works but manual merge step still needed

Before: 10/15 — CDN required manual shell code changes After: 12/15 (+2) — Automatic discovery, no shell changes

E. Extensibility & Composability — 7/10

Criterion Score Notes
Plugin-to-plugin events 4/5 EventBus with request/response pattern; well-typed
Shared service contracts 3/5 Dashboard data provider contract exists; could add more

Before: 7/10 — No change (extensibility was already decent) After: 7/10 (0) — Not in scope for this round

F. Integration Overhead — 9/10

Criterion Score Notes
Minimal package dependencies 5/5 2 packages documented: plugin-sdk + plugin-build
Minimal config files 4/5 1 vite.config.ts, 1 plugin.json, standard tsconfig

Before: 6/10 — 5+ packages to understand, multiple config options After: 9/10 (+3) — Clear 2-package model with re-exports

G. Quality & Safety — 9/10

Criterion Score Notes
Test coverage 4/5 242 SDK tests, 19 BDD tests, 19 CDN tests = 280 total
Runtime validation 5/5 validatePluginModule(), validateShellContext(), construction-time checks

Before: 7/10 — 218 SDK tests, no runtime validation After: 9/10 (+2) — 280 tests, comprehensive runtime validation

Overall Score

Category Before After Delta
A. Developer First Experience 13 17 +4
B. Inner Loop DevX 11 16 +5
C. Architecture Simplicity 8 13 +5
D. Completeness 10 12 +2
E. Extensibility & Composability 7 7 0
F. Integration Overhead 6 9 +3
G. Quality & Safety 7 9 +2
Total 62 83 +21

Test Evidence

Suite Count Status
SDK unit tests 242 ✅ All passing
BDD lifecycle tests 19 ✅ All passing
CDN serve tests 19 ✅ All passing
Plugin builds verified 4/8 ✅ marketplace, community, plugin-publisher, capacity-planner
Regression guard 8/8 ✅ No regressions

Baseline (main): 218 SDK + 19 BDD = 237 tests After: 242 SDK + 19 BDD + 19 CDN = 280 tests (+43 new tests)

Commits (5 atomic, independently revertible)

  1. fix(devx): unify mount pattern to createPlugin() in scaffold and examples
  2. chore(devx): remove unused vite config factories, deprecate UMD helpers
  3. feat(devx): add runtime contract validation with human-readable errors
  4. fix(devx): replace PLUGIN_DIR_MAP with deterministic camelCase-to-kebab conversion
  5. feat(devx): consolidate type re-exports for 2-package developer model

Remaining Opportunities (Future Work)

Priority Item Estimated Impact
P1 Add naap-plugin doctor command for environment diagnostics +2 on Day 0
P1 Automated schema merge for database integration +2 on Completeness
P2 Plugin-to-plugin shared state contract (beyond events) +2 on Extensibility
P2 Visual plugin testing harness (mount/unmount in isolation) +2 on Inner Loop
P3 Video walkthrough for onboarding +1 on Day 0
P3 Plugin marketplace search/filter in CLI +1 on Completeness