Skip to content

Coverage

Accuracy Matrix

Doc Claims Code sources Status Fix plan
01-overview.md Privacy‑first; realtime via Convex; Postgres history; OpenRouter cost trusted; no prompts/outputs app/api/ingest/route.ts (no prompts required; cost logic), convex/*, prisma/schema.prisma, lib/pricing.ts ✅ Keep aligned as features evolve
02-quickstart.md Track with RunForge.track; envs; streaming tip; sample data sdk-ts/index.ts, sdk-py/runforge.py, examples/* ✅ Generator script optional
03-architecture.md Dual‑store; ingestion & sync; mermaid diagrams; idempotency by id convex/runs.ts, app/api/internal/pg-sync/route.ts ✅ N/A
04-data-model.md Run fields include costUSD, latency, status, promptHash/preview prisma/schema.prisma ✅ Includes costSource, costEstimated
05-apis.md Ingest schema with costSource/Estimated; metrics ranges and shape; badge & sync app/api/ingest/route.ts, app/api/metrics/route.ts, app/api/badge/experiment/[id]/route.ts, app/api/internal/pg-sync/route.ts ✅ N/A
08-convex-realtime.md Tables/indices; functions; corrected fields list convex/schema.ts, convex/runs.ts, convex/kpis.ts, convex/runsActions.ts ✅ N/A
09-sdks-typescript.md RunForge.track docs; OpenRouter‑first; streaming tip sdk-ts/index.ts, sdk-ts/autoExtractor.ts ✅ N/A
10-sdks-python.md RunForge.track docs; OpenRouter‑first sdk-py/runforge.py ✅ N/A
11-configuration-env.md Env vars list lib/auth.ts, app/api/*, lib/crypto.ts ✅ N/A
12-security-privacy.md Never store prompts/outputs; hash optional; AES‑GCM BYOK app/api/ingest/route.ts (hashPrompt), lib/crypto.ts ✅ Note SHA‑256 (no HMAC) now; TODO keyed hashing
13-pricing-entitlements.md Plans + entitlements mirror convex/schema.ts ⚠️ Entitlements not enforced in OSS template
15-observability-alerts.md Metrics ranges; alerts scaffolding app/api/metrics/route.ts, convex/alerts.ts ⚠️ Alerts not implemented yet; keep TODO
api/ingest.md Auth, schema, cost recompute app/api/ingest/route.ts, lib/pricing.ts ✅ N/A
api/metrics.md Ranges, response app/api/metrics/route.ts ✅ N/A
api/projects.md Projects & keys creation app/api/projects/* ✅ N/A

Key contracts cross‑check

  • /api/ingest schema: matches code, including costSource and costEstimated.
  • Cost recompute: OpenRouter trusted; others via registry — implemented.
  • Convex realtime: runs_live includes costSource and costEstimated.
  • Prisma Run model: has costUSD, costSource, costEstimated.
  • SDKs: RunForge.track present in TS and Python; client marks estimated; server verifies.
  • /api/metrics: supports ?range=24h|7d|30d, aggregate, returns { points, costByProvider?, costByModel?, series }.

Notes

  • Naming: runs_live.id is the run id; docs reflect idempotency by id set from runId.
  • Privacy hashing: server supports promptHash and optional promptPreview; prefer hashing or omission.
  • Auth: Ingest uses per‑project API keys (see convex/apiKeys.ts).
  • Examples: minimal Node/Python examples included.

Final audit report

  • Files scanned: app/api/, convex/, prisma/, sdk-ts/, sdk-py/, docs/, examples/, lib/
  • Status: Core APIs and SDKs documented accurately; integration guides covered; minor follow-ups noted above.
  • New examples: examples/openrouter-node/index.ts, examples/openai-node/index.ts, examples/openai-python/main.py
  • New script: scripts/generate-sample-runs.ts