Skip to content

Overview

See also: - docs/api/ingest.md - docs/api/metrics.md - docs/api/projects.md - docs/api/internal-pg-sync.md - docs/api/badge-experiment.md

POST /api/ingest

  • Auth (OSS template):
  • Authorization: Bearer <INGEST_API_KEY> or header x-api-key: <INGEST_API_KEY>
  • projectId is required in the request body.
  • TODO: Switch to per‑project API keys (see convex/apiKeys.ts) and derive project from the key.
  • Rate limiting: RF_RATE_LIMIT_PER_MINUTE per project (default 1000/min).
  • Privacy: do not send prompts or outputs; only usage/cost/latency/metadata.
  • Request supports single or batch: 
    // excerpt from app/api/ingest/route.ts
const LLMRunInputSchema = z.object({
  id: z.string().optional(),
  provider: z.string(),
  model: z.string(),
  // privacy: do not send prompt/completion
  prompt: z.string().optional(),
  completion: z.string().optional(),
  tokens: z.object({ input: z.number(), output: z.number() }).optional(),
  cost: z.number().optional(),
  latency: z.number().optional(),
  timestamp: z.string().optional(),
  metadata: z.record(z.string(), z.any()).optional(),
  inputTokens: z.number().int().nonnegative().optional(),
  outputTokens: z.number().int().nonnegative().optional(),
  costUSD: z.number().nonnegative().optional(),
  latencyMs: z.number().int().nonnegative().optional(),
  status: z.enum(["success","error","timeout"]).optional(),
  errorCode: z.string().optional(),
  promptHash: z.string().optional(),
  promptPreview: z.string().max(160).optional(),
  experimentId: z.string().optional(),
  traceId: z.string().optional(),
  idempotencyKey: z.string().optional(),
  runId: z.string().optional(),
  // cost tracking
  costSource: z.enum(["provider", "catalog", "estimated"]).optional(),
  costEstimated: z.boolean().optional(),
})
  • Cost recompute behavior (server):
  • OpenRouter: if providerProvidesDirectCost(provider) and costUSD>0, trust provider cost → costSource="provider", costEstimated=false.
  • Others or when cost missing: compute from registry → calculateCostFromTokens(model, inputTokens, outputTokens) → costSource="catalog", costEstimated reflects pricing entry.
  • Responses:
  • Single: { ok: true, id }
  • Batch: { ok: true, processed, ids }

Example (single): 

curl -X POST "$BASE/api/ingest" \
  -H "Authorization: Bearer $INGEST_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "projectId": "<projectId>",
    "provider": "openai", "model": "gpt-4o",
    "inputTokens": 100, "outputTokens": 50,
    "costUSD": 0, "latencyMs": 1200, "status": "success"
  }'

GET /api/metrics

  • Auth: Bearer project key (dev mocked in lib/auth.ts when RUNFORGE_DEV_MODE=1).
  • Query params:
  • range=24h|7d|30d (default 7d)
  • projectId=<id> (optional; required when auth cannot infer project)
  • aggregate=1|true (optional; when set, aggregates across all projects)
  • Response shape: 
    {
  "points": [
    { "t": "2025-01-01T01:00:00.000Z", "costUSD": 0.12345, "errorRate": 0.01, "p95LatencyMs": 850, "runs": 42, "tokensIn": 1200, "tokensOut": 800 }
  ],
  "costByProvider": { "openai": 1.23, "openrouter": 0.45 },
  "costByModel": { "gpt-4o-mini": 0.67 },
  "series": [
    { "t": "2025-01-01T01:00:00.000Z", "costUSD": 0.12345, "errorRate": 0.01, "p95LatencyMs": 850 }
  ]
}

POST /api/internal/pg-sync

  • Purpose: Convex→Postgres durable sync webhook
  • Auth: HMAC header x-rf-signature computed with RUNFORGE_SYNC_SIGN
  • Body: JSON of a single runs_live doc (validated)

GET /api/badge/experiment/[id]

  • Returns SVG badge or { status: "pass"|"fail" } (JSON)

See also: api/ingest.md, api/metrics.md, convex/*, 07-backend.md.

  • OTel: optional traceId propagation — include in request metadata and it will be stored as traceId.
  • Privacy: RunForge never stores prompts or outputs. Only usage/cost/latency/metadata is ingested. Prefer promptHash over promptPreview.