From 49e63b9bcdd33ac4f4c7ee2d391ff2e4e637d4d6 Mon Sep 17 00:00:00 2001 From: Eric Allam Date: Wed, 11 Mar 2026 14:41:51 +0000 Subject: [PATCH 1/7] feat: add app concept for project-level crumb isolation Every crumb is now stamped with an app name, auto-detected from the nearest package.json. Crumbs are stored per-app at ~/.agentcrumbs// and all CLI commands scope to the current app by default. - Add app field to Crumb type and AgentCrumbsConfig - Add getApp() resolution: JSON config > AGENTCRUMBS_APP env > package.json - Per-app storage directories in collector/store - Collector routes incoming crumbs to per-app stores - All CLI commands support --app and --all-apps flags - New cli/app-store.ts shared helper for app context - Update format.ts to show app prefix in multi-app views - Add papertrail example app for demos - Update skills, docs, and homepage --- docs/content/docs/cli/other.mdx | 11 +- docs/content/docs/cli/query.mdx | 8 + docs/content/docs/cli/tail.mdx | 8 + docs/content/docs/config/env-var.mdx | 19 ++ docs/content/docs/crumb-format.mdx | 4 +- docs/next-env.d.ts | 2 +- docs/src/app/page.tsx | 1 + examples/papertrail/package.json | 17 ++ examples/papertrail/src/api.ts | 172 ++++++++++++++++ examples/papertrail/src/index.ts | 105 ++++++++++ examples/papertrail/src/pipeline.ts | 188 ++++++++++++++++++ examples/papertrail/src/store.ts | 122 ++++++++++++ examples/papertrail/src/types.ts | 40 ++++ examples/papertrail/tsconfig.json | 12 ++ packages/agentcrumbs/.tshy/build.json | 8 + packages/agentcrumbs/.tshy/commonjs.json | 17 ++ packages/agentcrumbs/.tshy/esm.json | 16 ++ packages/agentcrumbs/bin/intent.js | 0 .../agentcrumbs/skills/agentcrumbs/SKILL.md | 26 ++- .../skills/agentcrumbs/init/SKILL.md | 34 +++- .../agentcrumbs/src/__tests__/env.test.ts | 65 +++++- .../agentcrumbs/src/__tests__/trail.test.ts | 75 ++++++- packages/agentcrumbs/src/cli/app-store.ts | 50 +++++ .../agentcrumbs/src/cli/commands/clear.ts | 23 ++- .../agentcrumbs/src/cli/commands/collect.ts | 6 +- .../agentcrumbs/src/cli/commands/follow.ts | 14 +- .../agentcrumbs/src/cli/commands/query.ts | 12 +- .../agentcrumbs/src/cli/commands/replay.ts | 17 +- .../agentcrumbs/src/cli/commands/sessions.ts | 42 ++-- .../agentcrumbs/src/cli/commands/stats.ts | 55 ++++- packages/agentcrumbs/src/cli/commands/tail.ts | 18 +- packages/agentcrumbs/src/cli/format.ts | 29 ++- packages/agentcrumbs/src/cli/index.ts | 5 + packages/agentcrumbs/src/collector/server.ts | 30 ++- packages/agentcrumbs/src/collector/store.ts | 49 ++++- packages/agentcrumbs/src/env.ts | 63 ++++++ packages/agentcrumbs/src/trail.ts | 3 +- packages/agentcrumbs/src/types.ts | 2 + pnpm-lock.yaml | 67 +++++-- pnpm-workspace.yaml | 1 + 40 files changed, 1319 insertions(+), 117 deletions(-) create mode 100644 examples/papertrail/package.json create mode 100644 examples/papertrail/src/api.ts create mode 100644 examples/papertrail/src/index.ts create mode 100644 examples/papertrail/src/pipeline.ts create mode 100644 examples/papertrail/src/store.ts create mode 100644 examples/papertrail/src/types.ts create mode 100644 examples/papertrail/tsconfig.json create mode 100644 packages/agentcrumbs/.tshy/build.json create mode 100644 packages/agentcrumbs/.tshy/commonjs.json create mode 100644 packages/agentcrumbs/.tshy/esm.json mode change 100644 => 100755 packages/agentcrumbs/bin/intent.js create mode 100644 packages/agentcrumbs/src/cli/app-store.ts diff --git a/docs/content/docs/cli/other.mdx b/docs/content/docs/cli/other.mdx index c4ea0ac..a673810 100644 --- a/docs/content/docs/cli/other.mdx +++ b/docs/content/docs/cli/other.mdx @@ -3,20 +3,25 @@ title: "Other commands" description: "stats, clear, follow, replay, sessions" --- +All commands below accept `--app ` to scope to a specific app and `--all-apps` to include all apps. Default is auto-detect from `package.json`. + ## `agentcrumbs stats` Show crumb counts, file size, and active services. ```bash -agentcrumbs stats +agentcrumbs stats # current app +agentcrumbs stats --all-apps # per-app breakdown ``` ## `agentcrumbs clear` -Delete all stored crumbs. +Delete stored crumbs. ```bash -agentcrumbs clear +agentcrumbs clear # clear current app +agentcrumbs clear --all-apps # clear all apps +agentcrumbs clear --app foo # clear a specific app ``` ## `agentcrumbs sessions` diff --git a/docs/content/docs/cli/query.mdx b/docs/content/docs/cli/query.mdx index 3172d44..14e54ef 100644 --- a/docs/content/docs/cli/query.mdx +++ b/docs/content/docs/cli/query.mdx @@ -20,6 +20,8 @@ agentcrumbs query --since 5m | `--tag ` | Filter by tag | | `--session ` | Filter by session ID | | `--match ` | Text search | +| `--app ` | Scope to a specific app (default: auto-detect from package.json) | +| `--all-apps` | Query crumbs from all apps | | `--json` | JSON output | | `--limit ` | Maximum number of results | @@ -45,4 +47,10 @@ agentcrumbs query --since 1h --json --limit 50 # Text search agentcrumbs query --since 24h --match "connection refused" + +# Query a specific app +agentcrumbs query --since 1h --app my-project + +# Query across all apps +agentcrumbs query --since 5m --all-apps ``` diff --git a/docs/content/docs/cli/tail.mdx b/docs/content/docs/cli/tail.mdx index 5a81436..3b17194 100644 --- a/docs/content/docs/cli/tail.mdx +++ b/docs/content/docs/cli/tail.mdx @@ -19,6 +19,8 @@ agentcrumbs tail | `--tag ` | Filter by tag | | `--match ` | Filter by content | | `--session ` | Filter by session ID | +| `--app ` | Scope to a specific app (default: auto-detect from package.json) | +| `--all-apps` | Show crumbs from all apps | | `--json` | JSON output (for piping to jq, etc.) | ### Examples @@ -37,6 +39,12 @@ agentcrumbs tail --match "userId:123" # Filter by session agentcrumbs tail --session a1b2c3 +# Scope to a specific app +agentcrumbs tail --app my-project + +# Show crumbs from all apps +agentcrumbs tail --all-apps + # JSON output for piping agentcrumbs tail --json | jq '.data.userId' ``` diff --git a/docs/content/docs/config/env-var.mdx b/docs/content/docs/config/env-var.mdx index 77ca194..bff0832 100644 --- a/docs/content/docs/config/env-var.mdx +++ b/docs/content/docs/config/env-var.mdx @@ -43,16 +43,35 @@ AGENTCRUMBS='{"ns":"*","port":9999}' # JSON output format (instead of pretty) AGENTCRUMBS='{"ns":"*","format":"json"}' + +# Explicit app name +AGENTCRUMBS='{"app":"my-project","ns":"*"}' ``` ## Config schema | Field | Type | Default | Description | | --- | --- | --- | --- | +| `app` | `string` | (auto-detect) | App name. Defaults to nearest `package.json` name | | `ns` | `string` | (required) | Namespace filter pattern | | `port` | `number` | `8374` | Collector HTTP port | | `format` | `"pretty"` \| `"json"` | `"pretty"` | Output format for stderr | +## App name + +Every crumb is stamped with an app name. This keeps crumbs from different projects separate. + +The app name is resolved in this order: +1. `app` field in the JSON config +2. `AGENTCRUMBS_APP` environment variable +3. Auto-detected from the nearest `package.json` name field (walking up from `cwd`) +4. Fallback: `"unknown"` + +```bash +# Override via dedicated env var +AGENTCRUMBS_APP=my-project AGENTCRUMBS=1 node app.js +``` + ## Namespace patterns - `*` matches everything diff --git a/docs/content/docs/crumb-format.mdx b/docs/content/docs/crumb-format.mdx index ffbf45f..47b8517 100644 --- a/docs/content/docs/crumb-format.mdx +++ b/docs/content/docs/crumb-format.mdx @@ -9,6 +9,7 @@ Each crumb is a JSON object. When stored, they are written as JSONL (one JSON ob ```json { + "app": "my-project", "ts": "2026-03-07T10:00:00.123Z", "ns": "auth-service", "msg": "user logged in", @@ -28,6 +29,7 @@ Each crumb is a JSON object. When stored, they are written as JSONL (one JSON ob | Field | Type | Required | Description | | --- | --- | --- | --- | +| `app` | `string` | Yes | App name (auto-detected from `package.json` or explicit config) | | `ts` | `string` | Yes | ISO 8601 timestamp | | `ns` | `string` | Yes | Namespace | | `msg` | `string` | Yes | Message | @@ -57,4 +59,4 @@ Each crumb is a JSON object. When stored, they are written as JSONL (one JSON ob ## Storage -Crumbs are stored in `~/.agentcrumbs/crumbs.jsonl` by default. One JSON object per line, no trailing comma, no wrapping array. +Crumbs are stored per-app at `~/.agentcrumbs//crumbs.jsonl`. One JSON object per line, no trailing comma, no wrapping array. The app name is auto-detected from the nearest `package.json` by default. diff --git a/docs/next-env.d.ts b/docs/next-env.d.ts index c4b7818..9edff1c 100644 --- a/docs/next-env.d.ts +++ b/docs/next-env.d.ts @@ -1,6 +1,6 @@ /// /// -import "./.next/dev/types/routes.d.ts"; +import "./.next/types/routes.d.ts"; // NOTE: This file should not be edited // see https://nextjs.org/docs/app/api-reference/config/typescript for more information. diff --git a/docs/src/app/page.tsx b/docs/src/app/page.tsx index 5f2e587..483b25b 100644 --- a/docs/src/app/page.tsx +++ b/docs/src/app/page.tsx @@ -258,6 +258,7 @@ export default function HomePage() {
## agentcrumbs
+
App: my-project
### Namespaces
diff --git a/examples/papertrail/package.json b/examples/papertrail/package.json new file mode 100644 index 0000000..160d46a --- /dev/null +++ b/examples/papertrail/package.json @@ -0,0 +1,17 @@ +{ + "name": "papertrail", + "version": "0.0.0", + "private": true, + "type": "module", + "scripts": { + "dev": "tsx src/index.ts", + "dev:debug": "AGENTCRUMBS=1 tsx src/index.ts" + }, + "dependencies": { + "agentcrumbs": "workspace:*" + }, + "devDependencies": { + "tsx": "^4.19.0", + "typescript": "^5.7.0" + } +} diff --git a/examples/papertrail/src/api.ts b/examples/papertrail/src/api.ts new file mode 100644 index 0000000..3c2e955 --- /dev/null +++ b/examples/papertrail/src/api.ts @@ -0,0 +1,172 @@ +import { trail } from "agentcrumbs"; // @crumbs + +import type { Document, PaginatedResult, ProcessedDocument } from "./types.js"; +import { processDocument } from "./pipeline.js"; +import { getDocument, listDocuments, deleteDocument, getStats } from "./store.js"; + +const crumb = trail("api-gateway"); // @crumbs + +let nextId = 1; + +function generateId(): string { + return `doc_${String(nextId++).padStart(4, "0")}`; +} + +// Fake auth tokens +const validTokens = new Map([ + ["tok_admin_001", { userId: "user_1", role: "admin" as const }], + ["tok_user_002", { userId: "user_2", role: "viewer" as const }], +]); + +type AuthContext = { userId: string; role: "admin" | "viewer" }; + +function authenticate(token: string | undefined): AuthContext { + crumb("authenticating", { tokenPrefix: token?.slice(0, 8) }, { tags: ["auth"] }); // @crumbs + + if (!token) { + crumb("auth failed: no token", undefined, { tags: ["auth", "error"] }); // @crumbs + throw new Error("Missing authentication token"); + } + + const auth = validTokens.get(token); + if (!auth) { + crumb("auth failed: invalid token", { tokenPrefix: token.slice(0, 8) }, { tags: ["auth", "error"] }); // @crumbs + throw new Error("Invalid authentication token"); + } + + crumb("auth success", { userId: auth.userId, role: auth.role }, { tags: ["auth"] }); // @crumbs + return auth; +} + +export type ApiResponse = { + ok: boolean; + data?: T; + error?: string; +}; + +export async function uploadDocument( + token: string | undefined, + filename: string, + content: string, + mimeType: string = "text/plain" +): Promise> { + const reqCrumb = crumb.child({ requestId: generateId(), endpoint: "upload" }); // @crumbs + + return reqCrumb.scope("handle-upload", async () => { // @crumbs + try { + const auth = authenticate(token); + reqCrumb("upload request", { // @crumbs + filename, // @crumbs + mimeType, // @crumbs + contentLength: content.length, // @crumbs + userId: auth.userId, // @crumbs + }); // @crumbs + + if (auth.role !== "admin") { + reqCrumb("upload denied: insufficient permissions", { role: auth.role }, { tags: ["auth", "error"] }); // @crumbs + return { ok: false, error: "Insufficient permissions" }; + } + + const doc: Document = { + id: generateId(), + filename, + content, + mimeType, + uploadedAt: new Date().toISOString(), + }; + + reqCrumb("document created", { id: doc.id }); // @crumbs + + crumb.time("full-pipeline"); // @crumbs + const processed = await processDocument(doc); + crumb.timeEnd("full-pipeline", { id: doc.id }); // @crumbs + + reqCrumb("upload complete", { id: processed.id }, { tags: ["perf"] }); // @crumbs + return { ok: true, data: processed }; + } catch (err) { + reqCrumb("upload error", { // @crumbs + error: err instanceof Error ? err.message : String(err), // @crumbs + }, { tags: ["error"] }); // @crumbs + return { ok: false, error: err instanceof Error ? err.message : "Unknown error" }; + } + }); // @crumbs +} + +export async function fetchDocument( + token: string | undefined, + id: string +): Promise> { + const reqCrumb = crumb.child({ requestId: generateId(), endpoint: "fetch" }); // @crumbs + + return reqCrumb.scope("handle-fetch", async () => { // @crumbs + try { + authenticate(token); + reqCrumb("fetching document", { id }); // @crumbs + + const doc = await getDocument(id); + if (!doc) { + reqCrumb("document not found", { id }, { tags: ["error"] }); // @crumbs + return { ok: false, error: "Document not found" }; + } + + reqCrumb("fetch complete", { id }); // @crumbs + return { ok: true, data: doc }; + } catch (err) { + return { ok: false, error: err instanceof Error ? err.message : "Unknown error" }; + } + }); // @crumbs +} + +export async function listAllDocuments( + token: string | undefined, + page: number = 0, + pageSize: number = 10 +): Promise>> { + const reqCrumb = crumb.child({ requestId: generateId(), endpoint: "list" }); // @crumbs + + return reqCrumb.scope("handle-list", async () => { // @crumbs + try { + authenticate(token); + reqCrumb("listing documents", { page, pageSize }); // @crumbs + + const result = await listDocuments(page, pageSize); + + reqCrumb("list complete", { total: result.total, returned: result.items.length }); // @crumbs + return { ok: true, data: result }; + } catch (err) { + return { ok: false, error: err instanceof Error ? err.message : "Unknown error" }; + } + }); // @crumbs +} + +export async function removeDocument( + token: string | undefined, + id: string +): Promise> { + const reqCrumb = crumb.child({ requestId: generateId(), endpoint: "delete" }); // @crumbs + + return reqCrumb.scope("handle-delete", async () => { // @crumbs + try { + const auth = authenticate(token); + + if (auth.role !== "admin") { + reqCrumb("delete denied: insufficient permissions", { role: auth.role }, { tags: ["auth"] }); // @crumbs + return { ok: false, error: "Insufficient permissions" }; + } + + reqCrumb("deleting document", { id }); // @crumbs + const deleted = await deleteDocument(id); + + reqCrumb("delete complete", { id, deleted }); // @crumbs + return { ok: true, data: { deleted } }; + } catch (err) { + return { ok: false, error: err instanceof Error ? err.message : "Unknown error" }; + } + }); // @crumbs +} + +export function healthCheck(): ApiResponse<{ status: string; stats: ReturnType }> { + const stats = getStats(); + crumb("health check", stats); // @crumbs + return { ok: true, data: { status: "healthy", stats } }; +} diff --git a/examples/papertrail/src/index.ts b/examples/papertrail/src/index.ts new file mode 100644 index 0000000..33e16c9 --- /dev/null +++ b/examples/papertrail/src/index.ts @@ -0,0 +1,105 @@ +import { + uploadDocument, + fetchDocument, + listAllDocuments, + removeDocument, + healthCheck, +} from "./api.js"; + +const ADMIN_TOKEN = "tok_admin_001"; +const VIEWER_TOKEN = "tok_user_002"; + +function log(label: string, data: unknown) { + console.log(`\n--- ${label} ---`); + console.log(JSON.stringify(data, null, 2)); +} + +async function run() { + console.log("PaperTrail Demo — Document Processing Pipeline\n"); + + // Health check + log("Health Check", healthCheck()); + + // Upload a few documents + const doc1 = await uploadDocument( + ADMIN_TOKEN, + "q3-report.md", + `## Q3 Revenue Report + +The quarterly revenue showed excellent growth across all segments. +Server infrastructure costs declined by 12% due to improved deployment automation. +The engineering team shipped 47 features and resolved 183 bugs. +User feedback has been overwhelmingly positive about the new dashboard design. +Overall profit margins improved to 34%, exceeding our target by 6 points.`, + "text/markdown" + ); + log("Upload doc1", doc1); + + const doc2 = await uploadDocument( + ADMIN_TOKEN, + "incident-postmortem.txt", + `On March 3rd the primary database server experienced a connection pool exhaustion issue. +The problem was caused by a missing connection timeout in the API gateway code. +This resulted in 23 minutes of degraded performance for approximately 8000 users. +The team identified the root cause within 7 minutes and deployed a hotfix. +We have since added connection pool monitoring and alerting to prevent recurrence. +The risk of this happening again is low but we recommend a full review of all timeout configurations.`, + "text/plain" + ); + log("Upload doc2", doc2); + + const doc3 = await uploadDocument( + ADMIN_TOKEN, + "product-roadmap.txt", + `Product Roadmap Q4 2026 + +Launch the redesigned user onboarding flow by October 15th. +Ship the API v3 endpoints with improved rate limiting and authentication. +Deploy the new feature flag system for gradual rollouts. +Collect user feedback on the search improvements launched last quarter. +The design team will finalize the component library documentation.`, + "text/plain" + ); + log("Upload doc3", doc3); + + // Try uploading as a viewer (should fail) + const denied = await uploadDocument( + VIEWER_TOKEN, + "secret.txt", + "this should not work", + ); + log("Upload as viewer (should fail)", denied); + + // Try with no token + const noAuth = await uploadDocument(undefined, "nope.txt", "no token"); + log("Upload with no token (should fail)", noAuth); + + // Fetch a document (cache hit on second fetch) + if (doc1.data) { + const fetched1 = await fetchDocument(ADMIN_TOKEN, doc1.data.id); + log("Fetch doc1 (first - from cache)", fetched1); + + const fetched2 = await fetchDocument(VIEWER_TOKEN, doc1.data.id); + log("Fetch doc1 (second - cache hit)", fetched2); + } + + // List all documents + const all = await listAllDocuments(ADMIN_TOKEN, 0, 2); + log("List documents (page 0, size 2)", all); + + const page2 = await listAllDocuments(ADMIN_TOKEN, 1, 2); + log("List documents (page 1, size 2)", page2); + + // Delete a document + if (doc2.data) { + const deleted = await removeDocument(ADMIN_TOKEN, doc2.data.id); + log("Delete doc2", deleted); + } + + // Final health check + log("Final Health Check", healthCheck()); + + console.log("\n--- Done ---"); +} + +run().catch(console.error); diff --git a/examples/papertrail/src/pipeline.ts b/examples/papertrail/src/pipeline.ts new file mode 100644 index 0000000..a0e10ac --- /dev/null +++ b/examples/papertrail/src/pipeline.ts @@ -0,0 +1,188 @@ +import { trail } from "agentcrumbs"; // @crumbs + +import type { + Document, + DocumentMetadata, + DocumentSummary, + ProcessedDocument, +} from "./types.js"; +import { saveDocument } from "./store.js"; + +const crumb = trail("doc-pipeline"); // @crumbs + +function simulateLatency(min: number, max: number): Promise { + const ms = Math.floor(Math.random() * (max - min)) + min; + return new Promise((resolve) => setTimeout(resolve, ms)); +} + +async function parseDocument(doc: Document): Promise { + return crumb.scope("parse", async (ctx) => { // @crumbs + ctx.crumb("parsing document", { // @crumbs + filename: doc.filename, // @crumbs + mimeType: doc.mimeType, // @crumbs + contentLength: doc.content.length, // @crumbs + }); // @crumbs + + crumb.assert(doc.content.length > 0, "document content is empty"); // @crumbs + crumb.assert(doc.content.length < 1_000_000, "document exceeds 1MB limit"); // @crumbs + + crumb.time("parse"); // @crumbs + await simulateLatency(30, 100); + + // Simulate parsing based on mime type + let parsed: string; + if (doc.mimeType === "text/plain") { + parsed = doc.content; + } else if (doc.mimeType === "text/markdown") { + // Strip markdown syntax (simplified) + parsed = doc.content + .replace(/#{1,6}\s/g, "") + .replace(/\*\*/g, "") + .replace(/\*/g, ""); + } else { + parsed = doc.content; // fallback + } + + crumb.timeEnd("parse", { parsedLength: parsed.length }); // @crumbs + return parsed; + }); // @crumbs +} + +async function extractMetadata(text: string): Promise { + return crumb.scope("extract-metadata", async () => { // @crumbs + crumb.time("metadata-extraction"); // @crumbs + await simulateLatency(50, 150); + + const words = text.split(/\s+/).filter(Boolean); + const wordCount = words.length; + + // Fake topic extraction + const topicKeywords: Record = { + engineering: ["api", "server", "database", "deploy", "code", "function"], + finance: ["revenue", "cost", "budget", "quarterly", "profit", "margin"], + product: ["user", "feature", "roadmap", "launch", "feedback", "design"], + }; + + const lowerText = text.toLowerCase(); + const topics = Object.entries(topicKeywords) + .filter(([, keywords]) => keywords.some((kw) => lowerText.includes(kw))) + .map(([topic]) => topic); + + // Fake sentiment + const positiveWords = ["great", "excellent", "improved", "success", "growth"]; + const negativeWords = ["failed", "issue", "problem", "declined", "risk"]; + const posCount = positiveWords.filter((w) => lowerText.includes(w)).length; + const negCount = negativeWords.filter((w) => lowerText.includes(w)).length; + const sentiment = + posCount > negCount + ? "positive" + : negCount > posCount + ? "negative" + : "neutral"; + + const metadata: DocumentMetadata = { + wordCount, + language: "en", + topics: topics.length > 0 ? topics : ["general"], + sentiment, + }; + + crumb.timeEnd("metadata-extraction"); // @crumbs + crumb.snapshot("extracted-metadata", metadata); // @crumbs + + return metadata; + }); // @crumbs +} + +async function summarize( + text: string, + metadata: DocumentMetadata +): Promise { + return crumb.scope("summarize", async (ctx) => { // @crumbs + ctx.crumb("calling summarizer", { // @crumbs + wordCount: metadata.wordCount, // @crumbs + topics: metadata.topics, // @crumbs + }, { tags: ["ai"] }); // @crumbs + + crumb.time("ai-summarize"); // @crumbs + // Simulate AI API call latency + await simulateLatency(200, 500); + + const sentences = text.split(/[.!?]+/).filter((s) => s.trim().length > 10); + const title = + sentences[0]?.trim().slice(0, 80) || "Untitled Document"; + + const summary = + sentences.length > 2 + ? sentences.slice(0, 3).join(". ").trim() + "." + : text.slice(0, 200); + + const keyPoints = sentences + .filter((_, i) => i % Math.max(1, Math.floor(sentences.length / 3)) === 0) + .slice(0, 5) + .map((s) => s.trim()); + + crumb.timeEnd("ai-summarize", { // @crumbs + titleLength: title.length, // @crumbs + keyPointCount: keyPoints.length, // @crumbs + }); // @crumbs + + return { title, summary, keyPoints }; + }); // @crumbs +} + +export async function processDocument(doc: Document): Promise { + const session = crumb.session("process-doc"); // @crumbs + session.crumb("starting pipeline", { id: doc.id, filename: doc.filename }); // @crumbs + + try { + // Stage 1: Parse + const parsed = await parseDocument(doc); + + // #region @crumbs + crumb.snapshot("after-parse", { + id: doc.id, + parsedLength: parsed.length, + preview: parsed.slice(0, 100), + }); + // #endregion @crumbs + + // Stage 2: Extract metadata + const metadata = await extractMetadata(parsed); + + // Stage 3: Summarize + const summary = await summarize(parsed, metadata); + + // #region @crumbs + crumb.snapshot("after-summarize", { + id: doc.id, + title: summary.title, + keyPointCount: summary.keyPoints.length, + sentiment: metadata.sentiment, + }); + // #endregion @crumbs + + const processed: ProcessedDocument = { + id: doc.id, + document: doc, + metadata, + summary, + processedAt: new Date().toISOString(), + }; + + // Stage 4: Store + await saveDocument(processed); + + session.crumb("pipeline complete", { id: doc.id }); // @crumbs + session.end(); // @crumbs + + return processed; + } catch (err) { + session.crumb("pipeline failed", { // @crumbs + id: doc.id, // @crumbs + error: err instanceof Error ? err.message : String(err), // @crumbs + }); // @crumbs + session.end(); // @crumbs + throw err; + } +} diff --git a/examples/papertrail/src/store.ts b/examples/papertrail/src/store.ts new file mode 100644 index 0000000..99cce88 --- /dev/null +++ b/examples/papertrail/src/store.ts @@ -0,0 +1,122 @@ +import { trail } from "agentcrumbs"; // @crumbs + +import type { + CacheEntry, + PaginatedResult, + ProcessedDocument, +} from "./types.js"; + +const crumb = trail("doc-store"); // @crumbs + +// In-memory "database" +const documents = new Map(); + +// Simple cache layer +const cache = new Map>(); +const CACHE_TTL = 30_000; // 30 seconds + +// #region @crumbs +const cacheGet = crumb.wrap("cache-get", (key: string): T | undefined => { + const entry = cache.get(key) as CacheEntry | undefined; + if (!entry) { + crumb("cache miss", { key }, { tags: ["cache"] }); + return undefined; + } + if (Date.now() > entry.expiresAt) { + crumb("cache expired", { key, expiresAt: entry.expiresAt }, { tags: ["cache"] }); + cache.delete(key); + return undefined; + } + crumb("cache hit", { key }, { tags: ["cache"] }); + return entry.value; +}); + +function cacheSet(key: string, value: T): void { + crumb("cache set", { key, ttl: CACHE_TTL }, { tags: ["cache"] }); + cache.set(key, { value, expiresAt: Date.now() + CACHE_TTL }); +} +// #endregion @crumbs + +function simulateLatency(min: number, max: number): Promise { + const ms = Math.floor(Math.random() * (max - min)) + min; + return new Promise((resolve) => setTimeout(resolve, ms)); +} + +export async function saveDocument(doc: ProcessedDocument): Promise { + return crumb.scope("save-document", async () => { // @crumbs + crumb("writing to store", { id: doc.id, filename: doc.document.filename }); // @crumbs + + crumb.time("db-write"); // @crumbs + await simulateLatency(20, 80); + documents.set(doc.id, doc); + crumb.timeEnd("db-write", { documentsCount: documents.size }); // @crumbs + + cacheSet(`doc:${doc.id}`, doc); // @crumbs + + crumb("document saved", { id: doc.id }); // @crumbs + }); // @crumbs +} + +export async function getDocument( + id: string +): Promise { + return crumb.scope("get-document", async () => { // @crumbs + // Check cache first + const cached = cacheGet(`doc:${id}`); // @crumbs + if (cached) return cached; // @crumbs + + crumb.time("db-read"); // @crumbs + await simulateLatency(10, 40); + const doc = documents.get(id); + crumb.timeEnd("db-read", { found: !!doc }); // @crumbs + + if (doc) cacheSet(`doc:${id}`, doc); // @crumbs + return doc; + }); // @crumbs +} + +export async function listDocuments( + page: number, + pageSize: number +): Promise> { + return crumb.scope("list-documents", async () => { // @crumbs + crumb("listing documents", { page, pageSize }); // @crumbs + + crumb.time("db-list"); // @crumbs + await simulateLatency(15, 50); + + const all = Array.from(documents.values()); + const start = page * pageSize; + const items = all.slice(start, start + pageSize); + + crumb.timeEnd("db-list", { total: all.length, returned: items.length }); // @crumbs + + return { + items, + total: all.length, + page, + pageSize, + }; + }); // @crumbs +} + +export async function deleteDocument(id: string): Promise { + return crumb.scope("delete-document", async () => { // @crumbs + crumb("deleting document", { id }); // @crumbs + + crumb.time("db-delete"); // @crumbs + await simulateLatency(10, 30); + const existed = documents.delete(id); + cache.delete(`doc:${id}`); + crumb.timeEnd("db-delete", { existed }); // @crumbs + + return existed; + }); // @crumbs +} + +export function getStats() { + return { + documentCount: documents.size, + cacheSize: cache.size, + }; +} diff --git a/examples/papertrail/src/types.ts b/examples/papertrail/src/types.ts new file mode 100644 index 0000000..35b1abc --- /dev/null +++ b/examples/papertrail/src/types.ts @@ -0,0 +1,40 @@ +export type Document = { + id: string; + filename: string; + content: string; + mimeType: string; + uploadedAt: string; +}; + +export type DocumentMetadata = { + wordCount: number; + language: string; + topics: string[]; + sentiment: "positive" | "negative" | "neutral"; +}; + +export type DocumentSummary = { + title: string; + summary: string; + keyPoints: string[]; +}; + +export type ProcessedDocument = { + id: string; + document: Document; + metadata: DocumentMetadata; + summary: DocumentSummary; + processedAt: string; +}; + +export type CacheEntry = { + value: T; + expiresAt: number; +}; + +export type PaginatedResult = { + items: T[]; + total: number; + page: number; + pageSize: number; +}; diff --git a/examples/papertrail/tsconfig.json b/examples/papertrail/tsconfig.json new file mode 100644 index 0000000..e968634 --- /dev/null +++ b/examples/papertrail/tsconfig.json @@ -0,0 +1,12 @@ +{ + "compilerOptions": { + "target": "ES2022", + "module": "Node16", + "moduleResolution": "Node16", + "strict": true, + "esModuleInterop": true, + "skipLibCheck": true, + "outDir": "dist" + }, + "include": ["src"] +} diff --git a/packages/agentcrumbs/.tshy/build.json b/packages/agentcrumbs/.tshy/build.json new file mode 100644 index 0000000..aea1a9e --- /dev/null +++ b/packages/agentcrumbs/.tshy/build.json @@ -0,0 +1,8 @@ +{ + "extends": "../tsconfig.json", + "compilerOptions": { + "rootDir": "../src", + "module": "nodenext", + "moduleResolution": "nodenext" + } +} diff --git a/packages/agentcrumbs/.tshy/commonjs.json b/packages/agentcrumbs/.tshy/commonjs.json new file mode 100644 index 0000000..e0bd597 --- /dev/null +++ b/packages/agentcrumbs/.tshy/commonjs.json @@ -0,0 +1,17 @@ +{ + "extends": "./build.json", + "include": [ + "../src/**/*.ts", + "../src/**/*.cts", + "../src/**/*.tsx", + "../src/**/*.json" + ], + "exclude": [ + "../src/__tests__/**/*", + "../src/**/*.mts", + "../src/package.json" + ], + "compilerOptions": { + "outDir": "../.tshy-build/commonjs" + } +} diff --git a/packages/agentcrumbs/.tshy/esm.json b/packages/agentcrumbs/.tshy/esm.json new file mode 100644 index 0000000..d717e14 --- /dev/null +++ b/packages/agentcrumbs/.tshy/esm.json @@ -0,0 +1,16 @@ +{ + "extends": "./build.json", + "include": [ + "../src/**/*.ts", + "../src/**/*.mts", + "../src/**/*.tsx", + "../src/**/*.json" + ], + "exclude": [ + "../src/__tests__/**/*", + "../src/package.json" + ], + "compilerOptions": { + "outDir": "../.tshy-build/esm" + } +} diff --git a/packages/agentcrumbs/bin/intent.js b/packages/agentcrumbs/bin/intent.js old mode 100644 new mode 100755 diff --git a/packages/agentcrumbs/skills/agentcrumbs/SKILL.md b/packages/agentcrumbs/skills/agentcrumbs/SKILL.md index 8b04a4e..d43135d 100644 --- a/packages/agentcrumbs/skills/agentcrumbs/SKILL.md +++ b/packages/agentcrumbs/skills/agentcrumbs/SKILL.md @@ -79,13 +79,19 @@ const result = await crumb.scope("operation", async (ctx) => { ```bash agentcrumbs collect # Start HTTP collector (required for query/tail) -agentcrumbs tail # Live tail (--ns, --tag, --match filters) +agentcrumbs tail # Live tail (auto-scoped to current app) +agentcrumbs tail --app foo # Tail a specific app +agentcrumbs tail --all-apps # Tail all apps agentcrumbs query --since 5m # Query history (--ns, --tag, --session, --json) +agentcrumbs clear # Clear crumbs for current app +agentcrumbs clear --all-apps # Clear crumbs for all apps agentcrumbs strip # Remove all crumb markers from source agentcrumbs strip --check # CI gate — exits 1 if markers found agentcrumbs --help # Full command reference ``` +Most commands accept `--app ` and `--all-apps`. Default is auto-detect from `package.json`. + Run `agentcrumbs --help` for detailed options on any command. ## Enable tracing @@ -97,6 +103,24 @@ AGENTCRUMBS='{"ns":"auth-*"}' node app.js # Filter by namespace When `AGENTCRUMBS` is not set, `trail()` returns a frozen noop. No conditionals, no overhead. +## App isolation + +Every crumb is stamped with an `app` name. This keeps crumbs from different projects separate — storage, CLI queries, and tail all scope to the current app by default. + +**App name resolution** (first match wins): +1. `app` field in `AGENTCRUMBS` JSON config: `AGENTCRUMBS='{"app":"my-app","ns":"*"}'` +2. `AGENTCRUMBS_APP` env var +3. Auto-detected from the nearest `package.json` name field + +Crumbs are stored per-app at `~/.agentcrumbs//crumbs.jsonl`. + +```bash +agentcrumbs tail # Scoped to current app (auto-detected) +agentcrumbs tail --app my-app # Scope to a specific app +agentcrumbs tail --all-apps # See crumbs from all apps +agentcrumbs stats --all-apps # Per-app statistics +``` + ## Critical mistakes 1. **Missing markers** — Every crumb line needs `// @crumbs` or a `#region @crumbs` block. Without them, `strip` can't clean up. diff --git a/packages/agentcrumbs/skills/agentcrumbs/init/SKILL.md b/packages/agentcrumbs/skills/agentcrumbs/init/SKILL.md index 5aa1c04..b0b8f08 100644 --- a/packages/agentcrumbs/skills/agentcrumbs/init/SKILL.md +++ b/packages/agentcrumbs/skills/agentcrumbs/init/SKILL.md @@ -67,6 +67,15 @@ you have a reasonable set (typically 3–15 namespaces): - Match existing conventions in the repo if they exist - Group related subsystems with prefixes: `db-queries`, `db-migrations` +### Determine the app name + +The app name identifies this project in crumb storage and CLI filtering. +Read the root `package.json` `name` field (strip any `@scope/` prefix). +This is what gets stamped on every crumb as the `app` field. + +If the repo is a monorepo, use the root package name (not individual +workspace packages — those become namespaces, not apps). + ### What to capture for each namespace For each namespace, record: @@ -78,12 +87,13 @@ For each namespace, record: ## Phase 2 — Ask the maintainer -Present your discovered namespaces to the maintainer and ask: +Present the app name and discovered namespaces to the maintainer and ask: -1. "Here are the namespaces I found. Should I add, remove, or rename any?" -2. "Are there any cross-cutting concerns I should add?" (e.g., `cron`, +1. "The app name is `` (from package.json). Is that correct?" +2. "Here are the namespaces I found. Should I add, remove, or rename any?" +3. "Are there any cross-cutting concerns I should add?" (e.g., `cron`, `migrations`, `external-api`, `auth`, `billing`) -3. "Which agent config file should I write to — CLAUDE.md or AGENTS.md?" +4. "Which agent config file should I write to — CLAUDE.md or AGENTS.md?" Wait for their response before proceeding. @@ -103,6 +113,8 @@ The output should look like this: ````markdown ## agentcrumbs +App: `my-project` + Add crumbs as you write code — not just when debugging. Mark lines with `// @crumbs` or wrap blocks in `// #region @crumbs`. They stay on the branch throughout development and are stripped by `agentcrumbs strip` @@ -129,14 +141,18 @@ production. ### CLI ```bash -agentcrumbs collect # start collector (multi-service) -agentcrumbs tail # live tail -agentcrumbs strip # remove crumbs before merge +AGENTCRUMBS=1 node app.js # enable tracing (or AGENTCRUMBS_APP=my-project) +agentcrumbs collect # start collector (multi-service) +agentcrumbs tail # live tail (scoped to this app) +agentcrumbs tail --app my-project # tail a specific app +agentcrumbs clear # clear crumbs for this app +agentcrumbs strip # remove crumbs before merge ``` ```` -Adapt the example above to the actual discovered namespaces. Drop the -Path column if namespaces don't have meaningful directory paths. +Adapt the example above to the actual discovered namespaces and app name. +Drop the Path column if namespaces don't have meaningful directory paths. +Replace `my-project` with the actual app name detected from `package.json`. --- diff --git a/packages/agentcrumbs/src/__tests__/env.test.ts b/packages/agentcrumbs/src/__tests__/env.test.ts index 208f8eb..c8066d7 100644 --- a/packages/agentcrumbs/src/__tests__/env.test.ts +++ b/packages/agentcrumbs/src/__tests__/env.test.ts @@ -1,11 +1,13 @@ import { describe, it, expect, beforeEach, afterEach } from "vitest"; -import { isNamespaceEnabled, parseConfig, resetConfig } from "../env.js"; +import { isNamespaceEnabled, parseConfig, resetConfig, getApp, resetApp } from "../env.js"; describe("env", () => { const originalEnv = process.env.AGENTCRUMBS; + const originalAppEnv = process.env.AGENTCRUMBS_APP; beforeEach(() => { resetConfig(); + resetApp(); }); afterEach(() => { @@ -14,7 +16,13 @@ describe("env", () => { } else { process.env.AGENTCRUMBS = originalEnv; } + if (originalAppEnv === undefined) { + delete process.env.AGENTCRUMBS_APP; + } else { + process.env.AGENTCRUMBS_APP = originalAppEnv; + } resetConfig(); + resetApp(); }); it("returns disabled when AGENTCRUMBS is not set", () => { @@ -83,4 +91,59 @@ describe("env", () => { expect(config.format).toBe("json"); } }); + + it("parses app from JSON config", () => { + process.env.AGENTCRUMBS = '{"app":"my-app","ns":"*"}'; + const config = parseConfig(); + expect(config.enabled).toBe(true); + if (config.enabled) { + expect(config.app).toBe("my-app"); + } + }); + + describe("getApp", () => { + it("returns app from JSON config as highest priority", () => { + process.env.AGENTCRUMBS = '{"app":"json-app","ns":"*"}'; + process.env.AGENTCRUMBS_APP = "env-app"; + resetConfig(); + resetApp(); + expect(getApp()).toBe("json-app"); + }); + + it("returns AGENTCRUMBS_APP when no JSON config app", () => { + process.env.AGENTCRUMBS = '{"ns":"*"}'; + process.env.AGENTCRUMBS_APP = "env-app"; + resetConfig(); + resetApp(); + expect(getApp()).toBe("env-app"); + }); + + it("returns AGENTCRUMBS_APP when AGENTCRUMBS=1", () => { + process.env.AGENTCRUMBS = "1"; + process.env.AGENTCRUMBS_APP = "env-app"; + resetConfig(); + resetApp(); + expect(getApp()).toBe("env-app"); + }); + + it("auto-detects from package.json when no explicit config", () => { + delete process.env.AGENTCRUMBS_APP; + delete process.env.AGENTCRUMBS; + resetConfig(); + resetApp(); + const app = getApp(); + // Should find the agentcrumbs package.json in the repo + expect(app).not.toBe("unknown"); + expect(typeof app).toBe("string"); + }); + + it("caches the result", () => { + process.env.AGENTCRUMBS_APP = "cached-app"; + resetApp(); + expect(getApp()).toBe("cached-app"); + // Change env — should still return cached value + process.env.AGENTCRUMBS_APP = "different-app"; + expect(getApp()).toBe("cached-app"); + }); + }); }); diff --git a/packages/agentcrumbs/src/__tests__/trail.test.ts b/packages/agentcrumbs/src/__tests__/trail.test.ts index 7bb2e34..eee5a36 100644 --- a/packages/agentcrumbs/src/__tests__/trail.test.ts +++ b/packages/agentcrumbs/src/__tests__/trail.test.ts @@ -1,15 +1,17 @@ import { describe, it, expect, beforeEach, afterEach } from "vitest"; import { trail, addSink, removeSink, resetSinks } from "../trail.js"; -import { resetConfig } from "../env.js"; +import { resetConfig, resetApp } from "../env.js"; import { MemorySink } from "../sinks/memory.js"; import { NOOP } from "../noop.js"; describe("trail", () => { const originalEnv = process.env.AGENTCRUMBS; + const originalAppEnv = process.env.AGENTCRUMBS_APP; let sink: MemorySink; beforeEach(() => { resetConfig(); + resetApp(); resetSinks(); sink = new MemorySink(); addSink(sink); @@ -21,7 +23,13 @@ describe("trail", () => { } else { process.env.AGENTCRUMBS = originalEnv; } + if (originalAppEnv === undefined) { + delete process.env.AGENTCRUMBS_APP; + } else { + process.env.AGENTCRUMBS_APP = originalAppEnv; + } resetConfig(); + resetApp(); resetSinks(); }); @@ -98,6 +106,8 @@ describe("trail", () => { expect(sink.entries).toHaveLength(1); const entry = sink.entries[0]!; + expect(entry.app).toBeDefined(); + expect(typeof entry.app).toBe("string"); expect(entry.ns).toBe("test"); expect(entry.msg).toBe("hello"); expect(entry.data).toEqual({ key: "value" }); @@ -336,4 +346,67 @@ describe("trail", () => { expect(sink.entries.some((e) => e.type === "scope:exit" && e.msg === "add")).toBe(true); }); }); + + describe("app", () => { + beforeEach(() => { + process.env.AGENTCRUMBS = "1"; + resetConfig(); + resetApp(); + }); + + it("stamps every crumb with app field", () => { + const crumb = trail("test"); + crumb("hello"); + + expect(sink.entries).toHaveLength(1); + expect(sink.entries[0]!.app).toBeDefined(); + expect(typeof sink.entries[0]!.app).toBe("string"); + expect(sink.entries[0]!.app.length).toBeGreaterThan(0); + }); + + it("uses AGENTCRUMBS_APP env var", () => { + process.env.AGENTCRUMBS_APP = "my-test-app"; + resetApp(); + + const crumb = trail("test"); + crumb("hello"); + + expect(sink.entries[0]!.app).toBe("my-test-app"); + }); + + it("uses app from JSON config", () => { + process.env.AGENTCRUMBS = '{"app":"json-app","ns":"*"}'; + resetConfig(); + resetApp(); + + const crumb = trail("test"); + crumb("hello"); + + expect(sink.entries[0]!.app).toBe("json-app"); + }); + + it("JSON config app takes priority over env var", () => { + process.env.AGENTCRUMBS = '{"app":"json-app","ns":"*"}'; + process.env.AGENTCRUMBS_APP = "env-app"; + resetConfig(); + resetApp(); + + const crumb = trail("test"); + crumb("hello"); + + expect(sink.entries[0]!.app).toBe("json-app"); + }); + + it("auto-detects from package.json when no explicit app", () => { + delete process.env.AGENTCRUMBS_APP; + resetApp(); + + const crumb = trail("test"); + crumb("hello"); + + // Should find the agentcrumbs package.json + expect(sink.entries[0]!.app).toBeDefined(); + expect(sink.entries[0]!.app).not.toBe("unknown"); + }); + }); }); diff --git a/packages/agentcrumbs/src/cli/app-store.ts b/packages/agentcrumbs/src/cli/app-store.ts new file mode 100644 index 0000000..55ecdcd --- /dev/null +++ b/packages/agentcrumbs/src/cli/app-store.ts @@ -0,0 +1,50 @@ +import path from "node:path"; +import os from "node:os"; +import { CrumbStore } from "../collector/store.js"; +import { getApp } from "../env.js"; +import { getFlag, hasFlag } from "./args.js"; + +const DEFAULT_BASE = path.join(os.homedir(), ".agentcrumbs"); + +export type AppContext = { + app: string | undefined; + allApps: boolean; +}; + +export function parseAppFlags(args: string[]): AppContext { + const app = getFlag(args, "--app"); + const allApps = hasFlag(args, "--all-apps"); + return { app, allApps }; +} + +export function resolveApp(ctx: AppContext): string { + if (ctx.app) return ctx.app; + return getApp(); +} + +export function getStoreForContext(ctx: AppContext): CrumbStore { + if (ctx.allApps) { + // For --all-apps, callers should use readAllCrumbs instead + // Return a dummy store for the current app as fallback + return CrumbStore.forApp(resolveApp(ctx)); + } + const app = resolveApp(ctx); + return CrumbStore.forApp(app); +} + +export function readAllCrumbs(ctx: AppContext): import("../types.js").Crumb[] { + if (ctx.allApps) { + return CrumbStore.readAllApps(); + } + const store = getStoreForContext(ctx); + return store.readAll(); +} + +export function getStoreFilePath(ctx: AppContext): string { + const store = getStoreForContext(ctx); + return store.getFilePath(); +} + +export function getBaseDir(): string { + return DEFAULT_BASE; +} diff --git a/packages/agentcrumbs/src/cli/commands/clear.ts b/packages/agentcrumbs/src/cli/commands/clear.ts index b90c9c0..011cbd3 100644 --- a/packages/agentcrumbs/src/cli/commands/clear.ts +++ b/packages/agentcrumbs/src/cli/commands/clear.ts @@ -1,9 +1,22 @@ -import path from "node:path"; -import os from "node:os"; import { CrumbStore } from "../../collector/store.js"; +import { hasFlag } from "../args.js"; +import { parseAppFlags, getStoreForContext, resolveApp } from "../app-store.js"; -export async function clear(_args: string[]): Promise { - const store = new CrumbStore(path.join(os.homedir(), ".agentcrumbs")); +export async function clear(args: string[]): Promise { + const appCtx = parseAppFlags(args); + + if (appCtx.allApps || hasFlag(args, "--all-apps")) { + const apps = CrumbStore.listApps(); + for (const app of apps) { + const store = CrumbStore.forApp(app); + store.clear(); + } + process.stdout.write(`Crumb trail cleared for all apps (${apps.length}).\n`); + return; + } + + const app = resolveApp(appCtx); + const store = getStoreForContext(appCtx); store.clear(); - process.stdout.write("Crumb trail cleared.\n"); + process.stdout.write(`Crumb trail cleared for ${app}.\n`); } diff --git a/packages/agentcrumbs/src/cli/commands/collect.ts b/packages/agentcrumbs/src/cli/commands/collect.ts index daaa986..317b518 100644 --- a/packages/agentcrumbs/src/cli/commands/collect.ts +++ b/packages/agentcrumbs/src/cli/commands/collect.ts @@ -15,7 +15,7 @@ export async function collect(args: string[]): Promise { server.on("crumb", (crumb: Crumb) => { if (!quiet) { - process.stdout.write(formatCrumbPretty(crumb) + "\n"); + process.stdout.write(formatCrumbPretty(crumb, { showApp: true }) + "\n"); } }); @@ -25,10 +25,9 @@ export async function collect(args: string[]): Promise { await server.start(); - const store = server.getStore(); process.stdout.write(`agentcrumbs collector\n`); process.stdout.write(` http: http://localhost:${port}/crumb\n`); - process.stdout.write(` store: ${store.getFilePath()}\n`); + process.stdout.write(` crumbs stored per-app in ~/.agentcrumbs//\n`); process.stdout.write(` press ctrl+c to stop\n\n`); const shutdown = () => { @@ -40,4 +39,3 @@ export async function collect(args: string[]): Promise { process.on("SIGINT", shutdown); process.on("SIGTERM", shutdown); } - diff --git a/packages/agentcrumbs/src/cli/commands/follow.ts b/packages/agentcrumbs/src/cli/commands/follow.ts index 6aa39c7..273a9ca 100644 --- a/packages/agentcrumbs/src/cli/commands/follow.ts +++ b/packages/agentcrumbs/src/cli/commands/follow.ts @@ -1,21 +1,19 @@ -import path from "node:path"; -import os from "node:os"; -import { CrumbStore } from "../../collector/store.js"; import { formatCrumbPretty, formatCrumbJson } from "../format.js"; import { getFlag, hasFlag } from "../args.js"; +import { parseAppFlags, readAllCrumbs } from "../app-store.js"; export async function follow(args: string[]): Promise { const traceId = getFlag(args, "--trace"); const json = hasFlag(args, "--json"); + const appCtx = parseAppFlags(args); + const showApp = appCtx.allApps; if (!traceId) { - process.stderr.write("Usage: agentcrumbs follow --trace [--json]\n"); + process.stderr.write("Usage: agentcrumbs follow --trace [--app ] [--all-apps] [--json]\n"); process.exit(1); } - const store = new CrumbStore(path.join(os.homedir(), ".agentcrumbs")); - const allCrumbs = store.readAll(); - + const allCrumbs = readAllCrumbs(appCtx); const traceCrumbs = allCrumbs.filter((c) => c.traceId === traceId); if (traceCrumbs.length === 0) { @@ -35,7 +33,7 @@ export async function follow(args: string[]): Promise { if (json) { process.stdout.write(formatCrumbJson(crumb) + "\n"); } else { - process.stdout.write(formatCrumbPretty(crumb) + "\n"); + process.stdout.write(formatCrumbPretty(crumb, { showApp }) + "\n"); } } } diff --git a/packages/agentcrumbs/src/cli/commands/query.ts b/packages/agentcrumbs/src/cli/commands/query.ts index f8aae0b..ab3111b 100644 --- a/packages/agentcrumbs/src/cli/commands/query.ts +++ b/packages/agentcrumbs/src/cli/commands/query.ts @@ -1,9 +1,7 @@ -import path from "node:path"; -import os from "node:os"; import type { Crumb } from "../../types.js"; -import { CrumbStore } from "../../collector/store.js"; import { formatCrumbPretty, formatCrumbJson } from "../format.js"; import { getFlag, hasFlag } from "../args.js"; +import { parseAppFlags, readAllCrumbs } from "../app-store.js"; export async function query(args: string[]): Promise { const ns = getFlag(args, "--ns"); @@ -13,9 +11,10 @@ export async function query(args: string[]): Promise { const match = getFlag(args, "--match"); const json = hasFlag(args, "--json"); const limit = parseInt(getFlag(args, "--limit") ?? "100", 10); + const appCtx = parseAppFlags(args); + const showApp = appCtx.allApps; - const store = new CrumbStore(path.join(os.homedir(), ".agentcrumbs")); - const allCrumbs = store.readAll(); + const allCrumbs = readAllCrumbs(appCtx); let filtered = allCrumbs; @@ -52,7 +51,7 @@ export async function query(args: string[]): Promise { if (json) { process.stdout.write(formatCrumbJson(crumb) + "\n"); } else { - process.stdout.write(formatCrumbPretty(crumb) + "\n"); + process.stdout.write(formatCrumbPretty(crumb, { showApp }) + "\n"); } } @@ -79,4 +78,3 @@ function parseSince(since: string): number { return Date.now() - value * multipliers[unit]!; } - diff --git a/packages/agentcrumbs/src/cli/commands/replay.ts b/packages/agentcrumbs/src/cli/commands/replay.ts index 0383159..5eba22d 100644 --- a/packages/agentcrumbs/src/cli/commands/replay.ts +++ b/packages/agentcrumbs/src/cli/commands/replay.ts @@ -1,20 +1,19 @@ -import path from "node:path"; -import os from "node:os"; -import { CrumbStore } from "../../collector/store.js"; import { formatCrumbPretty, formatCrumbJson } from "../format.js"; +import { hasFlag } from "../args.js"; +import { parseAppFlags, readAllCrumbs } from "../app-store.js"; export async function replay(args: string[]): Promise { const sessionId = args[0]; - const json = args.includes("--json"); + const json = hasFlag(args, "--json"); + const appCtx = parseAppFlags(args); + const showApp = appCtx.allApps; if (!sessionId) { - process.stderr.write("Usage: agentcrumbs replay [--json]\n"); + process.stderr.write("Usage: agentcrumbs replay [--app ] [--all-apps] [--json]\n"); process.exit(1); } - const store = new CrumbStore(path.join(os.homedir(), ".agentcrumbs")); - const allCrumbs = store.readAll(); - + const allCrumbs = readAllCrumbs(appCtx); const sessionCrumbs = allCrumbs.filter((c) => c.sid === sessionId); if (sessionCrumbs.length === 0) { @@ -40,7 +39,7 @@ export async function replay(args: string[]): Promise { if (json) { process.stdout.write(formatCrumbJson(crumb) + "\n"); } else { - process.stdout.write(formatCrumbPretty(crumb) + "\n"); + process.stdout.write(formatCrumbPretty(crumb, { showApp }) + "\n"); } } } diff --git a/packages/agentcrumbs/src/cli/commands/sessions.ts b/packages/agentcrumbs/src/cli/commands/sessions.ts index be16c0e..16a3bf5 100644 --- a/packages/agentcrumbs/src/cli/commands/sessions.ts +++ b/packages/agentcrumbs/src/cli/commands/sessions.ts @@ -1,18 +1,16 @@ -import path from "node:path"; -import os from "node:os"; import fs from "node:fs"; -import { CrumbStore } from "../../collector/store.js"; +import { parseAppFlags, readAllCrumbs } from "../app-store.js"; const SESSION_FILE = "/tmp/agentcrumbs.session"; -export async function sessions(_args: string[]): Promise { - const store = new CrumbStore(path.join(os.homedir(), ".agentcrumbs")); - const allCrumbs = store.readAll(); +export async function sessions(args: string[]): Promise { + const appCtx = parseAppFlags(args); + const allCrumbs = readAllCrumbs(appCtx); // Find all sessions from crumbs const sessionMap = new Map< string, - { name: string; startedAt: string; endedAt?: string; count: number } + { name: string; app?: string; startedAt: string; endedAt?: string; count: number } >(); for (const crumb of allCrumbs) { @@ -22,6 +20,7 @@ export async function sessions(_args: string[]): Promise { if (!existing) { sessionMap.set(crumb.sid, { name: crumb.type === "session:start" ? crumb.msg : "unknown", + app: crumb.app, startedAt: crumb.ts, endedAt: crumb.type === "session:end" ? crumb.ts : undefined, count: 1, @@ -63,11 +62,20 @@ export async function sessions(_args: string[]): Promise { return; } + const showApp = appCtx.allApps; + // Print header - process.stdout.write( - `${"ID".padEnd(10)} ${"Name".padEnd(25)} ${"Duration".padEnd(12)} ${"Crumbs".padEnd(8)} Status\n` - ); - process.stdout.write("-".repeat(70) + "\n"); + if (showApp) { + process.stdout.write( + `${"ID".padEnd(10)} ${"App".padEnd(20)} ${"Name".padEnd(25)} ${"Duration".padEnd(12)} ${"Crumbs".padEnd(8)} Status\n` + ); + process.stdout.write("-".repeat(90) + "\n"); + } else { + process.stdout.write( + `${"ID".padEnd(10)} ${"Name".padEnd(25)} ${"Duration".padEnd(12)} ${"Crumbs".padEnd(8)} Status\n` + ); + process.stdout.write("-".repeat(70) + "\n"); + } for (const [id, info] of sessionMap) { const isActive = id === activeSessionId; @@ -80,9 +88,15 @@ export async function sessions(_args: string[]): Promise { : "?"; const status = isActive ? "active" : info.endedAt ? "stopped" : "?"; - process.stdout.write( - `${id.padEnd(10)} ${info.name.padEnd(25)} ${duration.padEnd(12)} ${String(info.count).padEnd(8)} ${status}\n` - ); + if (showApp) { + process.stdout.write( + `${id.padEnd(10)} ${(info.app ?? "unknown").padEnd(20)} ${info.name.padEnd(25)} ${duration.padEnd(12)} ${String(info.count).padEnd(8)} ${status}\n` + ); + } else { + process.stdout.write( + `${id.padEnd(10)} ${info.name.padEnd(25)} ${duration.padEnd(12)} ${String(info.count).padEnd(8)} ${status}\n` + ); + } } } diff --git a/packages/agentcrumbs/src/cli/commands/stats.ts b/packages/agentcrumbs/src/cli/commands/stats.ts index 2fb8e5c..2e46509 100644 --- a/packages/agentcrumbs/src/cli/commands/stats.ts +++ b/packages/agentcrumbs/src/cli/commands/stats.ts @@ -1,15 +1,53 @@ import fs from "node:fs"; -import path from "node:path"; -import os from "node:os"; import { CrumbStore } from "../../collector/store.js"; +import { parseAppFlags, readAllCrumbs, resolveApp, getStoreFilePath } from "../app-store.js"; -export async function stats(_args: string[]): Promise { - const storeDir = path.join(os.homedir(), ".agentcrumbs"); - const store = new CrumbStore(storeDir); - const filePath = store.getFilePath(); +export async function stats(args: string[]): Promise { + const appCtx = parseAppFlags(args); - const allCrumbs = store.readAll(); + if (appCtx.allApps) { + // Show stats for all apps + const apps = CrumbStore.listApps(); + if (apps.length === 0) { + process.stdout.write("No apps found.\n"); + return; + } + + process.stdout.write(`agentcrumbs stats (all apps)\n\n`); + + for (const app of apps) { + const store = CrumbStore.forApp(app); + const crumbs = store.readAll(); + const namespaces = new Set(crumbs.map((c) => c.ns)); + const filePath = store.getFilePath(); + + let fileSize = "0B"; + try { + const stat = fs.statSync(filePath); + fileSize = formatBytes(stat.size); + } catch { + // file might not exist + } + + const oneHourAgo = Date.now() - 60 * 60 * 1000; + const recentCount = crumbs.filter( + (c) => new Date(c.ts).getTime() >= oneHourAgo + ).length; + + process.stdout.write(` ${app}\n`); + process.stdout.write(` Services: ${[...namespaces].join(", ") || "none"}\n`); + process.stdout.write(` Total: ${crumbs.length} crumbs\n`); + process.stdout.write(` Last hour: ${recentCount} crumbs\n`); + process.stdout.write(` File: ${filePath} (${fileSize})\n\n`); + } + return; + } + + // Single app stats + const app = resolveApp(appCtx); + const allCrumbs = readAllCrumbs(appCtx); const namespaces = new Set(allCrumbs.map((c) => c.ns)); + const filePath = getStoreFilePath(appCtx); let fileSize = "0B"; try { @@ -24,7 +62,8 @@ export async function stats(_args: string[]): Promise { (c) => new Date(c.ts).getTime() >= oneHourAgo ).length; - process.stdout.write(`agentcrumbs stats\n`); + process.stdout.write(`agentcrumbs stats (${app})\n`); + process.stdout.write(` App: ${app}\n`); process.stdout.write(` Services: ${[...namespaces].join(", ") || "none"}\n`); process.stdout.write(` Total: ${allCrumbs.length} crumbs\n`); process.stdout.write(` Last hour: ${recentCount} crumbs\n`); diff --git a/packages/agentcrumbs/src/cli/commands/tail.ts b/packages/agentcrumbs/src/cli/commands/tail.ts index 0ec08fc..7ebb044 100644 --- a/packages/agentcrumbs/src/cli/commands/tail.ts +++ b/packages/agentcrumbs/src/cli/commands/tail.ts @@ -1,9 +1,8 @@ import fs from "node:fs"; -import path from "node:path"; -import os from "node:os"; import type { Crumb } from "../../types.js"; import { formatCrumbPretty, formatCrumbJson } from "../format.js"; import { getFlag, hasFlag } from "../args.js"; +import { parseAppFlags, getStoreFilePath } from "../app-store.js"; export async function tail(args: string[]): Promise { const ns = getFlag(args, "--ns"); @@ -11,9 +10,11 @@ export async function tail(args: string[]): Promise { const match = getFlag(args, "--match"); const session = getFlag(args, "--session"); const json = hasFlag(args, "--json"); + const appCtx = parseAppFlags(args); + const showApp = appCtx.allApps; - const dir = path.join(os.homedir(), ".agentcrumbs"); - const filePath = path.join(dir, "crumbs.jsonl"); + const filePath = getStoreFilePath(appCtx); + const dir = filePath.replace(/\/[^/]+$/, ""); if (!fs.existsSync(filePath)) { if (!fs.existsSync(dir)) { @@ -27,7 +28,7 @@ export async function tail(args: string[]): Promise { const existing = readLastLines(filePath, 50); for (const crumb of existing) { if (matchesCrumb(crumb, { ns, tag, match, session })) { - printCrumb(crumb, json); + printCrumb(crumb, json, showApp); } } @@ -53,7 +54,7 @@ export async function tail(args: string[]): Promise { try { const crumb = JSON.parse(line) as Crumb; if (matchesCrumb(crumb, { ns, tag, match, session })) { - printCrumb(crumb, json); + printCrumb(crumb, json, showApp); } } catch { // skip invalid lines @@ -98,11 +99,11 @@ function matchesCrumb(crumb: Crumb, filters: Filters): boolean { return true; } -function printCrumb(crumb: Crumb, json: boolean): void { +function printCrumb(crumb: Crumb, json: boolean, showApp: boolean): void { if (json) { process.stdout.write(formatCrumbJson(crumb) + "\n"); } else { - process.stdout.write(formatCrumbPretty(crumb) + "\n"); + process.stdout.write(formatCrumbPretty(crumb, { showApp }) + "\n"); } } @@ -124,4 +125,3 @@ function readLastLines(filePath: string, count: number): Crumb[] { return []; } } - diff --git a/packages/agentcrumbs/src/cli/format.ts b/packages/agentcrumbs/src/cli/format.ts index 6b9426a..1e74833 100644 --- a/packages/agentcrumbs/src/cli/format.ts +++ b/packages/agentcrumbs/src/cli/format.ts @@ -2,6 +2,10 @@ import { inspect } from "node:util"; import type { Crumb } from "../types.js"; import { colorize, dim, bold, getNamespaceColor } from "../colors.js"; +export type FormatOptions = { + showApp?: boolean; +}; + function formatDelta(dt: number): string { if (dt < 1000) return `+${Math.round(dt)}ms`; if (dt < 60000) return `+${(dt / 1000).toFixed(1)}s`; @@ -13,42 +17,47 @@ function formatData(data: unknown): string { return inspect(data, { colors: true, compact: true, depth: 4, breakLength: Infinity }); } -export function formatCrumbPretty(crumb: Crumb): string { +export function formatCrumbPretty(crumb: Crumb, options?: FormatOptions): string { const color = getNamespaceColor(crumb.ns); const ns = colorize(crumb.ns.padEnd(20), color); const dt = dim(formatDelta(crumb.dt)); const depth = crumb.depth ?? 0; const pad = " ".repeat(depth); + let prefix = ns; + if (options?.showApp && crumb.app) { + prefix = `${dim(crumb.app)} ${ns}`; + } + let line: string; switch (crumb.type) { case "scope:enter": - line = `${ns} ${pad}${bold(`[${crumb.msg}]`)} ${dim("->")} enter ${dt}`; + line = `${prefix} ${pad}${bold(`[${crumb.msg}]`)} ${dim("->")} enter ${dt}`; break; case "scope:exit": - line = `${ns} ${pad}${bold(`[${crumb.msg}]`)} ${dim("<-")} exit ${dt}`; + line = `${prefix} ${pad}${bold(`[${crumb.msg}]`)} ${dim("<-")} exit ${dt}`; break; case "scope:error": - line = `${ns} ${pad}${bold(`[${crumb.msg}]`)} ${dim("!!")} error ${dt}`; + line = `${prefix} ${pad}${bold(`[${crumb.msg}]`)} ${dim("!!")} error ${dt}`; break; case "snapshot": - line = `${ns} ${pad}${dim("snapshot:")} ${crumb.msg} ${dt}`; + line = `${prefix} ${pad}${dim("snapshot:")} ${crumb.msg} ${dt}`; break; case "assert": - line = `${ns} ${pad}${dim("assert:")} ${crumb.msg} ${dt}`; + line = `${prefix} ${pad}${dim("assert:")} ${crumb.msg} ${dt}`; break; case "time": - line = `${ns} ${pad}${dim("time:")} ${crumb.msg} ${dt}`; + line = `${prefix} ${pad}${dim("time:")} ${crumb.msg} ${dt}`; break; case "session:start": - line = `${ns} ${pad}${bold("session start:")} ${crumb.msg} ${dim(`[${crumb.sid}]`)} ${dt}`; + line = `${prefix} ${pad}${bold("session start:")} ${crumb.msg} ${dim(`[${crumb.sid}]`)} ${dt}`; break; case "session:end": - line = `${ns} ${pad}${bold("session end:")} ${crumb.msg} ${dim(`[${crumb.sid}]`)} ${dt}`; + line = `${prefix} ${pad}${bold("session end:")} ${crumb.msg} ${dim(`[${crumb.sid}]`)} ${dt}`; break; default: - line = `${ns} ${pad}${crumb.msg} ${dt}`; + line = `${prefix} ${pad}${crumb.msg} ${dt}`; } if (crumb.tags && crumb.tags.length > 0) { diff --git a/packages/agentcrumbs/src/cli/index.ts b/packages/agentcrumbs/src/cli/index.ts index e82db62..2e6d19e 100644 --- a/packages/agentcrumbs/src/cli/index.ts +++ b/packages/agentcrumbs/src/cli/index.ts @@ -54,8 +54,13 @@ Strip options: --dir Directory to scan (default: cwd) --ext File extensions (default: .ts,.tsx,.js,.jsx,.mjs,.mts) +App filtering (available on most commands): + --app Scope to a specific app (default: auto-detect from package.json) + --all-apps Show crumbs from all apps + Environment: AGENTCRUMBS Enable debug tracing (see docs for format) + AGENTCRUMBS_APP Override app name (default: auto-detect from package.json) `; async function main(): Promise { diff --git a/packages/agentcrumbs/src/collector/server.ts b/packages/agentcrumbs/src/collector/server.ts index 41ca7a8..b0192d2 100644 --- a/packages/agentcrumbs/src/collector/server.ts +++ b/packages/agentcrumbs/src/collector/server.ts @@ -8,13 +8,23 @@ const SESSION_FILE = "/tmp/agentcrumbs.session"; export class CollectorServer extends EventEmitter { private server: http.Server | undefined; - private store: CrumbStore; + private stores = new Map(); + private baseDir: string | undefined; private port: number; constructor(port: number, storeDir?: string) { super(); this.port = port; - this.store = new CrumbStore(storeDir); + this.baseDir = storeDir; + } + + private getStoreForApp(app: string): CrumbStore { + let store = this.stores.get(app); + if (!store) { + store = CrumbStore.forApp(app, this.baseDir); + this.stores.set(app, store); + } + return store; } start(): Promise { @@ -31,11 +41,12 @@ export class CollectorServer extends EventEmitter { } if (req.url === "/health" && req.method === "GET") { + const apps = CrumbStore.listApps(this.baseDir); res.writeHead(200, { "Content-Type": "application/json" }); res.end(JSON.stringify({ status: "ok", port: this.port, - store: this.store.getFilePath(), + apps, session: this.getActiveSession() ?? null, })); return; @@ -49,7 +60,9 @@ export class CollectorServer extends EventEmitter { req.on("end", () => { try { const crumb = JSON.parse(body) as Crumb; - this.store.appendRaw(body); + const app = crumb.app || "unknown"; + const store = this.getStoreForApp(app); + store.appendRaw(body); this.emit("crumb", crumb); res.writeHead(200); res.end("ok"); @@ -81,11 +94,10 @@ export class CollectorServer extends EventEmitter { this.server.close(); this.server = undefined; } - this.store.close(); - } - - getStore(): CrumbStore { - return this.store; + for (const store of this.stores.values()) { + store.close(); + } + this.stores.clear(); } getPort(): number { diff --git a/packages/agentcrumbs/src/collector/store.ts b/packages/agentcrumbs/src/collector/store.ts index 999f604..07c55c7 100644 --- a/packages/agentcrumbs/src/collector/store.ts +++ b/packages/agentcrumbs/src/collector/store.ts @@ -3,7 +3,7 @@ import path from "node:path"; import os from "node:os"; import type { Crumb } from "../types.js"; -const DEFAULT_DIR = path.join(os.homedir(), ".agentcrumbs"); +const DEFAULT_BASE = path.join(os.homedir(), ".agentcrumbs"); const DEFAULT_FILE = "crumbs.jsonl"; const MAX_FILE_SIZE = 50 * 1024 * 1024; // 50MB before rotation @@ -12,13 +12,58 @@ export class CrumbStore { private fd: number | undefined; constructor(dir?: string) { - const storeDir = dir ?? DEFAULT_DIR; + const storeDir = dir ?? DEFAULT_BASE; if (!fs.existsSync(storeDir)) { fs.mkdirSync(storeDir, { recursive: true }); } this.filePath = path.join(storeDir, DEFAULT_FILE); } + static forApp(app: string, baseDir?: string): CrumbStore { + const base = baseDir ?? DEFAULT_BASE; + return new CrumbStore(path.join(base, app)); + } + + static listApps(baseDir?: string): string[] { + const base = baseDir ?? DEFAULT_BASE; + try { + const entries = fs.readdirSync(base, { withFileTypes: true }); + return entries + .filter((e) => e.isDirectory()) + .map((e) => e.name) + .sort(); + } catch { + return []; + } + } + + static readAllApps(baseDir?: string): Crumb[] { + const apps = CrumbStore.listApps(baseDir); + const allCrumbs: Crumb[] = []; + for (const app of apps) { + const store = CrumbStore.forApp(app, baseDir); + allCrumbs.push(...store.readAll()); + } + // Also read legacy flat file if it exists + const base = baseDir ?? DEFAULT_BASE; + const legacyPath = path.join(base, DEFAULT_FILE); + try { + if (fs.existsSync(legacyPath)) { + const legacy = new CrumbStore(base); + const legacyCrumbs = legacy.readAll(); + // Tag legacy crumbs missing app field + for (const crumb of legacyCrumbs) { + if (!crumb.app) crumb.app = "unknown"; + allCrumbs.push(crumb); + } + } + } catch { + // ignore + } + allCrumbs.sort((a, b) => a.ts.localeCompare(b.ts)); + return allCrumbs; + } + private ensureFd(): number { if (this.fd === undefined) { this.fd = fs.openSync(this.filePath, "a"); diff --git a/packages/agentcrumbs/src/env.ts b/packages/agentcrumbs/src/env.ts index 6754dfe..b1eef22 100644 --- a/packages/agentcrumbs/src/env.ts +++ b/packages/agentcrumbs/src/env.ts @@ -1,3 +1,5 @@ +import fs from "node:fs"; +import path from "node:path"; import type { AgentCrumbsConfig } from "./types.js"; const DEFAULT_PORT = 8374; @@ -6,6 +8,7 @@ type ParsedConfig = { enabled: false; } | { enabled: true; + app?: string; includes: RegExp[]; excludes: RegExp[]; port: number; @@ -13,6 +16,7 @@ type ParsedConfig = { }; let cachedConfig: ParsedConfig | undefined; +let cachedApp: string | undefined; function namespaceToRegex(pattern: string): RegExp { const escaped = pattern.replace(/[.+?^${}()|[\]\\]/g, "\\$&").replace(/\*/g, ".*?"); @@ -69,6 +73,7 @@ export function parseConfig(): ParsedConfig { cachedConfig = { enabled: true, + app: config.app, includes, excludes, port: config.port ?? DEFAULT_PORT, @@ -100,7 +105,65 @@ export function getFormat(): "pretty" | "json" { return config.format; } +/** + * Resolve the app name. Priority: + * 1. `app` field from AGENTCRUMBS JSON config + * 2. AGENTCRUMBS_APP env var + * 3. Nearest package.json `name` field (walk up from cwd) + * 4. Fallback: "unknown" + */ +export function getApp(): string { + if (cachedApp !== undefined) return cachedApp; + + // 1. From parsed AGENTCRUMBS config + const config = parseConfig(); + if (config.enabled && config.app) { + cachedApp = config.app; + return cachedApp; + } + + // 2. From dedicated env var + const envApp = process.env.AGENTCRUMBS_APP; + if (envApp) { + cachedApp = envApp; + return cachedApp; + } + + // 3. Auto-detect from nearest package.json + cachedApp = detectAppFromPackageJson() ?? "unknown"; + return cachedApp; +} + +function detectAppFromPackageJson(): string | undefined { + let dir = process.cwd(); + + for (let i = 0; i < 50; i++) { + const pkgPath = path.join(dir, "package.json"); + try { + const content = fs.readFileSync(pkgPath, "utf-8"); + const pkg = JSON.parse(content) as { name?: string }; + if (pkg.name) { + // Strip @scope/ prefix + return pkg.name.replace(/^@[^/]+\//, ""); + } + } catch { + // no package.json here, keep walking + } + + const parent = path.dirname(dir); + if (parent === dir) break; + dir = parent; + } + + return undefined; +} + /** Reset cached config — useful for tests */ export function resetConfig(): void { cachedConfig = undefined; } + +/** Reset cached app — useful for tests */ +export function resetApp(): void { + cachedApp = undefined; +} diff --git a/packages/agentcrumbs/src/trail.ts b/packages/agentcrumbs/src/trail.ts index 1100e2d..352a626 100644 --- a/packages/agentcrumbs/src/trail.ts +++ b/packages/agentcrumbs/src/trail.ts @@ -9,7 +9,7 @@ import type { TrailFunction, } from "./types.js"; import { NOOP } from "./noop.js"; -import { isNamespaceEnabled, getCollectorUrl, getFormat } from "./env.js"; +import { isNamespaceEnabled, getCollectorUrl, getFormat, getApp } from "./env.js"; import { getContext, runWithContext, type DebugContext } from "./context.js"; import { ConsoleSink } from "./sinks/console.js"; import { HttpSink } from "./sinks/socket.js"; @@ -100,6 +100,7 @@ function createTrailFunction( const cliSession = getCliSessionId(); const crumb: Crumb = { + app: getApp(), ts: new Date().toISOString(), ns: namespace, msg, diff --git a/packages/agentcrumbs/src/types.ts b/packages/agentcrumbs/src/types.ts index 0c07392..3e5be18 100644 --- a/packages/agentcrumbs/src/types.ts +++ b/packages/agentcrumbs/src/types.ts @@ -10,6 +10,7 @@ export type CrumbType = | "session:end"; export type Crumb = { + app: string; ts: string; ns: string; msg: string; @@ -72,6 +73,7 @@ export type Formatter = { }; export type AgentCrumbsConfig = { + app?: string; ns: string; port?: number; format?: "pretty" | "json"; diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 9e8beee..b293c59 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -22,7 +22,7 @@ importers: version: 16.6.10(@mdx-js/mdx@3.1.1)(@types/estree-jsx@1.0.5)(@types/hast@3.0.4)(@types/mdast@4.0.4)(@types/react@19.2.14)(lucide-react@0.577.0(react@19.2.4))(next@16.1.6(react-dom@19.2.4(react@19.2.4))(react@19.2.4))(react-dom@19.2.4(react@19.2.4))(react@19.2.4)(zod@4.3.6) fumadocs-mdx: specifier: ^14.0.0 - version: 14.2.9(@types/mdast@4.0.4)(@types/mdx@2.0.13)(@types/react@19.2.14)(fumadocs-core@16.6.10(@mdx-js/mdx@3.1.1)(@types/estree-jsx@1.0.5)(@types/hast@3.0.4)(@types/mdast@4.0.4)(@types/react@19.2.14)(lucide-react@0.577.0(react@19.2.4))(next@16.1.6(react-dom@19.2.4(react@19.2.4))(react@19.2.4))(react-dom@19.2.4(react@19.2.4))(react@19.2.4)(zod@4.3.6))(next@16.1.6(react-dom@19.2.4(react@19.2.4))(react@19.2.4))(react@19.2.4)(vite@7.3.1(@types/node@22.19.15)(jiti@2.6.1)(lightningcss@1.31.1)) + version: 14.2.9(@types/mdast@4.0.4)(@types/mdx@2.0.13)(@types/react@19.2.14)(fumadocs-core@16.6.10(@mdx-js/mdx@3.1.1)(@types/estree-jsx@1.0.5)(@types/hast@3.0.4)(@types/mdast@4.0.4)(@types/react@19.2.14)(lucide-react@0.577.0(react@19.2.4))(next@16.1.6(react-dom@19.2.4(react@19.2.4))(react@19.2.4))(react-dom@19.2.4(react@19.2.4))(react@19.2.4)(zod@4.3.6))(next@16.1.6(react-dom@19.2.4(react@19.2.4))(react@19.2.4))(react@19.2.4)(vite@7.3.1(@types/node@22.19.15)(jiti@2.6.1)(lightningcss@1.31.1)(tsx@4.21.0)) fumadocs-ui: specifier: ^16.0.0 version: 16.6.10(@types/react-dom@19.2.3(@types/react@19.2.14))(@types/react@19.2.14)(fumadocs-core@16.6.10(@mdx-js/mdx@3.1.1)(@types/estree-jsx@1.0.5)(@types/hast@3.0.4)(@types/mdast@4.0.4)(@types/react@19.2.14)(lucide-react@0.577.0(react@19.2.4))(next@16.1.6(react-dom@19.2.4(react@19.2.4))(react@19.2.4))(react-dom@19.2.4(react@19.2.4))(react@19.2.4)(zod@4.3.6))(next@16.1.6(react-dom@19.2.4(react@19.2.4))(react@19.2.4))(react-dom@19.2.4(react@19.2.4))(react@19.2.4)(tailwindcss@4.2.1) @@ -58,6 +58,19 @@ importers: specifier: ^5.7.0 version: 5.9.3 + examples/papertrail: + dependencies: + agentcrumbs: + specifier: workspace:* + version: link:../../packages/agentcrumbs + devDependencies: + tsx: + specifier: ^4.19.0 + version: 4.21.0 + typescript: + specifier: ^5.7.0 + version: 5.9.3 + packages/agentcrumbs: devDependencies: '@types/node': @@ -71,7 +84,7 @@ importers: version: 5.9.3 vitest: specifier: ^3.0.0 - version: 3.2.4(@types/debug@4.1.12)(@types/node@25.3.3)(jiti@2.6.1)(lightningcss@1.31.1) + version: 3.2.4(@types/debug@4.1.12)(@types/node@25.3.3)(jiti@2.6.1)(lightningcss@1.31.1)(tsx@4.21.0) packages: @@ -1760,6 +1773,9 @@ packages: resolution: {integrity: sha512-FJhYRoDaiatfEkUK8HKlicmu/3SGFD51q3itKDGoSTysQJBnfOcxU5GxnhE1E6soB76MbT0MBtnKJuXyAx+96Q==} engines: {node: '>=6'} + get-tsconfig@4.13.6: + resolution: {integrity: sha512-shZT/QMiSHc/YBLxxOkMtgSid5HFoauqCE3/exfsEcwg1WkeqjG+V40yBbBrsD+jW2HDXcs28xOfcbm2jI8Ddw==} + github-slugger@2.0.0: resolution: {integrity: sha512-IaOQ9puYtjrkq7Y0Ygl9KDZnrf/aiUJYUpVf89y8kyaxbRG7Y1SrX/jaumrv81vc61+kiMempujsM3Yw7w5qcw==} @@ -2465,6 +2481,9 @@ packages: resolution: {integrity: sha512-gLWKdA5tiv5j/D7ipR47u3ovbVfzFPrctTdw2Ulnpmr6PPVVSvPKGNWu09jXVNlOSLLAeD6CA13bjIelpWttSw==} engines: {node: 20 || >=22} + resolve-pkg-maps@1.0.0: + resolution: {integrity: sha512-seS2Tj26TBVOC2NIc2rOe2y2ZO7efxITtLZcGSOnHHNOQ7CkiUBfw0Iw2ck6xkIhPwLhKNLS8BO+hEpngQlqzw==} + reusify@1.1.0: resolution: {integrity: sha512-g6QUff04oZpHs0eG5p83rFLhHeV00ug/Yf9nZM6fLeUrPguBTkTQOdpAWWspMh55TZfVQDPaN3NQJfbVRAxdIw==} engines: {iojs: '>=1.0.0', node: '>=0.10.0'} @@ -2642,6 +2661,11 @@ packages: tslib@2.8.1: resolution: {integrity: sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==} + tsx@4.21.0: + resolution: {integrity: sha512-5C1sg4USs1lfG0GFb2RLXsdpXqBSEhAaA/0kPL01wxzpMqLILNxIxIOKiILz+cdg/pLnOUxFYOR5yhHU666wbw==} + engines: {node: '>=18.0.0'} + hasBin: true + typescript@5.9.3: resolution: {integrity: sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==} engines: {node: '>=14.17'} @@ -3934,13 +3958,13 @@ snapshots: chai: 5.3.3 tinyrainbow: 2.0.0 - '@vitest/mocker@3.2.4(vite@7.3.1(@types/node@25.3.3)(jiti@2.6.1)(lightningcss@1.31.1))': + '@vitest/mocker@3.2.4(vite@7.3.1(@types/node@25.3.3)(jiti@2.6.1)(lightningcss@1.31.1)(tsx@4.21.0))': dependencies: '@vitest/spy': 3.2.4 estree-walker: 3.0.3 magic-string: 0.30.21 optionalDependencies: - vite: 7.3.1(@types/node@25.3.3)(jiti@2.6.1)(lightningcss@1.31.1) + vite: 7.3.1(@types/node@25.3.3)(jiti@2.6.1)(lightningcss@1.31.1)(tsx@4.21.0) '@vitest/pretty-format@3.2.4': dependencies: @@ -4294,7 +4318,7 @@ snapshots: transitivePeerDependencies: - supports-color - fumadocs-mdx@14.2.9(@types/mdast@4.0.4)(@types/mdx@2.0.13)(@types/react@19.2.14)(fumadocs-core@16.6.10(@mdx-js/mdx@3.1.1)(@types/estree-jsx@1.0.5)(@types/hast@3.0.4)(@types/mdast@4.0.4)(@types/react@19.2.14)(lucide-react@0.577.0(react@19.2.4))(next@16.1.6(react-dom@19.2.4(react@19.2.4))(react@19.2.4))(react-dom@19.2.4(react@19.2.4))(react@19.2.4)(zod@4.3.6))(next@16.1.6(react-dom@19.2.4(react@19.2.4))(react@19.2.4))(react@19.2.4)(vite@7.3.1(@types/node@22.19.15)(jiti@2.6.1)(lightningcss@1.31.1)): + fumadocs-mdx@14.2.9(@types/mdast@4.0.4)(@types/mdx@2.0.13)(@types/react@19.2.14)(fumadocs-core@16.6.10(@mdx-js/mdx@3.1.1)(@types/estree-jsx@1.0.5)(@types/hast@3.0.4)(@types/mdast@4.0.4)(@types/react@19.2.14)(lucide-react@0.577.0(react@19.2.4))(next@16.1.6(react-dom@19.2.4(react@19.2.4))(react@19.2.4))(react-dom@19.2.4(react@19.2.4))(react@19.2.4)(zod@4.3.6))(next@16.1.6(react-dom@19.2.4(react@19.2.4))(react@19.2.4))(react@19.2.4)(vite@7.3.1(@types/node@22.19.15)(jiti@2.6.1)(lightningcss@1.31.1)(tsx@4.21.0)): dependencies: '@mdx-js/mdx': 3.1.1 '@standard-schema/spec': 1.1.0 @@ -4320,7 +4344,7 @@ snapshots: '@types/react': 19.2.14 next: 16.1.6(react-dom@19.2.4(react@19.2.4))(react@19.2.4) react: 19.2.4 - vite: 7.3.1(@types/node@22.19.15)(jiti@2.6.1)(lightningcss@1.31.1) + vite: 7.3.1(@types/node@22.19.15)(jiti@2.6.1)(lightningcss@1.31.1)(tsx@4.21.0) transitivePeerDependencies: - supports-color @@ -4364,6 +4388,10 @@ snapshots: get-nonce@1.0.1: {} + get-tsconfig@4.13.6: + dependencies: + resolve-pkg-maps: 1.0.0 + github-slugger@2.0.0: {} glob-parent@5.1.2: @@ -5391,6 +5419,8 @@ snapshots: glob: 13.0.6 walk-up-path: 4.0.0 + resolve-pkg-maps@1.0.0: {} + reusify@1.1.0: {} rimraf@6.1.3: @@ -5601,6 +5631,13 @@ snapshots: tslib@2.8.1: {} + tsx@4.21.0: + dependencies: + esbuild: 0.27.3 + get-tsconfig: 4.13.6 + optionalDependencies: + fsevents: 2.3.3 + typescript@5.9.3: {} undici-types@6.21.0: {} @@ -5683,13 +5720,13 @@ snapshots: '@types/unist': 3.0.3 vfile-message: 4.0.3 - vite-node@3.2.4(@types/node@25.3.3)(jiti@2.6.1)(lightningcss@1.31.1): + vite-node@3.2.4(@types/node@25.3.3)(jiti@2.6.1)(lightningcss@1.31.1)(tsx@4.21.0): dependencies: cac: 6.7.14 debug: 4.4.3 es-module-lexer: 1.7.0 pathe: 2.0.3 - vite: 7.3.1(@types/node@25.3.3)(jiti@2.6.1)(lightningcss@1.31.1) + vite: 7.3.1(@types/node@25.3.3)(jiti@2.6.1)(lightningcss@1.31.1)(tsx@4.21.0) transitivePeerDependencies: - '@types/node' - jiti @@ -5704,7 +5741,7 @@ snapshots: - tsx - yaml - vite@7.3.1(@types/node@22.19.15)(jiti@2.6.1)(lightningcss@1.31.1): + vite@7.3.1(@types/node@22.19.15)(jiti@2.6.1)(lightningcss@1.31.1)(tsx@4.21.0): dependencies: esbuild: 0.27.3 fdir: 6.5.0(picomatch@4.0.3) @@ -5717,9 +5754,10 @@ snapshots: fsevents: 2.3.3 jiti: 2.6.1 lightningcss: 1.31.1 + tsx: 4.21.0 optional: true - vite@7.3.1(@types/node@25.3.3)(jiti@2.6.1)(lightningcss@1.31.1): + vite@7.3.1(@types/node@25.3.3)(jiti@2.6.1)(lightningcss@1.31.1)(tsx@4.21.0): dependencies: esbuild: 0.27.3 fdir: 6.5.0(picomatch@4.0.3) @@ -5732,12 +5770,13 @@ snapshots: fsevents: 2.3.3 jiti: 2.6.1 lightningcss: 1.31.1 + tsx: 4.21.0 - vitest@3.2.4(@types/debug@4.1.12)(@types/node@25.3.3)(jiti@2.6.1)(lightningcss@1.31.1): + vitest@3.2.4(@types/debug@4.1.12)(@types/node@25.3.3)(jiti@2.6.1)(lightningcss@1.31.1)(tsx@4.21.0): dependencies: '@types/chai': 5.2.3 '@vitest/expect': 3.2.4 - '@vitest/mocker': 3.2.4(vite@7.3.1(@types/node@25.3.3)(jiti@2.6.1)(lightningcss@1.31.1)) + '@vitest/mocker': 3.2.4(vite@7.3.1(@types/node@25.3.3)(jiti@2.6.1)(lightningcss@1.31.1)(tsx@4.21.0)) '@vitest/pretty-format': 3.2.4 '@vitest/runner': 3.2.4 '@vitest/snapshot': 3.2.4 @@ -5755,8 +5794,8 @@ snapshots: tinyglobby: 0.2.15 tinypool: 1.1.1 tinyrainbow: 2.0.0 - vite: 7.3.1(@types/node@25.3.3)(jiti@2.6.1)(lightningcss@1.31.1) - vite-node: 3.2.4(@types/node@25.3.3)(jiti@2.6.1)(lightningcss@1.31.1) + vite: 7.3.1(@types/node@25.3.3)(jiti@2.6.1)(lightningcss@1.31.1)(tsx@4.21.0) + vite-node: 3.2.4(@types/node@25.3.3)(jiti@2.6.1)(lightningcss@1.31.1)(tsx@4.21.0) why-is-node-running: 2.3.0 optionalDependencies: '@types/debug': 4.1.12 diff --git a/pnpm-workspace.yaml b/pnpm-workspace.yaml index 661bdfe..4bfe0be 100644 --- a/pnpm-workspace.yaml +++ b/pnpm-workspace.yaml @@ -1,3 +1,4 @@ packages: - "packages/*" + - "examples/*" - "docs" From b1a6a42bad9749d869d9294c7fd713ad3c308b70 Mon Sep 17 00:00:00 2001 From: Eric Allam Date: Wed, 11 Mar 2026 14:42:58 +0000 Subject: [PATCH 2/7] chore: ignore .tshy build artifacts --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index f2f5d43..cf97d90 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,6 @@ node_modules dist +.tshy *.tsbuildinfo .DS_Store .env From 6b0664eef8e989112ffd1d536a8fe160b8eea216 Mon Sep 17 00:00:00 2001 From: Eric Allam Date: Wed, 11 Mar 2026 15:10:19 +0000 Subject: [PATCH 3/7] feat: add cursor pagination and time windows to query command Replace the old 'last N results' approach with forward pagination: - Add --after and --before for absolute ISO timestamp bounds - Default limit reduced from 100 to 50 for manageable pages - Output includes 'Next page: --after ' when more results exist - Results returned oldest-first for natural forward pagination Update skill to guide agents toward broad queries with pagination instead of aggressive namespace/match filtering. --- docs/content/docs/cli/query.mdx | 38 ++++++++++++------ .../agentcrumbs/skills/agentcrumbs/SKILL.md | 25 +++++++++++- .../agentcrumbs/src/cli/commands/query.ts | 40 +++++++++++++++++-- packages/agentcrumbs/src/cli/index.ts | 6 ++- 4 files changed, 90 insertions(+), 19 deletions(-) diff --git a/docs/content/docs/cli/query.mdx b/docs/content/docs/cli/query.mdx index 14e54ef..28de078 100644 --- a/docs/content/docs/cli/query.mdx +++ b/docs/content/docs/cli/query.mdx @@ -5,7 +5,7 @@ description: "Search historical crumbs" ## `agentcrumbs query` -Query historical crumbs with time ranges and filters. +Query historical crumbs with time windows and cursor-based pagination. ```bash agentcrumbs query --since 5m @@ -15,7 +15,10 @@ agentcrumbs query --since 5m | Flag | Description | | --- | --- | -| `--since ` | Time range (e.g., `5m`, `1h`, `24h`, `7d`) | +| `--since ` | Relative time window (e.g., `5m`, `1h`, `24h`, `7d`) | +| `--after ` | Crumbs after this ISO timestamp (cursor for pagination) | +| `--before ` | Crumbs before this ISO timestamp | +| `--limit ` | Results per page (default: 50) | | `--ns ` | Filter by namespace | | `--tag ` | Filter by tag | | `--session ` | Filter by session ID | @@ -23,30 +26,39 @@ agentcrumbs query --since 5m | `--app ` | Scope to a specific app (default: auto-detect from package.json) | | `--all-apps` | Query crumbs from all apps | | `--json` | JSON output | -| `--limit ` | Maximum number of results | Time units: `s` (seconds), `m` (minutes), `h` (hours), `d` (days). +### Pagination + +Results are returned oldest-first, capped at `--limit` (default 50). When there are more results, the output includes a `Next page:` line with an `--after` timestamp you can use to fetch the next page. + +```bash +# First page +agentcrumbs query --since 5m +# Output: 50 of 128 crumbs. Next page: --after 2026-03-11T14:20:00.123Z + +# Next page +agentcrumbs query --since 5m --after 2026-03-11T14:20:00.123Z +``` + ### Examples ```bash -# Last 5 minutes +# Last 5 minutes (all namespaces) agentcrumbs query --since 5m -# Last hour, filtered by namespace -agentcrumbs query --since 1h --ns auth-service +# Time window with absolute timestamps +agentcrumbs query --after 2026-03-11T14:00:00Z --before 2026-03-11T14:05:00Z -# Filter by tag -agentcrumbs query --tag root-cause +# Paginate through results +agentcrumbs query --since 1h --limit 25 # Filter by session agentcrumbs query --session a1b2c3 -# JSON output with limit -agentcrumbs query --since 1h --json --limit 50 - -# Text search -agentcrumbs query --since 24h --match "connection refused" +# Filter by tag +agentcrumbs query --tag root-cause # Query a specific app agentcrumbs query --since 1h --app my-project diff --git a/packages/agentcrumbs/skills/agentcrumbs/SKILL.md b/packages/agentcrumbs/skills/agentcrumbs/SKILL.md index d43135d..15aef59 100644 --- a/packages/agentcrumbs/skills/agentcrumbs/SKILL.md +++ b/packages/agentcrumbs/skills/agentcrumbs/SKILL.md @@ -82,7 +82,8 @@ agentcrumbs collect # Start HTTP collector (required for query/tail) agentcrumbs tail # Live tail (auto-scoped to current app) agentcrumbs tail --app foo # Tail a specific app agentcrumbs tail --all-apps # Tail all apps -agentcrumbs query --since 5m # Query history (--ns, --tag, --session, --json) +agentcrumbs query --since 5m # Query last 5 minutes (all namespaces, 50 per page) +agentcrumbs query --since 5m --after # Next page agentcrumbs clear # Clear crumbs for current app agentcrumbs clear --all-apps # Clear crumbs for all apps agentcrumbs strip # Remove all crumb markers from source @@ -92,6 +93,28 @@ agentcrumbs --help # Full command reference Most commands accept `--app ` and `--all-apps`. Default is auto-detect from `package.json`. +## Querying crumbs + +Start broad — query a time window across all namespaces, then paginate if there are too many results. Do NOT filter by namespace or match text unless you are looking for something very specific. The whole point of crumbs is seeing the full picture across services. + +```bash +# Start here: get recent crumbs across all services +agentcrumbs query --since 5m + +# Paginate forward (timestamp from "Next page:" in output) +agentcrumbs query --since 5m --after 2026-03-11T14:20:00.123Z + +# Time window with absolute bounds +agentcrumbs query --after 2026-03-11T14:00:00Z --before 2026-03-11T14:05:00Z + +# Smaller pages if context is tight +agentcrumbs query --since 5m --limit 25 + +# Only filter by namespace/match when you have a specific reason +agentcrumbs query --since 5m --tag error +agentcrumbs query --session a1b2c3 +``` + Run `agentcrumbs --help` for detailed options on any command. ## Enable tracing diff --git a/packages/agentcrumbs/src/cli/commands/query.ts b/packages/agentcrumbs/src/cli/commands/query.ts index ab3111b..8258dfb 100644 --- a/packages/agentcrumbs/src/cli/commands/query.ts +++ b/packages/agentcrumbs/src/cli/commands/query.ts @@ -7,10 +7,12 @@ export async function query(args: string[]): Promise { const ns = getFlag(args, "--ns"); const tag = getFlag(args, "--tag"); const since = getFlag(args, "--since"); + const after = getFlag(args, "--after"); + const before = getFlag(args, "--before"); const session = getFlag(args, "--session"); const match = getFlag(args, "--match"); const json = hasFlag(args, "--json"); - const limit = parseInt(getFlag(args, "--limit") ?? "100", 10); + const limit = parseInt(getFlag(args, "--limit") ?? "50", 10); const appCtx = parseAppFlags(args); const showApp = appCtx.allApps; @@ -18,11 +20,31 @@ export async function query(args: string[]): Promise { let filtered = allCrumbs; + // Time window filters if (since) { const cutoff = parseSince(since); filtered = filtered.filter((c) => new Date(c.ts).getTime() >= cutoff); } + if (after) { + const afterMs = new Date(after).getTime(); + if (isNaN(afterMs)) { + process.stderr.write(`Invalid --after timestamp: "${after}". Use ISO 8601 format.\n`); + process.exit(1); + } + filtered = filtered.filter((c) => new Date(c.ts).getTime() > afterMs); + } + + if (before) { + const beforeMs = new Date(before).getTime(); + if (isNaN(beforeMs)) { + process.stderr.write(`Invalid --before timestamp: "${before}". Use ISO 8601 format.\n`); + process.exit(1); + } + filtered = filtered.filter((c) => new Date(c.ts).getTime() < beforeMs); + } + + // Content filters if (ns) { const pattern = new RegExp(`^${ns.replace(/\*/g, ".*")}$`); filtered = filtered.filter((c) => pattern.test(c.ns)); @@ -40,7 +62,10 @@ export async function query(args: string[]): Promise { filtered = filtered.filter((c) => JSON.stringify(c).includes(match)); } - const results = filtered.slice(-limit); + const total = filtered.length; + + // Take first `limit` crumbs (oldest first) for forward pagination + const results = filtered.slice(0, limit); if (results.length === 0) { process.stderr.write("No crumbs found matching filters.\n"); @@ -55,7 +80,16 @@ export async function query(args: string[]): Promise { } } - process.stderr.write(`\n${results.length} crumbs found.\n`); + // Pagination footer + const hasMore = total > results.length; + if (hasMore) { + const lastTs = results[results.length - 1]!.ts; + process.stderr.write( + `\n${results.length} of ${total} crumbs. Next page: --after ${lastTs}\n` + ); + } else { + process.stderr.write(`\n${results.length} crumbs.\n`); + } } function parseSince(since: string): number { diff --git a/packages/agentcrumbs/src/cli/index.ts b/packages/agentcrumbs/src/cli/index.ts index 2e6d19e..61b64e3 100644 --- a/packages/agentcrumbs/src/cli/index.ts +++ b/packages/agentcrumbs/src/cli/index.ts @@ -35,12 +35,14 @@ Tail options: --json Output as JSON Query options: - --since Time filter (e.g., 5m, 1h, 24h) + --since Relative time window (e.g., 5m, 1h, 24h) + --after Crumbs after this ISO timestamp (for pagination) + --before Crumbs before this ISO timestamp + --limit Max results per page (default: 50) --ns Filter by namespace --tag Filter by tag --session Filter by session ID --match Filter by text content - --limit Max results (default: 100) --json Output as JSON Collect options: From d1b9d1616e068df6e62c0989fcdf94b452f44cc8 Mon Sep 17 00:00:00 2001 From: Eric Allam Date: Wed, 11 Mar 2026 15:19:40 +0000 Subject: [PATCH 4/7] feat: add 8-char cursor IDs for query pagination MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace raw timestamp pagination with short cursor IDs. Cursors are 8-char hashes stored in ~/.agentcrumbs/.cursors.json with 1-hour TTL. Flow: query --since 5m → output shows 'Next: --cursor a1b2c3d4' → query --since 5m --cursor a1b2c3d4 for next page. Keep --after/--before for explicit time windows. --- docs/content/docs/cli/query.mdx | 17 ++++-- .../agentcrumbs/skills/agentcrumbs/SKILL.md | 6 +- .../agentcrumbs/src/cli/commands/query.ts | 28 +++++++-- packages/agentcrumbs/src/cli/cursor.ts | 59 +++++++++++++++++++ packages/agentcrumbs/src/cli/index.ts | 5 +- 5 files changed, 101 insertions(+), 14 deletions(-) create mode 100644 packages/agentcrumbs/src/cli/cursor.ts diff --git a/docs/content/docs/cli/query.mdx b/docs/content/docs/cli/query.mdx index 28de078..90182ef 100644 --- a/docs/content/docs/cli/query.mdx +++ b/docs/content/docs/cli/query.mdx @@ -16,8 +16,9 @@ agentcrumbs query --since 5m | Flag | Description | | --- | --- | | `--since ` | Relative time window (e.g., `5m`, `1h`, `24h`, `7d`) | -| `--after ` | Crumbs after this ISO timestamp (cursor for pagination) | +| `--after ` | Crumbs after this ISO timestamp | | `--before ` | Crumbs before this ISO timestamp | +| `--cursor ` | Resume from a previous page (8-char ID from output) | | `--limit ` | Results per page (default: 50) | | `--ns ` | Filter by namespace | | `--tag ` | Filter by tag | @@ -31,27 +32,33 @@ Time units: `s` (seconds), `m` (minutes), `h` (hours), `d` (days). ### Pagination -Results are returned oldest-first, capped at `--limit` (default 50). When there are more results, the output includes a `Next page:` line with an `--after` timestamp you can use to fetch the next page. +Results are returned oldest-first, capped at `--limit` (default 50). When there are more results, the output includes a short cursor ID for the next page. ```bash # First page agentcrumbs query --since 5m -# Output: 50 of 128 crumbs. Next page: --after 2026-03-11T14:20:00.123Z +# Output: 50 crumbs (1-50 of 128). Next: --cursor a1b2c3d4 # Next page -agentcrumbs query --since 5m --after 2026-03-11T14:20:00.123Z +agentcrumbs query --since 5m --cursor a1b2c3d4 +# Output: 50 crumbs (51-100 of 128). Next: --cursor e5f6g7h8 ``` +Cursors expire after 1 hour. You can also use `--after` / `--before` with ISO timestamps for explicit time windows without cursors. + ### Examples ```bash # Last 5 minutes (all namespaces) agentcrumbs query --since 5m +# Paginate through results +agentcrumbs query --since 5m --cursor a1b2c3d4 + # Time window with absolute timestamps agentcrumbs query --after 2026-03-11T14:00:00Z --before 2026-03-11T14:05:00Z -# Paginate through results +# Smaller pages agentcrumbs query --since 1h --limit 25 # Filter by session diff --git a/packages/agentcrumbs/skills/agentcrumbs/SKILL.md b/packages/agentcrumbs/skills/agentcrumbs/SKILL.md index 15aef59..36d995c 100644 --- a/packages/agentcrumbs/skills/agentcrumbs/SKILL.md +++ b/packages/agentcrumbs/skills/agentcrumbs/SKILL.md @@ -101,8 +101,8 @@ Start broad — query a time window across all namespaces, then paginate if ther # Start here: get recent crumbs across all services agentcrumbs query --since 5m -# Paginate forward (timestamp from "Next page:" in output) -agentcrumbs query --since 5m --after 2026-03-11T14:20:00.123Z +# Paginate forward using the cursor from the output +agentcrumbs query --since 5m --cursor a1b2c3d4 # Time window with absolute bounds agentcrumbs query --after 2026-03-11T14:00:00Z --before 2026-03-11T14:05:00Z @@ -115,6 +115,8 @@ agentcrumbs query --since 5m --tag error agentcrumbs query --session a1b2c3 ``` +Results are paginated (50 per page by default). When there are more results, the output includes a `--cursor` ID for the next page. Pass it back to get the next page. + Run `agentcrumbs --help` for detailed options on any command. ## Enable tracing diff --git a/packages/agentcrumbs/src/cli/commands/query.ts b/packages/agentcrumbs/src/cli/commands/query.ts index 8258dfb..869ff50 100644 --- a/packages/agentcrumbs/src/cli/commands/query.ts +++ b/packages/agentcrumbs/src/cli/commands/query.ts @@ -2,6 +2,7 @@ import type { Crumb } from "../../types.js"; import { formatCrumbPretty, formatCrumbJson } from "../format.js"; import { getFlag, hasFlag } from "../args.js"; import { parseAppFlags, readAllCrumbs } from "../app-store.js"; +import { saveCursor, resolveCursor } from "../cursor.js"; export async function query(args: string[]): Promise { const ns = getFlag(args, "--ns"); @@ -9,6 +10,7 @@ export async function query(args: string[]): Promise { const since = getFlag(args, "--since"); const after = getFlag(args, "--after"); const before = getFlag(args, "--before"); + const cursor = getFlag(args, "--cursor"); const session = getFlag(args, "--session"); const match = getFlag(args, "--match"); const json = hasFlag(args, "--json"); @@ -64,8 +66,18 @@ export async function query(args: string[]): Promise { const total = filtered.length; - // Take first `limit` crumbs (oldest first) for forward pagination - const results = filtered.slice(0, limit); + // Resolve cursor to skip offset + let startIndex = 0; + if (cursor) { + const entry = resolveCursor(cursor); + if (!entry) { + process.stderr.write(`Cursor expired or invalid: ${cursor}\n`); + process.exit(1); + } + startIndex = entry.offset; + } + + const results = filtered.slice(startIndex, startIndex + limit); if (results.length === 0) { process.stderr.write("No crumbs found matching filters.\n"); @@ -81,14 +93,20 @@ export async function query(args: string[]): Promise { } // Pagination footer - const hasMore = total > results.length; + const endIndex = startIndex + results.length; + const hasMore = endIndex < total; if (hasMore) { const lastTs = results[results.length - 1]!.ts; + const nextCursor = saveCursor(lastTs, endIndex); process.stderr.write( - `\n${results.length} of ${total} crumbs. Next page: --after ${lastTs}\n` + `\n${results.length} crumbs (${startIndex + 1}-${endIndex} of ${total}). Next: --cursor ${nextCursor}\n` ); } else { - process.stderr.write(`\n${results.length} crumbs.\n`); + if (startIndex > 0) { + process.stderr.write(`\n${results.length} crumbs (${startIndex + 1}-${endIndex} of ${total}).\n`); + } else { + process.stderr.write(`\n${results.length} crumbs.\n`); + } } } diff --git a/packages/agentcrumbs/src/cli/cursor.ts b/packages/agentcrumbs/src/cli/cursor.ts new file mode 100644 index 0000000..b8a18c7 --- /dev/null +++ b/packages/agentcrumbs/src/cli/cursor.ts @@ -0,0 +1,59 @@ +import fs from "node:fs"; +import path from "node:path"; +import os from "node:os"; +import { createHash } from "node:crypto"; + +const CURSOR_FILE = path.join(os.homedir(), ".agentcrumbs", ".cursors.json"); +const MAX_CURSORS = 50; +const CURSOR_TTL = 60 * 60 * 1000; // 1 hour + +type CursorEntry = { + ts: string; + offset: number; + createdAt: number; +}; + +type CursorMap = Record; + +function readCursors(): CursorMap { + try { + return JSON.parse(fs.readFileSync(CURSOR_FILE, "utf-8")) as CursorMap; + } catch { + return {}; + } +} + +function writeCursors(cursors: CursorMap): void { + const dir = path.dirname(CURSOR_FILE); + if (!fs.existsSync(dir)) { + fs.mkdirSync(dir, { recursive: true }); + } + fs.writeFileSync(CURSOR_FILE, JSON.stringify(cursors)); +} + +function generateId(ts: string, offset: number): string { + const hash = createHash("sha256").update(`${ts}:${offset}`).digest("hex"); + return hash.slice(0, 8); +} + +function pruneOld(cursors: CursorMap): CursorMap { + const now = Date.now(); + const entries = Object.entries(cursors) + .filter(([, v]) => now - v.createdAt < CURSOR_TTL) + .sort((a, b) => b[1].createdAt - a[1].createdAt) + .slice(0, MAX_CURSORS); + return Object.fromEntries(entries); +} + +export function saveCursor(ts: string, offset: number): string { + const id = generateId(ts, offset); + const cursors = pruneOld(readCursors()); + cursors[id] = { ts, offset, createdAt: Date.now() }; + writeCursors(cursors); + return id; +} + +export function resolveCursor(id: string): CursorEntry | undefined { + const cursors = readCursors(); + return cursors[id]; +} diff --git a/packages/agentcrumbs/src/cli/index.ts b/packages/agentcrumbs/src/cli/index.ts index 61b64e3..0a0e97c 100644 --- a/packages/agentcrumbs/src/cli/index.ts +++ b/packages/agentcrumbs/src/cli/index.ts @@ -36,9 +36,10 @@ Tail options: Query options: --since Relative time window (e.g., 5m, 1h, 24h) - --after Crumbs after this ISO timestamp (for pagination) + --after Crumbs after this ISO timestamp --before Crumbs before this ISO timestamp - --limit Max results per page (default: 50) + --cursor Resume from a previous page (8-char ID from output) + --limit Results per page (default: 50) --ns Filter by namespace --tag Filter by tag --session Filter by session ID From 642abbceb26f367e09636a8e9994d003ba1933b6 Mon Sep 17 00:00:00 2001 From: Eric Allam Date: Wed, 11 Mar 2026 15:24:57 +0000 Subject: [PATCH 5/7] docs: update all docs and READMEs for app concept and cursor pagination - Fix storage path to ~/.agentcrumbs//crumbs.jsonl everywhere - Add app field to cross-language examples (curl, Python, Go, Rust) - Update CLI examples with --app, --all-apps, --cursor flags - Add AGENTCRUMBS_APP env var to README env var table - Update collect.mdx to show per-app routing - Update multi-service guide with per-app storage --- README.md | 34 +++++++++++++-------- docs/content/docs/cli/collect.mdx | 6 ++-- docs/content/docs/guides/cross-language.mdx | 6 +++- docs/content/docs/guides/multi-service.mdx | 6 ++-- docs/content/docs/index.mdx | 2 +- packages/agentcrumbs/README.md | 34 +++++++++++++-------- 6 files changed, 54 insertions(+), 34 deletions(-) diff --git a/README.md b/README.md index 69f6413..f191bcc 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ Crumbs are development-only. They get stripped before merge and cost nothing whe ``` Service A ──┐ ┌── $ agentcrumbs tail Service B ──┤── fetch() ──> Collector :8374 ──┤── $ agentcrumbs query --since 5m -Service C ──┘ (fire & forget) └── ~/.agentcrumbs/crumbs.jsonl +Service C ──┘ (fire & forget) └── ~/.agentcrumbs//crumbs.jsonl ``` ## Getting started @@ -54,7 +54,7 @@ When something goes wrong, the agent starts the collector and queries the trail: ```bash agentcrumbs collect --quiet & AGENTCRUMBS=1 node app.js -agentcrumbs query --since 5m --ns auth-service +agentcrumbs query --since 5m ``` ``` @@ -62,6 +62,8 @@ auth-service login attempt +0ms { tokenPrefix: "eyJhbGci" } auth-service token decode ok +3ms { userId: "u_8f3k" } auth-service permissions check +8ms { roles: [] } auth-service rejected: no roles +8ms { status: 401 } + +4 crumbs. ``` Now the agent knows: the token is valid, but the user has no roles. The fix is in role assignment, not token validation. @@ -114,8 +116,11 @@ Everything is controlled by a single `AGENTCRUMBS` environment variable. | `auth-*,api-*` | Multiple patterns (comma or space separated) | | `* -internal-*` | Match all except excluded patterns | | `{"ns":"*","port":9999}` | JSON config with full control | +| `{"app":"my-app","ns":"*"}` | Explicit app name | + +JSON config fields: `app` (app name, default auto-detect from package.json), `ns` (namespace filter, required), `port` (collector port, default 8374), `format` (`"pretty"` or `"json"`, default `"pretty"`). -JSON config fields: `ns` (namespace filter, required), `port` (collector port, default 8374), `format` (`"pretty"` or `"json"`, default `"pretty"`). +You can also set `AGENTCRUMBS_APP` to override the app name independently. ## CLI @@ -127,15 +132,17 @@ agentcrumbs collect --quiet & # Start in background agentcrumbs collect --port 9999 # Custom port # Live tail -agentcrumbs tail # All namespaces +agentcrumbs tail # All namespaces (scoped to current app) agentcrumbs tail --ns auth-service # Filter by namespace -agentcrumbs tail --tag perf # Filter by tag +agentcrumbs tail --app my-app # Tail a specific app +agentcrumbs tail --all-apps # Tail all apps -# Query -agentcrumbs query --since 5m # Last 5 minutes -agentcrumbs query --ns auth-service --since 1h -agentcrumbs query --tag root-cause -agentcrumbs query --json --limit 50 +# Query (paginated, 50 per page) +agentcrumbs query --since 5m # Last 5 minutes +agentcrumbs query --since 5m --cursor a1b2c3d4 # Next page +agentcrumbs query --since 1h --limit 25 # Smaller pages +agentcrumbs query --session a1b2c3 # Filter by session +agentcrumbs query --tag root-cause # Filter by tag # Strip agentcrumbs strip --dry-run # Preview removals @@ -143,8 +150,9 @@ agentcrumbs strip # Remove all crumb code agentcrumbs strip --check # CI gate (exits 1 if markers found) # Utilities -agentcrumbs stats # Crumb counts, file size -agentcrumbs clear # Delete stored crumbs +agentcrumbs stats # Crumb counts (current app) +agentcrumbs stats --all-apps # Stats for all apps +agentcrumbs clear # Clear crumbs (current app) ``` Time units: `s` (seconds), `m` (minutes), `h` (hours), `d` (days). @@ -160,7 +168,7 @@ The collector is language-agnostic. Any language with HTTP support can send crum ```bash curl -X POST http://localhost:8374/crumb \ -H "Content-Type: application/json" \ - -d '{"ts":"2026-01-01T00:00:00Z","ns":"shell","msg":"hello","type":"crumb","dt":0,"pid":1}' + -d '{"app":"my-app","ts":"2026-01-01T00:00:00Z","ns":"shell","msg":"hello","type":"crumb","dt":0,"pid":1}' ``` ## Runtime compatibility diff --git a/docs/content/docs/cli/collect.mdx b/docs/content/docs/cli/collect.mdx index 2286625..57ed532 100644 --- a/docs/content/docs/cli/collect.mdx +++ b/docs/content/docs/cli/collect.mdx @@ -11,7 +11,7 @@ The collector is an HTTP server that receives crumbs via `POST /crumb` and write agentcrumbs collect # agentcrumbs collector # http: http://localhost:8374/crumb -# store: ~/.agentcrumbs/crumbs.jsonl +# crumbs stored per-app in ~/.agentcrumbs// # press ctrl+c to stop ``` @@ -50,8 +50,8 @@ agentcrumbs collect --quiet 1. Listens on the specified port for `POST /crumb` requests 2. Validates the JSON body matches the crumb schema -3. Appends each crumb as a line to `crumbs.jsonl` -4. The `tail` and `query` commands read from this file +3. Routes each crumb to per-app storage at `~/.agentcrumbs//crumbs.jsonl` +4. The `tail` and `query` commands read from these files ### Without the collector diff --git a/docs/content/docs/guides/cross-language.mdx b/docs/content/docs/guides/cross-language.mdx index 0f85646..85f176a 100644 --- a/docs/content/docs/guides/cross-language.mdx +++ b/docs/content/docs/guides/cross-language.mdx @@ -13,6 +13,7 @@ Send a `POST` request to `http://localhost:8374/crumb` with a JSON body matching | Field | Type | Description | | --- | --- | --- | +| `app` | `string` | App name (used for per-app storage routing) | | `ts` | `string` | ISO 8601 timestamp | | `ns` | `string` | Namespace | | `msg` | `string` | Message | @@ -38,7 +39,7 @@ Send a `POST` request to `http://localhost:8374/crumb` with a JSON body matching ```bash curl -X POST http://localhost:8374/crumb \ -H "Content-Type: application/json" \ - -d '{"ts":"2026-01-01T00:00:00Z","ns":"shell","msg":"hello","type":"crumb","dt":0,"pid":1}' + -d '{"app":"my-app","ts":"2026-01-01T00:00:00Z","ns":"shell","msg":"hello","type":"crumb","dt":0,"pid":1}' ``` ### Python @@ -50,6 +51,7 @@ from datetime import datetime def crumb(ns, msg, data=None): try: requests.post("http://localhost:8374/crumb", json={ + "app": "my-app", "ts": datetime.utcnow().isoformat() + "Z", "ns": ns, "msg": msg, @@ -69,6 +71,7 @@ crumb("python-service", "processing started", {"items": 42}) ```go func crumb(ns, msg string, data any) { body, _ := json.Marshal(map[string]any{ + "app": "my-app", "ts": time.Now().UTC().Format(time.RFC3339Nano), "ns": ns, "msg": msg, @@ -86,6 +89,7 @@ func crumb(ns, msg string, data any) { ```rust fn crumb(ns: &str, msg: &str) { let body = serde_json::json!({ + "app": "my-app", "ts": chrono::Utc::now().to_rfc3339(), "ns": ns, "msg": msg, diff --git a/docs/content/docs/guides/multi-service.mdx b/docs/content/docs/guides/multi-service.mdx index 9dfd1c1..a608423 100644 --- a/docs/content/docs/guides/multi-service.mdx +++ b/docs/content/docs/guides/multi-service.mdx @@ -10,13 +10,13 @@ agentcrumbs is designed for systems with multiple services running locally. ``` Service A ──┐ ┌── $ agentcrumbs tail Service B ──┤── fetch() ──> Collector :8374 ──┤── $ agentcrumbs query --since 5m -Service C ──┘ (fire & forget) └── ~/.agentcrumbs/crumbs.jsonl +Service C ──┘ (fire & forget) └── ~/.agentcrumbs//crumbs.jsonl ``` 1. Each service imports `agentcrumbs` and calls `trail()` to create namespaced trail functions 2. When `AGENTCRUMBS` is set, crumbs are sent via HTTP to the collector -3. The collector writes crumbs to a shared JSONL file -4. The CLI reads from the JSONL file to provide tail, query, and replay +3. The collector routes crumbs to per-app storage at `~/.agentcrumbs//crumbs.jsonl` +4. The CLI reads from these files to provide tail, query, and replay (auto-scoped to current app) ## Setup diff --git a/docs/content/docs/index.mdx b/docs/content/docs/index.mdx index a69067e..3b73b0e 100644 --- a/docs/content/docs/index.mdx +++ b/docs/content/docs/index.mdx @@ -14,7 +14,7 @@ Crumbs are development-only. They get stripped before merge and cost nothing whe ``` Service A ──┐ ┌── $ agentcrumbs tail Service B ──┤── fetch() ──> Collector :8374 ──┤── $ agentcrumbs query --since 5m -Service C ──┘ (fire & forget) └── ~/.agentcrumbs/crumbs.jsonl +Service C ──┘ (fire & forget) └── ~/.agentcrumbs//crumbs.jsonl ``` ## Why agents need this diff --git a/packages/agentcrumbs/README.md b/packages/agentcrumbs/README.md index 69f6413..f191bcc 100644 --- a/packages/agentcrumbs/README.md +++ b/packages/agentcrumbs/README.md @@ -7,7 +7,7 @@ Crumbs are development-only. They get stripped before merge and cost nothing whe ``` Service A ──┐ ┌── $ agentcrumbs tail Service B ──┤── fetch() ──> Collector :8374 ──┤── $ agentcrumbs query --since 5m -Service C ──┘ (fire & forget) └── ~/.agentcrumbs/crumbs.jsonl +Service C ──┘ (fire & forget) └── ~/.agentcrumbs//crumbs.jsonl ``` ## Getting started @@ -54,7 +54,7 @@ When something goes wrong, the agent starts the collector and queries the trail: ```bash agentcrumbs collect --quiet & AGENTCRUMBS=1 node app.js -agentcrumbs query --since 5m --ns auth-service +agentcrumbs query --since 5m ``` ``` @@ -62,6 +62,8 @@ auth-service login attempt +0ms { tokenPrefix: "eyJhbGci" } auth-service token decode ok +3ms { userId: "u_8f3k" } auth-service permissions check +8ms { roles: [] } auth-service rejected: no roles +8ms { status: 401 } + +4 crumbs. ``` Now the agent knows: the token is valid, but the user has no roles. The fix is in role assignment, not token validation. @@ -114,8 +116,11 @@ Everything is controlled by a single `AGENTCRUMBS` environment variable. | `auth-*,api-*` | Multiple patterns (comma or space separated) | | `* -internal-*` | Match all except excluded patterns | | `{"ns":"*","port":9999}` | JSON config with full control | +| `{"app":"my-app","ns":"*"}` | Explicit app name | + +JSON config fields: `app` (app name, default auto-detect from package.json), `ns` (namespace filter, required), `port` (collector port, default 8374), `format` (`"pretty"` or `"json"`, default `"pretty"`). -JSON config fields: `ns` (namespace filter, required), `port` (collector port, default 8374), `format` (`"pretty"` or `"json"`, default `"pretty"`). +You can also set `AGENTCRUMBS_APP` to override the app name independently. ## CLI @@ -127,15 +132,17 @@ agentcrumbs collect --quiet & # Start in background agentcrumbs collect --port 9999 # Custom port # Live tail -agentcrumbs tail # All namespaces +agentcrumbs tail # All namespaces (scoped to current app) agentcrumbs tail --ns auth-service # Filter by namespace -agentcrumbs tail --tag perf # Filter by tag +agentcrumbs tail --app my-app # Tail a specific app +agentcrumbs tail --all-apps # Tail all apps -# Query -agentcrumbs query --since 5m # Last 5 minutes -agentcrumbs query --ns auth-service --since 1h -agentcrumbs query --tag root-cause -agentcrumbs query --json --limit 50 +# Query (paginated, 50 per page) +agentcrumbs query --since 5m # Last 5 minutes +agentcrumbs query --since 5m --cursor a1b2c3d4 # Next page +agentcrumbs query --since 1h --limit 25 # Smaller pages +agentcrumbs query --session a1b2c3 # Filter by session +agentcrumbs query --tag root-cause # Filter by tag # Strip agentcrumbs strip --dry-run # Preview removals @@ -143,8 +150,9 @@ agentcrumbs strip # Remove all crumb code agentcrumbs strip --check # CI gate (exits 1 if markers found) # Utilities -agentcrumbs stats # Crumb counts, file size -agentcrumbs clear # Delete stored crumbs +agentcrumbs stats # Crumb counts (current app) +agentcrumbs stats --all-apps # Stats for all apps +agentcrumbs clear # Clear crumbs (current app) ``` Time units: `s` (seconds), `m` (minutes), `h` (hours), `d` (days). @@ -160,7 +168,7 @@ The collector is language-agnostic. Any language with HTTP support can send crum ```bash curl -X POST http://localhost:8374/crumb \ -H "Content-Type: application/json" \ - -d '{"ts":"2026-01-01T00:00:00Z","ns":"shell","msg":"hello","type":"crumb","dt":0,"pid":1}' + -d '{"app":"my-app","ts":"2026-01-01T00:00:00Z","ns":"shell","msg":"hello","type":"crumb","dt":0,"pid":1}' ``` ## Runtime compatibility From 4508610de0de09ce65554886f0ed7c5d1b3ead08 Mon Sep 17 00:00:00 2001 From: Eric Allam Date: Wed, 11 Mar 2026 15:27:55 +0000 Subject: [PATCH 6/7] docs: strengthen query guidance in skills against over-filtering - Add over-filtering as critical mistake #1 in core skill - Rewrite querying section with explicit CORRECT/AVOID examples - Fix CLI quick reference to show --cursor instead of --after - Add query + cursor examples to init skill template - Add inline guidance note to init template CLI section --- .../agentcrumbs/skills/agentcrumbs/SKILL.md | 37 +++++++++++-------- .../skills/agentcrumbs/init/SKILL.md | 15 +++++--- 2 files changed, 31 insertions(+), 21 deletions(-) diff --git a/packages/agentcrumbs/skills/agentcrumbs/SKILL.md b/packages/agentcrumbs/skills/agentcrumbs/SKILL.md index 36d995c..90b56b5 100644 --- a/packages/agentcrumbs/skills/agentcrumbs/SKILL.md +++ b/packages/agentcrumbs/skills/agentcrumbs/SKILL.md @@ -83,7 +83,7 @@ agentcrumbs tail # Live tail (auto-scoped to current app) agentcrumbs tail --app foo # Tail a specific app agentcrumbs tail --all-apps # Tail all apps agentcrumbs query --since 5m # Query last 5 minutes (all namespaces, 50 per page) -agentcrumbs query --since 5m --after # Next page +agentcrumbs query --since 5m --cursor # Next page (cursor from output) agentcrumbs clear # Clear crumbs for current app agentcrumbs clear --all-apps # Clear crumbs for all apps agentcrumbs strip # Remove all crumb markers from source @@ -95,27 +95,33 @@ Most commands accept `--app ` and `--all-apps`. Default is auto-detect fro ## Querying crumbs -Start broad — query a time window across all namespaces, then paginate if there are too many results. Do NOT filter by namespace or match text unless you are looking for something very specific. The whole point of crumbs is seeing the full picture across services. +**IMPORTANT: Query broadly, paginate — don't filter narrowly.** The value of crumbs is seeing what happened across ALL services, not just one. Filtering to a single namespace or adding match filters defeats the purpose — you'll miss the cross-service interactions that reveal the real bug. + +The right approach: +1. Query a time window with no namespace filter +2. Read the first page of results +3. Use `--cursor` to paginate forward if you need more ```bash -# Start here: get recent crumbs across all services +# CORRECT: broad query, paginate through results agentcrumbs query --since 5m +agentcrumbs query --since 5m --cursor a1b2c3d4 # cursor from previous output -# Paginate forward using the cursor from the output -agentcrumbs query --since 5m --cursor a1b2c3d4 - -# Time window with absolute bounds -agentcrumbs query --after 2026-03-11T14:00:00Z --before 2026-03-11T14:05:00Z +# CORRECT: narrow the time window, not the namespaces +agentcrumbs query --after 2026-03-11T14:00:00Z --before 2026-03-11T14:01:00Z -# Smaller pages if context is tight +# CORRECT: smaller pages to save context agentcrumbs query --since 5m --limit 25 -# Only filter by namespace/match when you have a specific reason -agentcrumbs query --since 5m --tag error +# CORRECT: filter by session (still shows all namespaces in that session) agentcrumbs query --session a1b2c3 + +# AVOID: don't filter to one namespace unless you already know the root cause +# agentcrumbs query --since 5m --ns auth-service # too narrow! +# agentcrumbs query --since 5m --match "userId:123" # too narrow! ``` -Results are paginated (50 per page by default). When there are more results, the output includes a `--cursor` ID for the next page. Pass it back to get the next page. +Results are paginated (50 per page by default). When there are more results, the output includes a short `--cursor` ID for the next page. Run `agentcrumbs --help` for detailed options on any command. @@ -148,9 +154,10 @@ agentcrumbs stats --all-apps # Per-app statistics ## Critical mistakes -1. **Missing markers** — Every crumb line needs `// @crumbs` or a `#region @crumbs` block. Without them, `strip` can't clean up. -2. **Creating trail() in hot paths** — `trail()` parses the env var each call. Create once at module scope, use `child()` for per-request context. -3. **No collector running** — Without `agentcrumbs collect`, crumbs go to stderr only and can't be queried. Start the collector before reproducing issues. +1. **Over-filtering queries** — Do NOT add `--ns` or `--match` filters to narrow results. Use `--limit` and `--cursor` to paginate instead. Filtering to one namespace hides cross-service bugs. If there are too many results, narrow the time window or reduce `--limit`, not the namespaces. +2. **Missing markers** — Every crumb line needs `// @crumbs` or a `#region @crumbs` block. Without them, `strip` can't clean up. +3. **Creating trail() in hot paths** — `trail()` parses the env var each call. Create once at module scope, use `child()` for per-request context. +4. **No collector running** — Without `agentcrumbs collect`, crumbs go to stderr only and can't be queried. Start the collector before reproducing issues. ## Further discovery diff --git a/packages/agentcrumbs/skills/agentcrumbs/init/SKILL.md b/packages/agentcrumbs/skills/agentcrumbs/init/SKILL.md index b0b8f08..febf3e7 100644 --- a/packages/agentcrumbs/skills/agentcrumbs/init/SKILL.md +++ b/packages/agentcrumbs/skills/agentcrumbs/init/SKILL.md @@ -141,13 +141,16 @@ production. ### CLI ```bash -AGENTCRUMBS=1 node app.js # enable tracing (or AGENTCRUMBS_APP=my-project) -agentcrumbs collect # start collector (multi-service) -agentcrumbs tail # live tail (scoped to this app) -agentcrumbs tail --app my-project # tail a specific app -agentcrumbs clear # clear crumbs for this app -agentcrumbs strip # remove crumbs before merge +AGENTCRUMBS=1 node app.js # enable tracing +agentcrumbs collect # start collector +agentcrumbs tail # live tail (scoped to this app) +agentcrumbs query --since 5m # query recent crumbs (all namespaces) +agentcrumbs query --since 5m --cursor # paginate (cursor from output) +agentcrumbs clear # clear crumbs for this app +agentcrumbs strip # remove crumbs before merge ``` + +When querying, always start broad (all namespaces) and paginate with `--cursor`. Do not filter by `--ns` or `--match` — the value is in seeing the full cross-service picture. ```` Adapt the example above to the actual discovered namespaces and app name. From ca4aace3fab53f5d535a606d619f2399bc1fae60 Mon Sep 17 00:00:00 2001 From: Eric Allam Date: Wed, 11 Mar 2026 15:30:02 +0000 Subject: [PATCH 7/7] chore: add changeset for app concept and query pagination --- .changeset/app-concept-and-pagination.md | 29 ++++++++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 .changeset/app-concept-and-pagination.md diff --git a/.changeset/app-concept-and-pagination.md b/.changeset/app-concept-and-pagination.md new file mode 100644 index 0000000..2ac610a --- /dev/null +++ b/.changeset/app-concept-and-pagination.md @@ -0,0 +1,29 @@ +--- +"agentcrumbs": minor +--- + +Add app concept for project-level crumb isolation and cursor-based query pagination. + +**App isolation:** +- Every crumb is stamped with an `app` name, auto-detected from the nearest `package.json` +- Crumbs stored per-app at `~/.agentcrumbs//crumbs.jsonl` +- Collector routes incoming crumbs to per-app stores +- All CLI commands scope to the current app by default +- Override with `--app `, `--all-apps`, `AGENTCRUMBS_APP` env var, or `app` field in JSON config + +**Query pagination:** +- New `--cursor` flag for forward pagination with short 8-char cursor IDs +- New `--after` and `--before` flags for absolute ISO timestamp windows +- Default limit reduced from 100 to 50 per page +- Results returned oldest-first with `Next: --cursor ` in output when more pages exist + +**New files:** +- `src/cli/app-store.ts` — shared helper for app context resolution across CLI commands +- `src/cli/cursor.ts` — cursor storage with 1-hour TTL + +**Breaking changes:** +- `Crumb` type now has a required `app: string` field +- `AgentCrumbsConfig` type now has an optional `app?: string` field +- `CollectorServer` no longer exposes `getStore()` (routes to per-app stores internally) +- Storage location changed from `~/.agentcrumbs/crumbs.jsonl` to `~/.agentcrumbs//crumbs.jsonl` +- Legacy flat-file crumbs (without `app` field) are still readable as app `"unknown"`