book-harvester — Document Knowledge Harvesting Platform

"Knowledge is power. This tool harvests the power — from books, not just videos."

Ingest long-form documents (books, application notes, scientific papers), preserve the full text verbatim as organized chapter-by-chapter markdown, and publish into the livewiki vault with a TOC.md map, a ≤1000-word document brief, ≤200-word chapter briefs, takeaways, and normalized tags — so Nax agents (and the owner) can retrieve exactly the right chapters into context before deciding anything.

Status: v0.1.0 (pipeline proof). The end-to-end path is real and tested: a text-layer PDF goes in; TOC.md + chapter files + chunk sidecars + provenance.json land in the vault. Reports on this page are limited to what shipped — see Status & roadmap for the v0.2 deferrals (OCR, EPUB/DOCX readers, the LangGraph StateGraph executor, webhooks/outbox, Postgres/Redis-backed stores).

---

Quick start

Backend

# 1. Install (dev extras: pytest, hypothesis, syrupy, ruff, mypy, …)
pip install -e ".[dev]"

# 2. Configure (secrets only in .env; .env.example has placeholders)
cp .env.example .env

# 3. Run the API (dev port 8000; production host port is 8320, PLAN §9)
uvicorn src.main:app --reload --port 8000

OpenAPI UI: http://127.0.0.1:8000/docs · ReDoc: /redoc · probes: /healthz, /readyz, /metrics.

Zero-infra smoke test (no Redis/Postgres/vault, no network — uses the offline stub LLM transport):

book-harvester quick ./sample.pdf --stub
book-harvester run ./sample.pdf --stub --vault-root ./vault

With real model credentials configured in .env, drop --stub (the router reads config/models.yaml).

Dashboard (Next.js operator console)

cd dashboard
npm install
npm run dev        # http://localhost:3000; proxies the API at http://127.0.0.1:8320

Override the proxy target with API_UPSTREAM_URL. Dashboard details, the five-theme token contract (default obsidian), and its browser e2e suite live in dashboard/README.md and dashboard/THEME.md.

---

Architecture overview

A document flows through seven ordered stages. The stage registry is open/closed: each stage registers with register_stage(name, depends_on=…, outputs=…), and the orchestrator walks the topological order. The v0.1 executor is the shipped OrchestratorJobRunner, which composes the real stage modules behind the JobRunner seam — the same seam the v0.2 LangGraph StateGraph + PostgresSaver executor will plug into without touching the API.

ingest → detect → normalize → chunk → brief → tag → publish
  │        │          │          │        │       │       │
  │        │          │          │        │       │       └─ validate vault tree,
  │        │          │          │        │       │          write TOC/chapters/
  │        │          │          │        │       │          sidecars/provenance,
  │        │          │          │        │       │          GraphStore edges, outbox
  │        │          │          │        │       └─ domain classification + kebab tags
  │        │          │          │        └─ TOC model, doc/chapter briefs, takeaways
  │        │          │          └─ sentence-boundary chunks + overlap markers
  │        │          └─ verbatim chapters + reversible offset sidecar
  │        └─ format route (pdf_text / pdf_scanned / epub / docx), privacy, sha256_text
  └─ streaming spool, 500 MB gate, server-side SHA-256, magic-byte sniff, dedup

Stage	Module	Notes
ingest	src/pipeline/ingest.py	Streaming (no full-file RAM buffering); server-side SHA-256; magic-byte MIME sniff; exact-duplicate short-circuit (D-13).
detect	src/pipeline/detect.py	Format route + privacy (G1) + sha256_text. Text-layer PDF is the v0.1 route; scanned/EPUB/DOCX quarantine with an operator note.
normalize	src/pipeline/normalize.py	Verbatim-only ops (encoding repair, de-hyphenation, whitespace collapse, header/footer strip, D-03), reversible-logged via page/char offsets.
chunk	src/pipeline/chunker.py	tiktoken o200k_base counter (D-10): target ≈ 4000 tokens, overlap ≥ 400 at every boundary, sentence-safe; 16 000-char backstop.
brief	src/pipeline/brief.py	TOC.md model, doc brief ≤1000 words, chapter briefs ≤200 words, takeaways both scopes (D-04).
tag	src/pipeline/tagger.py	LLM domain classification against config/domains.yaml (D-16 taxonomy), 8–15 doc / 3–8 chapter kebab tags.
publish	src/pipeline/publisher.py	Validates frontmatter pre-write, writes the vault tree, GraphStore edges, outbox rows; append-only ## Agent notes.

Pipeline state (PipelineState) is checkpointable and carries the identity (document_id uuid7, trace_id), source file facts, stage results, cost, and errors. Every pipeline log line carries document_id + trace_id (structlog JSON).

The vault (livewiki conformance)

Root: <vault>/<domain>/<doc-slug>/ — all writes conform to ~/workspace/livewiki/SPEC.md; frontmatter is validated before write and always carries source: agent + agent: book-harvester.

<vault>/<domain>/<doc-slug>/
├── TOC.md                        # chapter map + ≤1000-word doc brief + takeaways
├── 01-<chapter-slug>.md          # ≤200-word brief + ## Full text (verbatim) + takeaways
├── 01-<chapter-slug>.chunks.json # machine sidecar (offsets, encoder version) — never hand-edited
├── _doc.chunks.json              # document-level chunk index
└── provenance.json               # hashes, rights, edition chain

Body markdown is verbatim — LLM output lives only in the navigation/distillation layers (TOC, briefs, takeaways, tags, sidecars), never in the body (D-03).

---

CLI reference (book-harvester)

Command	Purpose
book-harvester run <file> [--vault-root PATH] [--privacy standard|local] [--stub]	Run the full chain to the vault and print the resulting paths + cost. --stub uses the network-free offline backend.
book-harvester quick <file> [--privacy …] [--stub]	Local-only fast path: ingest → normalize → brief to stdout with zero Redis/Postgres/vault side effects (D-15). Budget guard and the privacy: local seam still apply.
book-harvester upload <file> [--privacy …] [--domain HINT] [--in-process] [--api URL]	Upload to the API and print the 202 {document_id, job_id} payload.
book-harvester status [--document ID] [--in-process] [--api URL]	System readiness plus one document's job health.
book-harvester cost [--document ID] [--today] [--month] [--month-of YYYY-MM]	Per-document / daily / monthly ledger summaries.
book-harvester daemon [--mode observe|supervised|autonomous]	Report the runtime contract for a daemon mode.
book-harvester version	Print the installed package version.

upload/status/cost/daemon talk to a running API over HTTP by default (--api or BOOK_HARVESTER_API_URL), or run in-process with --in-process (BOOK_HARVESTER_CLI_MODE=in-process). Failures render the project envelope {"detail": "message", "code": "ERROR_CODE"} on stderr (exit 2 usage / 1 runtime).

---

API surface

REST under /api/v1/ (Pydantic response models; OpenAPI at /docs and /redoc, schema at /openapi.json). Error bodies are the flat {detail, code} envelope.

• POST /api/v1/documents — multipart upload, ≤500 MB (202; 413 beyond ceiling; 409 on exact duplicate).
• GET /api/v1/documents — list (?limit=20&offset=0&domain=&status=).
• GET /api/v1/documents/{id} — detail + stage statuses + cost summary.
• GET /api/v1/documents/{id}/toc — rendered TOC model (dashboard shortcut).
• DELETE /api/v1/documents/{id} — quarantine + vault tombstone (never silent).
• GET /api/v1/jobs/{job_id} — job tracking (stage, progress, error).
• GET /api/v1/cost/document/{id} · /daily · /monthly · /provider/{name} — ledger reads.
• POST /api/v1/webhooks/{source} — HMAC-SHA256 verified inbound ack.
• GET /healthz, /readyz, /metrics — liveness / honest readiness / Prometheus.

Write endpoints require an API key (X-API-Key/Bearer) outside dev/test; reads stay open for the dashboard.

---

Verification commands

# Lint + format
ruff check src/ tests/ && ruff format --check src/ tests/

# Types
mypy --strict src/

# Default suite (hermetic; coverage gate 80%)
pytest tests/ -v --cov=src --cov-fail-under=80 -m "not integration and not slow"

# Integration suite (self-contained e2e always runs; live Redis/Postgres probe
# runs when DATABASE_URL + REDIS_URL point at reachable services, else skips)
pytest tests/ -m integration

# Security
pip-audit && bandit -r src/ && gitleaks detect

# Local dev
uvicorn src.main:app --reload --port 8000
book-harvester quick sample.pdf --stub | head -50

# Deploy topology (mikulas; PLAN §9 — expect PORTS_CLEAR for the 832x family)
ssh -p 2231 agentuser@nazimyildiz.com 'ss -ltn | grep -E ":(832[0-4]|9102)" || echo PORTS_CLEAR'

---

Repo layout

• src/ — Python/FastAPI backend: pipeline stages, REST API, Typer CLI, LLM router, cost ledger, webhooks, observability
• dashboard/ — Next.js operator console (Obsidian theme family; THEME.md token contract)
• config/ — pipeline.yaml, models.yaml, domains.yaml, budget.yaml, pricing.yaml
• tests/ — tests/integration/ carries the -m integration suite
• deploy/ — compose, Caddy, Prometheus/Grafana, deploy + verification scripts
• .opencode/agents/ — project agent team (registry in .agents.local.md)

Deploy topology

Compose + Caddy serve the books service on the 832x family (zero collision with the sibling youtube-analyzer's 831x): API prod 8320, staging 8321, Prometheus 8322, Grafana 8323, dashboard 8324, worker metrics 9102. Ports verified PORTS_CLEAR on mikulas (D-02). See deploy/README.md.

---

Status & roadmap

Shipped in v0.1.0

• Full pipeline chain (ingest → detect → normalize → chunk → brief → tag → publish) with the shipped OrchestratorJobRunner; a real text-layer PDF produces vault artifacts via book-harvester run and via POST /api/v1/documents.
• REST API (documents/jobs/costs/health/webhooks) + OpenAPI.
• CLI (run, quick, upload, status, cost, daemon, version).
• Vault publisher: TOC.md, chapter files, chunk sidecars, provenance, append-only ## Agent notes (livewiki SPEC-conformant).
• LLM router with fallback chain + circuit breaker, per-document cost ledger, budget guard, and the G2 privacy seam (privacy: local never reaches a cloud provider).
• Tagging/domain taxonomy (D-16), dashboard shell + THEME.md, compose/Caddy 832x, Forgejo CI gates.

Deferred to v0.2 (documented, not claimed as shipped here)

• OCR path for scanned PDFs (worker-local ocrmypdf subprocess, D-11) — v0.1 routes pdf_scanned to quarantine with an operator note.
• EPUB and DOCX readers (EPUB via stdlib zipfile + lxml, D-09) — route stubs in v0.1.
• LangGraph StateGraph + PostgresSaver executor (the JobRunner seam is in place), Redis Streams queue, Postgres-backed stores, webhooks + transactional outbox relay.
• Staging encryption + orphan sweeper / secure wipe (monitor stage, D-12).

Docs

• PLAN.md — authoritative spec (vision, architecture, vault schema, CLI/API, milestones)
• DECISIONS.md — decision register (D-01 … D-16)
• REVIEW.md — review + security register
• TOOLS.md — tooling notes