yt-harvester

YouTube knowledge harvesting platform. Monitor channels, analyze content with fabric-derived patterns, publish structured knowledge to your livewiki vault, and build a cross-video knowledge graph.

Quick Start

# Install
pip install -e ".[dev]"

# Configure
cp .env.example .env  # fill in real values

Set API_KEY in .env — all mutation endpoints (POST/DELETE) require it via the X-API-Key header (or Authorization: Bearer).

# Run dev server
uvicorn src.main:app --reload --port 8000

# CLI
yt-harvester --help
yt-harvester quick https://youtube.com/watch?v=...
yt-harvester daemon
yt-harvester cost
yt-harvester status

Docker Deployment

# Production (app + postgres + redis + prometheus + grafana)
docker compose up -d

# Staging (isolated instances on port 8311)
docker compose -f docker-compose.staging.yml up -d

See docs/deployment.md for full deployment guide.

Architecture

Channel/Video URL → Ingestor → Transcriber → Analyzer → Domain Specialist → Publisher
                                                                              ├── livewiki vault
                                                                              ├── graphify graph
                                                                              └── webhooks

See PLAN.md for full architecture.

Environments

Env	URL	Branch
Production	yt.nazimyildiz.com	main
Staging	staging.yt.nazimyildiz.com	main (pre-release)
Development	dev.yt.nazimyildiz.com	develop
Local	localhost:8000	feature/*

Development

ruff check src/ tests/          # lint
mypy --strict src/              # typecheck
pytest tests/ -v --cov=src      # test

Project Structure

src/
├── pipeline/           # Core processing stages
│   ├── ingestor.py     # Video metadata fetch (yt-dlp)
│   ├── transcriber.py  # Captions → whisper fallback
│   ├── analyzer.py     # Fabric pattern execution
│   ├── publisher.py    # livewiki + graphify + webhooks
│   ├── monitor.py      # RSS feed polling
│   └── specialists/    # Domain enrichment agents
│       ├── finance.py
│       └── longevity.py
├── api/                # FastAPI REST endpoints
├── cli/                # Typer CLI
├── models/             # Pydantic schemas
├── config/             # Configuration loading
└── patterns/           # Vendored fabric patterns (markdown)
config/
├── domains.yaml        # Domain specialist registry
└── models.yaml         # LLM provider routing

License

MIT