docs: add ADR-007 for background vector database synchronization

Add comprehensive ADR-007 documenting background vector database
synchronization architecture using anyio TaskGroups for in-process
concurrency. This supersedes ADR-003's conceptual background worker.

Key decisions:
- In-process architecture using anyio TaskGroups (not Celery)
- Scanner task runs hourly, detects changes via timestamp comparison
- In-memory asyncio.Queue for pending documents
- Pool of 3 concurrent processor tasks for I/O-bound embedding workloads
- Qdrant metadata as single source of truth for indexing state
- Simple user controls: enable/disable with status visibility

Benefits:
- Single container deployment (was 3: mcp, celery-worker, celery-beat)
- No distributed task queue infrastructure
- Shared process state (no volume coordination)
- Sufficient throughput for I/O-bound embedding APIs
- Simpler debugging and deployment

Update ADR-003 status to "Superseded by ADR-007" with reference link.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2025-11-08 20:32:49 +01:00
parent 31ff8a71bf
commit dc93da2ea0
2 changed files with 940 additions and 1 deletions
@@ -1,7 +1,9 @@
# ADR-003: Vector Database and Semantic Search Architecture
## Status
Proposed
Superseded by ADR-007
**Note**: This ADR was never implemented. The core technical decisions (Qdrant, embeddings, hybrid search) remain valid and are incorporated into ADR-007, which adds user-controlled background job management, task queuing, multi-user scheduling, and web UI integration. See [ADR-007: Background Vector Sync with User-Controlled Job Management](./ADR-007-background-vector-sync-job-management.md) for the implemented architecture.
## Context