cb39b3fca4
Enable users to tune document chunking parameters to match their embedding model and content type by adding DOCUMENT_CHUNK_SIZE and DOCUMENT_CHUNK_OVERLAP environment variables. - **config.py**: Added `document_chunk_size` (default: 512) and `document_chunk_overlap` (default: 50) configuration fields with validation: - Ensures overlap < chunk_size - Warns if chunk_size < 100 words - Prevents negative overlap values - **processor.py**: Updated DocumentChunker instantiation to use config settings instead of hardcoded values (line 174-177) - **tests/unit/test_config.py**: Added TestChunkConfigValidation class with 9 tests covering: - Default values - Valid configurations - Validation errors (overlap >= chunk_size, negative overlap) - Warning for small chunk sizes - Environment variable loading - **docs/configuration.md**: Added comprehensive "Document Chunking Configuration" section with: - Chunk size selection guidance (256-384 vs 512 vs 768-1024 words) - Overlap recommendations (10-20% of chunk size) - Configuration examples for different use cases - Added env vars to reference table - **docs/semantic-search-architecture.md**: Added "Document Chunking Strategy" section with: - Chunking process explanation - Example showing sliding window behavior - Search behavior with chunks - Tuning recommendations - **env.sample**: Added complete "Semantic Search & Vector Sync Configuration" section with: - Vector sync settings - Qdrant configuration (3 modes) - Ollama embedding service - Document chunking configuration - **docker-compose.yml**: Added commented examples for DOCUMENT_CHUNK_SIZE and DOCUMENT_CHUNK_OVERLAP with usage notes \`\`\`bash DOCUMENT_CHUNK_SIZE=512 DOCUMENT_CHUNK_OVERLAP=50 \`\`\` 1. \`overlap\` must be less than \`chunk_size\` 2. \`overlap\` cannot be negative 3. Warning issued if \`chunk_size\` < 100 words **Precise matching** (small notes, specific queries): \`\`\`bash DOCUMENT_CHUNK_SIZE=256 DOCUMENT_CHUNK_OVERLAP=25 \`\`\` **Balanced** (default, general purpose): \`\`\`bash DOCUMENT_CHUNK_SIZE=512 DOCUMENT_CHUNK_OVERLAP=50 \`\`\` **Contextual** (long documents, broader topics): \`\`\`bash DOCUMENT_CHUNK_SIZE=1024 DOCUMENT_CHUNK_OVERLAP=100 \`\`\` ✅ **User control** - Tune chunking to match embedding model capabilities ✅ **Experimentation** - Test different chunk sizes for optimal results ✅ **Model alignment** - Match chunk size to embedding context window ✅ **Backward compatible** - Defaults maintain existing behavior ✅ **Well validated** - Comprehensive tests prevent misconfiguration All 22 config validation tests pass (9 new tests for chunking): - Default values work correctly - Validation prevents invalid configurations - Environment variables load properly - Warning system works as expected With configurable chunk sizes, users can now experiment with different Ollama embedding models and tune chunk parameters for optimal semantic search quality. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
199 lines
7.2 KiB
Plaintext
199 lines
7.2 KiB
Plaintext
# Nextcloud Instance
|
|
NEXTCLOUD_HOST=
|
|
|
|
# ===== AUTHENTICATION MODE =====
|
|
# Choose ONE of the following:
|
|
|
|
# Option 1: OAuth2/OIDC (RECOMMENDED - More Secure)
|
|
# - Requires Nextcloud OIDC app installed and configured
|
|
# - Admin must enable "Dynamic Client Registration" in OIDC app settings
|
|
# - Leave NEXTCLOUD_USERNAME and NEXTCLOUD_PASSWORD empty to use OAuth mode
|
|
# - OAuth client credentials are stored encrypted in SQLite (TOKEN_STORAGE_DB)
|
|
# - Optional: Pre-register client and provide credentials (otherwise auto-registers)
|
|
NEXTCLOUD_OIDC_CLIENT_ID=
|
|
NEXTCLOUD_OIDC_CLIENT_SECRET=
|
|
NEXTCLOUD_MCP_SERVER_URL=http://localhost:8000
|
|
|
|
# OAuth Storage Configuration (SQLite storage for OAuth clients and refresh tokens)
|
|
# TOKEN_ENCRYPTION_KEY: Required for encrypting OAuth client secrets and refresh tokens
|
|
# Generate with: python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
|
|
#TOKEN_ENCRYPTION_KEY=
|
|
# TOKEN_STORAGE_DB: Path to SQLite database (default: /app/data/tokens.db)
|
|
#TOKEN_STORAGE_DB=/app/data/tokens.db
|
|
|
|
# ===== ADR-004 PROGRESSIVE CONSENT CONFIGURATION =====
|
|
# Enable Progressive Consent mode (dual OAuth flows)
|
|
# When enabled: Flow 1 for client auth, Flow 2 for Nextcloud resource access
|
|
# When disabled: Uses existing hybrid flow (backward compatible)
|
|
|
|
# MCP Server OAuth Client Configuration
|
|
# The MCP server's own OAuth client credentials for Flow 2
|
|
# If not set, will use dynamic client registration
|
|
#MCP_SERVER_CLIENT_ID=
|
|
#MCP_SERVER_CLIENT_SECRET=
|
|
|
|
# Allowed MCP Client IDs (comma-separated list)
|
|
# Client IDs that are allowed to authenticate in Flow 1
|
|
# Examples: claude-desktop,continue-dev,zed-editor
|
|
#ALLOWED_MCP_CLIENTS=claude-desktop,continue-dev,zed-editor
|
|
|
|
# Token cache configuration for Token Broker Service
|
|
# Cache TTL in seconds (default: 300 = 5 minutes)
|
|
#TOKEN_CACHE_TTL=300
|
|
# Early refresh threshold in seconds (default: 30)
|
|
#TOKEN_CACHE_EARLY_REFRESH=30
|
|
|
|
# Option 2: Basic Authentication (LEGACY - Less Secure)
|
|
# - Requires username and password
|
|
# - Credentials stored in environment variables
|
|
# - Use only for backward compatibility or if OAuth unavailable
|
|
# - If these are set, OAuth mode is disabled
|
|
NEXTCLOUD_USERNAME=
|
|
NEXTCLOUD_PASSWORD=
|
|
|
|
# ============================================
|
|
# Document Processing Configuration
|
|
# ============================================
|
|
# Enable document processing (PDF, DOCX, images, etc.)
|
|
# Set to false to disable all document processing
|
|
ENABLE_DOCUMENT_PROCESSING=false
|
|
|
|
# Default processor to use when multiple are available
|
|
# Options: unstructured, tesseract, custom
|
|
DOCUMENT_PROCESSOR=unstructured
|
|
|
|
# ============================================
|
|
# Unstructured.io Processor
|
|
# ============================================
|
|
# Enable Unstructured processor (requires unstructured service in docker-compose)
|
|
# This is a cloud-based/API processor supporting many document types
|
|
ENABLE_UNSTRUCTURED=false
|
|
|
|
# Unstructured API endpoint
|
|
UNSTRUCTURED_API_URL=http://unstructured:8000
|
|
|
|
# Request timeout in seconds (default: 120)
|
|
# OCR operations can take 30-120 seconds for large documents
|
|
UNSTRUCTURED_TIMEOUT=120
|
|
|
|
# Parsing strategy: auto, fast, hi_res
|
|
# - auto: Automatically choose based on document type
|
|
# - fast: Fast parsing without OCR
|
|
# - hi_res: High-resolution with OCR (slowest, most accurate)
|
|
UNSTRUCTURED_STRATEGY=auto
|
|
|
|
# OCR languages (comma-separated ISO 639-3 codes)
|
|
# Common: eng=English, deu=German, fra=French, spa=Spanish
|
|
UNSTRUCTURED_LANGUAGES=eng,deu
|
|
|
|
# Progress reporting interval in seconds (default: 10)
|
|
# During long-running OCR operations, progress notifications are sent to the MCP client
|
|
# at this interval to prevent timeouts and provide status updates
|
|
PROGRESS_INTERVAL=10
|
|
|
|
# ============================================
|
|
# Tesseract Processor (Local OCR)
|
|
# ============================================
|
|
# Enable Tesseract processor (requires tesseract binary installed)
|
|
# This is a local, lightweight OCR solution for images only
|
|
ENABLE_TESSERACT=false
|
|
|
|
# Path to tesseract executable (optional, auto-detected if in PATH)
|
|
#TESSERACT_CMD=/usr/bin/tesseract
|
|
|
|
# OCR language (e.g., eng, deu, eng+deu for multiple)
|
|
TESSERACT_LANG=eng
|
|
|
|
# ============================================
|
|
# Custom Processor (Your own API)
|
|
# ============================================
|
|
# Enable custom document processor via HTTP API
|
|
ENABLE_CUSTOM_PROCESSOR=false
|
|
|
|
# Unique name for your processor
|
|
#CUSTOM_PROCESSOR_NAME=my_ocr
|
|
|
|
# Your custom processor API endpoint
|
|
#CUSTOM_PROCESSOR_URL=http://localhost:9000/process
|
|
|
|
# Optional API key for authentication
|
|
#CUSTOM_PROCESSOR_API_KEY=your-api-key-here
|
|
|
|
# Request timeout in seconds
|
|
#CUSTOM_PROCESSOR_TIMEOUT=60
|
|
|
|
# Comma-separated MIME types your processor supports
|
|
#CUSTOM_PROCESSOR_TYPES=application/pdf,image/jpeg,image/png
|
|
|
|
# ============================================
|
|
# Semantic Search & Vector Sync Configuration
|
|
# ============================================
|
|
# EXPERIMENTAL: Semantic search for Notes app (multi-app support planned)
|
|
# Requires: Qdrant vector database + Ollama embedding service
|
|
# Disabled by default
|
|
|
|
# Enable background vector indexing
|
|
VECTOR_SYNC_ENABLED=false
|
|
|
|
# Document scan interval in seconds (default: 300 = 5 minutes)
|
|
# How often to check for new/updated documents
|
|
#VECTOR_SYNC_SCAN_INTERVAL=300
|
|
|
|
# Concurrent indexing workers (default: 3)
|
|
# Number of parallel workers for embedding generation
|
|
#VECTOR_SYNC_PROCESSOR_WORKERS=3
|
|
|
|
# Max queued documents (default: 10000)
|
|
# Maximum documents waiting to be processed
|
|
#VECTOR_SYNC_QUEUE_MAX_SIZE=10000
|
|
|
|
# ============================================
|
|
# Qdrant Vector Database Configuration
|
|
# ============================================
|
|
# Choose ONE of three modes:
|
|
# 1. In-memory mode (default): Set neither QDRANT_URL nor QDRANT_LOCATION
|
|
# 2. Persistent local: Set QDRANT_LOCATION=/path/to/data
|
|
# 3. Network mode: Set QDRANT_URL=http://qdrant:6333
|
|
|
|
# Network mode: URL to Qdrant service
|
|
#QDRANT_URL=http://qdrant:6333
|
|
|
|
# Local mode: Path to store vectors (use :memory: for in-memory)
|
|
#QDRANT_LOCATION=:memory:
|
|
|
|
# API key for network mode (optional)
|
|
#QDRANT_API_KEY=
|
|
|
|
# Collection name (optional - auto-generated if not set)
|
|
# Auto-generation format: {deployment-id}-{model-name}
|
|
# Allows safe model switching and multi-server deployments
|
|
#QDRANT_COLLECTION=nextcloud_content
|
|
|
|
# ============================================
|
|
# Ollama Embedding Service Configuration
|
|
# ============================================
|
|
# Ollama endpoint for embeddings (if not set, uses SimpleEmbeddingProvider fallback)
|
|
#OLLAMA_BASE_URL=http://ollama:11434
|
|
|
|
# Embedding model to use (default: nomic-embed-text, 768 dimensions)
|
|
# Changing this creates a new collection (requires re-embedding all documents)
|
|
#OLLAMA_EMBEDDING_MODEL=nomic-embed-text
|
|
|
|
# Verify SSL certificates (default: true)
|
|
#OLLAMA_VERIFY_SSL=true
|
|
|
|
# ============================================
|
|
# Document Chunking Configuration
|
|
# ============================================
|
|
# Configure how documents are split before embedding
|
|
|
|
# Words per chunk (default: 512)
|
|
# Smaller chunks (256-384): More precise, less context, more storage
|
|
# Larger chunks (768-1024): More context, less precise, less storage
|
|
#DOCUMENT_CHUNK_SIZE=512
|
|
|
|
# Overlapping words between chunks (default: 50)
|
|
# Recommended: 10-20% of chunk size
|
|
# Preserves context across chunk boundaries
|
|
#DOCUMENT_CHUNK_OVERLAP=50
|