2147fc1696
Refactors PR #190's hardcoded Unstructured.io integration into a flexible, extensible plugin system supporting multiple text extraction engines. - **`DocumentProcessor` ABC**: Abstract interface for all processors - **`ProcessorRegistry`**: Central registry for discovery and routing - **`ProcessingResult`**: Standardized output format across processors - **`UnstructuredProcessor`**: Refactored from `UnstructuredClient` - **`TesseractProcessor`**: Local OCR for images (lightweight alternative) - **`CustomHTTPProcessor`**: Generic wrapper for custom HTTP APIs - New `get_document_processor_config()` returns structured config - Supports enabling/disabling individual processors - Per-processor configuration via environment variables - **Breaking Change**: `ENABLE_UNSTRUCTURED_PARSING` replaced with: - `ENABLE_DOCUMENT_PROCESSING=true/false` (master switch) - `ENABLE_UNSTRUCTURED=true/false` (per-processor) - `ENABLE_TESSERACT=true/false` - `ENABLE_CUSTOM_PROCESSOR=true/false` - `parse_document()` now uses `ProcessorRegistry` - Auto-selects appropriate processor based on MIME type - Processor priority system (Unstructured=10, Tesseract=5, Custom=1) - `initialize_document_processors()` registers processors at startup - Integrated into both BasicAuth and OAuth lifespans - Graceful degradation if processors fail to initialize ```env ENABLE_DOCUMENT_PROCESSING=false ENABLE_UNSTRUCTURED=false UNSTRUCTURED_API_URL=http://unstructured:8000 UNSTRUCTURED_STRATEGY=auto # auto|fast|hi_res UNSTRUCTURED_LANGUAGES=eng,deu ENABLE_TESSERACT=false TESSERACT_LANG=eng ENABLE_CUSTOM_PROCESSOR=false CUSTOM_PROCESSOR_URL=http://localhost:9000/process CUSTOM_PROCESSOR_TYPES=application/pdf,image/jpeg ``` - **Removed**: `tests/test_unstructured_config.py` (legacy tests) - **Added**: `tests/unit/test_document_processor_config.py` - 7 unit tests for new config system - Tests individual and multi-processor configurations - **Added**: - `nextcloud_mcp_server/document_processors/__init__.py` - `nextcloud_mcp_server/document_processors/base.py` - `nextcloud_mcp_server/document_processors/registry.py` - `nextcloud_mcp_server/document_processors/unstructured.py` - `nextcloud_mcp_server/document_processors/tesseract.py` - `nextcloud_mcp_server/document_processors/custom_http.py` - `tests/unit/test_document_processor_config.py` - **Modified**: - `nextcloud_mcp_server/config.py` - New plugin config system - `nextcloud_mcp_server/app.py` - Processor initialization - `nextcloud_mcp_server/utils/document_parser.py` - Uses registry - `nextcloud_mcp_server/server/webdav.py` - Import updates - `env.sample` - New configuration format - `docker-compose.yml` - (profile changes from previous work) - **Removed**: - `nextcloud_mcp_server/client/unstructured_client.py` - Replaced by UnstructuredProcessor - `tests/test_unstructured_config.py` - Replaced with new tests ✅ **Extensible**: Add processors without modifying core code ✅ **Testable**: Mock processors for unit tests ✅ **Configurable**: Enable only needed processors ✅ **Flexible**: Choose fast (Tesseract) vs accurate (Unstructured) ✅ **Opt-in**: Disabled by default, no mandatory dependencies Users upgrading from PR #190 need to update environment variables: ```bash ENABLE_UNSTRUCTURED_PARSING=true ENABLE_DOCUMENT_PROCESSING=true ENABLE_UNSTRUCTURED=true ``` 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
93 lines
2.8 KiB
Python
93 lines
2.8 KiB
Python
"""Document parsing utilities using pluggable processor registry."""
|
|
|
|
import base64
|
|
import logging
|
|
from typing import Optional, Tuple
|
|
|
|
from nextcloud_mcp_server.config import get_document_processor_config
|
|
from nextcloud_mcp_server.document_processors import (
|
|
ProcessorError,
|
|
get_registry,
|
|
)
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
|
|
def is_parseable_document(content_type: Optional[str]) -> bool:
|
|
"""Check if a document type can be parsed by any registered processor.
|
|
|
|
Args:
|
|
content_type: The MIME type of the document
|
|
|
|
Returns:
|
|
True if any processor can handle this type, False otherwise
|
|
"""
|
|
if not content_type:
|
|
return False
|
|
|
|
config = get_document_processor_config()
|
|
if not config["enabled"]:
|
|
return False
|
|
|
|
registry = get_registry()
|
|
processor = registry.find_processor(content_type)
|
|
return processor is not None
|
|
|
|
|
|
async def parse_document(
|
|
content: bytes, content_type: Optional[str], filename: Optional[str] = None
|
|
) -> Tuple[str, dict]:
|
|
"""Parse a document using registered processors.
|
|
|
|
This function uses the processor registry to find an appropriate
|
|
processor for the given document type and extract text from it.
|
|
|
|
Args:
|
|
content: The document content as bytes
|
|
content_type: The MIME type of the document
|
|
filename: Optional filename to help with format detection
|
|
|
|
Returns:
|
|
Tuple of (parsed_text, metadata) where:
|
|
- parsed_text: The extracted text content
|
|
- metadata: Additional metadata about the parsing
|
|
|
|
Raises:
|
|
ValueError: If the document type is not supported
|
|
Exception: If parsing fails
|
|
"""
|
|
if not content_type:
|
|
raise ValueError("Content type is required for document parsing")
|
|
|
|
config = get_document_processor_config()
|
|
if not config["enabled"]:
|
|
raise ValueError("Document processing is disabled")
|
|
|
|
registry = get_registry()
|
|
|
|
logger.debug(f"Parsing document of type '{content_type}'")
|
|
|
|
try:
|
|
# Process using registry (auto-selects processor based on MIME type)
|
|
result = await registry.process(
|
|
content=content,
|
|
content_type=content_type,
|
|
filename=filename,
|
|
)
|
|
|
|
logger.info(f"Successfully parsed document with '{result.processor}' processor")
|
|
|
|
return result.text, result.metadata
|
|
|
|
except ProcessorError as e:
|
|
logger.error(f"Document processing failed: {e}")
|
|
# Fallback to base64 with error metadata
|
|
parsed_text = f"Document could not be parsed. Base64 content: {base64.b64encode(content).decode('ascii')[:200]}..."
|
|
metadata = {
|
|
"mime_type": content_type,
|
|
"text_length": len(parsed_text),
|
|
"parsing_method": "fallback_base64",
|
|
"error": str(e),
|
|
}
|
|
return parsed_text, metadata
|