refactor: Transform document parsing into pluggable processor architecture

Refactors PR #190's hardcoded Unstructured.io integration into a flexible,
extensible plugin system supporting multiple text extraction engines.

- **`DocumentProcessor` ABC**: Abstract interface for all processors
- **`ProcessorRegistry`**: Central registry for discovery and routing
- **`ProcessingResult`**: Standardized output format across processors

- **`UnstructuredProcessor`**: Refactored from `UnstructuredClient`
- **`TesseractProcessor`**: Local OCR for images (lightweight alternative)
- **`CustomHTTPProcessor`**: Generic wrapper for custom HTTP APIs

- New `get_document_processor_config()` returns structured config
- Supports enabling/disabling individual processors
- Per-processor configuration via environment variables
- **Breaking Change**: `ENABLE_UNSTRUCTURED_PARSING` replaced with:
  - `ENABLE_DOCUMENT_PROCESSING=true/false` (master switch)
  - `ENABLE_UNSTRUCTURED=true/false` (per-processor)
  - `ENABLE_TESSERACT=true/false`
  - `ENABLE_CUSTOM_PROCESSOR=true/false`

- `parse_document()` now uses `ProcessorRegistry`
- Auto-selects appropriate processor based on MIME type
- Processor priority system (Unstructured=10, Tesseract=5, Custom=1)

- `initialize_document_processors()` registers processors at startup
- Integrated into both BasicAuth and OAuth lifespans
- Graceful degradation if processors fail to initialize

```env
ENABLE_DOCUMENT_PROCESSING=false

ENABLE_UNSTRUCTURED=false
UNSTRUCTURED_API_URL=http://unstructured:8000
UNSTRUCTURED_STRATEGY=auto  # auto|fast|hi_res
UNSTRUCTURED_LANGUAGES=eng,deu

ENABLE_TESSERACT=false
TESSERACT_LANG=eng

ENABLE_CUSTOM_PROCESSOR=false
CUSTOM_PROCESSOR_URL=http://localhost:9000/process
CUSTOM_PROCESSOR_TYPES=application/pdf,image/jpeg
```

- **Removed**: `tests/test_unstructured_config.py` (legacy tests)
- **Added**: `tests/unit/test_document_processor_config.py`
  - 7 unit tests for new config system
  - Tests individual and multi-processor configurations

- **Added**:
  - `nextcloud_mcp_server/document_processors/__init__.py`
  - `nextcloud_mcp_server/document_processors/base.py`
  - `nextcloud_mcp_server/document_processors/registry.py`
  - `nextcloud_mcp_server/document_processors/unstructured.py`
  - `nextcloud_mcp_server/document_processors/tesseract.py`
  - `nextcloud_mcp_server/document_processors/custom_http.py`
  - `tests/unit/test_document_processor_config.py`

- **Modified**:
  - `nextcloud_mcp_server/config.py` - New plugin config system
  - `nextcloud_mcp_server/app.py` - Processor initialization
  - `nextcloud_mcp_server/utils/document_parser.py` - Uses registry
  - `nextcloud_mcp_server/server/webdav.py` - Import updates
  - `env.sample` - New configuration format
  - `docker-compose.yml` - (profile changes from previous work)

- **Removed**:
  - `nextcloud_mcp_server/client/unstructured_client.py` - Replaced by UnstructuredProcessor
  - `tests/test_unstructured_config.py` - Replaced with new tests

 **Extensible**: Add processors without modifying core code
 **Testable**: Mock processors for unit tests
 **Configurable**: Enable only needed processors
 **Flexible**: Choose fast (Tesseract) vs accurate (Unstructured)
 **Opt-in**: Disabled by default, no mandatory dependencies

Users upgrading from PR #190 need to update environment variables:
```bash
ENABLE_UNSTRUCTURED_PARSING=true

ENABLE_DOCUMENT_PROCESSING=true
ENABLE_UNSTRUCTURED=true
```

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2025-10-25 19:21:38 +02:00
parent a19017c686
commit 2147fc1696
20 changed files with 2027 additions and 529 deletions
+41 -79
View File
@@ -1,62 +1,46 @@
"""Document parsing utilities based on the "unstructured" microservice"""
"""Document parsing utilities using pluggable processor registry."""
import base64
import logging
from typing import Optional, Tuple
from nextcloud_mcp_server.config import is_unstructured_parsing_enabled
from nextcloud_mcp_server.config import get_document_processor_config
from nextcloud_mcp_server.document_processors import (
ProcessorError,
get_registry,
)
logger = logging.getLogger(__name__)
# Mapping of MIME types to their corresponding parsing strategies
PARSEABLE_MIME_TYPES = {
# PDF documents
"application/pdf": "pdf",
# Microsoft Word documents
"application/vnd.openxmlformats-officedocument.wordprocessingml.document": "docx",
"application/msword": "doc",
# Microsoft PowerPoint
"application/vnd.openxmlformats-officedocument.presentationml.presentation": "pptx",
"application/vnd.ms-powerpoint": "ppt",
# Microsoft Excel
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet": "xlsx",
"application/vnd.ms-excel": "xls",
# Other document formats
"application/rtf": "rtf",
"text/rtf": "rtf",
"application/vnd.oasis.opendocument.text": "odt",
"application/epub+zip": "epub",
# Email formats
"message/rfc822": "eml",
"application/vnd.ms-outlook": "msg",
# Image formats (for OCR)
"image/jpeg": "image",
"image/png": "image",
"image/tiff": "image",
"image/bmp": "image",
}
def is_parseable_document(content_type: Optional[str]) -> bool:
"""Check if a document type can be parsed.
"""Check if a document type can be parsed by any registered processor.
Args:
content_type: The MIME type of the document
Returns:
True if the document can be parsed, False otherwise
True if any processor can handle this type, False otherwise
"""
if not content_type:
return False
# Handle content types with additional parameters (e.g., "application/pdf; charset=utf-8")
base_content_type = content_type.split(";")[0].strip().lower()
return base_content_type in PARSEABLE_MIME_TYPES
config = get_document_processor_config()
if not config["enabled"]:
return False
registry = get_registry()
processor = registry.find_processor(content_type)
return processor is not None
async def parse_document(
content: bytes, content_type: Optional[str], filename: Optional[str] = None
) -> Tuple[str, dict]:
"""Parse a document using the Unstructured API.
"""Parse a document using registered processors.
This function uses the processor registry to find an appropriate
processor for the given document type and extract text from it.
Args:
content: The document content as bytes
@@ -72,59 +56,37 @@ async def parse_document(
ValueError: If the document type is not supported
Exception: If parsing fails
"""
if not is_parseable_document(content_type):
raise ValueError(f"Document type '{content_type}' is not supported for parsing")
if not content_type:
raise ValueError("Content type is required for document parsing")
base_content_type = (
content_type.split(";")[0].strip().lower() if content_type else ""
)
doc_type = PARSEABLE_MIME_TYPES.get(base_content_type, "unknown")
config = get_document_processor_config()
if not config["enabled"]:
raise ValueError("Document processing is disabled")
logger.debug(f"Parsing document of type '{doc_type}' (MIME: {content_type})")
registry = get_registry()
# Check if unstructured parsing is enabled via environment
if is_unstructured_parsing_enabled():
logger.debug("Using Unstructured API for parsing")
try:
from nextcloud_mcp_server.client.unstructured_client import (
UnstructuredClient,
)
logger.debug(f"Parsing document of type '{content_type}'")
client = UnstructuredClient()
# The client will automatically use environment configuration
# (UNSTRUCTURED_STRATEGY and UNSTRUCTURED_LANGUAGES)
return await client.partition_document(
content=content,
filename=filename or f"document.{doc_type}",
content_type=content_type,
)
except Exception as e:
logger.error(f"Unstructured API parsing failed: {e}")
# If unstructured parsing fails, return base64 as fallback
import base64
parsed_text = f"Document could not be parsed. Base64 content: {base64.b64encode(content).decode('ascii')[:200]}..."
metadata = {
"document_type": doc_type,
"mime_type": content_type,
"element_count": 0,
"text_length": len(parsed_text),
"parsing_method": "fallback_base64",
"error": str(e),
}
return parsed_text, metadata
else:
logger.debug(
"Unstructured parsing is disabled, returning base64 encoded content as fallback"
try:
# Process using registry (auto-selects processor based on MIME type)
result = await registry.process(
content=content,
content_type=content_type,
filename=filename,
)
import base64
logger.info(f"Successfully parsed document with '{result.processor}' processor")
return result.text, result.metadata
except ProcessorError as e:
logger.error(f"Document processing failed: {e}")
# Fallback to base64 with error metadata
parsed_text = f"Document could not be parsed. Base64 content: {base64.b64encode(content).decode('ascii')[:200]}..."
metadata = {
"document_type": doc_type,
"mime_type": content_type,
"element_count": 0,
"text_length": len(parsed_text),
"parsing_method": "fallback_base64",
"error": str(e),
}
return parsed_text, metadata