nextcloud-mcp-server

brandon/nextcloud-mcp-server

Fork 0

Commit Graph

Author	SHA1	Message	Date
Chris Coutinho	e575c8e57b	feat(vector): Support multiple embedding models with auto-generated collection names This PR enables safe switching between embedding models and multi-server deployments by implementing auto-generated Qdrant collection names based on deployment ID and model name. ## Problem Previously, all deployments used a single hardcoded collection name "nextcloud_content", which caused two critical issues: 1. Dimension mismatches when switching models: Changing OLLAMA_EMBEDDING_MODEL (e.g., nomic-embed-text at 768D → all-minilm at 384D) would cause runtime errors as vectors couldn't be inserted into a collection with incompatible dimensions. 2. Collection collisions in multi-server setups: Multiple MCP servers sharing a single Qdrant instance would overwrite each other's data, making horizontal scaling impossible. ## Solution ### Auto-Generated Collection Naming Collections are now automatically named using the pattern: \`{deployment-id}-{model-name}\` Deployment ID: Uses \`OTEL_SERVICE_NAME\` if configured (and not default value), otherwise falls back to \`hostname\` for simple Docker deployments. Model Name: From \`OLLAMA_EMBEDDING_MODEL\` with path separators sanitized. Examples: - \`my-mcp-server-nomic-embed-text\` (with OTEL_SERVICE_NAME=my-mcp-server) - \`mcp-container-all-minilm\` (simple Docker, hostname=mcp-container) Override: Users can still set \`QDRANT_COLLECTION\` explicitly to bypass auto-generation for backward compatibility. ### Dimension Validation Added startup validation that checks collection dimensions match the embedding service. If a mismatch is detected, the server fails fast with a clear error message explaining: - Expected vs actual dimensions - Likely cause (model change) - Solutions (delete collection, use different name, or revert model) ### Improved Sampling Error Handling Enhanced MCP sampling rejection handling to treat user rejections as normal behavior rather than errors: - User rejections ("rejected", "denied") → INFO log, no traceback - Unsupported clients → INFO log, no traceback - Other MCP errors → WARNING log, no traceback - Unexpected errors → ERROR log WITH traceback This aligns with the MCP specification where clients SHOULD prompt users for approval/denial of sampling requests. ## Changes ### Core Implementation - nextcloud_mcp_server/config.py: Added \`get_collection_name()\` method with deployment ID detection and model name sanitization - nextcloud_mcp_server/vector/qdrant_client.py: Dimension validation on collection open with helpful error messages - nextcloud_mcp_server/vector/{scanner,processor}.py: Updated to use \`get_collection_name()\` - nextcloud_mcp_server/auth/userinfo_routes.py: Vector sync status uses \`get_collection_name()\` - nextcloud_mcp_server/server/semantic.py: - Updated semantic search tools to use \`get_collection_name()\` - Improved sampling rejection error handling (McpError vs Exception) ### Documentation - docs/semantic-search-architecture.md: New comprehensive architecture document (557 lines) covering background sync, semantic search flow, RAG implementation, and deployment modes - docs/configuration.md: Added detailed "Qdrant Collection Naming" section with examples and multi-server deployment guidance - docker-compose.yml: Added comments explaining collection naming behavior - README.md: Updated semantic search descriptions to clarify experimental status, Notes-only support, and infrastructure requirements ## Migration Guide For existing single-server deployments: Option 1 (Recommended): Use explicit collection name for continuity \`\`\`bash QDRANT_COLLECTION=nextcloud_content # Keep existing collection \`\`\` Option 2: Allow auto-generation and re-embed \`\`\`bash # Remove QDRANT_COLLECTION override # New collection will be created based on deployment ID + model # Requires re-embedding all documents (may take time) \`\`\` For new multi-server deployments: Set unique OTEL service names per server: \`\`\`bash # Server 1 OTEL_SERVICE_NAME=mcp-prod OLLAMA_EMBEDDING_MODEL=nomic-embed-text # → Collection: "mcp-prod-nomic-embed-text" # Server 2 OTEL_SERVICE_NAME=mcp-staging OLLAMA_EMBEDDING_MODEL=nomic-embed-text # → Collection: "mcp-staging-nomic-embed-text" \`\`\` ## Benefits ✅ Safe model switching: Each model gets its own collection, preventing dimension mismatch errors ✅ Multi-server support: Multiple MCP servers can share one Qdrant instance without conflicts ✅ Clear ownership: Collection names show which deployment and model owns the data ✅ Better error messages: Dimension validation provides actionable guidance ✅ Backward compatible: Existing deployments can continue using \`QDRANT_COLLECTION\` override ## Testing Validated with: - Single-server deployments (default hostname-based naming) - Multi-server deployments (OTEL service name-based naming) - Model switching scenarios (dimension validation) - Collection override scenarios (backward compatibility) Next steps: Testing various Ollama embedding models to investigate optimal chunk sizes and performance characteristics. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>	2025-11-10 01:18:30 +01:00
Chris Coutinho	72232f937a	refactor: migrate vector sync from asyncio.Queue to anyio memory object streams Replace asyncio.Queue with anyio.create_memory_object_stream() throughout the vector sync system for better library consistency and improved shutdown semantics. ## Changes Made scanner.py: - Changed parameter type from `asyncio.Queue` to `MemoryObjectSendStream[DocumentTask]` - Replaced all `await document_queue.put()` calls with `await send_stream.send()` - Wrapped scanner loop in `async with send_stream:` context manager for automatic cleanup - Updated log messages: "Queued" → "Sent" - Removed `import asyncio` (no longer needed) processor.py: - Changed parameter type from `asyncio.Queue` to `MemoryObjectReceiveStream[DocumentTask]` - Replaced `asyncio.wait_for(document_queue.get(), timeout=1.0)` with `anyio.fail_after(1.0)` + `await receive_stream.receive()` - Removed all `document_queue.task_done()` calls (not needed with streams) - Added `anyio.EndOfStream` exception handling for graceful shutdown when scanner closes - Removed `import asyncio` (no longer needed) app.py: - Removed `import asyncio` from top-level imports - Added `from anyio.streams.memory import MemoryObjectReceiveStream, MemoryObjectSendStream` - Updated AppContext dataclass: - Replaced `document_queue: Optional[asyncio.Queue]` with: - `document_send_stream: Optional[MemoryObjectSendStream]` - `document_receive_stream: Optional[MemoryObjectReceiveStream]` - Updated `app_lifespan_basic()`: - Replaced `asyncio.Queue(maxsize=...)` with `anyio.create_memory_object_stream(max_buffer_size=...)` - Pass `send_stream` to scanner_task - Pass `receive_stream.clone()` to each processor_task (enables multiple consumers) - Updated AppContext yield to include both streams - Updated `starlette_lifespan()`: - Same changes as app_lifespan_basic for streamable-http transport - Removed `import asyncio as asyncio_module` (no longer needed) - Updated app.state storage to use send_stream and receive_stream semantic.py: - Updated `nc_get_vector_sync_status()` tool: - Access `document_receive_stream` instead of `document_queue` from lifespan context - Use `stream_stats.current_buffer_used` instead of `queue.qsize()` for pending count - More reliable metrics (qsize() was not guaranteed accurate) ## Benefits 1. Library Consistency: Pure anyio throughout codebase (was mixing asyncio.Queue with anyio.Event and anyio.create_task_group) 2. Graceful Shutdown: `async with send_stream:` automatically closes stream on exit, signaling EndOfStream to all processors 3. Better Timeout Handling: `anyio.fail_after()` is more idiomatic than `asyncio.wait_for()` 4. Stream Cloning: Easy to add multiple consumers via `receive_stream.clone()` 5. Better Statistics: `.statistics()` provides accurate buffer metrics (qsize() was unreliable) 6. Type Safety: Separate send/receive types prevent accidental misuse 7. No task_done() tracking: Streams handle completion automatically ## Testing - ✅ All 69 unit tests passing - ✅ All 5 smoke tests passing - ✅ No regressions in functionality - ✅ Graceful shutdown behavior improved ## References - https://anyio.readthedocs.io/en/stable/why.html#queue-fix - https://anyio.readthedocs.io/en/stable/streams.html#memory-object-streams 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>	2025-11-09 06:43:44 +01:00
Chris Coutinho	4b026e9aa0	feat: implement ADR-009 - refactor semantic search to use generic semantic:read scope This implements ADR-009, which documents the decision to use a generic `semantic:read` OAuth scope instead of requiring all app-specific scopes for semantic search functionality. Changes: - Created new `nextcloud_mcp_server/models/semantic.py` with semantic search models - SemanticSearchResult (with new doc_type field for multi-app support) - SemanticSearchResponse - SamplingSearchResponse - VectorSyncStatusResponse - Created new `nextcloud_mcp_server/server/semantic.py` with semantic search tools - nc_semantic_search (renamed from nc_notes_semantic_search) - nc_semantic_search_answer (renamed from nc_notes_semantic_search_answer) - nc_get_vector_sync_status (renamed from nc_notes_get_vector_sync_status) - All tools now use @require_scopes("semantic:read") instead of "notes:read" - Updated `nextcloud_mcp_server/server/notes.py` - Removed semantic search tools (moved to semantic.py) - Removed semantic search model imports - Removed unused MCP imports (ModelHint, ModelPreferences, etc.) - Updated `nextcloud_mcp_server/models/notes.py` - Removed semantic search models (moved to semantic.py) - Updated `nextcloud_mcp_server/app.py` - Import configure_semantic_tools - Register semantic tools when VECTOR_SYNC_ENABLED=true - Updated `nextcloud_mcp_server/server/__init__.py` - Export configure_semantic_tools - Updated tests - tests/integration/test_sampling.py: Use new tool names - tests/unit/test_response_models.py: Import from semantic.py, add doc_type field Architecture: - Semantic search is now a cross-app feature, not tied to Notes - Uses dual-phase authorization: semantic:read scope + per-document verification - Supports future multi-app indexing (notes, calendar, deck, files, contacts) Test results: - All 69 unit tests passing - All 5 smoke tests passing 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>	2025-11-09 05:53:53 +01:00

Author

SHA1

Message

Date

Chris Coutinho

e575c8e57b

feat(vector): Support multiple embedding models with auto-generated collection names

This PR enables safe switching between embedding models and multi-server
deployments by implementing auto-generated Qdrant collection names based on
deployment ID and model name.

## Problem

Previously, all deployments used a single hardcoded collection name
"nextcloud_content", which caused two critical issues:

1. **Dimension mismatches when switching models**: Changing
   OLLAMA_EMBEDDING_MODEL (e.g., nomic-embed-text at 768D → all-minilm at
   384D) would cause runtime errors as vectors couldn't be inserted into a
   collection with incompatible dimensions.

2. **Collection collisions in multi-server setups**: Multiple MCP servers
   sharing a single Qdrant instance would overwrite each other's data,
   making horizontal scaling impossible.

## Solution

### Auto-Generated Collection Naming

Collections are now automatically named using the pattern:
\`{deployment-id}-{model-name}\`

**Deployment ID**: Uses \`OTEL_SERVICE_NAME\` if configured (and not default
value), otherwise falls back to \`hostname\` for simple Docker deployments.

**Model Name**: From \`OLLAMA_EMBEDDING_MODEL\` with path separators sanitized.

**Examples**:
- \`my-mcp-server-nomic-embed-text\` (with OTEL_SERVICE_NAME=my-mcp-server)
- \`mcp-container-all-minilm\` (simple Docker, hostname=mcp-container)

**Override**: Users can still set \`QDRANT_COLLECTION\` explicitly to bypass
auto-generation for backward compatibility.

### Dimension Validation

Added startup validation that checks collection dimensions match the
embedding service. If a mismatch is detected, the server fails fast with a
clear error message explaining:
- Expected vs actual dimensions
- Likely cause (model change)
- Solutions (delete collection, use different name, or revert model)

### Improved Sampling Error Handling

Enhanced MCP sampling rejection handling to treat user rejections as normal
behavior rather than errors:

- **User rejections** ("rejected", "denied") → INFO log, no traceback
- **Unsupported clients** → INFO log, no traceback
- **Other MCP errors** → WARNING log, no traceback
- **Unexpected errors** → ERROR log WITH traceback

This aligns with the MCP specification where clients SHOULD prompt users for
approval/denial of sampling requests.

## Changes

### Core Implementation

- **nextcloud_mcp_server/config.py**: Added \`get_collection_name()\` method
  with deployment ID detection and model name sanitization
- **nextcloud_mcp_server/vector/qdrant_client.py**: Dimension validation on
  collection open with helpful error messages
- **nextcloud_mcp_server/vector/{scanner,processor}.py**: Updated to use
  \`get_collection_name()\`
- **nextcloud_mcp_server/auth/userinfo_routes.py**: Vector sync status uses
  \`get_collection_name()\`
- **nextcloud_mcp_server/server/semantic.py**:
  - Updated semantic search tools to use \`get_collection_name()\`
  - Improved sampling rejection error handling (McpError vs Exception)

### Documentation

- **docs/semantic-search-architecture.md**: New comprehensive architecture
  document (557 lines) covering background sync, semantic search flow, RAG
  implementation, and deployment modes
- **docs/configuration.md**: Added detailed "Qdrant Collection Naming"
  section with examples and multi-server deployment guidance
- **docker-compose.yml**: Added comments explaining collection naming behavior
- **README.md**: Updated semantic search descriptions to clarify
  experimental status, Notes-only support, and infrastructure requirements

## Migration Guide

**For existing single-server deployments:**

Option 1 (Recommended): Use explicit collection name for continuity
\`\`\`bash
QDRANT_COLLECTION=nextcloud_content  # Keep existing collection
\`\`\`

Option 2: Allow auto-generation and re-embed
\`\`\`bash
# Remove QDRANT_COLLECTION override
# New collection will be created based on deployment ID + model
# Requires re-embedding all documents (may take time)
\`\`\`

**For new multi-server deployments:**

Set unique OTEL service names per server:
\`\`\`bash
# Server 1
OTEL_SERVICE_NAME=mcp-prod
OLLAMA_EMBEDDING_MODEL=nomic-embed-text
# → Collection: "mcp-prod-nomic-embed-text"

# Server 2
OTEL_SERVICE_NAME=mcp-staging
OLLAMA_EMBEDDING_MODEL=nomic-embed-text
# → Collection: "mcp-staging-nomic-embed-text"
\`\`\`

## Benefits

✅ **Safe model switching**: Each model gets its own collection, preventing
   dimension mismatch errors
✅ **Multi-server support**: Multiple MCP servers can share one Qdrant
   instance without conflicts
✅ **Clear ownership**: Collection names show which deployment and model owns
   the data
✅ **Better error messages**: Dimension validation provides actionable
   guidance
✅ **Backward compatible**: Existing deployments can continue using
   \`QDRANT_COLLECTION\` override

## Testing

Validated with:
- Single-server deployments (default hostname-based naming)
- Multi-server deployments (OTEL service name-based naming)
- Model switching scenarios (dimension validation)
- Collection override scenarios (backward compatibility)

Next steps: Testing various Ollama embedding models to investigate optimal
chunk sizes and performance characteristics.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

2025-11-10 01:18:30 +01:00

Chris Coutinho

72232f937a

refactor: migrate vector sync from asyncio.Queue to anyio memory object streams

Replace asyncio.Queue with anyio.create_memory_object_stream() throughout
the vector sync system for better library consistency and improved shutdown
semantics.

## Changes Made

**scanner.py**:
- Changed parameter type from `asyncio.Queue` to `MemoryObjectSendStream[DocumentTask]`
- Replaced all `await document_queue.put()` calls with `await send_stream.send()`
- Wrapped scanner loop in `async with send_stream:` context manager for automatic cleanup
- Updated log messages: "Queued" → "Sent"
- Removed `import asyncio` (no longer needed)

**processor.py**:
- Changed parameter type from `asyncio.Queue` to `MemoryObjectReceiveStream[DocumentTask]`
- Replaced `asyncio.wait_for(document_queue.get(), timeout=1.0)` with `anyio.fail_after(1.0)` + `await receive_stream.receive()`
- Removed all `document_queue.task_done()` calls (not needed with streams)
- Added `anyio.EndOfStream` exception handling for graceful shutdown when scanner closes
- Removed `import asyncio` (no longer needed)

**app.py**:
- Removed `import asyncio` from top-level imports
- Added `from anyio.streams.memory import MemoryObjectReceiveStream, MemoryObjectSendStream`
- Updated AppContext dataclass:
  - Replaced `document_queue: Optional[asyncio.Queue]` with:
    - `document_send_stream: Optional[MemoryObjectSendStream]`
    - `document_receive_stream: Optional[MemoryObjectReceiveStream]`
- Updated `app_lifespan_basic()`:
  - Replaced `asyncio.Queue(maxsize=...)` with `anyio.create_memory_object_stream(max_buffer_size=...)`
  - Pass `send_stream` to scanner_task
  - Pass `receive_stream.clone()` to each processor_task (enables multiple consumers)
  - Updated AppContext yield to include both streams
- Updated `starlette_lifespan()`:
  - Same changes as app_lifespan_basic for streamable-http transport
  - Removed `import asyncio as asyncio_module` (no longer needed)
  - Updated app.state storage to use send_stream and receive_stream

**semantic.py**:
- Updated `nc_get_vector_sync_status()` tool:
  - Access `document_receive_stream` instead of `document_queue` from lifespan context
  - Use `stream_stats.current_buffer_used` instead of `queue.qsize()` for pending count
  - More reliable metrics (qsize() was not guaranteed accurate)

## Benefits

1. **Library Consistency**: Pure anyio throughout codebase (was mixing asyncio.Queue with anyio.Event and anyio.create_task_group)
2. **Graceful Shutdown**: `async with send_stream:` automatically closes stream on exit, signaling EndOfStream to all processors
3. **Better Timeout Handling**: `anyio.fail_after()` is more idiomatic than `asyncio.wait_for()`
4. **Stream Cloning**: Easy to add multiple consumers via `receive_stream.clone()`
5. **Better Statistics**: `.statistics()` provides accurate buffer metrics (qsize() was unreliable)
6. **Type Safety**: Separate send/receive types prevent accidental misuse
7. **No task_done() tracking**: Streams handle completion automatically

## Testing

- ✅ All 69 unit tests passing
- ✅ All 5 smoke tests passing
- ✅ No regressions in functionality
- ✅ Graceful shutdown behavior improved

## References

- https://anyio.readthedocs.io/en/stable/why.html#queue-fix
- https://anyio.readthedocs.io/en/stable/streams.html#memory-object-streams

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

2025-11-09 06:43:44 +01:00

Chris Coutinho

4b026e9aa0

feat: implement ADR-009 - refactor semantic search to use generic semantic:read scope

This implements ADR-009, which documents the decision to use a generic
`semantic:read` OAuth scope instead of requiring all app-specific scopes
for semantic search functionality.

Changes:
- Created new `nextcloud_mcp_server/models/semantic.py` with semantic search models
  - SemanticSearchResult (with new doc_type field for multi-app support)
  - SemanticSearchResponse
  - SamplingSearchResponse
  - VectorSyncStatusResponse

- Created new `nextcloud_mcp_server/server/semantic.py` with semantic search tools
  - nc_semantic_search (renamed from nc_notes_semantic_search)
  - nc_semantic_search_answer (renamed from nc_notes_semantic_search_answer)
  - nc_get_vector_sync_status (renamed from nc_notes_get_vector_sync_status)
  - All tools now use @require_scopes("semantic:read") instead of "notes:read"

- Updated `nextcloud_mcp_server/server/notes.py`
  - Removed semantic search tools (moved to semantic.py)
  - Removed semantic search model imports
  - Removed unused MCP imports (ModelHint, ModelPreferences, etc.)

- Updated `nextcloud_mcp_server/models/notes.py`
  - Removed semantic search models (moved to semantic.py)

- Updated `nextcloud_mcp_server/app.py`
  - Import configure_semantic_tools
  - Register semantic tools when VECTOR_SYNC_ENABLED=true

- Updated `nextcloud_mcp_server/server/__init__.py`
  - Export configure_semantic_tools

- Updated tests
  - tests/integration/test_sampling.py: Use new tool names
  - tests/unit/test_response_models.py: Import from semantic.py, add doc_type field

Architecture:
- Semantic search is now a cross-app feature, not tied to Notes
- Uses dual-phase authorization: semantic:read scope + per-document verification
- Supports future multi-app indexing (notes, calendar, deck, files, contacts)

Test results:
- All 69 unit tests passing
- All 5 smoke tests passing

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

2025-11-09 05:53:53 +01:00

3 Commits