Compare commits

...

68 Commits

Author SHA1 Message Date
github-actions[bot] 747d297008 bump: version 0.32.0 → 0.32.1 2025-11-12 02:16:57 +00:00
Chris Coutinho ba8486b73b Merge pull request #289 from cbcoutinho/fix/dynamic-embedding-dimensions
fix: add dynamic dimension detection for Ollama embedding models
2025-11-12 03:16:29 +01:00
Chris Coutinho 6812e1aca7 fix: add dynamic dimension detection for Ollama embedding models
This fixes dimension mismatch errors when using embedding models with
non-standard dimensions (e.g., qwen3-embedding:4b produces 2560-dim
vectors instead of the hardcoded 768).

Changes:
- OllamaEmbeddingProvider: Detect dimensions dynamically by generating
  test embedding instead of hardcoding to 768
- qdrant_client: Call dimension detection before collection creation
- app.py: Initialize Qdrant collection before starting background tasks
  in streamable-http transport path
- tests: Fix integration tests to properly mock EmbeddingService wrapper

Fixes dimension mismatch error:
"could not broadcast input array from shape (2560,) into shape (768,)"

All integration tests passing (6/6).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-12 02:46:30 +01:00
github-actions[bot] 49a9dd43c6 bump: version 0.31.1 → 0.32.0 2025-11-11 23:54:43 +00:00
Chris Coutinho f6656fee06 Merge pull request #288 from cbcoutinho/feat/webhook-testing-validation
feat: webhook-based vector sync with management UI and validation
2025-11-12 00:54:20 +01:00
Chris Coutinho 7e93097137 feat(ollama): Pull model on startup if not available in ollama 2025-11-12 00:37:26 +01:00
Chris Coutinho 0eae33a918 ci: Fix logging warning and cli mock 2025-11-11 23:42:00 +01:00
Chris Coutinho 3430b2409d build: Set default logging to text 2025-11-11 23:19:37 +01:00
Chris Coutinho adde0e5623 fix: improve webapp tab UI with CSS Grid and viewport-filling container
Fixes layout issues on the webhooks admin tab:
- Add min-height to container to fill viewport consistently
- Use CSS Grid to overlay tab panes without jumpiness
- Add smooth htmx fade transitions for content swaps
- Adjust vector sync polling interval from 3s to 10s
- Add .playwright-mcp/ to gitignore for test screenshots

The CSS Grid approach allows tabs to overlay without absolute positioning,
preventing content cutoff while maintaining smooth transitions without
container resizing jumps.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-11 23:07:44 +01:00
Chris Coutinho 12c96af819 feat: add dynamic vector sync status updates with htmx polling
Implement real-time vector sync status updates in the /app UI without
requiring page refreshes. The status (indexed documents, pending
documents, sync state) now updates automatically every 3 seconds.

Changes:
- Add vector_sync_status_fragment() endpoint that returns HTML fragment
  with current vector sync status
- Modify user_info_html() to use htmx loading for vector sync section
  with hx-trigger="load" on initial render
- Status fragment includes hx-trigger="every 3s" for continuous polling
- Add /app/vector-sync/status route to browser_routes

The implementation uses htmx (already loaded on page) to poll the status
endpoint, providing near real-time updates with minimal overhead. The
endpoint queries Qdrant for indexed count and reads from memory streams
for pending count, returning only the status HTML fragment.

Pattern follows existing webhook management UI which also uses htmx
for dynamic loading.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-11 21:04:31 +01:00
Chris Coutinho d86a185e04 refactor: move webapp from /user/page to /app
Simplified the webapp routing structure by consolidating the admin UI
to a single clean endpoint.

Changes:
- Moved webapp from /user/page to /app (root of mount)
- Removed /user JSON endpoint (no longer needed)
- Updated mount point from /user to /app in app.py
- Updated all route path checks (3 locations)
- Updated OAuth redirects to point to /app
- Updated all HTMX endpoint references
- Updated documentation (ADR-007, CHANGELOG)
- Added redirect from /app to /app/ for trailing slash handling

New Route Structure:
- /app - Main webapp (HTML UI with tabs)
- /app/revoke - Revoke background access
- /app/webhooks - Webhook management UI
- /app/webhooks/enable/{preset_id} - Enable webhook preset
- /app/webhooks/disable/{preset_id} - Disable webhook preset

Breaking Change: Existing bookmarks to /user or /user/page will no longer work.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-11 20:53:43 +01:00
Chris Coutinho f4759e424d feat: add webhook management UI and BeforeNodeDeletedEvent support
Added comprehensive webhook management capabilities including:

Webhook Client & API:
- Added WebhooksClient for Nextcloud webhooks API integration
- Create, list, update, and delete webhooks programmatically
- Support for event filters in webhook registration

Webhook Presets:
- Added preset system for common webhook configurations
- notes_sync: BeforeNodeDeletedEvent for Notes file operations
- calendar_sync: Calendar events (create, update, delete)
- deck_sync: Deck card operations
- files_sync: File system changes
- forms_sync: Form submissions (conditional)
- Filter presets by installed apps

Admin UI:
- Added multi-pane app view with tabs (User Info, Vector Sync, Webhooks)
- Webhooks tab for admin users only
- Enable/disable preset webhooks via UI
- View currently registered webhooks
- Uses htmx for dynamic loading and Alpine.js for tab state
- Admin permission checking via OCS API

CLI Improvements:
- Refactored CLI to separate module (cli.py)
- Updated entry point in pyproject.toml

BeforeNodeDeletedEvent Fix:
- Updated ADR-010 to document NodeDeletedEvent issue
- BeforeNodeDeletedEvent includes node.id before deletion
- NodeDeletedEvent lacks node.id (file already deleted)
- Implemented per Nextcloud maintainer recommendation

Testing:
- Added comprehensive webhook client tests
- Added webhook preset filtering tests
- Added admin permission tests

Configuration:
- Updated docker-compose.yml Qdrant settings

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-11 20:35:08 +01:00
Chris Coutinho 1bced88c97 refactor: consolidate database storage for webhooks and OAuth tokens
Refactored the storage system to use a unified SQLite database for both
webhook tracking and OAuth token storage, available in both BasicAuth
and OAuth modes.

Changes:
- Renamed refresh_token_storage.py → storage.py
- Made TOKEN_ENCRYPTION_KEY optional (only required for OAuth token ops)
- Added registered_webhooks table with schema versioning
- Added webhook storage methods (store, get, delete, list, clear)
- Initialize storage in both BasicAuth and OAuth modes
- Updated webhook routes to persist registrations in database
- Database-first pattern for webhook status checks (performance)
- Updated all imports across codebase

Storage Behavior:
- Database created automatically at startup if needed
- Existing databases detected and reused
- Server fails fast if database initialization fails
- No migrations needed (OAuth feature is experimental)

Testing:
- Added 13 comprehensive unit tests for webhook storage
- All 118 unit tests pass
- All 5 smoke tests pass
- Verified fail-fast behavior on initialization errors

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-11 20:01:49 +01:00
Chris Coutinho b58e7238ae feat: validate Nextcloud webhook schemas and document findings
Manual testing of Nextcloud webhook_listeners app to validate webhook
payloads against ADR-010 expected schemas and document implementation
requirements for webhook-based vector synchronization.

## Changes

- Add test webhook endpoint at /webhooks/nextcloud in app.py
  - Captures and logs webhook payloads for analysis
  - Returns 200 OK immediately for webhook delivery confirmation

- Create webhook-testing-findings.md with comprehensive test results
  - Captured payloads for 5/6 webhook event types
  - Critical findings: missing node.id in deletions, type mismatches
  - Implementation recommendations with code examples

- Update ADR-010 with Appendix A: Manual Webhook Testing Results
  - Document actual vs expected webhook behavior
  - Update event mapping table with tested webhook status
  - Add 6 specific implementation recommendations
  - Include testing implications for future development

## Testing Results

 NodeCreatedEvent - fires correctly, includes node.id (integer)
 NodeWrittenEvent - fires correctly, includes node.id (integer)
 NodeDeletedEvent - fires but missing node.id field (path only)
 CalendarObjectCreatedEvent - fires correctly with full iCal
 CalendarObjectUpdatedEvent - fires correctly with full iCal
 CalendarObjectDeletedEvent - does not fire (potential NC bug)

## Key Findings

1. NodeDeletedEvent missing node.id field - requires path-based fallback
2. node.id returns integer not string - needs casting for consistency
3. Multiple webhooks fire per operation - needs deduplication logic
4. Calendar deletion webhooks don't fire - reported as issue #53497
5. Calendar webhooks include full iCal content - enables rich parsing

## GitHub Issues

- Created issue #56371: NodeDeletedEvent missing node.id field
- Commented on issue #53497: CalendarObjectDeletedEvent not firing

Closes #283

---

_This commit was generated with the help of AI, and reviewed by a Human_
2025-11-11 12:13:20 +01:00
Chris Coutinho 0005e0dce0 Merge pull request #286 from cbcoutinho/renovate/docker.io-library-mariadb-lts
chore(deps): update docker.io/library/mariadb:lts docker digest to 404ebf2
2025-11-11 09:17:23 +01:00
Chris Coutinho 636e5105c3 Merge pull request #287 from cbcoutinho/renovate/astral-sh-setup-uv-7.x
chore(deps): update astral-sh/setup-uv action to v7.1.3
2025-11-11 09:17:16 +01:00
renovate-bot-cbcoutinho[bot] ee7080afb3 chore(deps): update astral-sh/setup-uv action to v7.1.3 2025-11-10 23:10:10 +00:00
renovate-bot-cbcoutinho[bot] b52f482a51 chore(deps): update docker.io/library/mariadb:lts docker digest to 404ebf2 2025-11-10 23:10:04 +00:00
github-actions[bot] ce666934f2 bump: version 0.31.0 → 0.31.1 2025-11-10 22:21:48 +00:00
Chris Coutinho cdf69b3ea8 Merge pull request #285 from cbcoutinho/feat/otel-tracing-improvements
refactor: simplify OpenTelemetry tracing configuration
2025-11-10 23:21:18 +01:00
Chris Coutinho a6e5f3d8ff refactor: simplify OpenTelemetry tracing configuration
Simplifies the OpenTelemetry tracing setup by removing the redundant
OTEL_ENABLED flag and using the presence of OTEL_EXPORTER_OTLP_ENDPOINT
to determine if tracing should be enabled. This follows the standard
OpenTelemetry environment variable conventions more closely.

Changes:
- Remove OTEL_ENABLED/tracing_enabled flag in favor of checking if
  OTEL_EXPORTER_OTLP_ENDPOINT is set
- Add OTEL_EXPORTER_VERIFY_SSL configuration option for OTLP endpoints
  with self-signed certificates (defaults to false for development)
- Move HTTPXClientInstrumentor initialization to module level to ensure
  httpx calls are traced across all Nextcloud API requests
- Add tracing spans to vector sync operations (scan_user_documents)
- Fix authorization header logging to only warn about missing headers
  in OAuth mode (BasicAuth mode doesn't use Authorization headers)
- Update observability documentation to reflect simplified configuration
- Refactor Dockerfile to use --no-editable flag for uv sync

Breaking changes:
- OTEL_ENABLED environment variable is removed
- Tracing is now automatically enabled when OTEL_EXPORTER_OTLP_ENDPOINT
  is set

Migration guide:
- Remove OTEL_ENABLED=true from environment configuration
- Tracing will be enabled automatically if OTEL_EXPORTER_OTLP_ENDPOINT
  is configured

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-10 22:48:37 +01:00
github-actions[bot] f44bf3e8f2 bump: version 0.30.0 → 0.31.0 2025-11-10 07:02:49 +00:00
Chris Coutinho 37141003d8 Merge pull request #283 from cbcoutinho/feat/adr-010-webhook-vector-sync
docs: Add ADR-010 for webhook-based vector sync
2025-11-10 08:02:22 +01:00
Chris Coutinho c787abf2f3 fix: add retry logic for ETag conflicts in category change test
The test_attachments_category_change_handling test was failing in CI with
HTTP 412 Precondition Failed errors. This is caused by the background vector
scanner (runs every 10 seconds) modifying notes between when the test fetches
the ETag and when it attempts to update the category.

Solution: Added retry logic (up to 3 attempts) that refetches the latest ETag
and retries the update operation when encountering 412 errors. This handles
the race condition gracefully while still catching genuine errors.
2025-11-10 07:41:02 +01:00
Chris Coutinho b32324cb76 feat: skip tracing for health and metrics endpoints
Health check and metrics endpoints are frequently polled and don't
provide meaningful trace data. This change skips OpenTelemetry span
creation for:
- /health/* (liveness, readiness checks)
- /metrics (Prometheus metrics)

These endpoints still record Prometheus metrics (request count, latency,
in-flight requests) but no longer create trace spans, reducing tracing
noise and storage costs.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-10 07:24:27 +01:00
Chris Coutinho 640a7818f9 fix: optimize Notes API pagination with pruneBefore parameter
The Nextcloud Notes API intentionally returns all note IDs (with only 'id'
field) in the last chunk to enable deletion detection. Without using the
pruneBefore parameter, this causes duplicates - all notes appear with full
data in chunks, then again with minimal data in the last chunk.

This commit implements proper pruneBefore support:
- NotesClient.get_all_notes() now accepts prune_before timestamp parameter
- Scanner calculates max(indexed_at) from Qdrant to use as prune threshold
- Only notes modified after this timestamp are sent with full data
- Deduplication logic handles the API's deletion detection pattern
- Significantly reduces data transfer for incremental syncs

The behavior is documented in Notes API v1 spec - this is not an API bug,
but a feature we weren't utilizing correctly.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-10 07:19:26 +01:00
Chris Coutinho 8e5d0b5df1 Merge pull request #276 from cbcoutinho/renovate/pin-dependencies
chore(deps): pin qdrant/qdrant docker tag to 0fb8897
2025-11-10 06:48:01 +01:00
Chris Coutinho 851d21f56e Merge pull request #284 from cbcoutinho/renovate/lock-file-maintenance
chore(deps): lock file maintenance
2025-11-10 06:47:35 +01:00
renovate-bot-cbcoutinho[bot] fb1af697f7 chore(deps): lock file maintenance 2025-11-10 05:13:55 +00:00
renovate-bot-cbcoutinho[bot] bf4eed6007 chore(deps): pin qdrant/qdrant docker tag to 0fb8897 2025-11-10 05:12:36 +00:00
Chris Coutinho 3a41860d27 docs: Add ADR-010 for webhook-based vector sync
Add architecture decision record for integrating Nextcloud webhooks
into the vector database synchronization system.

Key features:
- Webhook endpoint at /webhooks/nextcloud receives push notifications
- Complements existing polling (ADR-007) without replacing it
- Optional authentication via WEBHOOK_SECRET
- Simple architecture: webhooks are just another DocumentTask producer
- Administrators can reduce polling frequency when webhooks are configured

Benefits:
- Reduced latency: seconds to minutes instead of up to 1 hour
- Lower API load: ~95% reduction when polling frequency is increased
- Better scalability: only process changed documents
- No changes required to scanner or processor components

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-10 05:28:36 +01:00
github-actions[bot] 126b5a7626 bump: version 0.29.2 → 0.30.0 2025-11-10 02:50:11 +00:00
Chris Coutinho 4d3ff1abe1 Merge pull request #282 from cbcoutinho/feat/multi-embedding-model-support
feat(vector): Support multiple embedding models with auto-generated collection names
2025-11-10 03:49:48 +01:00
Chris Coutinho d80e54ff97 feat(helm): Add document chunking configuration
Add support for configurable document chunking parameters to Helm chart
to match docker-compose and application capabilities.

Changes:
1. values.yaml:
   - Add documentChunking section with chunkSize (512) and chunkOverlap (50)
   - Include comprehensive comments explaining chunking strategies
   - Positioned between vectorSync and qdrant sections

2. templates/deployment.yaml:
   - Add DOCUMENT_CHUNK_SIZE and DOCUMENT_CHUNK_OVERLAP env vars
   - Always set (not conditional), used by vector sync processor
   - Environment variables follow same pattern as config.py defaults

3. README.md:
   - Add documentChunking parameter table in Vector Search section
   - Document chunking strategies (small/medium/large chunks)
   - Explain overlap recommendations (10-20% of chunk size)

Validation:
- helm lint: Passes
- helm template: Environment variables correctly generated
- Custom values: Work as expected (tested with chunkSize=1024)
- Always present: Not conditional on vectorSync.enabled

This maintains feature parity between Helm and docker-compose deployments,
allowing users to tune chunking for their embedding models and use cases.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-10 03:34:16 +01:00
Chris Coutinho 157e433d65 fix: Support in-memory Qdrant for CI testing
Changes to make tests work without external qdrant/ollama dependencies:

1. docker-compose.yml (mcp service):
   - Switch from QDRANT_URL (network mode) to QDRANT_LOCATION=":memory:"
   - Comment out QDRANT_URL and QDRANT_API_KEY (not needed for in-memory)
   - Keep OLLAMA_BASE_URL commented out (use SimpleEmbeddingProvider fallback)

2. nextcloud_mcp_server/vector/qdrant_client.py:
   - Fix collection creation bug in in-memory mode
   - Previously: All ValueError exceptions were re-raised
   - Now: Only dimension mismatch ValueError is re-raised
   - Allows "Collection not found" ValueError to trigger auto-creation

3. tests/integration/test_sampling.py:
   - Update test to handle all sampling unsupported cases
   - Check for multiple fallback search_method values
   - Skip test gracefully when sampling unavailable

This configuration enables:
- CI testing without external services (qdrant, ollama)
- In-memory vector database (ephemeral but sufficient for tests)
- SimpleEmbeddingProvider for embeddings (feature hashing, 384 dims)
- Automatic collection creation on first use

Test result: test_semantic_search_answer_successful_sampling now passes
(skipped with appropriate message when sampling unsupported)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-10 03:21:27 +01:00
Chris Coutinho 94d16092c0 ci: Add qdrant profile to docker compose up command 2025-11-10 03:09:50 +01:00
Chris Coutinho cb39b3fca4 feat(vector): Add configurable chunk size and overlap for document embedding
Enable users to tune document chunking parameters to match their embedding
model and content type by adding DOCUMENT_CHUNK_SIZE and DOCUMENT_CHUNK_OVERLAP
environment variables.

- **config.py**: Added `document_chunk_size` (default: 512) and
  `document_chunk_overlap` (default: 50) configuration fields with validation:
  - Ensures overlap < chunk_size
  - Warns if chunk_size < 100 words
  - Prevents negative overlap values

- **processor.py**: Updated DocumentChunker instantiation to use config
  settings instead of hardcoded values (line 174-177)

- **tests/unit/test_config.py**: Added TestChunkConfigValidation class with
  9 tests covering:
  - Default values
  - Valid configurations
  - Validation errors (overlap >= chunk_size, negative overlap)
  - Warning for small chunk sizes
  - Environment variable loading

- **docs/configuration.md**: Added comprehensive "Document Chunking
  Configuration" section with:
  - Chunk size selection guidance (256-384 vs 512 vs 768-1024 words)
  - Overlap recommendations (10-20% of chunk size)
  - Configuration examples for different use cases
  - Added env vars to reference table

- **docs/semantic-search-architecture.md**: Added "Document Chunking Strategy"
  section with:
  - Chunking process explanation
  - Example showing sliding window behavior
  - Search behavior with chunks
  - Tuning recommendations

- **env.sample**: Added complete "Semantic Search & Vector Sync Configuration"
  section with:
  - Vector sync settings
  - Qdrant configuration (3 modes)
  - Ollama embedding service
  - Document chunking configuration

- **docker-compose.yml**: Added commented examples for DOCUMENT_CHUNK_SIZE and
  DOCUMENT_CHUNK_OVERLAP with usage notes

\`\`\`bash
DOCUMENT_CHUNK_SIZE=512

DOCUMENT_CHUNK_OVERLAP=50
\`\`\`

1. \`overlap\` must be less than \`chunk_size\`
2. \`overlap\` cannot be negative
3. Warning issued if \`chunk_size\` < 100 words

**Precise matching** (small notes, specific queries):
\`\`\`bash
DOCUMENT_CHUNK_SIZE=256
DOCUMENT_CHUNK_OVERLAP=25
\`\`\`

**Balanced** (default, general purpose):
\`\`\`bash
DOCUMENT_CHUNK_SIZE=512
DOCUMENT_CHUNK_OVERLAP=50
\`\`\`

**Contextual** (long documents, broader topics):
\`\`\`bash
DOCUMENT_CHUNK_SIZE=1024
DOCUMENT_CHUNK_OVERLAP=100
\`\`\`

 **User control** - Tune chunking to match embedding model capabilities
 **Experimentation** - Test different chunk sizes for optimal results
 **Model alignment** - Match chunk size to embedding context window
 **Backward compatible** - Defaults maintain existing behavior
 **Well validated** - Comprehensive tests prevent misconfiguration

All 22 config validation tests pass (9 new tests for chunking):
- Default values work correctly
- Validation prevents invalid configurations
- Environment variables load properly
- Warning system works as expected

With configurable chunk sizes, users can now experiment with different Ollama
embedding models and tune chunk parameters for optimal semantic search quality.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-10 02:47:57 +01:00
Chris Coutinho f3050e9b45 chore: Remove /health and /metrics endpoints from logging 2025-11-10 02:07:45 +01:00
Chris Coutinho e575c8e57b feat(vector): Support multiple embedding models with auto-generated collection names
This PR enables safe switching between embedding models and multi-server
deployments by implementing auto-generated Qdrant collection names based on
deployment ID and model name.

## Problem

Previously, all deployments used a single hardcoded collection name
"nextcloud_content", which caused two critical issues:

1. **Dimension mismatches when switching models**: Changing
   OLLAMA_EMBEDDING_MODEL (e.g., nomic-embed-text at 768D → all-minilm at
   384D) would cause runtime errors as vectors couldn't be inserted into a
   collection with incompatible dimensions.

2. **Collection collisions in multi-server setups**: Multiple MCP servers
   sharing a single Qdrant instance would overwrite each other's data,
   making horizontal scaling impossible.

## Solution

### Auto-Generated Collection Naming

Collections are now automatically named using the pattern:
\`{deployment-id}-{model-name}\`

**Deployment ID**: Uses \`OTEL_SERVICE_NAME\` if configured (and not default
value), otherwise falls back to \`hostname\` for simple Docker deployments.

**Model Name**: From \`OLLAMA_EMBEDDING_MODEL\` with path separators sanitized.

**Examples**:
- \`my-mcp-server-nomic-embed-text\` (with OTEL_SERVICE_NAME=my-mcp-server)
- \`mcp-container-all-minilm\` (simple Docker, hostname=mcp-container)

**Override**: Users can still set \`QDRANT_COLLECTION\` explicitly to bypass
auto-generation for backward compatibility.

### Dimension Validation

Added startup validation that checks collection dimensions match the
embedding service. If a mismatch is detected, the server fails fast with a
clear error message explaining:
- Expected vs actual dimensions
- Likely cause (model change)
- Solutions (delete collection, use different name, or revert model)

### Improved Sampling Error Handling

Enhanced MCP sampling rejection handling to treat user rejections as normal
behavior rather than errors:

- **User rejections** ("rejected", "denied") → INFO log, no traceback
- **Unsupported clients** → INFO log, no traceback
- **Other MCP errors** → WARNING log, no traceback
- **Unexpected errors** → ERROR log WITH traceback

This aligns with the MCP specification where clients SHOULD prompt users for
approval/denial of sampling requests.

## Changes

### Core Implementation

- **nextcloud_mcp_server/config.py**: Added \`get_collection_name()\` method
  with deployment ID detection and model name sanitization
- **nextcloud_mcp_server/vector/qdrant_client.py**: Dimension validation on
  collection open with helpful error messages
- **nextcloud_mcp_server/vector/{scanner,processor}.py**: Updated to use
  \`get_collection_name()\`
- **nextcloud_mcp_server/auth/userinfo_routes.py**: Vector sync status uses
  \`get_collection_name()\`
- **nextcloud_mcp_server/server/semantic.py**:
  - Updated semantic search tools to use \`get_collection_name()\`
  - Improved sampling rejection error handling (McpError vs Exception)

### Documentation

- **docs/semantic-search-architecture.md**: New comprehensive architecture
  document (557 lines) covering background sync, semantic search flow, RAG
  implementation, and deployment modes
- **docs/configuration.md**: Added detailed "Qdrant Collection Naming"
  section with examples and multi-server deployment guidance
- **docker-compose.yml**: Added comments explaining collection naming behavior
- **README.md**: Updated semantic search descriptions to clarify
  experimental status, Notes-only support, and infrastructure requirements

## Migration Guide

**For existing single-server deployments:**

Option 1 (Recommended): Use explicit collection name for continuity
\`\`\`bash
QDRANT_COLLECTION=nextcloud_content  # Keep existing collection
\`\`\`

Option 2: Allow auto-generation and re-embed
\`\`\`bash
# Remove QDRANT_COLLECTION override
# New collection will be created based on deployment ID + model
# Requires re-embedding all documents (may take time)
\`\`\`

**For new multi-server deployments:**

Set unique OTEL service names per server:
\`\`\`bash
# Server 1
OTEL_SERVICE_NAME=mcp-prod
OLLAMA_EMBEDDING_MODEL=nomic-embed-text
# → Collection: "mcp-prod-nomic-embed-text"

# Server 2
OTEL_SERVICE_NAME=mcp-staging
OLLAMA_EMBEDDING_MODEL=nomic-embed-text
# → Collection: "mcp-staging-nomic-embed-text"
\`\`\`

## Benefits

 **Safe model switching**: Each model gets its own collection, preventing
   dimension mismatch errors
 **Multi-server support**: Multiple MCP servers can share one Qdrant
   instance without conflicts
 **Clear ownership**: Collection names show which deployment and model owns
   the data
 **Better error messages**: Dimension validation provides actionable
   guidance
 **Backward compatible**: Existing deployments can continue using
   \`QDRANT_COLLECTION\` override

## Testing

Validated with:
- Single-server deployments (default hostname-based naming)
- Multi-server deployments (OTEL service name-based naming)
- Model switching scenarios (dimension validation)
- Collection override scenarios (backward compatibility)

Next steps: Testing various Ollama embedding models to investigate optimal
chunk sizes and performance characteristics.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-10 01:18:30 +01:00
github-actions[bot] a0576aa9a2 bump: version 0.29.1 → 0.29.2 2025-11-09 18:28:34 +00:00
Chris Coutinho 4a6c60113b fix(helm): Set default strategy to Recreate 2025-11-09 19:27:55 +01:00
Chris Coutinho a0cb1ac9fe Merge pull request #281 from cbcoutinho/renovate/qdrant-1.x
chore(deps): update helm release qdrant to v1
2025-11-09 18:38:22 +01:00
renovate-bot-cbcoutinho[bot] de4f1032aa chore(deps): update helm release qdrant to v1 2025-11-09 17:08:13 +00:00
Chris Coutinho 178be5da6d Merge pull request #279 from cbcoutinho/renovate/ollama-1.x
chore(deps): update helm release ollama to v1.34.0
2025-11-09 18:04:08 +01:00
Chris Coutinho 61d8c851c9 Merge pull request #272 from cbcoutinho/renovate/softprops-action-gh-release-2.x
chore(deps): update softprops/action-gh-release action to v2.4.2
2025-11-09 17:02:19 +01:00
Chris Coutinho a8c63c8379 Merge pull request #278 from cbcoutinho/renovate/azure-setup-helm-4.x
chore(deps): update azure/setup-helm action to v4.3.1
2025-11-09 17:01:59 +01:00
renovate-bot-cbcoutinho[bot] 3147180ccd chore(deps): update helm release ollama to v1.34.0 2025-11-09 11:08:18 +00:00
renovate-bot-cbcoutinho[bot] 380578dd2e chore(deps): update softprops/action-gh-release action to v2.4.2 2025-11-09 11:07:57 +00:00
renovate-bot-cbcoutinho[bot] 10c5557aea chore(deps): update azure/setup-helm action to v4.3.1 2025-11-09 11:07:52 +00:00
github-actions[bot] 7772b1ac2e bump: version 0.29.0 → 0.29.1 2025-11-09 08:54:26 +00:00
Chris Coutinho 0513bec105 Merge pull request #275 from cbcoutinho/feature/observability-monitoring
fix(observability): isolate metrics endpoint to dedicated port
2025-11-09 09:54:00 +01:00
Chris Coutinho 4e89e92b65 fix(observability): isolate metrics endpoint to dedicated port
Security fix: Move Prometheus metrics endpoint from main HTTP port to
dedicated port 9090 to prevent external exposure of metrics data.

Changes:
- Use prometheus_client.start_http_server() for dedicated metrics server
- Remove /metrics route from main application routes
- Metrics now only accessible on port 9090 (configurable via METRICS_PORT)
- Main application port no longer serves /metrics endpoint

This follows security best practice of isolating monitoring endpoints
from application traffic.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-09 09:53:36 +01:00
github-actions[bot] af96378cb6 bump: version 0.28.0 → 0.29.0 2025-11-09 08:29:53 +00:00
Chris Coutinho c5da11aa4c Merge pull request #274 from cbcoutinho/feature/observability-monitoring
feature/observability monitoring
2025-11-09 09:29:25 +01:00
Chris Coutinho 5e4667a643 fix(readiness): Only check external Qdrant in network mode
The readiness probe incorrectly tried to connect to an external Qdrant service
even when using memory or persistent mode (embedded Qdrant). This caused pods
to never become ready in Kubernetes deployments using the default configuration.

Root cause:
- In memory/persistent modes, QDRANT_URL env var is NOT set
- Readiness check used default 'http://qdrant:6333' anyway
- Tried to connect to non-existent service
- Connection failed -> 503 -> pod stuck in not-ready state

Fix:
- Only check external Qdrant health if QDRANT_URL is explicitly set (network mode)
- For embedded modes (memory/persistent), report status as 'embedded' without blocking
- Background scanner tasks don't block readiness (already non-blocking via anyio.start_soon)

This allows pods to become ready immediately when using embedded Qdrant,
while still validating external Qdrant connectivity in network mode.

Fixes: Kubernetes pods failing readiness check with default Qdrant configuration

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-09 09:28:09 +01:00
Chris Coutinho 093ac5b5ba feat(helm): Add observability support with ServiceMonitor and Grafana dashboard
Add comprehensive observability configuration to Helm chart:

**Helm Values:**
- Add observability configuration section for metrics, tracing, and logging
- Add serviceMonitor configuration (disabled by default)
- Add prometheusRule configuration (disabled by default)

**Templates:**
- Update deployment to include observability environment variables
- Update deployment to expose metrics port (9090)
- Update service to expose metrics port
- Add ServiceMonitor template for Prometheus Operator
- Add PrometheusRule template with critical and warning alerts

**Dashboards:**
- Add comprehensive Grafana dashboard JSON with 6 panels:
  - Request Rate (by method and endpoint)
  - Error Rate (5xx errors percentage)
  - Request Latency (P50/P95 by endpoint)
  - Top MCP Tools (by invocation volume)
  - Nextcloud API Latency (by app)
  - Vector Sync Queue Size
- Add dashboard README with import instructions

**Alert Rules:**
- Critical: Server down, high error rate (>5%), high latency (>1s), dependency down
- Warning: Token validation errors (>1%), vector sync queue high (>100), Qdrant slow (>500ms)

All features are opt-in via values.yaml configuration.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-09 09:10:11 +01:00
github-actions[bot] ae81f0334e bump: version 0.27.3 → 0.28.0 2025-11-09 08:04:06 +00:00
Chris Coutinho 23f3a231a5 Merge pull request #273 from cbcoutinho/feature/observability-monitoring
Feature/observability monitoring
2025-11-09 09:03:40 +01:00
Chris Coutinho 7be40a33e1 fix(vector): Handle missing 'modified' field in notes gracefully
The vector scanner crashed when encountering notes without a 'modified' field,
causing KeyError and preventing initial sync from completing.

Changes:
- Use dict.get() with fallback value (0) instead of direct key access
- Log warnings for notes missing 'modified' field
- Apply fix to both initial sync and incremental sync code paths

This ensures the scanner continues processing all notes even if some have
missing metadata fields, preventing scanner crashes that could affect
deployment readiness.

Fixes: Notes without 'modified' field causing scanner crash and readiness check failure

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-09 09:03:05 +01:00
Chris Coutinho 578de4d7d6 feat(observability): Add comprehensive monitoring with Prometheus and OpenTelemetry
- Add Prometheus metrics for HTTP, MCP tools, Nextcloud API, OAuth, vector sync, and DB operations
- Add OpenTelemetry distributed tracing with OTLP export
- Add structured JSON logging with trace context correlation
- Add ObservabilityMiddleware for automatic HTTP instrumentation
- Add app_name attribute to all client classes for per-app metrics
- Add configuration for metrics, tracing, and logging via environment variables
- Add documentation in docs/observability.md
- Fix graceful degradation when tracing is disabled (default state)
- Fix uvicorn logging configuration to use observability formatters

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-09 08:54:04 +01:00
github-actions[bot] 8f0f989c6d bump: version 0.27.2 → 0.27.3 2025-11-09 06:52:31 +00:00
Chris Coutinho f8a2935c22 fix(ci): Use helm dependency build instead of update to use Chart.lock 2025-11-09 07:52:00 +01:00
github-actions[bot] 137dc80075 bump: version 0.27.1 → 0.27.2 2025-11-09 06:45:44 +00:00
Chris Coutinho 725ac65e6a fix(helm): update Qdrant dependency condition to match new mode structure
The Qdrant subchart was being included by default even in memory/persistent
modes. Changed the dependency condition from `qdrant.enabled` to
`qdrant.networkMode.deploySubchart` to align with the three-mode structure.

Now the Qdrant subchart is ONLY deployed when:
- qdrant.mode: "network"
- qdrant.networkMode.deploySubchart: true

Verified all three modes:
- Memory mode (:memory:): No subchart, QDRANT_LOCATION=:memory:
- Persistent mode (path): No subchart, QDRANT_LOCATION=/app/data/qdrant, PVC created
- Network mode (subchart): Qdrant subchart deployed, QDRANT_URL=http://...:6333
- Network mode (external): No subchart, QDRANT_URL=<external-url>

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-09 07:45:06 +01:00
github-actions[bot] f51edff25d bump: version 0.27.0 → 0.27.1 2025-11-09 06:22:00 +00:00
Chris Coutinho 50ba6ccc88 fix(ci): add Helm repository setup to chart release workflow
The chart-releaser was failing because it couldn't resolve the
dependencies (Qdrant and Ollama subcharts) when packaging.

Changes:
- Add azure/setup-helm action to install Helm v3.16.0
- Add step to add Qdrant and Ollama Helm repositories
- Run helm dependency update before chart-releaser runs

This fixes the error:
"Error: no repository definition for https://qdrant.github.io/qdrant-helm, https://otwld.github.io/ollama-helm"

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-09 07:21:17 +01:00
github-actions[bot] 538bbc375e bump: version 0.26.1 → 0.27.0 2025-11-09 06:15:27 +00:00
Chris Coutinho d4c686eba7 Merge pull request #271 from cbcoutinho/docs/adr-007-background-vector-sync
feat: implement ADR-007 background vector sync and semantic search
2025-11-09 07:15:00 +01:00
79 changed files with 9627 additions and 1269 deletions
+1 -1
View File
@@ -25,7 +25,7 @@ jobs:
github_token: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
changelog_increment_filename: body.md
- name: Release
uses: softprops/action-gh-release@6da8fa9354ddfdc4aeace5fc48d7f679b5214090 # v2.4.1
uses: softprops/action-gh-release@5be0e66d93ac7ed76da52eca8bb058f665c3a5fe # v2.4.2
with:
body_path: "body.md"
tag_name: v${{ env.REVISION }}
+12
View File
@@ -24,6 +24,18 @@ jobs:
git config user.name "$GITHUB_ACTOR"
git config user.email "$GITHUB_ACTOR@users.noreply.github.com"
- name: Install Helm
uses: azure/setup-helm@1a275c3b69536ee54be43f2070a358922e12c8d4 # v4.3.1
with:
version: v3.16.0
- name: Add Helm repositories and update dependencies
run: |
helm repo add qdrant https://qdrant.github.io/qdrant-helm
helm repo add ollama https://otwld.github.io/ollama-helm
helm repo update
helm dependency build charts/nextcloud-mcp-server
- name: Run chart-releaser
uses: helm/chart-releaser-action@cae68fefc6b5f367a0275617c9f83181ba54714f # v1.7.0
env:
+1 -1
View File
@@ -20,7 +20,7 @@ jobs:
- name: Checkout
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5
- name: Install uv
uses: astral-sh/setup-uv@85856786d1ce8acfbcc2f13a5f3fbd6b938f9f41 # v7.1.2
uses: astral-sh/setup-uv@5a7eac68fb9809dea845d802897dc5c723910fa3 # v7.1.3
- name: Install Python 3.11
run: uv python install 3.11
- name: Build
+3 -2
View File
@@ -11,7 +11,7 @@ jobs:
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- name: Install the latest version of uv
uses: astral-sh/setup-uv@85856786d1ce8acfbcc2f13a5f3fbd6b938f9f41 # v7.1.2
uses: astral-sh/setup-uv@5a7eac68fb9809dea845d802897dc5c723910fa3 # v7.1.3
- name: Check format
run: |
uv run --frozen ruff format --diff
@@ -52,10 +52,11 @@ jobs:
uses: hoverkraft-tech/compose-action@3846bcd61da338e9eaaf83e7ed0234a12b099b72 # v2.4.1
with:
compose-file: "./docker-compose.yml"
#compose-flags: "--profile qdrant"
up-flags: "--build"
- name: Install the latest version of uv
uses: astral-sh/setup-uv@85856786d1ce8acfbcc2f13a5f3fbd6b938f9f41 # v7.1.2
uses: astral-sh/setup-uv@5a7eac68fb9809dea845d802897dc5c723910fa3 # v7.1.3
- name: Install Playwright dependencies
run: |
+3
View File
@@ -5,5 +5,8 @@ __pycache__/
.env.local
.env.*.local
docker-compose.override.yml
# Generated by pytest used to login users
.nextcloud_oauth_*.json
.playwright-mcp/
+127
View File
@@ -1,3 +1,130 @@
## v0.32.1 (2025-11-12)
### Fix
- add dynamic dimension detection for Ollama embedding models
## v0.32.0 (2025-11-11)
### Feat
- **ollama**: Pull model on startup if not available in ollama
- add dynamic vector sync status updates with htmx polling
- add webhook management UI and BeforeNodeDeletedEvent support
- validate Nextcloud webhook schemas and document findings
### Fix
- improve webapp tab UI with CSS Grid and viewport-filling container
### Refactor
- move webapp from /user/page to /app
- consolidate database storage for webhooks and OAuth tokens
## v0.31.1 (2025-11-10)
### Refactor
- simplify OpenTelemetry tracing configuration
## v0.31.0 (2025-11-10)
### Feat
- skip tracing for health and metrics endpoints
### Fix
- add retry logic for ETag conflicts in category change test
- optimize Notes API pagination with pruneBefore parameter
## v0.30.0 (2025-11-10)
### Feat
- **helm**: Add document chunking configuration
- **vector**: Add configurable chunk size and overlap for document embedding
- **vector**: Support multiple embedding models with auto-generated collection names
### Fix
- Support in-memory Qdrant for CI testing
## v0.29.2 (2025-11-09)
### Fix
- **helm**: Set default strategy to Recreate
## v0.29.1 (2025-11-09)
### Fix
- **observability**: isolate metrics endpoint to dedicated port
## v0.29.0 (2025-11-09)
### Feat
- **helm**: Add observability support with ServiceMonitor and Grafana dashboard
### Fix
- **readiness**: Only check external Qdrant in network mode
## v0.28.0 (2025-11-09)
### Feat
- **observability**: Add comprehensive monitoring with Prometheus and OpenTelemetry
### Fix
- **vector**: Handle missing 'modified' field in notes gracefully
## v0.27.3 (2025-11-09)
### Fix
- **ci**: Use helm dependency build instead of update to use Chart.lock
## v0.27.2 (2025-11-09)
### Fix
- **helm**: update Qdrant dependency condition to match new mode structure
## v0.27.1 (2025-11-09)
### Fix
- **ci**: add Helm repository setup to chart release workflow
## v0.27.0 (2025-11-09)
### Feat
- **helm**: add Qdrant local mode support with three deployment options [skip ci]
- add Qdrant local mode support with in-memory and persistent storage
- implement ADR-009 - refactor semantic search to use generic semantic:read scope
- implement MCP sampling for semantic search RAG (ADR-008)
- add optional vector database and semantic search to helm chart
- add vector sync processing status to /app endpoint
- implement semantic search tool and fix vector sync issues (ADR-007 Phase 3)
- implement vector sync scanner and processor (ADR-007 Phase 2)
### Fix
- implement deletion grace period and vector sync status tool
- remove unnecessary urllib3<2.0 constraint
- integrate vector sync tasks with Starlette lifespan for streamable-http
### Refactor
- migrate vector sync from asyncio.Queue to anyio memory object streams
- update to Qdrant query_points API and fix Playwright Keycloak login
## v0.26.1 (2025-11-08)
### Fix
+4
View File
@@ -391,3 +391,7 @@ docker compose exec app php occ user_oidc:provider keycloak
- `docs/configuration.md` - Configuration options
- `docs/authentication.md` - Authentication modes
- `docs/running.md` - Running the server
**For additional information regarding MCP during development, see**:
- `../../Software/modelcontextprotocol/` - MCP spec
- `../../Software/python-sdk/` - Python MCP SDK
+1 -1
View File
@@ -9,7 +9,7 @@ WORKDIR /app
COPY . .
RUN uv sync --locked --no-dev
RUN uv sync --locked --no-dev --no-editable
ENV PYTHONUNBUFFERED=1
ENV VIRTUAL_ENV=/app/.venv
+114 -280
View File
@@ -2,286 +2,134 @@
[![Docker Image](https://img.shields.io/badge/docker-ghcr.io/cbcoutinho/nextcloud--mcp--server-blue)](https://github.com/cbcoutinho/nextcloud-mcp-server/pkgs/container/nextcloud-mcp-server)
**Enable AI assistants to interact with your Nextcloud instance.**
**A production-ready MCP server that connects AI assistants to your Nextcloud instance.**
The Nextcloud MCP (Model Context Protocol) server allows Large Language Models like Claude, GPT, and Gemini to interact with your Nextcloud data through a secure API. Create notes, manage calendars, organize contacts, work with files, and more - all through natural language.
Enable Large Language Models like Claude, GPT, and Gemini to interact with your Nextcloud data through a secure API. Create notes, manage calendars, organize contacts, work with files, and more - all through natural language conversations.
This is a **dedicated standalone MCP server** designed for external MCP clients like Claude Code and IDEs. It runs independently of Nextcloud (Docker, VM, Kubernetes, or local) and provides deep CRUD operations across Nextcloud apps.
> [!NOTE]
> **Nextcloud has two ways to enable AI access:** Nextcloud provides [Context Agent](https://github.com/nextcloud/context_agent), an AI agent backend that powers the [Assistant](https://github.com/nextcloud/assistant) app and allows AI to interact with Nextcloud apps like Calendar, Talk, and Contacts. Context Agent runs as an ExApp inside Nextcloud and also _[exposes an MCP server](https://docs.nextcloud.com/server/stable/admin_manual/ai/app_context_agent.html#using-nextcloud-mcp-server)_ for external MCP clients.
>
> This project (Nextcloud MCP Server) is a **dedicated standalone MCP server** designed specifically for external MCP clients like Claude Code and IDEs, with deep CRUD operations and OAuth support. It does not require any additional AI-features to be enabled in Nextcloud beyond the apps that you intend to interact with.
### High-level Comparison: Nextcloud MCP Server vs. Nextcloud AI Stack
| Aspect | **Nextcloud MCP Server**<br/>(This Project) | **Nextcloud AI Stack**<br/>(Assistant + Context Agent) |
|--------|---------------------------------------------|--------------------------------------------------------|
| **Purpose** | External MCP client access to Nextcloud | AI assistance within Nextcloud UI |
| **Deployment** | Standalone (Docker, VM, K8s) | Inside Nextcloud (ExApp via AppAPI) |
| **Primary Users** | Claude Code, IDEs, external developers | Nextcloud end users via Assistant app |
| **Authentication** | OAuth2/OIDC or Basic Auth | Session-based (integrated) |
| **Notes Support** | ✅ Full CRUD + keyword search (7 tools) | ❌ Not implemented |
| **Semantic Search** | ✅ Multi-app vector search (2+ tools) | ❌ Not implemented |
| **Calendar** | ✅ Full CalDAV + tasks (20+ tools) | ✅ Events, free/busy, tasks (4 tools) |
| **Contacts** | ✅ Full CardDAV (8 tools) | ✅ Find person, current user (2 tools) |
| **Files (WebDAV)** | ✅ Full filesystem access (12 tools) | ✅ Read, folder tree, sharing (3 tools) |
| **Document Processing** | ✅ OCR with progress (PDF, DOCX, images) | ❌ Not implemented |
| **Deck** | ✅ Full project management (15 tools) | ✅ Basic board/card ops (2 tools) |
| **Tables** | ✅ Row operations (5 tools) | ❌ Not implemented |
| **Cookbook** | ✅ Full recipe management (13 tools) | ❌ Not implemented |
| **Talk** | ❌ Not implemented | ✅ Messages, conversations (4 tools) |
| **Mail** | ❌ Not implemented | ✅ Send email (2 tools) |
| **AI Features** | ❌ Not implemented | ✅ Image gen, transcription, doc gen (4 tools) |
| **Web/Maps** | ❌ Not implemented | ✅ Search, weather, transit (5 tools) |
| **MCP Resources** | ✅ Structured data URIs | ❌ Not supported |
| **External MCP** | ❌ Pure server | ✅ Consumes external MCP servers |
| **Safety Model** | Client-controlled | Built-in safe/dangerous distinction |
| **Best For** | • Deep CRUD operations<br/>• External integrations<br/>• OAuth security<br/>• IDE/editor integration | • AI-driven actions in Nextcloud UI<br/>• Multi-service orchestration<br/>• User task automation<br/>• MCP aggregation hub |
See our [detailed comparison](docs/comparison-context-agent.md) for architecture diagrams, workflow examples, and guidance on when to use each approach.
Want to see another Nextcloud app supported? [Open an issue](https://github.com/cbcoutinho/nextcloud-mcp-server/issues) or contribute a pull request!
### Authentication
| Mode | Security | Best For |
|------|----------|----------|
| **OAuth2/OIDC** ⚠️ **Experimental** | 🔒 High | Testing, evaluation (requires patch for app-specific APIs) |
| **Basic Auth** ✅ | Lower | Development, testing, production |
> [!IMPORTANT]
> **OAuth is experimental** and requires a manual patch to the `user_oidc` app for full functionality:
> - **Required patch**: `user_oidc` app needs modifications for Bearer token support ([issue #1221](https://github.com/nextcloud/user_oidc/issues/1221))
> - **Impact**: Without the patch, most app-specific APIs (Notes, Calendar, Contacts, Deck, etc.) will fail with 401 errors
> - **What works without patches**: OAuth flow, PKCE support (with `oidc` v1.10.0+), OCS APIs
> - **Production use**: Wait for upstream patch to be merged into official releases
>
> See [OAuth Upstream Status](docs/oauth-upstream-status.md) for detailed information on required patches and workarounds.
OAuth2/OIDC provides secure, per-user authentication with access tokens. See [Authentication Guide](docs/authentication.md) for details.
> **Looking for AI features inside Nextcloud?** Nextcloud also provides [Context Agent](https://github.com/nextcloud/context_agent), which powers the Assistant app and runs as an ExApp inside Nextcloud. See [docs/comparison-context-agent.md](docs/comparison-context-agent.md) for a detailed comparison of use cases.
## Quick Start
### 1. Install
Get up and running in 60 seconds using Docker:
```bash
# Clone the repository
git clone https://github.com/cbcoutinho/nextcloud-mcp-server.git
cd nextcloud-mcp-server
# Install with uv (recommended)
uv sync
# Or using Docker
docker pull ghcr.io/cbcoutinho/nextcloud-mcp-server:latest
# Or deploy to Kubernetes with Helm
helm repo add nextcloud-mcp https://cbcoutinho.github.io/nextcloud-mcp-server
helm repo update
helm install nextcloud-mcp nextcloud-mcp/nextcloud-mcp-server \
--set nextcloud.host=https://cloud.example.com \
--set auth.basic.username=myuser \
--set auth.basic.password=mypassword
```
See [Installation Guide](docs/installation.md) for detailed instructions, or [Helm Chart README](charts/nextcloud-mcp-server/README.md) for Kubernetes deployment.
### 2. Configure
Create a `.env` file:
```bash
# Copy the sample
cp env.sample .env
```
**For Basic Auth (recommended for most users):**
```dotenv
# 1. Create a minimal configuration
cat > .env << EOF
NEXTCLOUD_HOST=https://your.nextcloud.instance.com
NEXTCLOUD_USERNAME=your_username
NEXTCLOUD_PASSWORD=your_app_password
```
EOF
**For OAuth (experimental - requires patches):**
```dotenv
NEXTCLOUD_HOST=https://your.nextcloud.instance.com
```
See [Configuration Guide](docs/configuration.md) for all options.
### 3. Set Up Authentication
**Basic Auth Setup (recommended):**
1. Create an app password in Nextcloud (Settings → Security → Devices & sessions)
2. Add credentials to `.env` file
3. Start the server
**OAuth Setup (experimental):**
1. Install Nextcloud OIDC apps (`oidc` v1.10.0+ + `user_oidc`)
2. **Apply required patch** to `user_oidc` app for Bearer token support (see [OAuth Upstream Status](docs/oauth-upstream-status.md))
3. Enable dynamic client registration or create an OIDC client with id & secret
4. Configure Bearer token validation in `user_oidc`
5. Start the server
See [OAuth Quick Start](docs/quickstart-oauth.md) for 5-minute setup or [OAuth Setup Guide](docs/oauth-setup.md) for detailed instructions.
### 4. Run the Server
```bash
# Load environment variables
export $(grep -v '^#' .env | xargs)
# Start with Basic Auth (default)
uv run nextcloud-mcp-server
# Or start with OAuth (experimental - requires patches)
uv run nextcloud-mcp-server --oauth
# Or with Docker
# 2. Start the server
docker run -p 127.0.0.1:8000:8000 --env-file .env --rm \
ghcr.io/cbcoutinho/nextcloud-mcp-server:latest
# 3. Test the connection
curl http://127.0.0.1:8000/health/ready
```
The server starts on `http://127.0.0.1:8000` by default.
**Next Steps:**
- Create an app password in Nextcloud: Settings → Security → Devices & sessions
- Connect your MCP client (Claude Desktop, IDEs, `mcp dev`, etc.)
- See [docs/installation.md](docs/installation.md) for other deployment options (local, Kubernetes)
See [Running the Server](docs/running.md) for more options.
## Key Features
### 5. Connect an MCP Client
- **90+ MCP Tools** - Comprehensive API coverage across 8 Nextcloud apps
- **MCP Resources** - Structured data URIs for browsing Nextcloud data
- **Semantic Search (Experimental)** - Optional vector-powered search for Notes (requires Qdrant + Ollama)
- **Document Processing** - OCR and text extraction from PDFs, DOCX, images with progress notifications
- **Flexible Deployment** - Docker, Kubernetes (Helm), VM, or local installation
- **Production-Ready Auth** - Basic Auth with app passwords (recommended) or OAuth2/OIDC (experimental)
- **Multiple Transports** - SSE, HTTP, and streamable-http support
Test with MCP Inspector:
## Supported Apps
```bash
uv run mcp dev
```
| App | Tools | Capabilities |
|-----|-------|--------------|
| **Notes** | 7 | Full CRUD, keyword search, semantic search |
| **Calendar** | 20+ | Events, todos (tasks), recurring events, attendees, availability |
| **Contacts** | 8 | Full CardDAV support, address books |
| **Files (WebDAV)** | 12 | Filesystem access, OCR/document processing |
| **Deck** | 15 | Boards, stacks, cards, labels, assignments |
| **Cookbook** | 13 | Recipe management, URL import (schema.org) |
| **Tables** | 5 | Row operations on Nextcloud Tables |
| **Sharing** | 10+ | Create and manage shares |
| **Semantic Search** | 2+ | Vector search for Notes (experimental, opt-in, requires infrastructure) |
Or connect from:
- Claude Desktop
- Any MCP-compatible client
Want to see another Nextcloud app supported? [Open an issue](https://github.com/cbcoutinho/nextcloud-mcp-server/issues) or contribute a pull request!
## Authentication
> [!IMPORTANT]
> **OAuth2/OIDC is experimental** and requires a manual patch to the `user_oidc` app:
> - **Required patch**: Bearer token support ([issue #1221](https://github.com/nextcloud/user_oidc/issues/1221))
> - **Impact**: Without the patch, most app-specific APIs fail with 401 errors
> - **Recommendation**: Use Basic Auth for production until upstream patches are merged
>
> See [docs/oauth-upstream-status.md](docs/oauth-upstream-status.md) for patch status and workarounds.
**Recommended:** Basic Auth with app-specific passwords provides secure, production-ready authentication. See [docs/authentication.md](docs/authentication.md) for setup details and OAuth configuration.
### Authentication Modes
The server supports two authentication modes:
**Single-User Mode (BasicAuth):**
- One set of credentials shared by all MCP clients
- Simple setup: username + app password in environment variables
- All clients access Nextcloud as the same user
- Best for: Personal use, development, single-user deployments
**Multi-User Mode (OAuth):**
- Each MCP client authenticates separately with their own Nextcloud account
- Per-user scopes and permissions (clients only see tools they're authorized for)
- More secure: tokens expire, credentials never shared with server
- Best for: Teams, multi-user deployments, production environments with multiple users
See [docs/authentication.md](docs/authentication.md) for detailed setup instructions.
## Semantic Search
The server provides an experimental RAG pipeline to enable _Semantic Search_ that enables MCP clients to find information in Nextcloud based on **meaning** rather than just keywords. Instead of matching "machine learning" only when those exact words appear, it understands that "neural networks," "AI models," and "deep learning" are semantically related concepts.
**Example:**
- **Keyword search**: Query "car" only finds notes containing "car"
- **Semantic search**: Query "car" also finds notes about "automobile," "vehicle," "sedan," "transportation"
This enables natural language queries and helps discover related content across your Nextcloud notes.
> [!NOTE]
> **Semantic Search is experimental and opt-in:**
> - Disabled by default (`VECTOR_SYNC_ENABLED=false`)
> - Currently supports Notes app only (multi-app support planned)
> - Requires additional infrastructure: vector database + embedding service
> - Answer generation (`nc_semantic_search_answer`) requires MCP client sampling support
>
> See [docs/semantic-search-architecture.md](docs/semantic-search-architecture.md) for architecture details and [docs/configuration.md](docs/configuration.md) for setup instructions.
## Documentation
### Getting Started
- **[Installation](docs/installation.md)** - Install the server
- **[Configuration](docs/configuration.md)** - Environment variables and settings
- **[Authentication](docs/authentication.md)** - OAuth vs BasicAuth
- **[Running the Server](docs/running.md)** - Start and manage the server
- **[Installation](docs/installation.md)** - Docker, Kubernetes, local, or VM deployment
- **[Configuration](docs/configuration.md)** - Environment variables and advanced options
- **[Authentication](docs/authentication.md)** - Basic Auth vs OAuth2/OIDC setup
- **[Running the Server](docs/running.md)** - Start, manage, and troubleshoot
### Architecture
- **[Comparison with Context Agent](docs/comparison-context-agent.md)** - How this MCP server differs from Nextcloud's Context Agent
### Features
- **[App Documentation](docs/)** - Notes, Calendar, Contacts, WebDAV, Deck, Cookbook, Tables
- **[Document Processing](docs/configuration.md#document-processing)** - OCR and text extraction setup
- **[Semantic Search Architecture](docs/semantic-search-architecture.md)** - Experimental vector search (Notes only, opt-in)
### OAuth Documentation (Experimental)
- **[OAuth Quick Start](docs/quickstart-oauth.md)** - 5-minute setup guide
- **[OAuth Setup Guide](docs/oauth-setup.md)** - Detailed setup instructions
- **[OAuth Architecture](docs/oauth-architecture.md)** - How OAuth works
- **[OAuth Troubleshooting](docs/oauth-troubleshooting.md)** - OAuth-specific issues
- **[Upstream Status](docs/oauth-upstream-status.md)** - **Required patches and PRs** ⚠️
### Reference
### Advanced Topics
- **[OAuth Architecture](docs/oauth-architecture.md)** - How OAuth works (experimental)
- **[OAuth Quick Start](docs/quickstart-oauth.md)** - 5-minute OAuth setup
- **[OAuth Setup Guide](docs/oauth-setup.md)** - Detailed OAuth configuration
- **[Troubleshooting](docs/troubleshooting.md)** - Common issues and solutions
### App-Specific Documentation
- [Notes API](docs/notes.md)
- [Calendar (CalDAV)](docs/calendar.md)
- [Contacts (CardDAV)](docs/contacts.md)
- [Cookbook](docs/cookbook.md)
- [Deck](docs/deck.md)
- [Tables](docs/table.md)
- [WebDAV](docs/webdav.md)
## MCP Tools & Resources
The server exposes Nextcloud functionality through MCP tools (for actions) and resources (for data browsing).
### Tools
The server provides 90+ tools across 8 Nextcloud apps. When using OAuth, tools are dynamically filtered based on your granted scopes.
For a complete list of all supported OAuth scopes and their descriptions, see [OAuth Scopes Documentation](docs/oauth-architecture.md#oauth-scopes).
#### Available Tool Categories
| App | Tools | Read Scope | Write Scope | Operations |
|-----|-------|-----------|-------------|------------|
| **Notes** | 7 | `notes:read` | `notes:write` | Create, read, update, delete, search notes (keyword search) |
| **Calendar** | 20+ | `calendar:read` `todo:read` | `calendar:write` `todo:write` | Events, todos (tasks), calendars, recurring events, attendees |
| **Contacts** | 8 | `contacts:read` | `contacts:write` | Create, read, update, delete contacts and address books |
| **Files (WebDAV)** | 12 | `files:read` | `files:write` | List, read, upload, delete, move files; **OCR/document processing** |
| **Deck** | 15 | `deck:read` | `deck:write` | Boards, stacks, cards, labels, assignments |
| **Cookbook** | 13 | `cookbook:read` | `cookbook:write` | Recipes, import from URLs, search, categories |
| **Tables** | 5 | `tables:read` | `tables:write` | Row operations on Nextcloud Tables |
| **Sharing** | 10+ | `sharing:read` | `sharing:write` | Create, manage, delete shares |
| **Semantic Search** | 2+ | `semantic:read` | `semantic:write` | Vector-powered semantic search across **all apps** (notes, calendar, deck, files, contacts), background indexing |
#### Document Processing (Optional)
The WebDAV file reading tool (`nc_webdav_read_file`) supports **automatic text extraction** from documents and images:
**Supported Formats:**
- **Documents**: PDF, DOCX, PPTX, XLSX, RTF, ODT, EPUB
- **Images**: PNG, JPEG, TIFF, BMP (with OCR)
- **Email**: EML, MSG files
**Features:**
- **Progress Notifications**: Long-running OCR operations (up to 120s) send progress updates every 10 seconds to prevent client timeouts
- **Pluggable Architecture**: Multiple processor backends (Unstructured.io, Tesseract, custom HTTP APIs)
- **Automatic Detection**: Files are processed based on MIME type
- **Graceful Fallback**: Returns base64-encoded content if processing fails
**Configuration:**
```dotenv
# Enable document processing (optional)
ENABLE_DOCUMENT_PROCESSING=true
# Unstructured.io processor (cloud/API-based, supports many formats)
ENABLE_UNSTRUCTURED=true
UNSTRUCTURED_API_URL=http://localhost:8002
UNSTRUCTURED_STRATEGY=auto # auto, fast, or hi_res
UNSTRUCTURED_LANGUAGES=eng,deu
PROGRESS_INTERVAL=10 # Progress update interval in seconds
# Tesseract processor (local OCR, images only)
ENABLE_TESSERACT=false
TESSERACT_LANG=eng
# Custom HTTP processor
ENABLE_CUSTOM_PROCESSOR=false
CUSTOM_PROCESSOR_URL=http://localhost:9000/process
CUSTOM_PROCESSOR_TYPES=application/pdf,image/jpeg
```
**Example Usage:**
```
AI: "Read the contents of Documents/report.pdf"
→ Uses nc_webdav_read_file tool with automatic OCR processing
→ Returns extracted text with parsing metadata
→ Sends progress updates during long operations
```
See [env.sample](env.sample) for complete configuration options.
**Example Tools:**
- `nc_notes_create_note` - Create a new note
- `nc_cookbook_import_recipe` - Import recipes from URLs with schema.org metadata
- `deck_create_card` - Create a Deck card
- `nc_calendar_create_event` - Create a calendar event
- `nc_calendar_create_todo` - Create a CalDAV task/todo
- `nc_contacts_create_contact` - Create a contact
- `nc_webdav_upload_file` - Upload a file to Nextcloud
- And 80+ more...
> [!TIP]
> **OAuth Scope Filtering**: When connecting via OAuth, MCP clients will only see tools for which you've granted access. For example, granting only `notes:read` and `notes:write` will show 7 Notes tools instead of all 90+ tools. See [OAuth Scopes Documentation](docs/oauth-architecture.md#oauth-scopes) for the complete scope reference, or [OAuth Troubleshooting - Limited Scopes](docs/oauth-troubleshooting.md#limited-scopes---only-seeing-notes-tools) if you're only seeing a subset of tools.
>
> **Known Issue**: Claude Code and some other MCP clients may only request/grant Notes scopes during initial connection. Track progress at [#234](https://github.com/cbcoutinho/nextcloud-mcp-server/issues/234).
### Resources
Resources provide read-only access to Nextcloud data:
- `nc://capabilities` - Server capabilities
- `cookbook://version` - Cookbook app version info
- `nc://Deck/boards/{board_id}` - Deck board data
- `notes://settings` - Notes app settings
- And more...
Run `uv run nextcloud-mcp-server --help` to see all available options.
- **[Comparison with Context Agent](docs/comparison-context-agent.md)** - When to use each approach
## Examples
@@ -291,45 +139,31 @@ AI: "Create a note called 'Meeting Notes' with today's agenda"
→ Uses nc_notes_create_note tool
```
### Manage Recipes
### Import Recipes
```
AI: "Import the recipe from this URL: https://www.example.com/recipe/chocolate-cake"
→ Uses nc_cookbook_import_recipe tool to extract schema.org metadata
AI: "Import the recipe from https://www.example.com/recipe/chocolate-cake"
→ Uses nc_cookbook_import_recipe tool with schema.org metadata extraction
```
### Manage Calendar
### Schedule Meetings
```
AI: "Schedule a team meeting for next Tuesday at 2pm"
→ Uses nc_calendar_create_event tool
```
### Organize Files
### Manage Files
```
AI: "Create a folder called 'Project X' and move all PDFs there"
→ Uses WebDAV tools (nc_webdav_create_directory, nc_webdav_move)
→ Uses nc_webdav_create_directory and nc_webdav_move tools
```
### Project Management
### Semantic Search (Experimental, Opt-in)
```
AI: "Create a new Deck board for Q1 planning with Todo, In Progress, and Done stacks"
→ Uses deck_create_board and deck_create_stack tools
AI: "Find notes related to machine learning concepts"
→ Uses nc_semantic_search to find semantically similar notes (requires Qdrant + Ollama setup)
```
## Transport Protocols
The server supports multiple MCP transport protocols:
- **streamable-http** (recommended) - Modern streaming protocol
- **sse** (default, deprecated) - Server-Sent Events for backward compatibility
- **http** - Standard HTTP protocol
```bash
# Use streamable-http (recommended)
uv run nextcloud-mcp-server --transport streamable-http
```
> [!WARNING]
> SSE transport is deprecated and will be removed in a future MCP specification version. Please migrate to `streamable-http`.
**Note:** For AI-generated answers with citations, use `nc_semantic_search_answer` (requires MCP client with sampling support).
## Contributing
@@ -337,17 +171,17 @@ Contributions are welcome!
- Report bugs or request features: [GitHub Issues](https://github.com/cbcoutinho/nextcloud-mcp-server/issues)
- Submit improvements: [Pull Requests](https://github.com/cbcoutinho/nextcloud-mcp-server/pulls)
- Read [CLAUDE.md](CLAUDE.md) for development guidelines
- Development guidelines: [CLAUDE.md](CLAUDE.md)
## Security
[![MseeP.ai Security Assessment](https://mseep.net/pr/cbcoutinho-nextcloud-mcp-server-badge.png)](https://mseep.ai/app/cbcoutinho-nextcloud-mcp-server)
This project takes security seriously:
- OAuth2/OIDC support (experimental - requires upstream patches)
- Basic Auth with app-specific passwords (recommended)
- No credential storage with OAuth mode
- Production-ready Basic Auth with app-specific passwords
- OAuth2/OIDC support (experimental, requires upstream patches)
- Per-user access tokens
- No credential storage in OAuth mode
- Regular security assessments
Found a security issue? Please report it privately to the maintainers.
+4 -4
View File
@@ -1,9 +1,9 @@
dependencies:
- name: qdrant
repository: https://qdrant.github.io/qdrant-helm
version: 0.9.0
version: 1.15.5
- name: ollama
repository: https://otwld.github.io/ollama-helm
version: 1.33.0
digest: sha256:c53b7a604d202460f60408a62025ae837cad8d4da970b1e5bb404e2b41289f94
generated: "2025-11-08T23:44:59.709689907+01:00"
version: 1.34.0
digest: sha256:d51c97d05be2614b751c0dd7267ef7dc959eff5ebef859c5f895c5c554b7a874
generated: "2025-11-09T17:08:02.86648061Z"
+5 -5
View File
@@ -2,8 +2,8 @@ apiVersion: v2
name: nextcloud-mcp-server
description: A Helm chart for Nextcloud MCP Server - enables AI assistants to interact with Nextcloud
type: application
version: 0.26.1
appVersion: "0.26.1"
version: 0.32.1
appVersion: "0.32.1"
keywords:
- nextcloud
- mcp
@@ -23,10 +23,10 @@ sources:
icon: https://raw.githubusercontent.com/nextcloud/server/master/core/img/logo/logo.svg
dependencies:
- name: qdrant
version: "0.9.0"
version: "1.15.5"
repository: https://qdrant.github.io/qdrant-helm
condition: qdrant.enabled
condition: qdrant.networkMode.deploySubchart
- name: ollama
version: "1.33.0"
version: "1.34.0"
repository: https://otwld.github.io/ollama-helm
condition: ollama.enabled
+13
View File
@@ -219,6 +219,19 @@ Enable semantic search capabilities by deploying a vector database (Qdrant) and
| `vectorSync.processorWorkers` | Number of concurrent processor workers | `3` |
| `vectorSync.queueMaxSize` | Maximum queue size for pending documents | `10000` |
**Document Chunking Configuration:**
| Parameter | Description | Default |
|-----------|-------------|---------|
| `documentChunking.chunkSize` | Number of words per chunk for embedding | `512` |
| `documentChunking.chunkOverlap` | Number of overlapping words between chunks | `50` |
**Chunking Strategy:**
- **Small chunks (256-384)**: Better precision for searches, more storage overhead
- **Medium chunks (512-768)**: Balanced approach (recommended for most use cases)
- **Large chunks (1024+)**: Better context preservation, less precise matching
- **Overlap**: Should be 10-20% of chunk size to preserve context across boundaries
**Qdrant Vector Database:**
Qdrant is deployed as a subchart when `qdrant.enabled` is `true`. All configuration values are passed through to the [qdrant/qdrant](https://github.com/qdrant/qdrant-helm) chart.
@@ -0,0 +1,90 @@
# Grafana Dashboards
This directory contains example Grafana dashboards for monitoring the Nextcloud MCP Server.
## Dashboards
### nextcloud-mcp-server.json
Comprehensive dashboard with the following panels:
- **Request Rate**: HTTP requests per second by method and endpoint
- **Error Rate**: Percentage of 5xx errors
- **Request Latency**: P50 and P95 latency by endpoint
- **Top MCP Tools**: Most frequently called tools
- **Nextcloud API Latency**: API call latency by app (notes, calendar, etc.)
- **Vector Sync Queue**: Queue size for background document processing
## Importing to Grafana
### Manual Import
1. Open Grafana UI
2. Navigate to Dashboards → Import
3. Upload `nextcloud-mcp-server.json`
4. Select your Prometheus data source
5. Click "Import"
### Automated Import (Kubernetes)
If using the Grafana Operator or kube-prometheus-stack, you can create a ConfigMap:
```bash
kubectl create configmap nextcloud-mcp-dashboards \
--from-file=nextcloud-mcp-server.json \
-n monitoring
# Add label for Grafana sidecar to discover
kubectl label configmap nextcloud-mcp-dashboards \
grafana_dashboard=1 \
-n monitoring
```
Or add to your Helm values:
```yaml
# values.yaml for kube-prometheus-stack
grafana:
dashboardProviders:
dashboardproviders.yaml:
apiVersion: 1
providers:
- name: 'nextcloud-mcp'
orgId: 1
folder: 'Nextcloud MCP'
type: file
disableDeletion: false
editable: true
options:
path: /var/lib/grafana/dashboards/nextcloud-mcp
dashboardsConfigMaps:
nextcloud-mcp: nextcloud-mcp-dashboards
```
## Dashboard Variables
The dashboard includes two variables:
- **Data Source**: Select your Prometheus data source
- **Namespace**: Filter metrics by Kubernetes namespace
## Customization
You can customize the dashboard by:
1. Adjusting refresh rate (default: 30s)
2. Modifying time range (default: last 6 hours)
3. Adding new panels for specific metrics
4. Adjusting thresholds in existing panels
## Metrics Reference
All metrics are documented in `/docs/observability.md`. Key metric prefixes:
- `mcp_http_*` - HTTP server metrics
- `mcp_tool_*` - MCP tool invocation metrics
- `mcp_nextcloud_api_*` - Nextcloud API call metrics
- `mcp_oauth_*` - OAuth token validation metrics
- `mcp_vector_sync_*` - Vector database sync metrics
- `mcp_db_*` - Database operation metrics
@@ -0,0 +1,630 @@
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "grafana",
"uid": "-- Grafana --"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
"id": null,
"links": [],
"liveNow": false,
"panels": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"tooltip": false,
"viz": false,
"legend": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "reqps"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 0
},
"id": 1,
"options": {
"legend": {
"calcs": ["mean", "max"],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"expr": "sum(rate(mcp_http_requests_total{namespace=\"$namespace\"}[5m])) by (method, endpoint)",
"legendFormat": "{{method}} {{endpoint}}",
"refId": "A"
}
],
"title": "Request Rate",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"tooltip": false,
"viz": false,
"legend": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "line"
}
},
"mappings": [],
"max": 100,
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "yellow",
"value": 1
},
{
"color": "red",
"value": 5
}
]
},
"unit": "percent"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 0
},
"id": 2,
"options": {
"legend": {
"calcs": ["mean", "max"],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"expr": "sum(rate(mcp_http_requests_total{status_code=~\"5..\", namespace=\"$namespace\"}[5m])) / sum(rate(mcp_http_requests_total{namespace=\"$namespace\"}[5m])) * 100",
"legendFormat": "Error Rate",
"refId": "A"
}
],
"title": "Error Rate (%)",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"tooltip": false,
"viz": false,
"legend": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 8
},
"id": 3,
"options": {
"legend": {
"calcs": ["mean", "max"],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"expr": "histogram_quantile(0.95, sum(rate(mcp_http_request_duration_seconds_bucket{namespace=\"$namespace\"}[5m])) by (le, endpoint))",
"legendFormat": "{{endpoint}} (p95)",
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"expr": "histogram_quantile(0.50, sum(rate(mcp_http_request_duration_seconds_bucket{namespace=\"$namespace\"}[5m])) by (le, endpoint))",
"legendFormat": "{{endpoint}} (p50)",
"refId": "B"
}
],
"title": "Request Latency (P50/P95)",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"tooltip": false,
"viz": false,
"legend": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 8
},
"id": 4,
"options": {
"legend": {
"calcs": ["mean", "max"],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"expr": "topk(10, sum(rate(mcp_tool_calls_total{namespace=\"$namespace\"}[5m])) by (tool_name))",
"legendFormat": "{{tool_name}}",
"refId": "A"
}
],
"title": "Top MCP Tools by Volume",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"tooltip": false,
"viz": false,
"legend": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 16
},
"id": 5,
"options": {
"legend": {
"calcs": ["mean", "max"],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"expr": "histogram_quantile(0.95, sum(rate(mcp_nextcloud_api_duration_seconds_bucket{namespace=\"$namespace\"}[5m])) by (le, app))",
"legendFormat": "{{app}} (p95)",
"refId": "A"
}
],
"title": "Nextcloud API Latency by App",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"tooltip": false,
"viz": false,
"legend": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 16
},
"id": 6,
"options": {
"legend": {
"calcs": ["mean", "lastNotNull"],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"expr": "mcp_vector_sync_queue_size{namespace=\"$namespace\"}",
"legendFormat": "Queue Size",
"refId": "A"
}
],
"title": "Vector Sync Queue Size",
"type": "timeseries"
}
],
"refresh": "30s",
"schemaVersion": 38,
"style": "dark",
"tags": ["nextcloud", "mcp", "observability"],
"templating": {
"list": [
{
"current": {
"selected": false,
"text": "Prometheus",
"value": "Prometheus"
},
"hide": 0,
"includeAll": false,
"label": "Data Source",
"multi": false,
"name": "datasource",
"options": [],
"query": "prometheus",
"refresh": 1,
"regex": "",
"skipUrlSync": false,
"type": "datasource"
},
{
"current": {
"selected": false,
"text": "default",
"value": "default"
},
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"definition": "label_values(mcp_http_requests_total, namespace)",
"hide": 0,
"includeAll": false,
"label": "Namespace",
"multi": false,
"name": "namespace",
"options": [],
"query": {
"query": "label_values(mcp_http_requests_total, namespace)",
"refId": "PrometheusVariableQueryEditor-VariableQuery"
},
"refresh": 1,
"regex": "",
"skipUrlSync": false,
"sort": 0,
"type": "query"
}
]
},
"time": {
"from": "now-6h",
"to": "now"
},
"timepicker": {},
"timezone": "",
"title": "Nextcloud MCP Server",
"uid": "nextcloud-mcp-server",
"version": 1,
"weekStart": ""
}
@@ -5,6 +5,8 @@ metadata:
labels:
{{- include "nextcloud-mcp-server.labels" . | nindent 4 }}
spec:
strategy:
type: Recreate
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount }}
{{- end }}
@@ -56,6 +58,11 @@ spec:
- name: http
containerPort: {{ include "nextcloud-mcp-server.port" . }}
protocol: TCP
{{- if .Values.observability.metrics.enabled }}
- name: metrics
containerPort: {{ .Values.observability.metrics.port }}
protocol: TCP
{{- end }}
env:
# Nextcloud connection
- name: NEXTCLOUD_HOST
@@ -151,6 +158,11 @@ spec:
- name: VECTOR_SYNC_QUEUE_MAX_SIZE
value: {{ .Values.vectorSync.queueMaxSize | quote }}
{{- end }}
# Document Chunking (always set, used by vector sync processor)
- name: DOCUMENT_CHUNK_SIZE
value: {{ .Values.documentChunking.chunkSize | quote }}
- name: DOCUMENT_CHUNK_OVERLAP
value: {{ .Values.documentChunking.chunkOverlap | quote }}
# Qdrant Vector Database
{{- if eq .Values.qdrant.mode "network" }}
# Network mode: Use dedicated Qdrant service
@@ -200,6 +212,25 @@ spec:
value: {{ .Values.openai.baseUrl | quote }}
{{- end }}
{{- end }}
# Observability
- name: METRICS_ENABLED
value: {{ .Values.observability.metrics.enabled | quote }}
- name: METRICS_PORT
value: {{ .Values.observability.metrics.port | quote }}
{{- if .Values.observability.tracing.enabled }}
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: {{ .Values.observability.tracing.endpoint | quote }}
- name: OTEL_SERVICE_NAME
value: {{ .Values.observability.tracing.serviceName | quote }}
- name: OTEL_TRACES_SAMPLER_ARG
value: {{ .Values.observability.tracing.samplingRate | quote }}
{{- end }}
- name: LOG_FORMAT
value: {{ .Values.observability.logging.format | quote }}
- name: LOG_LEVEL
value: {{ .Values.observability.logging.level | quote }}
- name: LOG_INCLUDE_TRACE_CONTEXT
value: {{ .Values.observability.logging.includeTraceContext | quote }}
{{- with .Values.extraEnv }}
{{- toYaml . | nindent 12 }}
{{- end }}
@@ -0,0 +1,92 @@
{{- if and .Values.observability.metrics.enabled .Values.prometheusRule.enabled }}
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: {{ include "nextcloud-mcp-server.fullname" . }}
namespace: {{ .Release.Namespace }}
labels:
{{- include "nextcloud-mcp-server.labels" . | nindent 4 }}
{{- with .Values.prometheusRule.labels }}
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
groups:
- name: nextcloud-mcp-server.critical
interval: 30s
rules:
- alert: NextcloudMCPServerDown
expr: up{job="{{ include "nextcloud-mcp-server.fullname" . }}"} == 0
for: 5m
labels:
severity: critical
annotations:
summary: "Nextcloud MCP Server is down"
description: "{{ `{{` }} $labels.pod {{ `}}` }} has been down for more than 5 minutes."
- alert: NextcloudMCPHighErrorRate
expr: |
sum(rate(mcp_http_requests_total{status_code=~"5..", job="{{ include "nextcloud-mcp-server.fullname" . }}"}[5m]))
/ sum(rate(mcp_http_requests_total{job="{{ include "nextcloud-mcp-server.fullname" . }}"}[5m])) > 0.05
for: 5m
labels:
severity: critical
annotations:
summary: "High error rate on Nextcloud MCP Server"
description: "Error rate is {{ `{{` }} printf \"%.2f%%\" (mul $value 100) {{ `}}` }} (threshold: 5%)"
- alert: NextcloudMCPHighLatency
expr: |
histogram_quantile(0.95,
sum(rate(mcp_http_request_duration_seconds_bucket{job="{{ include "nextcloud-mcp-server.fullname" . }}"}[5m])) by (le, endpoint)
) > 1
for: 5m
labels:
severity: critical
annotations:
summary: "High latency on Nextcloud MCP Server"
description: "P95 latency is {{ `{{` }} printf \"%.2fs\" $value {{ `}}` }} on {{ `{{` }} $labels.endpoint {{ `}}` }} (threshold: 1s)"
- alert: NextcloudMCPDependencyDown
expr: mcp_dependency_health{job="{{ include "nextcloud-mcp-server.fullname" . }}"} == 0
for: 2m
labels:
severity: critical
annotations:
summary: "Nextcloud MCP dependency is down"
description: "Dependency {{ `{{` }} $labels.dependency {{ `}}` }} has been down for more than 2 minutes."
- name: nextcloud-mcp-server.warning
interval: 30s
rules:
- alert: NextcloudMCPTokenValidationErrors
expr: |
sum(rate(mcp_oauth_token_validations_total{result="error", job="{{ include "nextcloud-mcp-server.fullname" . }}"}[10m]))
/ sum(rate(mcp_oauth_token_validations_total{job="{{ include "nextcloud-mcp-server.fullname" . }}"}[10m])) > 0.01
for: 10m
labels:
severity: warning
annotations:
summary: "High token validation error rate"
description: "Token validation error rate is {{ `{{` }} printf \"%.2f%%\" (mul $value 100) {{ `}}` }} (threshold: 1%)"
- alert: NextcloudMCPVectorSyncQueueHigh
expr: mcp_vector_sync_queue_size{job="{{ include "nextcloud-mcp-server.fullname" . }}"} > 100
for: 15m
labels:
severity: warning
annotations:
summary: "Vector sync queue is high"
description: "Vector sync queue size is {{ `{{` }} $value {{ `}}` }} (threshold: 100)"
- alert: NextcloudMCPQdrantSlowQueries
expr: |
histogram_quantile(0.95,
sum(rate(mcp_db_operation_duration_seconds_bucket{db="qdrant", job="{{ include "nextcloud-mcp-server.fullname" . }}"}[10m])) by (le)
) > 0.5
for: 10m
labels:
severity: warning
annotations:
summary: "Qdrant queries are slow"
description: "P95 Qdrant query latency is {{ `{{` }} printf \"%.2fs\" $value {{ `}}` }} (threshold: 0.5s)"
{{- end }}
@@ -15,5 +15,11 @@ spec:
targetPort: http
protocol: TCP
name: http
{{- if .Values.observability.metrics.enabled }}
- port: {{ .Values.observability.metrics.port }}
targetPort: metrics
protocol: TCP
name: metrics
{{- end }}
selector:
{{- include "nextcloud-mcp-server.selectorLabels" . | nindent 4 }}
@@ -0,0 +1,32 @@
{{- if and .Values.observability.metrics.enabled .Values.serviceMonitor.enabled }}
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: {{ include "nextcloud-mcp-server.fullname" . }}
namespace: {{ .Release.Namespace }}
labels:
{{- include "nextcloud-mcp-server.labels" . | nindent 4 }}
{{- with .Values.serviceMonitor.labels }}
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
selector:
matchLabels:
{{- include "nextcloud-mcp-server.selectorLabels" . | nindent 6 }}
endpoints:
- port: metrics
path: {{ .Values.observability.metrics.path }}
interval: {{ .Values.serviceMonitor.interval }}
scrapeTimeout: {{ .Values.serviceMonitor.scrapeTimeout }}
scheme: http
relabelings:
# Add namespace label
- sourceLabels: [__meta_kubernetes_namespace]
targetLabel: namespace
# Add pod label
- sourceLabels: [__meta_kubernetes_pod_name]
targetLabel: pod
# Add service label
- sourceLabels: [__meta_kubernetes_service_name]
targetLabel: service
{{- end }}
+51
View File
@@ -168,6 +168,43 @@ securityContext:
runAsNonRoot: true
runAsUser: 1000
# Observability Configuration
observability:
# Prometheus metrics
metrics:
enabled: true
port: 9090
path: /metrics
# OpenTelemetry tracing
tracing:
enabled: false
endpoint: "" # e.g., "http://opentelemetry-collector:4317"
serviceName: "nextcloud-mcp-server"
samplingRate: 1.0
# Logging configuration
logging:
format: json # "json" or "text"
level: INFO
includeTraceContext: true
# Prometheus ServiceMonitor (requires Prometheus Operator)
serviceMonitor:
enabled: false
interval: 30s
scrapeTimeout: 10s
labels: {}
# Additional labels for ServiceMonitor (e.g., for Prometheus selector)
# Example: { prometheus: kube-prometheus }
# Prometheus alert rules (requires Prometheus Operator)
prometheusRule:
enabled: false
labels: {}
# Additional labels for PrometheusRule (e.g., for Prometheus selector)
# Example: { prometheus: kube-prometheus }
service:
type: ClusterIP
port: 8000
@@ -277,6 +314,20 @@ vectorSync:
# Maximum queue size for documents pending indexing
queueMaxSize: 10000
# Document Chunking Configuration
# Controls how documents are split into chunks before embedding
# Only relevant when vectorSync.enabled is true
documentChunking:
# Number of words per chunk (default: 512)
# Smaller chunks (256-384): Better for precise searches, more chunks to store
# Medium chunks (512-768): Balanced approach (recommended for most use cases)
# Larger chunks (1024+): Better for context, less precise matching
chunkSize: 512
# Number of overlapping words between chunks (default: 50)
# Recommended: 10-20% of chunkSize for context preservation across boundaries
# Must be less than chunkSize
chunkOverlap: 50
# Qdrant Vector Database Configuration
# Three deployment modes available:
# 1. Local In-Memory: Fast, ephemeral, zero-config (mode: "memory")
+26 -8
View File
@@ -3,7 +3,7 @@ services:
# https://hub.docker.com/_/mariadb
db:
# Note: Check the recommend version here: https://docs.nextcloud.com/server/latest/admin_manual/installation/system_requirements.html#server
image: docker.io/library/mariadb:lts@sha256:ae6119716edac6998ae85508431b3d2e666530ddf4e94c61a10710caec9b0f71
image: docker.io/library/mariadb:lts@sha256:404ebf26ed7a56fbab05c29f6f1e70188e5eadb51bba8cee8d355775776deb08
restart: always
command: --transaction-isolation=READ-COMMITTED
volumes:
@@ -88,20 +88,38 @@ services:
- VECTOR_SYNC_SCAN_INTERVAL=10
- VECTOR_SYNC_PROCESSOR_WORKERS=1
#- LOG_FORMAT=json
# Qdrant configuration (three modes):
# 1. Network mode: Set QDRANT_URL=http://qdrant:6333 (requires qdrant service)
# 2. In-memory mode: Set QDRANT_LOCATION=:memory: (default if nothing set)
# 3. Persistent local: Set QDRANT_LOCATION=/app/data/qdrant (stored in mcp-data volume)
- QDRANT_LOCATION=:memory:
# - QDRANT_URL=http://qdrant:6333 # Uncomment for network mode
# - QDRANT_API_KEY=${QDRANT_API_KEY:-my_secret_api_key} # Only for network mode
- QDRANT_COLLECTION=nextcloud_content
#- QDRANT_LOCATION=/app/data/qdrant # In-memory mode used if not set
#- QDRANT_URL=http://qdrant:6333 # Uncomment for network mode
#- QDRANT_API_KEY=${QDRANT_API_KEY:-my_secret_api_key} # Only for network mode
# Observability
#- OTEL_SERVICE_NAME=nextcloud-mcp-docker-compose
#- OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
# Collection naming: Auto-generated as {deployment-id}-{model-name}
# - Deployment ID: OTEL_SERVICE_NAME (if set) or hostname (fallback)
# - Model name: OLLAMA_EMBEDDING_MODEL
# - Example: "nextcloud-mcp-server-nomic-embed-text"
# - Changing models creates new collection (requires re-embedding)
# - Set QDRANT_COLLECTION to override auto-generation:
#- QDRANT_COLLECTION=nextcloud_content
# Ollama configuration (optional - uses SimpleEmbeddingProvider if not set)
# - OLLAMA_BASE_URL=http://your-ollama-endpoint:port
# - OLLAMA_EMBEDDING_MODEL=nomic-embed-text
# - OLLAMA_BASE_URL=http://ollama:11434
# - OLLAMA_EMBEDDING_MODEL=nomic-embed-text # Changing this creates new collection
# - OLLAMA_VERIFY_SSL=false
# Document chunking configuration (for vector embeddings)
# Tune these based on your embedding model and content type
# - DOCUMENT_CHUNK_SIZE=512 # Words per chunk (default: 512)
# - DOCUMENT_CHUNK_OVERLAP=50 # Overlapping words (default: 50, recommended: 10-20% of chunk size)
mcp-oauth:
build: .
command: ["--transport", "streamable-http", "--oauth", "--port", "8001", "--oauth-token-type", "jwt"]
@@ -205,7 +223,7 @@ services:
- keycloak-oauth-storage:/app/.oauth
qdrant:
image: qdrant/qdrant:latest
image: qdrant/qdrant:v1.15.5@sha256:0fb8897412abc81d1c0430a899b9a81eb8328aa634e7242d1bc804c1fe8fe863
restart: always
ports:
- 127.0.0.1:6333:6333 # REST API
@@ -377,7 +377,7 @@ async def get_vector_sync_status(ctx: Context) -> dict:
}
```
The web UI (`/user/page` route) mirrors these controls with a simple toggle switch for enabling/disabling sync and a status display showing indexed counts and sync state. There is no job history, no detailed progress bars, no per-document status—just the essential information users need.
The web UI (`/app` route) mirrors these controls with a simple toggle switch for enabling/disabling sync and a status display showing indexed counts and sync state. There is no job history, no detailed progress bars, no per-document status—just the essential information users need.
### Authentication and Offline Access
+661
View File
@@ -0,0 +1,661 @@
# ADR-010: Webhook-Based Vector Database Synchronization
**Status**: Proposed
**Date**: 2025-01-10
**Depends On**: ADR-007 (Background Vector Sync)
## Context
ADR-007 established a background synchronization architecture for maintaining the vector database using periodic polling. The scanner task runs on a configurable interval (default 3600 seconds / 1 hour) to detect changed documents across Nextcloud apps. While this polling approach is simple and reliable, it introduces significant latency between content changes and vector database updates.
### Current Polling Architecture
The existing scanner implementation in `nextcloud_mcp_server/vector/scanner.py` operates as follows:
1. **Periodic Scanning**: The scanner task sleeps for `vector_sync_scan_interval` seconds between runs
2. **Change Detection**: For each scan, it:
- Fetches all documents from Nextcloud (notes, calendar events, etc.)
- Queries Qdrant for the last indexed timestamp of each document
- Compares modification timestamps to detect changes
- Queues changed documents for processing
3. **Document Processing**: Processor tasks pull from the queue, generate embeddings, and update Qdrant
This architecture works but has fundamental limitations:
**Latency**: With a 1-hour scan interval, content changes can take up to 1 hour to appear in semantic search results. For time-sensitive use cases (e.g., "What's on my calendar today?"), this delay is problematic.
**API Load**: Every scan fetches *all* documents for *all* enabled users, regardless of whether anything changed. For large deployments with thousands of documents, this generates significant unnecessary API traffic to Nextcloud.
**Resource Waste**: The scanner and processors consume compute resources even when no content has changed. During periods of low activity, the system performs wasteful polling.
**Scalability**: As the number of users and documents grows, the time required to complete a full scan increases. Eventually, the scan duration may exceed the scan interval, causing scans to run continuously without idle periods.
**Rate Limiting**: Fetching all documents for all users in rapid succession can trigger Nextcloud's rate limiting, especially on shared hosting environments with restrictive API quotas.
These limitations are inherent to any polling-based architecture. Reducing the scan interval (e.g., to 5 minutes) reduces latency but exacerbates API load, resource waste, and rate limiting issues. The fundamental problem is that the system has no way to know *when* content changes occur—it must repeatedly check to find out.
### Nextcloud Webhook Listeners
Nextcloud provides a webhook_listeners app (bundled with Nextcloud 30+) that enables push-based change notifications. Instead of polling for changes, external services can register webhook endpoints and receive HTTP POST requests when specific events occur. Administrators register these webhooks using Nextcloud's OCS API or occ commands.
The webhook_listeners app supports events for all Nextcloud apps relevant to this MCP server's vector database:
**Files/Notes Events** (notes are stored as files):
- `OCP\Files\Events\Node\NodeCreatedEvent`
- `OCP\Files\Events\Node\NodeWrittenEvent`
- `OCP\Files\Events\Node\BeforeNodeDeletedEvent`**Use this for deletion (includes node.id)**
- `OCP\Files\Events\Node\NodeDeletedEvent` (missing node.id - file already deleted)
- `OCP\Files\Events\Node\NodeRenamedEvent`
- `OCP\Files\Events\Node\NodeCopiedEvent`
**Calendar Events**:
- `OCP\Calendar\Events\CalendarObjectCreatedEvent`
- `OCP\Calendar\Events\CalendarObjectUpdatedEvent`
- `OCP\Calendar\Events\CalendarObjectDeletedEvent`
- `OCP\Calendar\Events\CalendarObjectMovedEvent`
**Tables Events**:
- `OCA\Tables\Event\RowAddedEvent`
- `OCA\Tables\Event\RowUpdatedEvent`
- `OCA\Tables\Event\RowDeletedEvent`
**Deck Events** (via file events since cards are stored as files in some configurations)
Each webhook notification includes rich metadata:
- User ID who triggered the event
- Timestamp of the event
- Document ID and metadata
- Operation type (create, update, delete)
- Path information (for files)
Webhook notifications are dispatched via background jobs, with configurable delivery guarantees. Administrators can set up dedicated webhook worker processes to achieve near-real-time delivery (within seconds of the triggering event).
### Why Not Replace Polling Entirely?
While webhooks provide superior latency and efficiency, they cannot fully replace polling:
**Missed Events**: If the MCP server is down when a webhook fires, the notification is lost. Nextcloud's background job system processes webhooks asynchronously, but does not queue failed deliveries indefinitely.
**Administrator Setup**: Webhooks must be registered by Nextcloud administrators using the OCS API or occ commands. This is an optional optimization that administrators can enable when they want to reduce polling frequency.
**Filter Configuration**: Webhook filters must be carefully configured to avoid notification floods. A poorly configured filter could send thousands of notifications for bulk operations (e.g., importing a calendar with hundreds of events).
**Graceful Degradation**: In environments where webhooks are not configured, the system continues using polling without any degradation in functionality.
**Deletion Detection**: Nextcloud's webhook system does not guarantee delivery of deletion events if the user's account is removed or the app is uninstalled. Periodic polling provides a safety mechanism to detect orphaned documents.
A complementary architecture where webhooks supplement (but don't replace) polling provides low-latency updates when configured, with polling ensuring reliability.
### Design Considerations
**Push vs Pull Trade-offs**:
Webhooks introduce new failure modes (network issues, endpoint unavailability, notification floods) that polling avoids. The webhook endpoint must handle failures gracefully without blocking semantic search functionality.
**Webhook Endpoint Security**:
The MCP server exposes an HTTP endpoint to receive webhooks. Authentication is optional—in production deployments, administrators can configure Nextcloud to send an `Authorization` header that the MCP server validates. For local development, authentication can be disabled for simplicity.
**Idempotency**:
The system may receive duplicate notifications (webhook + next scan) or out-of-order notifications (update fires before create completes). Document processing must be idempotent—processing the same document multiple times produces the same result.
**Asynchronous Processing**:
Nextcloud processes webhooks via background jobs, introducing delivery latency (typically seconds to minutes depending on background job configuration). This affects testing strategies—integration tests cannot rely on immediate webhook delivery.
**Deployment Patterns**:
The MCP server webhook endpoint is accessible at the same host/port as the MCP server itself. Administrators configure Nextcloud to POST to `https://<mcp-server-host>:<port>/webhooks/nextcloud` when registering webhook listeners.
## Decision
We will add a webhook endpoint to the MCP server that receives change notifications from Nextcloud and queues documents for vector database processing. This complements the existing polling architecture from ADR-007 without replacing it—webhooks provide low-latency updates when configured, while polling ensures reliability regardless of webhook availability.
The architecture is intentionally simple: the webhook endpoint is just another producer of `DocumentTask` objects that feed into the existing processor queue. The scanner task, processor pool, and queue management remain unchanged from ADR-007.
### Architecture Components
**1. Webhook Endpoint**
A new Starlette HTTP route will be added to receive webhook notifications from Nextcloud:
```python
from starlette.requests import Request
from starlette.responses import JSONResponse
@app.route("/webhooks/nextcloud", methods=["POST"])
async def handle_nextcloud_webhook(request: Request) -> JSONResponse:
"""
Receive webhook notifications from Nextcloud.
Parses event payload, extracts document metadata, and queues
changed documents for processing using the same queue as the scanner.
"""
# 1. Optional authentication validation
if settings.webhook_secret:
auth_header = request.headers.get("authorization", "")
if not auth_header.startswith("Bearer ") or \
auth_header[7:] != settings.webhook_secret:
logger.warning("Webhook authentication failed")
return JSONResponse(
{"status": "error", "message": "Unauthorized"},
status_code=401
)
# 2. Parse webhook payload
payload = await request.json()
event_class = payload["event"]["class"]
user_id = payload["user"]["uid"]
# 3. Extract document metadata from event
doc_task = extract_document_task(event_class, payload)
if not doc_task:
return JSONResponse({"status": "ignored", "reason": "unsupported event"})
# 4. Send to processor queue (same queue as scanner)
try:
await webhook_send_stream.send(doc_task)
logger.info(f"Queued document from webhook: {doc_task}")
return JSONResponse({"status": "queued"})
except Exception as e:
logger.error(f"Failed to queue webhook document: {e}")
return JSONResponse(
{"status": "error", "message": str(e)},
status_code=500
)
```
The endpoint:
- Validates optional authentication via `Authorization: Bearer <secret>` header
- Parses various event types (calendar, files, tables) into `DocumentTask` objects
- Sends to the same processing queue that the scanner uses
- Returns quickly (<50ms) to avoid blocking Nextcloud's webhook workers
- Handles errors gracefully (invalid payload, queue full, etc.)
**2. Webhook Registration Helper (Development Only)**
For development and testing purposes, a helper method will be added to `NextcloudClient` for registering webhooks via the OCS API. This is NOT exposed as an MCP tool—administrators register webhooks manually using Nextcloud's admin interface or the OCS API directly.
```python
class NextcloudClient:
async def register_webhook(
self,
event_type: str,
uri: str,
http_method: str = "POST",
auth_method: str = "none",
headers: dict[str, str] | None = None,
) -> dict:
"""
Register a webhook with Nextcloud (requires admin credentials).
Used for development/testing. Production admins should register
webhooks using Nextcloud's admin UI or occ commands.
"""
# Implementation uses OCS API: POST /ocs/v2.php/apps/webhook_listeners/api/v1/webhooks
...
```
This keeps webhook registration out of the MCP tool surface while providing a convenient API for integration tests.
**3. Event Parsing**
A helper function extracts `DocumentTask` from various Nextcloud event types:
```python
def extract_document_task(event_class: str, payload: dict) -> DocumentTask | None:
"""Extract DocumentTask from webhook event payload."""
user_id = payload["user"]["uid"]
event_data = payload["event"]
# File/Note events
if "NodeCreatedEvent" in event_class or "NodeWrittenEvent" in event_class:
# Only process markdown files (notes)
path = event_data["node"]["path"]
if not path.endswith(".md"):
return None
return DocumentTask(
user_id=user_id,
doc_id=event_data["node"]["id"],
doc_type="note",
operation="index",
modified_at=payload["time"],
)
# Calendar events
elif "CalendarObjectCreatedEvent" in event_class or \
"CalendarObjectUpdatedEvent" in event_class:
return DocumentTask(
user_id=user_id,
doc_id=str(event_data["objectData"]["id"]),
doc_type="calendar_event",
operation="index",
modified_at=event_data["objectData"]["lastmodified"],
)
# Deletion events (use BeforeNodeDeletedEvent for files to get node.id)
elif "BeforeNodeDeletedEvent" in event_class or \
"NodeDeletedEvent" in event_class or \
"CalendarObjectDeletedEvent" in event_class:
# Similar logic for delete operations
...
return None # Unsupported event type
```
**4. No Changes to Scanner or Processors**
The existing scanner task from ADR-007 continues operating unchanged. It polls Nextcloud on its configured interval (`VECTOR_SYNC_SCAN_INTERVAL`), discovers changed documents, and queues them for processing. The scanner is unaware of webhooks—it simply adds `DocumentTask` objects to the queue.
Similarly, the processor pool continues pulling `DocumentTask` objects from the queue, generating embeddings, and updating Qdrant. Processors don't know or care whether a task came from the scanner or a webhook.
This design keeps concerns separated: webhooks and scanner are independent producers, processors are independent consumers, and the queue mediates between them.
### Configuration
A new optional environment variable controls webhook authentication:
```bash
# Optional: Shared secret for webhook authentication
# If set, webhooks must include "Authorization: Bearer <secret>" header
# If unset, no authentication is required (useful for local development)
WEBHOOK_SECRET=<generate-random-secret>
```
The webhook endpoint is automatically available at `/webhooks/nextcloud` when the MCP server starts. No feature flags or additional configuration needed—if Nextcloud sends webhooks to this endpoint, they will be processed.
**Reducing Polling Frequency**: Administrators who configure webhooks may want to reduce polling frequency to minimize API load while maintaining safety reconciliation scans:
```bash
# Increase scan interval from 1 hour (default) to 24 hours
VECTOR_SYNC_SCAN_INTERVAL=86400
```
This is a manual configuration decision, not automatic—the scanner doesn't adapt based on webhook availability.
### Webhook Event Mapping
The webhook handler maps Nextcloud events to document types:
| Nextcloud Event | Document Type | Operation |
|----------------|---------------|-----------|
| `NodeCreatedEvent` (path: `*/files/*.md`) | `note` | `index` |
| `NodeWrittenEvent` (path: `*/files/*.md`) | `note` | `index` |
| `NodeDeletedEvent` (path: `*/files/*.md`) | `note` | `delete` |
| `CalendarObjectCreatedEvent` | `calendar_event` | `index` |
| `CalendarObjectUpdatedEvent` | `calendar_event` | `index` |
| `CalendarObjectDeletedEvent` | `calendar_event` | `delete` |
| `RowAddedEvent` | `table_row` | `index` |
| `RowUpdatedEvent` | `table_row` | `index` |
| `RowDeletedEvent` | `table_row` | `delete` |
Path filters in webhook registration ensure only relevant files trigger notifications (e.g., exclude `.jpg`, `.mp4` for file events).
### Administrator Setup
Administrators who want to enable webhooks:
1. **Enable webhook_listeners app** in Nextcloud: `occ app:enable webhook_listeners`
2. **Register webhook endpoints** using Nextcloud's OCS API or admin UI:
- Endpoint: `https://<mcp-server-host>:<port>/webhooks/nextcloud`
- Events: File created/updated/deleted, Calendar object events, Table row events
- Filters: Exclude non-content files (images, videos), system directories
- Optional: Configure `Authorization: Bearer <WEBHOOK_SECRET>` header
3. **Optionally reduce scanner frequency**: Set `VECTOR_SYNC_SCAN_INTERVAL=86400` (24 hours)
4. **Set up webhook workers** (optional): Configure dedicated background job workers for low-latency delivery
Existing deployments continue using polling without any changes. Webhooks are purely additive.
## Consequences
### Benefits
**Reduced Latency**: With webhooks configured, content changes appear in semantic search within seconds to minutes (depending on Nextcloud background job configuration) instead of up to 1 hour. Queries like "What meetings do I have today?" reflect recent calendar updates.
**Lower API Load**: Administrators who configure webhooks can reduce scanner frequency (e.g., 24-hour intervals), eliminating most polling API calls while maintaining safety reconciliation scans. This significantly reduces load on Nextcloud servers.
**Better Scalability**: Webhooks scale better than polling as content volume grows. The system only processes changed documents instead of checking all documents every hour.
**Simple Architecture**: The webhook endpoint is just another producer feeding the existing processor queue. No changes to scanner, processors, or queue management—webhooks integrate cleanly into the existing architecture.
**Improved User Experience**: Lower-latency semantic search feels more responsive and accurate, especially for time-sensitive queries about recent changes.
### Drawbacks
**Manual Configuration**: Administrators must configure webhooks outside the MCP server using Nextcloud's admin tools. This adds setup complexity compared to the zero-configuration polling approach.
**Deployment Requirements**: Webhooks require the MCP server to be reachable from Nextcloud via HTTP(S). Deployments behind NAT or with restrictive firewalls may not support webhooks without additional networking configuration.
**Asynchronous Delivery**: Nextcloud processes webhooks via background jobs, introducing delivery latency (typically seconds to minutes). The exact latency depends on background job worker configuration and system load.
**Testing Complexity**: Integration tests cannot rely on immediate webhook delivery due to asynchronous background job processing. Tests must either poll for results or mock webhook delivery directly.
**New Failure Modes**: Webhook endpoint downtime, network issues between Nextcloud and MCP server, webhook notification floods from bulk operations. The system must handle these gracefully.
**Version Dependencies**: The webhook_listeners app requires Nextcloud 30+. Older versions continue using polling exclusively.
### Monitoring and Observability
New metrics track webhook performance:
- `webhook_notifications_received_total{event_type}`: Count of webhook notifications by event type
- `webhook_processing_duration_seconds{event_type}`: Webhook handler latency
- `webhook_errors_total{error_type}`: Failed webhook processing by error type (auth failure, parse error, queue full)
Logs include:
- Successful webhook processing: `Queued document from webhook: DocumentTask(...)`
- Webhook authentication failures: `Webhook authentication failed`
- Parse errors: `Failed to parse webhook payload: ...`
- Unsupported events: `Ignoring webhook for unsupported event: ...`
### Security Considerations
**Optional Authentication**: When `WEBHOOK_SECRET` is configured, webhook requests must include `Authorization: Bearer <WEBHOOK_SECRET>` header. The server validates this before processing to prevent unauthorized document queueing. For local development, authentication can be disabled by leaving `WEBHOOK_SECRET` unset.
**Payload Validation**: Webhook payloads are parsed and validated against expected schemas. Malformed payloads are rejected with 400 Bad Request responses.
**No Scope Enforcement**: Unlike MCP tools, webhooks do not enforce progressive consent or check if users have enabled semantic search. Webhooks queue all document changes—administrators control which events trigger webhooks via Nextcloud filters. This keeps the webhook endpoint simple and stateless.
### Testing Strategy
**Unit Tests**: Test webhook handler logic, event parsing, and authentication validation using mocked payloads:
```python
async def test_webhook_endpoint_parses_note_created_event():
"""Unit test: webhook endpoint extracts DocumentTask from note created event."""
payload = {
"user": {"uid": "alice"},
"time": 1704067200,
"event": {
"class": "OCP\\Files\\Events\\Node\\NodeCreatedEvent",
"node": {"id": "123", "path": "/alice/files/test.md"}
}
}
# Mock send_stream and verify DocumentTask is queued
...
```
**Integration Tests (Without Real Webhooks)**: Since Nextcloud processes webhooks asynchronously via background jobs, integration tests should NOT rely on triggering real Nextcloud events and waiting for webhook delivery. Instead, tests should:
1. **Mock webhook delivery**: POST webhook payloads directly to the `/webhooks/nextcloud` endpoint
2. **Verify processing**: Check that documents are queued and eventually appear in Qdrant
3. **Test authentication**: Verify requests without valid auth header are rejected (when `WEBHOOK_SECRET` is set)
```python
async def test_webhook_integration_mocked_delivery():
"""Integration test: webhook handler queues document for processing."""
# POST webhook payload directly to endpoint (bypass Nextcloud)
response = await client.post("/webhooks/nextcloud", json=note_created_payload)
assert response.status_code == 200
# Wait for processor to handle document
await asyncio.sleep(2)
# Verify document appears in Qdrant
results = await qdrant_client.scroll(...)
assert len(results[0]) > 0
```
**Manual Testing (Real Webhooks)**: For end-to-end validation with real Nextcloud webhook delivery:
1. Register webhook via OCS API or `NextcloudClient.register_webhook()` helper
2. Configure webhook background job workers for low-latency delivery
3. Trigger Nextcloud events (create note, add calendar event)
4. Monitor MCP server logs for webhook delivery
5. Verify documents appear in Qdrant after background job processing
**Failure Mode Tests**:
- Invalid authentication: Verify 401 response when auth header is missing/incorrect
- Malformed payload: Verify 400 response for invalid JSON or missing required fields
- Unsupported event types: Verify graceful handling (ignored, not error)
- Queue full: Verify 500 response with appropriate error message
### Future Enhancements
**Batch Processing**: Group multiple webhook notifications within a short time window (e.g., 5 seconds) into a single batch before queueing. This reduces processor overhead during bulk operations like importing calendars.
**Webhook Payload Optimization**: For large documents, Nextcloud could be configured to send minimal metadata in webhooks (just user_id, doc_id, doc_type), with processors fetching full content lazily. This reduces webhook payload size and network bandwidth.
**Deduplication Window**: Track recently processed documents (last 5 minutes) to avoid redundant work when webhooks and scanner both detect the same change. The processor can check a simple in-memory cache before fetching document content.
## Appendix A: Manual Webhook Testing Results (2025-01-11)
### Testing Summary
Manual validation of Nextcloud webhook schemas and behavior confirmed that webhooks work as documented with several important findings for implementation. **5 out of 6** webhook types were successfully captured and validated.
**Test Environment:**
- Nextcloud 30+ (Docker compose)
- webhook_listeners app enabled
- Test endpoint: `http://mcp:8000/webhooks/nextcloud`
- Background webhook worker running (60s timeout)
**Results:**
- ✅ NodeCreatedEvent (file creation)
- ✅ NodeWrittenEvent (file update)
- ✅ NodeDeletedEvent (file deletion)
- ✅ CalendarObjectCreatedEvent
- ✅ CalendarObjectUpdatedEvent
- ❌ CalendarObjectDeletedEvent (webhook did not fire - potential Nextcloud bug)
### Critical Implementation Findings
#### 1. Deletion Events Lack `node.id` Field
**Finding:** `NodeDeletedEvent` payloads do NOT include `event.node.id`, only `event.node.path`.
**Example:**
```json
{
"user": {"uid": "admin", "displayName": "admin"},
"time": 1762851093,
"event": {
"class": "OCP\\Files\\Events\\Node\\NodeDeletedEvent",
"node": {
"path": "/admin/files/Notes/Webhooks/Webhook Test Note.md"
// NOTE: No "id" field present
}
}
}
```
**Impact:** The event parser in this ADR's example code assumes `event_data["node"]["id"]` exists for all file events. This will fail for deletions.
**Update (2025-11-11):** Nextcloud maintainer clarified that `BeforeNodeDeletedEvent` should be used instead of `NodeDeletedEvent` to access `node.id` before the file is deleted. See [issue #56371](https://github.com/nextcloud/server/issues/56371#issuecomment-2470896634).
> "Try using the `BeforeNodeDeletedEvent`. The `id` should still be available at that time. The reason `id` is not in `NodeDeletedEvent` is because the file is effectively guaranteed to be gone and, in turn, so is the FileInfo."
> — Josh Richards, Nextcloud maintainer
**Recommended Solution:** Use `OCP\Files\Events\Node\BeforeNodeDeletedEvent` for file deletion webhooks instead of `NodeDeletedEvent`.
**Alternative Fix (if using NodeDeletedEvent):** Check for `id` existence and fall back to path-based identification:
```python
def extract_document_task(event_class: str, payload: dict) -> DocumentTask | None:
user_id = payload["user"]["uid"]
event_data = payload["event"]
# File deletion events - NO node.id field
if "NodeDeletedEvent" in event_class:
path = event_data["node"]["path"]
if not path.endswith(".md"):
return None
# Use path-based ID since node.id is unavailable
return DocumentTask(
user_id=user_id,
doc_id=f"path:{path}", # Prefix to distinguish from numeric IDs
doc_type="note",
operation="delete",
modified_at=payload["time"],
)
# File creation/update events - node.id exists
elif "NodeCreatedEvent" in event_class or "NodeWrittenEvent" in event_class:
path = event_data["node"]["path"]
if not path.endswith(".md"):
return None
# Check if 'id' exists (should, but be defensive)
node_id = event_data["node"].get("id")
if not node_id:
# Fallback for missing ID
node_id = f"path:{path}"
return DocumentTask(
user_id=user_id,
doc_id=str(node_id),
doc_type="note",
operation="index",
modified_at=payload["time"],
)
```
**Qdrant Deletion Strategy:** When deleting by path-based ID, search Qdrant for documents with matching path metadata:
```python
async def delete_document_by_path(user_id: str, path: str):
"""Delete document from Qdrant using path (when ID unavailable)."""
points = await qdrant.scroll(
collection_name=collection,
scroll_filter=Filter(must=[
FieldCondition(key="user_id", match=MatchValue(value=user_id)),
FieldCondition(key="metadata.path", match=MatchValue(value=path)),
]),
)
# Delete found points...
```
#### 2. Multiple Webhooks Per Operation
**Finding:** Creating a single note triggers 3-5 separate webhook events in rapid succession:
1. `NodeCreatedEvent` for parent folder (if new)
2. `NodeWrittenEvent` for parent folder
3. `NodeCreatedEvent` for the note file
4. `NodeWrittenEvent` for the note file (sometimes fires twice)
**Impact:** Without deduplication, the processor will fetch and index the same note multiple times within seconds, wasting compute and API quota.
**Solution:** The processor queue should be idempotent. If the same document is queued multiple times, only the latest version needs processing. Implementation options:
1. **Queue-level deduplication:** Before adding to queue, check if a task for the same `(user_id, doc_id)` is already pending. Replace the existing task instead of adding duplicate.
2. **Processor-level deduplication:** Track recently processed documents in a short-lived cache (5 minutes). If a document was just processed, skip redundant fetch unless the `modified_at` timestamp is newer.
3. **Accept duplicates:** Let the processor handle duplicates naturally. Qdrant upserts are idempotent—reindexing with identical content is harmless but wasteful.
**Recommendation:** Implement queue-level deduplication by maintaining a map of pending tasks and replacing duplicates with newer timestamps.
#### 3. Type Discrepancy in `node.id`
**Finding:** Nextcloud documentation specifies `node.id` as type `string`, but actual payloads return `int`:
```json
"node": {
"id": 437, // integer, not "437"
"path": "/admin/files/Notes/Webhooks/Webhook Test Note.md"
}
```
**Impact:** Code that assumes `node.id` is always a string will work but may cause type confusion in strongly-typed languages.
**Solution:** Explicitly convert to string when extracting: `doc_id=str(event_data["node"]["id"])`
#### 4. Calendar Events Have Different ID Field Path
**Finding:** Calendar events store the document ID in a different location than file events:
- **File events:** `event.node.id`
- **Calendar events:** `event.objectData.id`
**Impact:** Event parser must handle different field paths for different event types. The example code in this ADR correctly shows this difference.
**Calendar Event Deletion:** Calendar deletion webhooks did NOT fire during testing. This may be a Nextcloud bug or require specific configuration (e.g., trash bin enabled). Until resolved, calendar deletions will only be detected via periodic scanner runs.
#### 5. Rich Metadata in Calendar Webhooks
**Finding:** Calendar webhook payloads include extensive metadata not present in file webhooks:
```json
{
"event": {
"calendarId": 1,
"calendarData": {
"id": 1,
"uri": "personal",
"{http://calendarserver.org/ns/}getctag": "...",
"{http://sabredav.org/ns}sync-token": 21,
// ... many calendar-level properties
},
"objectData": {
"id": 3,
"uri": "webhook-test-event-001.ics",
"lastmodified": 1762851169,
"etag": "\"2b937b7d77dc83c77329dfdb210ba9d0\"",
"calendarid": 1,
"size": 297,
"component": "vevent",
"classification": 0,
"uid": "webhook-test-event-001@nextcloud",
"calendardata": "BEGIN:VCALENDAR\r\nVERSION:2.0\r\n...", // Full iCal
"{http://nextcloud.com/ns}deleted-at": null
},
"shares": [] // Array of sharing info
}
}
```
**Opportunity:** The full iCal content is available in `objectData.calendardata`. The processor could extract metadata directly from the webhook payload instead of making an additional CalDAV request, reducing API load.
### Updated Event Mapping
Based on testing, the actual webhook behavior:
| Nextcloud Event | Fires? | `node.id`/`objectData.id` Present? | Notes |
|----------------|--------|-------------------------------------|-------|
| `NodeCreatedEvent` | ✅ Yes | ✅ Yes (`int`) | Fires for folders too |
| `NodeWrittenEvent` | ✅ Yes | ✅ Yes (`int`) | Fires 1-2x per operation |
| `NodeDeletedEvent` | ✅ Yes | ❌ **NO** (only `path`) | Critical difference |
| `CalendarObjectCreatedEvent` | ✅ Yes | ✅ Yes (`objectData.id`) | Full iCal included |
| `CalendarObjectUpdatedEvent` | ✅ Yes | ✅ Yes (`objectData.id`) | Full iCal included |
| `CalendarObjectDeletedEvent` | ❌ **DID NOT FIRE** | ❓ Unknown | Possible Nextcloud bug |
### Recommended Implementation Changes
The webhook handler code in this ADR requires these modifications:
1. **Handle missing `node.id` in deletions** (see code example in Finding #1)
2. **Add deduplication logic** to prevent redundant processing from multiple webhooks per operation
3. **Validate field existence** before accessing nested properties (`get()` with defaults)
4. **Log unsupported events** at DEBUG level (not WARNING) to avoid log noise
5. **Add calendar deletion fallback:** Since webhook unreliable, calendar deletions rely on scanner reconciliation
6. **Consider payload optimization:** Extract calendar metadata from webhook payload to reduce CalDAV API calls
### Testing Implications
**Integration Test Strategy:**
The asynchronous nature of Nextcloud webhooks makes real webhook delivery unreliable for automated tests:
-**DO:** POST webhook payloads directly to `/webhooks/nextcloud` endpoint in tests
-**DON'T:** Trigger Nextcloud events and wait for webhook delivery
-**DO:** Test authentication, payload parsing, and queue integration with mocked payloads
-**DON'T:** Assume webhooks fire immediately or reliably
**Manual Testing Required:**
- Real webhook delivery latency (depends on background job workers)
- Calendar deletion webhook behavior (confirm bug or configuration issue)
- Behavior under high-frequency updates (bulk operations)
- Network failure handling (Nextcloud can't reach MCP server)
### Complete Tested Payload Examples
See `webhook-testing-findings.md` in the repository root for:
- Complete JSON payloads for all tested events
- Detailed schema validation results
- Additional edge cases and observations
- Screenshots of webhook logs
## References
- ADR-007: Background Vector Database Synchronization (polling architecture)
- Nextcloud Documentation: `~/Software/documentation/admin_manual/webhook_listeners/index.rst`
- Nextcloud OCS API: Webhook registration endpoint
- Current scanner implementation: `nextcloud_mcp_server/vector/scanner.py:37`
- Webhook Testing Report: `webhook-testing-findings.md` (2025-01-11)
+159
View File
@@ -178,6 +178,111 @@ VECTOR_SYNC_ENABLED=true
- Requires separate Qdrant service
- More complex deployment
### Qdrant Collection Naming
Collection names are automatically generated to include the embedding model, ensuring safe model switching and preventing dimension mismatches.
#### Auto-Generated Naming (Default)
**Format:** `{deployment-id}-{model-name}`
**Components:**
- **Deployment ID:** `OTEL_SERVICE_NAME` (if configured) or `hostname` (fallback)
- **Model name:** `OLLAMA_EMBEDDING_MODEL`
**Examples:**
```bash
# With OTEL service name configured
OTEL_SERVICE_NAME=my-mcp-server
OLLAMA_EMBEDDING_MODEL=nomic-embed-text
# → Collection: "my-mcp-server-nomic-embed-text"
# Simple Docker deployment (OTEL not configured)
# hostname=mcp-container
OLLAMA_EMBEDDING_MODEL=all-minilm
# → Collection: "mcp-container-all-minilm"
```
#### Switching Embedding Models
When you change `OLLAMA_EMBEDDING_MODEL`, a new collection is automatically created:
```bash
# Initial setup
OLLAMA_EMBEDDING_MODEL=nomic-embed-text
# Collection: "my-server-nomic-embed-text" (768 dimensions)
# Change model
OLLAMA_EMBEDDING_MODEL=all-minilm
# Collection: "my-server-all-minilm" (384 dimensions)
# → New collection created, full re-embedding occurs
```
**Important:**
- **Collections are mutually exclusive** - vectors cannot be shared between different embedding models
- **Switching models requires re-embedding** all documents (may take time for large note collections)
- **Old collection remains** in Qdrant and can be deleted manually if no longer needed
#### Explicit Override
Set `QDRANT_COLLECTION` to use a specific collection name:
```bash
QDRANT_COLLECTION=my-custom-collection # Bypasses auto-generation
```
**Use cases:**
- Backward compatibility with existing deployments
- Custom naming schemes
- Sharing a collection across deployments (advanced)
#### Multi-Server Deployments
Each server should have a unique deployment ID to avoid collection collisions:
```bash
# Server 1 (Production)
OTEL_SERVICE_NAME=mcp-prod
OLLAMA_EMBEDDING_MODEL=nomic-embed-text
# → Collection: "mcp-prod-nomic-embed-text"
# Server 2 (Staging)
OTEL_SERVICE_NAME=mcp-staging
OLLAMA_EMBEDDING_MODEL=nomic-embed-text
# → Collection: "mcp-staging-nomic-embed-text"
# Server 3 (Different model)
OTEL_SERVICE_NAME=mcp-experimental
OLLAMA_EMBEDDING_MODEL=bge-large
# → Collection: "mcp-experimental-bge-large"
```
**Benefits:**
- Multiple MCP servers can share one Qdrant instance safely
- No naming collisions between deployments
- Clear collection ownership (can see which deployment and model)
#### Dimension Validation
The server validates collection dimensions on startup:
```
Dimension mismatch for collection 'my-server-nomic-embed-text':
Expected: 384 (from embedding model 'all-minilm')
Found: 768
This usually means you changed the embedding model.
Solutions:
1. Delete the old collection: Collection will be recreated with new dimensions
2. Set QDRANT_COLLECTION to use a different collection name
3. Revert OLLAMA_EMBEDDING_MODEL to the original model
```
**What this prevents:**
- Runtime errors from dimension mismatches
- Data corruption in Qdrant
- Confusing error messages during indexing
### Vector Sync Configuration
Control background indexing behavior:
@@ -188,6 +293,10 @@ VECTOR_SYNC_ENABLED=true # Enable background indexing
VECTOR_SYNC_SCAN_INTERVAL=300 # Scan interval in seconds (default: 5 minutes)
VECTOR_SYNC_PROCESSOR_WORKERS=3 # Concurrent indexing workers (default: 3)
VECTOR_SYNC_QUEUE_MAX_SIZE=10000 # Max queued documents (default: 10000)
# Document chunking settings (for vector embeddings)
DOCUMENT_CHUNK_SIZE=512 # Words per chunk (default: 512)
DOCUMENT_CHUNK_OVERLAP=50 # Overlapping words between chunks (default: 50)
```
### Embedding Service Configuration
@@ -208,6 +317,54 @@ OLLAMA_VERIFY_SSL=true # Verify SSL certificates
If `OLLAMA_BASE_URL` is not set, the server uses a simple random embedding provider for testing. This is **not suitable for production** as it generates random embeddings with no semantic meaning.
### Document Chunking Configuration
The server chunks documents before embedding to handle documents larger than the embedding model's context window. Chunk size and overlap can be tuned based on your embedding model and content type.
#### Choosing Chunk Size
**Smaller chunks (256-384 words)**:
- More precise matching
- Less context per chunk
- Better for finding specific information
- Higher storage requirements (more vectors)
**Larger chunks (768-1024 words)**:
- More context per chunk
- Less precise matching
- Better for understanding broader topics
- Lower storage requirements (fewer vectors)
**Default (512 words)**:
- Balanced approach suitable for most use cases
- Works well with typical note lengths
- Good compromise between precision and context
#### Choosing Overlap
Overlap preserves context across chunk boundaries. Recommended settings:
- **10-20% of chunk size** (e.g., 50-100 words for 512-word chunks)
- **Too small** (<10%): May lose context at boundaries
- **Too large** (>20%): Redundant storage, diminishing returns
**Examples**:
```dotenv
# Precise matching for short notes
DOCUMENT_CHUNK_SIZE=256
DOCUMENT_CHUNK_OVERLAP=25
# Default balanced configuration
DOCUMENT_CHUNK_SIZE=512
DOCUMENT_CHUNK_OVERLAP=50
# More context for long documents
DOCUMENT_CHUNK_SIZE=1024
DOCUMENT_CHUNK_OVERLAP=100
```
**Important**: Changing chunk size requires re-embedding all documents. The collection naming strategy (see "Qdrant Collection Naming" above) helps manage this by creating separate collections for different configurations.
### Environment Variables Reference
| Variable | Required | Default | Description |
@@ -223,6 +380,8 @@ If `OLLAMA_BASE_URL` is not set, the server uses a simple random embedding provi
| `OLLAMA_BASE_URL` | ⚠️ Optional | - | Ollama API endpoint for embeddings |
| `OLLAMA_EMBEDDING_MODEL` | ⚠️ Optional | `nomic-embed-text` | Embedding model to use |
| `OLLAMA_VERIFY_SSL` | ⚠️ Optional | `true` | Verify SSL certificates |
| `DOCUMENT_CHUNK_SIZE` | ⚠️ Optional | `512` | Words per chunk for document embedding |
| `DOCUMENT_CHUNK_OVERLAP` | ⚠️ Optional | `50` | Overlapping words between chunks (must be < chunk size) |
### Docker Compose Example
+258
View File
@@ -0,0 +1,258 @@
# Observability and Monitoring
The Nextcloud MCP Server includes comprehensive observability features for production deployments:
- **Prometheus metrics** for monitoring performance and health
- **OpenTelemetry distributed tracing** for debugging request flows
- **Structured JSON logging** with trace correlation
- **Kubernetes integration** via ServiceMonitor and PrometheusRule
## Quick Start
### Local Development with Prometheus
```bash
# Enable metrics (enabled by default)
export METRICS_ENABLED=true
export METRICS_PORT=9090
# Enable tracing (optional - tracing is enabled when OTEL_EXPORTER_OTLP_ENDPOINT is set)
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
# Start the server
docker-compose up -d mcp
```
Access metrics at: `http://localhost:9090/metrics`
### Kubernetes Deployment
Metrics are automatically scraped if you have Prometheus Operator installed:
```bash
helm install nextcloud-mcp charts/nextcloud-mcp-server \
--set observability.metrics.enabled=true \
--set observability.tracing.enabled=true \
--set observability.tracing.endpoint=http://opentelemetry-collector:4317 \
--set serviceMonitor.enabled=true
```
## Configuration
### Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `METRICS_ENABLED` | `true` | Enable Prometheus metrics |
| `METRICS_PORT` | `9090` | Port for metrics endpoint |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | - | OTLP gRPC endpoint (e.g., `http://otel-collector:4317`). Tracing is enabled when this is set. |
| `OTEL_SERVICE_NAME` | `nextcloud-mcp-server` | Service name in traces |
| `OTEL_TRACES_SAMPLER` | `always_on` | Trace sampling strategy |
| `OTEL_TRACES_SAMPLER_ARG` | `1.0` | Sampling rate (0.0-1.0) |
| `LOG_FORMAT` | `json` | Log format (`json` or `text`) |
| `LOG_LEVEL` | `INFO` | Minimum log level |
| `LOG_INCLUDE_TRACE_CONTEXT` | `true` | Include trace IDs in logs |
### Helm Chart Configuration
```yaml
observability:
metrics:
enabled: true
port: 9090
path: /metrics
tracing:
enabled: true
endpoint: "http://opentelemetry-collector:4317"
samplingRate: 1.0
logging:
format: json
level: INFO
includeTraceContext: true
serviceMonitor:
enabled: true
interval: 30s
scrapeTimeout: 10s
```
## Metrics
### HTTP Server Metrics (RED)
- `mcp_http_requests_total` - Total HTTP requests
- `mcp_http_request_duration_seconds` - Request latency histogram
- `mcp_http_requests_in_progress` - In-flight requests gauge
### MCP Tool Metrics
- `mcp_tool_calls_total` - Tool invocation count by status
- `mcp_tool_duration_seconds` - Tool execution latency
- `mcp_tool_errors_total` - Tool errors by type
### Nextcloud API Metrics
- `mcp_nextcloud_api_requests_total` - API calls by app and status
- `mcp_nextcloud_api_duration_seconds` - API latency by app
- `mcp_nextcloud_api_retries_total` - Retry count (429, timeout, etc.)
### OAuth Flow Metrics
- `mcp_oauth_token_validations_total` - Token validation count
- `mcp_oauth_token_exchange_total` - Token exchange operations
- `mcp_oauth_token_cache_hits_total` - Cache hit/miss rate
- `mcp_oauth_refresh_token_operations_total` - Refresh token storage ops
### Vector Sync Metrics (when enabled)
- `mcp_vector_sync_documents_scanned_total` - Documents discovered
- `mcp_vector_sync_documents_processed_total` - Processing results
- `mcp_vector_sync_processing_duration_seconds` - Processing latency
- `mcp_vector_sync_queue_size` - Current queue depth
- `mcp_qdrant_operations_total` - Qdrant DB operations
### Database Metrics
- `mcp_db_operations_total` - DB operations (SQLite, Qdrant)
- `mcp_db_operation_duration_seconds` - DB latency
### Dependency Health
- `mcp_dependency_health` - External dependency status (1=up, 0=down)
- `mcp_dependency_check_duration_seconds` - Health check latency
## Distributed Tracing
### Span Hierarchy
```
HTTP POST /messages
├── mcp.tool.nc_notes_create_note
│ └── nextcloud.api.notes.POST
│ └── httpx request (auto-instrumented)
└── oauth.token.validate (if OAuth mode)
└── httpx request to IdP
```
### Span Attributes
- **MCP tools**: `mcp.tool.name`, `mcp.tool.args` (sanitized)
- **Nextcloud API**: `nextcloud.app`, `http.method`, `http.status_code`
- **OAuth**: `oauth.operation`, `oauth.method`
- **Vector sync**: `vector_sync.operation`, `vector_sync.document_count`
### Trace Context in Logs
When tracing is enabled, all logs include `trace_id` and `span_id`:
```json
{
"timestamp": "2025-01-09T12:34:56.789Z",
"level": "INFO",
"logger": "nextcloud_mcp_server.server.notes",
"message": "Note created successfully",
"trace_id": "a1b2c3d4e5f6...",
"span_id": "123456789abc...",
"note_id": 42
}
```
## Dashboards
### Prometheus Queries
**Request Rate (req/s)**:
```promql
sum(rate(mcp_http_requests_total[5m])) by (method, endpoint)
```
**Error Rate (%)**:
```promql
sum(rate(mcp_http_requests_total{status_code=~"5.."}[5m]))
/ sum(rate(mcp_http_requests_total[5m])) * 100
```
**P95 Latency**:
```promql
histogram_quantile(0.95,
sum(rate(mcp_http_request_duration_seconds_bucket[5m])) by (le, endpoint)
)
```
**Top Tools by Volume**:
```promql
topk(10, sum(rate(mcp_tool_calls_total[5m])) by (tool_name))
```
**Nextcloud API Health**:
```promql
sum(rate(mcp_nextcloud_api_requests_total{status_code!~"2.."}[5m])) by (app)
```
## Alerts
### Recommended Alert Rules
**Critical**:
- Server down for >5min
- Error rate >5% for >5min
- P95 latency >1s for >5min
- Dependency down for >2min
**Warning**:
- Token validation errors >1% for >10min
- Vector sync queue >100 for >15min
- Qdrant slow (p95 >500ms) for >10min
See `charts/nextcloud-mcp-server/templates/prometheusrule.yaml` for complete definitions.
## Troubleshooting
### Metrics Not Appearing
1. Check metrics are enabled: `curl http://localhost:9090/metrics`
2. Verify ServiceMonitor labels match Prometheus selector
3. Check Prometheus target status: `http://prometheus:9090/targets`
### Traces Not Appearing
1. Verify OTLP endpoint is reachable: `curl http://otel-collector:4317`
2. Check collector logs for errors
3. Verify sampling rate is not 0.0
4. Check trace backend (Jaeger/Tempo) connectivity
### High Cardinality Metrics
If you see cardinality warnings:
- Middleware normalizes endpoints (e.g., `/user/123``/user/*`)
- OAuth tokens are never included in metric labels
- User IDs are not tracked (use tracing for per-user debugging)
## Performance Impact
- **Metrics**: <1% overhead (counters/histograms are very fast)
- **Tracing**: ~2-5% overhead at 100% sampling
- **JSON logging**: <1% overhead vs text logging
**Recommendation**: Always enable metrics. Enable tracing in staging/production with 10-50% sampling.
## Architecture
The observability stack integrates at multiple layers:
1. **HTTP Layer**: `ObservabilityMiddleware` tracks all HTTP requests
2. **MCP Layer**: Tools use `@trace_mcp_tool` for span creation
3. **Client Layer**: `BaseNextcloudClient` tracks all API calls
4. **OAuth Layer**: Token operations are traced and metered
5. **Background Tasks**: Vector sync operations emit metrics/traces
All components use shared Prometheus `Registry` and OpenTelemetry `TracerProvider`.
## References
- [Prometheus Best Practices](https://prometheus.io/docs/practices/)
- [OpenTelemetry Python SDK](https://opentelemetry.io/docs/languages/python/)
- [Prometheus Operator](https://prometheus-operator.dev/)
- [Grafana Dashboards](https://grafana.com/docs/grafana/latest/dashboards/)
+921
View File
@@ -0,0 +1,921 @@
# Semantic Search Architecture
This document explains the architecture of the semantic search feature in the Nextcloud MCP Server, including background synchronization, vector search, and optional AI-generated answers via MCP sampling.
> [!IMPORTANT]
> **Status: Experimental**
> - Disabled by default (`VECTOR_SYNC_ENABLED=false`)
> - Currently supports **Notes app only** (multi-app architecture ready, additional apps planned)
> - Requires additional infrastructure (Qdrant vector database + Ollama embedding service)
> - RAG answer generation requires MCP client sampling support
## Overview
### What is Semantic Search?
**Semantic search** finds information based on **meaning** rather than exact keyword matches. It uses vector embeddings to understand that "car" and "automobile" are similar, or that "bread recipe" matches "how to bake bread."
**Traditional keyword search:**
```
Query: "machine learning"
Matches: Only notes containing "machine learning" exactly
Misses: Notes with "neural networks", "AI models", "deep learning"
```
**Semantic search:**
```
Query: "machine learning"
Matches: Notes about machine learning, neural networks, AI, deep learning, etc.
Understanding: Semantic similarity via vector embeddings
```
### Why It Matters
Semantic search enables:
- **Natural language queries** - Ask questions in plain language
- **Conceptual discovery** - Find related content even with different terminology
- **Cross-reference insights** - Connect ideas across your knowledge base
- **AI-powered answers** - Generate summaries with citations (optional, requires MCP sampling)
### Current Support
- **Supported Apps**: Notes (fully implemented)
- **Planned Apps**: Calendar events, Calendar tasks, Deck cards, Files (with text extraction), Contacts
- **Architecture**: Multi-app plugin system ready, awaiting implementation
## System Components
```mermaid
graph TB
subgraph "MCP Client"
Client[Claude Desktop, IDEs, etc.]
end
subgraph "Nextcloud MCP Server"
MCP[MCP Server]
Scanner[Background Scanner<br/>Hourly Change Detection]
Queue[Document Queue]
Processor[Embedding Processors<br/>Concurrent Workers]
end
subgraph "Infrastructure"
Qdrant[(Qdrant<br/>Vector Database)]
Ollama[Ollama<br/>Embedding Service]
NC[Nextcloud<br/>Notes API, CalDAV, etc.]
end
Client <-->|MCP Protocol| MCP
Scanner -->|Fetch Changes| NC
Scanner -->|Enqueue Documents| Queue
Queue -->|Process Batch| Processor
Processor -->|Generate Embeddings| Ollama
Processor -->|Store Vectors| Qdrant
MCP -->|Search Queries| Qdrant
MCP -->|Verify Access| NC
```
**Component Roles:**
- **MCP Server**: Exposes semantic search tools (`nc_semantic_search`, `nc_semantic_search_answer`, `nc_get_vector_sync_status`)
- **Background Scanner**: Discovers changed documents every hour using ETag-based change detection
- **Document Queue**: Holds pending documents for embedding generation
- **Embedding Processors**: Generate vector embeddings via Ollama (concurrent workers)
- **Qdrant Vector Database**: Stores document vectors with metadata and user_id filtering
- **Ollama Embedding Service**: Converts text to 768-dimensional vectors (default: `nomic-embed-text` model)
- **Nextcloud APIs**: Source of truth for documents and access control verification
## How It Works: Background Synchronization
Background synchronization runs automatically when `VECTOR_SYNC_ENABLED=true`, discovering changes and indexing documents without user intervention.
```mermaid
sequenceDiagram
participant Timer
participant Scanner
participant NC as Nextcloud API
participant Queue
participant Processor
participant Ollama
participant Qdrant
Timer->>Scanner: Trigger (hourly)
Scanner->>NC: Fetch all notes<br/>(Notes API)
NC-->>Scanner: Notes with ETags
Scanner->>Qdrant: Check indexed documents
Qdrant-->>Scanner: Existing ETags
Scanner->>Scanner: Identify changes<br/>(new/modified/deleted)
Scanner->>Queue: Enqueue changed docs
loop Continuous Processing
Processor->>Queue: Fetch batch
Queue-->>Processor: Documents
Processor->>Ollama: Generate embeddings
Ollama-->>Processor: 768-dim vectors
Processor->>Qdrant: Upsert vectors<br/>(with user_id, doc_type)
end
```
### Scanner Behavior
**Hourly Trigger:**
- Runs every hour (configurable)
- Fetches all notes from Nextcloud Notes API
- Compares ETags with Qdrant's indexed state
- Enqueues new/modified documents
**Change Detection:**
- **New documents**: No entry in Qdrant → enqueue for indexing
- **Modified documents**: ETag mismatch → enqueue for re-indexing
- **Deleted documents**: In Qdrant but not in Nextcloud → delete from Qdrant
**Multi-App Plugin Architecture:**
```python
# Each app implements DocumentScanner interface
class NotesScanner(DocumentScanner):
async def scan(self) -> list[Document]:
# Fetch notes, detect changes, return documents
```
Currently only `NotesScanner` is implemented. Future: `CalendarScanner`, `DeckScanner`, `FilesScanner`, etc.
### Queue Processing
**Document Queue:**
- In-memory FIFO queue (not persistent across restarts)
- Holds documents pending embedding generation
- Batch processing for efficiency
**Processor Pool:**
- Concurrent workers using `anyio.TaskGroup`
- Process documents in parallel (default: 4 workers)
- Each worker: fetch document → generate embedding → store in Qdrant
**Backpressure Handling:**
- Queue size limits prevent memory exhaustion
- Slow consumers (Ollama) naturally pace the system
### Vector Storage
**Qdrant Collection Schema:**
```
{
"id": "note_123",
"vector": [768 dimensions],
"payload": {
"user_id": "alice",
"doc_type": "note",
"doc_id": "123",
"title": "Machine Learning Notes",
"content": "Neural networks are...",
"etag": "abc123",
"last_modified": "2025-01-15T10:30:00Z"
}
}
```
**Key Fields:**
- `user_id`: Multi-tenancy filtering (each user's vectors isolated)
- `doc_type`: App identifier ("note", "event", "card", etc.)
- `etag`: Change detection for incremental updates
- `chunk_index`: Position of this chunk within the document (0-indexed)
- `total_chunks`: Total number of chunks for this document
- `excerpt`: First 200 characters of chunk (for display)
### Document Chunking Strategy
Documents are chunked before embedding to handle content larger than the embedding model's context window and to improve search precision.
**Configuration:**
```dotenv
DOCUMENT_CHUNK_SIZE=512 # Words per chunk (default)
DOCUMENT_CHUNK_OVERLAP=50 # Overlapping words between chunks (default)
```
**Chunking Process:**
1. **Text combination**: Document title + content (e.g., `"Note Title\n\nNote content..."`)
2. **Word-based splitting**: Simple whitespace tokenization
3. **Sliding window**: Create overlapping chunks
4. **Individual embedding**: Each chunk gets its own vector
5. **Separate storage**: Each chunk stored as distinct point in Qdrant
**Example:**
```
Document (1000 words):
→ Chunk 0: words 0-511
→ Chunk 1: words 462-973 (overlaps by 50 words)
→ Chunk 2: words 924-999 (last chunk, partial)
Each chunk stored as separate vector with metadata:
- chunk_index: 0, 1, 2
- total_chunks: 3
- excerpt: First 200 chars of each chunk
```
**Search Behavior:**
- **Vector search** operates on chunks (not whole documents)
- **Deduplication** collapses multiple matching chunks from same document
- **Best match** returns highest-scoring chunk's excerpt
- **Access verification** still performed at document level
**Tuning Recommendations:**
- **Small chunks (256-384 words)**: More precise, less context, more storage
- **Large chunks (768-1024 words)**: More context, less precise, less storage
- **Overlap (10-20% of chunk size)**: Preserves context across boundaries
- **Match to embedding model**: Consider model's context window when sizing
**Important**: Changing chunk size requires re-embedding all documents. Use the collection naming strategy to manage different chunking configurations.
### Collection Naming and Model Switching
**Auto-generated collection names:**
- **Format:** `{deployment-id}-{model-name}`
- **Deployment ID:** `OTEL_SERVICE_NAME` (if configured) or `hostname` (fallback)
- **Model name:** `OLLAMA_EMBEDDING_MODEL`
- **Example:** `"my-mcp-server-nomic-embed-text"`, `"mcp-container-all-minilm"`
**Why model-based naming:**
- Ensures each embedding model gets its own collection
- Prevents dimension mismatches when switching models
- Enables safe model experimentation (new model = new collection)
- Supports multi-server deployments (different deployment IDs)
**Switching embedding models:**
Collections are **mutually exclusive** - vectors from one embedding model cannot be used with another. When you change the embedding model:
1. **New collection is created** with the new model's dimensions
2. **Full re-embedding occurs** - scanner processes all documents again
3. **Old collection remains** - can be deleted manually if no longer needed
4. **Dimension validation** - server fails fast if collection dimension doesn't match model
**Example workflow:**
```bash
# Start with nomic-embed-text (768 dimensions)
OLLAMA_EMBEDDING_MODEL=nomic-embed-text
# Collection: "my-server-nomic-embed-text"
# → Scanner indexes 1000 notes → 1000 vectors in collection
# Switch to all-minilm (384 dimensions)
OLLAMA_EMBEDDING_MODEL=all-minilm
# Collection: "my-server-all-minilm"
# → Scanner detects 0 indexed documents → re-embeds 1000 notes
# → Old collection "my-server-nomic-embed-text" still exists in Qdrant
```
**Re-embedding performance:**
- CPU-only: 1-5 notes/second
- With GPU: 50-200 notes/second
- 1000 notes: 3-16 minutes (CPU) or 5-20 seconds (GPU)
**Multi-server deployments:**
Multiple MCP servers can share one Qdrant instance safely:
```bash
# Server 1 (Production)
OTEL_SERVICE_NAME=mcp-prod
OLLAMA_EMBEDDING_MODEL=nomic-embed-text
# → Collection: "mcp-prod-nomic-embed-text"
# Server 2 (Staging with different model)
OTEL_SERVICE_NAME=mcp-staging
OLLAMA_EMBEDDING_MODEL=all-minilm
# → Collection: "mcp-staging-all-minilm"
```
Each deployment gets its own collection - no naming collisions or dimension conflicts.
## How It Works: Semantic Search
Semantic search converts user queries into vectors and finds similar documents using cosine similarity.
```mermaid
sequenceDiagram
participant User
participant MCP as MCP Server
participant Ollama
participant Qdrant
participant NC as Nextcloud API
User->>MCP: nc_semantic_search("machine learning")
MCP->>MCP: Check OAuth scope<br/>(semantic:read)
MCP->>Ollama: Generate query embedding
Ollama-->>MCP: Query vector (768-dim)
MCP->>Qdrant: Search similar vectors<br/>(filter: user_id=alice)
Qdrant-->>MCP: Top K results<br/>(with similarity scores)
loop For each result
MCP->>NC: Verify access<br/>(fetch note by ID)
alt Access granted
NC-->>MCP: Note metadata
else Access denied (404/401)
MCP->>MCP: Filter out result
end
end
MCP-->>User: Search results<br/>(with scores, excerpts)
```
### Dual-Phase Authorization
**Phase 1: OAuth Scope Check**
- Verify user has `semantic:read` scope
- Rejects unauthorized users immediately
**Phase 2: Per-Document Verification**
- For each search result, fetch document via app API (Notes, Calendar, etc.)
- If fetch succeeds (200 OK), user has access
- If fetch fails (404 Not Found, 401 Unauthorized), filter out result
- **Security**: Prevents information leakage from vector search alone
**Rationale:**
- Vector database doesn't know about sharing, permissions changes, or deleted documents
- App APIs are source of truth for access control
- Verification ensures users only see documents they can access
### Search Flow
1. **Query Embedding**: Convert user query to 768-dimensional vector via Ollama
2. **Vector Search**: Find top K similar vectors in Qdrant (cosine similarity)
3. **User Filtering**: Qdrant pre-filters by `user_id` (multi-tenancy)
4. **Access Verification**: Fetch each document via app API to verify current access
5. **Result Ranking**: Return results sorted by similarity score
6. **Response**: Include document excerpts, metadata, and similarity scores
### Performance
- **Query latency**: 50-200ms typical (embedding + vector search + verification)
- **Accuracy**: Depends on embedding model quality (`nomic-embed-text` recommended)
- **Scalability**: Qdrant handles millions of vectors efficiently
## How It Works: RAG with MCP Sampling (Optional)
The `nc_semantic_search_answer` tool generates AI-powered answers with citations using **MCP sampling** - requesting the MCP client's LLM to generate text.
```mermaid
sequenceDiagram
participant User
participant MCP as MCP Server
participant Client as MCP Client<br/>(Claude Desktop)
participant LLM as Client's LLM<br/>(Claude, GPT, etc.)
User->>MCP: nc_semantic_search_answer("What are my Q1 goals?")
MCP->>MCP: Semantic search<br/>(find relevant notes)
MCP->>MCP: Construct prompt<br/>(query + documents + instructions)
MCP->>Client: Sampling request<br/>(MCP Protocol)
Client->>User: Prompt for approval<br/>(optional, client-controlled)
User-->>Client: Approve
Client->>LLM: Generate answer<br/>(with context)
LLM-->>Client: Answer with citations
Client-->>MCP: Sampling response
MCP-->>User: Generated answer<br/>(with source documents)
```
### MCP Sampling Architecture
**Why MCP Sampling?**
- **No server-side LLM**: MCP server has no API keys, doesn't call LLMs directly
- **Client controls everything**: Which model, who pays, user approval prompts
- **Privacy**: Documents stay with the client's LLM provider, not a third-party
- **Flexibility**: Works with any MCP client that supports sampling (Claude Desktop, future clients)
**Prompt Construction:**
```
User Query: {query}
Relevant Documents:
1. Document: {title} (Note)
Content: {excerpt}
2. Document: {title} (Note)
Content: {excerpt}
Instructions:
- Provide a comprehensive answer to the user's query
- Use the documents above as context
- Include citations: "According to Document 1 (title)..."
- If documents don't contain enough information, say so
```
**Graceful Fallback:**
```python
try:
result = await ctx.session.create_message(...)
return answer_with_citations
except Exception as e:
# Fallback: Return documents without generated answer
return SearchResponse(
generated_answer=f"[Sampling unavailable: {e}]",
sources=search_results
)
```
**Client Support:**
- **Requires**: MCP client with sampling capability
- **Known support**: Claude Desktop (as of Claude 3.5+)
- **Graceful degradation**: Returns raw documents if sampling unavailable
## Authentication & Security
### OAuth Scopes
**`semantic:read`** - Search permission
- Allows using `nc_semantic_search` and `nc_semantic_search_answer` tools
- Does NOT grant access to documents (verified via app APIs)
- Required for any semantic search operation
**`semantic:write`** - Sync control permission
- Allows enabling/disabling background sync (`provision_vector_sync`, `deprovision_vector_sync`)
- Controls whether user's documents are indexed
- Currently not implemented in OAuth mode (BasicAuth only)
### Dual-Phase Authorization Pattern
**Phase 1: Scope Check** (semantic:read)
- Verifies user authorized to search
- Prevents unauthorized vector database access
**Phase 2: Document Verification** (app-specific APIs)
- For each search result, fetch via Notes API, CalDAV, etc.
- If user can fetch → include in results
- If user cannot fetch (404/401) → filter out
- **Security**: Vector search cannot leak documents user shouldn't see
**Example Scenario:**
1. Alice creates note "Secret Project X"
2. Background sync indexes note with `user_id=alice`
3. Bob searches for "project"
4. Vector search finds "Secret Project X" (vector similarity)
5. Qdrant filters by `user_id=bob` → no match (Alice's note excluded)
6. Even if Bob somehow got the doc_id, Phase 2 verification would fail (404 Not Found)
### Offline Access for Background Sync
**Why needed:**
- Background scanner runs hourly without user interaction
- Requires valid access tokens to fetch documents from Nextcloud APIs
- User's session token expires after hours/days
**OAuth Mode (ADR-004 Flow 2):**
- User explicitly provisions offline access via `provision_nextcloud_access` tool
- Server requests `offline_access` scope → receives refresh token
- Refresh token stored securely (database, encrypted)
- Background sync uses refresh tokens to obtain access tokens
**BasicAuth Mode:**
- Username/password stored in environment variables
- Always available for background operations
- Simpler but less secure (credentials never expire)
## Deployment Modes
### Authentication Modes
| Mode | Security | Offline Access | Background Sync | Best For |
|------|----------|----------------|-----------------|----------|
| **BasicAuth** | Lower (credentials in env) | Always available | ✅ Works immediately | Single-user, development, testing |
| **OAuth** | Higher (tokens, scopes) | User must provision | ⚠️ Not yet implemented | Multi-user, production |
**BasicAuth:**
- Set `NEXTCLOUD_USERNAME` and `NEXTCLOUD_PASSWORD`
- Background sync works immediately when `VECTOR_SYNC_ENABLED=true`
- Credentials stored in `.env` file (secure server access required)
**OAuth:**
- Client authenticates with `semantic:read` scope
- User must explicitly provision offline access (future: `provision_vector_sync` tool)
- Background sync only works for users who provisioned access
- More secure: tokens expire, user controls access
### Qdrant Deployment Modes
| Mode | Configuration | Persistence | Scalability | Best For |
|------|---------------|-------------|-------------|----------|
| **In-Memory** (default) | `QDRANT_LOCATION=:memory:` | ❌ Lost on restart | Single instance | Testing, development |
| **Persistent Local** | `QDRANT_LOCATION=/data/qdrant` | ✅ Survives restarts | Single instance | Small deployments |
| **Network** | `QDRANT_URL=http://qdrant:6333` | ✅ Dedicated service | ✅ Horizontal scaling | Production |
**In-Memory Mode:**
```bash
VECTOR_SYNC_ENABLED=true
# QDRANT_LOCATION not set → defaults to :memory:
```
- Fastest startup
- No disk I/O
- **Warning**: All vectors lost when server restarts (must re-index)
**Persistent Local Mode:**
```bash
VECTOR_SYNC_ENABLED=true
QDRANT_LOCATION=/var/lib/qdrant
```
- Vectors survive restarts
- Single server only (no distributed setup)
- Disk I/O for durability
**Network Mode (Recommended for Production):**
```bash
VECTOR_SYNC_ENABLED=true
QDRANT_URL=http://qdrant:6333
QDRANT_API_KEY=secret # optional
```
- Dedicated Qdrant service (Docker, Kubernetes)
- Horizontal scaling (multiple MCP servers → one Qdrant)
- High availability options
### Embedding Service Options
| Service | Configuration | Cost | Performance | Best For |
|---------|---------------|------|-------------|----------|
| **Ollama** (recommended) | `OLLAMA_BASE_URL=http://ollama:11434` | Free (self-hosted) | Fast (local GPU) | Production, development |
| **OpenAI** (future) | `OPENAI_API_KEY=sk-...` | Paid (API) | Fast (cloud) | Cloud deployments |
| **Fallback** | No config | Free | Slow (random) | Testing only (not production) |
**Ollama Setup (Recommended):**
```bash
# docker-compose.yml
services:
ollama:
image: ollama/ollama
volumes:
- ollama-data:/root/.ollama
ports:
- "11434:11434"
# Pull embedding model
docker compose exec ollama ollama pull nomic-embed-text
```
**Environment Configuration:**
```bash
OLLAMA_BASE_URL=http://ollama:11434
OLLAMA_EMBEDDING_MODEL=nomic-embed-text # 768-dimensional vectors
```
**Model Options:**
- `nomic-embed-text` (default): 768-dim, optimized for semantic search
- `all-minilm`: Smaller, faster, slightly less accurate
- `mxbai-embed-large`: Larger, more accurate, slower
## Configuration Overview
### Key Environment Variables
**Enable Semantic Search:**
```bash
VECTOR_SYNC_ENABLED=true # Default: false (opt-in)
```
**Qdrant Vector Database:**
```bash
# In-memory mode (default if VECTOR_SYNC_ENABLED=true)
# QDRANT_LOCATION not set → uses :memory:
# Persistent local mode
QDRANT_LOCATION=/var/lib/qdrant
# Network mode (production)
QDRANT_URL=http://qdrant:6333
QDRANT_API_KEY=secret # optional
```
**Ollama Embedding Service:**
```bash
OLLAMA_BASE_URL=http://ollama:11434
OLLAMA_EMBEDDING_MODEL=nomic-embed-text # Default
```
**Scanner Configuration:**
```bash
VECTOR_SYNC_INTERVAL=3600 # Scan interval in seconds (default: 1 hour)
```
### Resource Requirements
**Qdrant:**
- **Memory**: ~100-200 MB base + ~1 KB per vector (1M vectors ≈ 1 GB)
- **Disk**: Persistent mode only, ~200 bytes per vector
- **CPU**: Low (indexing) to moderate (search)
**Ollama:**
- **Memory**: 2-4 GB for `nomic-embed-text` model
- **CPU**: High during embedding generation, idle otherwise
- **GPU**: Optional but recommended (10-100x faster)
**MCP Server:**
- **Memory**: +50-100 MB for background sync workers
- **CPU**: Moderate during scanning/processing, low otherwise
### Trade-offs
| Consideration | In-Memory Qdrant | Persistent Qdrant | Network Qdrant |
|---------------|------------------|-------------------|----------------|
| Setup complexity | ✅ Minimal | ✅ Easy | ⚠️ Requires separate service |
| Durability | ❌ Lost on restart | ✅ Survives restarts | ✅ Survives restarts |
| Scalability | ❌ Single instance | ❌ Single instance | ✅ Horizontal scaling |
| Performance | ✅ Fastest | ✅ Fast | ⚠️ Network latency |
## Operational Behavior
### What Happens When VECTOR_SYNC_ENABLED=true
**Immediate (Server Startup):**
1. MCP server connects to Qdrant (creates collection if needed)
2. MCP server connects to Ollama (verifies embedding model available)
3. Background scanner starts (schedules hourly runs)
4. Document queue and processors initialize
**First Scan (Within 1 hour):**
1. Scanner fetches all notes from Nextcloud
2. Compares with Qdrant (likely empty on first run)
3. Enqueues all notes for indexing
4. Processors generate embeddings (may take minutes for large note collections)
5. Vectors stored in Qdrant with user_id filtering
**Hourly Thereafter:**
1. Scanner fetches all notes
2. Identifies new/modified/deleted notes (ETag comparison)
3. Enqueues changes only
4. Incremental updates processed
### Performance Expectations
**Embedding Generation:**
- **Without GPU**: 1-5 notes/second (CPU-bound)
- **With GPU**: 50-200 notes/second (highly parallel)
- **Initial indexing**: 100 notes ≈ 20-100 seconds (CPU), 1-2 seconds (GPU)
**Search Query:**
- **Embedding generation**: 50-100ms
- **Vector search**: 10-50ms (depends on collection size)
- **Access verification**: 20-100ms per document (Nextcloud API calls)
- **Total latency**: 100-300ms typical
**Resource Usage:**
- **Idle**: Minimal (background scanner sleeps)
- **Scanning**: Moderate CPU (ETag checks, API calls)
- **Processing**: High CPU/GPU (embedding generation)
- **Searching**: Low to moderate (depends on query frequency)
### Background Sync Behavior
**Scanner Triggers:**
- Hourly (configurable via `VECTOR_SYNC_INTERVAL`)
- Manual trigger via `nc_trigger_vector_sync` (future)
**Queue Processing:**
- Continuous (workers always running)
- Batch processing (fetch 10 documents at a time)
- Concurrent workers (4 by default)
**Error Handling:**
- Individual document failures logged but don't stop scanning
- Retries for transient errors (network timeouts, rate limits)
- Failed documents skipped, re-attempted on next scan
**What Gets Indexed:**
- **Notes**: All notes accessible to the authenticated user
- **Future**: Calendar events, tasks, deck cards, files with text extraction, contacts
## Monitoring & Observability
### MCP Tools
**`nc_get_vector_sync_status`** - Check sync status
```python
{
"total_documents": 1234,
"indexed_documents": 1200,
"pending_documents": 34,
"sync_enabled": true,
"last_scan": "2025-01-15T14:30:00Z",
"status": "syncing" # idle | syncing | error
}
```
**Interpreting Status:**
- `idle`: No pending work, last scan completed successfully
- `syncing`: Currently processing documents
- `error`: Last scan failed (check logs)
### Logs to Check
**Scanner Logs:**
```
[INFO] Vector sync scanner started (interval: 3600s)
[INFO] Scanning notes: found 150 documents
[INFO] Changes detected: 5 new, 2 modified, 1 deleted
[INFO] Enqueued 7 documents for processing
```
**Processor Logs:**
```
[INFO] Processing document: note_123
[DEBUG] Generated embedding (768 dimensions)
[INFO] Stored vector in Qdrant: note_123
```
**Error Logs:**
```
[ERROR] Failed to generate embedding for note_123: Connection timeout
[WARN] Qdrant connection lost, retrying...
[ERROR] Ollama embedding failed: Model not found
```
**Log Locations:**
- **Docker**: `docker compose logs mcp`
- **Local**: stdout (redirect to file if needed)
- **Kubernetes**: `kubectl logs -f deployment/nextcloud-mcp-server`
### Metrics to Monitor
**Indexing Progress:**
- Total documents vs indexed documents
- Pending queue size
- Processing rate (docs/second)
**Search Performance:**
- Query latency (p50, p95, p99)
- Results per query
- Verification overhead (API calls per query)
**Resource Usage:**
- Qdrant memory/disk usage
- Ollama CPU/GPU usage
- MCP server memory
For detailed observability setup, see [docs/observability.md](observability.md).
## Troubleshooting from Architecture Perspective
### Documents Not Appearing in Search
**Diagnosis Flow:**
1. Check sync status: `nc_get_vector_sync_status`
- `sync_enabled: false` → Enable with `VECTOR_SYNC_ENABLED=true`
- `status: error` → Check scanner logs for failures
2. Check queue size:
- `pending_documents > 0` → Processing in progress, wait
- `pending_documents == 0` but `indexed_documents` low → Scan hasn't run yet (wait up to 1 hour)
3. Check Qdrant:
- Connection errors in logs → Verify `QDRANT_URL` or `QDRANT_LOCATION`
- Collection empty → First scan hasn't completed
4. Check Ollama:
- Embedding errors in logs → Verify `OLLAMA_BASE_URL`
- Model not found → Pull model: `ollama pull nomic-embed-text`
**Common Causes:**
- Sync disabled (default): Enable `VECTOR_SYNC_ENABLED=true`
- Ollama not running: Start Ollama service
- Qdrant not accessible: Check network/URL
- First scan in progress: Wait up to 1 hour + processing time
### Slow Search Performance
**Diagnosis:**
1. **Query embedding slow (>500ms)**:
- Ollama overloaded or CPU-bound
- Solution: Use GPU, upgrade CPU, or reduce concurrent requests
2. **Vector search slow (>200ms)**:
- Large collection (millions of vectors)
- Solution: Use network Qdrant with SSDs, add indexing
3. **Verification slow (>500ms)**:
- Many results to verify (10+ documents)
- Nextcloud API slow or overloaded
- Solution: Reduce `limit` parameter, optimize Nextcloud
**Performance Tuning:**
- Reduce search `limit` (default: 10 results)
- Use network Qdrant for large collections
- Enable Ollama GPU acceleration
- Check Nextcloud API response times
### Background Sync Stopped
**Diagnosis:**
1. Check logs for errors:
- Authentication failures (401/403) → Token expired (OAuth) or credentials invalid (BasicAuth)
- Connection timeouts → Network issues with Nextcloud/Qdrant/Ollama
- Rate limiting (429) → Reduce scan frequency
2. Check `nc_get_vector_sync_status`:
- `status: error` → See logs for details
- `last_scan` timestamp old (>2 hours) → Scanner may have crashed
3. Verify services:
- Qdrant accessible: `curl http://qdrant:6333/`
- Ollama accessible: `curl http://ollama:11434/api/tags`
- Nextcloud accessible: Check API health
**OAuth Mode (Future):**
- Offline access token expired → Re-provision via `provision_vector_sync`
- User deprovisioned access → Sync stops intentionally
### Out of Memory
**Diagnosis:**
1. Check Qdrant mode:
- In-memory mode with large collection → Switch to persistent or network mode
2. Check embedding batch size:
- Too many documents processed simultaneously → Reduce worker count
3. Check Ollama memory:
- Large models loaded → Use smaller embedding model
**Solutions:**
- Use persistent or network Qdrant (frees server memory)
- Reduce concurrent processor workers
- Use smaller embedding model (`all-minilm` instead of `nomic-embed-text`)
- Increase server memory allocation
## Limitations & Future Work
### Current Limitations
1. **Notes App Only**
- Architecture supports multiple apps (plugin system ready)
- Only `NotesScanner` and `NotesProcessor` implemented
- Future: Calendar, Deck, Files, Contacts
2. **MCP Sampling Support**
- `nc_semantic_search_answer` requires client sampling capability
- Not all MCP clients support sampling yet
- Graceful fallback: Returns documents without generated answer
3. **OAuth Background Sync**
- User-controlled background jobs not yet implemented
- Currently works in BasicAuth mode only
- Future: Users opt-in via `provision_vector_sync` tool
4. **No Incremental Updates**
- Document changes trigger full re-embedding
- Cannot update just modified paragraphs
- Future: Paragraph-level chunking and incremental updates
5. **No Query Caching**
- Each search generates new query embedding
- Repeated queries re-search Qdrant
- Future: Cache recent query embeddings and results
6. **Single Embedding Model**
- Uses one model for all documents and queries
- Cannot customize per app or user
- Future: App-specific or user-selected models
### Future Enhancements
**Multi-App Support** (In Progress):
- Scanner plugins for Calendar, Deck, Files, Contacts
- Unified vector search across all apps
- App-specific metadata in vector payloads
**User-Controlled Sync (OAuth Mode)**:
- `provision_vector_sync` and `deprovision_vector_sync` tools
- Per-user background job scheduling
- User dashboard for sync status and controls
**Advanced Search Features**:
- Hybrid search (vector + keyword combined)
- Filtering by date range, app type, tags
- Aggregations and faceted search
- Search result explanations (why this matched)
**Performance Optimizations**:
- Query caching for repeated searches
- Incremental document updates (paragraph-level)
- Batch query processing
- Qdrant HNSW indexing tuning
**Embedding Improvements**:
- Support for OpenAI embeddings (ada-002, text-embedding-3)
- Multi-language embedding models
- Fine-tuned models for Nextcloud content
- Paragraph-level chunking for long documents
## References
### Architecture Decision Records (ADRs)
- **[ADR-003: Vector Database Semantic Search](ADR-003-vector-database-semantic-search.md)** - Qdrant selection rationale, embedding strategy, hybrid search (superseded by ADR-007 but technical decisions remain valid)
- **[ADR-007: Background Vector Sync Job Management](ADR-007-background-vector-sync-job-management.md)** - Current implementation, Scanner-Queue-Processor architecture, plugin system
- **[ADR-008: MCP Sampling for Semantic Search](ADR-008-mcp-sampling-for-semantic-search.md)** - RAG with MCP sampling, client-server separation, prompt construction
- **[ADR-009: Semantic Search OAuth Scope](ADR-009-semantic-search-oauth-scope.md)** - OAuth scope model, dual-phase authorization, security rationale
### Configuration & Setup
- **[Configuration Guide](configuration.md)** - Environment variables, Qdrant setup, Ollama setup, detailed configuration options
- **[Installation Guide](installation.md)** - Deployment options (Docker, Kubernetes, local)
- **[Running the Server](running.md)** - Starting the server, transport options, testing
### Monitoring & Troubleshooting
- **[Observability Guide](observability.md)** - Logging, metrics, tracing, debugging
- **[Troubleshooting](troubleshooting.md)** - General issues and solutions
### Related Documentation
- **[OAuth Architecture](oauth-architecture.md)** - OAuth flows, scopes, token management
- **[Comparison with Context Agent](comparison-context-agent.md)** - When to use Nextcloud MCP Server vs Context Agent
---
**Questions or Issues?**
- [Open an issue](https://github.com/cbcoutinho/nextcloud-mcp-server/issues)
- [Contribute improvements](https://github.com/cbcoutinho/nextcloud-mcp-server/pulls)
+72
View File
@@ -124,3 +124,75 @@ ENABLE_CUSTOM_PROCESSOR=false
# Comma-separated MIME types your processor supports
#CUSTOM_PROCESSOR_TYPES=application/pdf,image/jpeg,image/png
# ============================================
# Semantic Search & Vector Sync Configuration
# ============================================
# EXPERIMENTAL: Semantic search for Notes app (multi-app support planned)
# Requires: Qdrant vector database + Ollama embedding service
# Disabled by default
# Enable background vector indexing
VECTOR_SYNC_ENABLED=false
# Document scan interval in seconds (default: 300 = 5 minutes)
# How often to check for new/updated documents
#VECTOR_SYNC_SCAN_INTERVAL=300
# Concurrent indexing workers (default: 3)
# Number of parallel workers for embedding generation
#VECTOR_SYNC_PROCESSOR_WORKERS=3
# Max queued documents (default: 10000)
# Maximum documents waiting to be processed
#VECTOR_SYNC_QUEUE_MAX_SIZE=10000
# ============================================
# Qdrant Vector Database Configuration
# ============================================
# Choose ONE of three modes:
# 1. In-memory mode (default): Set neither QDRANT_URL nor QDRANT_LOCATION
# 2. Persistent local: Set QDRANT_LOCATION=/path/to/data
# 3. Network mode: Set QDRANT_URL=http://qdrant:6333
# Network mode: URL to Qdrant service
#QDRANT_URL=http://qdrant:6333
# Local mode: Path to store vectors (use :memory: for in-memory)
#QDRANT_LOCATION=:memory:
# API key for network mode (optional)
#QDRANT_API_KEY=
# Collection name (optional - auto-generated if not set)
# Auto-generation format: {deployment-id}-{model-name}
# Allows safe model switching and multi-server deployments
#QDRANT_COLLECTION=nextcloud_content
# ============================================
# Ollama Embedding Service Configuration
# ============================================
# Ollama endpoint for embeddings (if not set, uses SimpleEmbeddingProvider fallback)
#OLLAMA_BASE_URL=http://ollama:11434
# Embedding model to use (default: nomic-embed-text, 768 dimensions)
# Changing this creates a new collection (requires re-embedding all documents)
#OLLAMA_EMBEDDING_MODEL=nomic-embed-text
# Verify SSL certificates (default: true)
#OLLAMA_VERIFY_SSL=true
# ============================================
# Document Chunking Configuration
# ============================================
# Configure how documents are split before embedding
# Words per chunk (default: 512)
# Smaller chunks (256-384): More precise, less context, more storage
# Larger chunks (768-1024): More context, less precise, less storage
#DOCUMENT_CHUNK_SIZE=512
# Overlapping words between chunks (default: 50)
# Recommended: 10-20% of chunk size
# Preserves context across chunk boundaries
#DOCUMENT_CHUNK_OVERLAP=50
+228 -262
View File
@@ -5,13 +5,15 @@ from contextlib import AsyncExitStack, asynccontextmanager
from dataclasses import dataclass
from typing import TYPE_CHECKING, Optional
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor
if TYPE_CHECKING:
from nextcloud_mcp_server.auth.refresh_token_storage import RefreshTokenStorage
from nextcloud_mcp_server.auth.storage import RefreshTokenStorage
import anyio
import click
import httpx
import uvicorn
from anyio.streams.memory import MemoryObjectReceiveStream, MemoryObjectSendStream
from mcp.server.auth.settings import AuthSettings
from mcp.server.fastmcp import Context, FastMCP
@@ -19,7 +21,7 @@ from pydantic import AnyHttpUrl
from starlette.applications import Starlette
from starlette.middleware.authentication import AuthenticationMiddleware
from starlette.middleware.cors import CORSMiddleware
from starlette.responses import JSONResponse
from starlette.responses import JSONResponse, RedirectResponse
from starlette.routing import Mount, Route
from nextcloud_mcp_server.auth import (
@@ -32,13 +34,16 @@ from nextcloud_mcp_server.auth import (
from nextcloud_mcp_server.auth.unified_verifier import UnifiedTokenVerifier
from nextcloud_mcp_server.client import NextcloudClient
from nextcloud_mcp_server.config import (
LOGGING_CONFIG,
get_document_processor_config,
get_settings,
setup_logging,
)
from nextcloud_mcp_server.context import get_client as get_nextcloud_client
from nextcloud_mcp_server.document_processors import get_registry
from nextcloud_mcp_server.observability import (
ObservabilityMiddleware,
setup_metrics,
setup_tracing,
)
from nextcloud_mcp_server.server import (
configure_calendar_tools,
configure_contacts_tools,
@@ -54,6 +59,7 @@ from nextcloud_mcp_server.server.oauth_tools import register_oauth_tools
from nextcloud_mcp_server.vector import processor_task, scanner_task
logger = logging.getLogger(__name__)
HTTPXClientInstrumentor().instrument()
def initialize_document_processors():
@@ -211,6 +217,7 @@ class AppContext:
"""Application context for BasicAuth mode."""
client: NextcloudClient
storage: Optional["RefreshTokenStorage"] = None
document_send_stream: Optional[MemoryObjectSendStream] = None
document_receive_stream: Optional[MemoryObjectReceiveStream] = None
shutdown_event: Optional[anyio.Event] = None
@@ -284,7 +291,7 @@ async def load_oauth_client_credentials(
# Try loading from SQLite storage
try:
from nextcloud_mcp_server.auth.refresh_token_storage import RefreshTokenStorage
from nextcloud_mcp_server.auth.storage import RefreshTokenStorage
storage = RefreshTokenStorage.from_env()
await storage.initialize()
@@ -338,7 +345,7 @@ async def load_oauth_client_credentials(
# Ensure OAuth client in SQLite storage
from nextcloud_mcp_server.auth.client_registration import ensure_oauth_client
from nextcloud_mcp_server.auth.refresh_token_storage import RefreshTokenStorage
from nextcloud_mcp_server.auth.storage import RefreshTokenStorage
storage = RefreshTokenStorage.from_env()
await storage.initialize()
@@ -388,6 +395,13 @@ async def app_lifespan_basic(server: FastMCP) -> AsyncIterator[AppContext]:
client = NextcloudClient.from_env()
logger.info("Client initialization complete")
# Initialize persistent storage (for webhook tracking and future features)
from nextcloud_mcp_server.auth.storage import RefreshTokenStorage
storage = RefreshTokenStorage.from_env()
await storage.initialize()
logger.info("Persistent storage initialized (webhook tracking enabled)")
# Initialize document processors
initialize_document_processors()
@@ -404,6 +418,19 @@ async def app_lifespan_basic(server: FastMCP) -> AsyncIterator[AppContext]:
"NEXTCLOUD_USERNAME is required for vector sync in BasicAuth mode"
)
# Initialize Qdrant collection before starting background tasks
logger.info("Initializing Qdrant collection...")
from nextcloud_mcp_server.vector.qdrant_client import get_qdrant_client
try:
await get_qdrant_client() # Triggers collection creation if needed
logger.info("Qdrant collection ready")
except Exception as e:
logger.error(f"Failed to initialize Qdrant collection: {e}")
raise RuntimeError(
f"Cannot start vector sync - Qdrant initialization failed: {e}"
) from e
# Initialize shared state
send_stream, receive_stream = anyio.create_memory_object_stream(
max_buffer_size=settings.vector_sync_queue_max_size
@@ -442,6 +469,7 @@ async def app_lifespan_basic(server: FastMCP) -> AsyncIterator[AppContext]:
try:
yield AppContext(
client=client,
storage=storage,
document_send_stream=send_stream,
document_receive_stream=receive_stream,
shutdown_event=shutdown_event,
@@ -458,7 +486,7 @@ async def app_lifespan_basic(server: FastMCP) -> AsyncIterator[AppContext]:
else:
# No vector sync - simple lifecycle
try:
yield AppContext(client=client)
yield AppContext(client=client, storage=storage)
finally:
logger.info("Shutting down BasicAuth mode")
await client.close()
@@ -575,7 +603,7 @@ async def setup_oauth_config():
refresh_token_storage = None
if enable_offline_access:
try:
from nextcloud_mcp_server.auth.refresh_token_storage import (
from nextcloud_mcp_server.auth.storage import (
RefreshTokenStorage,
)
@@ -776,7 +804,31 @@ async def setup_oauth_config():
def get_app(transport: str = "sse", enabled_apps: list[str] | None = None):
setup_logging()
# Initialize observability (logging will be configured by uvicorn)
settings = get_settings()
# Setup Prometheus metrics (always enabled by default)
if settings.metrics_enabled:
setup_metrics(port=settings.metrics_port)
logger.info(
f"Prometheus metrics enabled on dedicated port {settings.metrics_port}"
)
# Setup OpenTelemetry tracing (optional)
if settings.otel_exporter_otlp_endpoint:
setup_tracing(
service_name=settings.otel_service_name,
otlp_endpoint=settings.otel_exporter_otlp_endpoint,
otlp_verify_ssl=settings.otel_exporter_verify_ssl,
sampling_rate=settings.otel_traces_sampler_arg,
)
logger.info(
f"OpenTelemetry tracing enabled (endpoint: {settings.otel_exporter_otlp_endpoint})"
)
else:
logger.info(
"OpenTelemetry tracing disabled (set OTEL_EXPORTER_OTLP_ENDPOINT to enable)"
)
# Determine authentication mode
oauth_enabled = is_oauth_mode()
@@ -999,7 +1051,7 @@ def get_app(transport: str = "sse", enabled_apps: list[str] | None = None):
# browser_app is in the same function scope (defined later in create_app)
# We need to find it in the mounted routes
for route in app.routes:
if isinstance(route, Mount) and route.path == "/user":
if isinstance(route, Mount) and route.path == "/app":
route.app.state.oauth_context = oauth_context_dict
logger.info(
"OAuth context shared with browser_app for session auth"
@@ -1009,6 +1061,23 @@ def get_app(transport: str = "sse", enabled_apps: list[str] | None = None):
logger.info(
f"OAuth context initialized for login routes (client_id={client_id[:16]}...)"
)
else:
# BasicAuth mode - share storage with browser_app for webhook management
from nextcloud_mcp_server.auth.storage import RefreshTokenStorage
storage = RefreshTokenStorage.from_env()
await storage.initialize()
app.state.storage = storage
# Also share with browser_app for webhook routes
for route in app.routes:
if isinstance(route, Mount) and route.path == "/app":
route.app.state.storage = storage
logger.info(
"Storage shared with browser_app for webhook management"
)
break
# Start background vector sync tasks for BasicAuth mode (ADR-007)
# For streamable-http transport, FastMCP lifespan isn't automatically triggered
@@ -1030,6 +1099,19 @@ def get_app(transport: str = "sse", enabled_apps: list[str] | None = None):
# Create client since we're outside FastMCP lifespan
client = NextcloudClient.from_env()
# Initialize Qdrant collection before starting background tasks
logger.info("Initializing Qdrant collection...")
from nextcloud_mcp_server.vector.qdrant_client import get_qdrant_client
try:
await get_qdrant_client() # Triggers collection creation if needed
logger.info("Qdrant collection ready")
except Exception as e:
logger.error(f"Failed to initialize Qdrant collection: {e}")
raise RuntimeError(
f"Cannot start vector sync - Qdrant initialization failed: {e}"
) from e
# Initialize shared state
send_stream, receive_stream = anyio_module.create_memory_object_stream(
max_buffer_size=settings.vector_sync_queue_max_size
@@ -1043,15 +1125,15 @@ def get_app(transport: str = "sse", enabled_apps: list[str] | None = None):
app.state.shutdown_event = shutdown_event
app.state.scanner_wake_event = scanner_wake_event
# Also share with browser_app for /user/page route
# Also share with browser_app for /app route
for route in app.routes:
if isinstance(route, Mount) and route.path == "/user":
if isinstance(route, Mount) and route.path == "/app":
route.app.state.document_send_stream = send_stream
route.app.state.document_receive_stream = receive_stream
route.app.state.shutdown_event = shutdown_event
route.app.state.scanner_wake_event = scanner_wake_event
logger.info(
"Vector sync state shared with browser_app for /user/page"
"Vector sync state shared with browser_app for /app"
)
break
@@ -1148,13 +1230,15 @@ def get_app(transport: str = "sse", enabled_apps: list[str] | None = None):
checks["auth_configured"] = "error: credentials not set"
is_ready = False
# Check Qdrant status if vector sync is enabled
# Check Qdrant status if using network mode (external Qdrant service)
# In-memory and persistent modes use embedded Qdrant, no external service to check
vector_sync_enabled = (
os.getenv("VECTOR_SYNC_ENABLED", "false").lower() == "true"
)
if vector_sync_enabled:
qdrant_url = os.getenv("QDRANT_URL") # Only set in network mode
if vector_sync_enabled and qdrant_url:
try:
qdrant_url = os.getenv("QDRANT_URL", "http://qdrant:6333")
async with httpx.AsyncClient(timeout=2.0) as client:
response = await client.get(f"{qdrant_url}/readyz")
if response.status_code == 200:
@@ -1165,6 +1249,9 @@ def get_app(transport: str = "sse", enabled_apps: list[str] | None = None):
except Exception as e:
checks["qdrant"] = f"error: {str(e)}"
is_ready = False
elif vector_sync_enabled:
# Using embedded Qdrant (memory or persistent mode)
checks["qdrant"] = "embedded"
status_code = 200 if is_ready else 503
return JSONResponse(
@@ -1175,6 +1262,31 @@ def get_app(transport: str = "sse", enabled_apps: list[str] | None = None):
status_code=status_code,
)
async def handle_nextcloud_webhook(request):
"""Test webhook endpoint to capture and log Nextcloud webhook payloads.
This is a temporary endpoint for testing webhook schemas and payloads.
It logs the full payload and returns 200 OK immediately.
"""
import json
try:
payload = await request.json()
logger.info("=" * 80)
logger.info("🔔 Webhook received from Nextcloud:")
logger.info(json.dumps(payload, indent=2, sort_keys=True))
logger.info("=" * 80)
return JSONResponse(
{"status": "received", "timestamp": payload.get("time")},
status_code=200,
)
except Exception as e:
logger.error(f"❌ Failed to parse webhook payload: {e}")
return JSONResponse(
{"error": "invalid_payload", "message": str(e)}, status_code=400
)
# Add Protected Resource Metadata (PRM) endpoint for OAuth mode
routes = []
@@ -1183,6 +1295,15 @@ def get_app(transport: str = "sse", enabled_apps: list[str] | None = None):
routes.append(Route("/health/ready", health_ready, methods=["GET"]))
logger.info("Health check endpoints enabled: /health/live, /health/ready")
# Add test webhook endpoint (for development/testing)
routes.append(
Route("/webhooks/nextcloud", handle_nextcloud_webhook, methods=["POST"])
)
logger.info("Test webhook endpoint enabled: /webhooks/nextcloud")
# Note: Metrics endpoint is NOT exposed on main HTTP port for security reasons.
# Metrics are served on dedicated port via setup_metrics() (default: 9090)
if oauth_enabled:
# Import OAuth routes (ADR-004 Progressive Consent)
from nextcloud_mcp_server.auth.oauth_routes import oauth_authorize
@@ -1315,17 +1436,37 @@ def get_app(transport: str = "sse", enabled_apps: list[str] | None = None):
from nextcloud_mcp_server.auth.userinfo_routes import (
revoke_session,
user_info_html,
user_info_json,
vector_sync_status_fragment,
)
from nextcloud_mcp_server.auth.webhook_routes import (
disable_webhook_preset,
enable_webhook_preset,
webhook_management_pane,
)
# Create a separate Starlette app for browser routes that need session auth
# This prevents SessionAuthBackend from interfering with FastMCP's OAuth
browser_routes = [
Route("/", user_info_json, methods=["GET"]), # /user/ → user_info_json
Route("/page", user_info_html, methods=["GET"]), # /user/page → user_info_html
Route("/", user_info_html, methods=["GET"]), # /app → webapp (HTML UI)
Route(
"/revoke", revoke_session, methods=["POST"], name="revoke_session_endpoint"
), # /user/revoke → revoke_session
), # /app/revoke → revoke_session
# Vector sync status fragment (htmx polling)
Route(
"/vector-sync/status",
vector_sync_status_fragment,
methods=["GET"],
), # /app/vector-sync/status
# Webhook management routes (admin-only)
Route("/webhooks", webhook_management_pane, methods=["GET"]), # /app/webhooks
Route(
"/webhooks/enable/{preset_id:str}", enable_webhook_preset, methods=["POST"]
),
Route(
"/webhooks/disable/{preset_id:str}",
disable_webhook_preset,
methods=["DELETE"],
),
]
browser_app = Starlette(routes=browser_routes)
@@ -1334,9 +1475,14 @@ def get_app(transport: str = "sse", enabled_apps: list[str] | None = None):
backend=SessionAuthBackend(oauth_enabled=oauth_enabled),
)
# Mount browser app at /user (so /user and /user/page work)
routes.append(Mount("/user", app=browser_app))
logger.info("User info routes with session auth: /user, /user/page")
# Add redirect from /app to /app/ (Starlette requires trailing slash for mounted apps)
routes.append(
Route("/app", lambda request: RedirectResponse("/app/", status_code=307))
)
# Mount browser app at /app (webapp and admin routes)
routes.append(Mount("/app", app=browser_app))
logger.info("App routes with session auth: /app, /app/webhooks, /app/revoke")
# Mount FastMCP at root last (catch-all, handles OAuth via token_verifier)
routes.append(Mount("/", app=mcp_app))
@@ -1346,7 +1492,7 @@ def get_app(transport: str = "sse", enabled_apps: list[str] | None = None):
"Routes: /user/* with SessionAuth, /mcp with FastMCP OAuth Bearer tokens"
)
# Add debugging middleware to log Authorization headers
# Add debugging middleware to log Authorization headers and client capabilities
@app.middleware("http")
async def log_auth_headers(request, call_next):
auth_header = request.headers.get("authorization")
@@ -1358,9 +1504,58 @@ def get_app(transport: str = "sse", enabled_apps: list[str] | None = None):
)
logger.info(f"🔑 /mcp request with Authorization: {token_preview}")
else:
logger.warning(
f"⚠️ /mcp request WITHOUT Authorization header from {request.client}"
)
# Only warn about missing Authorization in OAuth mode
# In BasicAuth mode, /mcp requests without Authorization are expected
if oauth_enabled:
logger.warning(
f"⚠️ /mcp request WITHOUT Authorization header from {request.client}"
)
# Log client capabilities on initialize request
if request.method == "POST":
# Read body to check for initialize request
# Starlette caches the body internally, so it's safe to read here
body = await request.body()
try:
import json
data = json.loads(body)
# Check if this is an initialize request
if data.get("method") == "initialize":
params = data.get("params", {})
capabilities = params.get("capabilities", {})
client_info = params.get("clientInfo", {})
logger.info(
f"🔌 MCP client connected: {client_info.get('name', 'unknown')} "
f"v{client_info.get('version', 'unknown')}"
)
# Log capabilities in a structured way
cap_summary = []
# Check for presence using 'in' not truthiness (empty dict {} counts as having capability)
if "roots" in capabilities:
cap_summary.append("roots")
if "sampling" in capabilities:
cap_summary.append("sampling")
if "experimental" in capabilities:
cap_summary.append(
f"experimental({len(capabilities['experimental'])} features)"
)
logger.info(
f"📋 Client capabilities: {', '.join(cap_summary) if cap_summary else 'none'}"
)
# Log full capabilities at INFO level to diagnose capability issues
logger.info(
f"Full capabilities JSON: {json.dumps(capabilities)}"
)
except Exception as e:
# Don't fail the request if logging fails
logger.debug(
f"Failed to parse MCP request for capability logging: {e}"
)
response = await call_next(request)
return response
@@ -1374,6 +1569,11 @@ def get_app(transport: str = "sse", enabled_apps: list[str] | None = None):
expose_headers=["*"],
)
# Add observability middleware (metrics + tracing)
if settings.metrics_enabled or settings.otel_exporter_otlp_endpoint:
app.add_middleware(ObservabilityMiddleware)
logger.info("Observability middleware enabled (metrics and/or tracing)")
# Add exception handler for scope challenges (OAuth mode only)
if oauth_enabled:
@@ -1403,237 +1603,3 @@ def get_app(transport: str = "sse", enabled_apps: list[str] | None = None):
logger.info("WWW-Authenticate scope challenge handler enabled")
return app
@click.command()
@click.option(
"--host", "-h", default="127.0.0.1", show_default=True, help="Server host"
)
@click.option(
"--port", "-p", type=int, default=8000, show_default=True, help="Server port"
)
@click.option(
"--log-level",
"-l",
default="info",
show_default=True,
type=click.Choice(["critical", "error", "warning", "info", "debug", "trace"]),
help="Logging level",
)
@click.option(
"--transport",
"-t",
default="sse",
show_default=True,
type=click.Choice(["sse", "streamable-http", "http"]),
help="MCP transport protocol",
)
@click.option(
"--enable-app",
"-e",
multiple=True,
type=click.Choice(
["notes", "tables", "webdav", "calendar", "contacts", "cookbook", "deck"]
),
help="Enable specific Nextcloud app APIs. Can be specified multiple times. If not specified, all apps are enabled.",
)
@click.option(
"--oauth/--no-oauth",
default=None,
help="Force OAuth mode (if enabled) or BasicAuth mode (if disabled). By default, auto-detected based on environment variables.",
)
@click.option(
"--oauth-client-id",
envvar="NEXTCLOUD_OIDC_CLIENT_ID",
help="OAuth client ID (can also use NEXTCLOUD_OIDC_CLIENT_ID env var)",
)
@click.option(
"--oauth-client-secret",
envvar="NEXTCLOUD_OIDC_CLIENT_SECRET",
help="OAuth client secret (can also use NEXTCLOUD_OIDC_CLIENT_SECRET env var)",
)
@click.option(
"--mcp-server-url",
envvar="NEXTCLOUD_MCP_SERVER_URL",
default="http://localhost:8000",
show_default=True,
help="MCP server URL for OAuth callbacks (can also use NEXTCLOUD_MCP_SERVER_URL env var)",
)
@click.option(
"--nextcloud-host",
envvar="NEXTCLOUD_HOST",
help="Nextcloud instance URL (can also use NEXTCLOUD_HOST env var)",
)
@click.option(
"--nextcloud-username",
envvar="NEXTCLOUD_USERNAME",
help="Nextcloud username for BasicAuth (can also use NEXTCLOUD_USERNAME env var)",
)
@click.option(
"--nextcloud-password",
envvar="NEXTCLOUD_PASSWORD",
help="Nextcloud password for BasicAuth (can also use NEXTCLOUD_PASSWORD env var)",
)
@click.option(
"--oauth-scopes",
envvar="NEXTCLOUD_OIDC_SCOPES",
default="openid profile email notes:read notes:write calendar:read calendar:write todo:read todo:write contacts:read contacts:write cookbook:read cookbook:write deck:read deck:write tables:read tables:write files:read files:write sharing:read sharing:write",
show_default=True,
help="OAuth scopes to request during client registration. These define the maximum allowed scopes for the client. Note: Actual supported scopes are discovered dynamically from MCP tools at runtime. (can also use NEXTCLOUD_OIDC_SCOPES env var)",
)
@click.option(
"--oauth-token-type",
envvar="NEXTCLOUD_OIDC_TOKEN_TYPE",
default="bearer",
show_default=True,
type=click.Choice(["bearer", "jwt"], case_sensitive=False),
help="OAuth token type (can also use NEXTCLOUD_OIDC_TOKEN_TYPE env var)",
)
@click.option(
"--public-issuer-url",
envvar="NEXTCLOUD_PUBLIC_ISSUER_URL",
help="Public issuer URL for OAuth (can also use NEXTCLOUD_PUBLIC_ISSUER_URL env var)",
)
def run(
host: str,
port: int,
log_level: str,
transport: str,
enable_app: tuple[str, ...],
oauth: bool | None,
oauth_client_id: str | None,
oauth_client_secret: str | None,
mcp_server_url: str,
nextcloud_host: str | None,
nextcloud_username: str | None,
nextcloud_password: str | None,
oauth_scopes: str,
oauth_token_type: str,
public_issuer_url: str | None,
):
"""
Run the Nextcloud MCP server.
\b
Authentication Modes:
- BasicAuth: Set NEXTCLOUD_USERNAME and NEXTCLOUD_PASSWORD
- OAuth: Leave USERNAME/PASSWORD unset (requires OIDC app enabled)
\b
Examples:
# BasicAuth mode with CLI options
$ nextcloud-mcp-server --nextcloud-host=https://cloud.example.com \\
--nextcloud-username=admin --nextcloud-password=secret
# BasicAuth mode with env vars (recommended for credentials)
$ export NEXTCLOUD_HOST=https://cloud.example.com
$ export NEXTCLOUD_USERNAME=admin
$ export NEXTCLOUD_PASSWORD=secret
$ nextcloud-mcp-server --host 0.0.0.0 --port 8000
# OAuth mode with auto-registration
$ nextcloud-mcp-server --nextcloud-host=https://cloud.example.com --oauth
# OAuth mode with pre-configured client
$ nextcloud-mcp-server --nextcloud-host=https://cloud.example.com --oauth \\
--oauth-client-id=xxx --oauth-client-secret=yyy
# OAuth mode with custom scopes and JWT tokens
$ nextcloud-mcp-server --nextcloud-host=https://cloud.example.com --oauth \\
--oauth-scopes="openid notes:read notes:write" --oauth-token-type=jwt
# OAuth with public issuer URL (for Docker/proxy setups)
$ nextcloud-mcp-server --nextcloud-host=http://app --oauth \\
--public-issuer-url=http://localhost:8080
"""
# Set env vars from CLI options if provided
if nextcloud_host:
os.environ["NEXTCLOUD_HOST"] = nextcloud_host
if nextcloud_username:
os.environ["NEXTCLOUD_USERNAME"] = nextcloud_username
if nextcloud_password:
os.environ["NEXTCLOUD_PASSWORD"] = nextcloud_password
if oauth_client_id:
os.environ["NEXTCLOUD_OIDC_CLIENT_ID"] = oauth_client_id
if oauth_client_secret:
os.environ["NEXTCLOUD_OIDC_CLIENT_SECRET"] = oauth_client_secret
if oauth_scopes:
os.environ["NEXTCLOUD_OIDC_SCOPES"] = oauth_scopes
if oauth_token_type:
os.environ["NEXTCLOUD_OIDC_TOKEN_TYPE"] = oauth_token_type
if mcp_server_url:
os.environ["NEXTCLOUD_MCP_SERVER_URL"] = mcp_server_url
if public_issuer_url:
os.environ["NEXTCLOUD_PUBLIC_ISSUER_URL"] = public_issuer_url
# Force OAuth mode if explicitly requested
if oauth is True:
# Clear username/password to force OAuth mode
if "NEXTCLOUD_USERNAME" in os.environ:
click.echo(
"Warning: --oauth flag set, ignoring NEXTCLOUD_USERNAME", err=True
)
del os.environ["NEXTCLOUD_USERNAME"]
if "NEXTCLOUD_PASSWORD" in os.environ:
click.echo(
"Warning: --oauth flag set, ignoring NEXTCLOUD_PASSWORD", err=True
)
del os.environ["NEXTCLOUD_PASSWORD"]
# Validate OAuth configuration
nextcloud_host = os.getenv("NEXTCLOUD_HOST")
if not nextcloud_host:
raise click.ClickException(
"OAuth mode requires NEXTCLOUD_HOST environment variable to be set"
)
# Check if we have client credentials OR if dynamic registration is possible
has_client_creds = os.getenv("NEXTCLOUD_OIDC_CLIENT_ID") and os.getenv(
"NEXTCLOUD_OIDC_CLIENT_SECRET"
)
if not has_client_creds:
# No client credentials - will attempt dynamic registration
# Show helpful message before server starts
click.echo("", err=True)
click.echo("OAuth Configuration:", err=True)
click.echo(" Mode: Dynamic Client Registration", err=True)
click.echo(" Host: " + nextcloud_host, err=True)
click.echo(" Storage: SQLite (TOKEN_STORAGE_DB)", err=True)
click.echo("", err=True)
click.echo(
"Note: Make sure 'Dynamic Client Registration' is enabled", err=True
)
click.echo(" in your Nextcloud OIDC app settings.", err=True)
click.echo("", err=True)
else:
click.echo("", err=True)
click.echo("OAuth Configuration:", err=True)
click.echo(" Mode: Pre-configured Client", err=True)
click.echo(" Host: " + nextcloud_host, err=True)
click.echo(
" Client ID: "
+ os.getenv("NEXTCLOUD_OIDC_CLIENT_ID", "")[:16]
+ "...",
err=True,
)
click.echo("", err=True)
elif oauth is False:
# Force BasicAuth mode - verify credentials exist
if not os.getenv("NEXTCLOUD_USERNAME") or not os.getenv("NEXTCLOUD_PASSWORD"):
raise click.ClickException(
"--no-oauth flag set but NEXTCLOUD_USERNAME or NEXTCLOUD_PASSWORD not set"
)
enabled_apps = list(enable_app) if enable_app else None
app = get_app(transport=transport, enabled_apps=enabled_apps)
uvicorn.run(
app=app, host=host, port=port, log_level=log_level, log_config=LOGGING_CONFIG
)
if __name__ == "__main__":
run()
@@ -1,7 +1,7 @@
"""Browser-based OAuth login routes for admin UI.
Separate from MCP OAuth flow - these routes establish browser sessions
for accessing admin UI endpoints like /user/page.
for accessing admin UI endpoints like /app.
"""
import hashlib
@@ -38,8 +38,8 @@ async def oauth_login(request: Request) -> RedirectResponse | JSONResponse:
"""
oauth_ctx = request.app.state.oauth_context
if not oauth_ctx:
# BasicAuth mode - no login needed, redirect to user page
return RedirectResponse("/user/page", status_code=302)
# BasicAuth mode - no login needed, redirect to app
return RedirectResponse("/app", status_code=302)
storage = oauth_ctx["storage"]
oauth_client = oauth_ctx["oauth_client"]
@@ -71,7 +71,7 @@ async def oauth_login(request: Request) -> RedirectResponse | JSONResponse:
await storage.store_oauth_session(
session_id=state, # Use state as session ID
client_id="browser-ui",
client_redirect_uri="/user/page",
client_redirect_uri="/app",
state=state,
code_challenge=code_challenge,
code_challenge_method="S256",
@@ -383,7 +383,7 @@ async def oauth_login_callback(request: Request) -> RedirectResponse | HTMLRespo
# Continue anyway - profile cache is optional for browser UI
# Create response and set session cookie
response = RedirectResponse("/user/page", status_code=302)
response = RedirectResponse("/app", status_code=302)
response.set_cookie(
key="mcp_session",
value=user_id,
@@ -8,7 +8,7 @@ from typing import Any
import anyio
import httpx
from nextcloud_mcp_server.auth.refresh_token_storage import RefreshTokenStorage
from nextcloud_mcp_server.auth.storage import RefreshTokenStorage
logger = logging.getLogger(__name__)
+1 -1
View File
@@ -32,7 +32,7 @@ from starlette.requests import Request
from starlette.responses import JSONResponse, RedirectResponse
from nextcloud_mcp_server.auth.client_registry import get_client_registry
from nextcloud_mcp_server.auth.refresh_token_storage import RefreshTokenStorage
from nextcloud_mcp_server.auth.storage import RefreshTokenStorage
logger = logging.getLogger(__name__)
+54
View File
@@ -0,0 +1,54 @@
"""Permission checking utilities for Nextcloud admin operations."""
import logging
from httpx import AsyncClient
from starlette.requests import Request
from nextcloud_mcp_server.client.users import UsersClient
logger = logging.getLogger(__name__)
async def is_nextcloud_admin(request: Request, http_client: AsyncClient) -> bool:
"""Check if the authenticated user is a Nextcloud administrator.
This function extracts the username from the session/request context
and checks if the user is a member of the "admin" group in Nextcloud.
Args:
request: Starlette request object with authenticated user
http_client: Authenticated HTTP client for Nextcloud API calls
Returns:
True if user is admin, False otherwise
Example:
```python
if await is_nextcloud_admin(request, http_client):
# Show admin-only features
pass
```
"""
try:
# Extract username from authenticated session
username = request.user.display_name
if not username:
logger.warning("No username found in authenticated session")
return False
# Query Nextcloud for user's group memberships
users_client = UsersClient(http_client, username)
user_groups = await users_client.get_user_groups(username)
# Check if user is in the admin group
is_admin = "admin" in user_groups
logger.debug(
f"Admin check for user '{username}': {is_admin} (groups: {user_groups})"
)
return is_admin
except Exception as e:
logger.error(f"Error checking admin permissions: {e}", exc_info=True)
return False
@@ -13,7 +13,7 @@ from mcp.server.fastmcp import Context
from mcp.shared.exceptions import McpError
from mcp.types import ErrorData
from nextcloud_mcp_server.auth.refresh_token_storage import RefreshTokenStorage
from nextcloud_mcp_server.auth.storage import RefreshTokenStorage
logger = logging.getLogger(__name__)
@@ -1,23 +1,28 @@
"""
Refresh Token Storage for ADR-002 Tier 1: Offline Access
Persistent Storage for MCP Server State
Manages two separate concerns for OAuth authentication:
This module provides SQLite-based storage for multiple concerns across both
BasicAuth and OAuth authentication modes:
1. **Refresh Tokens** (for background jobs ONLY)
1. **Refresh Tokens** (OAuth mode only, for background jobs)
- Securely stores encrypted refresh tokens for offline access
- Used ONLY by background jobs to obtain access tokens
- NEVER used within MCP client sessions or browser sessions
2. **User Profile Cache** (for browser UI display ONLY)
2. **User Profile Cache** (OAuth mode only, for browser UI display)
- Caches IdP user profile data for browser-based admin UI
- Queried ONCE at login, displayed from cache thereafter
- NOT used for authorization decisions or background jobs
IMPORTANT: These are separate concerns. Browser sessions read profile cache for
display purposes. Background jobs use refresh tokens for API access. Never mix
the two.
3. **Webhook Registration Tracking** (both modes, for webhook management)
- Tracks registered webhook IDs mapped to presets
- Enables persistent webhook state across restarts
- Avoids redundant Nextcloud API calls for webhook status
Tokens are encrypted at rest using Fernet symmetric encryption.
IMPORTANT: The database is initialized in both BasicAuth and OAuth modes.
Token storage requires TOKEN_ENCRYPTION_KEY, but webhook tracking does not.
Sensitive data (tokens, secrets) is encrypted at rest using Fernet symmetric encryption.
"""
import json
@@ -34,25 +39,34 @@ logger = logging.getLogger(__name__)
class RefreshTokenStorage:
"""Securely store and manage user refresh tokens and profile cache.
"""Persistent storage for MCP server state (tokens, webhooks, and future features).
This class manages two separate concerns:
- Refresh tokens: Encrypted storage for background job access (write-only by OAuth, read-only by background jobs)
- User profiles: Plain JSON cache for browser UI display (written at login, read by UI)
This class manages multiple concerns across both BasicAuth and OAuth modes:
These concerns are architecturally separate and should never be mixed.
**OAuth-specific concerns**:
- Refresh tokens: Encrypted storage for background job access (requires encryption key)
- User profiles: Plain JSON cache for browser UI display
- OAuth client credentials: Encrypted client secrets from DCR
- OAuth sessions: Temporary session state for progressive consent flow
**Both modes**:
- Webhook registration: Track registered webhooks mapped to presets
- Schema versioning: Handle database migrations automatically
Token-related operations require TOKEN_ENCRYPTION_KEY, but webhook operations do not.
"""
def __init__(self, db_path: str, encryption_key: bytes):
def __init__(self, db_path: str, encryption_key: bytes | None = None):
"""
Initialize refresh token storage.
Initialize persistent storage.
Args:
db_path: Path to SQLite database file
encryption_key: Fernet encryption key (32 bytes, base64-encoded)
encryption_key: Optional Fernet encryption key (32 bytes, base64-encoded).
Required for token storage operations, not required for webhook tracking.
"""
self.db_path = db_path
self.cipher = Fernet(encryption_key)
self.cipher = Fernet(encryption_key) if encryption_key else None
self._initialized = False
@classmethod
@@ -62,41 +76,42 @@ class RefreshTokenStorage:
Environment variables:
TOKEN_STORAGE_DB: Path to database file (default: /app/data/tokens.db)
TOKEN_ENCRYPTION_KEY: Base64-encoded Fernet key
TOKEN_ENCRYPTION_KEY: Optional base64-encoded Fernet key (required for token storage)
Returns:
RefreshTokenStorage instance
Raises:
ValueError: If TOKEN_ENCRYPTION_KEY is not set
Note:
If TOKEN_ENCRYPTION_KEY is not set, token storage operations will fail,
but webhook tracking will still work.
"""
db_path = os.getenv("TOKEN_STORAGE_DB", "/app/data/tokens.db")
encryption_key_b64 = os.getenv("TOKEN_ENCRYPTION_KEY")
if not encryption_key_b64:
raise ValueError(
"TOKEN_ENCRYPTION_KEY environment variable is required. "
"Generate one with: python -c 'from cryptography.fernet import Fernet; "
"print(Fernet.generate_key().decode())'"
encryption_key = None
if encryption_key_b64:
# Fernet expects a base64url-encoded key as bytes, not decoded bytes
# The key from Fernet.generate_key() is already base64url-encoded
try:
# Convert string to bytes if needed
if isinstance(encryption_key_b64, str):
encryption_key = encryption_key_b64.encode()
else:
encryption_key = encryption_key_b64
# Validate the key by trying to create a Fernet instance
Fernet(encryption_key)
except Exception as e:
raise ValueError(
f"Invalid TOKEN_ENCRYPTION_KEY: {e}. "
"Must be a valid Fernet key (base64url-encoded 32 bytes)."
) from e
else:
logger.info(
"TOKEN_ENCRYPTION_KEY not set - token storage operations will be unavailable, "
"but webhook tracking will still work"
)
# Fernet expects a base64url-encoded key as bytes, not decoded bytes
# The key from Fernet.generate_key() is already base64url-encoded
try:
# Convert string to bytes if needed
if isinstance(encryption_key_b64, str):
encryption_key = encryption_key_b64.encode()
else:
encryption_key = encryption_key_b64
# Validate the key by trying to create a Fernet instance
Fernet(encryption_key)
except Exception as e:
raise ValueError(
f"Invalid TOKEN_ENCRYPTION_KEY: {e}. "
"Must be a valid Fernet key (base64url-encoded 32 bytes)."
) from e
return cls(db_path=db_path, encryption_key=encryption_key)
async def initialize(self) -> None:
@@ -204,6 +219,38 @@ class RefreshTokenStorage:
"ON oauth_sessions(mcp_authorization_code)"
)
# Schema version tracking
await db.execute(
"""
CREATE TABLE IF NOT EXISTS schema_version (
version INTEGER PRIMARY KEY,
applied_at REAL NOT NULL
)
"""
)
# Registered webhooks tracking (both BasicAuth and OAuth modes)
await db.execute(
"""
CREATE TABLE IF NOT EXISTS registered_webhooks (
id INTEGER PRIMARY KEY AUTOINCREMENT,
webhook_id INTEGER NOT NULL UNIQUE,
preset_id TEXT NOT NULL,
created_at REAL NOT NULL
)
"""
)
# Create indexes for efficient webhook queries
await db.execute(
"CREATE INDEX IF NOT EXISTS idx_webhooks_preset "
"ON registered_webhooks(preset_id)"
)
await db.execute(
"CREATE INDEX IF NOT EXISTS idx_webhooks_created "
"ON registered_webhooks(created_at)"
)
await db.commit()
# Set restrictive permissions after creation
@@ -1104,6 +1151,123 @@ class RefreshTokenStorage:
return deleted
# ============================================================================
# Webhook Registration Tracking (both BasicAuth and OAuth modes)
# ============================================================================
async def store_webhook(self, webhook_id: int, preset_id: str) -> None:
"""
Store registered webhook ID for tracking.
Args:
webhook_id: Nextcloud webhook ID
preset_id: Preset identifier (e.g., "notes_sync", "calendar_sync")
"""
if not self._initialized:
await self.initialize()
async with aiosqlite.connect(self.db_path) as db:
await db.execute(
"INSERT OR REPLACE INTO registered_webhooks (webhook_id, preset_id, created_at) VALUES (?, ?, ?)",
(webhook_id, preset_id, time.time()),
)
await db.commit()
logger.debug(f"Stored webhook {webhook_id} for preset '{preset_id}'")
async def get_webhooks_by_preset(self, preset_id: str) -> list[int]:
"""
Get all webhook IDs registered for a preset.
Args:
preset_id: Preset identifier
Returns:
List of webhook IDs
"""
if not self._initialized:
await self.initialize()
async with aiosqlite.connect(self.db_path) as db:
cursor = await db.execute(
"SELECT webhook_id FROM registered_webhooks WHERE preset_id = ?",
(preset_id,),
)
rows = await cursor.fetchall()
return [row[0] for row in rows]
async def delete_webhook(self, webhook_id: int) -> bool:
"""
Remove webhook from tracking.
Args:
webhook_id: Nextcloud webhook ID to remove
Returns:
True if webhook was deleted, False if not found
"""
if not self._initialized:
await self.initialize()
async with aiosqlite.connect(self.db_path) as db:
cursor = await db.execute(
"DELETE FROM registered_webhooks WHERE webhook_id = ?", (webhook_id,)
)
await db.commit()
deleted = cursor.rowcount > 0
if deleted:
logger.debug(f"Deleted webhook {webhook_id} from tracking")
return deleted
async def list_all_webhooks(self) -> list[dict]:
"""
List all tracked webhooks with metadata.
Returns:
List of webhook dictionaries with keys: webhook_id, preset_id, created_at
"""
if not self._initialized:
await self.initialize()
async with aiosqlite.connect(self.db_path) as db:
cursor = await db.execute(
"SELECT webhook_id, preset_id, created_at FROM registered_webhooks ORDER BY created_at DESC"
)
rows = await cursor.fetchall()
return [
{"webhook_id": row[0], "preset_id": row[1], "created_at": row[2]}
for row in rows
]
async def clear_preset_webhooks(self, preset_id: str) -> int:
"""
Delete all webhooks for a preset (bulk operation).
Args:
preset_id: Preset identifier
Returns:
Number of webhooks deleted
"""
if not self._initialized:
await self.initialize()
async with aiosqlite.connect(self.db_path) as db:
cursor = await db.execute(
"DELETE FROM registered_webhooks WHERE preset_id = ?", (preset_id,)
)
await db.commit()
deleted = cursor.rowcount
if deleted > 0:
logger.debug(f"Cleared {deleted} webhook(s) for preset '{preset_id}'")
return deleted
async def generate_encryption_key() -> str:
"""
+1 -1
View File
@@ -23,7 +23,7 @@ import httpx
import jwt
from cryptography.fernet import Fernet
from nextcloud_mcp_server.auth.refresh_token_storage import RefreshTokenStorage
from nextcloud_mcp_server.auth.storage import RefreshTokenStorage
from nextcloud_mcp_server.auth.token_exchange import exchange_token_for_delegation
logger = logging.getLogger(__name__)
+1 -1
View File
@@ -20,7 +20,7 @@ import httpx
import jwt
from ..config import get_settings
from .refresh_token_storage import RefreshTokenStorage
from .storage import RefreshTokenStorage
logger = logging.getLogger(__name__)
+11 -7
View File
@@ -231,17 +231,21 @@ class UnifiedTokenVerifier(TokenVerifier):
token,
signing_key.key,
algorithms=["RS256"],
issuer=self.settings.oidc_issuer
if hasattr(self.settings, "oidc_issuer")
else None,
issuer=(
self.settings.oidc_issuer
if hasattr(self.settings, "oidc_issuer")
else None
),
options={
"verify_signature": True,
"verify_exp": True,
"verify_iat": True,
"verify_iss": True
if hasattr(self.settings, "oidc_issuer")
and self.settings.oidc_issuer
else False,
"verify_iss": (
True
if hasattr(self.settings, "oidc_issuer")
and self.settings.oidc_issuer
else False
),
"verify_aud": False, # We handle audience validation separately
},
)
+344 -67
View File
@@ -19,6 +19,57 @@ from starlette.responses import HTMLResponse, JSONResponse
logger = logging.getLogger(__name__)
async def _get_authenticated_client_for_userinfo(request: Request) -> httpx.AsyncClient:
"""Get an authenticated HTTP client for user info page operations.
Args:
request: Starlette request object
Returns:
Authenticated httpx.AsyncClient
"""
oauth_ctx = getattr(request.app.state, "oauth_context", None)
# BasicAuth mode - use credentials from environment
if not oauth_ctx:
nextcloud_host = os.getenv("NEXTCLOUD_HOST")
username = os.getenv("NEXTCLOUD_USERNAME")
password = os.getenv("NEXTCLOUD_PASSWORD")
if not all([nextcloud_host, username, password]):
raise RuntimeError("BasicAuth credentials not configured")
assert nextcloud_host is not None # Type narrowing for type checker
return httpx.AsyncClient(
base_url=nextcloud_host,
auth=(username, password),
timeout=30.0,
)
# OAuth mode - get token from session
storage = oauth_ctx.get("storage")
session_id = request.cookies.get("mcp_session")
if not storage or not session_id:
raise RuntimeError("Session not found")
token_data = await storage.get_refresh_token(session_id)
if not token_data or "access_token" not in token_data:
raise RuntimeError("No access token found in session")
access_token = token_data["access_token"]
nextcloud_host = oauth_ctx.get("config", {}).get("nextcloud_host", "")
if not nextcloud_host:
raise RuntimeError("Nextcloud host not configured")
return httpx.AsyncClient(
base_url=nextcloud_host,
headers={"Authorization": f"Bearer {access_token}"},
timeout=30.0,
)
async def _get_processing_status(request: Request) -> dict[str, Any] | None:
"""Get vector sync processing status.
@@ -43,14 +94,17 @@ async def _get_processing_status(request: Request) -> dict[str, Any] | None:
return None
try:
# Get document queue from app state
document_queue = getattr(request.app.state, "document_queue", None)
if document_queue is None:
logger.debug("document_queue not available in app state")
# Get document receive stream from app state
document_receive_stream = getattr(
request.app.state, "document_receive_stream", None
)
if document_receive_stream is None:
logger.debug("document_receive_stream not available in app state")
return None
# Get pending count from queue
pending_count = document_queue.qsize()
# Get pending count from stream statistics
stats = document_receive_stream.statistics()
pending_count = stats.current_buffer_used
# Get Qdrant client and query indexed count
indexed_count = 0
@@ -63,7 +117,7 @@ async def _get_processing_status(request: Request) -> dict[str, Any] | None:
# Count documents in collection
count_result = await qdrant_client.count(
collection_name=settings.qdrant_collection
collection_name=settings.get_collection_name()
)
indexed_count = count_result.count
@@ -85,6 +139,71 @@ async def _get_processing_status(request: Request) -> dict[str, Any] | None:
return None
@requires("authenticated", redirect="oauth_login")
async def vector_sync_status_fragment(request: Request) -> HTMLResponse:
"""Vector sync status fragment endpoint - returns HTML fragment with current status.
This endpoint is polled by htmx to provide real-time updates of vector sync processing
status without requiring a full page refresh.
Requires authentication via session cookie (redirects to oauth_login route if not authenticated).
Args:
request: Starlette request object
Returns:
HTML response with vector sync status table fragment
"""
processing_status = await _get_processing_status(request)
# If vector sync is disabled or unavailable, return empty fragment
if not processing_status:
return HTMLResponse(
"""
<div id="vector-sync-status" hx-get="/app/vector-sync/status" hx-trigger="every 10s" hx-swap="innerHTML">
<p style="color: #999;">Vector sync not available</p>
</div>
"""
)
indexed_count = processing_status["indexed_count"]
pending_count = processing_status["pending_count"]
status = processing_status["status"]
# Format numbers with commas for readability
indexed_count_str = f"{indexed_count:,}"
pending_count_str = f"{pending_count:,}"
# Status badge color and text
if status == "syncing":
status_badge = (
'<span style="color: #ff9800; font-weight: bold;">⟳ Syncing</span>'
)
else:
status_badge = '<span style="color: #4caf50; font-weight: bold;">✓ Idle</span>'
# Return inner content only (container div is in initial page render)
html = f"""
<h2>Vector Sync Status</h2>
<table>
<tr>
<td><strong>Indexed Documents</strong></td>
<td>{indexed_count_str}</td>
</tr>
<tr>
<td><strong>Pending Documents</strong></td>
<td>{pending_count_str}</td>
</tr>
<tr>
<td><strong>Status</strong></td>
<td>{status_badge}</td>
</tr>
</table>
"""
return HTMLResponse(html)
async def _get_userinfo_endpoint(oauth_ctx: dict[str, Any]) -> str | None:
"""Get the correct userinfo endpoint based on OAuth mode.
@@ -293,6 +412,19 @@ async def user_info_html(request: Request) -> HTMLResponse:
# Get vector sync processing status
processing_status = await _get_processing_status(request)
# Check if user is admin (for Webhooks tab)
is_admin = False
try:
from nextcloud_mcp_server.auth.permissions import is_nextcloud_admin
# Get authenticated HTTP client
http_client = await _get_authenticated_client_for_userinfo(request)
is_admin = await is_nextcloud_admin(request, http_client)
await http_client.aclose()
except Exception as e:
logger.warning(f"Failed to check admin status: {e}")
# Default to not admin if check fails
# Check for error
if "error" in user_context and user_context["error"] != "":
# Get login URL dynamically
@@ -440,43 +572,15 @@ async def user_info_html(request: Request) -> HTMLResponse:
</div>
"""
# Build vector sync status HTML
# Build vector sync status HTML (with htmx auto-refresh)
vector_status_html = ""
if processing_status:
indexed_count = processing_status["indexed_count"]
pending_count = processing_status["pending_count"]
status = processing_status["status"]
# Format numbers with commas for readability
indexed_count_str = f"{indexed_count:,}"
pending_count_str = f"{pending_count:,}"
# Status badge color and text
if status == "syncing":
status_badge = (
'<span style="color: #ff9800; font-weight: bold;">⟳ Syncing</span>'
)
else:
status_badge = (
'<span style="color: #4caf50; font-weight: bold;">✓ Idle</span>'
)
vector_status_html = f"""
<h2>Vector Sync Status</h2>
<table>
<tr>
<td><strong>Indexed Documents</strong></td>
<td>{indexed_count_str}</td>
</tr>
<tr>
<td><strong>Pending Documents</strong></td>
<td>{pending_count_str}</td>
</tr>
<tr>
<td><strong>Status</strong></td>
<td>{status_badge}</td>
</tr>
</table>
# Use htmx to load and auto-refresh the status fragment
# Container div stays stable, only inner content updates every 10s
vector_status_html = """
<div id="vector-sync-status" hx-get="/app/vector-sync/status" hx-trigger="load, every 10s" hx-swap="innerHTML">
<p style="color: #999;">Loading vector sync status...</p>
</div>
"""
# Build IdP profile HTML
@@ -503,17 +607,61 @@ async def user_info_html(request: Request) -> HTMLResponse:
<div class="warning">{user_context["idp_profile_error"]}</div>
"""
# Build user info tab content
user_info_tab_html = f"""
<h2>Authentication</h2>
<table>
<tr>
<td><strong>Username</strong></td>
<td>{username}</td>
</tr>
<tr>
<td><strong>Authentication Mode</strong></td>
<td><span class="badge badge-{auth_mode}">{auth_mode}</span></td>
</tr>
</table>
{host_info_html}
{session_info_html}
{idp_profile_html}
"""
# Determine which tabs to show
show_vector_sync_tab = processing_status is not None
show_webhooks_tab = is_admin
# Build vector sync tab content (only if enabled)
vector_sync_tab_html = ""
if show_vector_sync_tab:
vector_sync_tab_html = vector_status_html
# Build webhooks tab content (only if admin)
webhooks_tab_html = ""
if show_webhooks_tab:
webhooks_tab_html = """
<div hx-get="/app/webhooks" hx-trigger="load" hx-swap="outerHTML">
<p style="color: #999;">Loading webhook management...</p>
</div>
"""
html_content = f"""
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>User Info - Nextcloud MCP Server</title>
<title>Nextcloud MCP Server</title>
<!-- htmx for dynamic loading -->
<script src="https://unpkg.com/htmx.org@1.9.10"></script>
<!-- Alpine.js for tab state management -->
<script defer src="https://cdn.jsdelivr.net/npm/alpinejs@3.x.x/dist/cdn.min.js"></script>
<style>
body {{
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif;
max-width: 800px;
max-width: 900px;
margin: 50px auto;
padding: 20px;
background-color: #f5f5f5;
@@ -523,6 +671,7 @@ async def user_info_html(request: Request) -> HTMLResponse:
border-radius: 8px;
padding: 30px;
box-shadow: 0 2px 4px rgba(0,0,0,0.1);
min-height: calc(100vh - 200px);
}}
h1 {{
color: #0082c9;
@@ -532,10 +681,51 @@ async def user_info_html(request: Request) -> HTMLResponse:
}}
h2 {{
color: #333;
margin-top: 30px;
margin-top: 20px;
border-bottom: 1px solid #e0e0e0;
padding-bottom: 5px;
}}
/* Tab navigation */
.tabs {{
display: flex;
gap: 0;
margin: 20px 0 0 0;
border-bottom: 2px solid #e0e0e0;
}}
.tab {{
padding: 12px 24px;
cursor: pointer;
background: transparent;
border: none;
font-size: 14px;
font-weight: 500;
color: #666;
border-bottom: 2px solid transparent;
margin-bottom: -2px;
transition: all 0.2s;
}}
.tab:hover {{
color: #0082c9;
background-color: #f5f5f5;
}}
.tab.active {{
color: #0082c9;
border-bottom-color: #0082c9;
}}
/* Tab content - use grid to overlay panes */
.tab-content {{
padding: 20px 0;
display: grid;
}}
/* Tab panes - all occupy the same grid cell to overlay */
.tab-pane {{
grid-area: 1 / 1;
}}
/* Tables */
table {{
width: 100%;
border-collapse: collapse;
@@ -555,6 +745,8 @@ async def user_info_html(request: Request) -> HTMLResponse:
border-radius: 3px;
font-family: 'Courier New', monospace;
}}
/* Badges */
.badge {{
display: inline-block;
padding: 3px 8px;
@@ -571,6 +763,8 @@ async def user_info_html(request: Request) -> HTMLResponse:
background-color: #2196f3;
color: white;
}}
/* Messages */
.warning {{
background-color: #fff3cd;
border-left: 4px solid #ffc107;
@@ -578,11 +772,15 @@ async def user_info_html(request: Request) -> HTMLResponse:
margin: 15px 0;
color: #856404;
}}
.logout {{
margin-top: 30px;
padding-top: 20px;
border-top: 1px solid #e0e0e0;
.info-message {{
background-color: #e3f2fd;
border-left: 4px solid #2196f3;
padding: 15px;
margin: 15px 0;
color: #1565c0;
}}
/* Buttons */
.button {{
display: inline-block;
padding: 10px 20px;
@@ -591,34 +789,113 @@ async def user_info_html(request: Request) -> HTMLResponse:
text-decoration: none;
border-radius: 4px;
transition: background-color 0.3s;
border: none;
cursor: pointer;
font-size: 14px;
}}
.button:hover {{
background-color: #b71c1c;
}}
.button-primary {{
background-color: #0082c9;
}}
.button-primary:hover {{
background-color: #006ba3;
}}
/* Logout section */
.logout {{
margin-top: 30px;
padding-top: 20px;
border-top: 1px solid #e0e0e0;
}}
/* Smooth htmx content swaps */
.htmx-swapping {{
opacity: 0;
transition: opacity 200ms ease-out;
}}
/* Smooth htmx content settling */
.htmx-settling {{
opacity: 1;
transition: opacity 200ms ease-in;
}}
</style>
</head>
<body>
<div class="container">
<h1>Nextcloud MCP Server - User Info</h1>
<div class="container" x-data="{{ activeTab: 'user-info' }}">
<h1>Nextcloud MCP Server</h1>
<h2>Authentication</h2>
<table>
<tr>
<td><strong>Username</strong></td>
<td>{username}</td>
</tr>
<tr>
<td><strong>Authentication Mode</strong></td>
<td><span class="badge badge-{auth_mode}">{auth_mode}</span></td>
</tr>
</table>
<!-- Tab Navigation -->
<div class="tabs">
<button
class="tab"
:class="activeTab === 'user-info' ? 'active' : ''"
@click="activeTab = 'user-info'">
User Info
</button>
{
""
if not show_vector_sync_tab
else '''
<button
class="tab"
:class="activeTab === 'vector-sync' ? 'active' : ''"
@click="activeTab = 'vector-sync'">
Vector Sync
</button>
'''
}
{
""
if not show_webhooks_tab
else '''
<button
class="tab"
:class="activeTab === 'webhooks' ? 'active' : ''"
@click="activeTab = 'webhooks'">
Webhooks
</button>
'''
}
</div>
{host_info_html}
{session_info_html}
{vector_status_html}
{idp_profile_html}
<!-- Tab Content -->
<div class="tab-content">
<!-- User Info Tab -->
<div class="tab-pane" x-show="activeTab === 'user-info'" x-transition.opacity.duration.150ms>
{user_info_tab_html}
</div>
{f'<div class="logout"><a href="{logout_url}" class="button">Logout</a></div>' if auth_mode == "oauth" else ""}
{
""
if not show_vector_sync_tab
else f'''
<!-- Vector Sync Tab -->
<div class="tab-pane" x-show="activeTab === 'vector-sync'" x-transition.opacity.duration.150ms>
{vector_sync_tab_html}
</div>
'''
}
{
""
if not show_webhooks_tab
else f'''
<!-- Webhooks Tab (admin-only, loaded dynamically) -->
<div class="tab-pane" x-show="activeTab === 'webhooks'" x-transition.opacity.duration.150ms>
{webhooks_tab_html}
</div>
'''
}
</div>
{
f'<div class="logout"><a href="{logout_url}" class="button">Logout</a></div>'
if auth_mode == "oauth"
else ""
}
</div>
</body>
</html>
+540
View File
@@ -0,0 +1,540 @@
"""Webhook management routes for admin UI.
Provides browser-based endpoints for admin users to manage webhook configurations
using preset templates. Only accessible to Nextcloud administrators.
"""
import logging
import os
import httpx
from starlette.authentication import requires
from starlette.requests import Request
from starlette.responses import HTMLResponse
from nextcloud_mcp_server.auth.permissions import is_nextcloud_admin
from nextcloud_mcp_server.client.webhooks import WebhooksClient
from nextcloud_mcp_server.server.webhook_presets import (
WEBHOOK_PRESETS,
filter_presets_by_installed_apps,
get_preset,
)
logger = logging.getLogger(__name__)
def _get_storage(request: Request):
"""Get storage instance from app state.
Args:
request: Starlette request object
Returns:
RefreshTokenStorage instance or None
"""
# Try browser_app state first (for /app routes)
storage = getattr(request.app.state, "storage", None)
# Try oauth_context if in OAuth mode
if not storage:
oauth_ctx = getattr(request.app.state, "oauth_context", None)
if oauth_ctx:
storage = oauth_ctx.get("storage")
return storage
async def _get_installed_apps(http_client: httpx.AsyncClient) -> list[str]:
"""Get list of installed and enabled apps from Nextcloud capabilities.
Args:
http_client: Authenticated HTTP client
Returns:
List of installed app names (e.g., ["notes", "calendar", "forms"])
"""
try:
response = await http_client.get(
"/ocs/v2.php/cloud/capabilities",
headers={"OCS-APIRequest": "true", "Accept": "application/json"},
)
response.raise_for_status()
data = response.json()
# Extract app names from capabilities
capabilities = data.get("ocs", {}).get("data", {}).get("capabilities", {})
# Filter out core NC capabilities (not apps)
core_keys = {"version", "core"}
app_keys = set(capabilities.keys()) - core_keys
return sorted(app_keys)
except Exception as e:
logger.warning(f"Failed to get installed apps from capabilities: {e}")
return []
def _get_webhook_uri() -> str:
"""Get the webhook endpoint URI for this MCP server.
This function determines the correct webhook URL based on the environment:
1. Uses WEBHOOK_INTERNAL_URL if explicitly set (highest priority)
2. Detects Docker environment and uses internal service name
3. Falls back to NEXTCLOUD_MCP_SERVER_URL
In Docker environments, Nextcloud needs to reach the MCP service using
the internal Docker network hostname (e.g., http://mcp:8000), not localhost.
Returns:
Full webhook endpoint URL accessible from Nextcloud
"""
# Explicit override (highest priority)
webhook_url = os.getenv("WEBHOOK_INTERNAL_URL")
if webhook_url:
return f"{webhook_url}/webhooks/nextcloud"
# Detect Docker environment
# Check for common Docker indicators
is_docker = (
os.path.exists("/.dockerenv") # Docker container marker file
or os.path.exists("/run/.containerenv") # Podman marker
or os.getenv("DOCKER_CONTAINER") == "true" # Explicit flag
)
if is_docker:
# In Docker, use internal service name from NEXTCLOUD_MCP_SERVICE_NAME
# or default to 'mcp' (docker-compose service name)
service_name = os.getenv("NEXTCLOUD_MCP_SERVICE_NAME", "mcp")
port = os.getenv("NEXTCLOUD_MCP_PORT", "8000")
logger.debug(
f"Docker environment detected, using internal URL: http://{service_name}:{port}"
)
return f"http://{service_name}:{port}/webhooks/nextcloud"
# Fallback to configured server URL (for non-Docker deployments)
server_url = os.getenv("NEXTCLOUD_MCP_SERVER_URL", "http://localhost:8000")
return f"{server_url}/webhooks/nextcloud"
async def _get_authenticated_client(request: Request) -> httpx.AsyncClient:
"""Get an authenticated HTTP client for Nextcloud API calls.
Args:
request: Starlette request object
Returns:
Authenticated httpx.AsyncClient
Raises:
RuntimeError: If unable to create authenticated client
"""
# Get OAuth context from app state
oauth_ctx = getattr(request.app.state, "oauth_context", None)
# BasicAuth mode - use credentials from environment
if not oauth_ctx:
nextcloud_host = os.getenv("NEXTCLOUD_HOST")
username = os.getenv("NEXTCLOUD_USERNAME")
password = os.getenv("NEXTCLOUD_PASSWORD")
if not all([nextcloud_host, username, password]):
raise RuntimeError("BasicAuth credentials not configured")
assert nextcloud_host is not None # Type narrowing for type checker
return httpx.AsyncClient(
base_url=nextcloud_host,
auth=(username, password),
timeout=30.0,
)
# OAuth mode - get token from session
storage = oauth_ctx.get("storage")
session_id = request.cookies.get("mcp_session")
if not storage or not session_id:
raise RuntimeError("Session not found")
token_data = await storage.get_refresh_token(session_id)
if not token_data or "access_token" not in token_data:
raise RuntimeError("No access token found in session")
access_token = token_data["access_token"]
nextcloud_host = oauth_ctx.get("config", {}).get("nextcloud_host", "")
if not nextcloud_host:
raise RuntimeError("Nextcloud host not configured")
return httpx.AsyncClient(
base_url=nextcloud_host,
headers={"Authorization": f"Bearer {access_token}"},
timeout=30.0,
)
async def _get_enabled_presets(
webhooks_client: WebhooksClient,
storage=None,
) -> dict[str, list[int]]:
"""Get currently enabled webhook presets.
Reads from database first for better performance. Falls back to API if needed.
Args:
webhooks_client: Webhooks API client
storage: Optional RefreshTokenStorage instance
Returns:
Dictionary mapping preset_id to list of webhook IDs
"""
try:
# Try database first (faster, works offline)
if storage:
all_webhooks = await storage.list_all_webhooks()
enabled_presets: dict[str, list[int]] = {}
for webhook in all_webhooks:
preset_id = webhook["preset_id"]
webhook_id = webhook["webhook_id"]
if preset_id not in enabled_presets:
enabled_presets[preset_id] = []
enabled_presets[preset_id].append(webhook_id)
return enabled_presets
# Fallback to API query
registered_webhooks = await webhooks_client.list_webhooks()
webhook_uri = _get_webhook_uri()
# Group webhooks by preset based on matching events
enabled_presets: dict[str, list[int]] = {}
for preset_id, preset in WEBHOOK_PRESETS.items():
preset_event_classes = {event["event"] for event in preset["events"]}
matching_webhooks = []
for webhook in registered_webhooks:
# Check if webhook matches this preset
if (
webhook.get("uri") == webhook_uri
and webhook.get("event") in preset_event_classes
):
matching_webhooks.append(webhook["id"])
if matching_webhooks:
enabled_presets[preset_id] = matching_webhooks
return enabled_presets
except Exception as e:
logger.error(f"Failed to list webhooks: {e}")
return {}
@requires("authenticated", redirect="oauth_login")
async def webhook_management_pane(request: Request) -> HTMLResponse:
"""Webhook management pane - returns HTML for webhook configuration.
This endpoint checks if the user is an admin and returns either:
- Admin view: Webhook management interface with preset controls
- Non-admin view: Message indicating admin-only access
Args:
request: Starlette request object
Returns:
HTML response with webhook management interface or access denied message
"""
try:
# Get authenticated HTTP client
http_client = await _get_authenticated_client(request)
username = request.user.display_name
# Check admin permissions
is_admin = await is_nextcloud_admin(request, http_client)
if not is_admin:
return HTMLResponse(
content="""
<div class="info-message">
<p><strong>Admin Access Required</strong></p>
<p>Webhook management is only available to Nextcloud administrators.</p>
<p>Your account does not have admin privileges.</p>
</div>
"""
)
# Get webhooks client
webhooks_client = WebhooksClient(http_client, username)
# Get storage for database-backed webhook tracking
storage = _get_storage(request)
# Get installed apps to filter presets
installed_apps = await _get_installed_apps(http_client)
logger.debug(f"Installed apps: {installed_apps}")
# Get currently enabled presets (from database or API)
enabled_presets = await _get_enabled_presets(webhooks_client, storage)
# Filter presets based on installed apps
available_presets = filter_presets_by_installed_apps(installed_apps)
# Build preset cards HTML
preset_cards_html = ""
for preset_id, preset in available_presets:
is_enabled = preset_id in enabled_presets
num_webhooks = len(enabled_presets.get(preset_id, []))
# Status badge
if is_enabled:
status_badge = f'<span style="color: #4caf50; font-weight: bold;">✓ Enabled ({num_webhooks} webhooks)</span>'
action_button = f"""
<button
hx-delete="/app/webhooks/disable/{preset_id}"
hx-target="#preset-{preset_id}"
hx-swap="outerHTML"
class="button"
style="background-color: #ff9800;">
Disable
</button>
"""
else:
status_badge = '<span style="color: #999;">Not Enabled</span>'
action_button = f"""
<button
hx-post="/app/webhooks/enable/{preset_id}"
hx-target="#preset-{preset_id}"
hx-swap="outerHTML"
class="button button-primary">
Enable
</button>
"""
preset_cards_html += f"""
<div id="preset-{preset_id}" style="border: 1px solid #e0e0e0; border-radius: 6px; padding: 20px; margin: 15px 0;">
<h3 style="margin-top: 0; color: #0082c9;">{preset["name"]}</h3>
<p style="color: #666; margin: 10px 0;">{preset["description"]}</p>
<p style="font-size: 13px; color: #999;">
<strong>App:</strong> {preset["app"]} |
<strong>Events:</strong> {len(preset["events"])}
</p>
<div style="margin-top: 15px; display: flex; align-items: center; gap: 15px;">
<div>{status_badge}</div>
<div>{action_button}</div>
</div>
</div>
"""
# Get webhook endpoint URL for display
webhook_uri = _get_webhook_uri()
html_content = f"""
<h2>Webhook Management</h2>
<div class="info-message">
<p><strong>About Webhooks</strong></p>
<p>Webhooks enable real-time synchronization by notifying this server when content changes in Nextcloud.</p>
<p><strong>Endpoint:</strong> <code>{webhook_uri}</code></p>
</div>
<h3 style="margin-top: 30px;">Available Presets</h3>
<p style="color: #666;">Enable webhook presets with one click for common synchronization scenarios.</p>
<p style="color: #999; font-size: 13px; margin-top: 5px;">Showing {len(available_presets)} preset(s) for your installed apps ({len(installed_apps)} detected)</p>
{preset_cards_html}
"""
return HTMLResponse(content=html_content)
except Exception as e:
logger.error(f"Error loading webhook management pane: {e}", exc_info=True)
return HTMLResponse(
content=f"""
<div class="warning">
<p><strong>Error Loading Webhooks</strong></p>
<p>{str(e)}</p>
</div>
""",
status_code=500,
)
@requires("authenticated", redirect="oauth_login")
async def enable_webhook_preset(request: Request) -> HTMLResponse:
"""Enable a webhook preset by registering all webhooks.
Args:
request: Starlette request object (preset_id in path)
Returns:
HTML response with updated preset card
"""
preset_id = request.path_params["preset_id"]
try:
# Get authenticated HTTP client
http_client = await _get_authenticated_client(request)
username = request.user.display_name
# Check admin permissions
is_admin = await is_nextcloud_admin(request, http_client)
if not is_admin:
return HTMLResponse(
content='<div class="warning">Admin access required</div>',
status_code=403,
)
# Get preset configuration
preset = get_preset(preset_id)
if not preset:
return HTMLResponse(
content=f'<div class="warning">Unknown preset: {preset_id}</div>',
status_code=404,
)
# Register webhooks
webhooks_client = WebhooksClient(http_client, username)
webhook_uri = _get_webhook_uri()
registered_ids = []
for event_config in preset["events"]:
webhook_data = await webhooks_client.create_webhook(
event=event_config["event"],
uri=webhook_uri,
event_filter=event_config["filter"] if event_config["filter"] else None,
)
webhook_id = webhook_data["id"]
registered_ids.append(webhook_id)
logger.info(f"Registered webhook {webhook_id} for {event_config['event']}")
# Persist webhook IDs to database
storage = _get_storage(request)
if storage:
for webhook_id in registered_ids:
await storage.store_webhook(webhook_id, preset_id)
logger.info(
f"Persisted {len(registered_ids)} webhook(s) for preset '{preset_id}' to database"
)
# Return updated card
num_webhooks = len(registered_ids)
return HTMLResponse(
content=f"""
<div id="preset-{preset_id}" style="border: 1px solid #e0e0e0; border-radius: 6px; padding: 20px; margin: 15px 0;">
<h3 style="margin-top: 0; color: #0082c9;">{preset["name"]}</h3>
<p style="color: #666; margin: 10px 0;">{preset["description"]}</p>
<p style="font-size: 13px; color: #999;">
<strong>App:</strong> {preset["app"]} |
<strong>Events:</strong> {len(preset["events"])}
</p>
<div style="margin-top: 15px; display: flex; align-items: center; gap: 15px;">
<div><span style="color: #4caf50; font-weight: bold;">✓ Enabled ({num_webhooks} webhooks)</span></div>
<div>
<button
hx-delete="/app/webhooks/disable/{preset_id}"
hx-target="#preset-{preset_id}"
hx-swap="outerHTML"
class="button"
style="background-color: #ff9800;">
Disable
</button>
</div>
</div>
</div>
"""
)
except Exception as e:
logger.error(f"Failed to enable preset {preset_id}: {e}", exc_info=True)
return HTMLResponse(
content=f'<div class="warning">Failed to enable preset: {str(e)}</div>',
status_code=500,
)
@requires("authenticated", redirect="oauth_login")
async def disable_webhook_preset(request: Request) -> HTMLResponse:
"""Disable a webhook preset by deleting all registered webhooks.
Args:
request: Starlette request object (preset_id in path)
Returns:
HTML response with updated preset card
"""
preset_id = request.path_params["preset_id"]
try:
# Get authenticated HTTP client
http_client = await _get_authenticated_client(request)
username = request.user.display_name
# Check admin permissions
is_admin = await is_nextcloud_admin(request, http_client)
if not is_admin:
return HTMLResponse(
content='<div class="warning">Admin access required</div>',
status_code=403,
)
# Get preset configuration
preset = get_preset(preset_id)
if not preset:
return HTMLResponse(
content=f'<div class="warning">Unknown preset: {preset_id}</div>',
status_code=404,
)
# Find and delete matching webhooks
webhooks_client = WebhooksClient(http_client, username)
# Get webhook IDs from database first (more reliable)
storage = _get_storage(request)
if storage:
webhook_ids = await storage.get_webhooks_by_preset(preset_id)
else:
# Fallback to API query if storage not available
enabled_presets = await _get_enabled_presets(webhooks_client)
webhook_ids = enabled_presets.get(preset_id, [])
for webhook_id in webhook_ids:
await webhooks_client.delete_webhook(webhook_id)
logger.info(f"Deleted webhook {webhook_id} from preset {preset_id}")
# Remove from database
if storage:
deleted_count = await storage.clear_preset_webhooks(preset_id)
logger.info(
f"Removed {deleted_count} webhook(s) for preset '{preset_id}' from database"
)
# Return updated card
return HTMLResponse(
content=f"""
<div id="preset-{preset_id}" style="border: 1px solid #e0e0e0; border-radius: 6px; padding: 20px; margin: 15px 0;">
<h3 style="margin-top: 0; color: #0082c9;">{preset["name"]}</h3>
<p style="color: #666; margin: 10px 0;">{preset["description"]}</p>
<p style="font-size: 13px; color: #999;">
<strong>App:</strong> {preset["app"]} |
<strong>Events:</strong> {len(preset["events"])}
</p>
<div style="margin-top: 15px; display: flex; align-items: center; gap: 15px;">
<div><span style="color: #999;">Not Enabled</span></div>
<div>
<button
hx-post="/app/webhooks/enable/{preset_id}"
hx-target="#preset-{preset_id}"
hx-swap="outerHTML"
class="button button-primary">
Enable
</button>
</div>
</div>
</div>
"""
)
except Exception as e:
logger.error(f"Failed to disable preset {preset_id}: {e}", exc_info=True)
return HTMLResponse(
content=f'<div class="warning">Failed to disable preset: {str(e)}</div>',
status_code=500,
)
+257
View File
@@ -0,0 +1,257 @@
import os
import click
import uvicorn
from nextcloud_mcp_server.config import (
get_settings,
)
from nextcloud_mcp_server.observability import get_uvicorn_logging_config
from .app import get_app
@click.command()
@click.option(
"--host", "-h", default="127.0.0.1", show_default=True, help="Server host"
)
@click.option(
"--port", "-p", type=int, default=8000, show_default=True, help="Server port"
)
@click.option(
"--log-level",
"-l",
default="info",
show_default=True,
type=click.Choice(["critical", "error", "warning", "info", "debug", "trace"]),
help="Logging level",
)
@click.option(
"--transport",
"-t",
default="sse",
show_default=True,
type=click.Choice(["sse", "streamable-http", "http"]),
help="MCP transport protocol",
)
@click.option(
"--enable-app",
"-e",
multiple=True,
type=click.Choice(
["notes", "tables", "webdav", "calendar", "contacts", "cookbook", "deck"]
),
help="Enable specific Nextcloud app APIs. Can be specified multiple times. If not specified, all apps are enabled.",
)
@click.option(
"--oauth/--no-oauth",
default=None,
help="Force OAuth mode (if enabled) or BasicAuth mode (if disabled). By default, auto-detected based on environment variables.",
)
@click.option(
"--oauth-client-id",
envvar="NEXTCLOUD_OIDC_CLIENT_ID",
help="OAuth client ID (can also use NEXTCLOUD_OIDC_CLIENT_ID env var)",
)
@click.option(
"--oauth-client-secret",
envvar="NEXTCLOUD_OIDC_CLIENT_SECRET",
help="OAuth client secret (can also use NEXTCLOUD_OIDC_CLIENT_SECRET env var)",
)
@click.option(
"--mcp-server-url",
envvar="NEXTCLOUD_MCP_SERVER_URL",
default="http://localhost:8000",
show_default=True,
help="MCP server URL for OAuth callbacks (can also use NEXTCLOUD_MCP_SERVER_URL env var)",
)
@click.option(
"--nextcloud-host",
envvar="NEXTCLOUD_HOST",
help="Nextcloud instance URL (can also use NEXTCLOUD_HOST env var)",
)
@click.option(
"--nextcloud-username",
envvar="NEXTCLOUD_USERNAME",
help="Nextcloud username for BasicAuth (can also use NEXTCLOUD_USERNAME env var)",
)
@click.option(
"--nextcloud-password",
envvar="NEXTCLOUD_PASSWORD",
help="Nextcloud password for BasicAuth (can also use NEXTCLOUD_PASSWORD env var)",
)
@click.option(
"--oauth-scopes",
envvar="NEXTCLOUD_OIDC_SCOPES",
default="openid profile email notes:read notes:write calendar:read calendar:write todo:read todo:write contacts:read contacts:write cookbook:read cookbook:write deck:read deck:write tables:read tables:write files:read files:write sharing:read sharing:write",
show_default=True,
help="OAuth scopes to request during client registration. These define the maximum allowed scopes for the client. Note: Actual supported scopes are discovered dynamically from MCP tools at runtime. (can also use NEXTCLOUD_OIDC_SCOPES env var)",
)
@click.option(
"--oauth-token-type",
envvar="NEXTCLOUD_OIDC_TOKEN_TYPE",
default="bearer",
show_default=True,
type=click.Choice(["bearer", "jwt"], case_sensitive=False),
help="OAuth token type (can also use NEXTCLOUD_OIDC_TOKEN_TYPE env var)",
)
@click.option(
"--public-issuer-url",
envvar="NEXTCLOUD_PUBLIC_ISSUER_URL",
help="Public issuer URL for OAuth (can also use NEXTCLOUD_PUBLIC_ISSUER_URL env var)",
)
def run(
host: str,
port: int,
log_level: str,
transport: str,
enable_app: tuple[str, ...],
oauth: bool | None,
oauth_client_id: str | None,
oauth_client_secret: str | None,
mcp_server_url: str,
nextcloud_host: str | None,
nextcloud_username: str | None,
nextcloud_password: str | None,
oauth_scopes: str,
oauth_token_type: str,
public_issuer_url: str | None,
):
"""
Run the Nextcloud MCP server.
\b
Authentication Modes:
- BasicAuth: Set NEXTCLOUD_USERNAME and NEXTCLOUD_PASSWORD
- OAuth: Leave USERNAME/PASSWORD unset (requires OIDC app enabled)
\b
Examples:
# BasicAuth mode with CLI options
$ nextcloud-mcp-server --nextcloud-host=https://cloud.example.com \\
--nextcloud-username=admin --nextcloud-password=secret
# BasicAuth mode with env vars (recommended for credentials)
$ export NEXTCLOUD_HOST=https://cloud.example.com
$ export NEXTCLOUD_USERNAME=admin
$ export NEXTCLOUD_PASSWORD=secret
$ nextcloud-mcp-server --host 0.0.0.0 --port 8000
# OAuth mode with auto-registration
$ nextcloud-mcp-server --nextcloud-host=https://cloud.example.com --oauth
# OAuth mode with pre-configured client
$ nextcloud-mcp-server --nextcloud-host=https://cloud.example.com --oauth \\
--oauth-client-id=xxx --oauth-client-secret=yyy
# OAuth mode with custom scopes and JWT tokens
$ nextcloud-mcp-server --nextcloud-host=https://cloud.example.com --oauth \\
--oauth-scopes="openid notes:read notes:write" --oauth-token-type=jwt
# OAuth with public issuer URL (for Docker/proxy setups)
$ nextcloud-mcp-server --nextcloud-host=http://app --oauth \\
--public-issuer-url=http://localhost:8080
"""
# Set env vars from CLI options if provided
if nextcloud_host:
os.environ["NEXTCLOUD_HOST"] = nextcloud_host
if nextcloud_username:
os.environ["NEXTCLOUD_USERNAME"] = nextcloud_username
if nextcloud_password:
os.environ["NEXTCLOUD_PASSWORD"] = nextcloud_password
if oauth_client_id:
os.environ["NEXTCLOUD_OIDC_CLIENT_ID"] = oauth_client_id
if oauth_client_secret:
os.environ["NEXTCLOUD_OIDC_CLIENT_SECRET"] = oauth_client_secret
if oauth_scopes:
os.environ["NEXTCLOUD_OIDC_SCOPES"] = oauth_scopes
if oauth_token_type:
os.environ["NEXTCLOUD_OIDC_TOKEN_TYPE"] = oauth_token_type
if mcp_server_url:
os.environ["NEXTCLOUD_MCP_SERVER_URL"] = mcp_server_url
if public_issuer_url:
os.environ["NEXTCLOUD_PUBLIC_ISSUER_URL"] = public_issuer_url
# Force OAuth mode if explicitly requested
if oauth is True:
# Clear username/password to force OAuth mode
if "NEXTCLOUD_USERNAME" in os.environ:
click.echo(
"Warning: --oauth flag set, ignoring NEXTCLOUD_USERNAME", err=True
)
del os.environ["NEXTCLOUD_USERNAME"]
if "NEXTCLOUD_PASSWORD" in os.environ:
click.echo(
"Warning: --oauth flag set, ignoring NEXTCLOUD_PASSWORD", err=True
)
del os.environ["NEXTCLOUD_PASSWORD"]
# Validate OAuth configuration
nextcloud_host = os.getenv("NEXTCLOUD_HOST")
if not nextcloud_host:
raise click.ClickException(
"OAuth mode requires NEXTCLOUD_HOST environment variable to be set"
)
# Check if we have client credentials OR if dynamic registration is possible
has_client_creds = os.getenv("NEXTCLOUD_OIDC_CLIENT_ID") and os.getenv(
"NEXTCLOUD_OIDC_CLIENT_SECRET"
)
if not has_client_creds:
# No client credentials - will attempt dynamic registration
# Show helpful message before server starts
click.echo("", err=True)
click.echo("OAuth Configuration:", err=True)
click.echo(" Mode: Dynamic Client Registration", err=True)
click.echo(" Host: " + nextcloud_host, err=True)
click.echo(" Storage: SQLite (TOKEN_STORAGE_DB)", err=True)
click.echo("", err=True)
click.echo(
"Note: Make sure 'Dynamic Client Registration' is enabled", err=True
)
click.echo(" in your Nextcloud OIDC app settings.", err=True)
click.echo("", err=True)
else:
click.echo("", err=True)
click.echo("OAuth Configuration:", err=True)
click.echo(" Mode: Pre-configured Client", err=True)
click.echo(" Host: " + nextcloud_host, err=True)
click.echo(
" Client ID: "
+ os.getenv("NEXTCLOUD_OIDC_CLIENT_ID", "")[:16]
+ "...",
err=True,
)
click.echo("", err=True)
elif oauth is False:
# Force BasicAuth mode - verify credentials exist
if not os.getenv("NEXTCLOUD_USERNAME") or not os.getenv("NEXTCLOUD_PASSWORD"):
raise click.ClickException(
"--no-oauth flag set but NEXTCLOUD_USERNAME or NEXTCLOUD_PASSWORD not set"
)
enabled_apps = list(enable_app) if enable_app else None
app = get_app(transport=transport, enabled_apps=enabled_apps)
# Get observability settings and create uvicorn logging config
settings = get_settings()
uvicorn_log_config = get_uvicorn_logging_config(
log_format=settings.log_format,
log_level=settings.log_level,
include_trace_context=settings.log_include_trace_context,
)
uvicorn.run(
app=app,
host=host,
port=port,
log_level=log_level,
log_config=uvicorn_log_config,
)
if __name__ == "__main__":
run()
+4
View File
@@ -9,6 +9,7 @@ from httpx import (
BasicAuth,
Request,
Response,
Timeout,
)
from ..controllers.notes_search import NotesSearchController
@@ -22,6 +23,7 @@ from .sharing import SharingClient
from .tables import TablesClient
from .users import UsersClient
from .webdav import WebDAVClient
from .webhooks import WebhooksClient
logger = logging.getLogger(__name__)
@@ -66,6 +68,7 @@ class NextcloudClient:
auth=auth,
transport=AsyncDisableCookieTransport(AsyncHTTPTransport()),
event_hooks={"request": [log_request], "response": [log_response]},
timeout=Timeout(timeout=30, connect=5),
)
# Initialize app clients
@@ -81,6 +84,7 @@ class NextcloudClient:
self.users = UsersClient(self._client, username)
self.groups = GroupsClient(self._client, username)
self.sharing = SharingClient(self._client, username)
self.webhooks = WebhooksClient(self._client, username)
# Initialize controllers
self._notes_search = NotesSearchController()
+57 -4
View File
@@ -7,6 +7,12 @@ from functools import wraps
from httpx import AsyncClient, HTTPStatusError, RequestError, codes
from nextcloud_mcp_server.observability.metrics import (
record_nextcloud_api_call,
record_nextcloud_api_retry,
)
from nextcloud_mcp_server.observability.tracing import trace_nextcloud_api_call
logger = logging.getLogger(__name__)
@@ -38,6 +44,9 @@ def retry_on_429(func):
logger.warning(
f"429 Client Error: Too Many Requests, Number of attempts: {retries}"
)
# Record retry metric (extract app name from args if available)
if len(args) > 0 and hasattr(args[0], "app_name"):
record_nextcloud_api_retry(app=args[0].app_name, reason="429")
time.sleep(5)
elif e.response.status_code == 404:
# 404 errors are often expected (e.g., checking if attachments exist)
@@ -72,6 +81,9 @@ def retry_on_429(func):
class BaseNextcloudClient(ABC):
"""Base class for all Nextcloud app clients."""
# Subclasses should set this to identify the app for metrics/tracing
app_name: str = "unknown"
def __init__(self, http_client: AsyncClient, username: str):
"""Initialize with shared HTTP client and username.
@@ -88,7 +100,7 @@ class BaseNextcloudClient(ABC):
@retry_on_429
async def _make_request(self, method: str, url: str, **kwargs):
"""Common request wrapper with logging and error handling.
"""Common request wrapper with logging, tracing, and error handling.
Args:
method: HTTP method
@@ -99,6 +111,47 @@ class BaseNextcloudClient(ABC):
Response object
"""
logger.debug(f"Making {method} request to {url}")
response = await self._client.request(method, url, **kwargs)
response.raise_for_status()
return response
# Start timer for metrics
start_time = time.time()
status_code = 0
try:
# Wrap request in trace span
with trace_nextcloud_api_call(
app=self.app_name,
method=method,
path=url,
):
response = await self._client.request(method, url, **kwargs)
status_code = response.status_code
response.raise_for_status()
# Record successful API call metrics
duration = time.time() - start_time
record_nextcloud_api_call(
app=self.app_name,
method=method,
status_code=status_code,
duration=duration,
)
return response
except (HTTPStatusError, RequestError) as e:
# Record error metrics
if isinstance(e, HTTPStatusError):
status_code = e.response.status_code
else:
status_code = 0 # Connection error, no status code
duration = time.time() - start_time
record_nextcloud_api_call(
app=self.app_name,
method=method,
status_code=status_code,
duration=duration,
)
# Re-raise the exception
raise
+2
View File
@@ -13,6 +13,8 @@ logger = logging.getLogger(__name__)
class ContactsClient(BaseNextcloudClient):
"""Client for NextCloud CardDAV contact operations."""
app_name = "contacts"
def _get_carddav_base_path(self) -> str:
"""Helper to get the base CardDAV path for contacts."""
return f"/remote.php/dav/addressbooks/users/{self.username}"
+2
View File
@@ -13,6 +13,8 @@ logger = logging.getLogger(__name__)
class CookbookClient(BaseNextcloudClient):
"""Client for Nextcloud Cookbook app operations."""
app_name = "cookbook"
async def get_version(self) -> Dict[str, Any]:
"""Get Cookbook app and API version."""
response = await self._make_request("GET", "/apps/cookbook/api/version")
+2
View File
@@ -17,6 +17,8 @@ from nextcloud_mcp_server.models.deck import (
class DeckClient(BaseNextcloudClient):
"""Client for Nextcloud Deck app operations."""
app_name = "deck"
def _get_deck_headers(
self, additional_headers: Optional[Dict[str, str]] = None
) -> Dict[str, str]:
+2
View File
@@ -11,6 +11,8 @@ logger = logging.getLogger(__name__)
class GroupsClient(BaseNextcloudClient):
"""Client for Nextcloud Groups API operations."""
app_name = "groups"
@retry_on_429
async def search_groups(
self,
+45 -4
View File
@@ -11,23 +11,64 @@ logger = logging.getLogger(__name__)
class NotesClient(BaseNextcloudClient):
"""Client for Nextcloud Notes app operations."""
app_name = "notes"
async def get_settings(self) -> Dict[str, Any]:
"""Get Notes app settings."""
response = await self._make_request("GET", "/apps/notes/api/v1/settings")
return response.json()
async def get_all_notes(self) -> AsyncIterator[Dict[str, Any]]:
"""Get all notes, yielding them one at a time."""
async def get_all_notes(
self, prune_before: Optional[int] = None
) -> AsyncIterator[Dict[str, Any]]:
"""Get all notes, yielding them one at a time.
The Notes API returns changed notes with full data in chunks, and ALL note IDs
(with only 'id' field) in the last chunk for deletion detection. This causes
duplicates which we handle by tracking seen IDs (first occurrence with full
data is kept, later pruned duplicates are skipped).
Args:
prune_before: Optional Unix timestamp. Notes unchanged since this time
are pruned (only 'id' field returned in last chunk).
Reduces data transfer for large note collections.
Yields:
Note dictionaries with full data (deduplicated).
"""
cursor = ""
seen_ids: set[int] = set()
while True:
params: Dict[str, Any] = {"chunkSize": 10}
if cursor:
params["chunkCursor"] = cursor
if prune_before is not None:
params["pruneBefore"] = prune_before
response = await self._make_request(
"GET",
"/apps/notes/api/v1/notes",
params={"chunkSize": 10, "chunkCursor": cursor},
params=params,
)
for note in response.json():
response_data = response.json()
for note in response_data:
note_id = note.get("id")
if note_id is None:
logger.warning(f"Skipping note without ID: {note}")
continue
# Skip duplicates (API returns all IDs in last chunk for deletion detection)
if note_id in seen_ids:
logger.debug(
f"Skipping duplicate note {note_id} (pruned version in last chunk)"
)
continue
seen_ids.add(note_id)
yield note
if "X-Notes-Chunk-Cursor" not in response.headers:
break
cursor = response.headers["X-Notes-Chunk-Cursor"]
+2
View File
@@ -11,6 +11,8 @@ logger = logging.getLogger(__name__)
class SharingClient(BaseNextcloudClient):
"""Client for Nextcloud OCS Sharing API operations."""
app_name = "sharing"
@retry_on_429
async def create_share(
self,
+2
View File
@@ -11,6 +11,8 @@ logger = logging.getLogger(__name__)
class TablesClient(BaseNextcloudClient):
"""Client for Nextcloud Tables app operations."""
app_name = "tables"
async def list_tables(self) -> List[Dict[str, Any]]:
"""List all tables available to the user."""
response = await self._make_request(
+2
View File
@@ -7,6 +7,8 @@ from nextcloud_mcp_server.models.users import UserDetails
class UsersClient(BaseNextcloudClient):
"""Client for Nextcloud User API operations."""
app_name = "users"
def _get_user_headers(
self, additional_headers: Optional[Dict[str, str]] = None
) -> Dict[str, str]:
+2
View File
@@ -15,6 +15,8 @@ logger = logging.getLogger(__name__)
class WebDAVClient(BaseNextcloudClient):
"""Client for Nextcloud WebDAV operations."""
app_name = "webdav"
async def delete_resource(self, path: str) -> Dict[str, Any]:
"""Delete a resource (file or directory) via WebDAV DELETE."""
# Ensure path ends with a slash if it's a directory
+109
View File
@@ -0,0 +1,109 @@
"""Client for Nextcloud Webhook Listeners API operations."""
from typing import Any, Dict, List, Optional
from nextcloud_mcp_server.client.base import BaseNextcloudClient
class WebhooksClient(BaseNextcloudClient):
"""Client for Nextcloud webhook_listeners app API operations."""
app_name = "webhooks"
def _get_webhook_headers(
self, additional_headers: Optional[Dict[str, str]] = None
) -> Dict[str, str]:
"""Get standard headers required for Webhook Listeners API calls."""
headers = {"OCS-APIRequest": "true", "Accept": "application/json"}
if additional_headers:
headers.update(additional_headers)
return headers
async def list_webhooks(self) -> List[Dict[str, Any]]:
"""List all registered webhooks for the current user.
Returns:
List of webhook registrations with id, uri, event, filters, etc.
"""
headers = self._get_webhook_headers()
response = await self._make_request(
"GET",
"/ocs/v2.php/apps/webhook_listeners/api/v1/webhooks",
headers=headers,
)
data = response.json()["ocs"]["data"]
return data if isinstance(data, list) else []
async def create_webhook(
self,
event: str,
uri: str,
http_method: str = "POST",
auth_method: str = "none",
headers: Optional[Dict[str, str]] = None,
event_filter: Optional[Dict[str, Any]] = None,
) -> Dict[str, Any]:
"""Register a new webhook for the specified event.
Args:
event: Fully qualified event class name (e.g., "OCP\\Files\\Events\\Node\\NodeCreatedEvent")
uri: Webhook endpoint URL to receive event notifications
http_method: HTTP method for webhook delivery (default: "POST")
auth_method: Authentication method ("none", "bearer", etc.)
headers: Custom headers to include in webhook requests (e.g., Authorization header)
event_filter: JSON object specifying event filters (e.g., {"user.uid": "bob"})
Returns:
Webhook registration details including webhook ID
"""
data: Dict[str, Any] = {
"httpMethod": http_method,
"uri": uri,
"event": event,
"authMethod": auth_method,
}
if headers:
data["headers"] = headers
if event_filter:
data["eventFilter"] = event_filter
request_headers = self._get_webhook_headers()
response = await self._make_request(
"POST",
"/ocs/v2.php/apps/webhook_listeners/api/v1/webhooks",
json=data,
headers=request_headers,
)
return response.json()["ocs"]["data"]
async def delete_webhook(self, webhook_id: int) -> None:
"""Delete a webhook registration.
Args:
webhook_id: ID of the webhook to delete
"""
headers = self._get_webhook_headers()
await self._make_request(
"DELETE",
f"/ocs/v2.php/apps/webhook_listeners/api/v1/webhooks/{webhook_id}",
headers=headers,
)
async def get_webhook(self, webhook_id: int) -> Dict[str, Any]:
"""Get details of a specific webhook registration.
Args:
webhook_id: ID of the webhook to retrieve
Returns:
Webhook registration details
"""
headers = self._get_webhook_headers()
response = await self._make_request(
"GET",
f"/ocs/v2.php/apps/webhook_listeners/api/v1/webhooks/{webhook_id}",
headers=headers,
)
return response.json()["ocs"]["data"]
+100 -3
View File
@@ -153,7 +153,13 @@ class Settings:
# Token exchange cache settings
token_exchange_cache_ttl: int = 300 # seconds (5 minutes default)
# Token settings
# Token and webhook storage settings
# TOKEN_ENCRYPTION_KEY: Optional - Only required for OAuth token storage operations.
# Webhook tracking works without encryption key.
# If set, must be a valid base64-encoded Fernet key (32 bytes).
# TOKEN_STORAGE_DB: Path to SQLite database for persistent storage.
# Used for webhook tracking (all modes) and OAuth token storage.
# Defaults to /tmp/tokens.db
token_encryption_key: Optional[str] = None
token_storage_db: Optional[str] = None
@@ -174,6 +180,22 @@ class Settings:
ollama_embedding_model: str = "nomic-embed-text"
ollama_verify_ssl: bool = True
# Document chunking settings (for vector embeddings)
document_chunk_size: int = 512 # Words per chunk
document_chunk_overlap: int = 50 # Overlapping words between chunks
# Observability settings
metrics_enabled: bool = True
metrics_port: int = 9090
otel_exporter_otlp_endpoint: Optional[str] = None
otel_exporter_verify_ssl: bool = False
otel_service_name: str = "nextcloud-mcp-server"
otel_traces_sampler: str = "always_on"
otel_traces_sampler_arg: float = 1.0
log_format: str = "text" # "json" or "text"
log_level: str = "INFO"
log_include_trace_context: bool = True
def __post_init__(self):
"""Validate Qdrant configuration and set defaults."""
logger = logging.getLogger(__name__)
@@ -188,7 +210,7 @@ class Settings:
# Default to :memory: if neither set
if not self.qdrant_url and not self.qdrant_location:
self.qdrant_location = ":memory:"
logger.info("Using default Qdrant mode: in-memory (:memory:)")
logger.debug("Using default Qdrant mode: in-memory (:memory:)")
# Warn if API key set in local mode
if self.qdrant_location and self.qdrant_api_key:
@@ -197,6 +219,65 @@ class Settings:
"API key is only relevant for network mode and will be ignored."
)
# Validate chunking configuration
if self.document_chunk_overlap >= self.document_chunk_size:
raise ValueError(
f"DOCUMENT_CHUNK_OVERLAP ({self.document_chunk_overlap}) must be less than "
f"DOCUMENT_CHUNK_SIZE ({self.document_chunk_size}). "
f"Overlap should be 10-20% of chunk size for optimal results."
)
if self.document_chunk_size < 100:
logger.warning(
f"DOCUMENT_CHUNK_SIZE is set to {self.document_chunk_size} words, which is quite small. "
f"Smaller chunks may lose context. Consider using at least 256 words."
)
if self.document_chunk_overlap < 0:
raise ValueError(
f"DOCUMENT_CHUNK_OVERLAP ({self.document_chunk_overlap}) cannot be negative."
)
def get_collection_name(self) -> str:
"""
Get Qdrant collection name.
Auto-generates from deployment ID + model name unless explicitly set.
Deployment ID uses OTEL_SERVICE_NAME if configured, otherwise hostname.
This enables:
- Safe embedding model switching (new model new collection)
- Multi-server deployments (unique deployment IDs)
- Clear collection naming (shows deployment and model)
Format: {deployment-id}-{model-name}
Examples:
- "my-deployment-nomic-embed-text" (OTEL_SERVICE_NAME set)
- "mcp-container-all-minilm" (hostname fallback)
Returns:
Collection name string
"""
import socket
# Use explicit override if user configured non-default value
if self.qdrant_collection != "nextcloud_content":
return self.qdrant_collection
# Determine deployment ID (OTEL service name or hostname fallback)
if self.otel_service_name != "nextcloud-mcp-server": # Non-default
deployment_id = self.otel_service_name
else:
# Fallback to hostname for simple Docker deployments without OTEL config
deployment_id = socket.gethostname()
# Sanitize deployment ID and model name
deployment_id = deployment_id.lower().replace(" ", "-").replace("_", "-")
model_name = self.ollama_embedding_model.replace("/", "-").replace(":", "-")
return f"{deployment_id}-{model_name}"
def get_settings() -> Settings:
"""Get application settings from environment variables.
@@ -230,7 +311,7 @@ def get_settings() -> Settings:
),
# Token exchange cache settings
token_exchange_cache_ttl=int(os.getenv("TOKEN_EXCHANGE_CACHE_TTL", "300")),
# Token settings
# Token and webhook storage settings (encryption key optional for webhook-only usage)
token_encryption_key=os.getenv("TOKEN_ENCRYPTION_KEY"),
token_storage_db=os.getenv("TOKEN_STORAGE_DB", "/tmp/tokens.db"),
# Vector sync settings (ADR-007)
@@ -253,4 +334,20 @@ def get_settings() -> Settings:
ollama_base_url=os.getenv("OLLAMA_BASE_URL"),
ollama_embedding_model=os.getenv("OLLAMA_EMBEDDING_MODEL", "nomic-embed-text"),
ollama_verify_ssl=os.getenv("OLLAMA_VERIFY_SSL", "true").lower() == "true",
# Document chunking settings
document_chunk_size=int(os.getenv("DOCUMENT_CHUNK_SIZE", "512")),
document_chunk_overlap=int(os.getenv("DOCUMENT_CHUNK_OVERLAP", "50")),
# Observability settings
metrics_enabled=os.getenv("METRICS_ENABLED", "true").lower() == "true",
metrics_port=int(os.getenv("METRICS_PORT", "9090")),
otel_exporter_otlp_endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT"),
otel_exporter_verify_ssl=os.getenv("OTEL_EXPORTER_VERIFY_SSL", "false").lower()
== "true",
otel_service_name=os.getenv("OTEL_SERVICE_NAME", "nextcloud-mcp-server"),
otel_traces_sampler=os.getenv("OTEL_TRACES_SAMPLER", "always_on"),
otel_traces_sampler_arg=float(os.getenv("OTEL_TRACES_SAMPLER_ARG", "1.0")),
log_format=os.getenv("LOG_FORMAT", "text"),
log_level=os.getenv("LOG_LEVEL", "INFO"),
log_include_trace_context=os.getenv("LOG_INCLUDE_TRACE_CONTEXT", "true").lower()
== "true",
)
@@ -17,6 +17,7 @@ class OllamaEmbeddingProvider(EmbeddingProvider):
base_url: str,
model: str = "nomic-embed-text",
verify_ssl: bool = True,
timeout=httpx.Timeout(timeout=120, connect=5),
):
"""
Initialize Ollama embedding provider.
@@ -29,12 +30,14 @@ class OllamaEmbeddingProvider(EmbeddingProvider):
self.base_url = base_url.rstrip("/")
self.model = model
self.verify_ssl = verify_ssl
self.client = httpx.AsyncClient(verify=verify_ssl, timeout=30.0)
self._dimension = 768 # nomic-embed-text default
self.client = httpx.AsyncClient(verify=verify_ssl, timeout=timeout)
self._dimension: int | None = None # Will be detected dynamically
logger.info(
f"Initialized Ollama provider: {base_url} (model={model}, verify_ssl={verify_ssl})"
)
self._check_model_is_loaded(autoload=True)
async def embed(self, text: str) -> list[float]:
"""
Generate embedding vector for text.
@@ -71,15 +74,55 @@ class OllamaEmbeddingProvider(EmbeddingProvider):
embeddings.append(embedding)
return embeddings
async def _detect_dimension(self):
"""
Detect embedding dimension by generating a test embedding.
This method queries the model to determine the actual dimension
instead of relying on hardcoded values.
"""
if self._dimension is None:
logger.debug(f"Detecting embedding dimension for model {self.model}...")
test_embedding = await self.embed("test")
self._dimension = len(test_embedding)
logger.info(
f"Detected embedding dimension: {self._dimension} for model {self.model}"
)
def get_dimension(self) -> int:
"""
Get embedding dimension.
Returns:
Vector dimension (768 for nomic-embed-text)
Vector dimension for the configured model
Raises:
RuntimeError: If dimension not detected yet (call _detect_dimension first)
"""
if self._dimension is None:
raise RuntimeError(
f"Embedding dimension not detected yet for model {self.model}. "
"Call _detect_dimension() first or generate an embedding."
)
return self._dimension
def _check_model_is_loaded(self, autoload: bool = True):
response = httpx.get(f"{self.base_url}/api/tags")
response.raise_for_status()
models = [model["name"] for model in response.json().get("models", [])]
logger.info("Ollama has following models pre-loaded: %s", models)
if (self.model not in models) and autoload:
logger.warning(
"Embedding model '%s' not yet available in ollama, attempting to pull now...",
self.model,
)
response = httpx.post(
f"{self.base_url}/api/pull", json={"model": self.model}
)
response.raise_for_status()
async def close(self):
"""Close HTTP client."""
await self.client.aclose()
@@ -0,0 +1,31 @@
"""
Observability module for the Nextcloud MCP Server.
This module provides:
- Prometheus metrics collection
- OpenTelemetry distributed tracing
- Enhanced structured logging with trace correlation
- Monitoring middleware for Starlette/FastAPI
Usage:
from nextcloud_mcp_server.observability import setup_observability
# In app.py lifespan
setup_observability(app, config)
"""
from nextcloud_mcp_server.observability.logging_config import (
get_uvicorn_logging_config,
setup_logging,
)
from nextcloud_mcp_server.observability.metrics import setup_metrics
from nextcloud_mcp_server.observability.middleware import ObservabilityMiddleware
from nextcloud_mcp_server.observability.tracing import setup_tracing
__all__ = [
"setup_logging",
"get_uvicorn_logging_config",
"setup_metrics",
"setup_tracing",
"ObservabilityMiddleware",
]
@@ -0,0 +1,327 @@
"""
Enhanced logging configuration for the Nextcloud MCP Server.
This module provides:
- Structured JSON logging with python-json-logger
- Trace context injection (trace_id, span_id) for correlation with distributed traces
- Configurable log formats (JSON or text)
- Log level configuration per component
"""
import logging
import sys
from typing import Any
from pythonjsonlogger.json import JsonFormatter
from nextcloud_mcp_server.observability.tracing import get_trace_context
class HealthCheckFilter(logging.Filter):
"""
Logging filter that excludes health check endpoint requests.
This prevents health check polls from cluttering logs while keeping
access logs for all other endpoints.
"""
def filter(self, record: logging.LogRecord) -> bool:
"""
Filter out health check requests from uvicorn access logs.
Args:
record: LogRecord instance
Returns:
False if this is a health check request, True otherwise
"""
# Check if the log message contains health check endpoints
message = record.getMessage()
return not any(
endpoint in message
for endpoint in ["/health/live", "/health/ready", "/metrics"]
)
class TraceContextFormatter(JsonFormatter):
"""
JSON formatter that injects OpenTelemetry trace context into log records.
This allows logs to be correlated with distributed traces by including
trace_id and span_id in each log entry.
"""
def add_fields(
self,
log_record: dict[str, Any],
record: logging.LogRecord,
message_dict: dict[str, Any],
) -> None:
"""
Add custom fields to the log record, including trace context.
Args:
log_record: Dictionary to be serialized as JSON
record: LogRecord instance
message_dict: Dictionary of extra fields from log call
"""
# Call parent to add standard fields
super().add_fields(log_record, record, message_dict)
# Add trace context if available
trace_context = get_trace_context()
if trace_context:
log_record["trace_id"] = trace_context.get("trace_id")
log_record["span_id"] = trace_context.get("span_id")
# Add standard fields with consistent naming
log_record["timestamp"] = self.formatTime(record)
log_record["level"] = record.levelname
log_record["logger"] = record.name
log_record["message"] = record.getMessage()
# Include exception info if present
if record.exc_info:
log_record["exception"] = self.formatException(record.exc_info)
class TraceContextTextFormatter(logging.Formatter):
"""
Text formatter that includes OpenTelemetry trace context.
Format: [LEVEL] [timestamp] logger - message [trace_id=xxx span_id=yyy]
"""
def format(self, record: logging.LogRecord) -> str:
"""
Format log record with trace context.
Args:
record: LogRecord instance
Returns:
Formatted log string
"""
# Format base message
base_message = super().format(record)
# Add trace context if available
trace_context = get_trace_context()
if trace_context:
trace_id = trace_context.get("trace_id", "")
span_id = trace_context.get("span_id", "")
return f"{base_message} [trace_id={trace_id} span_id={span_id}]"
return base_message
def setup_logging(
log_format: str = "json",
log_level: str = "INFO",
include_trace_context: bool = True,
) -> None:
"""
Configure logging for the Nextcloud MCP Server.
Args:
log_format: "json" for JSON logging, "text" for human-readable text (default: "json")
log_level: Minimum log level (DEBUG, INFO, WARNING, ERROR, CRITICAL) (default: "INFO")
include_trace_context: Whether to include trace context in logs (default: True)
"""
# Get root logger
root_logger = logging.getLogger()
root_logger.setLevel(getattr(logging, log_level.upper(), logging.INFO))
# Remove existing handlers
root_logger.handlers.clear()
# Create console handler
console_handler = logging.StreamHandler(sys.stdout)
console_handler.setLevel(getattr(logging, log_level.upper(), logging.INFO))
# Configure formatter based on format preference
if log_format.lower() == "json":
if include_trace_context:
formatter = TraceContextFormatter(
"%(timestamp)s %(level)s %(name)s %(message)s",
datefmt="%Y-%m-%dT%H:%M:%S",
)
else:
formatter = JsonFormatter(
"%(timestamp)s %(level)s %(name)s %(message)s",
datefmt="%Y-%m-%dT%H:%M:%S",
)
else: # text format
if include_trace_context:
formatter = TraceContextTextFormatter(
"%(levelname)s [%(asctime)s] %(name)s - %(message)s",
datefmt="%Y-%m-%d %H:%M:%S",
)
else:
formatter = logging.Formatter(
"%(levelname)s [%(asctime)s] %(name)s - %(message)s",
datefmt="%Y-%m-%d %H:%M:%S",
)
console_handler.setFormatter(formatter)
root_logger.addHandler(console_handler)
# Configure specific logger levels
configure_component_loggers(log_level)
root_logger.info(
f"Logging configured: format={log_format}, level={log_level}, "
f"trace_context={include_trace_context}"
)
def configure_component_loggers(default_level: str = "INFO") -> None:
"""
Configure log levels for specific components.
This allows fine-grained control over logging verbosity for different
parts of the application.
Args:
default_level: Default log level for most components
"""
# Map of logger names to log levels
logger_levels = {
# Application loggers
"nextcloud_mcp_server": default_level,
"nextcloud_mcp_server.server": default_level,
"nextcloud_mcp_server.client": default_level,
"nextcloud_mcp_server.auth": default_level,
"nextcloud_mcp_server.observability": default_level,
# HTTP client loggers (less verbose by default)
"httpx": "WARNING",
"httpcore": "WARNING",
# Server loggers
"uvicorn": "INFO",
"uvicorn.access": "INFO",
"uvicorn.error": "INFO",
# MCP framework
"mcp": "INFO",
# OpenTelemetry (less verbose)
"opentelemetry": "WARNING",
}
for logger_name, level in logger_levels.items():
logger = logging.getLogger(logger_name)
logger.setLevel(getattr(logging, level.upper(), logging.INFO))
def get_logger(name: str) -> logging.Logger:
"""
Get a logger instance for a specific module.
This is a convenience function that wraps logging.getLogger()
to ensure consistent logger configuration.
Args:
name: Logger name (typically __name__)
Returns:
Logger instance
"""
return logging.getLogger(name)
def get_uvicorn_logging_config(
log_format: str = "json",
log_level: str = "INFO",
include_trace_context: bool = True,
) -> dict:
"""
Get uvicorn-compatible logging configuration.
This creates a logging config dict that uvicorn can use while maintaining
our observability setup (JSON format, trace context, etc.).
Args:
log_format: "json" or "text"
log_level: Minimum log level
include_trace_context: Whether to include trace IDs in logs
Returns:
Logging config dict compatible with uvicorn's log_config parameter
"""
# Determine formatter class based on format and trace context
if log_format.lower() == "json":
if include_trace_context:
formatter_class = "nextcloud_mcp_server.observability.logging_config.TraceContextFormatter"
else:
formatter_class = "pythonjsonlogger.json.JsonFormatter"
format_string = "%(timestamp)s %(level)s %(name)s %(message)s"
else:
if include_trace_context:
formatter_class = "nextcloud_mcp_server.observability.logging_config.TraceContextTextFormatter"
else:
formatter_class = "logging.Formatter"
format_string = "%(levelname)s [%(asctime)s] %(name)s - %(message)s"
return {
"version": 1,
"disable_existing_loggers": False,
"formatters": {
"default": {
"()": formatter_class,
"format": format_string,
"datefmt": "%Y-%m-%d %H:%M:%S",
},
},
"filters": {
"health_check_filter": {
"()": "nextcloud_mcp_server.observability.logging_config.HealthCheckFilter",
},
},
"handlers": {
"default": {
"formatter": "default",
"class": "logging.StreamHandler",
"stream": "ext://sys.stdout",
},
"access": {
"formatter": "default",
"class": "logging.StreamHandler",
"stream": "ext://sys.stdout",
"filters": ["health_check_filter"],
},
},
"loggers": {
"": {
"handlers": ["default"],
"level": log_level.upper(),
},
"uvicorn": {
"handlers": ["default"],
"level": "INFO",
"propagate": False,
},
"uvicorn.access": {
"handlers": ["access"],
"level": "INFO",
"propagate": False,
},
"uvicorn.error": {
"handlers": ["default"],
"level": "INFO",
"propagate": False,
},
"httpx": {
"handlers": ["default"],
"level": "WARNING",
"propagate": False,
},
"httpcore": {
"handlers": ["default"],
"level": "WARNING",
"propagate": False,
},
"opentelemetry": {
"handlers": ["default"],
"level": "WARNING",
"propagate": False,
},
},
}
@@ -0,0 +1,354 @@
"""
Prometheus metrics for the Nextcloud MCP Server.
This module defines all Prometheus metrics for monitoring server health, performance,
and resource usage. Metrics are organized by category:
- HTTP Server Metrics (RED: Rate, Errors, Duration)
- MCP Tool Metrics (per-tool invocation tracking)
- MCP Resource Metrics
- Nextcloud API Client Metrics
- OAuth Flow Metrics
- Vector Sync Metrics (conditional on feature flag)
- Database Operation Metrics
- External Dependency Health Metrics
"""
import logging
from prometheus_client import (
Counter,
Gauge,
Histogram,
start_http_server,
)
logger = logging.getLogger(__name__)
# =============================================================================
# HTTP Server Metrics (RED + System)
# =============================================================================
http_requests_total = Counter(
"mcp_http_requests_total",
"Total HTTP requests received",
["method", "endpoint", "status_code"],
)
http_request_duration_seconds = Histogram(
"mcp_http_request_duration_seconds",
"HTTP request latency in seconds",
["method", "endpoint"],
buckets=(0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0),
)
http_requests_in_progress = Gauge(
"mcp_http_requests_in_progress",
"Number of HTTP requests currently being processed",
["method", "endpoint"],
)
# =============================================================================
# MCP Tool Metrics
# =============================================================================
mcp_tool_calls_total = Counter(
"mcp_tool_calls_total",
"Total MCP tool invocations",
["tool_name", "status"], # status: success | error
)
mcp_tool_duration_seconds = Histogram(
"mcp_tool_duration_seconds",
"MCP tool execution duration in seconds",
["tool_name"],
buckets=(0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, 30.0),
)
mcp_tool_errors_total = Counter(
"mcp_tool_errors_total",
"Total MCP tool errors by type",
["tool_name", "error_type"],
)
# =============================================================================
# MCP Resource Metrics
# =============================================================================
mcp_resource_requests_total = Counter(
"mcp_resource_requests_total",
"Total MCP resource requests",
["resource_uri", "status"],
)
mcp_resource_duration_seconds = Histogram(
"mcp_resource_duration_seconds",
"MCP resource request duration in seconds",
["resource_uri"],
buckets=(0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5),
)
# =============================================================================
# Nextcloud API Client Metrics
# =============================================================================
nextcloud_api_requests_total = Counter(
"mcp_nextcloud_api_requests_total",
"Total Nextcloud API requests",
["app", "method", "status_code"], # app: notes, calendar, contacts, etc.
)
nextcloud_api_duration_seconds = Histogram(
"mcp_nextcloud_api_duration_seconds",
"Nextcloud API request duration in seconds",
["app", "method"],
buckets=(0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0),
)
nextcloud_api_retries_total = Counter(
"mcp_nextcloud_api_retries_total",
"Total Nextcloud API retries",
["app", "reason"], # reason: 429 | timeout | connection_error
)
# =============================================================================
# OAuth Flow Metrics
# =============================================================================
oauth_token_validations_total = Counter(
"mcp_oauth_token_validations_total",
"Total OAuth token validation attempts",
["method", "result"], # method: introspect | jwt; result: valid | invalid | error
)
oauth_token_exchange_total = Counter(
"mcp_oauth_token_exchange_total",
"Total OAuth token exchange operations (RFC 8693)",
["status"], # status: success | error
)
oauth_token_cache_hits_total = Counter(
"mcp_oauth_token_cache_hits_total",
"Total OAuth token cache lookups",
["hit"], # hit: true | false
)
oauth_refresh_token_operations_total = Counter(
"mcp_oauth_refresh_token_operations_total",
"Total refresh token storage operations",
[
"operation",
"status",
], # operation: store | retrieve | delete; status: success | error
)
# =============================================================================
# Vector Sync Metrics (optional feature)
# =============================================================================
vector_sync_documents_scanned_total = Counter(
"mcp_vector_sync_documents_scanned_total",
"Total documents scanned for vector sync",
)
vector_sync_documents_processed_total = Counter(
"mcp_vector_sync_documents_processed_total",
"Total documents processed for vector sync",
["status"], # status: success | error
)
vector_sync_processing_duration_seconds = Histogram(
"mcp_vector_sync_processing_duration_seconds",
"Document processing duration in seconds",
buckets=(0.1, 0.5, 1.0, 2.5, 5.0, 10.0, 30.0, 60.0),
)
vector_sync_queue_size = Gauge(
"mcp_vector_sync_queue_size",
"Current number of documents in processing queue",
)
qdrant_operations_total = Counter(
"mcp_qdrant_operations_total",
"Total Qdrant vector database operations",
[
"operation",
"status",
], # operation: upsert | search | delete; status: success | error
)
# =============================================================================
# Database Metrics
# =============================================================================
db_operations_total = Counter(
"mcp_db_operations_total",
"Total database operations",
["db", "operation", "status"], # db: sqlite | qdrant; operation varies
)
db_operation_duration_seconds = Histogram(
"mcp_db_operation_duration_seconds",
"Database operation duration in seconds",
["db", "operation"],
buckets=(0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0),
)
# =============================================================================
# External Dependency Health Metrics
# =============================================================================
dependency_health = Gauge(
"mcp_dependency_health",
"External dependency health status (1=up, 0=down)",
["dependency"], # dependency: nextcloud | keycloak | qdrant | unstructured
)
dependency_check_duration_seconds = Histogram(
"mcp_dependency_check_duration_seconds",
"Dependency health check duration in seconds",
["dependency"],
buckets=(0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5),
)
# =============================================================================
# Metrics Setup and HTTP Handler
# =============================================================================
def setup_metrics(port: int = 9090) -> None:
"""
Initialize Prometheus metrics collection and start HTTP server.
Starts a dedicated HTTP server on the specified port to serve metrics.
This server runs in a separate thread and is isolated from the main application.
Args:
port: Port to serve metrics on (default: 9090)
Note:
Metrics endpoint (/metrics) is ONLY accessible on this dedicated port,
not on the main application HTTP port. This is a security best practice
to prevent external exposure of metrics.
"""
try:
start_http_server(port)
logger.info(f"Prometheus metrics server started on port {port}")
except OSError as e:
if "Address already in use" in str(e):
logger.warning(
f"Metrics port {port} already in use (metrics server likely already running)"
)
else:
logger.error(f"Failed to start metrics server on port {port}: {e}")
raise
# =============================================================================
# Convenience Functions for Common Metric Updates
# =============================================================================
def record_tool_call(tool_name: str, duration: float, status: str = "success") -> None:
"""
Record metrics for an MCP tool call.
Args:
tool_name: Name of the MCP tool
duration: Execution duration in seconds
status: "success" or "error"
"""
mcp_tool_calls_total.labels(tool_name=tool_name, status=status).inc()
mcp_tool_duration_seconds.labels(tool_name=tool_name).observe(duration)
def record_tool_error(tool_name: str, error_type: str) -> None:
"""
Record an MCP tool error.
Args:
tool_name: Name of the MCP tool
error_type: Type of error (e.g., "HTTPStatusError", "ValueError")
"""
mcp_tool_errors_total.labels(tool_name=tool_name, error_type=error_type).inc()
def record_nextcloud_api_call(
app: str,
method: str,
status_code: int,
duration: float,
) -> None:
"""
Record metrics for a Nextcloud API call.
Args:
app: Nextcloud app name (notes, calendar, contacts, etc.)
method: HTTP method (GET, POST, PUT, DELETE, PROPFIND, etc.)
status_code: HTTP status code
duration: Request duration in seconds
"""
nextcloud_api_requests_total.labels(
app=app, method=method, status_code=str(status_code)
).inc()
nextcloud_api_duration_seconds.labels(app=app, method=method).observe(duration)
def record_nextcloud_api_retry(app: str, reason: str) -> None:
"""
Record a Nextcloud API retry.
Args:
app: Nextcloud app name
reason: Retry reason (429, timeout, connection_error)
"""
nextcloud_api_retries_total.labels(app=app, reason=reason).inc()
def record_oauth_token_validation(method: str, result: str) -> None:
"""
Record an OAuth token validation.
Args:
method: Validation method ("introspect" or "jwt")
result: Validation result ("valid", "invalid", or "error")
"""
oauth_token_validations_total.labels(method=method, result=result).inc()
def record_db_operation(
db: str, operation: str, duration: float, status: str = "success"
) -> None:
"""
Record a database operation.
Args:
db: Database type ("sqlite" or "qdrant")
operation: Operation type (e.g., "insert", "select", "upsert", "search")
duration: Operation duration in seconds
status: "success" or "error"
"""
db_operations_total.labels(db=db, operation=operation, status=status).inc()
db_operation_duration_seconds.labels(db=db, operation=operation).observe(duration)
def set_dependency_health(dependency: str, is_healthy: bool) -> None:
"""
Update external dependency health status.
Args:
dependency: Dependency name (nextcloud, keycloak, qdrant, unstructured)
is_healthy: True if dependency is healthy, False otherwise
"""
dependency_health.labels(dependency=dependency).set(1 if is_healthy else 0)
def record_dependency_check(dependency: str, duration: float) -> None:
"""
Record a dependency health check duration.
Args:
dependency: Dependency name
duration: Check duration in seconds
"""
dependency_check_duration_seconds.labels(dependency=dependency).observe(duration)
@@ -0,0 +1,218 @@
"""
Observability middleware for the Nextcloud MCP Server.
This module provides Starlette middleware that automatically instruments
HTTP requests with:
- Prometheus metrics (request count, latency, in-flight requests)
- OpenTelemetry distributed tracing
- Request/response timing and error tracking
"""
import logging
import time
from typing import Callable
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.requests import Request
from starlette.responses import Response
from nextcloud_mcp_server.observability.metrics import (
http_request_duration_seconds,
http_requests_in_progress,
http_requests_total,
)
from nextcloud_mcp_server.observability.tracing import (
add_span_attribute,
trace_operation,
)
logger = logging.getLogger(__name__)
class ObservabilityMiddleware(BaseHTTPMiddleware):
"""
Starlette middleware for automatic HTTP request instrumentation.
This middleware:
- Records Prometheus metrics for each request (RED metrics)
- Creates OpenTelemetry spans for distributed tracing
- Tracks request timing and errors
- Handles in-flight request counting
"""
async def dispatch(
self,
request: Request,
call_next: Callable,
) -> Response:
"""
Process HTTP request with observability instrumentation.
Args:
request: Starlette request object
call_next: Next middleware or route handler
Returns:
Response from downstream handler
"""
# Extract request details
method = request.method
path = request.url.path
endpoint = self._get_endpoint_label(path)
# Increment in-flight requests counter
http_requests_in_progress.labels(method=method, endpoint=endpoint).inc()
# Record start time
start_time = time.time()
# Skip tracing for health/metrics endpoints to reduce noise
should_trace = not (path.startswith("/health/") or path == "/metrics")
try:
if should_trace:
# Create span for request (OpenTelemetry auto-instrumentation will create parent span)
with trace_operation(
f"HTTP {method} {endpoint}",
attributes={
"http.method": method,
"http.path": path,
"http.scheme": request.url.scheme,
"http.host": request.url.hostname,
},
):
# Process request
response = await call_next(request)
# Add response status to span
add_span_attribute("http.status_code", response.status_code)
# Record metrics
duration = time.time() - start_time
self._record_request_metrics(
method=method,
endpoint=endpoint,
status_code=response.status_code,
duration=duration,
)
return response
else:
# No tracing for health/metrics endpoints, but still record metrics
response = await call_next(request)
# Record metrics
duration = time.time() - start_time
self._record_request_metrics(
method=method,
endpoint=endpoint,
status_code=response.status_code,
duration=duration,
)
return response
except Exception:
# Record error metrics
duration = time.time() - start_time
self._record_request_metrics(
method=method,
endpoint=endpoint,
status_code=500, # Internal server error
duration=duration,
)
logger.error(
f"Request failed: {method} {path}",
exc_info=True,
extra={
"method": method,
"path": path,
"duration_seconds": duration,
},
)
# Re-raise exception to be handled by error middleware
raise
finally:
# Decrement in-flight requests counter
http_requests_in_progress.labels(method=method, endpoint=endpoint).dec()
def _get_endpoint_label(self, path: str) -> str:
"""
Get endpoint label for metrics, normalizing dynamic path segments.
This prevents metric cardinality explosion by grouping similar paths.
Args:
path: Request path
Returns:
Normalized endpoint label
"""
# Health check endpoints
if path.startswith("/health/"):
return "/health/*"
# Metrics endpoint
if path == "/metrics":
return "/metrics"
# MCP protocol endpoints
if path == "/sse" or path.startswith("/sse/"):
return "/sse"
if path == "/messages" or path.startswith("/messages/"):
return "/messages"
# OAuth/OIDC endpoints
if path.startswith("/oauth/"):
return "/oauth/*"
if path.startswith("/oidc/"):
return "/oidc/*"
# Catch-all for other paths
return path
def _record_request_metrics(
self,
method: str,
endpoint: str,
status_code: int,
duration: float,
) -> None:
"""
Record Prometheus metrics for an HTTP request.
Args:
method: HTTP method
endpoint: Normalized endpoint label
status_code: HTTP status code
duration: Request duration in seconds
"""
# Record request count
http_requests_total.labels(
method=method,
endpoint=endpoint,
status_code=str(status_code),
).inc()
# Record request duration
http_request_duration_seconds.labels(
method=method,
endpoint=endpoint,
).observe(duration)
# Log slow requests (>1 second)
if duration > 1.0:
logger.warning(
f"Slow request: {method} {endpoint} took {duration:.3f}s",
extra={
"method": method,
"endpoint": endpoint,
"status_code": status_code,
"duration_seconds": duration,
},
)
@@ -0,0 +1,367 @@
"""
OpenTelemetry distributed tracing for the Nextcloud MCP Server.
This module provides:
- OpenTelemetry SDK initialization with OTLP exporter
- Auto-instrumentation for ASGI (Starlette/FastAPI) and httpx
- Helper functions for creating custom spans
- Context propagation utilities
- Span attribute standardization
"""
import logging
from contextlib import contextmanager
from typing import Any
from importlib_metadata import version
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.logging import LoggingInstrumentor
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.trace import Status, StatusCode, Tracer
logger = logging.getLogger(__name__)
# Global tracer instance (initialized in setup_tracing)
_tracer: Tracer | None = None
# Auto-instrument httpx for Nextcloud API calls
def setup_tracing(
service_name: str = "nextcloud-mcp-server",
otlp_endpoint: str | None = None,
otlp_verify_ssl: bool = False,
sampling_rate: float = 1.0,
) -> Tracer:
"""
Initialize OpenTelemetry tracing with OTLP exporter.
Args:
service_name: Service name for traces (default: "nextcloud-mcp-server")
otlp_endpoint: OTLP gRPC endpoint (e.g., "http://otel-collector:4317")
If None, tracing is initialized but no exporter is configured
otlp_verify_ssl: Enable TLS verification for otlp_endpoint. If True,
`insecure` will eval to False
sampling_rate: Sampling rate (0.0-1.0). Default 1.0 (100% sampling)
Returns:
Tracer instance for creating custom spans
"""
global _tracer
# Create resource with service name
resource = Resource.create(
{
"service.name": service_name,
"service.version": version(__package__.split(".")[0]),
}
)
# Create tracer provider
provider = TracerProvider(resource=resource)
# Configure OTLP exporter if endpoint is provided
if otlp_endpoint:
try:
otlp_exporter = OTLPSpanExporter(
endpoint=otlp_endpoint, insecure=not otlp_verify_ssl
)
span_processor = BatchSpanProcessor(otlp_exporter)
provider.add_span_processor(span_processor)
logger.info(
f"OpenTelemetry tracing enabled with OTLP endpoint: {otlp_endpoint}"
)
except Exception as e:
logger.warning(
f"Failed to initialize OTLP exporter: {e}. Continuing without trace export."
)
else:
logger.info(
"OpenTelemetry tracing initialized without OTLP exporter (traces will be generated but not exported)"
)
# Set global tracer provider
trace.set_tracer_provider(provider)
# Auto-instrument logging to inject trace context
LoggingInstrumentor().instrument(set_logging_format=True)
# Get and store tracer
_tracer = trace.get_tracer(__name__)
logger.info(f"OpenTelemetry tracing initialized for service: {service_name}")
return _tracer
def get_tracer() -> Tracer | None:
"""
Get the global tracer instance.
Returns:
Tracer instance for creating custom spans, or None if tracing is not enabled
Note:
Returns None if setup_tracing() was never called (tracing disabled).
Calling code should handle None gracefully.
"""
return _tracer
@contextmanager
def trace_operation(
operation_name: str,
attributes: dict[str, Any] | None = None,
record_exception: bool = True,
):
"""
Context manager for tracing an operation with automatic error handling.
Usage:
with trace_operation("mcp.tool.nc_notes_create_note", {"note.title": "My Note"}):
# Your code here
pass
Args:
operation_name: Name of the operation (span name)
attributes: Optional attributes to add to the span
record_exception: Whether to record exceptions in the span (default: True)
Yields:
Span instance for adding additional attributes (or None if tracing disabled)
"""
tracer = get_tracer()
# If tracing is not enabled, just yield without creating a span
if tracer is None:
yield None
return
with tracer.start_as_current_span(operation_name) as span:
# Set initial attributes
if attributes:
for key, value in attributes.items():
span.set_attribute(key, value)
try:
yield span
span.set_status(Status(StatusCode.OK))
except Exception as e:
if record_exception:
span.record_exception(e)
span.set_status(Status(StatusCode.ERROR, str(e)))
raise
def trace_mcp_tool(tool_name: str, tool_args: dict[str, Any] | None = None):
"""
Create a span for an MCP tool invocation.
Usage:
with trace_mcp_tool("nc_notes_create_note", {"title": "My Note"}):
# Tool implementation
pass
Args:
tool_name: Name of the MCP tool
tool_args: Optional tool arguments (sensitive data will be sanitized)
Returns:
Context manager for the span
"""
attributes = {
"mcp.tool.name": tool_name,
}
# Add sanitized tool args (avoid logging sensitive data)
if tool_args:
# Only include non-sensitive arguments
safe_args = {
k: v
for k, v in tool_args.items()
if k not in ("password", "token", "secret", "api_key", "etag")
}
if safe_args:
attributes["mcp.tool.args"] = str(safe_args)
return trace_operation(f"mcp.tool.{tool_name}", attributes)
def trace_nextcloud_api_call(
app: str,
method: str,
path: str | None = None,
):
"""
Create a span for a Nextcloud API call.
Usage:
with trace_nextcloud_api_call("notes", "POST", "/apps/notes/api/v1/notes"):
# API call implementation
pass
Args:
app: Nextcloud app name (notes, calendar, contacts, etc.)
method: HTTP method (GET, POST, PUT, DELETE, etc.)
path: Optional API path
Returns:
Context manager for the span
"""
attributes = {
"nextcloud.app": app,
"http.method": method,
}
if path:
attributes["http.path"] = path
return trace_operation(f"nextcloud.api.{app}.{method}", attributes)
def trace_oauth_operation(operation: str, details: dict[str, Any] | None = None):
"""
Create a span for an OAuth operation.
Usage:
with trace_oauth_operation("token.validate", {"method": "jwt"}):
# OAuth validation logic
pass
Args:
operation: OAuth operation name (e.g., "token.validate", "token.exchange")
details: Optional operation details (sensitive data will be sanitized)
Returns:
Context manager for the span
"""
attributes = {"oauth.operation": operation}
if details:
# Only include non-sensitive details
safe_details = {
k: v
for k, v in details.items()
if k not in ("token", "refresh_token", "access_token", "client_secret")
}
if safe_details:
attributes.update(safe_details)
return trace_operation(f"oauth.{operation}", attributes)
def trace_vector_sync_operation(
operation: str,
document_count: int | None = None,
):
"""
Create a span for a vector sync operation.
Usage:
with trace_vector_sync_operation("scan", document_count=10):
# Vector sync logic
pass
Args:
operation: Operation name (scan, process, embed, upsert)
document_count: Optional number of documents being processed
Returns:
Context manager for the span
"""
attributes = {"vector_sync.operation": operation}
if document_count is not None:
attributes["vector_sync.document_count"] = document_count
return trace_operation(f"vector_sync.{operation}", attributes)
def trace_db_operation(
db: str,
operation: str,
table: str | None = None,
):
"""
Create a span for a database operation.
Usage:
with trace_db_operation("sqlite", "insert", "refresh_tokens"):
# Database operation
pass
Args:
db: Database type (sqlite, qdrant)
operation: Operation type (insert, select, update, delete, upsert, search)
table: Optional table/collection name
Returns:
Context manager for the span
"""
attributes = {
"db.system": db,
"db.operation": operation,
}
if table:
attributes["db.table"] = table
return trace_operation(f"db.{db}.{operation}", attributes)
def add_span_attribute(key: str, value: Any) -> None:
"""
Add an attribute to the current span (if any).
Args:
key: Attribute key
value: Attribute value
Note:
This is a no-op if tracing is not enabled or there's no active span.
"""
if _tracer is None:
return # Tracing not enabled
span = trace.get_current_span()
if span.is_recording():
span.set_attribute(key, value)
def add_span_event(name: str, attributes: dict[str, Any] | None = None) -> None:
"""
Add an event to the current span (if any).
Args:
name: Event name
attributes: Optional event attributes
Note:
This is a no-op if tracing is not enabled or there's no active span.
"""
if _tracer is None:
return # Tracing not enabled
span = trace.get_current_span()
if span.is_recording():
span.add_event(name, attributes=attributes or {})
def get_trace_context() -> dict[str, str]:
"""
Get current trace context as a dictionary.
Returns:
Dictionary with trace_id and span_id (or empty dict if tracing disabled or no active span)
"""
if _tracer is None:
return {} # Tracing not enabled
span = trace.get_current_span()
if span.is_recording():
span_context = span.get_span_context()
return {
"trace_id": format(span_context.trace_id, "032x"),
"span_id": format(span_context.span_id, "016x"),
}
return {}
+1 -1
View File
@@ -18,7 +18,7 @@ from mcp.server.fastmcp import Context
from pydantic import BaseModel, Field
from nextcloud_mcp_server.auth import require_scopes
from nextcloud_mcp_server.auth.refresh_token_storage import RefreshTokenStorage
from nextcloud_mcp_server.auth.storage import RefreshTokenStorage
from nextcloud_mcp_server.auth.token_broker import TokenBrokerService
from nextcloud_mcp_server.auth.userinfo_routes import _query_idp_userinfo
+164 -32
View File
@@ -68,17 +68,25 @@ def configure_semantic_tools(mcp: FastMCP):
client = await get_client(ctx)
username = client.username
logger.info(
f"Semantic search: query='{query}', user={username}, "
f"limit={limit}, score_threshold={score_threshold}"
)
try:
# Generate embedding for query
embedding_service = get_embedding_service()
query_embedding = await embedding_service.embed(query)
logger.debug(
f"Generated embedding for query (dimension={len(query_embedding)})"
)
# Search Qdrant with user filtering
# Note: Currently only searching notes (doc_type="note")
# Future: Remove doc_type filter to search all apps
qdrant_client = await get_qdrant_client()
search_response = await qdrant_client.query_points(
collection_name=settings.qdrant_collection,
collection_name=settings.get_collection_name(),
query=query_embedding,
query_filter=Filter(
must=[
@@ -98,6 +106,15 @@ def configure_semantic_tools(mcp: FastMCP):
with_vectors=False, # Don't return vectors to save bandwidth
)
logger.info(
f"Qdrant returned {len(search_response.points)} results "
f"(before deduplication and access verification)"
)
if search_response.points:
# Log top 3 scores to help with threshold tuning
top_scores = [p.score for p in search_response.points[:3]]
logger.debug(f"Top 3 similarity scores: {top_scores}")
# Deduplicate by document ID (multiple chunks per document)
seen_doc_ids = set()
results = []
@@ -137,9 +154,14 @@ def configure_semantic_tools(mcp: FastMCP):
except HTTPStatusError as e:
if e.response.status_code == 403:
# User lost access, skip this document
logger.debug(f"Skipping note {doc_id}: access denied (403)")
continue
elif e.response.status_code == 404:
# Document was deleted but not yet removed from vector DB
logger.debug(
f"Skipping note {doc_id}: not found (404), "
f"likely deleted after indexing"
)
continue
else:
# Log other errors but continue processing
@@ -148,6 +170,16 @@ def configure_semantic_tools(mcp: FastMCP):
)
continue
logger.info(
f"Returning {len(results)} results after deduplication and access verification"
)
if results:
result_details = [
f"note_{r.id} (score={r.score:.3f}, title='{r.title}')"
for r in results[:5] # Show top 5
]
logger.debug(f"Top results: {', '.join(result_details)}")
return SemanticSearchResponse(
results=results,
query=query,
@@ -259,7 +291,47 @@ def configure_semantic_tools(mcp: FastMCP):
success=True,
)
# 3. Construct context from retrieved documents
# 3. Check if client supports sampling
from mcp.types import ClientCapabilities, SamplingCapability
client_has_sampling = ctx.session.check_client_capability(
ClientCapabilities(sampling=SamplingCapability())
)
# Log capability check result for debugging
logger.info(
f"Sampling capability check: client_has_sampling={client_has_sampling}, "
f"query='{query}'"
)
if hasattr(ctx.session, "_client_params") and ctx.session._client_params:
client_caps = ctx.session._client_params.capabilities
logger.debug(
f"Client advertised capabilities: "
f"roots={client_caps.roots is not None}, "
f"sampling={client_caps.sampling is not None}, "
f"experimental={client_caps.experimental is not None}"
)
if not client_has_sampling:
logger.info(
f"Client does not support sampling (query: '{query}'), "
f"returning {len(search_response.results)} documents"
)
return SamplingSearchResponse(
query=query,
generated_answer=(
f"[Sampling not supported by client]\n\n"
f"Your MCP client doesn't support answer generation. "
f"Found {search_response.total_found} relevant documents. "
f"Please review the sources below."
),
sources=search_response.results,
total_found=search_response.total_found,
search_method="semantic_sampling_unsupported",
success=True,
)
# 4. Construct context from retrieved documents
context_parts = []
for idx, result in enumerate(search_response.results, 1):
context_parts.append(
@@ -273,7 +345,7 @@ def configure_semantic_tools(mcp: FastMCP):
context = "\n".join(context_parts)
# 4. Construct prompt - reuse user's query, add context and instructions
# 5. Construct prompt - reuse user's query, add context and instructions
prompt = (
f"{query}\n\n"
f"Here are relevant documents from Nextcloud (notes, calendar events, deck cards, files, contacts):\n\n"
@@ -282,31 +354,35 @@ def configure_semantic_tools(mcp: FastMCP):
f"Cite the document numbers when referencing specific information."
)
logger.debug(
f"Requesting sampling for query: {query} "
f"({len(search_response.results)} documents retrieved)"
logger.info(
f"Initiating sampling request: query_length={len(query)}, "
f"documents={len(search_response.results)}, "
f"prompt_length={len(prompt)}, max_tokens={max_answer_tokens}"
)
# 5. Request LLM completion via MCP sampling
try:
sampling_result = await ctx.session.create_message(
messages=[
SamplingMessage(
role="user",
content=TextContent(type="text", text=prompt),
)
],
max_tokens=max_answer_tokens,
temperature=0.7,
model_preferences=ModelPreferences(
hints=[ModelHint(name="claude-3-5-sonnet")],
intelligencePriority=0.8,
speedPriority=0.5,
),
include_context="thisServer",
)
# 6. Request LLM completion via MCP sampling with timeout
import anyio
# 6. Extract answer from sampling response
try:
with anyio.fail_after(30):
sampling_result = await ctx.session.create_message(
messages=[
SamplingMessage(
role="user",
content=TextContent(type="text", text=prompt),
)
],
max_tokens=max_answer_tokens,
temperature=0.7,
model_preferences=ModelPreferences(
hints=[ModelHint(name="claude-3-5-sonnet")],
intelligencePriority=0.8,
speedPriority=0.5,
),
include_context="thisServer",
)
# 7. Extract answer from sampling response
if sampling_result.content.type == "text":
generated_answer = sampling_result.content.text
else:
@@ -318,7 +394,8 @@ def configure_semantic_tools(mcp: FastMCP):
logger.info(
f"Sampling successful: model={sampling_result.model}, "
f"stop_reason={sampling_result.stopReason}"
f"stop_reason={sampling_result.stopReason}, "
f"answer_length={len(generated_answer)}"
)
return SamplingSearchResponse(
@@ -332,23 +409,78 @@ def configure_semantic_tools(mcp: FastMCP):
success=True,
)
except Exception as e:
# Fallback: Return documents without generated answer
except TimeoutError:
logger.warning(
f"Sampling failed ({type(e).__name__}: {e}), "
f"Sampling request timed out after 30 seconds for query: '{query}', "
f"returning search results only"
)
return SamplingSearchResponse(
query=query,
generated_answer=(
f"[Sampling request timed out]\n\n"
f"The answer generation took too long (>30s). "
f"Found {search_response.total_found} relevant documents. "
f"Please review the sources below or try a simpler query."
),
sources=search_response.results,
total_found=search_response.total_found,
search_method="semantic_sampling_timeout",
success=True,
)
except McpError as e:
# Expected MCP protocol errors (user rejection, unsupported, etc.)
error_msg = str(e)
if "rejected" in error_msg.lower() or "denied" in error_msg.lower():
# User explicitly declined - this is normal, not an error
logger.info(f"User declined sampling request for query: '{query}'")
search_method = "semantic_sampling_user_declined"
user_message = "User declined to generate an answer"
elif "not supported" in error_msg.lower():
# Client doesn't support sampling - also normal
logger.info(f"Sampling not supported by client for query: '{query}'")
search_method = "semantic_sampling_unsupported"
user_message = "Sampling not supported by this client"
else:
# Other MCP protocol errors
logger.warning(
f"MCP error during sampling for query '{query}': {error_msg}"
)
search_method = "semantic_sampling_mcp_error"
user_message = f"Sampling unavailable: {error_msg}"
return SamplingSearchResponse(
query=query,
generated_answer=(
f"[Sampling unavailable: {str(e)}]\n\n"
f"[{user_message}]\n\n"
f"Found {search_response.total_found} relevant documents. "
f"Please review the sources below."
),
sources=search_response.results,
total_found=search_response.total_found,
search_method="semantic_sampling_fallback",
search_method=search_method,
success=True,
)
except Exception as e:
# Truly unexpected errors - these SHOULD have tracebacks
logger.error(
f"Unexpected error during sampling for query '{query}': "
f"{type(e).__name__}: {e}",
exc_info=True,
)
return SamplingSearchResponse(
query=query,
generated_answer=(
f"[Unexpected error during sampling]\n\n"
f"Found {search_response.total_found} relevant documents. "
f"Please review the sources below."
),
sources=search_response.results,
total_found=search_response.total_found,
search_method="semantic_sampling_error",
success=True,
)
@@ -413,7 +545,7 @@ def configure_semantic_tools(mcp: FastMCP):
# Count documents in collection
count_result = await qdrant_client.count(
collection_name=settings.qdrant_collection
collection_name=settings.get_collection_name()
)
indexed_count = count_result.count
@@ -0,0 +1,197 @@
"""Webhook preset configurations for common sync scenarios.
This module defines pre-configured webhook bundles that simplify
webhook setup for common use cases like Notes sync, Calendar sync, etc.
"""
from typing import Any, Dict, List, TypedDict
class WebhookEventConfig(TypedDict):
"""Configuration for a single webhook event."""
event: str # Fully qualified event class name
filter: Dict[str, Any] # Event filter (optional)
class WebhookPreset(TypedDict):
"""Definition of a webhook preset."""
name: str # Display name
description: str # User-friendly description
events: List[WebhookEventConfig] # List of events to register
app: str # Nextcloud app this preset is for
# File/Notes webhook events
FILE_EVENT_CREATED = "OCP\\Files\\Events\\Node\\NodeCreatedEvent"
FILE_EVENT_WRITTEN = "OCP\\Files\\Events\\Node\\NodeWrittenEvent"
# Use BeforeNodeDeletedEvent instead of NodeDeletedEvent to get node.id
# See: https://github.com/nextcloud/server/issues/56371
FILE_EVENT_DELETED = "OCP\\Files\\Events\\Node\\BeforeNodeDeletedEvent"
# Calendar webhook events
CALENDAR_EVENT_CREATED = "OCP\\Calendar\\Events\\CalendarObjectCreatedEvent"
CALENDAR_EVENT_UPDATED = "OCP\\Calendar\\Events\\CalendarObjectUpdatedEvent"
CALENDAR_EVENT_DELETED = "OCP\\Calendar\\Events\\CalendarObjectDeletedEvent"
# Tables webhook events (Nextcloud 30+)
TABLES_EVENT_ROW_ADDED = "OCA\\Tables\\Event\\RowAddedEvent"
TABLES_EVENT_ROW_UPDATED = "OCA\\Tables\\Event\\RowUpdatedEvent"
TABLES_EVENT_ROW_DELETED = "OCA\\Tables\\Event\\RowDeletedEvent"
# Forms webhook events (Nextcloud 30+)
FORMS_EVENT_FORM_SUBMITTED = "OCA\\Forms\\Events\\FormSubmittedEvent"
# NOTE: Deck and Contacts do NOT support webhooks
# Their event classes do not implement IWebhookCompatibleEvent interface.
# Alternative sync strategies:
# - Deck: Use polling with ETag-based change detection
# - Contacts: Use CardDAV sync-token mechanism for efficient syncing
WEBHOOK_PRESETS: Dict[str, WebhookPreset] = {
"notes_sync": {
"name": "Notes Sync",
"description": "Real-time synchronization for Notes app (create, update, delete)",
"app": "notes",
"events": [
{
"event": FILE_EVENT_CREATED,
"filter": {"event.node.path": "/^\\/.*\\/files\\/Notes\\//"},
},
{
"event": FILE_EVENT_WRITTEN,
"filter": {"event.node.path": "/^\\/.*\\/files\\/Notes\\//"},
},
{
"event": FILE_EVENT_DELETED,
"filter": {"event.node.path": "/^\\/.*\\/files\\/Notes\\//"},
},
],
},
"calendar_sync": {
"name": "Calendar Sync",
"description": "Real-time synchronization for Calendar events (create, update, delete)",
"app": "calendar",
"events": [
{
"event": CALENDAR_EVENT_CREATED,
"filter": {},
},
{
"event": CALENDAR_EVENT_UPDATED,
"filter": {},
},
{
"event": CALENDAR_EVENT_DELETED,
"filter": {},
},
],
},
"tables_sync": {
"name": "Tables Sync",
"description": "Real-time synchronization for Tables rows (add, update, delete)",
"app": "tables",
"events": [
{
"event": TABLES_EVENT_ROW_ADDED,
"filter": {},
},
{
"event": TABLES_EVENT_ROW_UPDATED,
"filter": {},
},
{
"event": TABLES_EVENT_ROW_DELETED,
"filter": {},
},
],
},
"forms_sync": {
"name": "Forms Sync",
"description": "Real-time synchronization for Forms submissions",
"app": "forms",
"events": [
{
"event": FORMS_EVENT_FORM_SUBMITTED,
"filter": {},
},
],
},
"files_sync": {
"name": "All Files Sync",
"description": "Real-time synchronization for all file operations (create, update, delete)",
"app": "files",
"events": [
{
"event": FILE_EVENT_CREATED,
"filter": {},
},
{
"event": FILE_EVENT_WRITTEN,
"filter": {},
},
{
"event": FILE_EVENT_DELETED,
"filter": {},
},
],
},
}
def get_preset(preset_id: str) -> WebhookPreset | None:
"""Get a webhook preset by ID.
Args:
preset_id: Preset identifier (e.g., "notes_sync", "calendar_sync")
Returns:
Webhook preset configuration or None if not found
"""
return WEBHOOK_PRESETS.get(preset_id)
def list_presets() -> List[tuple[str, WebhookPreset]]:
"""Get all available webhook presets.
Returns:
List of (preset_id, preset_config) tuples
"""
return list(WEBHOOK_PRESETS.items())
def get_preset_events(preset_id: str) -> List[str]:
"""Get list of event class names for a preset.
Args:
preset_id: Preset identifier
Returns:
List of fully qualified event class names
"""
preset = get_preset(preset_id)
if not preset:
return []
return [event_config["event"] for event_config in preset["events"]]
def filter_presets_by_installed_apps(
installed_apps: list[str],
) -> List[tuple[str, WebhookPreset]]:
"""Filter webhook presets to only show those for installed apps.
Args:
installed_apps: List of installed app names (e.g., ["notes", "calendar", "forms"])
Returns:
List of (preset_id, preset_config) tuples for presets whose apps are installed
"""
filtered = []
for preset_id, preset in WEBHOOK_PRESETS.items():
app_name = preset["app"]
# "files" is always available (core functionality)
if app_name == "files" or app_name in installed_apps:
filtered.append((preset_id, preset))
return filtered
+65 -51
View File
@@ -15,6 +15,7 @@ from qdrant_client.models import FieldCondition, Filter, MatchValue, PointStruct
from nextcloud_mcp_server.client import NextcloudClient
from nextcloud_mcp_server.config import get_settings
from nextcloud_mcp_server.embedding import get_embedding_service
from nextcloud_mcp_server.observability.tracing import trace_operation
from nextcloud_mcp_server.vector.document_chunker import DocumentChunker
from nextcloud_mcp_server.vector.qdrant_client import get_qdrant_client
from nextcloud_mcp_server.vector.scanner import DocumentTask
@@ -94,58 +95,68 @@ async def process_document(doc_task: DocumentTask, nc_client: NextcloudClient):
f"for {doc_task.user_id} ({doc_task.operation})"
)
qdrant_client = await get_qdrant_client()
settings = get_settings()
with trace_operation(
"vector_sync.process_document",
attributes={
"vector_sync.operation": "process",
"vector_sync.user_id": doc_task.user_id,
"vector_sync.doc_id": doc_task.doc_id,
"vector_sync.doc_type": doc_task.doc_type,
"vector_sync.doc_operation": doc_task.operation,
},
):
qdrant_client = await get_qdrant_client()
settings = get_settings()
# Handle deletion
if doc_task.operation == "delete":
await qdrant_client.delete(
collection_name=settings.qdrant_collection,
points_selector=Filter(
must=[
FieldCondition(
key="user_id",
match=MatchValue(value=doc_task.user_id),
),
FieldCondition(
key="doc_id",
match=MatchValue(value=doc_task.doc_id),
),
FieldCondition(
key="doc_type",
match=MatchValue(value=doc_task.doc_type),
),
]
),
)
logger.info(
f"Deleted {doc_task.doc_type}_{doc_task.doc_id} for {doc_task.user_id}"
)
return
# Handle deletion
if doc_task.operation == "delete":
await qdrant_client.delete(
collection_name=settings.get_collection_name(),
points_selector=Filter(
must=[
FieldCondition(
key="user_id",
match=MatchValue(value=doc_task.user_id),
),
FieldCondition(
key="doc_id",
match=MatchValue(value=doc_task.doc_id),
),
FieldCondition(
key="doc_type",
match=MatchValue(value=doc_task.doc_type),
),
]
),
)
logger.info(
f"Deleted {doc_task.doc_type}_{doc_task.doc_id} for {doc_task.user_id}"
)
return
# Handle indexing with retry
max_retries = 3
retry_delay = 1.0
# Handle indexing with retry
max_retries = 3
retry_delay = 1.0
for attempt in range(max_retries):
try:
await _index_document(doc_task, nc_client, qdrant_client)
return # Success
for attempt in range(max_retries):
try:
await _index_document(doc_task, nc_client, qdrant_client)
return # Success
except (HTTPStatusError, Exception) as e:
if attempt < max_retries - 1:
logger.warning(
f"Retry {attempt + 1}/{max_retries} for "
f"{doc_task.doc_type}_{doc_task.doc_id}: {e}"
)
await anyio.sleep(retry_delay)
retry_delay *= 2 # Exponential backoff
else:
logger.error(
f"Failed to index {doc_task.doc_type}_{doc_task.doc_id} "
f"after {max_retries} retries: {e}"
)
raise
except (HTTPStatusError, Exception) as e:
if attempt < max_retries - 1:
logger.warning(
f"Retry {attempt + 1}/{max_retries} for "
f"{doc_task.doc_type}_{doc_task.doc_id}: {e}"
)
await anyio.sleep(retry_delay)
retry_delay *= 2 # Exponential backoff
else:
logger.error(
f"Failed to index {doc_task.doc_type}_{doc_task.doc_id} "
f"after {max_retries} retries: {e}"
)
raise
async def _index_document(
@@ -170,8 +181,11 @@ async def _index_document(
else:
raise ValueError(f"Unsupported doc_type: {doc_task.doc_type}")
# Tokenize and chunk
chunker = DocumentChunker(chunk_size=512, overlap=50)
# Tokenize and chunk (using configured chunk size and overlap)
chunker = DocumentChunker(
chunk_size=settings.document_chunk_size,
overlap=settings.document_chunk_overlap,
)
chunks = chunker.chunk_text(content)
# Generate embeddings (I/O bound - external API call)
@@ -209,7 +223,7 @@ async def _index_document(
# Upsert to Qdrant
await qdrant_client.upsert(
collection_name=settings.qdrant_collection,
collection_name=settings.get_collection_name(),
points=points,
wait=True,
)
+51 -11
View File
@@ -59,30 +59,70 @@ async def get_qdrant_client() -> AsyncQdrantClient:
logger.warning("No Qdrant mode configured, defaulting to :memory:")
_qdrant_client = AsyncQdrantClient(":memory:")
# Ensure collection exists
collection_name = settings.qdrant_collection
# Get collection name (auto-generated from deployment ID + model)
collection_name = settings.get_collection_name()
# Import here to avoid circular dependency
from nextcloud_mcp_server.embedding import get_embedding_service
embedding_service = get_embedding_service()
dimension = embedding_service.get_dimension()
try:
await _qdrant_client.get_collection(collection_name)
logger.info(f"Using existing Qdrant collection: {collection_name}")
except Exception:
# Collection doesn't exist, create it
# Detect dimension dynamically (for OllamaEmbeddingProvider)
if hasattr(embedding_service.provider, "_detect_dimension"):
await embedding_service.provider._detect_dimension() # type: ignore[call-non-callable]
expected_dimension = embedding_service.get_dimension()
# Explicitly check if collection exists
logger.debug(f"Checking if collection '{collection_name}' exists...")
collections = await _qdrant_client.get_collections()
collection_names = [c.name for c in collections.collections]
if collection_name in collection_names:
# Collection exists - validate dimensions
logger.debug(
f"Collection '{collection_name}' found, validating dimensions..."
)
collection_info = await _qdrant_client.get_collection(collection_name)
actual_dimension = collection_info.config.params.vectors.size
# Validate dimension matches
if actual_dimension != expected_dimension:
raise ValueError(
f"Dimension mismatch for collection '{collection_name}':\n"
f" Expected: {expected_dimension} (from embedding model '{settings.ollama_embedding_model}')\n"
f" Found: {actual_dimension}\n"
f"This usually means you changed the embedding model.\n"
f"Solutions:\n"
f" 1. Delete the old collection: Collection will be recreated with new dimensions\n"
f" 2. Set QDRANT_COLLECTION to use a different collection name\n"
f" 3. Revert OLLAMA_EMBEDDING_MODEL to the original model"
)
logger.info(
f"Using existing Qdrant collection: {collection_name} "
f"(dimension={actual_dimension}, model={settings.ollama_embedding_model})"
)
else:
# Collection doesn't exist - create it
logger.info(
f"Collection '{collection_name}' not found, creating with "
f"dimension={expected_dimension}, model={settings.ollama_embedding_model}..."
)
await _qdrant_client.create_collection(
collection_name=collection_name,
vectors_config=VectorParams(
size=dimension,
size=expected_dimension,
distance=Distance.COSINE,
),
)
logger.info(
f"Created Qdrant collection: {collection_name} "
f"(dimension={dimension}, distance=COSINE)"
f"Created Qdrant collection: {collection_name}\n"
f" Dimension: {expected_dimension}\n"
f" Model: {settings.ollama_embedding_model}\n"
f" Distance: COSINE\n"
f"Background sync will index all documents with this embedding model."
)
return _qdrant_client
+191 -108
View File
@@ -13,6 +13,7 @@ from qdrant_client.models import FieldCondition, Filter, MatchValue
from nextcloud_mcp_server.client import NextcloudClient
from nextcloud_mcp_server.config import get_settings
from nextcloud_mcp_server.observability.tracing import trace_operation
from nextcloud_mcp_server.vector.qdrant_client import get_qdrant_client
logger = logging.getLogger(__name__)
@@ -34,6 +35,57 @@ class DocumentTask:
_potentially_deleted: dict[tuple[str, str], float] = {}
async def get_last_indexed_timestamp(user_id: str) -> int | None:
"""Get the most recent indexed_at timestamp for user's notes in Qdrant.
This timestamp can be used as pruneBefore parameter to optimize data transfer
when fetching notes - only notes modified after this timestamp will be sent
with full data.
Args:
user_id: User to query
Returns:
Unix timestamp of most recently indexed note, or None if no notes indexed yet
"""
try:
qdrant_client = await get_qdrant_client()
# Query for user's notes, ordered by indexed_at descending, limit 1
scroll_result = await qdrant_client.scroll(
collection_name=get_settings().get_collection_name(),
scroll_filter=Filter(
must=[
FieldCondition(key="user_id", match=MatchValue(value=user_id)),
FieldCondition(key="doc_type", match=MatchValue(value="note")),
]
),
with_payload=["indexed_at"],
with_vectors=False,
limit=10000, # Get all to find max
)
# Find max indexed_at across all results
num_points = len(scroll_result[0]) if scroll_result[0] else 0
logger.info(f"Found {num_points} indexed notes in Qdrant for user {user_id}")
if scroll_result[0]:
timestamps = [
point.payload.get("indexed_at", 0) for point in scroll_result[0]
]
max_timestamp = max(timestamps)
logger.info(
f"Max indexed_at: {max_timestamp}, timestamps sample: {timestamps[:3]}"
)
return int(max_timestamp) if max_timestamp > 0 else None
logger.info(f"No indexed notes found for user {user_id}")
return None
except Exception as e:
logger.warning(f"Failed to get last indexed timestamp: {e}", exc_info=True)
return None
async def scanner_task(
send_stream: MemoryObjectSendStream[DocumentTask],
shutdown_event: anyio.Event,
@@ -96,124 +148,155 @@ async def scan_user_documents(
nc_client: Authenticated Nextcloud client
initial_sync: If True, send all documents (first-time sync)
"""
logger.info(f"Scanning documents for user: {user_id}")
import random
# Fetch all notes from Nextcloud
notes = [note async for note in nc_client.notes.get_all_notes()]
logger.debug(f"Found {len(notes)} notes for {user_id}")
if initial_sync:
# Send everything on first sync
for note in notes:
await send_stream.send(
DocumentTask(
user_id=user_id,
doc_id=str(note["id"]),
doc_type="note",
operation="index",
modified_at=note["modified"],
)
)
logger.info(f"Sent {len(notes)} documents for initial sync: {user_id}")
return
# Get indexed state from Qdrant
qdrant_client = await get_qdrant_client()
scroll_result = await qdrant_client.scroll(
collection_name=get_settings().qdrant_collection,
scroll_filter=Filter(
must=[
FieldCondition(key="user_id", match=MatchValue(value=user_id)),
FieldCondition(key="doc_type", match=MatchValue(value="note")),
]
),
with_payload=["doc_id", "indexed_at"],
with_vectors=False,
limit=10000,
scan_id = random.randint(1000, 9999)
logger.info(
f"[SCAN-{scan_id}] Starting scan for user: {user_id}, initial_sync={initial_sync}"
)
indexed_docs = {
point.payload["doc_id"]: point.payload["indexed_at"]
for point in scroll_result[0]
}
logger.debug(f"Found {len(indexed_docs)} indexed documents in Qdrant")
# Compare and queue changes
queued = 0
nextcloud_doc_ids = {str(note["id"]) for note in notes}
for note in notes:
doc_id = str(note["id"])
indexed_at = indexed_docs.get(doc_id)
# If document reappeared, remove from potentially_deleted
doc_key = (user_id, doc_id)
if doc_key in _potentially_deleted:
logger.debug(
f"Document {doc_id} reappeared, removing from deletion grace period"
with trace_operation(
"vector_sync.scan_user_documents",
attributes={
"vector_sync.operation": "scan",
"vector_sync.user_id": user_id,
"vector_sync.initial_sync": initial_sync,
"vector_sync.scan_id": scan_id,
},
):
# Calculate prune timestamp for optimized data transfer
# Only notes modified after this will be sent with full data
prune_before = (
None if initial_sync else await get_last_indexed_timestamp(user_id)
)
if prune_before:
logger.info(
f"[SCAN-{scan_id}] Using pruneBefore={prune_before} to optimize data transfer"
)
del _potentially_deleted[doc_key]
# Send if never indexed or modified since last index
if indexed_at is None or note["modified"] > indexed_at:
await send_stream.send(
DocumentTask(
user_id=user_id,
doc_id=doc_id,
doc_type="note",
operation="index",
modified_at=note["modified"],
# Fetch all notes from Nextcloud
notes = [
note
async for note in nc_client.notes.get_all_notes(prune_before=prune_before)
]
logger.info(f"[SCAN-{scan_id}] Found {len(notes)} notes for {user_id}")
if initial_sync:
# Send everything on first sync
for note in notes:
modified_at = note.get("modified", 0)
await send_stream.send(
DocumentTask(
user_id=user_id,
doc_id=str(note["id"]),
doc_type="note",
operation="index",
modified_at=modified_at,
)
)
)
queued += 1
logger.info(f"Sent {len(notes)} documents for initial sync: {user_id}")
return
# Check for deleted documents (in Qdrant but not in Nextcloud)
# Use grace period: only delete after 2 consecutive scans confirm absence
settings = get_settings()
grace_period = settings.vector_sync_scan_interval * 1.5 # Allow 1.5 scan intervals
current_time = time.time()
# Get indexed state from Qdrant
qdrant_client = await get_qdrant_client()
scroll_result = await qdrant_client.scroll(
collection_name=get_settings().get_collection_name(),
scroll_filter=Filter(
must=[
FieldCondition(key="user_id", match=MatchValue(value=user_id)),
FieldCondition(key="doc_type", match=MatchValue(value="note")),
]
),
with_payload=["doc_id", "indexed_at"],
with_vectors=False,
limit=10000,
)
for doc_id in indexed_docs:
if doc_id not in nextcloud_doc_ids:
indexed_docs = {
point.payload["doc_id"]: point.payload["indexed_at"]
for point in scroll_result[0]
}
logger.debug(f"Found {len(indexed_docs)} indexed documents in Qdrant")
# Compare and queue changes
queued = 0
nextcloud_doc_ids = {str(note["id"]) for note in notes}
for note in notes:
doc_id = str(note["id"])
indexed_at = indexed_docs.get(doc_id)
modified_at = note.get("modified", 0)
# If document reappeared, remove from potentially_deleted
doc_key = (user_id, doc_id)
if doc_key in _potentially_deleted:
# Already marked as potentially deleted, check if grace period elapsed
first_missing_time = _potentially_deleted[doc_key]
time_missing = current_time - first_missing_time
if time_missing >= grace_period:
# Grace period elapsed, send for deletion
logger.info(
f"Document {doc_id} missing for {time_missing:.1f}s "
f"(>{grace_period:.1f}s grace period), sending deletion"
)
await send_stream.send(
DocumentTask(
user_id=user_id,
doc_id=doc_id,
doc_type="note",
operation="delete",
modified_at=0,
)
)
queued += 1
# Remove from tracking after sending deletion
del _potentially_deleted[doc_key]
else:
logger.debug(
f"Document {doc_id} still missing "
f"({time_missing:.1f}s/{grace_period:.1f}s grace period)"
)
else:
# First time missing, add to grace period tracking
logger.debug(
f"Document {doc_id} missing for first time, starting grace period"
f"Document {doc_id} reappeared, removing from deletion grace period"
)
_potentially_deleted[doc_key] = current_time
del _potentially_deleted[doc_key]
if queued > 0:
logger.info(f"Sent {queued} documents for incremental sync: {user_id}")
else:
logger.debug(f"No changes detected for {user_id}")
# Send if never indexed or modified since last index
if indexed_at is None or modified_at > indexed_at:
await send_stream.send(
DocumentTask(
user_id=user_id,
doc_id=doc_id,
doc_type="note",
operation="index",
modified_at=modified_at,
)
)
queued += 1
# Check for deleted documents (in Qdrant but not in Nextcloud)
# Use grace period: only delete after 2 consecutive scans confirm absence
settings = get_settings()
grace_period = (
settings.vector_sync_scan_interval * 1.5
) # Allow 1.5 scan intervals
current_time = time.time()
for doc_id in indexed_docs:
if doc_id not in nextcloud_doc_ids:
doc_key = (user_id, doc_id)
if doc_key in _potentially_deleted:
# Already marked as potentially deleted, check if grace period elapsed
first_missing_time = _potentially_deleted[doc_key]
time_missing = current_time - first_missing_time
if time_missing >= grace_period:
# Grace period elapsed, send for deletion
logger.info(
f"Document {doc_id} missing for {time_missing:.1f}s "
f"(>{grace_period:.1f}s grace period), sending deletion"
)
await send_stream.send(
DocumentTask(
user_id=user_id,
doc_id=doc_id,
doc_type="note",
operation="delete",
modified_at=0,
)
)
queued += 1
# Remove from tracking after sending deletion
del _potentially_deleted[doc_key]
else:
logger.debug(
f"Document {doc_id} still missing "
f"({time_missing:.1f}s/{grace_period:.1f}s grace period)"
)
else:
# First time missing, add to grace period tracking
logger.debug(
f"Document {doc_id} missing for first time, starting grace period"
)
_potentially_deleted[doc_key] = current_time
if queued > 0:
logger.info(f"Sent {queued} documents for incremental sync: {user_id}")
else:
logger.debug(f"No changes detected for {user_id}")
+11 -2
View File
@@ -1,6 +1,6 @@
[project]
name = "nextcloud-mcp-server"
version = "0.26.1"
version = "0.32.1"
description = "Model Context Protocol (MCP) server for Nextcloud integration - enables AI assistants to interact with Nextcloud data"
authors = [
{name = "Chris Coutinho", email = "chris@coutinho.io"}
@@ -22,6 +22,15 @@ dependencies = [
"aiosqlite>=0.20.0", # Async SQLite for refresh token storage
"authlib>=1.6.5",
"qdrant-client>=1.7.0",
# Observability dependencies
"prometheus-client>=0.21.0", # Prometheus metrics
"opentelemetry-api>=1.28.2", # OpenTelemetry API
"opentelemetry-sdk>=1.28.2", # OpenTelemetry SDK
"opentelemetry-instrumentation-asgi>=0.49b2", # Auto-instrument ASGI/Starlette
"opentelemetry-instrumentation-httpx>=0.49b2", # Auto-instrument httpx client
"opentelemetry-instrumentation-logging>=0.49b2", # Logging integration
"opentelemetry-exporter-otlp-proto-grpc>=1.28.2", # OTLP gRPC exporter
"python-json-logger>=3.2.0", # Structured JSON logging
]
classifiers = [
"Development Status :: 4 - Beta",
@@ -107,7 +116,7 @@ dev = [
]
[project.scripts]
nextcloud-mcp-server = "nextcloud_mcp_server.app:run"
nextcloud-mcp-server = "nextcloud_mcp_server.cli:run"
[[tool.uv.index]]
name = "testpypi"
+112
View File
@@ -0,0 +1,112 @@
"""Unit tests for permission checking."""
import pytest
from httpx import AsyncClient
from nextcloud_mcp_server.auth.permissions import is_nextcloud_admin
from nextcloud_mcp_server.client.users import UsersClient
@pytest.fixture
def mock_request(mocker):
"""Create a mock Starlette request."""
request = mocker.Mock()
request.user = mocker.Mock()
request.user.display_name = "testuser"
return request
@pytest.fixture
def mock_http_client(mocker):
"""Create a mock HTTP client."""
return mocker.AsyncMock(spec=AsyncClient)
@pytest.mark.unit
async def test_is_nextcloud_admin_true(mock_request, mock_http_client, mocker):
"""Test checking if user is admin (admin group membership)."""
# Mock the get_user_groups method to return admin group
mock_get_user_groups = mocker.patch.object(
UsersClient, "get_user_groups", return_value=["admin", "users"]
)
is_admin = await is_nextcloud_admin(mock_request, mock_http_client)
assert is_admin is True
mock_get_user_groups.assert_called_once_with("testuser")
@pytest.mark.unit
async def test_is_nextcloud_admin_false(mock_request, mock_http_client, mocker):
"""Test checking if user is not admin (no admin group membership)."""
# Mock the get_user_groups method to return no admin group
mock_get_user_groups = mocker.patch.object(
UsersClient, "get_user_groups", return_value=["users", "editors"]
)
is_admin = await is_nextcloud_admin(mock_request, mock_http_client)
assert is_admin is False
mock_get_user_groups.assert_called_once_with("testuser")
@pytest.mark.unit
async def test_is_nextcloud_admin_empty_groups(mock_request, mock_http_client, mocker):
"""Test checking admin status when user has no groups."""
# Mock the get_user_groups method to return empty list
mock_get_user_groups = mocker.patch.object(
UsersClient, "get_user_groups", return_value=[]
)
is_admin = await is_nextcloud_admin(mock_request, mock_http_client)
assert is_admin is False
mock_get_user_groups.assert_called_once_with("testuser")
@pytest.mark.unit
async def test_is_nextcloud_admin_no_username(mock_request, mock_http_client, mocker):
"""Test checking admin status when username is missing."""
# Set username to None
mock_request.user.display_name = None
mock_get_user_groups = mocker.patch.object(UsersClient, "get_user_groups")
is_admin = await is_nextcloud_admin(mock_request, mock_http_client)
assert is_admin is False
# Ensure get_user_groups was not called
mock_get_user_groups.assert_not_called()
@pytest.mark.unit
async def test_is_nextcloud_admin_api_error(mock_request, mock_http_client, mocker):
"""Test checking admin status when API call fails."""
# Mock the get_user_groups method to raise an exception
mock_get_user_groups = mocker.patch.object(
UsersClient,
"get_user_groups",
side_effect=Exception("API error"),
)
is_admin = await is_nextcloud_admin(mock_request, mock_http_client)
assert is_admin is False
mock_get_user_groups.assert_called_once_with("testuser")
@pytest.mark.unit
async def test_is_nextcloud_admin_case_sensitive(
mock_request, mock_http_client, mocker
):
"""Test that admin group check is case-sensitive."""
# Mock with "Admin" (capital A) instead of "admin"
mock_get_user_groups = mocker.patch.object(
UsersClient, "get_user_groups", return_value=["Admin", "users"]
)
is_admin = await is_nextcloud_admin(mock_request, mock_http_client)
# Should be False because Nextcloud uses lowercase "admin"
assert is_admin is False
mock_get_user_groups.assert_called_once_with("testuser")
+37 -14
View File
@@ -239,23 +239,46 @@ async def test_attachments_category_change_handling(nc_client: NextcloudClient):
assert retrieved_content1 == attachment_content
logger.info("Attachment retrieved successfully from initial category.")
# 4. Update note category
# 4. Update note category (with retry for ETag conflicts from background scanner)
logger.info(
f"Updating note {note_id} category from '{initial_category}' to '{new_category}'"
)
# Need to fetch the latest etag after attachment add (WebDAV ops don't update note etag)
current_note_data = await nc_client.notes.get_note(note_id=note_id)
current_etag = current_note_data["etag"]
updated_note = await nc_client.notes.update(
note_id=note_id,
etag=current_etag,
category=new_category,
title=note_title,
content="Updated content", # Pass required fields
)
etag3 = updated_note["etag"]
assert updated_note["category"] == new_category
logger.info(f"Note category updated successfully. New Etag: {etag3}")
# Retry logic for 412 Precondition Failed (ETag conflict)
# This can happen if the background vector scanner touches the note
max_update_attempts = 3
for attempt in range(max_update_attempts):
try:
# Fetch the latest etag
current_note_data = await nc_client.notes.get_note(note_id=note_id)
current_etag = current_note_data["etag"]
logger.info(
f"Update attempt {attempt + 1}/{max_update_attempts}, current etag: {current_etag}"
)
updated_note = await nc_client.notes.update(
note_id=note_id,
etag=current_etag,
category=new_category,
title=note_title,
content="Updated content", # Pass required fields
)
etag3 = updated_note["etag"]
assert updated_note["category"] == new_category
logger.info(f"Note category updated successfully. New Etag: {etag3}")
break # Success, exit retry loop
except HTTPStatusError as e:
if e.response.status_code == 412 and attempt < max_update_attempts - 1:
# ETag conflict (likely from background scanner), retry
logger.warning(
f"ETag conflict (412) on attempt {attempt + 1}, retrying..."
)
time.sleep(1) # Brief delay before retry
continue
else:
# Not a 412 or out of retries, re-raise
raise
time.sleep(1)
# 5. Verify attachment retrieval from *new* category (passing new category)
+218
View File
@@ -0,0 +1,218 @@
"""Unit tests for WebhooksClient."""
import pytest
from httpx import AsyncClient
from nextcloud_mcp_server.client.webhooks import WebhooksClient
@pytest.fixture
def webhooks_client(mocker):
"""Create a WebhooksClient with mocked HTTP client."""
mock_http_client = mocker.AsyncMock(spec=AsyncClient)
return WebhooksClient(mock_http_client, "testuser")
@pytest.mark.unit
async def test_list_webhooks(webhooks_client, mocker):
"""Test listing registered webhooks."""
mock_response = mocker.Mock()
mock_response.json.return_value = {
"ocs": {
"data": [
{
"id": 1,
"uri": "http://example.com/webhook",
"event": "OCP\\Files\\Events\\Node\\NodeCreatedEvent",
"httpMethod": "POST",
},
{
"id": 2,
"uri": "http://example.com/webhook",
"event": "OCP\\Files\\Events\\Node\\NodeWrittenEvent",
"httpMethod": "POST",
},
]
}
}
mock_make_request = mocker.patch.object(
WebhooksClient, "_make_request", return_value=mock_response
)
webhooks = await webhooks_client.list_webhooks()
assert len(webhooks) == 2
assert webhooks[0]["id"] == 1
assert webhooks[0]["event"] == "OCP\\Files\\Events\\Node\\NodeCreatedEvent"
assert webhooks[1]["id"] == 2
mock_make_request.assert_called_once_with(
"GET",
"/ocs/v2.php/apps/webhook_listeners/api/v1/webhooks",
headers={"OCS-APIRequest": "true", "Accept": "application/json"},
)
@pytest.mark.unit
async def test_list_webhooks_empty(webhooks_client, mocker):
"""Test listing webhooks when none are registered."""
mock_response = mocker.Mock()
mock_response.json.return_value = {"ocs": {"data": []}}
mocker.patch.object(WebhooksClient, "_make_request", return_value=mock_response)
webhooks = await webhooks_client.list_webhooks()
assert webhooks == []
@pytest.mark.unit
async def test_create_webhook(webhooks_client, mocker):
"""Test creating a webhook registration."""
mock_response = mocker.Mock()
mock_response.json.return_value = {
"ocs": {
"data": {
"id": 123,
"uri": "http://example.com/webhook",
"event": "OCP\\Files\\Events\\Node\\NodeCreatedEvent",
"httpMethod": "POST",
"authMethod": "none",
}
}
}
mock_make_request = mocker.patch.object(
WebhooksClient, "_make_request", return_value=mock_response
)
webhook_data = await webhooks_client.create_webhook(
event="OCP\\Files\\Events\\Node\\NodeCreatedEvent",
uri="http://example.com/webhook",
)
assert webhook_data["id"] == 123
assert webhook_data["event"] == "OCP\\Files\\Events\\Node\\NodeCreatedEvent"
mock_make_request.assert_called_once()
call_args = mock_make_request.call_args
assert call_args[0][0] == "POST"
assert call_args[0][1] == "/ocs/v2.php/apps/webhook_listeners/api/v1/webhooks"
@pytest.mark.unit
async def test_create_webhook_with_filter(webhooks_client, mocker):
"""Test creating a webhook with event filter."""
mock_response = mocker.Mock()
mock_response.json.return_value = {
"ocs": {
"data": {
"id": 124,
"uri": "http://example.com/webhook",
"event": "OCP\\Files\\Events\\Node\\NodeCreatedEvent",
"eventFilter": {"user.uid": "bob"},
}
}
}
mock_make_request = mocker.patch.object(
WebhooksClient, "_make_request", return_value=mock_response
)
webhook_data = await webhooks_client.create_webhook(
event="OCP\\Files\\Events\\Node\\NodeCreatedEvent",
uri="http://example.com/webhook",
event_filter={"user.uid": "bob"},
)
assert webhook_data["id"] == 124
assert webhook_data["eventFilter"] == {"user.uid": "bob"}
mock_make_request.assert_called_once()
call_args = mock_make_request.call_args
assert call_args[1]["json"]["eventFilter"] == {"user.uid": "bob"}
@pytest.mark.unit
async def test_create_webhook_with_auth_headers(webhooks_client, mocker):
"""Test creating a webhook with authentication headers."""
mock_response = mocker.Mock()
mock_response.json.return_value = {
"ocs": {
"data": {
"id": 125,
"uri": "http://example.com/webhook",
"event": "OCP\\Files\\Events\\Node\\NodeCreatedEvent",
"authMethod": "bearer",
}
}
}
mock_make_request = mocker.patch.object(
WebhooksClient, "_make_request", return_value=mock_response
)
webhook_data = await webhooks_client.create_webhook(
event="OCP\\Files\\Events\\Node\\NodeCreatedEvent",
uri="http://example.com/webhook",
auth_method="bearer",
headers={"Authorization": "Bearer secret-token"},
)
assert webhook_data["id"] == 125
assert webhook_data["authMethod"] == "bearer"
mock_make_request.assert_called_once()
call_args = mock_make_request.call_args
assert call_args[1]["json"]["authMethod"] == "bearer"
assert call_args[1]["json"]["headers"] == {"Authorization": "Bearer secret-token"}
@pytest.mark.unit
async def test_delete_webhook(webhooks_client, mocker):
"""Test deleting a webhook registration."""
mock_response = mocker.Mock()
mock_make_request = mocker.patch.object(
WebhooksClient, "_make_request", return_value=mock_response
)
await webhooks_client.delete_webhook(webhook_id=123)
mock_make_request.assert_called_once_with(
"DELETE",
"/ocs/v2.php/apps/webhook_listeners/api/v1/webhooks/123",
headers={"OCS-APIRequest": "true", "Accept": "application/json"},
)
@pytest.mark.unit
async def test_get_webhook(webhooks_client, mocker):
"""Test getting a specific webhook by ID."""
mock_response = mocker.Mock()
mock_response.json.return_value = {
"ocs": {
"data": {
"id": 123,
"uri": "http://example.com/webhook",
"event": "OCP\\Files\\Events\\Node\\NodeCreatedEvent",
"httpMethod": "POST",
}
}
}
mock_make_request = mocker.patch.object(
WebhooksClient, "_make_request", return_value=mock_response
)
webhook_data = await webhooks_client.get_webhook(webhook_id=123)
assert webhook_data["id"] == 123
assert webhook_data["event"] == "OCP\\Files\\Events\\Node\\NodeCreatedEvent"
mock_make_request.assert_called_once_with(
"GET",
"/ocs/v2.php/apps/webhook_listeners/api/v1/webhooks/123",
headers={"OCS-APIRequest": "true", "Accept": "application/json"},
)
@@ -0,0 +1,322 @@
"""Integration tests for Qdrant collection auto-creation.
These tests validate that:
1. Collections are automatically created on first access
2. Dimension validation detects mismatches
3. Idempotent initialization (multiple calls don't fail)
4. Proper error handling and logging
"""
from unittest.mock import Mock
import pytest
from nextcloud_mcp_server.vector.qdrant_client import get_qdrant_client
pytestmark = pytest.mark.integration
@pytest.fixture(autouse=True)
async def reset_singleton():
"""Reset the global Qdrant client singleton between tests."""
global _qdrant_client
import nextcloud_mcp_server.vector.qdrant_client as qdrant_module
# Store original
original = qdrant_module._qdrant_client
# Reset for test
qdrant_module._qdrant_client = None
yield
# Restore original
qdrant_module._qdrant_client = original
@pytest.mark.integration
async def test_collection_auto_created_on_first_access(monkeypatch):
"""Test that collection is automatically created if it doesn't exist."""
# Mock settings
from nextcloud_mcp_server.config import Settings
mock_settings = Settings(
qdrant_location=":memory:",
ollama_embedding_model="nomic-embed-text",
vector_sync_enabled=False, # Disable background sync for test
)
monkeypatch.setattr(
"nextcloud_mcp_server.vector.qdrant_client.get_settings", lambda: mock_settings
)
# Mock embedding service - must have .provider attribute
from nextcloud_mcp_server.embedding import SimpleEmbeddingProvider
mock_provider = SimpleEmbeddingProvider(dimension=384)
mock_embedding_service = Mock()
mock_embedding_service.provider = mock_provider
mock_embedding_service.get_dimension = lambda: mock_provider.get_dimension()
monkeypatch.setattr(
"nextcloud_mcp_server.embedding.get_embedding_service",
lambda: mock_embedding_service,
)
# Get client (should trigger collection creation)
client = await get_qdrant_client()
# Verify client is initialized
assert client is not None
# Verify collection was created
collection_name = mock_settings.get_collection_name()
collections = await client.get_collections()
collection_names = [c.name for c in collections.collections]
assert collection_name in collection_names
# Verify collection has correct dimensions
collection_info = await client.get_collection(collection_name)
assert collection_info.config.params.vectors.size == 384
@pytest.mark.integration
async def test_existing_collection_reused(monkeypatch):
"""Test that existing collection is reused without error."""
# Mock settings
from nextcloud_mcp_server.config import Settings
mock_settings = Settings(
qdrant_location=":memory:",
ollama_embedding_model="nomic-embed-text",
vector_sync_enabled=False,
)
monkeypatch.setattr(
"nextcloud_mcp_server.vector.qdrant_client.get_settings", lambda: mock_settings
)
# Mock embedding service - must have .provider attribute
from nextcloud_mcp_server.embedding import SimpleEmbeddingProvider
mock_provider = SimpleEmbeddingProvider(dimension=384)
mock_embedding_service = Mock()
mock_embedding_service.provider = mock_provider
mock_embedding_service.get_dimension = lambda: mock_provider.get_dimension()
monkeypatch.setattr(
"nextcloud_mcp_server.embedding.get_embedding_service",
lambda: mock_embedding_service,
)
# First call - creates collection
_ = await get_qdrant_client()
collection_name = mock_settings.get_collection_name()
# Reset singleton to simulate second initialization
import nextcloud_mcp_server.vector.qdrant_client as qdrant_module
qdrant_module._qdrant_client = None
# Second call - should reuse existing collection
client2 = await get_qdrant_client()
# Verify both clients work
assert client2 is not None
# Verify collection still exists and wasn't recreated
collections = await client2.get_collections()
collection_names = [c.name for c in collections.collections]
assert collection_name in collection_names
# Verify dimensions unchanged
collection_info = await client2.get_collection(collection_name)
assert collection_info.config.params.vectors.size == 384
@pytest.mark.integration
async def test_dimension_mismatch_detected(monkeypatch, tmp_path):
"""Test that dimension mismatch raises clear error."""
# Use persistent temp directory so collection survives client reset
from nextcloud_mcp_server.config import Settings
qdrant_path = str(tmp_path / "qdrant_data")
mock_settings = Settings(
qdrant_location=qdrant_path,
ollama_embedding_model="nomic-embed-text",
vector_sync_enabled=False,
)
monkeypatch.setattr(
"nextcloud_mcp_server.vector.qdrant_client.get_settings", lambda: mock_settings
)
# First embedding service: 384 dimensions
from nextcloud_mcp_server.embedding import SimpleEmbeddingProvider
mock_provider_1 = SimpleEmbeddingProvider(dimension=384)
mock_embedding_service_1 = Mock()
mock_embedding_service_1.provider = mock_provider_1
mock_embedding_service_1.get_dimension = lambda: mock_provider_1.get_dimension()
monkeypatch.setattr(
"nextcloud_mcp_server.embedding.get_embedding_service",
lambda: mock_embedding_service_1,
)
# First call - creates collection with 384 dimensions
client1 = await get_qdrant_client()
collection_name = mock_settings.get_collection_name()
# Verify collection created
collection_info = await client1.get_collection(collection_name)
assert collection_info.config.params.vectors.size == 384
# Close client1 to release file lock
await client1.close()
# Reset singleton (but collection persists in temp directory)
import nextcloud_mcp_server.vector.qdrant_client as qdrant_module
qdrant_module._qdrant_client = None
# Change embedding service to different dimension (768)
mock_provider_2 = SimpleEmbeddingProvider(dimension=768)
mock_embedding_service_2 = Mock()
mock_embedding_service_2.provider = mock_provider_2
mock_embedding_service_2.get_dimension = lambda: mock_provider_2.get_dimension()
monkeypatch.setattr(
"nextcloud_mcp_server.embedding.get_embedding_service",
lambda: mock_embedding_service_2,
)
# Second call - should detect dimension mismatch and raise error
with pytest.raises(ValueError) as exc_info:
await get_qdrant_client()
# Verify error message is helpful
error_msg = str(exc_info.value)
assert "Dimension mismatch" in error_msg
assert "384" in error_msg # Old dimension
assert "768" in error_msg # New dimension
assert "Solutions:" in error_msg # Includes helpful solutions
@pytest.mark.integration
async def test_idempotent_initialization(monkeypatch):
"""Test that multiple calls to get_qdrant_client() are idempotent."""
# Mock settings
from nextcloud_mcp_server.config import Settings
mock_settings = Settings(
qdrant_location=":memory:",
ollama_embedding_model="nomic-embed-text",
vector_sync_enabled=False,
)
monkeypatch.setattr(
"nextcloud_mcp_server.vector.qdrant_client.get_settings", lambda: mock_settings
)
# Mock embedding service - must have .provider attribute
from nextcloud_mcp_server.embedding import SimpleEmbeddingProvider
mock_provider = SimpleEmbeddingProvider(dimension=384)
mock_embedding_service = Mock()
mock_embedding_service.provider = mock_provider
mock_embedding_service.get_dimension = lambda: mock_provider.get_dimension()
monkeypatch.setattr(
"nextcloud_mcp_server.embedding.get_embedding_service",
lambda: mock_embedding_service,
)
# Call multiple times
client1 = await get_qdrant_client()
client2 = await get_qdrant_client()
client3 = await get_qdrant_client()
# All should return same singleton instance
assert client1 is client2
assert client2 is client3
# Collection should exist
collection_name = mock_settings.get_collection_name()
collections = await client1.get_collections()
collection_names = [c.name for c in collections.collections]
assert collection_name in collection_names
@pytest.mark.integration
async def test_collection_name_generation(monkeypatch):
"""Test that collection name is correctly generated from deployment ID and model."""
# Mock settings with custom deployment ID
from nextcloud_mcp_server.config import Settings
mock_settings = Settings(
qdrant_location=":memory:",
ollama_embedding_model="test-model",
vector_sync_enabled=False,
)
# Mock deployment ID
monkeypatch.setenv("MCP_DEPLOYMENT_ID", "test-deployment")
monkeypatch.setattr(
"nextcloud_mcp_server.vector.qdrant_client.get_settings", lambda: mock_settings
)
# Mock embedding service - must have .provider attribute
from nextcloud_mcp_server.embedding import SimpleEmbeddingProvider
mock_provider = SimpleEmbeddingProvider(dimension=384)
mock_embedding_service = Mock()
mock_embedding_service.provider = mock_provider
mock_embedding_service.get_dimension = lambda: mock_provider.get_dimension()
monkeypatch.setattr(
"nextcloud_mcp_server.embedding.get_embedding_service",
lambda: mock_embedding_service,
)
# Get client
client = await get_qdrant_client()
# Verify collection name includes deployment ID and model
collection_name = mock_settings.get_collection_name()
assert "test-deployment" in collection_name or "test-model" in collection_name
# Verify collection was created with that name
collections = await client.get_collections()
collection_names = [c.name for c in collections.collections]
assert collection_name in collection_names
@pytest.mark.integration
async def test_collection_uses_cosine_distance(monkeypatch):
"""Test that created collection uses COSINE distance metric."""
# Mock settings
from nextcloud_mcp_server.config import Settings
mock_settings = Settings(
qdrant_location=":memory:",
ollama_embedding_model="nomic-embed-text",
vector_sync_enabled=False,
)
monkeypatch.setattr(
"nextcloud_mcp_server.vector.qdrant_client.get_settings", lambda: mock_settings
)
# Mock embedding service - must have .provider attribute
from nextcloud_mcp_server.embedding import SimpleEmbeddingProvider
mock_provider = SimpleEmbeddingProvider(dimension=384)
mock_embedding_service = Mock()
mock_embedding_service.provider = mock_provider
mock_embedding_service.get_dimension = lambda: mock_provider.get_dimension()
monkeypatch.setattr(
"nextcloud_mcp_server.embedding.get_embedding_service",
lambda: mock_embedding_service,
)
# Get client (creates collection)
client = await get_qdrant_client()
# Verify collection uses COSINE distance
collection_name = mock_settings.get_collection_name()
collection_info = await client.get_collection(collection_name)
from qdrant_client.models import Distance
assert collection_info.config.params.vectors.distance == Distance.COSINE
+16 -5
View File
@@ -146,12 +146,23 @@ Avoid blocking operations in async code.""",
assert "search_method" in result
# For this test, sampling might fail (no real LLM client)
# So we check for either success or fallback
if "[Sampling unavailable" in result["generated_answer"]:
# Fallback mode - should still have sources
assert result["search_method"] == "semantic_sampling_fallback"
# So we check for either success or various fallback states
unsupported_methods = {
"semantic_sampling_unsupported",
"semantic_sampling_user_declined",
"semantic_sampling_timeout",
"semantic_sampling_mcp_error",
"semantic_sampling_fallback",
}
if result["search_method"] in unsupported_methods:
# Fallback/unsupported mode - should still have sources
assert len(result["sources"]) > 0
pytest.skip("Sampling not supported by test client (expected fallback)")
assert result["total_found"] > 0
pytest.skip(
f"Sampling not available (method: {result['search_method']}), "
f"but search results returned successfully"
)
else:
# Successful sampling
assert result["search_method"] == "semantic_sampling"
+1 -1
View File
@@ -28,7 +28,7 @@ import httpx
from playwright.async_api import async_playwright
from nextcloud_mcp_server.auth.client_registration import ensure_oauth_client
from nextcloud_mcp_server.auth.refresh_token_storage import RefreshTokenStorage
from nextcloud_mcp_server.auth.storage import RefreshTokenStorage
from nextcloud_mcp_server.client import NextcloudClient
from tests.load.oauth_metrics import OAuthBenchmarkMetrics
from tests.load.oauth_pool import (
+1 -1
View File
@@ -11,7 +11,7 @@ from unittest.mock import AsyncMock, MagicMock, patch
import jwt
import pytest
from nextcloud_mcp_server.auth.refresh_token_storage import RefreshTokenStorage
from nextcloud_mcp_server.auth.storage import RefreshTokenStorage
from nextcloud_mcp_server.auth.token_broker import TokenBrokerService
from nextcloud_mcp_server.auth.token_exchange import TokenExchangeService
+6 -6
View File
@@ -5,7 +5,7 @@ import os
import pytest
from click.testing import CliRunner
from nextcloud_mcp_server.app import run
from nextcloud_mcp_server.cli import run
@pytest.fixture
@@ -103,7 +103,7 @@ def test_cli_options_set_environment_variables(runner, clean_env, monkeypatch):
raise SystemExit(0)
# Patch get_app to capture env vars
monkeypatch.setattr("nextcloud_mcp_server.app.get_app", mock_get_app)
monkeypatch.setattr("nextcloud_mcp_server.cli.get_app", mock_get_app)
_ = runner.invoke(
run,
@@ -158,7 +158,7 @@ def test_cli_options_override_environment_variables(runner, monkeypatch):
)
raise SystemExit(0)
monkeypatch.setattr("nextcloud_mcp_server.app.get_app", mock_get_app)
monkeypatch.setattr("nextcloud_mcp_server.cli.get_app", mock_get_app)
# Provide CLI options that should override env vars
_ = runner.invoke(
@@ -211,7 +211,7 @@ def test_environment_variables_used_when_cli_not_provided(runner, monkeypatch):
)
raise SystemExit(0)
monkeypatch.setattr("nextcloud_mcp_server.app.get_app", mock_get_app)
monkeypatch.setattr("nextcloud_mcp_server.cli.get_app", mock_get_app)
# Don't provide any CLI options - should use env vars
_ = runner.invoke(run, [])
@@ -243,7 +243,7 @@ def test_default_values(runner, clean_env, monkeypatch):
)
raise SystemExit(0)
monkeypatch.setattr("nextcloud_mcp_server.app.get_app", mock_get_app)
monkeypatch.setattr("nextcloud_mcp_server.cli.get_app", mock_get_app)
# Don't provide CLI options or env vars - should use defaults
_ = runner.invoke(run, [])
@@ -275,7 +275,7 @@ def test_oauth_token_type_case_normalization(runner, clean_env, monkeypatch):
)
raise SystemExit(0)
monkeypatch.setattr("nextcloud_mcp_server.app.get_app", mock_get_app)
monkeypatch.setattr("nextcloud_mcp_server.cli.get_app", mock_get_app)
# Test uppercase JWT
runner.invoke(run, ["--oauth-token-type", "JWT"])
+108
View File
@@ -151,3 +151,111 @@ class TestGetSettings:
assert settings.vector_sync_scan_interval == 600
assert settings.vector_sync_processor_workers == 5
assert settings.vector_sync_queue_max_size == 5000
class TestChunkConfigValidation:
"""Test document chunking configuration validation."""
def test_default_chunk_settings(self):
"""Test default chunk size and overlap values."""
settings = Settings()
assert settings.document_chunk_size == 512
assert settings.document_chunk_overlap == 50
def test_valid_chunk_settings(self):
"""Test valid chunk size and overlap configuration."""
settings = Settings(
document_chunk_size=1024,
document_chunk_overlap=100,
)
assert settings.document_chunk_size == 1024
assert settings.document_chunk_overlap == 100
def test_overlap_greater_than_or_equal_to_chunk_size_raises_error(self):
"""Test that overlap >= chunk size raises ValueError."""
with pytest.raises(
ValueError,
match="DOCUMENT_CHUNK_OVERLAP .* must be less than DOCUMENT_CHUNK_SIZE",
):
Settings(
document_chunk_size=512,
document_chunk_overlap=512,
)
def test_overlap_larger_than_chunk_size_raises_error(self):
"""Test that overlap > chunk size raises ValueError."""
with pytest.raises(
ValueError,
match="DOCUMENT_CHUNK_OVERLAP .* must be less than DOCUMENT_CHUNK_SIZE",
):
Settings(
document_chunk_size=256,
document_chunk_overlap=300,
)
def test_negative_overlap_raises_error(self):
"""Test that negative overlap raises ValueError."""
with pytest.raises(
ValueError,
match="DOCUMENT_CHUNK_OVERLAP .* cannot be negative",
):
Settings(
document_chunk_size=512,
document_chunk_overlap=-10,
)
def test_small_chunk_size_warning(self, caplog):
"""Test that chunk size < 100 triggers warning."""
import logging
caplog.set_level(logging.WARNING, logger="nextcloud_mcp_server.config")
Settings(
document_chunk_size=64,
document_chunk_overlap=10,
)
assert (
"DOCUMENT_CHUNK_SIZE is set to 64 words, which is quite small"
in caplog.text
)
assert "Consider using at least 256 words" in caplog.text
def test_reasonable_chunk_size_no_warning(self, caplog):
"""Test that chunk size >= 100 doesn't trigger warning."""
import logging
caplog.set_level(logging.WARNING, logger="nextcloud_mcp_server.config")
Settings(
document_chunk_size=256,
document_chunk_overlap=25,
)
assert "DOCUMENT_CHUNK_SIZE" not in caplog.text
@patch.dict(
os.environ,
{
"DOCUMENT_CHUNK_SIZE": "1024",
"DOCUMENT_CHUNK_OVERLAP": "102",
},
clear=True,
)
def test_get_settings_chunk_config(self):
"""Test get_settings() with chunk configuration."""
settings = get_settings()
assert settings.document_chunk_size == 1024
assert settings.document_chunk_overlap == 102
@patch.dict(
os.environ,
{
"DOCUMENT_CHUNK_SIZE": "256",
"DOCUMENT_CHUNK_OVERLAP": "256",
},
clear=True,
)
def test_get_settings_invalid_chunk_config_raises_error(self):
"""Test get_settings() raises error for invalid chunk config."""
with pytest.raises(
ValueError,
match="DOCUMENT_CHUNK_OVERLAP .* must be less than DOCUMENT_CHUNK_SIZE",
):
get_settings()
+88
View File
@@ -0,0 +1,88 @@
"""Unit tests for logging filters."""
import logging
import pytest
from nextcloud_mcp_server.observability.logging_config import HealthCheckFilter
@pytest.mark.unit
class TestHealthCheckFilter:
"""Tests for the HealthCheckFilter."""
def test_filters_health_live_requests(self):
"""Test that /health/live requests are filtered out."""
# Create a log record that looks like a uvicorn access log for /health/live
record = logging.LogRecord(
name="uvicorn.access",
level=logging.INFO,
pathname="",
lineno=0,
msg='127.0.0.1:12345 - "GET /health/live HTTP/1.1" 200',
args=(),
exc_info=None,
)
filter_instance = HealthCheckFilter()
assert filter_instance.filter(record) is False
def test_filters_health_ready_requests(self):
"""Test that /health/ready requests are filtered out."""
record = logging.LogRecord(
name="uvicorn.access",
level=logging.INFO,
pathname="",
lineno=0,
msg='127.0.0.1:12345 - "GET /health/ready HTTP/1.1" 200',
args=(),
exc_info=None,
)
filter_instance = HealthCheckFilter()
assert filter_instance.filter(record) is False
def test_filters_metrics_requests(self):
"""Test that /metrics requests are filtered out."""
record = logging.LogRecord(
name="uvicorn.access",
level=logging.INFO,
pathname="",
lineno=0,
msg='127.0.0.1:12345 - "GET /metrics HTTP/1.1" 200',
args=(),
exc_info=None,
)
filter_instance = HealthCheckFilter()
assert filter_instance.filter(record) is False
def test_allows_other_requests(self):
"""Test that non-health-check requests are not filtered."""
record = logging.LogRecord(
name="uvicorn.access",
level=logging.INFO,
pathname="",
lineno=0,
msg='127.0.0.1:12345 - "GET /mcp/messages HTTP/1.1" 200',
args=(),
exc_info=None,
)
filter_instance = HealthCheckFilter()
assert filter_instance.filter(record) is True
def test_allows_api_requests(self):
"""Test that API requests are not filtered."""
record = logging.LogRecord(
name="uvicorn.access",
level=logging.INFO,
pathname="",
lineno=0,
msg='127.0.0.1:12345 - "POST /oauth/login HTTP/1.1" 302',
args=(),
exc_info=None,
)
filter_instance = HealthCheckFilter()
assert filter_instance.filter(record) is True
+112
View File
@@ -0,0 +1,112 @@
"""Unit tests for webhook preset filtering."""
import pytest
from nextcloud_mcp_server.server.webhook_presets import (
filter_presets_by_installed_apps,
get_preset,
list_presets,
)
@pytest.mark.unit
def test_list_all_presets():
"""Test listing all presets returns 5 presets."""
presets = list_presets()
assert len(presets) == 5
preset_ids = [preset_id for preset_id, _ in presets]
assert "notes_sync" in preset_ids
assert "calendar_sync" in preset_ids
assert "tables_sync" in preset_ids
assert "forms_sync" in preset_ids
assert "files_sync" in preset_ids
@pytest.mark.unit
def test_get_preset_existing():
"""Test getting an existing preset."""
preset = get_preset("notes_sync")
assert preset is not None
assert preset["name"] == "Notes Sync"
assert preset["app"] == "notes"
assert len(preset["events"]) == 3
@pytest.mark.unit
def test_get_preset_nonexistent():
"""Test getting a nonexistent preset returns None."""
preset = get_preset("nonexistent_sync")
assert preset is None
@pytest.mark.unit
def test_filter_presets_all_apps_installed():
"""Test filtering when all apps are installed."""
installed_apps = ["notes", "calendar", "tables", "forms"]
filtered = filter_presets_by_installed_apps(installed_apps)
assert len(filtered) == 5 # All 5 presets (files is always included)
preset_ids = [preset_id for preset_id, _ in filtered]
assert "notes_sync" in preset_ids
assert "calendar_sync" in preset_ids
assert "tables_sync" in preset_ids
assert "forms_sync" in preset_ids
assert "files_sync" in preset_ids
@pytest.mark.unit
def test_filter_presets_subset_installed():
"""Test filtering when only some apps are installed."""
installed_apps = ["notes", "calendar"]
filtered = filter_presets_by_installed_apps(installed_apps)
assert len(filtered) == 3 # notes, calendar, files
preset_ids = [preset_id for preset_id, _ in filtered]
assert "notes_sync" in preset_ids
assert "calendar_sync" in preset_ids
assert "files_sync" in preset_ids
assert "tables_sync" not in preset_ids
assert "forms_sync" not in preset_ids
@pytest.mark.unit
def test_filter_presets_no_apps_installed():
"""Test filtering when no optional apps are installed."""
installed_apps = []
filtered = filter_presets_by_installed_apps(installed_apps)
assert len(filtered) == 1 # Only files
preset_ids = [preset_id for preset_id, _ in filtered]
assert "files_sync" in preset_ids
assert "notes_sync" not in preset_ids
assert "calendar_sync" not in preset_ids
@pytest.mark.unit
def test_filter_presets_files_always_included():
"""Test that files preset is always included regardless of installed apps."""
# Empty list
filtered = filter_presets_by_installed_apps([])
preset_ids = [preset_id for preset_id, _ in filtered]
assert "files_sync" in preset_ids
# List with other apps but not explicitly "files"
filtered = filter_presets_by_installed_apps(["notes", "calendar"])
preset_ids = [preset_id for preset_id, _ in filtered]
assert "files_sync" in preset_ids
@pytest.mark.unit
def test_filter_presets_forms_included_when_installed():
"""Test that forms preset is included when Forms app is installed."""
installed_apps = ["forms"]
filtered = filter_presets_by_installed_apps(installed_apps)
preset_ids = [preset_id for preset_id, _ in filtered]
assert "forms_sync" in preset_ids
assert len(filtered) == 2 # forms + files
@pytest.mark.unit
def test_filter_presets_forms_excluded_when_not_installed():
"""Test that forms preset is excluded when Forms app is not installed."""
installed_apps = ["notes", "calendar", "tables"]
filtered = filter_presets_by_installed_apps(installed_apps)
preset_ids = [preset_id for preset_id, _ in filtered]
assert "forms_sync" not in preset_ids
+195
View File
@@ -0,0 +1,195 @@
"""
Unit tests for Webhook Storage functionality.
Tests the webhook tracking methods in RefreshTokenStorage without
requiring real database connections or network calls.
"""
import tempfile
import time
from pathlib import Path
import pytest
from nextcloud_mcp_server.auth.storage import RefreshTokenStorage
pytestmark = pytest.mark.unit
@pytest.fixture
async def temp_storage():
"""Create temporary storage instance for testing."""
with tempfile.TemporaryDirectory() as tmpdir:
db_path = Path(tmpdir) / "test_webhooks.db"
# No encryption key needed for webhook tracking
storage = RefreshTokenStorage(db_path=str(db_path), encryption_key=None)
await storage.initialize()
yield storage
async def test_store_webhook(temp_storage):
"""Test storing a webhook."""
await temp_storage.store_webhook(webhook_id=123, preset_id="notes_sync")
webhooks = await temp_storage.list_all_webhooks()
assert len(webhooks) == 1
assert webhooks[0]["webhook_id"] == 123
assert webhooks[0]["preset_id"] == "notes_sync"
assert "created_at" in webhooks[0]
async def test_store_webhook_duplicate(temp_storage):
"""Test storing duplicate webhook replaces existing."""
await temp_storage.store_webhook(webhook_id=123, preset_id="notes_sync")
await temp_storage.store_webhook(webhook_id=123, preset_id="calendar_sync")
webhooks = await temp_storage.list_all_webhooks()
# Should only have one entry due to UNIQUE constraint
assert len(webhooks) == 1
assert webhooks[0]["preset_id"] == "calendar_sync"
async def test_get_webhooks_by_preset(temp_storage):
"""Test retrieving webhooks by preset."""
await temp_storage.store_webhook(webhook_id=123, preset_id="notes_sync")
await temp_storage.store_webhook(webhook_id=456, preset_id="notes_sync")
await temp_storage.store_webhook(webhook_id=789, preset_id="calendar_sync")
notes_webhooks = await temp_storage.get_webhooks_by_preset("notes_sync")
assert len(notes_webhooks) == 2
assert 123 in notes_webhooks
assert 456 in notes_webhooks
calendar_webhooks = await temp_storage.get_webhooks_by_preset("calendar_sync")
assert len(calendar_webhooks) == 1
assert 789 in calendar_webhooks
async def test_get_webhooks_by_preset_empty(temp_storage):
"""Test retrieving webhooks for non-existent preset."""
webhooks = await temp_storage.get_webhooks_by_preset("nonexistent")
assert len(webhooks) == 0
async def test_delete_webhook(temp_storage):
"""Test deleting a webhook."""
await temp_storage.store_webhook(webhook_id=123, preset_id="notes_sync")
await temp_storage.store_webhook(webhook_id=456, preset_id="notes_sync")
deleted = await temp_storage.delete_webhook(webhook_id=123)
assert deleted is True
webhooks = await temp_storage.get_webhooks_by_preset("notes_sync")
assert len(webhooks) == 1
assert 456 in webhooks
async def test_delete_webhook_nonexistent(temp_storage):
"""Test deleting non-existent webhook."""
deleted = await temp_storage.delete_webhook(webhook_id=999)
assert deleted is False
async def test_list_all_webhooks(temp_storage):
"""Test listing all webhooks."""
await temp_storage.store_webhook(webhook_id=123, preset_id="notes_sync")
await temp_storage.store_webhook(webhook_id=456, preset_id="calendar_sync")
await temp_storage.store_webhook(webhook_id=789, preset_id="notes_sync")
webhooks = await temp_storage.list_all_webhooks()
assert len(webhooks) == 3
# Verify all expected fields present
for webhook in webhooks:
assert "webhook_id" in webhook
assert "preset_id" in webhook
assert "created_at" in webhook
# Verify webhook IDs
webhook_ids = [w["webhook_id"] for w in webhooks]
assert 123 in webhook_ids
assert 456 in webhook_ids
assert 789 in webhook_ids
async def test_list_all_webhooks_empty(temp_storage):
"""Test listing webhooks when none exist."""
webhooks = await temp_storage.list_all_webhooks()
assert len(webhooks) == 0
async def test_clear_preset_webhooks(temp_storage):
"""Test clearing all webhooks for a preset."""
await temp_storage.store_webhook(webhook_id=123, preset_id="notes_sync")
await temp_storage.store_webhook(webhook_id=456, preset_id="notes_sync")
await temp_storage.store_webhook(webhook_id=789, preset_id="calendar_sync")
deleted_count = await temp_storage.clear_preset_webhooks("notes_sync")
assert deleted_count == 2
# Verify notes_sync webhooks are gone
notes_webhooks = await temp_storage.get_webhooks_by_preset("notes_sync")
assert len(notes_webhooks) == 0
# Verify calendar_sync webhook still exists
calendar_webhooks = await temp_storage.get_webhooks_by_preset("calendar_sync")
assert len(calendar_webhooks) == 1
assert 789 in calendar_webhooks
async def test_clear_preset_webhooks_nonexistent(temp_storage):
"""Test clearing webhooks for non-existent preset."""
deleted_count = await temp_storage.clear_preset_webhooks("nonexistent")
assert deleted_count == 0
async def test_webhook_timestamps(temp_storage):
"""Test that webhook timestamps are properly stored."""
start_time = time.time()
await temp_storage.store_webhook(webhook_id=123, preset_id="notes_sync")
end_time = time.time()
webhooks = await temp_storage.list_all_webhooks()
assert len(webhooks) == 1
created_at = webhooks[0]["created_at"]
assert start_time <= created_at <= end_time
async def test_storage_without_encryption_key():
"""Test that storage can be initialized without encryption key."""
with tempfile.TemporaryDirectory() as tmpdir:
db_path = Path(tmpdir) / "test_no_encryption.db"
storage = RefreshTokenStorage(db_path=str(db_path), encryption_key=None)
await storage.initialize()
# Webhook operations should work without encryption key
await storage.store_webhook(webhook_id=123, preset_id="notes_sync")
webhooks = await storage.get_webhooks_by_preset("notes_sync")
assert len(webhooks) == 1
assert 123 in webhooks
async def test_multiple_presets_independence(temp_storage):
"""Test that different presets maintain independent webhook lists."""
presets = ["notes_sync", "calendar_sync", "deck_sync", "files_sync"]
# Store webhooks for each preset
for i, preset in enumerate(presets):
webhook_id = 100 + i
await temp_storage.store_webhook(webhook_id=webhook_id, preset_id=preset)
# Verify each preset has exactly one webhook
for i, preset in enumerate(presets):
webhooks = await temp_storage.get_webhooks_by_preset(preset)
assert len(webhooks) == 1
assert (100 + i) in webhooks
# Clear one preset
deleted = await temp_storage.clear_preset_webhooks("notes_sync")
assert deleted == 1
# Verify other presets unchanged
for preset in ["calendar_sync", "deck_sync", "files_sync"]:
webhooks = await temp_storage.get_webhooks_by_preset(preset)
assert len(webhooks) == 1
Generated
+557 -332
View File
File diff suppressed because it is too large Load Diff
+532
View File
@@ -0,0 +1,532 @@
# Nextcloud Webhook Testing Findings
**Date:** 2025-11-11
**Purpose:** Manual validation of Nextcloud webhook schemas and behavior for vector sync integration (ADR-010)
## Executive Summary
Successfully tested and validated Nextcloud webhook payloads for file/note events and calendar events. **5 out of 6** webhook types were captured and validated against expected schemas from ADR-010 and Nextcloud documentation. One calendar deletion webhook did not fire during testing (potential Nextcloud issue or configuration).
## Test Environment
- **Nextcloud Version:** 30+ (Docker compose setup)
- **Webhook App:** `webhook_listeners` (bundled, enabled)
- **MCP Server:** Test endpoint at `http://mcp:8000/webhooks/nextcloud`
- **Background Worker:** Running with 60s timeout
- **Authentication:** None (test environment)
## Webhooks Registered
| ID | Event Class | Status |
|----|------------|--------|
| 1 | `OCP\Files\Events\Node\NodeCreatedEvent` | ✓ Tested |
| 2 | `OCP\Files\Events\Node\NodeWrittenEvent` | ✓ Tested |
| 3 | `OCP\Files\Events\Node\NodeDeletedEvent` | ✓ Tested |
| 4 | `OCP\Calendar\Events\CalendarObjectCreatedEvent` | ✓ Tested |
| 5 | `OCP\Calendar\Events\CalendarObjectUpdatedEvent` | ✓ Tested |
| 6 | `OCP\Calendar\Events\CalendarObjectDeletedEvent` | ✗ Not received |
## Captured Webhook Payloads
### 1. NodeCreatedEvent (File/Note Creation)
**Test Action:** Created note via Notes API
**Trigger Time:** 2025-11-11 08:37:25
**Webhooks Fired:** 3 events (folder creation + file creation + file written)
**Payload:**
```json
{
"user": {
"uid": "admin",
"displayName": "admin"
},
"time": 1762850245,
"event": {
"class": "OCP\\Files\\Events\\Node\\NodeCreatedEvent",
"node": {
"id": 437,
"path": "/admin/files/Notes/Webhooks/Webhook Test Note.md"
}
}
}
```
**Validation:**
- ✅ Schema matches ADR-010 specification
- ✅ Contains `user` object with `uid` and `displayName`
- ✅ Contains `time` (Unix timestamp)
- ✅ Contains `event.class` (fully qualified event name)
- ✅ Contains `event.node.id` (file ID)
- ✅ Contains `event.node.path` (absolute path)
**Observations:**
- Creating a note via Notes API triggers 3 webhook events:
1. `NodeCreatedEvent` for the parent folder (if new)
2. `NodeWrittenEvent` for the parent folder
3. `NodeCreatedEvent` for the actual file
4. `NodeWrittenEvent` for the file (sometimes fired 2x)
### 2. NodeWrittenEvent (File/Note Update)
**Test Action:** Updated note content via Notes API
**Trigger Time:** 2025-11-11 08:49:20
**Payload:**
```json
{
"user": {
"uid": "admin",
"displayName": "admin"
},
"time": 1762850960,
"event": {
"class": "OCP\\Files\\Events\\Node\\NodeWrittenEvent",
"node": {
"id": 437,
"path": "/admin/files/Notes/Webhooks/Webhook Test Note.md"
}
}
}
```
**Validation:**
- ✅ Schema identical to `NodeCreatedEvent` except for `event.class`
- ✅ Same file ID (437) as creation event
- ✅ Updated timestamp reflects actual modification time
**Observations:**
- File updates trigger a single `NodeWrittenEvent`
- No duplicate events fired for update operations
### 3. NodeDeletedEvent (File/Note Deletion)
**Test Action:** Deleted note via Notes API
**Trigger Time:** 2025-11-11 08:51:34
**Webhooks Fired:** 2 events (file + folder deletion)
**Payload:**
```json
{
"user": {
"uid": "admin",
"displayName": "admin"
},
"time": 1762851093,
"event": {
"class": "OCP\\Files\\Events\\Node\\NodeDeletedEvent",
"node": {
"path": "/admin/files/Notes/Webhooks/Webhook Test Note.md"
}
}
}
```
**Validation:**
- ✅ Schema matches ADR-010 specification
- ⚠️ **IMPORTANT:** No `node.id` field in deletion events (only `path`)
- ✅ Folder deletion triggered after file deletion (empty folder cleanup)
**Observations:**
- **Critical Difference:** Deletion events do NOT include `node.id`, only `node.path`
- This differs from Create/Write events which include both `id` and `path`
- ADR-010 implementation must handle missing `id` field for deletions
- Deleting a file also triggers deletion of empty parent folders
### 4. CalendarObjectCreatedEvent (Calendar Event Creation)
**Test Action:** Created calendar event via CalDAV PUT
**Trigger Time:** 2025-11-11 08:52:50
**Payload (partial - calendarData omitted for brevity):**
```json
{
"user": {
"uid": "admin",
"displayName": "admin"
},
"time": 1762851169,
"event": {
"calendarId": 1,
"class": "OCP\\Calendar\\Events\\CalendarObjectCreatedEvent",
"calendarData": {
"id": 1,
"uri": "personal",
"{http://calendarserver.org/ns/}getctag": "...",
"{http://sabredav.org/ns}sync-token": 21,
"{urn:ietf:params:xml:ns:caldav}supported-calendar-component-set": [],
"{urn:ietf:params:xml:ns:caldav}schedule-calendar-transp": [],
"{urn:ietf:params:xml:ns:caldav}calendar-timezone": null
},
"objectData": {
"id": 3,
"uri": "webhook-test-event-001.ics",
"lastmodified": 1762851169,
"etag": "\"2b937b7d77dc83c77329dfdb210ba9d0\"",
"calendarid": 1,
"size": 297,
"component": "vevent",
"classification": 0,
"uid": "webhook-test-event-001@nextcloud",
"calendardata": "BEGIN:VCALENDAR\r\nVERSION:2.0\r\n...",
"{http://nextcloud.com/ns}deleted-at": null
},
"shares": []
}
}
```
**Validation:**
- ✅ Schema matches Nextcloud documentation
- ✅ Contains complete calendar metadata (`calendarData`)
- ✅ Contains complete event data (`objectData`)
- ✅ Includes full iCal data in `objectData.calendardata`
- ✅ Includes `objectData.id` for database lookups
- ⚠️ **Complex:** Much more metadata than file events
**Observations:**
- Calendar webhooks include significantly more data than file webhooks
- Full iCal content is embedded in `objectData.calendardata`
- Event ID is in `objectData.id` (NOT `event.id`)
- `calendarData` contains calendar-level metadata
- `shares` array contains sharing information (empty in this test)
### 5. CalendarObjectUpdatedEvent (Calendar Event Update)
**Test Action:** Updated calendar event via CalDAV PUT
**Trigger Time:** 2025-11-11 08:53:28
**Payload (partial):**
```json
{
"user": {
"uid": "admin",
"displayName": "admin"
},
"time": 1762851207,
"event": {
"calendarId": 1,
"class": "OCP\\Calendar\\Events\\CalendarObjectUpdatedEvent",
"calendarData": { /* same structure as creation */ },
"objectData": {
"id": 3,
"uri": "webhook-test-event-001.ics",
"lastmodified": 1762851207,
"etag": "\"2695a18013e0991e4212b07b61d5e1e2\"",
"calendarid": 1,
"size": 315,
"component": "vevent",
"classification": 0,
"uid": "webhook-test-event-001@nextcloud",
"calendardata": "BEGIN:VCALENDAR\r\nVERSION:2.0\r\n...",
"{http://nextcloud.com/ns}deleted-at": null
},
"shares": []
}
}
```
**Validation:**
- ✅ Schema identical to `CalendarObjectCreatedEvent` except `event.class`
- ✅ Same event ID (3) as creation
- ✅ Updated `lastmodified` timestamp
- ✅ Different `etag` (changed from creation)
- ✅ Larger `size` (315 vs 297 bytes)
**Observations:**
- Update events contain full new state (not delta)
- ETag changes on updates (useful for conflict detection)
- Size field reflects actual iCal size
### 6. CalendarObjectDeletedEvent (Calendar Event Deletion)
**Test Action:** Deleted calendar event via CalDAV DELETE
**Trigger Time:** 2025-11-11 08:54:47
**Status:** ❌ **WEBHOOK DID NOT FIRE**
**Expected Payload (from Nextcloud docs):**
```json
{
"user": {
"uid": "admin",
"displayName": "admin"
},
"time": <timestamp>,
"event": {
"calendarId": 1,
"class": "OCP\\Calendar\\Events\\CalendarObjectDeletedEvent",
"calendarData": { /* calendar metadata */ },
"objectData": {
"id": 3,
"uri": "webhook-test-event-001.ics",
/* ... other fields ... */
},
"shares": []
}
}
```
**Issue:**
- Calendar event was successfully deleted (verified via CalDAV PROPFIND)
- Webhook registration confirmed (ID #6 in `webhook_listeners:list`)
- Background worker running and processing other events
- **No webhook notification received after 2+ minutes**
**Possible Causes:**
1. Known Nextcloud bug with calendar deletion webhooks
2. CalDAV DELETE may not trigger event system properly
3. Deletion event may require trash bin enabled
4. Background job may have silently failed
**Recommended Actions:**
- File Nextcloud issue report
- Test with trash bin enabled (`CalendarObjectMovedToTrashEvent`)
- Check Nextcloud error logs for webhook failures
- Verify with Nextcloud 31+ if issue persists
## Schema Comparison: Expected vs Actual
### File Events
| Field | Expected (ADR-010) | Actual | Match |
|-------|-------------------|--------|-------|
| `user.uid` | string | string | ✅ |
| `user.displayName` | string | string | ✅ |
| `time` | int | int | ✅ |
| `event.class` | string | string | ✅ |
| `event.node.id` | string | int | ⚠️ Type mismatch |
| `event.node.path` | string | string | ✅ |
**Type Discrepancy:** `node.id` is documented as `string` but returns as `int` (437 instead of "437")
### Calendar Events
| Field | Expected (Nextcloud docs) | Actual | Match |
|-------|-------------------------|--------|-------|
| `user.uid` | string | string | ✅ |
| `user.displayName` | string | string | ✅ |
| `time` | int | int | ✅ |
| `event.class` | string | string | ✅ |
| `event.calendarId` | int | int | ✅ |
| `event.calendarData.*` | object | object | ✅ |
| `event.objectData.id` | int | int | ✅ |
| `event.objectData.uri` | string | string | ✅ |
| `event.objectData.calendardata` | string | string | ✅ |
| `event.objectData.lastmodified` | int | int | ✅ |
| `event.objectData.etag` | string | string | ✅ |
| `event.objectData.component` | string\|null | string | ✅ |
| `event.shares` | array | array | ✅ |
All calendar event fields match expected schemas.
## Key Findings for ADR-010 Implementation
### 1. Deletion Events Have Different Schema
- **File Deletions:** No `node.id` field, only `node.path`
- **Calendar Deletions:** Not tested (webhook didn't fire)
- **Impact:** Webhook handler must check for `node.id` existence before using it
### 2. Multiple Webhooks Per Operation
- Creating a note triggers 3-5 webhook events
- Deleting a note triggers 2 events (file + folder)
- **Impact:** Deduplication logic needed in webhook handler
### 3. Event-Specific ID Fields
- **File events:** `event.node.id`
- **Calendar events:** `event.objectData.id`
- **Impact:** Event parser must handle different ID field locations
### 4. Full State vs Delta
- All webhooks contain complete current state (not delta)
- **Impact:** No need for "previous state" tracking in webhook handler
### 5. Calendar Data Richness
- Calendar webhooks include full iCal content
- **Impact:** Can extract all event metadata without additional API calls
## Recommendations for ADR-010 Implementation
### 1. Webhook Event Parser (`webhook_parser.py`)
```python
def extract_document_task(event_class: str, payload: dict) -> DocumentTask | None:
"""Extract DocumentTask from webhook event payload."""
user_id = payload["user"]["uid"]
event_data = payload["event"]
# File/Note events
if "NodeCreatedEvent" in event_class or "NodeWrittenEvent" in event_class:
path = event_data["node"]["path"]
# Only process markdown files for notes
if not path.endswith(".md"):
return None
# IMPORTANT: Check if 'id' exists (missing in deletion events)
doc_id = str(event_data["node"].get("id", ""))
if not doc_id:
# For missing ID, use path-based identifier
doc_id = f"path:{path}"
return DocumentTask(
user_id=user_id,
doc_id=doc_id,
doc_type="note",
operation="index",
modified_at=payload["time"],
)
# File deletion events
elif "NodeDeletedEvent" in event_class:
path = event_data["node"]["path"]
if not path.endswith(".md"):
return None
# Deletion events DON'T have node.id - use path
return DocumentTask(
user_id=user_id,
doc_id=f"path:{path}", # Path-based since ID unavailable
doc_type="note",
operation="delete",
modified_at=payload["time"],
)
# Calendar creation/update events
elif "CalendarObjectCreatedEvent" in event_class or \
"CalendarObjectUpdatedEvent" in event_class:
return DocumentTask(
user_id=user_id,
doc_id=str(event_data["objectData"]["id"]),
doc_type="calendar_event",
operation="index",
modified_at=event_data["objectData"]["lastmodified"],
)
# Calendar deletion events
elif "CalendarObjectDeletedEvent" in event_class:
return DocumentTask(
user_id=user_id,
doc_id=str(event_data["objectData"]["id"]),
doc_type="calendar_event",
operation="delete",
modified_at=payload["time"],
)
return None # Unsupported event type
```
### 2. Deduplication Strategy
**Problem:** Creating a note triggers 3-5 webhooks
**Solution:** Idempotent processing + task deduplication
```python
# In webhook handler
async def handle_nextcloud_webhook(request: Request) -> JSONResponse:
payload = await request.json()
task = extract_document_task(
payload["event"]["class"],
payload
)
if task:
# Idempotent: Queue will only process latest version
await document_queue.send(task)
return JSONResponse({"status": "received"}, status_code=200)
```
### 3. Path-Based Fallback for Deletions
Since deletion events lack `node.id`, use path-based identification:
```python
# In Qdrant delete logic
async def delete_document(user_id: str, doc_id: str, doc_type: str):
if doc_id.startswith("path:"):
# Path-based deletion
path = doc_id.removeprefix("path:")
# Search Qdrant for document with matching path in metadata
points = await qdrant.scroll(
collection_name=collection,
scroll_filter=Filter(must=[
FieldCondition(
key="user_id",
match=MatchValue(value=user_id),
),
FieldCondition(
key="metadata.path",
match=MatchValue(value=path),
),
]),
)
# Delete found points
else:
# ID-based deletion (normal case)
...
```
### 4. Webhook Registration Filters
To reduce webhook volume, add filters:
```json
{
"httpMethod": "POST",
"uri": "http://mcp:8000/webhooks/nextcloud",
"event": "OCP\\Files\\Events\\Node\\NodeCreatedEvent",
"eventFilter": {
"event.node.path": "/^.*\\.md$/"
}
}
```
This filters to only `.md` files at the webhook registration level (not handler level).
### 5. Monitoring and Metrics
Add webhook-specific metrics:
```python
webhook_notifications_received_total{event_type="note_created"} 42
webhook_processing_duration_seconds{event_type="note_created"} 0.023
webhook_errors_total{error_type="parse_error"} 2
webhook_duplicates_filtered_total{doc_type="note"} 15
```
## Testing Checklist for Implementation
- [x] File creation webhook triggers document indexing
- [x] File update webhook triggers reindexing
- [x] File deletion webhook triggers document removal
- [ ] File deletion without ID successfully removes document (path-based)
- [x] Calendar creation webhook triggers event indexing
- [x] Calendar update webhook triggers event reindexing
- [ ] Calendar deletion webhook triggers event removal (NOT TESTED - webhook didn't fire)
- [ ] Duplicate webhooks are deduplicated
- [ ] Non-markdown file webhooks are ignored
- [ ] Malformed webhook payloads return 400 error
- [ ] Webhook authentication validates shared secret
- [ ] Webhook processing completes within 50ms
## Appendix: Raw Webhook Logs
Complete webhook logs with full payloads are available in MCP container logs:
```bash
docker compose logs mcp | grep -A 30 "🔔 Webhook received"
```
## Conclusion
Nextcloud webhooks work as documented with minor exceptions:
1. ✅ **File/Note Events:** Fully functional and match expected schemas
2. ✅ **Calendar Creation/Update:** Fully functional with rich metadata
3. ❌ **Calendar Deletion:** Webhook did not fire (requires investigation)
4. ⚠️ **Schema Discrepancy:** `node.id` is integer (not string as documented)
5. ⚠️ **Deletion Schema:** Missing `node.id` field (only `path` provided)
**Overall Status:** Ready for ADR-010 implementation with noted caveats. Calendar deletion webhook issue should be reported to Nextcloud and may require alternative approach (polling or trash bin events).