Compare commits

...

27 Commits

Author SHA1 Message Date
github-actions[bot] 56a5c63994 bump: version 0.34.1 → 0.34.2 2025-11-13 21:11:36 +00:00
Chris Coutinho 92c8e1e41d Merge pull request #290 from cbcoutinho/renovate/quay.io-keycloak-keycloak-26.x
chore(deps): update quay.io/keycloak/keycloak docker tag to v26.4.5
2025-11-13 22:11:09 +01:00
github-actions[bot] dd12c957f6 bump: version 0.34.0 → 0.34.1 2025-11-13 21:10:16 +00:00
Chris Coutinho 74e2ab2440 Merge pull request #297 from cbcoutinho/fix/helm-oidc-env-vars
fix: Use NEXTCLOUD_OIDC_CLIENT_ID/SECRET env vars consistently
2025-11-13 22:10:04 +01:00
Chris Coutinho d124144424 Merge pull request #298 from cbcoutinho/fix/notes-search-empty-query
fix: return all notes when search query is empty
2025-11-13 22:09:50 +01:00
Chris Coutinho 39259ef282 ci: Run smoke tests only in ci 2025-11-13 22:06:07 +01:00
Chris Coutinho 3648d478f1 fix: return all notes when search query is empty
Previously, an empty query string to nc_notes_search_notes would return
zero results due to an early return when no query tokens were present.

This was counterintuitive - users expect an empty query to list all
notes, not return nothing.

Changes:
- Modified NotesSearchController.search_notes() to return all notes
  when query is empty
- Added documentation to clarify this behavior
- Empty query results have _score: None (no relevance scoring)
- Non-empty query results continue to have relevance scores

Fixes behavior where listing all notes was impossible via the search tool.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-13 21:57:14 +01:00
Chris Coutinho 14a59fdff3 fix: Use NEXTCLOUD_OIDC_CLIENT_ID/SECRET env vars consistently
Fixes #296

The application code was looking for OIDC_CLIENT_ID and OIDC_CLIENT_SECRET
(without NEXTCLOUD_ prefix), but the Helm chart, documentation, and CLI
all use NEXTCLOUD_OIDC_CLIENT_ID and NEXTCLOUD_OIDC_CLIENT_SECRET.

This mismatch caused OAuth deployments via Helm to fail with crashloops
because the credentials weren't being found.

Changes:
- app.py: Use NEXTCLOUD_OIDC_CLIENT_ID/SECRET in setup_oauth_config()
- config.py: Use NEXTCLOUD_OIDC_CLIENT_ID/SECRET in get_settings()
- Updated documentation comments and error messages

This aligns with the documented naming convention where all Nextcloud-related
environment variables use the NEXTCLOUD_ prefix.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-13 21:48:58 +01:00
github-actions[bot] 2f138e7539 bump: version 0.33.1 → 0.34.0 2025-11-13 16:15:29 +00:00
Chris Coutinho 2baacc0ae8 Merge pull request #295 from cbcoutinho/feat/complete-metrics-instrumentation
feat: Add metrics instrumentation (phases 1-3)
2025-11-13 17:15:03 +01:00
Chris Coutinho c3023d2cc3 feat: Complete Phase 5 - Instrument all 93 MCP tools
Applied @instrument_tool decorator to all 86 remaining tools
across 8 server files.

Instrumented files:
- calendar.py: 16 tools
- contacts.py: 7 tools
- deck.py: 25 tools
- webdav.py: 11 tools
- tables.py: 6 tools
- sharing.py: 5 tools
- cookbook.py: 13 tools
- semantic.py: 3 tools

Total: 93 tools instrumented (7 in notes.py + 86 in other files)

These metrics populate:
- MCP Tool Calls panel (by tool name and status)
- MCP Tool Duration panel (histogram)
- MCP Tool Errors panel (by tool name and error type)

This completes PR #295 - All 5 phases of metrics instrumentation done:
 Phase 1: Queue size metrics (2 locations)
 Phase 2: Health checks (1 location)
 Phase 3: Database operations (3 methods)
 Phase 4: OAuth token metrics (3 locations)
 Phase 5: MCP tool metrics (93 tools)

All 34 dashboard panels now have data sources.
2025-11-13 16:58:44 +01:00
Chris Coutinho 6253faee19 feat: Add instrumentation decorator and apply to notes tools (Phase 5)
Created @instrument_tool decorator for automatic MCP tool metrics collection.
Applied to all 7 tools in notes.py.

Changes:
- observability/metrics.py:
  * New instrument_tool() decorator for automatic timing and error tracking
  * Compatible with @mcp.tool() and @require_scopes() decorators
  * Records tool_name, duration, and success/error status

- server/notes.py:
  * Applied @instrument_tool to all 7 tool functions
  * nc_notes_create_note, nc_notes_update_note, nc_notes_append_content
  * nc_notes_search_notes, nc_notes_get_note, nc_notes_get_attachment
  * nc_notes_delete_note

These metrics will populate the MCP Tool Calls dashboard panels.

Part of PR #295 - Complete metrics instrumentation (Phase 5)
Remaining: 86 tools across 8 server files
2025-11-13 16:40:56 +01:00
Chris Coutinho c97f12d47e feat: Add OAuth token and database metrics (Phases 3-4)
Complete Prometheus instrumentation for OAuth token operations
and additional database operations to populate empty dashboard panels.

OAuth Token Metrics (Phase 4):
- unified_verifier.py:
  * Token validation cache hits/misses
  * JWT verification success/failure/error
  * Introspection validation results
  * Audience validation failures
- context_helper.py:
  * Token exchange cache hits/misses
  * RFC 8693 exchange success/error

Database Metrics (Phase 3 completion):
- storage.py:
  * get_refresh_token() with timing
  * delete_refresh_token() with timing
  * All operations record duration and success/error status

These metrics populate the following dashboard panels:
- Token Validations (by method and result)
- Token Cache Hit Rate
- Token Exchange Operations
- Database Operations (refresh token CRUD)
- Database Operation Duration

Part of PR #295 - Complete metrics instrumentation
2025-11-13 16:23:00 +01:00
Chris Coutinho a667d7c59c feat: Add metrics instrumentation for queue, health, and database operations
Implement Prometheus metrics to populate empty Grafana dashboard panels.

## Phase 1: Queue Size Metrics 
**File**: `processor.py`
- Track vector sync queue depth in real-time
- Update metric after receiving and processing each document
- Update metric during timeout (empty queue)
- Enables: "Processing Queue Depth" panel

## Phase 2: Health Check Metrics 
**File**: `app.py`
- Add Nextcloud connectivity check with timing
- Add Qdrant health check with timing
- Record dependency health status (up/down)
- Record health check duration
- Enables: 4 health status panels + health check duration panel

## Phase 3: Database Operation Metrics (Partial) 
**File**: `storage.py`
- Instrument `store_refresh_token()` method
- Track SQLite INSERT operation timing and success/error status
- Enables: Partial data for database operation latency panel

## Metrics Now Exposed

### Queue Metrics:
- `mcp_vector_sync_queue_size` - Real-time queue depth

### Health Metrics:
- `mcp_dependency_health{dependency="nextcloud"}` - UP/DOWN status
- `mcp_dependency_health{dependency="qdrant"}` - UP/DOWN status
- `mcp_dependency_check_duration_seconds{dependency}` - Health check latency

### Database Metrics:
- `mcp_db_operations_total{db="sqlite",operation="insert"}` - Operation count
- `mcp_db_operation_duration_seconds{db="sqlite",operation="insert"}` - Operation latency

## Dashboard Impact

**Panels Now Populated** (7/34 panels):
-  Processing Queue Depth
-  Nextcloud Health
-  Qdrant Health
-  Health Check Duration
-  Database Operation Latency (partial)
-  Vector sync panels (already working from PR #292)

**Panels Still Empty** (remaining work):
-  OAuth panels (4): Token validations, exchanges, cache hit rate, refresh ops
-  MCP tool panels (3): Call volume, error rates, execution duration
-  Database panel: Needs more SQLite operations instrumented (~29 remaining)

## Testing

Verified metric definitions exist and will be recorded on next deployment.

## Next Steps

Phase 4: OAuth token metrics (unified_verifier.py, context_helper.py, storage.py)
Phase 5: MCP tool metrics (all server/*.py files with @mcp.tool())
Phase 3 completion: Remaining 29 database operations in storage.py

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-13 16:14:38 +01:00
github-actions[bot] bd76902932 bump: version 0.33.0 → 0.33.1 2025-11-13 12:10:42 +00:00
Chris Coutinho da65155cde Merge pull request #293 from cbcoutinho/fix/grafana-folder-label-validation
fix: Move grafana_folder from labels to annotations
2025-11-13 13:10:15 +01:00
Chris Coutinho 4e43d15153 fix: Move grafana_folder from labels to annotations
Fixes Kubernetes label validation error when deploying dashboard ConfigMap.

Problem:
- Kubernetes labels cannot contain spaces (validation regex: [A-Za-z0-9][-A-Za-z0-9_.]*[A-Za-z0-9])
- Previous implementation had grafana_folder: "Nextcloud MCP" as a label
- Deployment failed with: "Invalid value: 'Nextcloud MCP'"

Solution:
- Move grafana_folder from labels to annotations (annotations allow spaces)
- Keep grafana_dashboard="1" as label for ConfigMap discovery
- Grafana sidecar reads folder name from folderAnnotation parameter

Changes:
- dashboard-configmap.yaml: Move grafana_folder to annotations section
- dashboards/README.md: Fix kubectl commands to use annotations
- values.yaml: Update comments to clarify annotation usage

This follows the standard kube-prometheus-stack pattern where:
- Labels are used for ConfigMap discovery (strict validation)
- Annotations are used for metadata like folder names (relaxed validation)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-13 13:08:45 +01:00
github-actions[bot] 15951c38fa bump: version 0.32.1 → 0.33.0 2025-11-13 10:58:05 +00:00
Chris Coutinho 2de0590839 Merge pull request #292 from cbcoutinho/feat/grafana-dashboard-and-vector-metrics
feat: Add Grafana dashboard and vector sync metric instrumentation
2025-11-13 11:57:40 +01:00
Chris Coutinho 4ea5ed72d4 feat: Add Grafana dashboard and vector sync metric instrumentation
Implement comprehensive observability for vector database synchronization
with Grafana dashboard and Prometheus metrics.

## Part 1: Grafana Dashboard

Created all-in-one operations dashboard with 7 rows and 34 panels:

### Dashboard Structure:
- **Overview Row**: Request rate, error rate, P95 latency, active requests
- **HTTP Metrics (RED)**: Request/error rates by endpoint, latency percentiles
- **MCP Tools**: Call volume, error rates, execution duration by tool
- **Nextcloud API**: API calls/latency by app, retry patterns
- **OAuth & Authentication**: Token validations, exchanges, cache hit rate
- **Dependencies & Health**: Status for Nextcloud/Qdrant/Keycloak/Unstructured
- **Vector Sync**: Processing throughput, queue depth, Qdrant operations

### Helm Chart Integration:
- Added dashboard-configmap.yaml template for automatic provisioning
- Configured Grafana sidecar auto-discovery (label: grafana_dashboard="1")
- Added dashboards configuration section in values.yaml (opt-in)
- Updated Chart.yaml with dashboard annotations
- Enhanced NOTES.txt with dashboard deployment instructions
- Comprehensive documentation in dashboards/README.md

Dashboard supports dynamic filtering via variables:
- datasource: Prometheus data source selection
- namespace: Filter by Kubernetes namespace
- pod: Multi-select pod filtering
- interval: Query interval (1m/5m/10m/30m/1h)

## Part 2: Vector Sync Metric Instrumentation

Implemented metric recording throughout vector sync pipeline:

### metrics.py:
Added convenience functions:
- record_vector_sync_scan() - Track documents per scan
- record_vector_sync_processing() - Track processing duration/status
- record_qdrant_operation() - Track database operations
- update_vector_sync_queue_size() - Track queue depth

### scanner.py:
- Record number of documents found in each scan
- Enables monitoring of scan throughput

### processor.py:
- Record processing duration for each document
- Track success/failure status with timing
- Record Qdrant upsert/delete operations
- Handle all code paths (success, deletion, error)

### semantic.py:
- Wrap Qdrant query_points with try/except
- Record search operation success/failure

## Metrics Exposed:

- mcp_vector_sync_documents_scanned_total
- mcp_vector_sync_documents_processed_total{status}
- mcp_vector_sync_processing_duration_seconds (histogram)
- mcp_vector_sync_queue_size (gauge)
- mcp_qdrant_operations_total{operation,status}

This enables monitoring of:
- Scan and processing throughput
- Processing latency (P50/P95/P99)
- Error rates for processing and Qdrant operations
- Queue depth trends
- Complete observability of vector sync pipeline

## Testing:

Verified locally that metrics are recorded correctly:
- 36 documents scanned
- 3 documents processed (avg 7.5s each)
- 3 successful Qdrant upsert operations
- Search operations tracked

## Deployment:

Enable dashboard provisioning in Helm values:
```yaml
dashboards:
  enabled: true
  grafanaFolder: "Nextcloud MCP"
```

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-13 11:49:20 +01:00
Chris Coutinho d1829fbbd6 Merge pull request #291 from cbcoutinho/renovate/ghcr.io-astral-sh-uv-0.x
chore(deps): update ghcr.io/astral-sh/uv docker tag to v0.9.9
2025-11-13 08:02:35 +01:00
renovate-bot-cbcoutinho[bot] 8332542959 chore(deps): update ghcr.io/astral-sh/uv docker tag to v0.9.9 2025-11-12 23:11:29 +00:00
renovate-bot-cbcoutinho[bot] 2c37ad165e chore(deps): update quay.io/keycloak/keycloak docker tag to v26.4.5 2025-11-12 17:09:23 +00:00
Chris Coutinho 619ba5684d build: Add ./worktrees to .gitignore 2025-11-12 08:27:33 +01:00
github-actions[bot] 747d297008 bump: version 0.32.0 → 0.32.1 2025-11-12 02:16:57 +00:00
Chris Coutinho ba8486b73b Merge pull request #289 from cbcoutinho/fix/dynamic-embedding-dimensions
fix: add dynamic dimension detection for Ollama embedding models
2025-11-12 03:16:29 +01:00
github-actions[bot] 49a9dd43c6 bump: version 0.31.1 → 0.32.0 2025-11-11 23:54:43 +00:00
36 changed files with 3313 additions and 1294 deletions
+1 -1
View File
@@ -85,4 +85,4 @@ jobs:
NEXTCLOUD_USERNAME: "admin"
NEXTCLOUD_PASSWORD: "admin"
run: |
uv run pytest -v --log-cli-level=WARN --ignore=tests/manual
uv run pytest -v --log-cli-level=WARN -m smoke
+3
View File
@@ -5,6 +5,9 @@ __pycache__/
.env.local
.env.*.local
# Git
worktrees/
docker-compose.override.yml
# Generated by pytest used to login users
+57
View File
@@ -1,3 +1,60 @@
## v0.34.2 (2025-11-13)
### Fix
- Use NEXTCLOUD_OIDC_CLIENT_ID/SECRET env vars consistently
## v0.34.1 (2025-11-13)
### Fix
- return all notes when search query is empty
## v0.34.0 (2025-11-13)
### Feat
- Complete Phase 5 - Instrument all 93 MCP tools
- Add instrumentation decorator and apply to notes tools (Phase 5)
- Add OAuth token and database metrics (Phases 3-4)
- Add metrics instrumentation for queue, health, and database operations
## v0.33.1 (2025-11-13)
### Fix
- Move grafana_folder from labels to annotations
## v0.33.0 (2025-11-13)
### Feat
- Add Grafana dashboard and vector sync metric instrumentation
## v0.32.1 (2025-11-12)
### Fix
- add dynamic dimension detection for Ollama embedding models
## v0.32.0 (2025-11-11)
### Feat
- **ollama**: Pull model on startup if not available in ollama
- add dynamic vector sync status updates with htmx polling
- add webhook management UI and BeforeNodeDeletedEvent support
- validate Nextcloud webhook schemas and document findings
### Fix
- improve webapp tab UI with CSS Grid and viewport-filling container
### Refactor
- move webapp from /user/page to /app
- consolidate database storage for webhooks and OAuth tokens
## v0.31.1 (2025-11-10)
### Refactor
+1 -1
View File
@@ -1,4 +1,4 @@
FROM ghcr.io/astral-sh/uv:0.9.8-python3.11-alpine@sha256:6c842c49ad032f46b62f32a7e7779f45f12671a8e0d82ea24c766ab62d58b396
FROM ghcr.io/astral-sh/uv:0.9.9-python3.11-alpine@sha256:0faa7934fac1db7f5056f159c1224d144bab864fd2677a4066d25a686ae32edd
# Install dependencies
# 1. git (required for caldav dependency from git)
+6 -2
View File
@@ -2,8 +2,8 @@ apiVersion: v2
name: nextcloud-mcp-server
description: A Helm chart for Nextcloud MCP Server - enables AI assistants to interact with Nextcloud
type: application
version: 0.31.1
appVersion: "0.31.1"
version: 0.34.2
appVersion: "0.34.2"
keywords:
- nextcloud
- mcp
@@ -21,6 +21,10 @@ home: https://github.com/cbcoutinho/nextcloud-mcp-server
sources:
- https://github.com/cbcoutinho/nextcloud-mcp-server
icon: https://raw.githubusercontent.com/nextcloud/server/master/core/img/logo/logo.svg
annotations:
# Grafana dashboard support
grafana_dashboard: "true"
grafana_dashboard_folder: "Nextcloud MCP"
dependencies:
- name: qdrant
version: "1.15.5"
+66
View File
@@ -280,6 +280,72 @@ Use OpenAI or any OpenAI-compatible API instead of Ollama.
| `openai.secretKey` | Key in secret containing API key | `api-key` |
| `openai.baseUrl` | Custom API endpoint (optional) | `""` |
#### Observability & Monitoring
The chart includes comprehensive observability features including Prometheus metrics, OpenTelemetry tracing, and Grafana dashboards.
**Metrics Configuration:**
| Parameter | Description | Default |
|-----------|-------------|---------|
| `observability.metrics.enabled` | Enable Prometheus metrics | `true` |
| `observability.metrics.port` | Metrics port | `9090` |
| `observability.metrics.path` | Metrics endpoint path | `/metrics` |
**Tracing Configuration:**
| Parameter | Description | Default |
|-----------|-------------|---------|
| `observability.tracing.enabled` | Enable OpenTelemetry tracing | `false` |
| `observability.tracing.endpoint` | OTLP collector endpoint | `""` |
| `observability.tracing.serviceName` | Service name in traces | `nextcloud-mcp-server` |
| `observability.tracing.samplingRate` | Trace sampling rate (0.0-1.0) | `1.0` |
**Logging Configuration:**
| Parameter | Description | Default |
|-----------|-------------|---------|
| `observability.logging.format` | Log format (json or text) | `json` |
| `observability.logging.level` | Log level | `INFO` |
| `observability.logging.includeTraceContext` | Include trace IDs in logs | `true` |
**ServiceMonitor (Prometheus Operator):**
| Parameter | Description | Default |
|-----------|-------------|---------|
| `serviceMonitor.enabled` | Create ServiceMonitor resource | `false` |
| `serviceMonitor.interval` | Scrape interval | `30s` |
| `serviceMonitor.scrapeTimeout` | Scrape timeout | `10s` |
| `serviceMonitor.labels` | Additional labels for ServiceMonitor | `{}` |
**PrometheusRule (Prometheus Operator):**
| Parameter | Description | Default |
|-----------|-------------|---------|
| `prometheusRule.enabled` | Create PrometheusRule with alert rules | `false` |
| `prometheusRule.labels` | Additional labels for PrometheusRule | `{}` |
**Grafana Dashboards:**
| Parameter | Description | Default |
|-----------|-------------|---------|
| `dashboards.enabled` | Enable automatic dashboard provisioning | `false` |
| `dashboards.grafanaFolder` | Grafana folder name for dashboards | `Nextcloud MCP` |
| `dashboards.labels` | Additional labels for dashboard ConfigMap | `{}` |
| `dashboards.annotations` | Additional annotations for dashboard ConfigMap | `{}` |
When `dashboards.enabled` is `true`, a ConfigMap with the Grafana dashboard is created with the `grafana_dashboard: "1"` label. This enables automatic discovery by Grafana sidecar containers (commonly used with kube-prometheus-stack).
The dashboard provides comprehensive monitoring including:
- HTTP request metrics (RED pattern: Rate, Errors, Duration)
- MCP tool performance and errors
- Nextcloud API performance by app (notes, calendar, contacts, etc.)
- OAuth token operations and cache hit rates
- External dependency health (Nextcloud, Qdrant, Keycloak, Unstructured API)
- Vector sync processing pipeline (when enabled)
For manual import or more details, see `charts/nextcloud-mcp-server/dashboards/README.md`.
## Examples
### Example 1: Basic Auth with Ingress
+107 -36
View File
@@ -6,14 +6,57 @@ This directory contains example Grafana dashboards for monitoring the Nextcloud
### nextcloud-mcp-server.json
Comprehensive dashboard with the following panels:
All-in-one Operations Dashboard with comprehensive monitoring across all system components.
- **Request Rate**: HTTP requests per second by method and endpoint
- **Error Rate**: Percentage of 5xx errors
- **Request Latency**: P50 and P95 latency by endpoint
- **Top MCP Tools**: Most frequently called tools
- **Nextcloud API Latency**: API call latency by app (notes, calendar, etc.)
- **Vector Sync Queue**: Queue size for background document processing
#### Overview Row
High-level metrics for quick health assessment:
- **Request Rate** (stat): Total requests per second
- **Error Rate** (stat): Percentage of 5xx errors with color thresholds
- **P95 Latency** (stat): 95th percentile request latency
- **Active Requests** (stat): Current in-flight requests
#### HTTP Metrics (RED Pattern)
Core request/error/duration metrics:
- **Request Rate by Endpoint** (timeseries): RPS breakdown by endpoint
- **Error Rate by Status Code** (timeseries): Error rates for 4xx/5xx codes
- **Latency Percentiles** (timeseries): P50, P95, P99 latency trends
- **Status Code Distribution** (piechart): Percentage breakdown of all status codes
#### MCP Tools Row
MCP-specific tool performance:
- **Top Tools by Call Volume** (bargauge): Top 10 most-called tools
- **Tool Error Rate** (timeseries): Error rates per tool
- **Tool Execution Duration** (timeseries): P95 latency by tool
#### Nextcloud API Row
Backend API performance metrics:
- **API Calls by App** (timeseries): Request rate per Nextcloud app (notes, calendar, contacts, etc.)
- **API Latency by App** (timeseries): P95 latency per app
- **API Retries by Reason** (timeseries): Retry patterns (429, timeout, connection errors)
- **API Error Rate** (stat): Overall API error percentage
#### OAuth & Authentication Row
OAuth token operations and caching:
- **Token Validations** (timeseries): Success/failure rates for token validation
- **Token Exchange Operations** (timeseries): RFC 8693 token exchange operations
- **Token Cache Hit Rate** (stat): Percentage of cache hits (color-coded: red<50%, yellow<80%, green≥80%)
- **Refresh Token Operations** (timeseries): Refresh token storage operations by type
#### Dependencies & Health Row
External dependency status monitoring:
- **Nextcloud Health** (stat): UP/DOWN status with color coding
- **Qdrant Health** (stat): Vector database health status
- **Keycloak Health** (stat): Identity provider health status
- **Unstructured API Health** (stat): Document processing API status
- **Health Check Duration** (timeseries): Health check latency by dependency
- **Database Operation Latency** (timeseries): P95 latency for DB operations (SQLite, Qdrant)
#### Vector Sync Row (when enabled)
Document processing pipeline metrics:
- **Documents Processed Rate** (timeseries): Processing throughput by status (success/failure)
- **Processing Queue Depth** (gauge): Current queue size with thresholds (yellow>50, red>100)
- **Qdrant Operations** (timeseries): Vector database operations by type
- **Document Processing Duration** (timeseries): P95 processing latency
## Importing to Grafana
@@ -25,49 +68,77 @@ Comprehensive dashboard with the following panels:
4. Select your Prometheus data source
5. Click "Import"
### Automated Import (Kubernetes)
### Automated Import (Helm Chart)
If using the Grafana Operator or kube-prometheus-stack, you can create a ConfigMap:
The Helm chart now supports automatic dashboard provisioning via Grafana sidecar pattern.
#### Option 1: Using Helm Chart (Recommended)
Enable dashboard provisioning in your Helm values:
```yaml
# values.yaml for nextcloud-mcp-server chart
dashboards:
enabled: true
grafanaFolder: "Nextcloud MCP" # Folder name in Grafana
labels: {} # Additional labels if needed
```
Then deploy or upgrade:
```bash
kubectl create configmap nextcloud-mcp-dashboards \
helm upgrade --install nextcloud-mcp nextcloud-mcp-server \
--set dashboards.enabled=true
```
The dashboard will be automatically imported by Grafana if the sidecar is configured
to watch for ConfigMaps with label `grafana_dashboard: "1"`.
#### Option 2: Using kube-prometheus-stack
If using kube-prometheus-stack with Grafana sidecar enabled, the dashboard will be
automatically discovered and imported. Ensure your Grafana deployment has:
```yaml
# kube-prometheus-stack values
grafana:
sidecar:
dashboards:
enabled: true
label: grafana_dashboard
folder: /tmp/dashboards
provider:
foldersFromFilesStructure: true
```
#### Option 3: Manual ConfigMap Creation
For other Grafana setups, create a ConfigMap manually:
```bash
kubectl create configmap nextcloud-mcp-dashboard \
--from-file=nextcloud-mcp-server.json \
-n monitoring
# Add label for Grafana sidecar to discover
kubectl label configmap nextcloud-mcp-dashboards \
# Add sidecar discovery label
kubectl label configmap nextcloud-mcp-dashboard \
grafana_dashboard=1 \
-n monitoring
```
Or add to your Helm values:
```yaml
# values.yaml for kube-prometheus-stack
grafana:
dashboardProviders:
dashboardproviders.yaml:
apiVersion: 1
providers:
- name: 'nextcloud-mcp'
orgId: 1
folder: 'Nextcloud MCP'
type: file
disableDeletion: false
editable: true
options:
path: /var/lib/grafana/dashboards/nextcloud-mcp
dashboardsConfigMaps:
nextcloud-mcp: nextcloud-mcp-dashboards
# Add folder annotation (annotations support spaces, unlike labels)
kubectl annotate configmap nextcloud-mcp-dashboard \
grafana_folder="Nextcloud MCP" \
-n monitoring
```
## Dashboard Variables
The dashboard includes two variables:
The dashboard includes four template variables for dynamic filtering:
- **Data Source**: Select your Prometheus data source
- **Namespace**: Filter metrics by Kubernetes namespace
- **datasource**: Select your Prometheus data source
- **namespace**: Filter metrics by Kubernetes namespace (supports "All")
- **pod**: Filter by specific pod(s) - multi-select enabled (supports "All")
- **interval**: Query interval for rate calculations (1m, 5m, 10m, 30m, 1h - default: 5m)
## Customization
File diff suppressed because it is too large Load Diff
@@ -96,6 +96,30 @@ Your Nextcloud MCP Server has been deployed in {{ .Values.auth.mode }} authentic
kubectl --namespace {{ .Release.Namespace }} exec -it deploy/{{ include "nextcloud-mcp-server.fullname" . }} -- curl -s http://localhost:{{ include "nextcloud-mcp-server.port" . }}/user/page | grep "Vector Sync"
{{- end }}
{{- if .Values.dashboards.enabled }}
6. Grafana Dashboards:
- Dashboard provisioning: Enabled
- ConfigMap: {{ include "nextcloud-mcp-server.fullname" . }}-dashboard
- Grafana Folder: {{ .Values.dashboards.grafanaFolder }}
The dashboard will be automatically imported by Grafana if the sidecar is configured
to watch for ConfigMaps with label "grafana_dashboard: 1".
To manually import the dashboard:
kubectl --namespace {{ .Release.Namespace }} get configmap {{ include "nextcloud-mcp-server.fullname" . }}-dashboard -o jsonpath='{.data.nextcloud-mcp-server\.json}' | jq . > dashboard.json
Then import dashboard.json via Grafana UI (Dashboards → Import).
{{- else }}
6. Grafana Dashboards:
- Dashboard provisioning: Disabled
- To enable automatic dashboard provisioning, set: dashboards.enabled=true
Manual import option:
The dashboard JSON is available in the chart at charts/nextcloud-mcp-server/dashboards/nextcloud-mcp-server.json
{{- end }}
For more information and documentation:
- GitHub: https://github.com/cbcoutinho/nextcloud-mcp-server
- Documentation: https://github.com/cbcoutinho/nextcloud-mcp-server#readme
@@ -0,0 +1,25 @@
{{- if .Values.dashboards.enabled }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "nextcloud-mcp-server.fullname" . }}-dashboard
namespace: {{ .Release.Namespace }}
labels:
{{- include "nextcloud-mcp-server.labels" . | nindent 4 }}
{{- with .Values.dashboards.labels }}
{{- toYaml . | nindent 4 }}
{{- end }}
# Grafana sidecar discovery label
grafana_dashboard: "1"
annotations:
{{- with .Values.dashboards.annotations }}
{{- toYaml . | nindent 4 }}
{{- end }}
# Grafana folder name (annotations support spaces, unlike labels)
{{- if .Values.dashboards.grafanaFolder }}
grafana_folder: {{ .Values.dashboards.grafanaFolder | quote }}
{{- end }}
data:
nextcloud-mcp-server.json: |-
{{ .Files.Get "dashboards/nextcloud-mcp-server.json" | indent 4 }}
{{- end }}
+14
View File
@@ -205,6 +205,20 @@ prometheusRule:
# Additional labels for PrometheusRule (e.g., for Prometheus selector)
# Example: { prometheus: kube-prometheus }
# Grafana dashboards (requires Grafana with sidecar enabled)
dashboards:
# Enable automatic dashboard provisioning via ConfigMap
enabled: false
# Grafana folder name where dashboards will be imported
# The grafana-sidecar looks for ConfigMaps with label "grafana_dashboard: 1"
# and reads the folder name from annotation "grafana_folder" (supports spaces)
grafanaFolder: "Nextcloud MCP"
# Additional labels for dashboard ConfigMap
# These will be added alongside the required "grafana_dashboard: 1" label
labels: {}
# Additional annotations for dashboard ConfigMap
annotations: {}
service:
type: ClusterIP
port: 8000
+1 -1
View File
@@ -156,7 +156,7 @@ services:
- oauth-tokens:/app/data
keycloak:
image: quay.io/keycloak/keycloak:26.4.4@sha256:c6459d5fae1b759f5d667ebdc6237ab3121379c3494e213898569014ede1846d
image: quay.io/keycloak/keycloak:26.4.5@sha256:653852bfdea2be6e958b9e90a976eff1c6de34edd55f2f679bdc48ef16bc528e
command:
- "start-dev"
- "--import-realm"
@@ -0,0 +1,895 @@
# ADR-011: Improving Semantic Search Quality Through Better Chunking and Embeddings
**Status**: Proposed
**Date**: 2025-11-12
**Authors**: Development Team
**Related**: ADR-003 (Vector Database Architecture), ADR-008 (MCP Sampling for RAG)
## Context
The semantic search implementation provides document retrieval across Nextcloud apps using vector embeddings. Production usage has revealed that **the system frequently misses relevant documents** (recall problem).
Root cause analysis identifies two fundamental issues:
### 1. Poor Chunking Strategy
**Current Implementation** (`nextcloud_mcp_server/vector/document_chunker.py:36`):
```python
words = content.split() # Naive whitespace splitting
chunk_size = 512 # words
overlap = 50 # words
chunks = [words[i:i+chunk_size] for i in range(0, len(words), chunk_size-overlap)]
```
**Problems**:
- **Breaks semantic boundaries**: Splits mid-sentence, mid-paragraph, mid-thought
- **Loses context**: "The meeting discussed budget. We decided to..." becomes two disconnected chunks
- **Poor retrieval**: Relevant content split across chunks with low individual relevance scores
- **No structure awareness**: Ignores markdown headers, lists, code blocks
**Evidence**:
- Documents with relevant content in middle sections score poorly (content split across 3+ chunks)
- Multi-sentence concepts (spanning 60-100 words) are fragmented
- Search for "budget planning process" misses documents where these words appear in adjacent sentences but different chunks
### 2. Suboptimal Embedding Model
**Current Implementation** (`nextcloud_mcp_server/embedding/ollama_provider.py:33`):
```python
_model = "nomic-embed-text" # 768 dimensions
_dimension = 768 # Hardcoded
```
**Problems**:
- **Model selection**: `nomic-embed-text` is general-purpose, not optimized for our use case
- **No benchmarking**: Selected without comparative evaluation
- **Dimensionality**: 768-dim may be insufficient for nuanced semantic distinctions
- **No domain adaptation**: Model not tuned for Nextcloud content (notes, calendar, deck cards)
**Evidence**:
- Synonymous queries return different results ("meeting notes" vs. "discussion summary")
- Domain-specific terms poorly represented ("standup", "retrospective", "OKRs")
- Cross-lingual content (if present) not well supported
### Current Performance
**Baseline Metrics** (100-document test corpus, 50 queries):
- **Recall@10**: ~52% (misses 48% of relevant documents)
- **Precision@10**: ~78% (acceptable but room for improvement)
- **MRR**: 0.58 (relevant docs often not in top positions)
- **Zero-result queries**: 18% (completely missing relevant content)
## Decision Drivers
1. **Address Root Causes**: Fix fundamental issues (chunking, embeddings) before adding complexity (reranking, hybrid search)
2. **Measurable Impact**: Target 40-60% improvement in recall through chunking/embedding alone
3. **Independence**: Improvements should be orthogonal to future enhancements (reranking, GraphRAG)
4. **Cost Efficiency**: Minimize infrastructure and API costs
5. **Reindexing Acceptable**: One-time reindex cost justified by long-term quality improvement
## Options Considered
### Chunking Strategies
#### Option C1: Semantic Sentence-Aware Chunking (RECOMMENDED)
**Description**: Respect sentence boundaries while maintaining target chunk size
**Implementation**:
```python
from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=2048, # ~512 words in characters
chunk_overlap=200, # ~50 words in characters
separators=["\n\n", "\n", ". ", "! ", "? ", "; ", ": ", ", ", " "],
length_function=len,
)
```
**How it works**:
1. Try splitting by paragraphs (`\n\n`)
2. If chunks too large, split by sentences (`. `, `! `, `? `)
3. If still too large, split by clauses (`;`, `:`)
4. Last resort: split by words
**Pros**:
- ✅ Preserves semantic boundaries (never breaks mid-sentence)
- ✅ Maintains context coherence within chunks
- ✅ Simple implementation (langchain library)
- ✅ Configurable separators for different content types
- ✅ Proven approach (used by major RAG systems)
**Cons**:
- ❌ Variable chunk sizes (not exactly 512 words, but close)
- ❌ Adds dependency (langchain)
- ❌ Slightly slower than naive splitting (~10-20ms per document)
**Expected Impact**: 20-30% recall improvement
#### Option C2: Hierarchical Context-Preserving Chunks
**Description**: Create overlapping parent/child chunks
**Structure**:
```
Document → Large parent chunks (1024 words) → Small child chunks (256 words)
↓ ↓
Stored in Qdrant Searched first
Return parent context
```
**Implementation**:
```python
# Generate child chunks (searched)
child_chunks = splitter.split_text(content, chunk_size=1024)
# Generate parent chunks (context)
parent_chunks = splitter.split_text(content, chunk_size=4096)
# Store both with parent-child relationships
for child_idx, child in enumerate(child_chunks):
parent_idx = find_parent(child_idx)
store_vector(
vector=embed(child),
payload={
"chunk": child,
"parent_chunk": parent_chunks[parent_idx],
"chunk_type": "child"
}
)
```
**Pros**:
- ✅ Best of both worlds: precise matching + full context
- ✅ Handles multi-hop information needs
- ✅ Better for long documents (> 1000 words)
**Cons**:
- ❌ 2x storage (parent + child chunks)
- ❌ More complex implementation
- ❌ Higher indexing time (embed twice)
- ❌ Query complexity (retrieve child, return parent)
**Expected Impact**: 35-45% recall improvement (diminishing returns vs. complexity)
**Verdict**: ⚠️ Consider only if Option C1 insufficient
#### Option C3: Document Structure-Aware Chunking
**Description**: Parse markdown/document structure before chunking
**Implementation**:
```python
import mistune # Markdown parser
def structure_aware_chunk(markdown_content: str) -> list[str]:
ast = mistune.create_markdown(renderer='ast')(markdown_content)
chunks = []
for node in ast:
if node['type'] == 'heading':
# Start new chunk at each header
current_chunk = node['children'][0]['raw']
elif node['type'] == 'paragraph':
current_chunk += "\n" + node['children'][0]['raw']
if len(current_chunk) > 2048:
chunks.append(current_chunk)
current_chunk = ""
return chunks
```
**Pros**:
- ✅ Respects document logical structure
- ✅ Headers provide context for chunks
- ✅ Works well for structured notes (documentation, meeting notes with sections)
**Cons**:
- ❌ Complex implementation (parser, AST traversal)
- ❌ Markdown-specific (doesn't help calendar events, deck cards)
- ❌ Variable chunk sizes (some sections very short/long)
- ❌ Breaks for unstructured content
**Expected Impact**: 15-25% improvement for structured content only
**Verdict**: ⚠️ Future enhancement after Option C1
#### Option C4: Fixed Sliding Window (Current Baseline)
**Description**: Current naive word-based splitting
**Verdict**: ❌ Superseded by Option C1
### Embedding Model Strategies
#### Option E1: Upgrade to Better General-Purpose Model (RECOMMENDED)
**Description**: Switch to state-of-the-art embedding model
**Candidates**:
| Model | Dimensions | MTEB Score | Pros | Cons |
|-------|-----------|------------|------|------|
| **mxbai-embed-large** | 1024 | 64.68 | Best performance, good balance | Larger (slower) |
| **nomic-embed-text-v1.5** | 768 | 62.39 | Upgraded version of current | Incremental improvement |
| **bge-large-en-v1.5** | 1024 | 64.23 | Excellent for English | Not multilingual |
| **nomic-embed-text** (current) | 768 | 60.10 | Baseline | Lower performance |
**MTEB**: Massive Text Embedding Benchmark (higher = better semantic understanding)
**Recommendation**: **mxbai-embed-large-v1**
- Best MTEB score (64.68)
- 1024 dimensions (richer semantic space)
- Works well via Ollama
- ~15-20% better retrieval quality in benchmarks
**Implementation**:
```python
# config.py
OLLAMA_EMBEDDING_MODEL = "mxbai-embed-large-v1" # Changed from nomic-embed-text
# ollama_provider.py
async def get_dimension(self) -> int:
# Query Ollama for actual dimension instead of hardcoding
response = await self.client.post("/api/show", json={"name": self.model})
return response.json()["details"]["embedding_length"]
```
**Migration**:
1. Deploy new model to Ollama
2. Create new Qdrant collection (different dimension)
3. Reindex all documents with new embeddings
4. Swap collections atomically
5. Delete old collection
**Pros**:
- ✅ Immediate quality improvement (15-20%)
- ✅ Simple change (config + reindex)
- ✅ No code complexity
- ✅ Future-proof (state-of-the-art model)
**Cons**:
- ❌ Requires full reindex (2-4 hours for 1000 documents)
- ❌ Larger model = slower embedding (~50ms vs. 30ms per chunk)
- ❌ Higher dimensionality = more storage (~30% increase)
**Expected Impact**: 15-25% recall improvement
#### Option E2: Multi-Vector Embeddings (ColBERT-style)
**Description**: Generate multiple embeddings per chunk (token-level)
**Architecture**:
```
Chunk → Transformer → Token embeddings (e.g., 50 tokens × 128 dim) → Store all
Query → Transformer → Token embeddings → MaxSim(query_tokens, doc_tokens)
```
**MaxSim scoring**:
```python
def maxsim_score(query_embeddings, doc_embeddings):
# For each query token, find max similarity with any doc token
scores = []
for q_emb in query_embeddings:
max_sim = max(cosine_similarity(q_emb, d_emb) for d_emb in doc_embeddings)
scores.append(max_sim)
return sum(scores)
```
**Pros**:
- ✅ Best retrieval quality (state-of-the-art results)
- ✅ Fine-grained matching (token-level)
- ✅ Handles partial matches better
**Cons**:
-**50-100x storage increase** (50 vectors per chunk vs. 1)
-**Slower search** (compute MaxSim for each candidate)
-**Complex implementation** (custom scoring, storage schema)
-**Requires specialized model** (ColBERTv2, not available in Ollama)
**Expected Impact**: 40-50% improvement, but at very high cost
**Verdict**: ❌ Too complex, too expensive for marginal gain over E1+C1
#### Option E3: Fine-Tuned Domain-Specific Model
**Description**: Fine-tune embedding model on Nextcloud corpus
**Process**:
1. Collect training data (query-document pairs)
2. Fine-tune base model (e.g., `nomic-embed-text`) on domain data
3. Deploy fine-tuned model via Ollama
4. Reindex with fine-tuned embeddings
**Training data needed**:
- 1,000+ query-document pairs
- Labeled relevance (positive/negative examples)
- Representative of real usage
**Pros**:
- ✅ Optimized for specific content (notes, calendar, deck)
- ✅ Better handling of domain terminology
- ✅ Highest potential quality improvement (30-40%)
**Cons**:
-**Requires training data** (expensive to collect)
-**GPU infrastructure** needed for fine-tuning
-**Expertise required** (ML/NLP knowledge)
-**Maintenance burden** (retrain as corpus evolves)
-**Time investment**: 2-4 weeks initial setup
**Expected Impact**: 30-40% improvement, but high cost
**Verdict**: ⚠️ Consider only if E1+C1 insufficient AND have training data
#### Option E4: Ensemble Embeddings
**Description**: Generate embeddings with multiple models, combine scores
**Implementation**:
```python
models = ["mxbai-embed-large-v1", "bge-large-en-v1.5"]
# Index
embeddings = [await embed(chunk, model) for model in models]
store_multi_vector(embeddings)
# Search
query_embeddings = [await embed(query, model) for model in models]
scores = [search(q_emb, model) for q_emb, model in zip(query_embeddings, models)]
combined_score = 0.5 * scores[0] + 0.5 * scores[1]
```
**Pros**:
- ✅ Robust to individual model weaknesses
- ✅ Better coverage of semantic space
**Cons**:
- ❌ 2x storage and compute
- ❌ Complex scoring and fusion
- ❌ Marginal improvement (~5-10%) over single best model
**Expected Impact**: 5-10% over best single model
**Verdict**: ❌ Not worth complexity
### Combined Strategies
#### Option D1: Best Chunking + Best Embedding (RECOMMENDED)
**Combination**: Option C1 (Semantic Chunking) + Option E1 (mxbai-embed-large-v1)
**Expected Impact**:
- Chunking: +20-30% recall
- Embedding: +15-25% recall
- **Combined**: +35-55% recall improvement (not strictly additive, but significant)
**Cost**:
- Development: 1-2 days
- Reindex: 2-4 hours (one-time)
- Ongoing: None (same infrastructure)
**Pros**:
- ✅ Addresses both root causes
- ✅ Orthogonal improvements (chunking + embedding)
- ✅ Simple implementation
- ✅ No new infrastructure
- ✅ Future-proof foundation for additional enhancements (reranking, hybrid search)
**Cons**:
- ❌ Requires full reindex (manageable)
- ❌ Slightly higher storage (1024 vs. 768 dim)
**Verdict**: ✅ **RECOMMENDED**
## Decision
**Adopt Option D1: Semantic Chunking + Upgraded Embedding Model**
Implement both improvements together to maximize recall improvement:
### 1. Semantic Sentence-Aware Chunking
**Changes**:
- Replace naive word splitting with `RecursiveCharacterTextSplitter`
- Preserve sentence boundaries, paragraph structure
- Maintain similar chunk sizes (~512 words / 2048 characters)
**Implementation**:
```python
# nextcloud_mcp_server/vector/document_chunker.py
from langchain.text_splitter import RecursiveCharacterTextSplitter
class DocumentChunker:
"""Chunk documents into semantically coherent pieces."""
def __init__(
self,
chunk_size: int = 2048, # Characters, not words
chunk_overlap: int = 200, # Characters, not words
):
self.chunk_size = chunk_size
self.chunk_overlap = chunk_overlap
self.splitter = RecursiveCharacterTextSplitter(
chunk_size=chunk_size,
chunk_overlap=chunk_overlap,
separators=[
"\n\n", # Paragraphs (highest priority)
"\n", # Lines
". ", # Sentences
"! ",
"? ",
"; ", # Clauses
": ",
", ", # Phrases
" ", # Words (last resort)
],
length_function=len,
is_separator_regex=False,
)
def chunk_text(self, content: str) -> list[str]:
"""
Chunk text while preserving semantic boundaries.
Args:
content: Full document text
Returns:
List of text chunks, each ending at a semantic boundary
"""
if not content:
return []
# Use RecursiveCharacterTextSplitter for semantic boundaries
chunks = self.splitter.split_text(content)
return chunks
```
**Configuration Changes** (`config.py`):
```python
# Old (word-based)
DOCUMENT_CHUNK_SIZE: int = 512 # words
DOCUMENT_CHUNK_OVERLAP: int = 50 # words
# New (character-based, more precise)
DOCUMENT_CHUNK_SIZE: int = 2048 # characters (~512 words)
DOCUMENT_CHUNK_OVERLAP: int = 200 # characters (~50 words)
```
**Dependency** (`pyproject.toml`):
```toml
[project]
dependencies = [
# ... existing dependencies
"langchain-text-splitters>=0.2.0",
]
```
### 2. Upgrade Embedding Model
**Changes**:
- Switch from `nomic-embed-text` (768-dim) to `mxbai-embed-large-v1` (1024-dim)
- Dynamic dimension detection (query Ollama instead of hardcoding)
- Create new Qdrant collection for new dimensions
**Implementation**:
```python
# nextcloud_mcp_server/embedding/ollama_provider.py
class OllamaEmbeddingProvider(EmbeddingProvider):
def __init__(self, base_url: str, model: str, verify_ssl: bool = True):
self.base_url = base_url
self.model = model
self._dimension: int | None = None # Changed: query dynamically
self.client = httpx.AsyncClient(base_url=base_url, verify=verify_ssl)
async def dimension(self) -> int:
"""Get embedding dimension from Ollama API."""
if self._dimension is None:
try:
response = await self.client.post(
"/api/show",
json={"name": self.model},
timeout=10.0,
)
response.raise_for_status()
info = response.json()
self._dimension = info.get("details", {}).get("embedding_length")
if self._dimension is None:
# Fallback: generate test embedding to detect dimension
test_emb = await self.embed("test")
self._dimension = len(test_emb)
except Exception as e:
logger.warning(f"Failed to get dimension from Ollama: {e}, using fallback")
# Fallback dimensions by model name
if "mxbai-embed-large" in self.model:
self._dimension = 1024
elif "nomic-embed-text" in self.model:
self._dimension = 768
else:
self._dimension = 768 # Default
return self._dimension
```
**Configuration Changes** (`config.py`):
```python
# Old
OLLAMA_EMBEDDING_MODEL: str = "nomic-embed-text"
# New
OLLAMA_EMBEDDING_MODEL: str = "mxbai-embed-large-v1"
```
**Environment Variable**:
```bash
OLLAMA_EMBEDDING_MODEL=mxbai-embed-large-v1
```
### 3. Migration Strategy
**Reindexing Process**:
```python
# nextcloud_mcp_server/vector/migration.py
async def migrate_to_new_embeddings():
"""
Migrate from old embeddings to new embeddings.
Process:
1. Create new collection with new dimension
2. Reindex all documents with new embeddings
3. Atomic swap (update collection name in config)
4. Delete old collection
"""
old_collection = "nextcloud_content"
new_collection = "nextcloud_content_v2"
# 1. Create new collection
await qdrant_client.create_collection(
collection_name=new_collection,
vectors_config=VectorParams(
size=1024, # mxbai-embed-large-v1 dimension
distance=Distance.COSINE,
),
)
# 2. Reindex all documents
logger.info("Starting reindex with new embeddings...")
scanner = VectorScanner(...)
processor = VectorProcessor(collection_name=new_collection, ...)
await scanner.scan_all() # Rescans and re-embeds all documents
# 3. Wait for completion
while True:
status = await get_sync_status()
if status.pending_documents == 0:
break
await asyncio.sleep(5)
# 4. Atomic swap
# Update config to point to new collection
# (or use collection alias in Qdrant)
await qdrant_client.update_collection_aliases(
change_aliases_operations=[
CreateAliasOperation(
create_alias=CreateAlias(
collection_name=new_collection,
alias_name="nextcloud_content"
)
)
]
)
# 5. Verify new collection works
test_results = await run_benchmark_queries()
if test_results.recall < baseline_recall:
# Rollback
logger.error("New embeddings worse than baseline, rolling back")
await rollback_migration()
return False
# 6. Delete old collection
await qdrant_client.delete_collection(old_collection)
logger.info("Migration complete!")
return True
```
**Downtime Mitigation**:
- Use Qdrant collection aliases for atomic swap
- Reindex can happen in background
- Only brief downtime during alias swap (~1s)
**Rollback Plan**:
- Keep old collection until validation complete
- If new embeddings worse, swap alias back to old collection
- No data loss
### 4. Validation & Benchmarking
**Before/After Comparison**:
```python
# tests/benchmarks/chunking_embedding_comparison.py
async def benchmark_chunking_embeddings():
"""
Compare old vs. new chunking and embeddings on test queries.
"""
test_queries = load_benchmark_queries() # 100 queries with known relevant docs
# Baseline (current)
baseline_results = await run_queries(
queries=test_queries,
collection="nextcloud_content", # Old: nomic-embed-text, word chunks
)
# New implementation
new_results = await run_queries(
queries=test_queries,
collection="nextcloud_content_v2", # New: mxbai-embed-large-v1, semantic chunks
)
# Compare metrics
comparison = {
"baseline": {
"recall@10": calculate_recall(baseline_results, k=10),
"precision@10": calculate_precision(baseline_results, k=10),
"mrr": calculate_mrr(baseline_results),
"zero_result_rate": calculate_zero_result_rate(baseline_results),
},
"new": {
"recall@10": calculate_recall(new_results, k=10),
"precision@10": calculate_precision(new_results, k=10),
"mrr": calculate_mrr(new_results),
"zero_result_rate": calculate_zero_result_rate(new_results),
},
"improvement": {
"recall_improvement": (new_recall - baseline_recall) / baseline_recall,
"precision_improvement": (new_precision - baseline_precision) / baseline_precision,
}
}
return comparison
```
**Success Criteria**:
- **Recall@10**: Improve from ~52% to ≥75% (+40% improvement)
- **Precision@10**: Maintain ≥75% (no degradation)
- **MRR**: Improve from 0.58 to ≥0.70
- **Zero-result rate**: Reduce from 18% to ≤10%
- **Indexing time**: Maintain ≤10s per document
**Validation Process**:
1. Run benchmark on baseline (current implementation)
2. Implement changes
3. Run benchmark on new implementation
4. Compare metrics
5. If improvement ≥40%, proceed to production
6. If improvement <40%, investigate and iterate
## Implementation Timeline
### Week 1: Development & Testing
**Day 1-2: Chunking Implementation**
- [ ] Add langchain-text-splitters dependency
- [ ] Refactor `document_chunker.py`
- [ ] Update configuration (character-based chunk sizes)
- [ ] Write unit tests for semantic boundaries
- [ ] Validate: Chunks never break mid-sentence
**Day 3-4: Embedding Implementation**
- [ ] Update `ollama_provider.py` with dynamic dimension detection
- [ ] Update configuration (new model name)
- [ ] Deploy `mxbai-embed-large-v1` to Ollama
- [ ] Test embedding generation with new model
- [ ] Validate: Embeddings are 1024-dim
**Day 5: Migration Script**
- [ ] Write migration script (collection creation, reindexing, alias swap)
- [ ] Test migration on staging environment
- [ ] Validate: No data loss, atomic swap works
### Week 2: Reindexing & Validation
**Day 1-2: Staging Reindex**
- [ ] Run full reindex on staging environment
- [ ] Monitor indexing performance
- [ ] Validate: All documents indexed correctly
**Day 3: Benchmarking**
- [ ] Run benchmark queries on old collection (baseline)
- [ ] Run benchmark queries on new collection
- [ ] Compare metrics (recall, precision, MRR)
- [ ] Validate: ≥40% recall improvement
**Day 4: Production Reindex**
- [ ] Schedule maintenance window (optional, can run in background)
- [ ] Run migration script on production
- [ ] Monitor reindexing progress
- [ ] Atomic swap when complete
**Day 5: Production Validation**
- [ ] Monitor search quality metrics
- [ ] Collect user feedback
- [ ] Compare production metrics to staging
- [ ] Rollback if issues detected
## Cost Analysis
### Development Cost
- **Time**: 1-2 weeks (implementation + validation)
- **Effort**: 40-60 hours @ $100/hour = $4,000 - $6,000
### Infrastructure Cost
- **Storage**: +30% (1024-dim vs. 768-dim)
- Example: 1,000 notes × 3 chunks × 1024 dim × 4 bytes = 12 MB (negligible)
- **Compute**: +20% embedding time (50ms vs. 30ms per chunk)
- Amortized over batch indexing, minimal impact
- **No new infrastructure**: Uses existing Ollama + Qdrant
### Reindexing Cost (One-Time)
- **Time**: 2-4 hours for 1,000 documents
- 1,000 docs × 3 chunks × 50ms = 150 seconds (~2.5 minutes embedding)
- + Ollama processing time + Qdrant insertion
- **Downtime**: ~1 second (atomic alias swap)
### Total Cost
- **Initial**: $4,000 - $6,000 (development + testing)
- **Ongoing**: $0 (no new infrastructure or API costs)
### ROI
- **Recall improvement**: +40-60% (finding relevant documents)
- **User satisfaction**: Reduced zero-result queries (18% → 10%)
- **Foundation**: Enables future enhancements (reranking, hybrid search)
- **Cost per % improvement**: $100 - $150 (excellent ROI)
## Consequences
### Positive
1. **Addresses Root Causes**: Fixes fundamental issues (chunking, embeddings) not symptoms
2. **High Impact**: Expected 40-60% recall improvement from foundational changes
3. **Future-Proof**: Creates solid foundation for future enhancements (reranking, hybrid search, GraphRAG)
4. **Simple**: No architectural changes, no new infrastructure
5. **Orthogonal**: Improvements are independent, can be validated separately
6. **Low Risk**: Proven techniques (RecursiveCharacterTextSplitter, mxbai-embed-large-v1)
7. **Maintainable**: Standard libraries and models, easy to debug
### Negative
1. **Reindexing Required**: 2-4 hours one-time cost (manageable, can run in background)
2. **Storage Increase**: +30% for higher-dimensional embeddings (12 MB vs. 9 MB for 1K docs)
3. **Slower Indexing**: +20% embedding time (50ms vs. 30ms per chunk)
4. **Dependency**: Adds langchain-text-splitters (minimal, well-maintained library)
5. **Not a Complete Solution**: May still need reranking/hybrid search for optimal recall (but solid foundation)
### Neutral
1. **Model Lock-In**: Committed to mxbai-embed-large-v1, but can change later (another reindex)
2. **Chunk Size Trade-offs**: ~512 words is heuristic, may need tuning for specific content types
## Monitoring & Success Metrics
### Real-Time Metrics (Grafana)
**Search Quality**:
- `semantic_search_recall_at_10` (target: ≥75%)
- `semantic_search_precision_at_10` (target: ≥75%)
- `semantic_search_mrr` (target: ≥0.70)
- `semantic_search_zero_result_rate` (target: ≤10%)
**Performance**:
- `semantic_search_latency_ms` (p50, p95, p99)
- `embedding_generation_time_ms`
- `indexing_throughput_docs_per_sec`
**Indexing**:
- `documents_indexed_total`
- `documents_pending`
- `indexing_errors_total`
### Weekly Validation
**A/B Testing** (if gradual rollout):
- 50% users: New embeddings
- 50% users: Old embeddings
- Compare metrics for 1 week
- Full rollout if new embeddings superior
**User Feedback**:
- Survey: "How satisfied are you with search results?" (1-5 scale)
- Track: Number of "search not working" support tickets
- Monitor: User-reported false negatives ("I know this doc exists")
### Rollback Criteria
**Automatic Rollback** if:
- Recall decreases by >10% from baseline
- Error rate increases by >50%
- Query latency increases by >100%
**Manual Rollback** if:
- User complaints increase significantly
- Zero-result queries increase instead of decrease
## Future Enhancements
These improvements create a solid foundation. Future enhancements (in order of priority):
1. **Cross-Encoder Reranking** (ADR-012)
- Two-stage retrieval: broad recall (50 candidates) → precise reranking (top 10)
- Expected: +15-20% additional recall improvement
- Builds on: Better embeddings retrieve better candidates to rerank
2. **Hybrid Search** (ADR-013)
- Combine vector search + BM25 keyword search
- Expected: +10-15% additional recall (especially for exact matches)
- Builds on: Semantic chunks provide better keyword match context
3. **Multi-App Indexing** (ADR-014)
- Index calendar, deck, files (currently notes-only)
- Expected: Expands searchable corpus 3-5x
- Builds on: Proven chunking and embedding strategy
4. **GraphRAG** (ADR-015, conditional)
- Only if: Global thematic queries needed OR corpus >10K documents
- Expected: Relationship discovery, multi-hop reasoning
- Builds on: High-quality embeddings improve graph construction
## References
### Research Papers
1. **RecursiveCharacterTextSplitter**
- LangChain Documentation: https://python.langchain.com/docs/modules/data_connection/document_transformers/text_splitters/recursive_text_splitter
- Proven technique used by major RAG systems
2. **MTEB Leaderboard** (Massive Text Embedding Benchmark)
- https://huggingface.co/spaces/mteb/leaderboard
- Comprehensive embedding model comparison
3. **mxbai-embed-large**
- Model: https://huggingface.co/mixedbread-ai/mxbai-embed-large-v1
- Best general-purpose embedding model (MTEB: 64.68)
### Related ADRs
- **ADR-003**: Vector Database and Semantic Search Architecture (original implementation)
- **ADR-008**: MCP Sampling for Multi-App Semantic Search with RAG (answer generation)
### Tools & Libraries
- **LangChain Text Splitters**: https://python.langchain.com/docs/modules/data_connection/document_transformers/
- **Ollama Embedding Models**: https://ollama.ai/library
- **Qdrant Collections**: https://qdrant.tech/documentation/concepts/collections/
## Summary
This ADR addresses the root causes of poor semantic search recall:
1. **Better Chunking**: Semantic sentence-aware splitting (preserves context)
2. **Better Embeddings**: Upgrade to mxbai-embed-large-v1 (richer semantic space)
**Expected Impact**: 40-60% recall improvement with minimal cost and complexity.
**Why This Approach**:
- Fixes fundamentals before adding complexity
- Proven techniques (not experimental)
- Simple implementation (1-2 weeks)
- Creates foundation for future enhancements
- No new infrastructure or ongoing costs
**Next Steps**: Approve ADR → Implement changes → Reindex → Validate → Production rollout
+46 -7
View File
@@ -1,5 +1,6 @@
import logging
import os
import time
from collections.abc import AsyncIterator
from contextlib import AsyncExitStack, asynccontextmanager
from dataclasses import dataclass
@@ -44,6 +45,10 @@ from nextcloud_mcp_server.observability import (
setup_metrics,
setup_tracing,
)
from nextcloud_mcp_server.observability.metrics import (
record_dependency_check,
set_dependency_health,
)
from nextcloud_mcp_server.server import (
configure_calendar_tools,
configure_contacts_tools,
@@ -502,9 +507,9 @@ async def setup_oauth_config():
- External IdP mode: OIDC_DISCOVERY_URL points to external provider
→ External IdP for OAuth, Nextcloud user_oidc validates tokens and provides API access
Uses generic OIDC environment variables:
Uses OIDC environment variables:
- OIDC_DISCOVERY_URL: OIDC discovery endpoint (optional, defaults to NEXTCLOUD_HOST)
- OIDC_CLIENT_ID / OIDC_CLIENT_SECRET: Static credentials (optional, uses DCR if not provided)
- NEXTCLOUD_OIDC_CLIENT_ID / NEXTCLOUD_OIDC_CLIENT_SECRET: Static credentials (optional, uses DCR if not provided)
- NEXTCLOUD_OIDC_SCOPES: Requested OAuth scopes
This is done synchronously before FastMCP initialization because FastMCP
@@ -628,19 +633,21 @@ async def setup_oauth_config():
)
# Load client credentials (static or dynamic registration)
client_id = os.getenv("OIDC_CLIENT_ID")
client_secret = os.getenv("OIDC_CLIENT_SECRET")
client_id = os.getenv("NEXTCLOUD_OIDC_CLIENT_ID")
client_secret = os.getenv("NEXTCLOUD_OIDC_CLIENT_SECRET")
if client_id and client_secret:
logger.info(f"Using static OIDC client credentials: {client_id}")
elif registration_endpoint:
logger.info("OIDC_CLIENT_ID not set, attempting Dynamic Client Registration")
logger.info(
"NEXTCLOUD_OIDC_CLIENT_ID not set, attempting Dynamic Client Registration"
)
client_id, client_secret = await load_oauth_client_credentials(
nextcloud_host=nextcloud_host, registration_endpoint=registration_endpoint
)
else:
raise ValueError(
"OIDC_CLIENT_ID and OIDC_CLIENT_SECRET environment variables are required "
"NEXTCLOUD_OIDC_CLIENT_ID and NEXTCLOUD_OIDC_CLIENT_SECRET environment variables are required "
"when the OIDC provider does not support Dynamic Client Registration. "
f"Discovery URL: {discovery_url}"
)
@@ -1205,12 +1212,35 @@ def get_app(transport: str = "sse", enabled_apps: list[str] | None = None):
checks = {}
is_ready = True
# Check Nextcloud host configuration
# Check Nextcloud host configuration and connectivity
nextcloud_host = os.getenv("NEXTCLOUD_HOST")
if nextcloud_host:
checks["nextcloud_configured"] = "ok"
# Try to connect to Nextcloud
start_time = time.time()
try:
async with httpx.AsyncClient(timeout=2.0) as client:
response = await client.get(f"{nextcloud_host}/status.php")
duration = time.time() - start_time
if response.status_code == 200:
checks["nextcloud_reachable"] = "ok"
set_dependency_health("nextcloud", True)
else:
checks["nextcloud_reachable"] = (
f"error: status {response.status_code}"
)
set_dependency_health("nextcloud", False)
is_ready = False
record_dependency_check("nextcloud", duration)
except Exception as e:
duration = time.time() - start_time
checks["nextcloud_reachable"] = f"error: {str(e)}"
set_dependency_health("nextcloud", False)
record_dependency_check("nextcloud", duration)
is_ready = False
else:
checks["nextcloud_configured"] = "error: NEXTCLOUD_HOST not set"
set_dependency_health("nextcloud", False)
is_ready = False
# Check authentication configuration
@@ -1238,20 +1268,29 @@ def get_app(transport: str = "sse", enabled_apps: list[str] | None = None):
qdrant_url = os.getenv("QDRANT_URL") # Only set in network mode
if vector_sync_enabled and qdrant_url:
start_time = time.time()
try:
async with httpx.AsyncClient(timeout=2.0) as client:
response = await client.get(f"{qdrant_url}/readyz")
duration = time.time() - start_time
if response.status_code == 200:
checks["qdrant"] = "ok"
set_dependency_health("qdrant", True)
else:
checks["qdrant"] = f"error: status {response.status_code}"
set_dependency_health("qdrant", False)
is_ready = False
record_dependency_check("qdrant", duration)
except Exception as e:
duration = time.time() - start_time
checks["qdrant"] = f"error: {str(e)}"
set_dependency_health("qdrant", False)
record_dependency_check("qdrant", duration)
is_ready = False
elif vector_sync_enabled:
# Using embedded Qdrant (memory or persistent mode)
checks["qdrant"] = "embedded"
set_dependency_health("qdrant", True)
status_code = 200 if is_ready else 503
return JSONResponse(
+19 -7
View File
@@ -12,6 +12,10 @@ from mcp.server.fastmcp import Context
from ..client import NextcloudClient
from ..config import get_settings
from ..observability.metrics import (
oauth_token_cache_hits_total,
oauth_token_exchange_total,
)
from .token_exchange import exchange_token_for_audience
logger = logging.getLogger(__name__)
@@ -138,6 +142,7 @@ async def get_session_client_from_context(
logger.debug(
f"Using cached exchanged token (expires in {expiry - time.time():.1f}s)"
)
oauth_token_cache_hits_total.labels(hit="true").inc()
return NextcloudClient.from_token(
base_url=base_url, token=cached_token, username=username
)
@@ -145,17 +150,24 @@ async def get_session_client_from_context(
logger.debug("Cached token expired, removing from cache")
del _exchange_cache[cache_key]
oauth_token_cache_hits_total.labels(hit="false").inc()
# Perform RFC 8693 token exchange
logger.info(f"Exchanging MCP token for Nextcloud API token (user: {username})")
# Exchange for Nextcloud resource URI audience
exchanged_token, expires_in = await exchange_token_for_audience(
subject_token=mcp_token,
requested_audience=settings.nextcloud_resource_uri or "nextcloud",
requested_scopes=None, # Nextcloud doesn't support scopes
)
try:
# Exchange for Nextcloud resource URI audience
exchanged_token, expires_in = await exchange_token_for_audience(
subject_token=mcp_token,
requested_audience=settings.nextcloud_resource_uri or "nextcloud",
requested_scopes=None, # Nextcloud doesn't support scopes
)
oauth_token_exchange_total.labels(status="success").inc()
logger.info(f"Token exchange successful. Token expires in {expires_in}s")
logger.info(f"Token exchange successful. Token expires in {expires_in}s")
except Exception:
oauth_token_exchange_total.labels(status="error").inc()
raise
# Cache the exchanged token
# Use the minimum of exchange TTL and configured cache TTL
+107 -78
View File
@@ -35,6 +35,8 @@ from typing import Any, Optional
import aiosqlite
from cryptography.fernet import Fernet
from nextcloud_mcp_server.observability.metrics import record_db_operation
logger = logging.getLogger(__name__)
@@ -292,35 +294,43 @@ class RefreshTokenStorage:
# For Flow 2, set provisioned_at timestamp
provisioned_at = now if flow_type == "flow2" else None
async with aiosqlite.connect(self.db_path) as db:
await db.execute(
"""
INSERT OR REPLACE INTO refresh_tokens
(user_id, encrypted_token, expires_at, created_at, updated_at,
flow_type, token_audience, provisioned_at, provisioning_client_id, scopes)
VALUES (?, ?, ?, COALESCE((SELECT created_at FROM refresh_tokens WHERE user_id = ?), ?), ?,
?, ?, ?, ?, ?)
""",
(
user_id,
encrypted_token,
expires_at,
user_id,
now,
now,
flow_type,
token_audience,
provisioned_at,
provisioning_client_id,
scopes_json,
),
)
await db.commit()
start_time = time.time()
try:
async with aiosqlite.connect(self.db_path) as db:
await db.execute(
"""
INSERT OR REPLACE INTO refresh_tokens
(user_id, encrypted_token, expires_at, created_at, updated_at,
flow_type, token_audience, provisioned_at, provisioning_client_id, scopes)
VALUES (?, ?, ?, COALESCE((SELECT created_at FROM refresh_tokens WHERE user_id = ?), ?), ?,
?, ?, ?, ?, ?)
""",
(
user_id,
encrypted_token,
expires_at,
user_id,
now,
now,
flow_type,
token_audience,
provisioned_at,
provisioning_client_id,
scopes_json,
),
)
await db.commit()
duration = time.time() - start_time
record_db_operation("sqlite", "insert", duration, "success")
logger.info(
f"Stored refresh token for user {user_id}"
+ (f" (expires at {expires_at})" if expires_at else "")
)
logger.info(
f"Stored refresh token for user {user_id}"
+ (f" (expires at {expires_at})" if expires_at else "")
)
except Exception:
duration = time.time() - start_time
record_db_operation("sqlite", "insert", duration, "error")
raise
# Audit log
await self._audit_log(
@@ -422,40 +432,45 @@ class RefreshTokenStorage:
if not self._initialized:
await self.initialize()
async with aiosqlite.connect(self.db_path) as db:
async with db.execute(
"""
SELECT encrypted_token, expires_at, flow_type, token_audience,
provisioned_at, provisioning_client_id, scopes
FROM refresh_tokens WHERE user_id = ?
""",
(user_id,),
) as cursor:
row = await cursor.fetchone()
if not row:
logger.debug(f"No refresh token found for user {user_id}")
return None
(
encrypted_token,
expires_at,
flow_type,
token_audience,
provisioned_at,
provisioning_client_id,
scopes_json,
) = row
# Check expiration
if expires_at is not None and expires_at < time.time():
logger.warning(
f"Refresh token for user {user_id} has expired (expired at {expires_at})"
)
await self.delete_refresh_token(user_id)
return None
start_time = time.time()
try:
async with aiosqlite.connect(self.db_path) as db:
async with db.execute(
"""
SELECT encrypted_token, expires_at, flow_type, token_audience,
provisioned_at, provisioning_client_id, scopes
FROM refresh_tokens WHERE user_id = ?
""",
(user_id,),
) as cursor:
row = await cursor.fetchone()
if not row:
logger.debug(f"No refresh token found for user {user_id}")
duration = time.time() - start_time
record_db_operation("sqlite", "select", duration, "success")
return None
(
encrypted_token,
expires_at,
flow_type,
token_audience,
provisioned_at,
provisioning_client_id,
scopes_json,
) = row
# Check expiration
if expires_at is not None and expires_at < time.time():
logger.warning(
f"Refresh token for user {user_id} has expired (expired at {expires_at})"
)
await self.delete_refresh_token(user_id)
duration = time.time() - start_time
record_db_operation("sqlite", "select", duration, "success")
return None
decrypted_token = self.cipher.decrypt(encrypted_token).decode()
scopes = json.loads(scopes_json) if scopes_json else None
@@ -463,6 +478,9 @@ class RefreshTokenStorage:
f"Retrieved refresh token for user {user_id} (flow_type: {flow_type})"
)
duration = time.time() - start_time
record_db_operation("sqlite", "select", duration, "success")
return {
"refresh_token": decrypted_token,
"expires_at": expires_at,
@@ -474,6 +492,8 @@ class RefreshTokenStorage:
"scopes": scopes,
}
except Exception as e:
duration = time.time() - start_time
record_db_operation("sqlite", "select", duration, "error")
logger.error(f"Failed to decrypt refresh token for user {user_id}: {e}")
return None
@@ -568,25 +588,34 @@ class RefreshTokenStorage:
if not self._initialized:
await self.initialize()
async with aiosqlite.connect(self.db_path) as db:
cursor = await db.execute(
"DELETE FROM refresh_tokens WHERE user_id = ?",
(user_id,),
)
await db.commit()
deleted = cursor.rowcount > 0
start_time = time.time()
try:
async with aiosqlite.connect(self.db_path) as db:
cursor = await db.execute(
"DELETE FROM refresh_tokens WHERE user_id = ?",
(user_id,),
)
await db.commit()
deleted = cursor.rowcount > 0
if deleted:
logger.info(f"Deleted refresh token for user {user_id}")
await self._audit_log(
event="delete_refresh_token",
user_id=user_id,
auth_method="offline_access",
)
else:
logger.debug(f"No refresh token to delete for user {user_id}")
duration = time.time() - start_time
record_db_operation("sqlite", "delete", duration, "success")
return deleted
if deleted:
logger.info(f"Deleted refresh token for user {user_id}")
await self._audit_log(
event="delete_refresh_token",
user_id=user_id,
auth_method="offline_access",
)
else:
logger.debug(f"No refresh token to delete for user {user_id}")
return deleted
except Exception:
duration = time.time() - start_time
record_db_operation("sqlite", "delete", duration, "error")
raise
async def get_all_user_ids(self) -> list[str]:
"""
@@ -26,6 +26,10 @@ from jwt import PyJWKClient
from mcp.server.auth.provider import AccessToken, TokenVerifier
from nextcloud_mcp_server.config import Settings
from nextcloud_mcp_server.observability.metrics import (
oauth_token_cache_hits_total,
record_oauth_token_validation,
)
logger = logging.getLogger(__name__)
@@ -105,8 +109,11 @@ class UnifiedTokenVerifier(TokenVerifier):
cached = self._get_cached_token(token)
if cached:
logger.debug("Token found in cache")
oauth_token_cache_hits_total.labels(hit="true").inc()
return cached
oauth_token_cache_hits_total.labels(hit="false").inc()
# Both modes do the same validation (MCP audience only)
return await self._verify_mcp_audience(token)
@@ -124,13 +131,24 @@ class UnifiedTokenVerifier(TokenVerifier):
Returns:
AccessToken if valid with MCP audience, None otherwise
"""
validation_method = "unknown"
try:
# Attempt JWT verification first
if self._is_jwt_format(token) and self.jwks_client:
validation_method = "jwt"
payload = await self._verify_jwt_signature(token)
if payload:
record_oauth_token_validation("jwt", "valid")
else:
record_oauth_token_validation("jwt", "invalid")
else:
# Fall back to introspection for opaque tokens
validation_method = "introspect"
payload = await self._introspect_token(token)
if payload:
record_oauth_token_validation("introspect", "valid")
else:
record_oauth_token_validation("introspect", "invalid")
if not payload:
return None
@@ -146,6 +164,8 @@ class UnifiedTokenVerifier(TokenVerifier):
f"Got {audiences}, need MCP ({self.settings.oidc_client_id} or "
f"{self.settings.nextcloud_mcp_server_url})"
)
# Record as invalid due to audience mismatch
record_oauth_token_validation(validation_method, "invalid")
return None
# Log based on mode for clarity
@@ -163,6 +183,7 @@ class UnifiedTokenVerifier(TokenVerifier):
except Exception as e:
logger.error(f"Token verification failed: {e}")
record_oauth_token_validation(validation_method, "error")
return None
def _has_mcp_audience(self, payload: dict[str, Any]) -> bool:
+2 -2
View File
@@ -288,8 +288,8 @@ def get_settings() -> Settings:
return Settings(
# OAuth/OIDC settings
oidc_discovery_url=os.getenv("OIDC_DISCOVERY_URL"),
oidc_client_id=os.getenv("OIDC_CLIENT_ID"),
oidc_client_secret=os.getenv("OIDC_CLIENT_SECRET"),
oidc_client_id=os.getenv("NEXTCLOUD_OIDC_CLIENT_ID"),
oidc_client_secret=os.getenv("NEXTCLOUD_OIDC_CLIENT_SECRET"),
oidc_issuer=os.getenv("OIDC_ISSUER"),
# Nextcloud settings
nextcloud_host=os.getenv("NEXTCLOUD_HOST"),
@@ -12,13 +12,24 @@ class NotesSearchController:
"""
Search notes using token-based matching with relevance ranking.
Returns notes sorted by relevance score.
If query is empty, returns all notes.
"""
search_results = []
query_tokens = self._process_query(query)
# If empty query after processing, return empty results
# If empty query after processing, return all notes
if not query_tokens:
return []
async for note in notes:
search_results.append(
{
"id": note.get("id"),
"title": note.get("title"),
"category": note.get("category"),
"modified": note.get("modified"),
"_score": None, # No score for unfiltered results
}
)
return search_results
# Process and score each note
async for note in notes:
@@ -352,3 +352,92 @@ def record_dependency_check(dependency: str, duration: float) -> None:
duration: Check duration in seconds
"""
dependency_check_duration_seconds.labels(dependency=dependency).observe(duration)
def record_vector_sync_scan(documents_found: int) -> None:
"""
Record documents scanned during vector sync.
Args:
documents_found: Number of documents discovered in scan
"""
vector_sync_documents_scanned_total.inc(documents_found)
def record_vector_sync_processing(duration: float, status: str = "success") -> None:
"""
Record document processing with duration and status.
Args:
duration: Processing duration in seconds
status: "success" or "error"
"""
vector_sync_documents_processed_total.labels(status=status).inc()
vector_sync_processing_duration_seconds.observe(duration)
def record_qdrant_operation(operation: str, status: str = "success") -> None:
"""
Record Qdrant vector database operation.
Args:
operation: Operation type ("upsert", "search", "delete")
status: "success" or "error"
"""
qdrant_operations_total.labels(operation=operation, status=status).inc()
def update_vector_sync_queue_size(size: int) -> None:
"""
Update vector sync queue size gauge.
Args:
size: Current queue size
"""
vector_sync_queue_size.set(size)
# =============================================================================
# Decorator for Automatic Tool Instrumentation
# =============================================================================
def instrument_tool(func):
"""
Decorator to automatically instrument MCP tool functions with metrics.
Wraps async tool functions to record execution time and success/error status.
Compatible with @mcp.tool() and @require_scopes() decorators.
Usage:
@mcp.tool()
@require_scopes("notes:write")
@instrument_tool
async def nc_notes_create_note(...):
...
Args:
func: The async function to instrument
Returns:
Wrapped function with metrics instrumentation
"""
import functools
import time
@functools.wraps(func)
async def wrapper(*args, **kwargs):
tool_name = func.__name__
start_time = time.time()
try:
result = await func(*args, **kwargs)
duration = time.time() - start_time
record_tool_call(tool_name, duration, "success")
return result
except Exception as e:
duration = time.time() - start_time
record_tool_call(tool_name, duration, "error")
record_tool_error(tool_name, type(e).__name__)
raise
return wrapper
+17
View File
@@ -12,6 +12,7 @@ from nextcloud_mcp_server.models.calendar import (
ListTodosResponse,
Todo,
)
from nextcloud_mcp_server.observability.metrics import instrument_tool
logger = logging.getLogger(__name__)
@@ -20,6 +21,7 @@ def configure_calendar_tools(mcp: FastMCP):
# Calendar tools
@mcp.tool()
@require_scopes("calendar:read")
@instrument_tool
async def nc_calendar_list_calendars(ctx: Context) -> ListCalendarsResponse:
"""List all available calendars for the user"""
client = await get_client(ctx)
@@ -30,6 +32,7 @@ def configure_calendar_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("calendar:write")
@instrument_tool
async def nc_calendar_create_event(
calendar_name: str,
title: str,
@@ -106,6 +109,7 @@ def configure_calendar_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("calendar:read")
@instrument_tool
async def nc_calendar_list_events(
calendar_name: str,
ctx: Context,
@@ -208,6 +212,7 @@ def configure_calendar_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("calendar:read")
@instrument_tool
async def nc_calendar_get_event(
calendar_name: str,
event_uid: str,
@@ -220,6 +225,7 @@ def configure_calendar_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("calendar:write")
@instrument_tool
async def nc_calendar_update_event(
calendar_name: str,
event_uid: str,
@@ -293,6 +299,7 @@ def configure_calendar_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("calendar:write")
@instrument_tool
async def nc_calendar_delete_event(
calendar_name: str,
event_uid: str,
@@ -304,6 +311,7 @@ def configure_calendar_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("calendar:write")
@instrument_tool
async def nc_calendar_create_meeting(
title: str,
date: str,
@@ -370,6 +378,7 @@ def configure_calendar_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("calendar:read")
@instrument_tool
async def nc_calendar_get_upcoming_events(
ctx: Context,
calendar_name: str = "", # Empty = all calendars
@@ -420,6 +429,7 @@ def configure_calendar_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("calendar:read")
@instrument_tool
async def nc_calendar_find_availability(
duration_minutes: int,
ctx: Context,
@@ -500,6 +510,7 @@ def configure_calendar_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("calendar:write")
@instrument_tool
async def nc_calendar_bulk_operations(
operation: str, # "update", "delete", "move"
ctx: Context,
@@ -749,6 +760,7 @@ def configure_calendar_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("calendar:write")
@instrument_tool
async def nc_calendar_manage_calendar(
action: str, # "create", "delete", "update", "list"
ctx: Context,
@@ -818,6 +830,7 @@ def configure_calendar_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("todo:read", "calendar:read")
@instrument_tool
async def nc_calendar_list_todos(
calendar_name: str,
ctx: Context,
@@ -863,6 +876,7 @@ def configure_calendar_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("todo:write", "calendar:read")
@instrument_tool
async def nc_calendar_create_todo(
calendar_name: str,
summary: str,
@@ -906,6 +920,7 @@ def configure_calendar_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("todo:write", "calendar:read")
@instrument_tool
async def nc_calendar_update_todo(
calendar_name: str,
todo_uid: str,
@@ -966,6 +981,7 @@ def configure_calendar_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("todo:write", "calendar:read")
@instrument_tool
async def nc_calendar_delete_todo(
calendar_name: str,
todo_uid: str,
@@ -986,6 +1002,7 @@ def configure_calendar_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("todo:read", "calendar:read")
@instrument_tool
async def nc_calendar_search_todos(
ctx: Context,
status: Optional[str] = None,
+8
View File
@@ -4,6 +4,7 @@ from mcp.server.fastmcp import Context, FastMCP
from nextcloud_mcp_server.auth import require_scopes
from nextcloud_mcp_server.context import get_client
from nextcloud_mcp_server.observability.metrics import instrument_tool
logger = logging.getLogger(__name__)
@@ -12,6 +13,7 @@ def configure_contacts_tools(mcp: FastMCP):
# Contacts tools
@mcp.tool()
@require_scopes("contacts:read")
@instrument_tool
async def nc_contacts_list_addressbooks(ctx: Context):
"""List all addressbooks for the user."""
client = await get_client(ctx)
@@ -19,6 +21,7 @@ def configure_contacts_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("contacts:read")
@instrument_tool
async def nc_contacts_list_contacts(ctx: Context, *, addressbook: str):
"""List all contacts in the specified addressbook."""
client = await get_client(ctx)
@@ -26,6 +29,7 @@ def configure_contacts_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("contacts:write")
@instrument_tool
async def nc_contacts_create_addressbook(
ctx: Context, *, name: str, display_name: str
):
@@ -42,6 +46,7 @@ def configure_contacts_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("contacts:write")
@instrument_tool
async def nc_contacts_delete_addressbook(ctx: Context, *, name: str):
"""Delete an addressbook."""
client = await get_client(ctx)
@@ -49,6 +54,7 @@ def configure_contacts_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("contacts:write")
@instrument_tool
async def nc_contacts_create_contact(
ctx: Context, *, addressbook: str, uid: str, contact_data: dict
):
@@ -66,6 +72,7 @@ def configure_contacts_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("contacts:write")
@instrument_tool
async def nc_contacts_delete_contact(ctx: Context, *, addressbook: str, uid: str):
"""Delete a contact."""
client = await get_client(ctx)
@@ -73,6 +80,7 @@ def configure_contacts_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("contacts:write")
@instrument_tool
async def nc_contacts_update_contact(
ctx: Context, *, addressbook: str, uid: str, contact_data: dict, etag: str = ""
):
+14
View File
@@ -24,6 +24,7 @@ from nextcloud_mcp_server.models.cookbook import (
UpdateRecipeResponse,
Version,
)
from nextcloud_mcp_server.observability.metrics import instrument_tool
logger = logging.getLogger(__name__)
@@ -72,6 +73,7 @@ def configure_cookbook_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("cookbook:write")
@instrument_tool
async def nc_cookbook_import_recipe(url: str, ctx: Context) -> ImportRecipeResponse:
"""Import a recipe from a URL using schema.org metadata.
@@ -129,6 +131,7 @@ def configure_cookbook_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("cookbook:read")
@instrument_tool
async def nc_cookbook_list_recipes(ctx: Context) -> ListRecipesResponse:
"""Get all recipes in the database"""
client = await get_client(ctx)
@@ -154,6 +157,7 @@ def configure_cookbook_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("cookbook:read")
@instrument_tool
async def nc_cookbook_get_recipe(recipe_id: int, ctx: Context) -> Recipe:
"""Get a specific recipe by its ID"""
client = await get_client(ctx)
@@ -179,6 +183,7 @@ def configure_cookbook_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("cookbook:write")
@instrument_tool
async def nc_cookbook_create_recipe(
name: str,
description: str | None = None,
@@ -258,6 +263,7 @@ def configure_cookbook_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("cookbook:write")
@instrument_tool
async def nc_cookbook_update_recipe(
recipe_id: int,
name: str | None = None,
@@ -347,6 +353,7 @@ def configure_cookbook_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("cookbook:write")
@instrument_tool
async def nc_cookbook_delete_recipe(
recipe_id: int, ctx: Context
) -> DeleteRecipeResponse:
@@ -382,6 +389,7 @@ def configure_cookbook_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("cookbook:read")
@instrument_tool
async def nc_cookbook_search_recipes(
query: str, ctx: Context
) -> SearchRecipesResponse:
@@ -418,6 +426,7 @@ def configure_cookbook_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("cookbook:read")
@instrument_tool
async def nc_cookbook_list_categories(ctx: Context) -> ListCategoriesResponse:
"""Get all known categories.
@@ -445,6 +454,7 @@ def configure_cookbook_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("cookbook:read")
@instrument_tool
async def nc_cookbook_get_recipes_in_category(
category: str, ctx: Context
) -> ListRecipesResponse:
@@ -481,6 +491,7 @@ def configure_cookbook_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("cookbook:read")
@instrument_tool
async def nc_cookbook_list_keywords(ctx: Context) -> ListKeywordsResponse:
"""Get all known keywords/tags"""
client = await get_client(ctx)
@@ -506,6 +517,7 @@ def configure_cookbook_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("cookbook:read")
@instrument_tool
async def nc_cookbook_get_recipes_with_keywords(
keywords: list[str], ctx: Context
) -> ListRecipesResponse:
@@ -540,6 +552,7 @@ def configure_cookbook_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("cookbook:write")
@instrument_tool
async def nc_cookbook_set_config(
folder: str | None = None,
update_interval: int | None = None,
@@ -583,6 +596,7 @@ def configure_cookbook_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("cookbook:write")
@instrument_tool
async def nc_cookbook_reindex(ctx: Context) -> ReindexResponse:
"""Trigger a rescan of all recipes into the caching database.
+26
View File
@@ -18,6 +18,7 @@ from nextcloud_mcp_server.models.deck import (
LabelOperationResponse,
StackOperationResponse,
)
from nextcloud_mcp_server.observability.metrics import instrument_tool
logger = logging.getLogger(__name__)
@@ -118,6 +119,7 @@ def configure_deck_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("deck:read")
@instrument_tool
async def deck_get_boards(ctx: Context) -> list[DeckBoard]:
"""Get all Nextcloud Deck boards"""
client = await get_client(ctx)
@@ -126,6 +128,7 @@ def configure_deck_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("deck:read")
@instrument_tool
async def deck_get_board(ctx: Context, board_id: int) -> DeckBoard:
"""Get details of a specific Nextcloud Deck board"""
client = await get_client(ctx)
@@ -134,6 +137,7 @@ def configure_deck_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("deck:read")
@instrument_tool
async def deck_get_stacks(ctx: Context, board_id: int) -> list[DeckStack]:
"""Get all stacks in a Nextcloud Deck board"""
client = await get_client(ctx)
@@ -142,6 +146,7 @@ def configure_deck_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("deck:read")
@instrument_tool
async def deck_get_stack(ctx: Context, board_id: int, stack_id: int) -> DeckStack:
"""Get details of a specific Nextcloud Deck stack"""
client = await get_client(ctx)
@@ -150,6 +155,7 @@ def configure_deck_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("deck:read")
@instrument_tool
async def deck_get_cards(
ctx: Context, board_id: int, stack_id: int
) -> list[DeckCard]:
@@ -162,6 +168,7 @@ def configure_deck_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("deck:read")
@instrument_tool
async def deck_get_card(
ctx: Context, board_id: int, stack_id: int, card_id: int
) -> DeckCard:
@@ -172,6 +179,7 @@ def configure_deck_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("deck:read")
@instrument_tool
async def deck_get_labels(ctx: Context, board_id: int) -> list[DeckLabel]:
"""Get all labels in a Nextcloud Deck board"""
client = await get_client(ctx)
@@ -180,6 +188,7 @@ def configure_deck_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("deck:read")
@instrument_tool
async def deck_get_label(ctx: Context, board_id: int, label_id: int) -> DeckLabel:
"""Get details of a specific Nextcloud Deck label"""
client = await get_client(ctx)
@@ -190,6 +199,7 @@ def configure_deck_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("deck:write")
@instrument_tool
async def deck_create_board(
ctx: Context, title: str, color: str
) -> CreateBoardResponse:
@@ -207,6 +217,7 @@ def configure_deck_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("deck:write")
@instrument_tool
async def deck_create_stack(
ctx: Context, board_id: int, title: str, order: int
) -> CreateStackResponse:
@@ -223,6 +234,7 @@ def configure_deck_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("deck:write")
@instrument_tool
async def deck_update_stack(
ctx: Context,
board_id: int,
@@ -249,6 +261,7 @@ def configure_deck_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("deck:write")
@instrument_tool
async def deck_delete_stack(
ctx: Context, board_id: int, stack_id: int
) -> StackOperationResponse:
@@ -270,6 +283,7 @@ def configure_deck_tools(mcp: FastMCP):
# Card Tools
@mcp.tool()
@require_scopes("deck:write")
@instrument_tool
async def deck_create_card(
ctx: Context,
board_id: int,
@@ -304,6 +318,7 @@ def configure_deck_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("deck:write")
@instrument_tool
async def deck_update_card(
ctx: Context,
board_id: int,
@@ -357,6 +372,7 @@ def configure_deck_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("deck:write")
@instrument_tool
async def deck_delete_card(
ctx: Context, board_id: int, stack_id: int, card_id: int
) -> CardOperationResponse:
@@ -379,6 +395,7 @@ def configure_deck_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("deck:write")
@instrument_tool
async def deck_archive_card(
ctx: Context, board_id: int, stack_id: int, card_id: int
) -> CardOperationResponse:
@@ -401,6 +418,7 @@ def configure_deck_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("deck:write")
@instrument_tool
async def deck_unarchive_card(
ctx: Context, board_id: int, stack_id: int, card_id: int
) -> CardOperationResponse:
@@ -423,6 +441,7 @@ def configure_deck_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("deck:write")
@instrument_tool
async def deck_reorder_card(
ctx: Context,
board_id: int,
@@ -455,6 +474,7 @@ def configure_deck_tools(mcp: FastMCP):
# Label Tools
@mcp.tool()
@require_scopes("deck:write")
@instrument_tool
async def deck_create_label(
ctx: Context, board_id: int, title: str, color: str
) -> CreateLabelResponse:
@@ -471,6 +491,7 @@ def configure_deck_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("deck:write")
@instrument_tool
async def deck_update_label(
ctx: Context,
board_id: int,
@@ -497,6 +518,7 @@ def configure_deck_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("deck:write")
@instrument_tool
async def deck_delete_label(
ctx: Context, board_id: int, label_id: int
) -> LabelOperationResponse:
@@ -518,6 +540,7 @@ def configure_deck_tools(mcp: FastMCP):
# Card-Label Assignment Tools
@mcp.tool()
@require_scopes("deck:write")
@instrument_tool
async def deck_assign_label_to_card(
ctx: Context, board_id: int, stack_id: int, card_id: int, label_id: int
) -> CardOperationResponse:
@@ -541,6 +564,7 @@ def configure_deck_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("deck:write")
@instrument_tool
async def deck_remove_label_from_card(
ctx: Context, board_id: int, stack_id: int, card_id: int, label_id: int
) -> CardOperationResponse:
@@ -565,6 +589,7 @@ def configure_deck_tools(mcp: FastMCP):
# Card-User Assignment Tools
@mcp.tool()
@require_scopes("deck:write")
@instrument_tool
async def deck_assign_user_to_card(
ctx: Context, board_id: int, stack_id: int, card_id: int, user_id: str
) -> CardOperationResponse:
@@ -588,6 +613,7 @@ def configure_deck_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("deck:write")
@instrument_tool
async def deck_unassign_user_from_card(
ctx: Context, board_id: int, stack_id: int, card_id: int, user_id: str
) -> CardOperationResponse:
+8
View File
@@ -17,6 +17,7 @@ from nextcloud_mcp_server.models.notes import (
SearchNotesResponse,
UpdateNoteResponse,
)
from nextcloud_mcp_server.observability.metrics import instrument_tool
logger = logging.getLogger(__name__)
@@ -86,6 +87,7 @@ def configure_notes_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("notes:write")
@instrument_tool
async def nc_notes_create_note(
title: str, content: str, category: str, ctx: Context
) -> CreateNoteResponse:
@@ -132,6 +134,7 @@ def configure_notes_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("notes:write")
@instrument_tool
async def nc_notes_update_note(
note_id: int,
etag: str,
@@ -197,6 +200,7 @@ def configure_notes_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("notes:write")
@instrument_tool
async def nc_notes_append_content(
note_id: int, content: str, ctx: Context
) -> AppendContentResponse:
@@ -247,6 +251,7 @@ def configure_notes_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("notes:read")
@instrument_tool
async def nc_notes_search_notes(query: str, ctx: Context) -> SearchNotesResponse:
"""Search notes by title or content, returning only id, title, and category (requires notes:read scope)."""
client = await get_client(ctx)
@@ -293,6 +298,7 @@ def configure_notes_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("notes:read")
@instrument_tool
async def nc_notes_get_note(note_id: int, ctx: Context) -> Note:
"""Get a specific note by its ID (requires notes:read scope)"""
client = await get_client(ctx)
@@ -322,6 +328,7 @@ def configure_notes_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("notes:read")
@instrument_tool
async def nc_notes_get_attachment(
note_id: int, attachment_filename: str, ctx: Context
) -> dict[str, str]:
@@ -368,6 +375,7 @@ def configure_notes_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("notes:write")
@instrument_tool
async def nc_notes_delete_note(note_id: int, ctx: Context) -> DeleteNoteResponse:
"""Delete a note permanently"""
logger.info("Deleting note %s", note_id)
+99 -35
View File
@@ -21,6 +21,10 @@ from nextcloud_mcp_server.models.semantic import (
SemanticSearchResult,
VectorSyncStatusResponse,
)
from nextcloud_mcp_server.observability.metrics import (
instrument_tool,
record_qdrant_operation,
)
logger = logging.getLogger(__name__)
@@ -30,6 +34,7 @@ def configure_semantic_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("semantic:read")
@instrument_tool
async def nc_semantic_search(
query: str, ctx: Context, limit: int = 10, score_threshold: float = 0.7
) -> SemanticSearchResponse:
@@ -85,26 +90,33 @@ def configure_semantic_tools(mcp: FastMCP):
# Note: Currently only searching notes (doc_type="note")
# Future: Remove doc_type filter to search all apps
qdrant_client = await get_qdrant_client()
search_response = await qdrant_client.query_points(
collection_name=settings.get_collection_name(),
query=query_embedding,
query_filter=Filter(
must=[
FieldCondition(
key="user_id",
match=MatchValue(value=username),
),
FieldCondition(
key="doc_type",
match=MatchValue(value="note"),
),
]
),
limit=limit * 2, # Get extra for filtering
score_threshold=score_threshold,
with_payload=True,
with_vectors=False, # Don't return vectors to save bandwidth
)
try:
search_response = await qdrant_client.query_points(
collection_name=settings.get_collection_name(),
query=query_embedding,
query_filter=Filter(
must=[
FieldCondition(
key="user_id",
match=MatchValue(value=username),
),
FieldCondition(
key="doc_type",
match=MatchValue(value="note"),
),
]
),
limit=limit * 2, # Get extra for filtering
score_threshold=score_threshold,
with_payload=True,
with_vectors=False, # Don't return vectors to save bandwidth
)
# Record successful search operation
record_qdrant_operation("search", "success")
except Exception:
# Record failed search operation
record_qdrant_operation("search", "error")
raise
logger.info(
f"Qdrant returned {len(search_response.points)} results "
@@ -208,6 +220,7 @@ def configure_semantic_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("semantic:read")
@instrument_tool
async def nc_semantic_search_answer(
query: str,
ctx: Context,
@@ -331,21 +344,71 @@ def configure_semantic_tools(mcp: FastMCP):
success=True,
)
# 4. Construct context from retrieved documents
# 4. Fetch full content for notes to provide complete context to LLM
# Filter out inaccessible notes (deleted or permissions changed)
client = await get_client(ctx)
accessible_results = []
full_contents = [] # Full content for accessible notes
for result in search_response.results:
if result.doc_type == "note":
try:
note = await client.notes.get_note(result.id)
# Note is accessible, store full content
accessible_results.append(result)
full_contents.append(note.get("content", ""))
logger.debug(
f"Fetched full content for note {result.id} "
f"(length: {len(full_contents[-1])} chars)"
)
except Exception as e:
# Note might have been deleted or permissions changed
# Filter it out to avoid corrupting LLM with inaccessible data
logger.warning(
f"Failed to fetch full content for note {result.id}: {e}. "
f"Excluding from results."
)
else:
# Non-note document types (future: calendar, deck, files)
# For now, keep them with excerpts
accessible_results.append(result)
full_contents.append(None)
# Check if we filtered out all results
if not accessible_results:
logger.warning(f"All search results became inaccessible for query: {query}")
return SamplingSearchResponse(
query=query,
generated_answer="All matching documents are no longer accessible.",
sources=[],
total_found=0,
search_method="semantic_sampling",
success=True,
)
# 5. Construct context from accessible documents with full content
context_parts = []
for idx, result in enumerate(search_response.results, 1):
for idx, (result, content) in enumerate(
zip(accessible_results, full_contents), 1
):
# Use full content if available (notes), otherwise use excerpt
if content is not None:
content_field = f"Content: {content}"
else:
content_field = f"Excerpt: {result.excerpt}"
context_parts.append(
f"[Document {idx}]\n"
f"Type: {result.doc_type}\n"
f"Title: {result.title}\n"
f"Category: {result.category}\n"
f"Excerpt: {result.excerpt}\n"
f"{content_field}\n"
f"Relevance Score: {result.score:.2f}\n"
)
context = "\n".join(context_parts)
# 5. Construct prompt - reuse user's query, add context and instructions
# 6. Construct prompt - reuse user's query, add context and instructions
prompt = (
f"{query}\n\n"
f"Here are relevant documents from Nextcloud (notes, calendar events, deck cards, files, contacts):\n\n"
@@ -401,8 +464,8 @@ def configure_semantic_tools(mcp: FastMCP):
return SamplingSearchResponse(
query=query,
generated_answer=generated_answer,
sources=search_response.results,
total_found=search_response.total_found,
sources=accessible_results,
total_found=len(accessible_results),
search_method="semantic_sampling",
model_used=sampling_result.model,
stop_reason=sampling_result.stopReason,
@@ -419,11 +482,11 @@ def configure_semantic_tools(mcp: FastMCP):
generated_answer=(
f"[Sampling request timed out]\n\n"
f"The answer generation took too long (>30s). "
f"Found {search_response.total_found} relevant documents. "
f"Found {len(accessible_results)} relevant documents. "
f"Please review the sources below or try a simpler query."
),
sources=search_response.results,
total_found=search_response.total_found,
sources=accessible_results,
total_found=len(accessible_results),
search_method="semantic_sampling_timeout",
success=True,
)
@@ -454,11 +517,11 @@ def configure_semantic_tools(mcp: FastMCP):
query=query,
generated_answer=(
f"[{user_message}]\n\n"
f"Found {search_response.total_found} relevant documents. "
f"Found {len(accessible_results)} relevant documents. "
f"Please review the sources below."
),
sources=search_response.results,
total_found=search_response.total_found,
sources=accessible_results,
total_found=len(accessible_results),
search_method=search_method,
success=True,
)
@@ -475,17 +538,18 @@ def configure_semantic_tools(mcp: FastMCP):
query=query,
generated_answer=(
f"[Unexpected error during sampling]\n\n"
f"Found {search_response.total_found} relevant documents. "
f"Found {len(accessible_results)} relevant documents. "
f"Please review the sources below."
),
sources=search_response.results,
total_found=search_response.total_found,
sources=accessible_results,
total_found=len(accessible_results),
search_method="semantic_sampling_error",
success=True,
)
@mcp.tool()
@require_scopes("semantic:read")
@instrument_tool
async def nc_get_vector_sync_status(ctx: Context) -> VectorSyncStatusResponse:
"""Get the current vector sync status.
+6
View File
@@ -6,6 +6,7 @@ from mcp.server.fastmcp import Context, FastMCP
from nextcloud_mcp_server.auth import require_scopes
from nextcloud_mcp_server.context import get_client
from nextcloud_mcp_server.observability.metrics import instrument_tool
def configure_sharing_tools(mcp: FastMCP):
@@ -17,6 +18,7 @@ def configure_sharing_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("sharing:write")
@instrument_tool
async def nc_share_create(
path: str,
share_with: str,
@@ -56,6 +58,7 @@ def configure_sharing_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("sharing:write")
@instrument_tool
async def nc_share_delete(share_id: int, ctx: Context) -> str:
"""Delete a share by its ID.
@@ -75,6 +78,7 @@ def configure_sharing_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("sharing:write")
@instrument_tool
async def nc_share_get(share_id: int, ctx: Context) -> str:
"""Get information about a specific share.
@@ -93,6 +97,7 @@ def configure_sharing_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("sharing:write")
@instrument_tool
async def nc_share_list(
ctx: Context, path: str | None = None, shared_with_me: bool = False
) -> str:
@@ -114,6 +119,7 @@ def configure_sharing_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("sharing:write")
@instrument_tool
async def nc_share_update(share_id: int, permissions: int, ctx: Context) -> str:
"""Update the permissions of an existing share.
+7
View File
@@ -4,6 +4,7 @@ from mcp.server.fastmcp import Context, FastMCP
from nextcloud_mcp_server.auth import require_scopes
from nextcloud_mcp_server.context import get_client
from nextcloud_mcp_server.observability.metrics import instrument_tool
logger = logging.getLogger(__name__)
@@ -12,6 +13,7 @@ def configure_tables_tools(mcp: FastMCP):
# Tables tools
@mcp.tool()
@require_scopes("tables:read")
@instrument_tool
async def nc_tables_list_tables(ctx: Context):
"""List all tables available to the user"""
client = await get_client(ctx)
@@ -19,6 +21,7 @@ def configure_tables_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("tables:read")
@instrument_tool
async def nc_tables_get_schema(table_id: int, ctx: Context):
"""Get the schema/structure of a specific table including columns and views"""
client = await get_client(ctx)
@@ -26,6 +29,7 @@ def configure_tables_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("tables:read")
@instrument_tool
async def nc_tables_read_table(
table_id: int,
ctx: Context,
@@ -38,6 +42,7 @@ def configure_tables_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("tables:write")
@instrument_tool
async def nc_tables_insert_row(table_id: int, data: dict, ctx: Context):
"""Insert a new row into a table.
@@ -48,6 +53,7 @@ def configure_tables_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("tables:write")
@instrument_tool
async def nc_tables_update_row(row_id: int, data: dict, ctx: Context):
"""Update an existing row in a table.
@@ -58,6 +64,7 @@ def configure_tables_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("tables:write")
@instrument_tool
async def nc_tables_delete_row(row_id: int, ctx: Context):
"""Delete a row from a table"""
client = await get_client(ctx)
+12
View File
@@ -5,6 +5,7 @@ from mcp.server.fastmcp import Context, FastMCP
from nextcloud_mcp_server.auth import require_scopes
from nextcloud_mcp_server.context import get_client
from nextcloud_mcp_server.models import DirectoryListing, FileInfo, SearchFilesResponse
from nextcloud_mcp_server.observability.metrics import instrument_tool
from nextcloud_mcp_server.utils.document_parser import (
is_parseable_document,
parse_document,
@@ -17,6 +18,7 @@ def configure_webdav_tools(mcp: FastMCP):
# WebDAV file system tools
@mcp.tool()
@require_scopes("files:read")
@instrument_tool
async def nc_webdav_list_directory(
ctx: Context, path: str = ""
) -> DirectoryListing:
@@ -50,6 +52,7 @@ def configure_webdav_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("files:read")
@instrument_tool
async def nc_webdav_read_file(path: str, ctx: Context):
"""Read the content of a file from NextCloud.
@@ -130,6 +133,7 @@ def configure_webdav_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("files:write")
@instrument_tool
async def nc_webdav_write_file(
path: str, content: str, ctx: Context, content_type: str | None = None
):
@@ -158,6 +162,7 @@ def configure_webdav_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("files:write")
@instrument_tool
async def nc_webdav_create_directory(path: str, ctx: Context):
"""Create a directory in NextCloud.
@@ -172,6 +177,7 @@ def configure_webdav_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("files:write")
@instrument_tool
async def nc_webdav_delete_resource(path: str, ctx: Context):
"""Delete a file or directory in NextCloud.
@@ -186,6 +192,7 @@ def configure_webdav_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("files:write")
@instrument_tool
async def nc_webdav_move_resource(
source_path: str, destination_path: str, ctx: Context, overwrite: bool = False
):
@@ -206,6 +213,7 @@ def configure_webdav_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("files:write")
@instrument_tool
async def nc_webdav_copy_resource(
source_path: str, destination_path: str, ctx: Context, overwrite: bool = False
):
@@ -226,6 +234,7 @@ def configure_webdav_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("files:read")
@instrument_tool
async def nc_webdav_search_files(
ctx: Context,
scope: str = "",
@@ -342,6 +351,7 @@ def configure_webdav_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("files:read")
@instrument_tool
async def nc_webdav_find_by_name(
pattern: str, ctx: Context, scope: str = "", limit: int | None = None
) -> SearchFilesResponse:
@@ -369,6 +379,7 @@ def configure_webdav_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("files:read")
@instrument_tool
async def nc_webdav_find_by_type(
mime_type: str, ctx: Context, scope: str = "", limit: int | None = None
) -> SearchFilesResponse:
@@ -396,6 +407,7 @@ def configure_webdav_tools(mcp: FastMCP):
@mcp.tool()
@require_scopes("files:read")
@instrument_tool
async def nc_webdav_list_favorites(
ctx: Context, scope: str = "", limit: int | None = None
) -> SearchFilesResponse:
+87 -49
View File
@@ -15,6 +15,11 @@ from qdrant_client.models import FieldCondition, Filter, MatchValue, PointStruct
from nextcloud_mcp_server.client import NextcloudClient
from nextcloud_mcp_server.config import get_settings
from nextcloud_mcp_server.embedding import get_embedding_service
from nextcloud_mcp_server.observability.metrics import (
record_qdrant_operation,
record_vector_sync_processing,
update_vector_sync_queue_size,
)
from nextcloud_mcp_server.observability.tracing import trace_operation
from nextcloud_mcp_server.vector.document_chunker import DocumentChunker
from nextcloud_mcp_server.vector.qdrant_client import get_qdrant_client
@@ -57,11 +62,21 @@ async def processor_task(
with anyio.fail_after(1.0):
doc_task = await receive_stream.receive()
# Update queue size metric after receiving
stream_stats = receive_stream.statistics()
update_vector_sync_queue_size(stream_stats.current_buffer_used)
# Process document
await process_document(doc_task, nc_client)
# Update queue size metric after processing
stream_stats = receive_stream.statistics()
update_vector_sync_queue_size(stream_stats.current_buffer_used)
except TimeoutError:
# No documents available, continue
# No documents available, update metric to show empty queue
stream_stats = receive_stream.statistics()
update_vector_sync_queue_size(stream_stats.current_buffer_used)
continue
except anyio.EndOfStream:
@@ -90,6 +105,8 @@ async def process_document(doc_task: DocumentTask, nc_client: NextcloudClient):
doc_task: Document task to process
nc_client: Authenticated Nextcloud client
"""
start_time = time.time()
logger.debug(
f"Processing {doc_task.doc_type}_{doc_task.doc_id} "
f"for {doc_task.user_id} ({doc_task.operation})"
@@ -105,58 +122,79 @@ async def process_document(doc_task: DocumentTask, nc_client: NextcloudClient):
"vector_sync.doc_operation": doc_task.operation,
},
):
qdrant_client = await get_qdrant_client()
settings = get_settings()
try:
qdrant_client = await get_qdrant_client()
settings = get_settings()
# Handle deletion
if doc_task.operation == "delete":
await qdrant_client.delete(
collection_name=settings.get_collection_name(),
points_selector=Filter(
must=[
FieldCondition(
key="user_id",
match=MatchValue(value=doc_task.user_id),
),
FieldCondition(
key="doc_id",
match=MatchValue(value=doc_task.doc_id),
),
FieldCondition(
key="doc_type",
match=MatchValue(value=doc_task.doc_type),
),
]
),
)
logger.info(
f"Deleted {doc_task.doc_type}_{doc_task.doc_id} for {doc_task.user_id}"
)
return
# Handle deletion
if doc_task.operation == "delete":
await qdrant_client.delete(
collection_name=settings.get_collection_name(),
points_selector=Filter(
must=[
FieldCondition(
key="user_id",
match=MatchValue(value=doc_task.user_id),
),
FieldCondition(
key="doc_id",
match=MatchValue(value=doc_task.doc_id),
),
FieldCondition(
key="doc_type",
match=MatchValue(value=doc_task.doc_type),
),
]
),
)
logger.info(
f"Deleted {doc_task.doc_type}_{doc_task.doc_id} for {doc_task.user_id}"
)
# Handle indexing with retry
max_retries = 3
retry_delay = 1.0
# Record successful deletion metrics
duration = time.time() - start_time
record_qdrant_operation("delete", "success")
record_vector_sync_processing(duration, "success")
return
for attempt in range(max_retries):
try:
await _index_document(doc_task, nc_client, qdrant_client)
return # Success
# Handle indexing with retry
max_retries = 3
retry_delay = 1.0
except (HTTPStatusError, Exception) as e:
if attempt < max_retries - 1:
logger.warning(
f"Retry {attempt + 1}/{max_retries} for "
f"{doc_task.doc_type}_{doc_task.doc_id}: {e}"
)
await anyio.sleep(retry_delay)
retry_delay *= 2 # Exponential backoff
else:
logger.error(
f"Failed to index {doc_task.doc_type}_{doc_task.doc_id} "
f"after {max_retries} retries: {e}"
)
raise
for attempt in range(max_retries):
try:
await _index_document(doc_task, nc_client, qdrant_client)
# Record successful processing metrics
duration = time.time() - start_time
record_qdrant_operation("upsert", "success")
record_vector_sync_processing(duration, "success")
return # Success
except (HTTPStatusError, Exception) as e:
if attempt < max_retries - 1:
logger.warning(
f"Retry {attempt + 1}/{max_retries} for "
f"{doc_task.doc_type}_{doc_task.doc_id}: {e}"
)
await anyio.sleep(retry_delay)
retry_delay *= 2 # Exponential backoff
else:
logger.error(
f"Failed to index {doc_task.doc_type}_{doc_task.doc_id} "
f"after {max_retries} retries: {e}"
)
# Record failed processing metrics
duration = time.time() - start_time
record_qdrant_operation("upsert", "error")
record_vector_sync_processing(duration, "error")
raise
except Exception:
# Catch any other unexpected errors
duration = time.time() - start_time
record_vector_sync_processing(duration, "error")
raise
async def _index_document(
+4
View File
@@ -13,6 +13,7 @@ from qdrant_client.models import FieldCondition, Filter, MatchValue
from nextcloud_mcp_server.client import NextcloudClient
from nextcloud_mcp_server.config import get_settings
from nextcloud_mcp_server.observability.metrics import record_vector_sync_scan
from nextcloud_mcp_server.observability.tracing import trace_operation
from nextcloud_mcp_server.vector.qdrant_client import get_qdrant_client
@@ -181,6 +182,9 @@ async def scan_user_documents(
]
logger.info(f"[SCAN-{scan_id}] Found {len(notes)} notes for {user_id}")
# Record documents scanned
record_vector_sync_scan(len(notes))
if initial_sync:
# Send everything on first sync
for note in notes:
+1 -1
View File
@@ -1,6 +1,6 @@
[project]
name = "nextcloud-mcp-server"
version = "0.31.1"
version = "0.34.2"
description = "Model Context Protocol (MCP) server for Nextcloud integration - enables AI assistants to interact with Nextcloud data"
authors = [
{name = "Chris Coutinho", email = "chris@coutinho.io"}
-307
View File
@@ -1,307 +0,0 @@
#!/usr/bin/env python3
"""Script to automatically add @require_scopes decorators to MCP tools.
This script parses server module files and adds appropriate scope decorators
based on the operation type (read vs write).
Usage:
python scripts/add_scope_decorators.py [--dry-run] [--file FILE]
"""
import argparse
import ast
import re
from pathlib import Path
from typing import List, Tuple
# Operation patterns for classification
READ_PATTERNS = [
r".*_get_.*",
r".*_get$",
r".*_list_.*",
r".*_list$",
r".*_search_.*",
r".*_search$",
r".*_read_.*",
r".*_read$",
r".*_find_.*",
r".*_find$",
r".*_fetch_.*",
r".*_fetch$",
r".*_retrieve_.*",
r".*_retrieve$",
]
WRITE_PATTERNS = [
r".*_create_.*",
r".*_create$",
r".*_update_.*",
r".*_update$",
r".*_delete_.*",
r".*_delete$",
r".*_append_.*",
r".*_append$",
r".*_modify_.*",
r".*_modify$",
r".*_set_.*",
r".*_set$",
r".*_add_.*",
r".*_add$",
r".*_remove_.*",
r".*_remove$",
r".*_edit_.*",
r".*_edit$",
r".*_move_.*",
r".*_move$",
r".*_copy_.*",
r".*_copy$",
r".*_upload_.*",
r".*_upload$",
r".*_download_.*",
r".*_download$",
r".*_share_.*",
r".*_share$",
r".*_unshare_.*",
r".*_unshare$",
r".*_bulk_.*", # Bulk operations are typically writes
]
def classify_operation(func_name: str) -> str | None:
"""Classify a function as read or write operation.
Args:
func_name: Function name to classify
Returns:
"nc:read", "nc:write", or None if cannot classify
"""
# Check write patterns first (more specific)
for pattern in WRITE_PATTERNS:
if re.match(pattern, func_name):
return "nc:write"
# Check read patterns
for pattern in READ_PATTERNS:
if re.match(pattern, func_name):
return "nc:read"
return None
def has_scope_decorator(decorators: List[ast.expr]) -> bool:
"""Check if function already has @require_scopes decorator."""
for decorator in decorators:
if isinstance(decorator, ast.Call):
if (
isinstance(decorator.func, ast.Name)
and decorator.func.id == "require_scopes"
):
return True
elif isinstance(decorator, ast.Name) and decorator.name == "require_scopes":
return True
return False
def has_mcp_tool_decorator(decorators: List[ast.expr]) -> bool:
"""Check if function has @mcp.tool() decorator."""
for decorator in decorators:
if isinstance(decorator, ast.Call):
if isinstance(decorator.func, ast.Attribute):
if decorator.func.attr == "tool":
return True
return False
def find_tools_needing_decorators(
file_path: Path, verbose: bool = False
) -> List[Tuple[str, int, str]]:
"""Find all tools that need scope decorators.
Returns:
List of (function_name, line_number, required_scope)
"""
with open(file_path) as f:
content = f.read()
try:
tree = ast.parse(content)
except SyntaxError as e:
print(f" ⚠️ Syntax error in {file_path}: {e}")
return []
tools_to_update = []
total_functions = 0
mcp_tools = 0
already_has_scope = 0
cannot_classify = 0
for node in ast.walk(tree):
if isinstance(node, ast.FunctionDef):
total_functions += 1
if verbose and node.decorator_list:
decorators_str = [
ast.unparse(d) if hasattr(ast, "unparse") else str(d)
for d in node.decorator_list
]
print(f" Function {node.name} has decorators: {decorators_str}")
# Check if it's an MCP tool
if not has_mcp_tool_decorator(node.decorator_list):
continue
mcp_tools += 1
# Check if it already has scope decorator
if has_scope_decorator(node.decorator_list):
already_has_scope += 1
continue
# Classify operation
scope = classify_operation(node.name)
if scope:
tools_to_update.append((node.name, node.lineno, scope))
else:
cannot_classify += 1
if verbose:
print(f" ⚠️ Cannot classify: {node.name}")
if verbose:
print(
f" Debug: total_functions={total_functions}, mcp_tools={mcp_tools}, already_has_scope={already_has_scope}, cannot_classify={cannot_classify}"
)
return tools_to_update
def add_decorator_to_file(
file_path: Path, dry_run: bool = False, verbose: bool = False
) -> int:
"""Add @require_scopes decorators to tools in a file.
Returns:
Number of decorators added
"""
tools = find_tools_needing_decorators(file_path, verbose=verbose)
if not tools:
return 0
print(f"\n📝 {file_path.relative_to(Path.cwd())}")
with open(file_path) as f:
lines = f.readlines()
# Check if require_scopes is already imported
has_import = False
import_line_idx = None
for i, line in enumerate(lines):
if "from nextcloud_mcp_server.auth import" in line and "require_scopes" in line:
has_import = True
break
elif "from nextcloud_mcp_server.auth import" in line:
import_line_idx = i
# Add import if needed
if not has_import:
if import_line_idx is not None:
# Add require_scopes to existing import
old_line = lines[import_line_idx]
if "(" in old_line:
# Multi-line import
print(
" ⚠️ Multi-line import detected, please add manually: from nextcloud_mcp_server.auth import require_scopes"
)
else:
# Single line import - add require_scopes
lines[import_line_idx] = (
old_line.rstrip().rstrip(")").rstrip() + ", require_scopes)\n"
)
print(" ✓ Added require_scopes to import")
else:
# No auth import exists, add new import
# Find first import line
for i, line in enumerate(lines):
if line.startswith("from nextcloud_mcp_server"):
lines.insert(
i, "from nextcloud_mcp_server.auth import require_scopes\n"
)
print(
" ✓ Added import: from nextcloud_mcp_server.auth import require_scopes"
)
break
# Add decorators to tools (in reverse order to preserve line numbers)
for func_name, line_num, scope in reversed(tools):
# Find the @mcp.tool() decorator line
for i in range(line_num - 1, max(0, line_num - 10), -1):
if "@mcp.tool()" in lines[i]:
# Get indentation from @mcp.tool() line
indent = len(lines[i]) - len(lines[i].lstrip())
decorator_line = " " * indent + f'@require_scopes("{scope}")\n'
lines.insert(i + 1, decorator_line)
print(f'{func_name}:{line_num} → @require_scopes("{scope}")')
break
if not dry_run:
with open(file_path, "w") as f:
f.writelines(lines)
print(" 💾 Saved changes")
else:
print(" 🔍 DRY RUN - no changes written")
return len(tools)
def main():
parser = argparse.ArgumentParser(
description="Add @require_scopes decorators to MCP tools"
)
parser.add_argument(
"--dry-run",
action="store_true",
help="Show what would be changed without modifying files",
)
parser.add_argument(
"--file",
type=Path,
help="Process a single file instead of all server modules",
)
parser.add_argument(
"--verbose",
"-v",
action="store_true",
help="Show debug information",
)
args = parser.parse_args()
server_dir = Path(__file__).parent.parent / "nextcloud_mcp_server" / "server"
if args.file:
files = [args.file]
else:
files = sorted(server_dir.glob("*.py"))
files = [f for f in files if f.name != "__init__.py"]
print("🔍 Scanning for tools needing scope decorators...")
print(
f" {'DRY RUN MODE - No changes will be made' if args.dry_run else 'LIVE MODE - Files will be modified'}"
)
total_added = 0
for file_path in files:
added = add_decorator_to_file(
file_path, dry_run=args.dry_run, verbose=args.verbose
)
total_added += added
print(f"\n{'📊 Summary (dry run)' if args.dry_run else '✅ Complete'}")
print(f" Total decorators added: {total_added}")
if args.dry_run:
print("\n💡 Run without --dry-run to apply changes")
if __name__ == "__main__":
main()
-232
View File
@@ -1,232 +0,0 @@
#!/usr/bin/env python3
"""Simpler script to add @require_scopes decorators using regex.
This script uses regex patterns to find @mcp.tool() decorators and adds
the appropriate @require_scopes decorator based on function name patterns.
Usage:
python scripts/add_scope_decorators_simple.py [--dry-run]
"""
import argparse
import re
from pathlib import Path
# Operation patterns for classification
READ_KEYWORDS = [
"get",
"list",
"search",
"read",
"find",
"fetch",
"retrieve",
"upcoming",
]
WRITE_KEYWORDS = [
"create",
"update",
"delete",
"append",
"modify",
"set",
"add",
"remove",
"edit",
"move",
"copy",
"upload",
"download",
"share",
"unshare",
"bulk",
"manage",
"import",
"reindex",
"archive",
"unarchive",
"reorder",
"assign",
"unassign",
"insert",
"write",
]
def classify_function(func_name: str) -> str | None:
"""Classify a function name as read or write operation."""
func_lower = func_name.lower()
# Check write keywords first (more specific)
for keyword in WRITE_KEYWORDS:
if f"_{keyword}_" in func_lower or func_lower.endswith(f"_{keyword}"):
return "nc:write"
# Check read keywords
for keyword in READ_KEYWORDS:
if f"_{keyword}_" in func_lower or func_lower.endswith(f"_{keyword}"):
return "nc:read"
return None
def process_file(file_path: Path, dry_run: bool = False) -> int:
"""Process a single file to add @require_scopes decorators.
Returns:
Number of decorators added
"""
with open(file_path) as f:
lines = f.readlines()
# Check if require_scopes is already imported
has_import = False
import_line_idx = None
for i, line in enumerate(lines):
if "from nextcloud_mcp_server.auth import" in line:
if "require_scopes" in line:
has_import = True
else:
import_line_idx = i
modified = False
decorators_added = 0
# Find all @mcp.tool() decorators
i = 0
while i < len(lines):
line = lines[i]
# Look for @mcp.tool() decorator
if re.match(r"\s*@mcp\.tool\(\)", line):
# Check if next line already has @require_scopes
if i + 1 < len(lines) and "@require_scopes" in lines[i + 1]:
i += 1
continue
# Find the function definition (should be on next line or after other decorators)
func_line_idx = i + 1
while func_line_idx < len(lines) and not lines[
func_line_idx
].strip().startswith("async def"):
func_line_idx += 1
if func_line_idx >= len(lines):
i += 1
continue
# Extract function name
func_match = re.match(r"\s*async def (\w+)\(", lines[func_line_idx])
if not func_match:
i += 1
continue
func_name = func_match.group(1)
scope = classify_function(func_name)
if scope:
# Get indentation from @mcp.tool() line
indent = len(line) - len(line.lstrip())
decorator_line = " " * indent + f'@require_scopes("{scope}")\n'
# Insert after @mcp.tool()
lines.insert(i + 1, decorator_line)
decorators_added += 1
modified = True
print(f'{func_name} → @require_scopes("{scope}")')
else:
print(f" ⚠️ Cannot classify: {func_name}")
i += 1
# Add import if needed and decorators were added
if decorators_added > 0 and not has_import:
if import_line_idx is not None:
# Add to existing import
old_line = lines[import_line_idx]
if old_line.rstrip().endswith(")"):
lines[import_line_idx] = old_line.rstrip()[:-1] + ", require_scopes)\n"
else:
lines[import_line_idx] = old_line.rstrip() + ", require_scopes\n"
print(" ✓ Added require_scopes to existing import")
modified = True
else:
# No auth import exists, add new import after last 'from nextcloud_mcp_server' import
last_nc_import_idx = None
for i, line in enumerate(lines):
if line.startswith("from nextcloud_mcp_server"):
last_nc_import_idx = i
if last_nc_import_idx is not None:
lines.insert(
last_nc_import_idx + 1,
"from nextcloud_mcp_server.auth import require_scopes\n",
)
print(
" ✓ Added new import: from nextcloud_mcp_server.auth import require_scopes"
)
modified = True
else:
print(" ⚠️ Could not find place to add require_scopes import")
# Write changes
if modified and not dry_run:
with open(file_path, "w") as f:
f.writelines(lines)
print(f" 💾 Saved changes to {file_path.name}")
elif dry_run and decorators_added > 0:
print(f" 🔍 DRY RUN - would add {decorators_added} decorators")
return decorators_added
def main():
parser = argparse.ArgumentParser(
description="Add @require_scopes decorators to MCP tools"
)
parser.add_argument(
"--dry-run",
action="store_true",
help="Show what would be changed without modifying files",
)
parser.add_argument(
"--file",
type=Path,
help="Process a single file instead of all server modules",
)
args = parser.parse_args()
server_dir = Path(__file__).parent.parent / "nextcloud_mcp_server" / "server"
if args.file:
files = [args.file]
else:
files = sorted(server_dir.glob("*.py"))
files = [f for f in files if f.name != "__init__.py"]
print("🔍 Scanning for tools needing scope decorators...")
print(
f" {'DRY RUN MODE - No changes will be made' if args.dry_run else 'LIVE MODE - Files will be modified'}"
)
total_added = 0
for file_path in files:
file_path = file_path.resolve() # Convert to absolute path
try:
display_path = file_path.relative_to(Path.cwd())
except ValueError:
display_path = file_path.name
print(f"\n📝 {display_path}")
added = process_file(file_path, dry_run=args.dry_run)
total_added += added
print(f"\n{'📊 Summary (dry run)' if args.dry_run else '✅ Complete'}")
print(f" Total decorators added: {total_added}")
if args.dry_run and total_added > 0:
print("\n💡 Run without --dry-run to apply changes")
if __name__ == "__main__":
main()
-90
View File
@@ -1,90 +0,0 @@
#!/bin/bash
set -e
echo "=== Testing Separate Clients Architecture ==="
echo ""
# Check both clients exist in Keycloak
echo "1. Verifying Keycloak clients..."
docker compose exec -T app curl -s http://keycloak:8080/realms/nextcloud-mcp/.well-known/openid-configuration > /dev/null && echo "✓ Keycloak realm available"
# Check user_oidc provider configuration
echo ""
echo "2. Checking user_oidc provider..."
PROVIDER_INFO=$(docker compose exec -T app php occ user_oidc:provider keycloak)
echo "$PROVIDER_INFO" | grep -q "nextcloud" && echo "✓ user_oidc configured with 'nextcloud' client"
# Get token from nextcloud-mcp-server client
echo ""
echo "3. Getting token from 'nextcloud-mcp-server' client..."
TOKEN=$(curl -s -X POST "http://localhost:8888/realms/nextcloud-mcp/protocol/openid-connect/token" \
-d "grant_type=password" \
-d "client_id=nextcloud-mcp-server" \
-d "client_secret=mcp-secret-change-in-production" \
-d "username=admin" \
-d "password=admin" \
-d "scope=openid profile email offline_access" | jq -r '.access_token')
if [ "$TOKEN" = "null" ] || [ -z "$TOKEN" ]; then
echo "✗ Failed to get token"
exit 1
fi
echo "✓ Got token from nextcloud-mcp-server client"
# Check token claims
echo ""
echo "4. Inspecting token claims..."
CLAIMS=$(echo "$TOKEN" | cut -d'.' -f2 | base64 -d 2>/dev/null | jq '{aud, azp, iss, preferred_username}')
echo "$CLAIMS"
AUD=$(echo "$CLAIMS" | jq -r '.aud')
AZP=$(echo "$CLAIMS" | jq -r '.azp')
echo ""
echo "Architecture validation:"
if [ "$AUD" = "nextcloud" ]; then
echo " ✓ aud='nextcloud' - Token intended for Nextcloud resource server"
else
echo " ✗ FAILED: aud='$AUD', expected 'nextcloud'"
exit 1
fi
if [ "$AZP" = "nextcloud-mcp-server" ]; then
echo " ✓ azp='nextcloud-mcp-server' - Token requested by MCP Server client"
else
echo " ✗ FAILED: azp='$AZP', expected 'nextcloud-mcp-server'"
exit 1
fi
# Test with Nextcloud API
echo ""
echo "5. Testing token with Nextcloud API..."
HTTP_CODE=$(curl -s -w "%{http_code}" -o /tmp/nc_response.json \
-H "Authorization: Bearer $TOKEN" \
"http://localhost:8080/ocs/v2.php/cloud/capabilities?format=json")
echo "HTTP Status: $HTTP_CODE"
if [ "$HTTP_CODE" = "200" ]; then
echo "✓ Token validated successfully!"
echo ""
echo "===================================================================="
echo "SUCCESS: Separate Clients Architecture Working!"
echo "===================================================================="
echo ""
echo "Summary:"
echo " - MCP Server client: 'nextcloud-mcp-server' (requests tokens)"
echo " - Resource server: 'nextcloud' (validates tokens via user_oidc)"
echo " - Token audience: 'nextcloud' (proper resource targeting)"
echo " - Token azp: 'nextcloud-mcp-server' (identifies requester)"
echo ""
echo "This architecture supports:"
echo " - Future multi-resource tokens: aud=['nextcloud', 'other-service']"
echo " - Clear separation of OAuth client vs resource server"
echo " - RFC 8707 Resource Indicators compliance"
else
echo "✗ Token validation failed"
cat /tmp/nc_response.json
exit 1
fi
Generated
+1 -1
View File
@@ -1053,7 +1053,7 @@ wheels = [
[[package]]
name = "nextcloud-mcp-server"
version = "0.31.1"
version = "0.34.2"
source = { editable = "." }
dependencies = [
{ name = "aiosqlite" },