Compare commits
6 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Chris Coutinho
|
4e43d15153 | ||
|
github-actions[bot]
|
15951c38fa | ||
|
Chris Coutinho
|
2de0590839 | ||
|
Chris Coutinho
|
4ea5ed72d4 | ||
|
Chris Coutinho
|
d1829fbbd6 | ||
|
renovate-bot-cbcoutinho[bot]
|
8332542959 |
@@ -1,3 +1,9 @@
|
||||
## v0.33.0 (2025-11-13)
|
||||
|
||||
### Feat
|
||||
|
||||
- Add Grafana dashboard and vector sync metric instrumentation
|
||||
|
||||
## v0.32.1 (2025-11-12)
|
||||
|
||||
### Fix
|
||||
|
||||
+1
-1
@@ -1,4 +1,4 @@
|
||||
FROM ghcr.io/astral-sh/uv:0.9.8-python3.11-alpine@sha256:6c842c49ad032f46b62f32a7e7779f45f12671a8e0d82ea24c766ab62d58b396
|
||||
FROM ghcr.io/astral-sh/uv:0.9.9-python3.11-alpine@sha256:0faa7934fac1db7f5056f159c1224d144bab864fd2677a4066d25a686ae32edd
|
||||
|
||||
# Install dependencies
|
||||
# 1. git (required for caldav dependency from git)
|
||||
|
||||
@@ -2,8 +2,8 @@ apiVersion: v2
|
||||
name: nextcloud-mcp-server
|
||||
description: A Helm chart for Nextcloud MCP Server - enables AI assistants to interact with Nextcloud
|
||||
type: application
|
||||
version: 0.32.1
|
||||
appVersion: "0.32.1"
|
||||
version: 0.33.0
|
||||
appVersion: "0.33.0"
|
||||
keywords:
|
||||
- nextcloud
|
||||
- mcp
|
||||
@@ -21,6 +21,10 @@ home: https://github.com/cbcoutinho/nextcloud-mcp-server
|
||||
sources:
|
||||
- https://github.com/cbcoutinho/nextcloud-mcp-server
|
||||
icon: https://raw.githubusercontent.com/nextcloud/server/master/core/img/logo/logo.svg
|
||||
annotations:
|
||||
# Grafana dashboard support
|
||||
grafana_dashboard: "true"
|
||||
grafana_dashboard_folder: "Nextcloud MCP"
|
||||
dependencies:
|
||||
- name: qdrant
|
||||
version: "1.15.5"
|
||||
|
||||
@@ -280,6 +280,72 @@ Use OpenAI or any OpenAI-compatible API instead of Ollama.
|
||||
| `openai.secretKey` | Key in secret containing API key | `api-key` |
|
||||
| `openai.baseUrl` | Custom API endpoint (optional) | `""` |
|
||||
|
||||
#### Observability & Monitoring
|
||||
|
||||
The chart includes comprehensive observability features including Prometheus metrics, OpenTelemetry tracing, and Grafana dashboards.
|
||||
|
||||
**Metrics Configuration:**
|
||||
|
||||
| Parameter | Description | Default |
|
||||
|-----------|-------------|---------|
|
||||
| `observability.metrics.enabled` | Enable Prometheus metrics | `true` |
|
||||
| `observability.metrics.port` | Metrics port | `9090` |
|
||||
| `observability.metrics.path` | Metrics endpoint path | `/metrics` |
|
||||
|
||||
**Tracing Configuration:**
|
||||
|
||||
| Parameter | Description | Default |
|
||||
|-----------|-------------|---------|
|
||||
| `observability.tracing.enabled` | Enable OpenTelemetry tracing | `false` |
|
||||
| `observability.tracing.endpoint` | OTLP collector endpoint | `""` |
|
||||
| `observability.tracing.serviceName` | Service name in traces | `nextcloud-mcp-server` |
|
||||
| `observability.tracing.samplingRate` | Trace sampling rate (0.0-1.0) | `1.0` |
|
||||
|
||||
**Logging Configuration:**
|
||||
|
||||
| Parameter | Description | Default |
|
||||
|-----------|-------------|---------|
|
||||
| `observability.logging.format` | Log format (json or text) | `json` |
|
||||
| `observability.logging.level` | Log level | `INFO` |
|
||||
| `observability.logging.includeTraceContext` | Include trace IDs in logs | `true` |
|
||||
|
||||
**ServiceMonitor (Prometheus Operator):**
|
||||
|
||||
| Parameter | Description | Default |
|
||||
|-----------|-------------|---------|
|
||||
| `serviceMonitor.enabled` | Create ServiceMonitor resource | `false` |
|
||||
| `serviceMonitor.interval` | Scrape interval | `30s` |
|
||||
| `serviceMonitor.scrapeTimeout` | Scrape timeout | `10s` |
|
||||
| `serviceMonitor.labels` | Additional labels for ServiceMonitor | `{}` |
|
||||
|
||||
**PrometheusRule (Prometheus Operator):**
|
||||
|
||||
| Parameter | Description | Default |
|
||||
|-----------|-------------|---------|
|
||||
| `prometheusRule.enabled` | Create PrometheusRule with alert rules | `false` |
|
||||
| `prometheusRule.labels` | Additional labels for PrometheusRule | `{}` |
|
||||
|
||||
**Grafana Dashboards:**
|
||||
|
||||
| Parameter | Description | Default |
|
||||
|-----------|-------------|---------|
|
||||
| `dashboards.enabled` | Enable automatic dashboard provisioning | `false` |
|
||||
| `dashboards.grafanaFolder` | Grafana folder name for dashboards | `Nextcloud MCP` |
|
||||
| `dashboards.labels` | Additional labels for dashboard ConfigMap | `{}` |
|
||||
| `dashboards.annotations` | Additional annotations for dashboard ConfigMap | `{}` |
|
||||
|
||||
When `dashboards.enabled` is `true`, a ConfigMap with the Grafana dashboard is created with the `grafana_dashboard: "1"` label. This enables automatic discovery by Grafana sidecar containers (commonly used with kube-prometheus-stack).
|
||||
|
||||
The dashboard provides comprehensive monitoring including:
|
||||
- HTTP request metrics (RED pattern: Rate, Errors, Duration)
|
||||
- MCP tool performance and errors
|
||||
- Nextcloud API performance by app (notes, calendar, contacts, etc.)
|
||||
- OAuth token operations and cache hit rates
|
||||
- External dependency health (Nextcloud, Qdrant, Keycloak, Unstructured API)
|
||||
- Vector sync processing pipeline (when enabled)
|
||||
|
||||
For manual import or more details, see `charts/nextcloud-mcp-server/dashboards/README.md`.
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Basic Auth with Ingress
|
||||
|
||||
@@ -6,14 +6,57 @@ This directory contains example Grafana dashboards for monitoring the Nextcloud
|
||||
|
||||
### nextcloud-mcp-server.json
|
||||
|
||||
Comprehensive dashboard with the following panels:
|
||||
All-in-one Operations Dashboard with comprehensive monitoring across all system components.
|
||||
|
||||
- **Request Rate**: HTTP requests per second by method and endpoint
|
||||
- **Error Rate**: Percentage of 5xx errors
|
||||
- **Request Latency**: P50 and P95 latency by endpoint
|
||||
- **Top MCP Tools**: Most frequently called tools
|
||||
- **Nextcloud API Latency**: API call latency by app (notes, calendar, etc.)
|
||||
- **Vector Sync Queue**: Queue size for background document processing
|
||||
#### Overview Row
|
||||
High-level metrics for quick health assessment:
|
||||
- **Request Rate** (stat): Total requests per second
|
||||
- **Error Rate** (stat): Percentage of 5xx errors with color thresholds
|
||||
- **P95 Latency** (stat): 95th percentile request latency
|
||||
- **Active Requests** (stat): Current in-flight requests
|
||||
|
||||
#### HTTP Metrics (RED Pattern)
|
||||
Core request/error/duration metrics:
|
||||
- **Request Rate by Endpoint** (timeseries): RPS breakdown by endpoint
|
||||
- **Error Rate by Status Code** (timeseries): Error rates for 4xx/5xx codes
|
||||
- **Latency Percentiles** (timeseries): P50, P95, P99 latency trends
|
||||
- **Status Code Distribution** (piechart): Percentage breakdown of all status codes
|
||||
|
||||
#### MCP Tools Row
|
||||
MCP-specific tool performance:
|
||||
- **Top Tools by Call Volume** (bargauge): Top 10 most-called tools
|
||||
- **Tool Error Rate** (timeseries): Error rates per tool
|
||||
- **Tool Execution Duration** (timeseries): P95 latency by tool
|
||||
|
||||
#### Nextcloud API Row
|
||||
Backend API performance metrics:
|
||||
- **API Calls by App** (timeseries): Request rate per Nextcloud app (notes, calendar, contacts, etc.)
|
||||
- **API Latency by App** (timeseries): P95 latency per app
|
||||
- **API Retries by Reason** (timeseries): Retry patterns (429, timeout, connection errors)
|
||||
- **API Error Rate** (stat): Overall API error percentage
|
||||
|
||||
#### OAuth & Authentication Row
|
||||
OAuth token operations and caching:
|
||||
- **Token Validations** (timeseries): Success/failure rates for token validation
|
||||
- **Token Exchange Operations** (timeseries): RFC 8693 token exchange operations
|
||||
- **Token Cache Hit Rate** (stat): Percentage of cache hits (color-coded: red<50%, yellow<80%, green≥80%)
|
||||
- **Refresh Token Operations** (timeseries): Refresh token storage operations by type
|
||||
|
||||
#### Dependencies & Health Row
|
||||
External dependency status monitoring:
|
||||
- **Nextcloud Health** (stat): UP/DOWN status with color coding
|
||||
- **Qdrant Health** (stat): Vector database health status
|
||||
- **Keycloak Health** (stat): Identity provider health status
|
||||
- **Unstructured API Health** (stat): Document processing API status
|
||||
- **Health Check Duration** (timeseries): Health check latency by dependency
|
||||
- **Database Operation Latency** (timeseries): P95 latency for DB operations (SQLite, Qdrant)
|
||||
|
||||
#### Vector Sync Row (when enabled)
|
||||
Document processing pipeline metrics:
|
||||
- **Documents Processed Rate** (timeseries): Processing throughput by status (success/failure)
|
||||
- **Processing Queue Depth** (gauge): Current queue size with thresholds (yellow>50, red>100)
|
||||
- **Qdrant Operations** (timeseries): Vector database operations by type
|
||||
- **Document Processing Duration** (timeseries): P95 processing latency
|
||||
|
||||
## Importing to Grafana
|
||||
|
||||
@@ -25,49 +68,77 @@ Comprehensive dashboard with the following panels:
|
||||
4. Select your Prometheus data source
|
||||
5. Click "Import"
|
||||
|
||||
### Automated Import (Kubernetes)
|
||||
### Automated Import (Helm Chart)
|
||||
|
||||
If using the Grafana Operator or kube-prometheus-stack, you can create a ConfigMap:
|
||||
The Helm chart now supports automatic dashboard provisioning via Grafana sidecar pattern.
|
||||
|
||||
#### Option 1: Using Helm Chart (Recommended)
|
||||
|
||||
Enable dashboard provisioning in your Helm values:
|
||||
|
||||
```yaml
|
||||
# values.yaml for nextcloud-mcp-server chart
|
||||
dashboards:
|
||||
enabled: true
|
||||
grafanaFolder: "Nextcloud MCP" # Folder name in Grafana
|
||||
labels: {} # Additional labels if needed
|
||||
```
|
||||
|
||||
Then deploy or upgrade:
|
||||
|
||||
```bash
|
||||
kubectl create configmap nextcloud-mcp-dashboards \
|
||||
helm upgrade --install nextcloud-mcp nextcloud-mcp-server \
|
||||
--set dashboards.enabled=true
|
||||
```
|
||||
|
||||
The dashboard will be automatically imported by Grafana if the sidecar is configured
|
||||
to watch for ConfigMaps with label `grafana_dashboard: "1"`.
|
||||
|
||||
#### Option 2: Using kube-prometheus-stack
|
||||
|
||||
If using kube-prometheus-stack with Grafana sidecar enabled, the dashboard will be
|
||||
automatically discovered and imported. Ensure your Grafana deployment has:
|
||||
|
||||
```yaml
|
||||
# kube-prometheus-stack values
|
||||
grafana:
|
||||
sidecar:
|
||||
dashboards:
|
||||
enabled: true
|
||||
label: grafana_dashboard
|
||||
folder: /tmp/dashboards
|
||||
provider:
|
||||
foldersFromFilesStructure: true
|
||||
```
|
||||
|
||||
#### Option 3: Manual ConfigMap Creation
|
||||
|
||||
For other Grafana setups, create a ConfigMap manually:
|
||||
|
||||
```bash
|
||||
kubectl create configmap nextcloud-mcp-dashboard \
|
||||
--from-file=nextcloud-mcp-server.json \
|
||||
-n monitoring
|
||||
|
||||
# Add label for Grafana sidecar to discover
|
||||
kubectl label configmap nextcloud-mcp-dashboards \
|
||||
# Add sidecar discovery label
|
||||
kubectl label configmap nextcloud-mcp-dashboard \
|
||||
grafana_dashboard=1 \
|
||||
-n monitoring
|
||||
```
|
||||
|
||||
Or add to your Helm values:
|
||||
|
||||
```yaml
|
||||
# values.yaml for kube-prometheus-stack
|
||||
grafana:
|
||||
dashboardProviders:
|
||||
dashboardproviders.yaml:
|
||||
apiVersion: 1
|
||||
providers:
|
||||
- name: 'nextcloud-mcp'
|
||||
orgId: 1
|
||||
folder: 'Nextcloud MCP'
|
||||
type: file
|
||||
disableDeletion: false
|
||||
editable: true
|
||||
options:
|
||||
path: /var/lib/grafana/dashboards/nextcloud-mcp
|
||||
|
||||
dashboardsConfigMaps:
|
||||
nextcloud-mcp: nextcloud-mcp-dashboards
|
||||
# Add folder annotation (annotations support spaces, unlike labels)
|
||||
kubectl annotate configmap nextcloud-mcp-dashboard \
|
||||
grafana_folder="Nextcloud MCP" \
|
||||
-n monitoring
|
||||
```
|
||||
|
||||
## Dashboard Variables
|
||||
|
||||
The dashboard includes two variables:
|
||||
The dashboard includes four template variables for dynamic filtering:
|
||||
|
||||
- **Data Source**: Select your Prometheus data source
|
||||
- **Namespace**: Filter metrics by Kubernetes namespace
|
||||
- **datasource**: Select your Prometheus data source
|
||||
- **namespace**: Filter metrics by Kubernetes namespace (supports "All")
|
||||
- **pod**: Filter by specific pod(s) - multi-select enabled (supports "All")
|
||||
- **interval**: Query interval for rate calculations (1m, 5m, 10m, 30m, 1h - default: 5m)
|
||||
|
||||
## Customization
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -96,6 +96,30 @@ Your Nextcloud MCP Server has been deployed in {{ .Values.auth.mode }} authentic
|
||||
kubectl --namespace {{ .Release.Namespace }} exec -it deploy/{{ include "nextcloud-mcp-server.fullname" . }} -- curl -s http://localhost:{{ include "nextcloud-mcp-server.port" . }}/user/page | grep "Vector Sync"
|
||||
{{- end }}
|
||||
|
||||
{{- if .Values.dashboards.enabled }}
|
||||
|
||||
6. Grafana Dashboards:
|
||||
- Dashboard provisioning: Enabled
|
||||
- ConfigMap: {{ include "nextcloud-mcp-server.fullname" . }}-dashboard
|
||||
- Grafana Folder: {{ .Values.dashboards.grafanaFolder }}
|
||||
|
||||
The dashboard will be automatically imported by Grafana if the sidecar is configured
|
||||
to watch for ConfigMaps with label "grafana_dashboard: 1".
|
||||
|
||||
To manually import the dashboard:
|
||||
kubectl --namespace {{ .Release.Namespace }} get configmap {{ include "nextcloud-mcp-server.fullname" . }}-dashboard -o jsonpath='{.data.nextcloud-mcp-server\.json}' | jq . > dashboard.json
|
||||
|
||||
Then import dashboard.json via Grafana UI (Dashboards → Import).
|
||||
{{- else }}
|
||||
|
||||
6. Grafana Dashboards:
|
||||
- Dashboard provisioning: Disabled
|
||||
- To enable automatic dashboard provisioning, set: dashboards.enabled=true
|
||||
|
||||
Manual import option:
|
||||
The dashboard JSON is available in the chart at charts/nextcloud-mcp-server/dashboards/nextcloud-mcp-server.json
|
||||
{{- end }}
|
||||
|
||||
For more information and documentation:
|
||||
- GitHub: https://github.com/cbcoutinho/nextcloud-mcp-server
|
||||
- Documentation: https://github.com/cbcoutinho/nextcloud-mcp-server#readme
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
{{- if .Values.dashboards.enabled }}
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: {{ include "nextcloud-mcp-server.fullname" . }}-dashboard
|
||||
namespace: {{ .Release.Namespace }}
|
||||
labels:
|
||||
{{- include "nextcloud-mcp-server.labels" . | nindent 4 }}
|
||||
{{- with .Values.dashboards.labels }}
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
# Grafana sidecar discovery label
|
||||
grafana_dashboard: "1"
|
||||
annotations:
|
||||
{{- with .Values.dashboards.annotations }}
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
# Grafana folder name (annotations support spaces, unlike labels)
|
||||
{{- if .Values.dashboards.grafanaFolder }}
|
||||
grafana_folder: {{ .Values.dashboards.grafanaFolder | quote }}
|
||||
{{- end }}
|
||||
data:
|
||||
nextcloud-mcp-server.json: |-
|
||||
{{ .Files.Get "dashboards/nextcloud-mcp-server.json" | indent 4 }}
|
||||
{{- end }}
|
||||
@@ -205,6 +205,20 @@ prometheusRule:
|
||||
# Additional labels for PrometheusRule (e.g., for Prometheus selector)
|
||||
# Example: { prometheus: kube-prometheus }
|
||||
|
||||
# Grafana dashboards (requires Grafana with sidecar enabled)
|
||||
dashboards:
|
||||
# Enable automatic dashboard provisioning via ConfigMap
|
||||
enabled: false
|
||||
# Grafana folder name where dashboards will be imported
|
||||
# The grafana-sidecar looks for ConfigMaps with label "grafana_dashboard: 1"
|
||||
# and reads the folder name from annotation "grafana_folder" (supports spaces)
|
||||
grafanaFolder: "Nextcloud MCP"
|
||||
# Additional labels for dashboard ConfigMap
|
||||
# These will be added alongside the required "grafana_dashboard: 1" label
|
||||
labels: {}
|
||||
# Additional annotations for dashboard ConfigMap
|
||||
annotations: {}
|
||||
|
||||
service:
|
||||
type: ClusterIP
|
||||
port: 8000
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -352,3 +352,46 @@ def record_dependency_check(dependency: str, duration: float) -> None:
|
||||
duration: Check duration in seconds
|
||||
"""
|
||||
dependency_check_duration_seconds.labels(dependency=dependency).observe(duration)
|
||||
|
||||
|
||||
def record_vector_sync_scan(documents_found: int) -> None:
|
||||
"""
|
||||
Record documents scanned during vector sync.
|
||||
|
||||
Args:
|
||||
documents_found: Number of documents discovered in scan
|
||||
"""
|
||||
vector_sync_documents_scanned_total.inc(documents_found)
|
||||
|
||||
|
||||
def record_vector_sync_processing(duration: float, status: str = "success") -> None:
|
||||
"""
|
||||
Record document processing with duration and status.
|
||||
|
||||
Args:
|
||||
duration: Processing duration in seconds
|
||||
status: "success" or "error"
|
||||
"""
|
||||
vector_sync_documents_processed_total.labels(status=status).inc()
|
||||
vector_sync_processing_duration_seconds.observe(duration)
|
||||
|
||||
|
||||
def record_qdrant_operation(operation: str, status: str = "success") -> None:
|
||||
"""
|
||||
Record Qdrant vector database operation.
|
||||
|
||||
Args:
|
||||
operation: Operation type ("upsert", "search", "delete")
|
||||
status: "success" or "error"
|
||||
"""
|
||||
qdrant_operations_total.labels(operation=operation, status=status).inc()
|
||||
|
||||
|
||||
def update_vector_sync_queue_size(size: int) -> None:
|
||||
"""
|
||||
Update vector sync queue size gauge.
|
||||
|
||||
Args:
|
||||
size: Current queue size
|
||||
"""
|
||||
vector_sync_queue_size.set(size)
|
||||
|
||||
@@ -21,6 +21,7 @@ from nextcloud_mcp_server.models.semantic import (
|
||||
SemanticSearchResult,
|
||||
VectorSyncStatusResponse,
|
||||
)
|
||||
from nextcloud_mcp_server.observability.metrics import record_qdrant_operation
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -85,26 +86,33 @@ def configure_semantic_tools(mcp: FastMCP):
|
||||
# Note: Currently only searching notes (doc_type="note")
|
||||
# Future: Remove doc_type filter to search all apps
|
||||
qdrant_client = await get_qdrant_client()
|
||||
search_response = await qdrant_client.query_points(
|
||||
collection_name=settings.get_collection_name(),
|
||||
query=query_embedding,
|
||||
query_filter=Filter(
|
||||
must=[
|
||||
FieldCondition(
|
||||
key="user_id",
|
||||
match=MatchValue(value=username),
|
||||
),
|
||||
FieldCondition(
|
||||
key="doc_type",
|
||||
match=MatchValue(value="note"),
|
||||
),
|
||||
]
|
||||
),
|
||||
limit=limit * 2, # Get extra for filtering
|
||||
score_threshold=score_threshold,
|
||||
with_payload=True,
|
||||
with_vectors=False, # Don't return vectors to save bandwidth
|
||||
)
|
||||
try:
|
||||
search_response = await qdrant_client.query_points(
|
||||
collection_name=settings.get_collection_name(),
|
||||
query=query_embedding,
|
||||
query_filter=Filter(
|
||||
must=[
|
||||
FieldCondition(
|
||||
key="user_id",
|
||||
match=MatchValue(value=username),
|
||||
),
|
||||
FieldCondition(
|
||||
key="doc_type",
|
||||
match=MatchValue(value="note"),
|
||||
),
|
||||
]
|
||||
),
|
||||
limit=limit * 2, # Get extra for filtering
|
||||
score_threshold=score_threshold,
|
||||
with_payload=True,
|
||||
with_vectors=False, # Don't return vectors to save bandwidth
|
||||
)
|
||||
# Record successful search operation
|
||||
record_qdrant_operation("search", "success")
|
||||
except Exception:
|
||||
# Record failed search operation
|
||||
record_qdrant_operation("search", "error")
|
||||
raise
|
||||
|
||||
logger.info(
|
||||
f"Qdrant returned {len(search_response.points)} results "
|
||||
@@ -331,21 +339,71 @@ def configure_semantic_tools(mcp: FastMCP):
|
||||
success=True,
|
||||
)
|
||||
|
||||
# 4. Construct context from retrieved documents
|
||||
# 4. Fetch full content for notes to provide complete context to LLM
|
||||
# Filter out inaccessible notes (deleted or permissions changed)
|
||||
client = await get_client(ctx)
|
||||
accessible_results = []
|
||||
full_contents = [] # Full content for accessible notes
|
||||
|
||||
for result in search_response.results:
|
||||
if result.doc_type == "note":
|
||||
try:
|
||||
note = await client.notes.get_note(result.id)
|
||||
# Note is accessible, store full content
|
||||
accessible_results.append(result)
|
||||
full_contents.append(note.get("content", ""))
|
||||
logger.debug(
|
||||
f"Fetched full content for note {result.id} "
|
||||
f"(length: {len(full_contents[-1])} chars)"
|
||||
)
|
||||
except Exception as e:
|
||||
# Note might have been deleted or permissions changed
|
||||
# Filter it out to avoid corrupting LLM with inaccessible data
|
||||
logger.warning(
|
||||
f"Failed to fetch full content for note {result.id}: {e}. "
|
||||
f"Excluding from results."
|
||||
)
|
||||
else:
|
||||
# Non-note document types (future: calendar, deck, files)
|
||||
# For now, keep them with excerpts
|
||||
accessible_results.append(result)
|
||||
full_contents.append(None)
|
||||
|
||||
# Check if we filtered out all results
|
||||
if not accessible_results:
|
||||
logger.warning(f"All search results became inaccessible for query: {query}")
|
||||
return SamplingSearchResponse(
|
||||
query=query,
|
||||
generated_answer="All matching documents are no longer accessible.",
|
||||
sources=[],
|
||||
total_found=0,
|
||||
search_method="semantic_sampling",
|
||||
success=True,
|
||||
)
|
||||
|
||||
# 5. Construct context from accessible documents with full content
|
||||
context_parts = []
|
||||
for idx, result in enumerate(search_response.results, 1):
|
||||
for idx, (result, content) in enumerate(
|
||||
zip(accessible_results, full_contents), 1
|
||||
):
|
||||
# Use full content if available (notes), otherwise use excerpt
|
||||
if content is not None:
|
||||
content_field = f"Content: {content}"
|
||||
else:
|
||||
content_field = f"Excerpt: {result.excerpt}"
|
||||
|
||||
context_parts.append(
|
||||
f"[Document {idx}]\n"
|
||||
f"Type: {result.doc_type}\n"
|
||||
f"Title: {result.title}\n"
|
||||
f"Category: {result.category}\n"
|
||||
f"Excerpt: {result.excerpt}\n"
|
||||
f"{content_field}\n"
|
||||
f"Relevance Score: {result.score:.2f}\n"
|
||||
)
|
||||
|
||||
context = "\n".join(context_parts)
|
||||
|
||||
# 5. Construct prompt - reuse user's query, add context and instructions
|
||||
# 6. Construct prompt - reuse user's query, add context and instructions
|
||||
prompt = (
|
||||
f"{query}\n\n"
|
||||
f"Here are relevant documents from Nextcloud (notes, calendar events, deck cards, files, contacts):\n\n"
|
||||
@@ -401,8 +459,8 @@ def configure_semantic_tools(mcp: FastMCP):
|
||||
return SamplingSearchResponse(
|
||||
query=query,
|
||||
generated_answer=generated_answer,
|
||||
sources=search_response.results,
|
||||
total_found=search_response.total_found,
|
||||
sources=accessible_results,
|
||||
total_found=len(accessible_results),
|
||||
search_method="semantic_sampling",
|
||||
model_used=sampling_result.model,
|
||||
stop_reason=sampling_result.stopReason,
|
||||
@@ -419,11 +477,11 @@ def configure_semantic_tools(mcp: FastMCP):
|
||||
generated_answer=(
|
||||
f"[Sampling request timed out]\n\n"
|
||||
f"The answer generation took too long (>30s). "
|
||||
f"Found {search_response.total_found} relevant documents. "
|
||||
f"Found {len(accessible_results)} relevant documents. "
|
||||
f"Please review the sources below or try a simpler query."
|
||||
),
|
||||
sources=search_response.results,
|
||||
total_found=search_response.total_found,
|
||||
sources=accessible_results,
|
||||
total_found=len(accessible_results),
|
||||
search_method="semantic_sampling_timeout",
|
||||
success=True,
|
||||
)
|
||||
@@ -454,11 +512,11 @@ def configure_semantic_tools(mcp: FastMCP):
|
||||
query=query,
|
||||
generated_answer=(
|
||||
f"[{user_message}]\n\n"
|
||||
f"Found {search_response.total_found} relevant documents. "
|
||||
f"Found {len(accessible_results)} relevant documents. "
|
||||
f"Please review the sources below."
|
||||
),
|
||||
sources=search_response.results,
|
||||
total_found=search_response.total_found,
|
||||
sources=accessible_results,
|
||||
total_found=len(accessible_results),
|
||||
search_method=search_method,
|
||||
success=True,
|
||||
)
|
||||
@@ -475,11 +533,11 @@ def configure_semantic_tools(mcp: FastMCP):
|
||||
query=query,
|
||||
generated_answer=(
|
||||
f"[Unexpected error during sampling]\n\n"
|
||||
f"Found {search_response.total_found} relevant documents. "
|
||||
f"Found {len(accessible_results)} relevant documents. "
|
||||
f"Please review the sources below."
|
||||
),
|
||||
sources=search_response.results,
|
||||
total_found=search_response.total_found,
|
||||
sources=accessible_results,
|
||||
total_found=len(accessible_results),
|
||||
search_method="semantic_sampling_error",
|
||||
success=True,
|
||||
)
|
||||
|
||||
@@ -15,6 +15,10 @@ from qdrant_client.models import FieldCondition, Filter, MatchValue, PointStruct
|
||||
from nextcloud_mcp_server.client import NextcloudClient
|
||||
from nextcloud_mcp_server.config import get_settings
|
||||
from nextcloud_mcp_server.embedding import get_embedding_service
|
||||
from nextcloud_mcp_server.observability.metrics import (
|
||||
record_qdrant_operation,
|
||||
record_vector_sync_processing,
|
||||
)
|
||||
from nextcloud_mcp_server.observability.tracing import trace_operation
|
||||
from nextcloud_mcp_server.vector.document_chunker import DocumentChunker
|
||||
from nextcloud_mcp_server.vector.qdrant_client import get_qdrant_client
|
||||
@@ -90,6 +94,8 @@ async def process_document(doc_task: DocumentTask, nc_client: NextcloudClient):
|
||||
doc_task: Document task to process
|
||||
nc_client: Authenticated Nextcloud client
|
||||
"""
|
||||
start_time = time.time()
|
||||
|
||||
logger.debug(
|
||||
f"Processing {doc_task.doc_type}_{doc_task.doc_id} "
|
||||
f"for {doc_task.user_id} ({doc_task.operation})"
|
||||
@@ -105,58 +111,79 @@ async def process_document(doc_task: DocumentTask, nc_client: NextcloudClient):
|
||||
"vector_sync.doc_operation": doc_task.operation,
|
||||
},
|
||||
):
|
||||
qdrant_client = await get_qdrant_client()
|
||||
settings = get_settings()
|
||||
try:
|
||||
qdrant_client = await get_qdrant_client()
|
||||
settings = get_settings()
|
||||
|
||||
# Handle deletion
|
||||
if doc_task.operation == "delete":
|
||||
await qdrant_client.delete(
|
||||
collection_name=settings.get_collection_name(),
|
||||
points_selector=Filter(
|
||||
must=[
|
||||
FieldCondition(
|
||||
key="user_id",
|
||||
match=MatchValue(value=doc_task.user_id),
|
||||
),
|
||||
FieldCondition(
|
||||
key="doc_id",
|
||||
match=MatchValue(value=doc_task.doc_id),
|
||||
),
|
||||
FieldCondition(
|
||||
key="doc_type",
|
||||
match=MatchValue(value=doc_task.doc_type),
|
||||
),
|
||||
]
|
||||
),
|
||||
)
|
||||
logger.info(
|
||||
f"Deleted {doc_task.doc_type}_{doc_task.doc_id} for {doc_task.user_id}"
|
||||
)
|
||||
return
|
||||
# Handle deletion
|
||||
if doc_task.operation == "delete":
|
||||
await qdrant_client.delete(
|
||||
collection_name=settings.get_collection_name(),
|
||||
points_selector=Filter(
|
||||
must=[
|
||||
FieldCondition(
|
||||
key="user_id",
|
||||
match=MatchValue(value=doc_task.user_id),
|
||||
),
|
||||
FieldCondition(
|
||||
key="doc_id",
|
||||
match=MatchValue(value=doc_task.doc_id),
|
||||
),
|
||||
FieldCondition(
|
||||
key="doc_type",
|
||||
match=MatchValue(value=doc_task.doc_type),
|
||||
),
|
||||
]
|
||||
),
|
||||
)
|
||||
logger.info(
|
||||
f"Deleted {doc_task.doc_type}_{doc_task.doc_id} for {doc_task.user_id}"
|
||||
)
|
||||
|
||||
# Handle indexing with retry
|
||||
max_retries = 3
|
||||
retry_delay = 1.0
|
||||
# Record successful deletion metrics
|
||||
duration = time.time() - start_time
|
||||
record_qdrant_operation("delete", "success")
|
||||
record_vector_sync_processing(duration, "success")
|
||||
return
|
||||
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
await _index_document(doc_task, nc_client, qdrant_client)
|
||||
return # Success
|
||||
# Handle indexing with retry
|
||||
max_retries = 3
|
||||
retry_delay = 1.0
|
||||
|
||||
except (HTTPStatusError, Exception) as e:
|
||||
if attempt < max_retries - 1:
|
||||
logger.warning(
|
||||
f"Retry {attempt + 1}/{max_retries} for "
|
||||
f"{doc_task.doc_type}_{doc_task.doc_id}: {e}"
|
||||
)
|
||||
await anyio.sleep(retry_delay)
|
||||
retry_delay *= 2 # Exponential backoff
|
||||
else:
|
||||
logger.error(
|
||||
f"Failed to index {doc_task.doc_type}_{doc_task.doc_id} "
|
||||
f"after {max_retries} retries: {e}"
|
||||
)
|
||||
raise
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
await _index_document(doc_task, nc_client, qdrant_client)
|
||||
|
||||
# Record successful processing metrics
|
||||
duration = time.time() - start_time
|
||||
record_qdrant_operation("upsert", "success")
|
||||
record_vector_sync_processing(duration, "success")
|
||||
return # Success
|
||||
|
||||
except (HTTPStatusError, Exception) as e:
|
||||
if attempt < max_retries - 1:
|
||||
logger.warning(
|
||||
f"Retry {attempt + 1}/{max_retries} for "
|
||||
f"{doc_task.doc_type}_{doc_task.doc_id}: {e}"
|
||||
)
|
||||
await anyio.sleep(retry_delay)
|
||||
retry_delay *= 2 # Exponential backoff
|
||||
else:
|
||||
logger.error(
|
||||
f"Failed to index {doc_task.doc_type}_{doc_task.doc_id} "
|
||||
f"after {max_retries} retries: {e}"
|
||||
)
|
||||
# Record failed processing metrics
|
||||
duration = time.time() - start_time
|
||||
record_qdrant_operation("upsert", "error")
|
||||
record_vector_sync_processing(duration, "error")
|
||||
raise
|
||||
|
||||
except Exception:
|
||||
# Catch any other unexpected errors
|
||||
duration = time.time() - start_time
|
||||
record_vector_sync_processing(duration, "error")
|
||||
raise
|
||||
|
||||
|
||||
async def _index_document(
|
||||
|
||||
@@ -13,6 +13,7 @@ from qdrant_client.models import FieldCondition, Filter, MatchValue
|
||||
|
||||
from nextcloud_mcp_server.client import NextcloudClient
|
||||
from nextcloud_mcp_server.config import get_settings
|
||||
from nextcloud_mcp_server.observability.metrics import record_vector_sync_scan
|
||||
from nextcloud_mcp_server.observability.tracing import trace_operation
|
||||
from nextcloud_mcp_server.vector.qdrant_client import get_qdrant_client
|
||||
|
||||
@@ -181,6 +182,9 @@ async def scan_user_documents(
|
||||
]
|
||||
logger.info(f"[SCAN-{scan_id}] Found {len(notes)} notes for {user_id}")
|
||||
|
||||
# Record documents scanned
|
||||
record_vector_sync_scan(len(notes))
|
||||
|
||||
if initial_sync:
|
||||
# Send everything on first sync
|
||||
for note in notes:
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "nextcloud-mcp-server"
|
||||
version = "0.32.1"
|
||||
version = "0.33.0"
|
||||
description = "Model Context Protocol (MCP) server for Nextcloud integration - enables AI assistants to interact with Nextcloud data"
|
||||
authors = [
|
||||
{name = "Chris Coutinho", email = "chris@coutinho.io"}
|
||||
|
||||
Reference in New Issue
Block a user