docs: refactor semantic search from notes-specific to multi-app architecture
Update ADRs to reflect that vector database and semantic search support multiple Nextcloud apps (notes, calendar, deck, files, contacts) rather than being notes-specific. Introduce semantic:read/write OAuth scopes to replace app-specific scope requirements for cross-app search. Changes: - ADR-007: Add plugin architecture (DocumentScanner, DocumentProcessor, DocumentVerifier) for multi-app vector sync - ADR-008: Rename tools from nc_notes_semantic_* to nc_semantic_*, update scope from notes:read to semantic:read - ADR-009: NEW - Document decision to use generic semantic:read scope with dual-phase authorization instead of requiring all app scopes - oauth-architecture.md: Add semantic:read/write scope documentation - README.md: Move semantic search to dedicated section separate from Notes This is a breaking change that correctly positions semantic search as a cross-app capability before broader adoption. Existing deployments will need to re-authenticate with the new semantic:read scope. Relates to user request to decouple vector database from notes-only model and establish proper OAuth scope boundaries for multi-app semantic search. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
@@ -19,7 +19,8 @@ The Nextcloud MCP (Model Context Protocol) server allows Large Language Models l
|
||||
| **Deployment** | Standalone (Docker, VM, K8s) | Inside Nextcloud (ExApp via AppAPI) |
|
||||
| **Primary Users** | Claude Code, IDEs, external developers | Nextcloud end users via Assistant app |
|
||||
| **Authentication** | OAuth2/OIDC or Basic Auth | Session-based (integrated) |
|
||||
| **Notes Support** | ✅ Full CRUD + search + semantic search (9 tools) | ❌ Not implemented |
|
||||
| **Notes Support** | ✅ Full CRUD + keyword search (7 tools) | ❌ Not implemented |
|
||||
| **Semantic Search** | ✅ Multi-app vector search (2+ tools) | ❌ Not implemented |
|
||||
| **Calendar** | ✅ Full CalDAV + tasks (20+ tools) | ✅ Events, free/busy, tasks (4 tools) |
|
||||
| **Contacts** | ✅ Full CardDAV (8 tools) | ✅ Find person, current user (2 tools) |
|
||||
| **Files (WebDAV)** | ✅ Full filesystem access (12 tools) | ✅ Read, folder tree, sharing (3 tools) |
|
||||
@@ -200,7 +201,7 @@ For a complete list of all supported OAuth scopes and their descriptions, see [O
|
||||
|
||||
| App | Tools | Read Scope | Write Scope | Operations |
|
||||
|-----|-------|-----------|-------------|------------|
|
||||
| **Notes** | 9 | `notes:read` | `notes:write` | Create, read, update, delete, search notes (keyword + semantic) |
|
||||
| **Notes** | 7 | `notes:read` | `notes:write` | Create, read, update, delete, search notes (keyword search) |
|
||||
| **Calendar** | 20+ | `calendar:read` `todo:read` | `calendar:write` `todo:write` | Events, todos (tasks), calendars, recurring events, attendees |
|
||||
| **Contacts** | 8 | `contacts:read` | `contacts:write` | Create, read, update, delete contacts and address books |
|
||||
| **Files (WebDAV)** | 12 | `files:read` | `files:write` | List, read, upload, delete, move files; **OCR/document processing** |
|
||||
@@ -208,6 +209,7 @@ For a complete list of all supported OAuth scopes and their descriptions, see [O
|
||||
| **Cookbook** | 13 | `cookbook:read` | `cookbook:write` | Recipes, import from URLs, search, categories |
|
||||
| **Tables** | 5 | `tables:read` | `tables:write` | Row operations on Nextcloud Tables |
|
||||
| **Sharing** | 10+ | `sharing:read` | `sharing:write` | Create, manage, delete shares |
|
||||
| **Semantic Search** | 2+ | `semantic:read` | `semantic:write` | Vector-powered semantic search across **all apps** (notes, calendar, deck, files, contacts), background indexing |
|
||||
|
||||
#### Document Processing (Optional)
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@
|
||||
|
||||
ADR-003 proposed a vector database architecture for semantic search over Nextcloud content, introducing Qdrant as the vector store, configurable embedding strategies, and hybrid search combining semantic and keyword matching. While these technical decisions remain sound, ADR-003 was never implemented because it lacked a critical component: a practical system for keeping the vector database synchronized with changing Nextcloud content.
|
||||
|
||||
The challenge is not simply indexing content once, but maintaining an up-to-date vector database as users create, modify, and delete notes, files, and other documents. This synchronization must happen in the background, outside of active MCP sessions, and must operate efficiently across multiple users without manual intervention. Users should not need to understand the mechanics of vector indexing—they simply enable semantic search and the system handles the rest.
|
||||
The challenge is not simply indexing content once, but maintaining an up-to-date vector database as users create, modify, and delete documents across multiple Nextcloud apps (notes, calendar events, deck cards, files, contacts). This synchronization must happen in the background, outside of active MCP sessions, and must operate efficiently across multiple users and content types without manual intervention. Users should not need to understand the mechanics of vector indexing—they simply enable semantic search and the system handles the rest.
|
||||
|
||||
ADR-003's conceptual description of a "background sync worker" left several fundamental questions unanswered:
|
||||
|
||||
@@ -57,6 +57,87 @@ The in-process model also simplifies state access. Background tasks and MCP tool
|
||||
|
||||
This architecture is not suitable for CPU-bound workloads (video transcoding, image processing, ML training) where separate worker processes or machines would be necessary. But for embedding-based semantic search, where the bottleneck is I/O latency to external APIs, in-process async concurrency provides an excellent balance of simplicity and performance.
|
||||
|
||||
### Multi-App Plugin Architecture
|
||||
|
||||
The vector sync system supports multiple Nextcloud apps through a plugin-based design. Each app that provides searchable content implements three interfaces:
|
||||
|
||||
**DocumentScanner Interface**: Responsible for discovering documents in the app and extracting basic metadata for change detection.
|
||||
|
||||
```python
|
||||
class DocumentScanner(ABC):
|
||||
@abstractmethod
|
||||
async def get_all_documents(self, nc_client: NextcloudClient) -> list[dict]:
|
||||
"""Fetch all documents for this app."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_doc_type(self) -> str:
|
||||
"""Return doc_type identifier (e.g., 'note', 'calendar_event')."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def extract_doc_id(self, doc: dict) -> str:
|
||||
"""Extract document ID from document dict."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def extract_modified_at(self, doc: dict) -> int:
|
||||
"""Extract modification timestamp."""
|
||||
pass
|
||||
```
|
||||
|
||||
**DocumentProcessor Interface**: Responsible for fetching full document content and extracting searchable text.
|
||||
|
||||
```python
|
||||
class DocumentProcessor(ABC):
|
||||
@abstractmethod
|
||||
def get_doc_type(self) -> str:
|
||||
"""Return doc_type this processor handles."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
async def fetch_document(self, doc_task: DocumentTask, nc_client: NextcloudClient) -> dict:
|
||||
"""Fetch full document from Nextcloud."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def extract_content(self, document: dict) -> str:
|
||||
"""Extract searchable text content."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def extract_title(self, document: dict) -> str:
|
||||
"""Extract document title."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def extract_metadata(self, document: dict) -> dict:
|
||||
"""Extract app-specific metadata for Qdrant payload."""
|
||||
pass
|
||||
```
|
||||
|
||||
**DocumentVerifier Interface**: Responsible for verifying user access during semantic search (dual-phase authorization).
|
||||
|
||||
```python
|
||||
class DocumentVerifier(ABC):
|
||||
@abstractmethod
|
||||
async def verify_access(self, doc_id: str, nc_client: NextcloudClient) -> bool:
|
||||
"""Verify user has access to document. Return True if accessible."""
|
||||
pass
|
||||
```
|
||||
|
||||
Concrete implementations for each app are registered in central registries (`SCANNERS`, `PROCESSORS`, `VERIFIERS`). The scanner task iterates through registered scanners for enabled apps, the processor tasks dispatch to registered processors based on `doc_type`, and semantic search tools use registered verifiers to check access.
|
||||
|
||||
**Supported Document Types**:
|
||||
- `note`: Notes app documents (implemented)
|
||||
- `calendar_event`: Calendar events (VEVENT)
|
||||
- `calendar_todo`: Calendar tasks (VTODO)
|
||||
- `deck_card`: Deck cards
|
||||
- `file`: WebDAV files with text extraction (leverages ADR-006 document processing)
|
||||
- `contact`: CardDAV contacts (VCARD)
|
||||
|
||||
New apps can be added by implementing the three interfaces and registering the implementations—no changes to core sync logic are required. The `VECTOR_SYNC_ENABLED_APPS` environment variable controls which apps are actually indexed.
|
||||
|
||||
### Change Detection: ETag and Modification Timestamps
|
||||
|
||||
Rather than polling every document's content on every sync or attempting to configure complex webhooks, we use a timestamp comparison approach. Each vector stored in Qdrant includes an `indexed_at` field in its metadata payload, recording when the document was last processed. When the scanner runs, it fetches the list of documents from Nextcloud (which includes each document's `modified_at` timestamp and `etag`) and compares these values against the stored `indexed_at` timestamps from Qdrant.
|
||||
@@ -74,7 +155,7 @@ The task queue is implemented using Python's built-in `asyncio.Queue`, which pro
|
||||
class DocumentTask:
|
||||
user_id: str
|
||||
doc_id: str
|
||||
doc_type: str # "note", "file", "calendar"
|
||||
doc_type: str # "note", "calendar_event", "calendar_todo", "deck_card", "file", "contact"
|
||||
operation: str # "index" or "delete"
|
||||
modified_at: int
|
||||
```
|
||||
@@ -159,14 +240,15 @@ The MCP tool interface reflects the simplicity of the user model:
|
||||
|
||||
```python
|
||||
@mcp.tool()
|
||||
@require_scopes("sync:write")
|
||||
@require_scopes("semantic:write")
|
||||
async def enable_vector_sync(ctx: Context) -> dict:
|
||||
"""
|
||||
Enable automatic background vector synchronization for semantic search.
|
||||
|
||||
Once enabled, the system will automatically maintain a vector database
|
||||
of your Nextcloud content, enabling semantic search capabilities. No
|
||||
further action is required - synchronization happens in the background.
|
||||
of your Nextcloud content across all enabled apps (notes, calendar, deck,
|
||||
files, contacts), enabling semantic search capabilities. No further action
|
||||
is required - synchronization happens in the background.
|
||||
|
||||
Returns:
|
||||
Status message and current indexed document count
|
||||
@@ -201,7 +283,7 @@ async def enable_vector_sync(ctx: Context) -> dict:
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
@require_scopes("sync:write")
|
||||
@require_scopes("semantic:write")
|
||||
async def disable_vector_sync(ctx: Context) -> dict:
|
||||
"""
|
||||
Disable vector synchronization and remove all indexed vectors.
|
||||
@@ -240,7 +322,7 @@ async def disable_vector_sync(ctx: Context) -> dict:
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
@require_scopes("sync:read")
|
||||
@require_scopes("semantic:read")
|
||||
async def get_vector_sync_status(ctx: Context) -> dict:
|
||||
"""
|
||||
Get current vector synchronization status.
|
||||
@@ -480,79 +562,93 @@ async def scan_user_documents(
|
||||
username=user_id
|
||||
)
|
||||
|
||||
# Fetch all notes
|
||||
notes = await client.notes.list_notes()
|
||||
# Get list of enabled document types from configuration
|
||||
enabled_apps = settings.vector_sync_enabled_apps # ["note", "calendar_event", "deck_card", ...]
|
||||
|
||||
queued = 0
|
||||
|
||||
# Scan each enabled app using registered scanners
|
||||
for scanner in get_registered_scanners():
|
||||
doc_type = scanner.get_doc_type()
|
||||
|
||||
if doc_type not in enabled_apps:
|
||||
continue # Skip disabled apps
|
||||
|
||||
# Fetch all documents for this app
|
||||
documents = await scanner.get_all_documents(client)
|
||||
|
||||
if initial_sync:
|
||||
# Queue everything on first sync
|
||||
for doc in documents:
|
||||
await document_queue.put(
|
||||
DocumentTask(
|
||||
user_id=user_id,
|
||||
doc_id=scanner.extract_doc_id(doc),
|
||||
doc_type=doc_type,
|
||||
operation="index",
|
||||
modified_at=scanner.extract_modified_at(doc)
|
||||
)
|
||||
)
|
||||
queued += 1
|
||||
continue # Move to next scanner
|
||||
|
||||
# Get indexed state from Qdrant for this doc_type
|
||||
qdrant_client = get_qdrant_client()
|
||||
scroll_result = await qdrant_client.scroll(
|
||||
collection_name="nextcloud_content",
|
||||
scroll_filter=Filter(
|
||||
must=[
|
||||
FieldCondition(key="user_id", match=MatchValue(value=user_id)),
|
||||
FieldCondition(key="doc_type", match=MatchValue(value=doc_type))
|
||||
]
|
||||
),
|
||||
with_payload=["doc_id", "indexed_at"],
|
||||
with_vectors=False,
|
||||
limit=10000
|
||||
)
|
||||
|
||||
indexed_docs = {
|
||||
point.payload["doc_id"]: point.payload["indexed_at"]
|
||||
for point, _ in scroll_result[0]
|
||||
}
|
||||
|
||||
# Compare and queue changes
|
||||
for doc in documents:
|
||||
doc_id = scanner.extract_doc_id(doc)
|
||||
indexed_at = indexed_docs.get(doc_id)
|
||||
|
||||
# Queue if never indexed or modified since last index
|
||||
if indexed_at is None or scanner.extract_modified_at(doc) > indexed_at:
|
||||
await document_queue.put(
|
||||
DocumentTask(
|
||||
user_id=user_id,
|
||||
doc_id=doc_id,
|
||||
doc_type=doc_type,
|
||||
operation="index",
|
||||
modified_at=scanner.extract_modified_at(doc)
|
||||
)
|
||||
)
|
||||
queued += 1
|
||||
|
||||
# Check for deleted documents (in Qdrant but not in Nextcloud)
|
||||
nextcloud_doc_ids = {scanner.extract_doc_id(doc) for doc in documents}
|
||||
for doc_id in indexed_docs:
|
||||
if doc_id not in nextcloud_doc_ids:
|
||||
await document_queue.put(
|
||||
DocumentTask(
|
||||
user_id=user_id,
|
||||
doc_id=doc_id,
|
||||
doc_type=doc_type,
|
||||
operation="delete",
|
||||
modified_at=0
|
||||
)
|
||||
)
|
||||
queued += 1
|
||||
|
||||
if initial_sync:
|
||||
# Queue everything on first sync
|
||||
for note in notes:
|
||||
await document_queue.put(
|
||||
DocumentTask(
|
||||
user_id=user_id,
|
||||
doc_id=str(note.id),
|
||||
doc_type="note",
|
||||
operation="index",
|
||||
modified_at=note.modified
|
||||
)
|
||||
)
|
||||
logger.info(f"Queued {len(notes)} documents for initial sync: {user_id}")
|
||||
return
|
||||
|
||||
# Get indexed state from Qdrant
|
||||
qdrant_client = get_qdrant_client()
|
||||
scroll_result = await qdrant_client.scroll(
|
||||
collection_name="nextcloud_content",
|
||||
scroll_filter=Filter(
|
||||
must=[
|
||||
FieldCondition(key="user_id", match=MatchValue(value=user_id)),
|
||||
FieldCondition(key="doc_type", match=MatchValue(value="note"))
|
||||
]
|
||||
),
|
||||
with_payload=["doc_id", "indexed_at"],
|
||||
with_vectors=False,
|
||||
limit=10000
|
||||
)
|
||||
|
||||
indexed_docs = {
|
||||
point.payload["doc_id"]: point.payload["indexed_at"]
|
||||
for point, _ in scroll_result[0]
|
||||
}
|
||||
|
||||
# Compare and queue changes
|
||||
queued = 0
|
||||
for note in notes:
|
||||
doc_id = str(note.id)
|
||||
indexed_at = indexed_docs.get(doc_id)
|
||||
|
||||
# Queue if never indexed or modified since last index
|
||||
if indexed_at is None or note.modified > indexed_at:
|
||||
await document_queue.put(
|
||||
DocumentTask(
|
||||
user_id=user_id,
|
||||
doc_id=doc_id,
|
||||
doc_type="note",
|
||||
operation="index",
|
||||
modified_at=note.modified
|
||||
)
|
||||
)
|
||||
queued += 1
|
||||
|
||||
# Check for deleted documents (in Qdrant but not in Nextcloud)
|
||||
nextcloud_doc_ids = {str(note.id) for note in notes}
|
||||
for doc_id in indexed_docs:
|
||||
if doc_id not in nextcloud_doc_ids:
|
||||
await document_queue.put(
|
||||
DocumentTask(
|
||||
user_id=user_id,
|
||||
doc_id=doc_id,
|
||||
doc_type="note",
|
||||
operation="delete",
|
||||
modified_at=0
|
||||
)
|
||||
)
|
||||
queued += 1
|
||||
|
||||
logger.info(f"Queued {queued} documents for incremental sync: {user_id}")
|
||||
logger.info(f"Queued {queued} documents for initial sync: {user_id}")
|
||||
else:
|
||||
logger.info(f"Queued {queued} documents for incremental sync: {user_id}")
|
||||
|
||||
# Update settings
|
||||
settings_repo = VectorSyncSettingsRepository()
|
||||
@@ -707,14 +803,16 @@ async def _index_document(doc_task: DocumentTask, qdrant_client):
|
||||
username=doc_task.user_id
|
||||
)
|
||||
|
||||
# Fetch document content
|
||||
if doc_task.doc_type == "note":
|
||||
document = await client.notes.get_note(int(doc_task.doc_id))
|
||||
content = f"{document['title']}\n\n{document['content']}"
|
||||
title = document['title']
|
||||
etag = document.get('etag', '')
|
||||
else:
|
||||
raise ValueError(f"Unsupported doc_type: {doc_task.doc_type}")
|
||||
# Get processor for this document type
|
||||
processor = get_registered_processor(doc_task.doc_type)
|
||||
if not processor:
|
||||
raise ValueError(f"No processor registered for doc_type: {doc_task.doc_type}")
|
||||
|
||||
# Fetch document content using processor
|
||||
document = await processor.fetch_document(doc_task, client)
|
||||
content = processor.extract_content(document)
|
||||
title = processor.extract_title(document)
|
||||
metadata = processor.extract_metadata(document) # App-specific fields
|
||||
|
||||
# Tokenize and chunk
|
||||
chunker = DocumentChunker(chunk_size=512, overlap=50)
|
||||
@@ -741,9 +839,10 @@ async def _index_document(doc_task: DocumentTask, qdrant_client):
|
||||
"excerpt": chunk[:200],
|
||||
"indexed_at": indexed_at,
|
||||
"modified_at": doc_task.modified_at,
|
||||
"etag": etag,
|
||||
"chunk_index": i,
|
||||
"total_chunks": len(chunks)
|
||||
"total_chunks": len(chunks),
|
||||
# App-specific metadata (e.g., category for notes, location for calendar)
|
||||
"metadata": metadata
|
||||
}
|
||||
)
|
||||
)
|
||||
@@ -766,6 +865,7 @@ async def _index_document(doc_task: DocumentTask, qdrant_client):
|
||||
```bash
|
||||
# Vector Sync Configuration
|
||||
VECTOR_SYNC_ENABLED=true
|
||||
VECTOR_SYNC_ENABLED_APPS=note,calendar_event,calendar_todo,deck_card,file,contact # Apps to index
|
||||
VECTOR_SYNC_SCAN_INTERVAL=3600 # Scanner runs every 3600 seconds (1 hour)
|
||||
VECTOR_SYNC_PROCESSOR_WORKERS=3 # Number of concurrent processor tasks
|
||||
VECTOR_SYNC_QUEUE_MAX_SIZE=10000 # Maximum documents in queue
|
||||
@@ -865,9 +965,11 @@ The authentication dependency on Flow 2 refresh tokens means users must complete
|
||||
|
||||
### Performance Characteristics
|
||||
|
||||
With three concurrent processor tasks and OpenAI's embedding API (100ms average latency), the system can process approximately 30 documents per second under ideal conditions. This translates to 1,800 documents per minute or 108,000 documents per hour. For a deployment with 100 users averaging 1,000 notes each, full initial indexing would complete within one hour of enabling semantic search.
|
||||
With three concurrent processor tasks and OpenAI's embedding API (100ms average latency), the system can process approximately 30 documents per second under ideal conditions. This translates to 1,800 documents per minute or 108,000 documents per hour. For a deployment with 100 users averaging 1,000 documents each across all enabled apps (notes, calendar events, deck cards, etc.), full initial indexing would complete within one hour of enabling semantic search.
|
||||
|
||||
Incremental syncs are much faster because most documents haven't changed between scanner runs. If the typical change rate is 1% of documents per hour (10 notes per user), the system processes 1,000 documents per scan cycle with the same 100 users, completing within 30 seconds. This keeps the vector database current with minimal lag.
|
||||
Incremental syncs are much faster because most documents haven't changed between scanner runs. If the typical change rate is 1% of documents per hour (10 documents per user across all apps), the system processes 1,000 documents per scan cycle with the same 100 users, completing within 30 seconds. This keeps the vector database current with minimal lag.
|
||||
|
||||
Performance scales linearly with the number of enabled apps. Enabling calendar and deck in addition to notes will approximately triple the initial indexing time, but incremental syncs remain fast because each app's change rate is independent.
|
||||
|
||||
The scanner itself is lightweight, making only API calls to list documents and scroll Qdrant metadata. With efficient API design (batch fetching, minimal payloads), a single scanner invocation for 100 users completes within minutes. The hourly scan interval provides ample time for completion even with occasional slowdowns.
|
||||
|
||||
@@ -875,7 +977,7 @@ The in-memory queue has negligible memory overhead. Each `DocumentTask` is appro
|
||||
|
||||
### Cost Estimates
|
||||
|
||||
For a deployment using OpenAI embeddings with 100 users averaging 500 notes each (50,000 total documents):
|
||||
For a deployment using OpenAI embeddings with 100 users, with notes only enabled (500 notes/user = 50,000 total documents):
|
||||
|
||||
Initial indexing cost: 50,000 documents × 250 words/document × $0.00002/1000 tokens ≈ $2.50
|
||||
|
||||
@@ -883,7 +985,9 @@ Monthly incremental sync cost (assuming 1% daily change rate): 50,000 × 0.01 ×
|
||||
|
||||
Total first month: $4.38, subsequent months: $1.88
|
||||
|
||||
Infrastructure costs (self-hosted): Qdrant requires approximately 200MB RAM for 50,000 vectors (4KB per document), the MCP server with background tasks uses approximately 512MB RAM (same as without background sync because tasks are I/O-bound), total infrastructure cost is dominated by Qdrant storage.
|
||||
**With multiple apps enabled** (notes + calendar + deck), costs scale proportionally. If each user has 500 notes, 200 calendar events, and 100 deck cards, the total document count becomes 80,000, and costs increase by 60% (first month: $7.00, subsequent months: $3.00).
|
||||
|
||||
Infrastructure costs (self-hosted): Qdrant requires approximately 200MB RAM for 50,000 vectors (4KB per document), scaling to 320MB RAM for 80,000 vectors. The MCP server with background tasks uses approximately 512MB RAM (same as without background sync because tasks are I/O-bound), total infrastructure cost is dominated by Qdrant storage.
|
||||
|
||||
Alternative with self-hosted embeddings: Zero per-document costs, requires GPU instance ($0.50/hour = $360/month for 24/7 operation) or CPU-only processing (negligible cost, ~10x slower embedding generation, can be run via `anyio.to_thread.run_sync()` in processor tasks).
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# ADR-008: MCP Sampling for Semantic Search Enhancement
|
||||
# ADR-008: MCP Sampling for Multi-App Semantic Search with RAG
|
||||
|
||||
**Status**: Proposed
|
||||
**Date**: 2025-01-11
|
||||
@@ -6,9 +6,9 @@
|
||||
|
||||
## Context
|
||||
|
||||
ADR-007 established a background synchronization architecture that maintains a vector database of Nextcloud content, enabling semantic search via the `nc_notes_semantic_search` tool. This tool returns a list of relevant documents with excerpts, similarity scores, and metadata—providing the raw materials for answering user questions.
|
||||
ADR-007 established a background synchronization architecture that maintains a vector database of Nextcloud content across multiple apps (notes, calendar, deck, files, contacts), enabling semantic search via the `nc_semantic_search` tool. This tool returns a list of relevant documents with excerpts, similarity scores, and metadata—providing the raw materials for answering user questions.
|
||||
|
||||
However, users typically don't want a list of documents—they want answers to their questions. When a user asks "What are my project goals?" or "What did I learn about Python last month?", they expect a natural language response that synthesizes information from multiple sources, not a ranked list of note excerpts. This is the pattern of Retrieval-Augmented Generation (RAG): retrieve relevant context, then generate a cohesive answer.
|
||||
However, users typically don't want a list of documents—they want answers to their questions. When a user asks "What are my project goals?" or "When is my next dentist appointment?", they expect a natural language response that synthesizes information from multiple sources and document types, not a ranked list of excerpts. This is the pattern of Retrieval-Augmented Generation (RAG): retrieve relevant context from all Nextcloud apps, then generate a cohesive answer.
|
||||
|
||||
The challenge is: who should generate the answer, and how?
|
||||
|
||||
@@ -54,21 +54,21 @@ However, sampling introduces new considerations:
|
||||
|
||||
Despite these considerations, MCP sampling provides the most principled solution for RAG-enhanced semantic search. It respects the client-server boundary, avoids duplicate infrastructure, and delivers the user experience users expect from semantic search tools.
|
||||
|
||||
This ADR proposes adding a new tool, `nc_notes_semantic_search_answer`, that uses MCP sampling to generate natural language answers from retrieved Nextcloud content.
|
||||
This ADR proposes adding a new tool, `nc_semantic_search_answer`, that uses MCP sampling to generate natural language answers from retrieved Nextcloud content across all indexed apps (notes, calendar, deck, files, contacts).
|
||||
|
||||
## Decision
|
||||
|
||||
We will implement a new MCP tool `nc_notes_semantic_search_answer` that retrieves relevant documents via vector similarity search and uses MCP sampling to generate natural language answers. The tool will construct a prompt that includes the user's original query and excerpts from retrieved documents, request an LLM completion via `ctx.session.create_message()`, and return the generated answer along with source citations.
|
||||
We will implement a new MCP tool `nc_semantic_search_answer` that retrieves relevant documents via vector similarity search across all indexed Nextcloud apps and uses MCP sampling to generate natural language answers. The tool will construct a prompt that includes the user's original query and excerpts from retrieved documents (notes, calendar events, deck cards, files, contacts), request an LLM completion via `ctx.session.create_message()`, and return the generated answer along with source citations.
|
||||
|
||||
The existing `nc_notes_semantic_search` tool will remain unchanged, providing users with a choice: call the original tool for raw document results, or call the new sampling-enhanced tool for generated answers. This dual-tool approach respects different use cases—some users want to browse documents, others want direct answers.
|
||||
The existing `nc_semantic_search` tool will remain unchanged, providing users with a choice: call the original tool for raw document results, or call the new sampling-enhanced tool for generated answers. This dual-tool approach respects different use cases—some users want to browse documents, others want direct answers.
|
||||
|
||||
### API Design
|
||||
|
||||
**Tool Signature**:
|
||||
```python
|
||||
@mcp.tool()
|
||||
@require_scopes("notes:read")
|
||||
async def nc_notes_semantic_search_answer(
|
||||
@require_scopes("semantic:read")
|
||||
async def nc_semantic_search_answer(
|
||||
query: str,
|
||||
ctx: Context,
|
||||
limit: int = 5,
|
||||
@@ -108,7 +108,7 @@ from mcp.types import SamplingMessage, TextContent, ModelPreferences, ModelHint
|
||||
# Construct prompt with retrieved context
|
||||
prompt = (
|
||||
f"{query}\n\n"
|
||||
f"Here are relevant documents from Nextcloud Notes:\n\n"
|
||||
f"Here are relevant documents from Nextcloud (notes, calendar events, deck cards, files, contacts):\n\n"
|
||||
f"{context}\n\n"
|
||||
f"Based on the documents above, please provide a comprehensive answer. "
|
||||
f"Cite the document numbers when referencing specific information."
|
||||
@@ -153,20 +153,29 @@ The prompt construction follows a structured template:
|
||||
```
|
||||
[User's original query]
|
||||
|
||||
Here are relevant documents from Nextcloud Notes:
|
||||
Here are relevant documents from Nextcloud (notes, calendar events, deck cards, files, contacts):
|
||||
|
||||
[Document 1]
|
||||
Type: note
|
||||
Title: Project Kickoff Notes
|
||||
Category: Work
|
||||
Excerpt: The primary goal for Q1 2025 is to improve semantic search...
|
||||
Relevance Score: 0.92
|
||||
|
||||
[Document 2]
|
||||
Title: Meeting Notes - Jan 5
|
||||
Category: Work
|
||||
Excerpt: Team agreed on three key objectives...
|
||||
Type: calendar_event
|
||||
Title: Team Planning Meeting
|
||||
Location: Conference Room A
|
||||
Excerpt: Scheduled for Jan 15 at 2pm. Agenda: Discuss Q1 objectives and timeline...
|
||||
Relevance Score: 0.88
|
||||
|
||||
[Document 3]
|
||||
Type: deck_card
|
||||
Title: Implement semantic search
|
||||
Labels: feature, high-priority
|
||||
Excerpt: This card tracks the semantic search implementation. Due: Jan 30...
|
||||
Relevance Score: 0.85
|
||||
|
||||
Based on the documents above, please provide a comprehensive answer.
|
||||
Cite the document numbers when referencing specific information.
|
||||
```
|
||||
@@ -211,7 +220,7 @@ When semantic search finds no relevant documents (all below `score_threshold`),
|
||||
if not search_response.results:
|
||||
return SamplingSearchResponse(
|
||||
query=query,
|
||||
generated_answer="No relevant documents found in your Nextcloud Notes for this query.",
|
||||
generated_answer="No relevant documents found in your Nextcloud content for this query.",
|
||||
sources=[],
|
||||
total_found=0,
|
||||
search_method="semantic_sampling",
|
||||
@@ -224,17 +233,17 @@ This avoids wasting a sampling call (and user approval) when there's no content
|
||||
### User Experience Flow
|
||||
|
||||
**Typical successful flow**:
|
||||
1. User calls `nc_notes_semantic_search_answer` with query "What are my project goals?"
|
||||
2. Server retrieves 5 relevant notes via vector search
|
||||
3. Server constructs prompt with document excerpts
|
||||
1. User calls `nc_semantic_search_answer` with query "What are my Q1 2025 objectives?"
|
||||
2. Server retrieves 5 relevant documents via vector search (2 notes, 2 calendar events, 1 deck card)
|
||||
3. Server constructs prompt with document excerpts showing mixed content types
|
||||
4. Server sends `sampling/createMessage` request to client
|
||||
5. Client prompts user: "MCP server wants to generate an answer using these documents. Allow?"
|
||||
6. User approves (or client auto-approves based on configuration)
|
||||
7. Client sends prompt to LLM (Claude, GPT-4, etc.)
|
||||
8. LLM generates answer with citations: "Based on Document 1 and Document 3..."
|
||||
8. LLM generates answer with citations: "Based on Document 1 (note: Project Kickoff), Document 2 (calendar: Team Planning Meeting), and Document 3 (deck card: Implement semantic search)..."
|
||||
9. Client returns answer to server
|
||||
10. Server returns `SamplingSearchResponse` with answer and sources
|
||||
11. User sees complete answer with citations
|
||||
11. User sees complete answer with citations across multiple Nextcloud apps
|
||||
|
||||
**Fallback flow** (sampling unavailable):
|
||||
1-3. Same as above
|
||||
@@ -256,7 +265,7 @@ This three-tier approach (answer → documents → error message) ensures users
|
||||
|
||||
### Response Model
|
||||
|
||||
Add to `nextcloud_mcp_server/models/notes.py`:
|
||||
Add to `nextcloud_mcp_server/models/semantic.py` (new file for semantic search models):
|
||||
|
||||
```python
|
||||
from pydantic import Field
|
||||
@@ -305,7 +314,7 @@ class SamplingSearchResponse(BaseResponse):
|
||||
|
||||
### Tool Implementation
|
||||
|
||||
Add to `nextcloud_mcp_server/server/notes.py`:
|
||||
Add to `nextcloud_mcp_server/server/semantic.py` (new file for semantic search tools):
|
||||
|
||||
```python
|
||||
import logging
|
||||
@@ -315,8 +324,8 @@ logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
@require_scopes("notes:read")
|
||||
async def nc_notes_semantic_search_answer(
|
||||
@require_scopes("semantic:read")
|
||||
async def nc_semantic_search_answer(
|
||||
query: str,
|
||||
ctx: Context,
|
||||
limit: int = 5,
|
||||
@@ -326,14 +335,16 @@ async def nc_notes_semantic_search_answer(
|
||||
"""
|
||||
Semantic search with LLM-generated answer using MCP sampling.
|
||||
|
||||
Retrieves relevant documents from Nextcloud Notes using vector similarity
|
||||
search, then uses MCP sampling to request the client's LLM to generate
|
||||
a natural language answer based on the retrieved context.
|
||||
Retrieves relevant documents from Nextcloud across all indexed apps (notes,
|
||||
calendar, deck, files, contacts) using vector similarity search, then uses
|
||||
MCP sampling to request the client's LLM to generate a natural language
|
||||
answer based on the retrieved context.
|
||||
|
||||
This tool combines the power of semantic search (finding relevant content)
|
||||
with LLM generation (synthesizing that content into coherent answers). The
|
||||
generated answer includes citations to specific documents, allowing users
|
||||
to verify claims and explore sources.
|
||||
This tool combines the power of semantic search (finding relevant content
|
||||
across all your Nextcloud apps) with LLM generation (synthesizing that
|
||||
content into coherent answers). The generated answer includes citations
|
||||
to specific documents with their types, allowing users to verify claims
|
||||
and explore sources.
|
||||
|
||||
The LLM generation happens client-side via MCP sampling. The MCP client
|
||||
controls which model is used, who pays for it, and whether to prompt the
|
||||
@@ -341,7 +352,7 @@ async def nc_notes_semantic_search_answer(
|
||||
while giving users full control over their LLM interactions.
|
||||
|
||||
Args:
|
||||
query: Natural language question to answer (e.g., "What are my project goals?")
|
||||
query: Natural language question to answer (e.g., "What are my Q1 objectives?" or "When is my next dentist appointment?")
|
||||
ctx: MCP context for session access
|
||||
limit: Maximum number of documents to retrieve (default: 5)
|
||||
score_threshold: Minimum similarity score 0-1 (default: 0.7)
|
||||
@@ -359,27 +370,28 @@ async def nc_notes_semantic_search_answer(
|
||||
The client may prompt the user to approve the sampling request.
|
||||
|
||||
Examples:
|
||||
>>> # Query about project goals
|
||||
>>> result = await nc_notes_semantic_search_answer(
|
||||
>>> # Query about objectives across multiple apps
|
||||
>>> result = await nc_semantic_search_answer(
|
||||
... query="What are my Q1 2025 project goals?",
|
||||
... ctx=ctx
|
||||
... )
|
||||
>>> print(result.generated_answer)
|
||||
"Based on Document 1 (Project Kickoff) and Document 3 (Q1 Planning),
|
||||
"Based on Document 1 (note: Project Kickoff), Document 2 (calendar event:
|
||||
Q1 Planning Meeting), and Document 3 (deck card: Implement semantic search),
|
||||
your main goals are: 1) Improve semantic search accuracy by 20%,
|
||||
2) Deploy new embedding model, 3) Reduce indexing latency..."
|
||||
|
||||
>>> # Query about learning
|
||||
>>> result = await nc_notes_semantic_search_answer(
|
||||
... query="What did I learn about Python async/await last month?",
|
||||
>>> # Query about appointments
|
||||
>>> result = await nc_semantic_search_answer(
|
||||
... query="When is my next dentist appointment?",
|
||||
... ctx=ctx,
|
||||
... limit=10
|
||||
... )
|
||||
>>> len(result.sources) # Up to 10 documents
|
||||
7
|
||||
>>> len(result.sources) # Calendar events and related notes
|
||||
3
|
||||
"""
|
||||
# 1. Retrieve relevant documents via existing semantic search
|
||||
search_response = await nc_notes_semantic_search(
|
||||
search_response = await nc_semantic_search(
|
||||
query=query,
|
||||
ctx=ctx,
|
||||
limit=limit,
|
||||
@@ -391,7 +403,7 @@ async def nc_notes_semantic_search_answer(
|
||||
logger.debug(f"No documents found for query: {query}")
|
||||
return SamplingSearchResponse(
|
||||
query=query,
|
||||
generated_answer="No relevant documents found in your Nextcloud Notes for this query.",
|
||||
generated_answer="No relevant documents found in your Nextcloud content for this query.",
|
||||
sources=[],
|
||||
total_found=0,
|
||||
search_method="semantic_sampling",
|
||||
@@ -414,7 +426,7 @@ async def nc_notes_semantic_search_answer(
|
||||
# 4. Construct prompt - reuse user's query, add context and instructions
|
||||
prompt = (
|
||||
f"{query}\n\n"
|
||||
f"Here are relevant documents from Nextcloud Notes:\n\n"
|
||||
f"Here are relevant documents from Nextcloud (notes, calendar events, deck cards, files, contacts):\n\n"
|
||||
f"{context}\n\n"
|
||||
f"Based on the documents above, please provide a comprehensive answer. "
|
||||
f"Cite the document numbers when referencing specific information."
|
||||
@@ -495,17 +507,18 @@ async def nc_notes_semantic_search_answer(
|
||||
|
||||
### Import Updates
|
||||
|
||||
Add to top of `nextcloud_mcp_server/server/notes.py`:
|
||||
Add to top of `nextcloud_mcp_server/server/semantic.py`:
|
||||
|
||||
```python
|
||||
from mcp.types import ModelHint, ModelPreferences, SamplingMessage, TextContent
|
||||
```
|
||||
|
||||
Add to `nextcloud_mcp_server/models/notes.py` exports:
|
||||
Add to `nextcloud_mcp_server/models/semantic.py` exports:
|
||||
|
||||
```python
|
||||
__all__ = [
|
||||
# ... existing exports
|
||||
"SemanticSearchResult",
|
||||
"SemanticSearchResponse",
|
||||
"SamplingSearchResponse",
|
||||
]
|
||||
```
|
||||
@@ -619,12 +632,16 @@ __all__ = [
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] Create ADR-008 document (this file)
|
||||
- [ ] Add `SamplingSearchResponse` model to `nextcloud_mcp_server/models/notes.py`
|
||||
- [ ] Implement `nc_notes_semantic_search_answer` tool in `nextcloud_mcp_server/server/notes.py`
|
||||
- [ ] Create `nextcloud_mcp_server/models/semantic.py` for semantic search models
|
||||
- [ ] Add `SamplingSearchResponse` model to `nextcloud_mcp_server/models/semantic.py`
|
||||
- [ ] Create `nextcloud_mcp_server/server/semantic.py` for semantic search tools
|
||||
- [ ] Implement `nc_semantic_search_answer` tool in `nextcloud_mcp_server/server/semantic.py`
|
||||
- [ ] Add MCP sampling type imports (`SamplingMessage`, `TextContent`, etc.)
|
||||
- [ ] Write unit tests with mocked sampling (`tests/unit/server/test_notes.py`)
|
||||
- [ ] Write unit tests with mocked sampling (`tests/unit/server/test_semantic.py`)
|
||||
- [ ] Create integration tests (`tests/integration/test_sampling.py`)
|
||||
- [ ] Update `README.md` with new tool documentation
|
||||
- [ ] Update `README.md` with new tool documentation in dedicated Semantic Search section
|
||||
- [ ] Update `CLAUDE.md` with sampling pattern guidance
|
||||
- [ ] Test with MCP client supporting sampling (Claude Desktop, MCP Inspector with callbacks)
|
||||
- [ ] Document client requirements and fallback behavior
|
||||
- [ ] Update oauth-architecture.md to add semantic:read scope
|
||||
- [ ] Create ADR-009 to document semantic:read scope decision
|
||||
|
||||
@@ -0,0 +1,268 @@
|
||||
# ADR-009: Generic `semantic:read` OAuth Scope for Multi-App Vector Search
|
||||
|
||||
**Status**: Proposed
|
||||
**Date**: 2025-01-11
|
||||
**Depends On**: ADR-007 (Background Vector Sync), ADR-008 (MCP Sampling for Semantic Search)
|
||||
|
||||
## Context
|
||||
|
||||
ADR-007 established a background vector synchronization architecture that indexes content from multiple Nextcloud apps (notes, calendar events, deck cards, files, contacts) into a unified vector database. ADR-008 introduced semantic search tools (`nc_semantic_search`, `nc_semantic_search_answer`) that query this vector database and use MCP sampling to generate natural language answers.
|
||||
|
||||
The question is: **What OAuth scopes should protect semantic search operations?**
|
||||
|
||||
### Option 1: App-Specific Scopes
|
||||
|
||||
Require users to have scopes for each app they want to search:
|
||||
|
||||
```python
|
||||
@mcp.tool()
|
||||
@require_scopes("notes:read", "calendar:read", "deck:read", "files:read", "contacts:read")
|
||||
async def nc_semantic_search(query: str, ctx: Context) -> SemanticSearchResponse:
|
||||
"""Search across all indexed apps"""
|
||||
```
|
||||
|
||||
**Advantages**:
|
||||
- Granular control - users explicitly consent to searching each app
|
||||
- Aligns with app-specific authorization model
|
||||
- Clear security boundary - can only search apps you can access
|
||||
|
||||
**Disadvantages**:
|
||||
- **Brittle user experience**: If a user grants only `notes:read` but the tool requires all 5 scopes, the tool becomes invisible/unusable
|
||||
- **All-or-nothing enforcement**: Can't search notes alone - must grant all scopes or none
|
||||
- **Poor progressive consent**: User can't start with notes search and later add calendar
|
||||
- **Scope inflation**: Every new app adds another required scope
|
||||
- **Mismatched semantics**: User thinks "I want to search my notes" but must grant calendar, deck, files, contacts just to make the tool appear
|
||||
|
||||
### Option 2: Single Generic Scope (Chosen)
|
||||
|
||||
Introduce a new semantic search-specific scope:
|
||||
|
||||
```python
|
||||
@mcp.tool()
|
||||
@require_scopes("semantic:read")
|
||||
async def nc_semantic_search(query: str, ctx: Context) -> SemanticSearchResponse:
|
||||
"""Search across all indexed apps"""
|
||||
```
|
||||
|
||||
**Advantages**:
|
||||
- **Simple authorization**: One scope grants semantic search capability
|
||||
- **Progressive enablement**: User grants `semantic:read`, searches notes initially, then enables calendar indexing later
|
||||
- **Logical grouping**: Semantic search is a cross-app feature, deserving its own scope
|
||||
- **Future-proof**: New apps can be added to vector sync without changing OAuth scopes
|
||||
- **Matches user mental model**: "I want semantic search" → grant `semantic:read` (not "I want semantic search" → grant 5 unrelated app scopes)
|
||||
|
||||
**Considerations**:
|
||||
- User could search apps they can't directly access via app-specific tools
|
||||
- **Mitigation**: Dual-phase authorization (Phase 1: scope check passes with `semantic:read`, Phase 2: verify user can access each returned document via app-specific permissions)
|
||||
- Less granular than app-specific scopes
|
||||
- **Counterpoint**: Semantic search is inherently cross-app - forcing per-app authorization defeats its purpose
|
||||
|
||||
### Option 3: Hybrid Approach (Rejected)
|
||||
|
||||
Support both: semantic search works with either `semantic:read` OR all app-specific scopes:
|
||||
|
||||
```python
|
||||
@mcp.tool()
|
||||
@require_scopes("semantic:read", alternative_scopes=["notes:read", "calendar:read", ...])
|
||||
async def nc_semantic_search(query: str, ctx: Context) -> SemanticSearchResponse:
|
||||
"""Search across all indexed apps"""
|
||||
```
|
||||
|
||||
**Rejected Because**:
|
||||
- Adds complexity to scope validation logic
|
||||
- Unclear to users which scopes they should grant
|
||||
- Alternative scopes still suffer from all-or-nothing problem
|
||||
- No significant benefit over Option 2 with dual-phase authorization
|
||||
|
||||
## Decision
|
||||
|
||||
We will introduce two new OAuth scopes specifically for semantic search operations:
|
||||
|
||||
- **`semantic:read`**: Query vector database, perform semantic search, generate answers
|
||||
- **`semantic:write`**: Enable/disable background vector synchronization, manage indexing settings
|
||||
|
||||
These scopes are **independent** of app-specific scopes (notes:read, calendar:read, etc.).
|
||||
|
||||
### Tool Scope Assignments
|
||||
|
||||
**Read Operations**:
|
||||
```python
|
||||
@mcp.tool()
|
||||
@require_scopes("semantic:read")
|
||||
async def nc_semantic_search(query: str, ctx: Context, limit: int = 10, score_threshold: float = 0.7) -> SemanticSearchResponse:
|
||||
"""Semantic search across all indexed Nextcloud apps"""
|
||||
|
||||
@mcp.tool()
|
||||
@require_scopes("semantic:read")
|
||||
async def nc_semantic_search_answer(query: str, ctx: Context, limit: int = 5, max_answer_tokens: int = 500) -> SamplingSearchResponse:
|
||||
"""Semantic search with LLM-generated answer via MCP sampling"""
|
||||
|
||||
@mcp.tool()
|
||||
@require_scopes("semantic:read")
|
||||
async def nc_get_vector_sync_status(ctx: Context) -> VectorSyncStatusResponse:
|
||||
"""Get current vector synchronization status (indexed count, pending count, status)"""
|
||||
```
|
||||
|
||||
**Write Operations**:
|
||||
```python
|
||||
@mcp.tool()
|
||||
@require_scopes("semantic:write")
|
||||
async def nc_enable_vector_sync(ctx: Context) -> VectorSyncResponse:
|
||||
"""Enable background vector synchronization for this user"""
|
||||
|
||||
@mcp.tool()
|
||||
@require_scopes("semantic:write")
|
||||
async def nc_disable_vector_sync(ctx: Context) -> VectorSyncResponse:
|
||||
"""Disable background vector synchronization"""
|
||||
```
|
||||
|
||||
### Dual-Phase Authorization
|
||||
|
||||
To ensure users can only access documents they have permission to view, semantic search implements **dual-phase authorization**:
|
||||
|
||||
**Phase 1: Scope Check** (MCP Server)
|
||||
- User must have `semantic:read` scope to call semantic search tools
|
||||
- This grants permission to query the vector database
|
||||
|
||||
**Phase 2: Document Verification** (Per-Result Filtering)
|
||||
- For each returned document, verify user has access via app-specific permissions
|
||||
- Uses `DocumentVerifier` interface per app:
|
||||
- Notes: Call `/apps/notes/api/v1/notes/{id}` - if 404/403, exclude from results
|
||||
- Calendar: Call `/remote.php/dav/calendars/username/calendar/event.ics` - if 404/403, exclude
|
||||
- Deck: Call `/apps/deck/api/v1.0/boards/{board_id}/stacks/{stack_id}/cards/{card_id}` - if 404/403, exclude
|
||||
- Files: Call `/remote.php/dav/files/username/path` with PROPFIND - if 404/403, exclude
|
||||
- Contacts: Call `/remote.php/dav/addressbooks/username/addressbook/contact.vcf` - if 404/403, exclude
|
||||
|
||||
This two-phase approach ensures:
|
||||
1. Semantic search is a **distinct capability** (like "global search") requiring explicit consent
|
||||
2. Results are **filtered** to only include documents the user can access
|
||||
3. No privilege escalation - users can't discover content they shouldn't see
|
||||
|
||||
**Implementation**: See ADR-007 Phase 3 (Document Verification) and `DocumentVerifier` interface.
|
||||
|
||||
### Scope Discovery
|
||||
|
||||
The new scopes will be:
|
||||
- **Advertised** via PRM endpoint (`/.well-known/oauth-protected-resource/mcp`)
|
||||
- **Dynamically discovered** from `@require_scopes` decorators on semantic search tools
|
||||
- **Documented** in OAuth architecture (oauth-architecture.md)
|
||||
- **Included** in default client registration scopes
|
||||
|
||||
## Consequences
|
||||
|
||||
### Benefits
|
||||
|
||||
**User Experience**:
|
||||
- Simple authorization: one scope for semantic search capability
|
||||
- Progressive enablement: grant `semantic:read`, enable indexing for apps later
|
||||
- Natural mental model: "semantic search" is a distinct feature deserving its own scope
|
||||
|
||||
**Security**:
|
||||
- Dual-phase authorization prevents privilege escalation
|
||||
- Users explicitly consent to cross-app search capability
|
||||
- Per-document verification ensures users only see accessible content
|
||||
|
||||
**Maintainability**:
|
||||
- Adding new apps to vector sync doesn't require OAuth scope changes
|
||||
- Clear separation between app access (notes:read) and search capability (semantic:read)
|
||||
- Logical grouping of related operations (search, sync status, enable/disable)
|
||||
|
||||
**Future-Proof**:
|
||||
- Can add new document types without breaking existing OAuth flows
|
||||
- Supports future semantic features (recommendations, clustering) under same scope
|
||||
- Aligns with potential future Nextcloud semantic capabilities
|
||||
|
||||
### Trade-offs
|
||||
|
||||
**Less Granular Than App-Specific Scopes**:
|
||||
- User can't grant "semantic search notes only"
|
||||
- Semantic search is all-or-nothing across enabled apps
|
||||
- **Mitigation**: Dual-phase verification ensures users only see documents they can access
|
||||
|
||||
**New Scope to Learn**:
|
||||
- Users must understand `semantic:read` is distinct from app scopes
|
||||
- MCP clients must present scope clearly during consent
|
||||
- **Mitigation**: Clear scope descriptions in OAuth consent UI and documentation
|
||||
|
||||
**Backend Complexity**:
|
||||
- Requires dual-phase authorization implementation
|
||||
- DocumentVerifier interface needed for each app
|
||||
- **Benefit**: Enforces proper security regardless of scope model
|
||||
|
||||
### Migration Impact
|
||||
|
||||
**Breaking Change**: Existing deployments using notes-specific semantic search will break.
|
||||
|
||||
**Before (OLD - Breaking)**:
|
||||
```python
|
||||
@mcp.tool()
|
||||
@require_scopes("notes:read")
|
||||
async def nc_notes_semantic_search(query: str, ctx: Context) -> SemanticSearchResponse:
|
||||
"""Semantic search notes"""
|
||||
```
|
||||
|
||||
**After (NEW)**:
|
||||
```python
|
||||
@mcp.tool()
|
||||
@require_scopes("semantic:read")
|
||||
async def nc_semantic_search(query: str, ctx: Context) -> SemanticSearchResponse:
|
||||
"""Semantic search across all apps"""
|
||||
```
|
||||
|
||||
**Migration Path**:
|
||||
1. Deploy server with new `semantic:read` scope
|
||||
2. Users re-authenticate, granting `semantic:read` scope
|
||||
3. Semantic search tools become visible/usable again
|
||||
4. **No data loss**: Vector database and indexed documents remain unchanged
|
||||
|
||||
**Backward Compatibility**: None. This is an intentional breaking change to correct the scope model before broader adoption.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
### Keep Notes-Specific Scopes
|
||||
|
||||
**Approach**: Continue using `notes:read` for semantic search, even when searching other apps.
|
||||
|
||||
**Rejected Because**:
|
||||
- Semantically incorrect - searching calendar events is not "reading notes"
|
||||
- Confuses users - why does searching calendar require notes:read?
|
||||
- Doesn't scale - what scope for multi-app search?
|
||||
|
||||
### Create Per-App Semantic Scopes
|
||||
|
||||
**Approach**: Introduce `notes:semantic`, `calendar:semantic`, `deck:semantic`, etc.
|
||||
|
||||
**Rejected Because**:
|
||||
- Scope proliferation - doubles the number of scopes
|
||||
- Defeats purpose of unified vector search
|
||||
- Users would need to grant 5+ scopes for cross-app search
|
||||
- No clear benefit over dual-phase authorization with `semantic:read`
|
||||
|
||||
### Require All App Scopes (Already Rejected in Option 1)
|
||||
|
||||
**Approach**: Require `notes:read AND calendar:read AND deck:read AND files:read AND contacts:read`
|
||||
|
||||
**Rejected Because**: Unusable UX (see Option 1 disadvantages above)
|
||||
|
||||
## Related Decisions
|
||||
|
||||
**ADR-007**: Background Vector Sync provides the indexing architecture that semantic scopes protect. The DocumentVerifier interface from ADR-007 Phase 3 implements dual-phase authorization.
|
||||
|
||||
**ADR-008**: MCP Sampling for semantic search uses `semantic:read` to protect the sampling-enhanced search tool.
|
||||
|
||||
**ADR-004**: Progressive Consent architecture supports users granting `semantic:read` initially, then enabling per-app indexing via `semantic:write` (enable_vector_sync with app selection).
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] Create ADR-009 document (this file)
|
||||
- [ ] Update `oauth-architecture.md` to document `semantic:read` and `semantic:write` scopes ✅
|
||||
- [ ] Update `README.md` to show Semantic Search as separate tool category ✅
|
||||
- [ ] Update ADR-007 to reference `semantic:*` scopes instead of `sync:*` ✅
|
||||
- [ ] Update ADR-008 to use `semantic:read` instead of `notes:read` ✅
|
||||
- [ ] Implement DocumentVerifier interface for all apps (notes, calendar, deck, files, contacts)
|
||||
- [ ] Update semantic search tools to use `@require_scopes("semantic:read")`
|
||||
- [ ] Update vector sync tools to use `@require_scopes("semantic:write")`
|
||||
- [ ] Add dual-phase authorization to semantic search implementation
|
||||
- [ ] Test OAuth flow with `semantic:read` scope
|
||||
- [ ] Update scope discovery in PRM endpoint
|
||||
- [ ] Document migration path for existing deployments
|
||||
@@ -634,6 +634,12 @@ The server supports the following OAuth scopes, organized by Nextcloud app:
|
||||
- `sharing:read` - List shares and read share information
|
||||
- `sharing:write` - Create, update, and delete shares
|
||||
|
||||
#### Semantic Search (Multi-App Vector Database)
|
||||
- `semantic:read` - Query vector database, perform semantic search across all indexed Nextcloud apps (notes, calendar, deck, files, contacts)
|
||||
- `semantic:write` - Enable/disable background vector synchronization, manage indexing settings
|
||||
|
||||
> **Note**: Semantic search scopes provide access to the vector database that indexes content across **all** Nextcloud apps. Unlike app-specific scopes (e.g., `notes:read`), semantic scopes grant cross-app search capabilities powered by background vector synchronization (ADR-007).
|
||||
|
||||
### Scope Discovery
|
||||
|
||||
The MCP server provides scope discovery through two mechanisms:
|
||||
|
||||
Reference in New Issue
Block a user