feat: implement MCP sampling for semantic search RAG (ADR-008)
Add nc_notes_semantic_search_answer tool that combines semantic search with MCP sampling to generate natural language answers from retrieved Nextcloud Notes. This enables Retrieval-Augmented Generation (RAG) patterns without requiring a server-side LLM. Key features: - Client-side LLM generation via ctx.session.create_message() - Graceful fallback when sampling unavailable - Proper source citations in generated answers - No results optimization (skips sampling when no docs found) - Comprehensive unit and integration tests Implementation details: - SamplingSearchResponse model with generated_answer and sources - Fixed prompt template with document context and citation instructions - Model preferences hint Claude Sonnet for balanced performance - Falls back to returning documents without answer on sampling failure Updates: - Add ADR-008 documenting sampling architecture decision - Add MCP sampling pattern guidance to CLAUDE.md - Update README.md and docs/notes.md (7 → 9 tools) - Add 4 unit tests and 6 integration tests 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
@@ -224,6 +224,82 @@ docker compose exec db mariadb -u root -ppassword nextcloud -e \
|
||||
|
||||
**Testing**: Extract `data["results"]` from MCP responses, not `data` directly.
|
||||
|
||||
## MCP Sampling for RAG (ADR-008)
|
||||
|
||||
**What is MCP Sampling?**
|
||||
MCP sampling allows servers to request LLM completions from their clients. This enables Retrieval-Augmented Generation (RAG) patterns where the server retrieves context and the client's LLM generates answers.
|
||||
|
||||
**When to use sampling:**
|
||||
- Generating natural language answers from retrieved documents
|
||||
- Synthesizing information from multiple sources
|
||||
- Creating summaries with citations
|
||||
|
||||
**Implementation Pattern** (see ADR-008 for details):
|
||||
|
||||
```python
|
||||
from mcp.types import ModelHint, ModelPreferences, SamplingMessage, TextContent
|
||||
|
||||
@mcp.tool()
|
||||
@require_scopes("notes:read")
|
||||
async def nc_notes_semantic_search_answer(
|
||||
query: str, ctx: Context, limit: int = 5, max_answer_tokens: int = 500
|
||||
) -> SamplingSearchResponse:
|
||||
# 1. Retrieve documents
|
||||
search_response = await nc_notes_semantic_search(query, ctx, limit)
|
||||
|
||||
# 2. Check for no results (don't waste sampling call)
|
||||
if not search_response.results:
|
||||
return SamplingSearchResponse(
|
||||
query=query,
|
||||
generated_answer="No relevant documents found.",
|
||||
sources=[], total_found=0, success=True
|
||||
)
|
||||
|
||||
# 3. Construct prompt with retrieved context
|
||||
prompt = f"{query}\n\nDocuments:\n{format_sources(search_response.results)}\n\nProvide answer with citations."
|
||||
|
||||
# 4. Request LLM completion via sampling
|
||||
try:
|
||||
result = await ctx.session.create_message(
|
||||
messages=[SamplingMessage(role="user", content=TextContent(type="text", text=prompt))],
|
||||
max_tokens=max_answer_tokens,
|
||||
temperature=0.7,
|
||||
model_preferences=ModelPreferences(
|
||||
hints=[ModelHint(name="claude-3-5-sonnet")],
|
||||
intelligencePriority=0.8,
|
||||
speedPriority=0.5,
|
||||
),
|
||||
include_context="thisServer",
|
||||
)
|
||||
|
||||
return SamplingSearchResponse(
|
||||
query=query,
|
||||
generated_answer=result.content.text,
|
||||
sources=search_response.results,
|
||||
model_used=result.model,
|
||||
stop_reason=result.stopReason,
|
||||
success=True
|
||||
)
|
||||
except Exception as e:
|
||||
# Fallback: Return documents without generated answer
|
||||
return SamplingSearchResponse(
|
||||
query=query,
|
||||
generated_answer=f"[Sampling unavailable: {e}]\n\nFound {len(search_response.results)} documents.",
|
||||
sources=search_response.results,
|
||||
search_method="semantic_sampling_fallback",
|
||||
success=True
|
||||
)
|
||||
```
|
||||
|
||||
**Key Points**:
|
||||
- **No server-side LLM**: Server has no API keys, client controls which model is used
|
||||
- **Graceful degradation**: Tool always returns useful results even if sampling fails
|
||||
- **User control**: MCP clients SHOULD prompt users to approve sampling requests
|
||||
- **No results optimization**: Skip sampling call when no documents found
|
||||
- **Fixed prompts**: Prompts are not user-configurable to avoid injection risks
|
||||
|
||||
**Reference**: See `nc_notes_semantic_search_answer` in `nextcloud_mcp_server/server/notes.py:517` and ADR-008 for complete implementation.
|
||||
|
||||
## Testing Best Practices (MANDATORY)
|
||||
|
||||
### Always Run Tests
|
||||
|
||||
@@ -19,7 +19,7 @@ The Nextcloud MCP (Model Context Protocol) server allows Large Language Models l
|
||||
| **Deployment** | Standalone (Docker, VM, K8s) | Inside Nextcloud (ExApp via AppAPI) |
|
||||
| **Primary Users** | Claude Code, IDEs, external developers | Nextcloud end users via Assistant app |
|
||||
| **Authentication** | OAuth2/OIDC or Basic Auth | Session-based (integrated) |
|
||||
| **Notes Support** | ✅ Full CRUD + search (7 tools) | ❌ Not implemented |
|
||||
| **Notes Support** | ✅ Full CRUD + search + semantic search (9 tools) | ❌ Not implemented |
|
||||
| **Calendar** | ✅ Full CalDAV + tasks (20+ tools) | ✅ Events, free/busy, tasks (4 tools) |
|
||||
| **Contacts** | ✅ Full CardDAV (8 tools) | ✅ Find person, current user (2 tools) |
|
||||
| **Files (WebDAV)** | ✅ Full filesystem access (12 tools) | ✅ Read, folder tree, sharing (3 tools) |
|
||||
@@ -200,7 +200,7 @@ For a complete list of all supported OAuth scopes and their descriptions, see [O
|
||||
|
||||
| App | Tools | Read Scope | Write Scope | Operations |
|
||||
|-----|-------|-----------|-------------|------------|
|
||||
| **Notes** | 7 | `notes:read` | `notes:write` | Create, read, update, delete, search notes |
|
||||
| **Notes** | 9 | `notes:read` | `notes:write` | Create, read, update, delete, search notes (keyword + semantic) |
|
||||
| **Calendar** | 20+ | `calendar:read` `todo:read` | `calendar:write` `todo:write` | Events, todos (tasks), calendars, recurring events, attendees |
|
||||
| **Contacts** | 8 | `contacts:read` | `contacts:write` | Create, read, update, delete contacts and address books |
|
||||
| **Files (WebDAV)** | 12 | `files:read` | `files:write` | List, read, upload, delete, move files; **OCR/document processing** |
|
||||
|
||||
@@ -0,0 +1,630 @@
|
||||
# ADR-008: MCP Sampling for Semantic Search Enhancement
|
||||
|
||||
**Status**: Proposed
|
||||
**Date**: 2025-01-11
|
||||
**Depends On**: ADR-007 (Background Vector Sync)
|
||||
|
||||
## Context
|
||||
|
||||
ADR-007 established a background synchronization architecture that maintains a vector database of Nextcloud content, enabling semantic search via the `nc_notes_semantic_search` tool. This tool returns a list of relevant documents with excerpts, similarity scores, and metadata—providing the raw materials for answering user questions.
|
||||
|
||||
However, users typically don't want a list of documents—they want answers to their questions. When a user asks "What are my project goals?" or "What did I learn about Python last month?", they expect a natural language response that synthesizes information from multiple sources, not a ranked list of note excerpts. This is the pattern of Retrieval-Augmented Generation (RAG): retrieve relevant context, then generate a cohesive answer.
|
||||
|
||||
The challenge is: who should generate the answer, and how?
|
||||
|
||||
**Option 1: Server-side LLM**
|
||||
The MCP server could maintain its own LLM connection (OpenAI API, Ollama, etc.), construct prompts from retrieved documents, and return generated answers directly. This approach has significant drawbacks:
|
||||
|
||||
- **Duplicate infrastructure**: MCP clients (like Claude Desktop) already have LLM capabilities. The server would duplicate this with its own LLM integration, API keys, and configuration.
|
||||
- **Cost and billing**: The server operator bears LLM costs for all users, creating billing and quota management challenges.
|
||||
- **Limited model choice**: Users are locked into whatever LLM the server configures. They cannot choose their preferred model or provider.
|
||||
- **Privacy concerns**: User queries and document contents flow through a server-controlled LLM, creating a potential privacy boundary.
|
||||
- **Configuration complexity**: Server operators must configure embedding services (for search) AND generation models (for answers), each with different API keys, rate limits, and failure modes.
|
||||
|
||||
**Option 2: Return documents, let client generate**
|
||||
The server could simply return retrieved documents and rely on the MCP client's existing LLM to generate answers. The user would call `nc_notes_semantic_search`, receive documents, and then the client would include those documents in its context when responding to the user's original question. This approach also has limitations:
|
||||
|
||||
- **Context window waste**: The client must include all document content in its context window, even if only small excerpts are relevant. For 5-10 documents, this can consume significant context space.
|
||||
- **Inconsistent behavior**: Whether the client synthesizes an answer or just displays documents depends on the client's implementation and the user's conversational style. There's no guaranteed answer generation.
|
||||
- **Poor citations**: The client may generate an answer but fail to cite which specific documents were used, making it hard to verify claims.
|
||||
- **User confusion**: Users see a tool that returns "search results" rather than "answers", requiring them to explicitly ask for synthesis.
|
||||
|
||||
**Option 3: MCP Sampling**
|
||||
The Model Context Protocol specification includes a **sampling** capability that allows MCP servers to request LLM completions from their clients. The server constructs a prompt with retrieved context, sends it to the client via `sampling/createMessage`, and the client's LLM generates a response that the server can return as a tool result.
|
||||
|
||||
This approach combines the best of both options:
|
||||
|
||||
- **No server-side LLM**: The server has no API keys, no LLM configuration, no billing concerns.
|
||||
- **User choice**: The MCP client controls which LLM is used (Claude, GPT-4, local Ollama) and who pays for it.
|
||||
- **User transparency**: MCP clients SHOULD present sampling requests to users for approval, making it clear when the server is requesting an LLM call.
|
||||
- **Consistent citations**: The server constructs a prompt that explicitly includes document references, ensuring generated answers cite sources.
|
||||
- **Single tool call**: Users call one tool (`nc_notes_semantic_search_answer`) and receive a complete answer with citations—no multi-turn conversation needed.
|
||||
|
||||
The sampling approach shifts responsibility appropriately: the MCP server is responsible for information retrieval and context construction (its expertise), while the MCP client is responsible for LLM access and user preferences (its expertise). This follows the MCP design philosophy of separating concerns between servers (data access) and clients (user interaction).
|
||||
|
||||
However, sampling introduces new considerations:
|
||||
|
||||
**Client compatibility**: Not all MCP clients implement sampling. The server must gracefully degrade when sampling is unavailable, falling back to returning documents without generated answers.
|
||||
|
||||
**Latency**: Sampling adds a full round-trip to the client and back, plus LLM generation time. A typical flow involves: (1) client calls tool, (2) server retrieves documents, (3) server requests sampling from client, (4) client generates answer, (5) server returns answer to client. This can take 2-5 seconds depending on LLM speed, compared to 100-500ms for document retrieval alone.
|
||||
|
||||
**User approval**: MCP clients SHOULD prompt users to approve sampling requests, allowing users to review the prompt before sending it to their LLM. This is a privacy and security feature (prevents servers from making arbitrary LLM requests) but adds interaction friction.
|
||||
|
||||
**Prompt engineering**: The server must construct effective prompts that guide the LLM to generate useful, well-cited answers. Unlike Option 1 where the server controls the LLM directly, the server has less control over how the prompt is interpreted.
|
||||
|
||||
Despite these considerations, MCP sampling provides the most principled solution for RAG-enhanced semantic search. It respects the client-server boundary, avoids duplicate infrastructure, and delivers the user experience users expect from semantic search tools.
|
||||
|
||||
This ADR proposes adding a new tool, `nc_notes_semantic_search_answer`, that uses MCP sampling to generate natural language answers from retrieved Nextcloud content.
|
||||
|
||||
## Decision
|
||||
|
||||
We will implement a new MCP tool `nc_notes_semantic_search_answer` that retrieves relevant documents via vector similarity search and uses MCP sampling to generate natural language answers. The tool will construct a prompt that includes the user's original query and excerpts from retrieved documents, request an LLM completion via `ctx.session.create_message()`, and return the generated answer along with source citations.
|
||||
|
||||
The existing `nc_notes_semantic_search` tool will remain unchanged, providing users with a choice: call the original tool for raw document results, or call the new sampling-enhanced tool for generated answers. This dual-tool approach respects different use cases—some users want to browse documents, others want direct answers.
|
||||
|
||||
### API Design
|
||||
|
||||
**Tool Signature**:
|
||||
```python
|
||||
@mcp.tool()
|
||||
@require_scopes("notes:read")
|
||||
async def nc_notes_semantic_search_answer(
|
||||
query: str,
|
||||
ctx: Context,
|
||||
limit: int = 5,
|
||||
score_threshold: float = 0.7,
|
||||
max_answer_tokens: int = 500,
|
||||
) -> SamplingSearchResponse
|
||||
```
|
||||
|
||||
**Parameters**:
|
||||
- `query`: The user's natural language question
|
||||
- `ctx`: MCP context for session access
|
||||
- `limit`: Maximum documents to retrieve (default 5)
|
||||
- `score_threshold`: Minimum similarity score 0-1 (default 0.7)
|
||||
- `max_answer_tokens`: Maximum tokens for generated answer (default 500)
|
||||
|
||||
**Response Model**:
|
||||
```python
|
||||
class SamplingSearchResponse(BaseResponse):
|
||||
query: str # Original user query
|
||||
generated_answer: str # LLM-generated answer
|
||||
sources: list[SemanticSearchResult] # Supporting documents
|
||||
total_found: int # Total matching documents
|
||||
search_method: str = "semantic_sampling"
|
||||
model_used: str | None = None # Model that generated answer
|
||||
stop_reason: str | None = None # Why generation stopped
|
||||
```
|
||||
|
||||
The response includes both the generated answer (for direct user consumption) and the source documents (for verification and citation). The `model_used` field records which LLM generated the answer, allowing users to understand which model provided the response.
|
||||
|
||||
### Sampling API Usage
|
||||
|
||||
The tool uses the MCP Python SDK's `ServerSession.create_message()` API:
|
||||
|
||||
```python
|
||||
from mcp.types import SamplingMessage, TextContent, ModelPreferences, ModelHint
|
||||
|
||||
# Construct prompt with retrieved context
|
||||
prompt = (
|
||||
f"{query}\n\n"
|
||||
f"Here are relevant documents from Nextcloud Notes:\n\n"
|
||||
f"{context}\n\n"
|
||||
f"Based on the documents above, please provide a comprehensive answer. "
|
||||
f"Cite the document numbers when referencing specific information."
|
||||
)
|
||||
|
||||
# Request LLM completion via MCP sampling
|
||||
sampling_result = await ctx.session.create_message(
|
||||
messages=[
|
||||
SamplingMessage(
|
||||
role="user",
|
||||
content=TextContent(type="text", text=prompt),
|
||||
)
|
||||
],
|
||||
max_tokens=max_answer_tokens,
|
||||
temperature=0.7,
|
||||
model_preferences=ModelPreferences(
|
||||
hints=[ModelHint(name="claude-3-5-sonnet")],
|
||||
intelligencePriority=0.8,
|
||||
speedPriority=0.5,
|
||||
),
|
||||
include_context="thisServer",
|
||||
)
|
||||
|
||||
# Extract answer from response
|
||||
if sampling_result.content.type == "text":
|
||||
generated_answer = sampling_result.content.text
|
||||
```
|
||||
|
||||
**Key parameters**:
|
||||
- `messages`: Chat-style messages with role ("user" or "assistant") and content
|
||||
- `max_tokens`: Limits response length to control costs and latency
|
||||
- `temperature`: 0.7 balances creativity with consistency for factual answers
|
||||
- `model_preferences`: Hints suggest Claude Sonnet for balanced intelligence/speed
|
||||
- `include_context`: "thisServer" includes MCP server context in client's LLM call
|
||||
|
||||
The `include_context` parameter is particularly important. When set to "thisServer", the MCP client provides its LLM with context about the server's capabilities, tools, and resources. This allows the LLM to reference the Nextcloud MCP server when generating answers, creating more contextually appropriate responses. For example, the LLM might say "Based on your Nextcloud Notes..." rather than generic phrasing.
|
||||
|
||||
### Prompt Construction
|
||||
|
||||
The prompt construction follows a structured template:
|
||||
|
||||
```
|
||||
[User's original query]
|
||||
|
||||
Here are relevant documents from Nextcloud Notes:
|
||||
|
||||
[Document 1]
|
||||
Title: Project Kickoff Notes
|
||||
Category: Work
|
||||
Excerpt: The primary goal for Q1 2025 is to improve semantic search...
|
||||
Relevance Score: 0.92
|
||||
|
||||
[Document 2]
|
||||
Title: Meeting Notes - Jan 5
|
||||
Category: Work
|
||||
Excerpt: Team agreed on three key objectives...
|
||||
Relevance Score: 0.88
|
||||
|
||||
Based on the documents above, please provide a comprehensive answer.
|
||||
Cite the document numbers when referencing specific information.
|
||||
```
|
||||
|
||||
This structure ensures:
|
||||
- The user's original query is preserved verbatim
|
||||
- Documents are clearly delineated and numbered for citation
|
||||
- Metadata (title, category, score) provides context
|
||||
- Explicit instruction to cite sources encourages proper attribution
|
||||
|
||||
The prompt is intentionally simple and fixed (not configurable). Allowing users to customize the prompt would complicate the API and introduce prompt injection risks. The fixed structure ensures consistent, well-cited answers across all users.
|
||||
|
||||
### Fallback Behavior
|
||||
|
||||
Sampling may fail for several reasons:
|
||||
- Client doesn't support sampling (e.g., MCP Inspector without callbacks)
|
||||
- User declines the sampling request
|
||||
- Network errors during sampling round-trip
|
||||
- LLM generation errors
|
||||
|
||||
The tool handles all failures gracefully by falling back to returning documents without a generated answer:
|
||||
|
||||
```python
|
||||
try:
|
||||
sampling_result = await ctx.session.create_message(...)
|
||||
generated_answer = sampling_result.content.text
|
||||
except Exception as e:
|
||||
logger.warning(f"Sampling failed: {e}, returning search results only")
|
||||
generated_answer = (
|
||||
f"[Sampling unavailable: {str(e)}]\n\n"
|
||||
f"Found {total_found} relevant documents. Please review the sources below."
|
||||
)
|
||||
```
|
||||
|
||||
This ensures the tool always returns useful information—either a generated answer or the underlying documents—rather than failing completely. The user knows sampling was attempted (via the `[Sampling unavailable]` prefix) and can still access the retrieved context.
|
||||
|
||||
### No Results Handling
|
||||
|
||||
When semantic search finds no relevant documents (all below `score_threshold`), the tool returns a clear message without attempting sampling:
|
||||
|
||||
```python
|
||||
if not search_response.results:
|
||||
return SamplingSearchResponse(
|
||||
query=query,
|
||||
generated_answer="No relevant documents found in your Nextcloud Notes for this query.",
|
||||
sources=[],
|
||||
total_found=0,
|
||||
search_method="semantic_sampling",
|
||||
success=True,
|
||||
)
|
||||
```
|
||||
|
||||
This avoids wasting a sampling call (and user approval) when there's no content to base an answer on.
|
||||
|
||||
### User Experience Flow
|
||||
|
||||
**Typical successful flow**:
|
||||
1. User calls `nc_notes_semantic_search_answer` with query "What are my project goals?"
|
||||
2. Server retrieves 5 relevant notes via vector search
|
||||
3. Server constructs prompt with document excerpts
|
||||
4. Server sends `sampling/createMessage` request to client
|
||||
5. Client prompts user: "MCP server wants to generate an answer using these documents. Allow?"
|
||||
6. User approves (or client auto-approves based on configuration)
|
||||
7. Client sends prompt to LLM (Claude, GPT-4, etc.)
|
||||
8. LLM generates answer with citations: "Based on Document 1 and Document 3..."
|
||||
9. Client returns answer to server
|
||||
10. Server returns `SamplingSearchResponse` with answer and sources
|
||||
11. User sees complete answer with citations
|
||||
|
||||
**Fallback flow** (sampling unavailable):
|
||||
1-3. Same as above
|
||||
4. Server attempts `ctx.session.create_message()`
|
||||
5. Client raises exception: "Sampling not supported"
|
||||
6. Server catches exception, logs warning
|
||||
7. Server returns `SamplingSearchResponse` with documents and "[Sampling unavailable]" message
|
||||
8. User sees raw documents instead of generated answer
|
||||
|
||||
**No results flow**:
|
||||
1-2. Same as above but no documents match threshold
|
||||
3. Server returns `SamplingSearchResponse` with "No relevant documents" message
|
||||
4. No sampling attempted (no prompt sent)
|
||||
5. User sees clear "not found" message
|
||||
|
||||
This three-tier approach (answer → documents → error message) ensures users always receive useful feedback appropriate to the situation.
|
||||
|
||||
## Implementation
|
||||
|
||||
### Response Model
|
||||
|
||||
Add to `nextcloud_mcp_server/models/notes.py`:
|
||||
|
||||
```python
|
||||
from pydantic import Field
|
||||
|
||||
class SamplingSearchResponse(BaseResponse):
|
||||
"""Response from semantic search with LLM-generated answer via MCP sampling.
|
||||
|
||||
This response includes both a generated natural language answer (created by
|
||||
the MCP client's LLM via sampling) and the source documents used to generate
|
||||
that answer. Users can read the answer for quick information and review
|
||||
sources for verification and deeper exploration.
|
||||
|
||||
Attributes:
|
||||
query: The original user query
|
||||
generated_answer: Natural language answer generated by client's LLM
|
||||
sources: List of semantic search results used as context
|
||||
total_found: Total number of matching documents found
|
||||
search_method: Always "semantic_sampling" for this response type
|
||||
model_used: Name of model that generated the answer (e.g., "claude-3-5-sonnet")
|
||||
stop_reason: Why generation stopped ("endTurn", "maxTokens", etc.)
|
||||
"""
|
||||
|
||||
query: str = Field(..., description="Original user query")
|
||||
generated_answer: str = Field(
|
||||
...,
|
||||
description="LLM-generated answer based on retrieved documents"
|
||||
)
|
||||
sources: list[SemanticSearchResult] = Field(
|
||||
default_factory=list,
|
||||
description="Source documents with excerpts and relevance scores"
|
||||
)
|
||||
total_found: int = Field(..., description="Total matching documents")
|
||||
search_method: str = Field(
|
||||
default="semantic_sampling",
|
||||
description="Search method used"
|
||||
)
|
||||
model_used: str | None = Field(
|
||||
default=None,
|
||||
description="Model that generated the answer"
|
||||
)
|
||||
stop_reason: str | None = Field(
|
||||
default=None,
|
||||
description="Reason generation stopped"
|
||||
)
|
||||
```
|
||||
|
||||
### Tool Implementation
|
||||
|
||||
Add to `nextcloud_mcp_server/server/notes.py`:
|
||||
|
||||
```python
|
||||
import logging
|
||||
from mcp.types import ModelHint, ModelPreferences, SamplingMessage, TextContent
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
@require_scopes("notes:read")
|
||||
async def nc_notes_semantic_search_answer(
|
||||
query: str,
|
||||
ctx: Context,
|
||||
limit: int = 5,
|
||||
score_threshold: float = 0.7,
|
||||
max_answer_tokens: int = 500,
|
||||
) -> SamplingSearchResponse:
|
||||
"""
|
||||
Semantic search with LLM-generated answer using MCP sampling.
|
||||
|
||||
Retrieves relevant documents from Nextcloud Notes using vector similarity
|
||||
search, then uses MCP sampling to request the client's LLM to generate
|
||||
a natural language answer based on the retrieved context.
|
||||
|
||||
This tool combines the power of semantic search (finding relevant content)
|
||||
with LLM generation (synthesizing that content into coherent answers). The
|
||||
generated answer includes citations to specific documents, allowing users
|
||||
to verify claims and explore sources.
|
||||
|
||||
The LLM generation happens client-side via MCP sampling. The MCP client
|
||||
controls which model is used, who pays for it, and whether to prompt the
|
||||
user for approval. This keeps the server simple (no LLM API keys needed)
|
||||
while giving users full control over their LLM interactions.
|
||||
|
||||
Args:
|
||||
query: Natural language question to answer (e.g., "What are my project goals?")
|
||||
ctx: MCP context for session access
|
||||
limit: Maximum number of documents to retrieve (default: 5)
|
||||
score_threshold: Minimum similarity score 0-1 (default: 0.7)
|
||||
max_answer_tokens: Maximum tokens for generated answer (default: 500)
|
||||
|
||||
Returns:
|
||||
SamplingSearchResponse containing:
|
||||
- generated_answer: Natural language answer with citations
|
||||
- sources: List of documents with excerpts and relevance scores
|
||||
- model_used: Which model generated the answer
|
||||
- stop_reason: Why generation stopped
|
||||
|
||||
Note: Requires MCP client to support sampling. If sampling is unavailable,
|
||||
the tool gracefully degrades to returning documents with an explanation.
|
||||
The client may prompt the user to approve the sampling request.
|
||||
|
||||
Examples:
|
||||
>>> # Query about project goals
|
||||
>>> result = await nc_notes_semantic_search_answer(
|
||||
... query="What are my Q1 2025 project goals?",
|
||||
... ctx=ctx
|
||||
... )
|
||||
>>> print(result.generated_answer)
|
||||
"Based on Document 1 (Project Kickoff) and Document 3 (Q1 Planning),
|
||||
your main goals are: 1) Improve semantic search accuracy by 20%,
|
||||
2) Deploy new embedding model, 3) Reduce indexing latency..."
|
||||
|
||||
>>> # Query about learning
|
||||
>>> result = await nc_notes_semantic_search_answer(
|
||||
... query="What did I learn about Python async/await last month?",
|
||||
... ctx=ctx,
|
||||
... limit=10
|
||||
... )
|
||||
>>> len(result.sources) # Up to 10 documents
|
||||
7
|
||||
"""
|
||||
# 1. Retrieve relevant documents via existing semantic search
|
||||
search_response = await nc_notes_semantic_search(
|
||||
query=query,
|
||||
ctx=ctx,
|
||||
limit=limit,
|
||||
score_threshold=score_threshold,
|
||||
)
|
||||
|
||||
# 2. Handle no results case - don't waste a sampling call
|
||||
if not search_response.results:
|
||||
logger.debug(f"No documents found for query: {query}")
|
||||
return SamplingSearchResponse(
|
||||
query=query,
|
||||
generated_answer="No relevant documents found in your Nextcloud Notes for this query.",
|
||||
sources=[],
|
||||
total_found=0,
|
||||
search_method="semantic_sampling",
|
||||
success=True,
|
||||
)
|
||||
|
||||
# 3. Construct context from retrieved documents
|
||||
context_parts = []
|
||||
for idx, result in enumerate(search_response.results, 1):
|
||||
context_parts.append(
|
||||
f"[Document {idx}]\n"
|
||||
f"Title: {result.title}\n"
|
||||
f"Category: {result.category}\n"
|
||||
f"Excerpt: {result.excerpt}\n"
|
||||
f"Relevance Score: {result.score:.2f}\n"
|
||||
)
|
||||
|
||||
context = "\n".join(context_parts)
|
||||
|
||||
# 4. Construct prompt - reuse user's query, add context and instructions
|
||||
prompt = (
|
||||
f"{query}\n\n"
|
||||
f"Here are relevant documents from Nextcloud Notes:\n\n"
|
||||
f"{context}\n\n"
|
||||
f"Based on the documents above, please provide a comprehensive answer. "
|
||||
f"Cite the document numbers when referencing specific information."
|
||||
)
|
||||
|
||||
logger.debug(
|
||||
f"Requesting sampling for query: {query} "
|
||||
f"({len(search_response.results)} documents retrieved)"
|
||||
)
|
||||
|
||||
# 5. Request LLM completion via MCP sampling
|
||||
try:
|
||||
sampling_result = await ctx.session.create_message(
|
||||
messages=[
|
||||
SamplingMessage(
|
||||
role="user",
|
||||
content=TextContent(type="text", text=prompt),
|
||||
)
|
||||
],
|
||||
max_tokens=max_answer_tokens,
|
||||
temperature=0.7,
|
||||
model_preferences=ModelPreferences(
|
||||
hints=[ModelHint(name="claude-3-5-sonnet")],
|
||||
intelligencePriority=0.8,
|
||||
speedPriority=0.5,
|
||||
),
|
||||
include_context="thisServer",
|
||||
)
|
||||
|
||||
# 6. Extract answer from sampling response
|
||||
if sampling_result.content.type == "text":
|
||||
generated_answer = sampling_result.content.text
|
||||
else:
|
||||
# Handle non-text responses (shouldn't happen for text prompts)
|
||||
generated_answer = (
|
||||
f"Received non-text response of type: {sampling_result.content.type}"
|
||||
)
|
||||
logger.warning(
|
||||
f"Unexpected content type from sampling: {sampling_result.content.type}"
|
||||
)
|
||||
|
||||
logger.info(
|
||||
f"Sampling successful: model={sampling_result.model}, "
|
||||
f"stop_reason={sampling_result.stopReason}"
|
||||
)
|
||||
|
||||
return SamplingSearchResponse(
|
||||
query=query,
|
||||
generated_answer=generated_answer,
|
||||
sources=search_response.results,
|
||||
total_found=search_response.total_found,
|
||||
search_method="semantic_sampling",
|
||||
model_used=sampling_result.model,
|
||||
stop_reason=sampling_result.stopReason,
|
||||
success=True,
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
# Fallback: Return documents without generated answer
|
||||
logger.warning(
|
||||
f"Sampling failed ({type(e).__name__}: {e}), "
|
||||
f"returning search results only"
|
||||
)
|
||||
|
||||
return SamplingSearchResponse(
|
||||
query=query,
|
||||
generated_answer=(
|
||||
f"[Sampling unavailable: {str(e)}]\n\n"
|
||||
f"Found {search_response.total_found} relevant documents. "
|
||||
f"Please review the sources below."
|
||||
),
|
||||
sources=search_response.results,
|
||||
total_found=search_response.total_found,
|
||||
search_method="semantic_sampling_fallback",
|
||||
success=True,
|
||||
)
|
||||
```
|
||||
|
||||
### Import Updates
|
||||
|
||||
Add to top of `nextcloud_mcp_server/server/notes.py`:
|
||||
|
||||
```python
|
||||
from mcp.types import ModelHint, ModelPreferences, SamplingMessage, TextContent
|
||||
```
|
||||
|
||||
Add to `nextcloud_mcp_server/models/notes.py` exports:
|
||||
|
||||
```python
|
||||
__all__ = [
|
||||
# ... existing exports
|
||||
"SamplingSearchResponse",
|
||||
]
|
||||
```
|
||||
|
||||
## Consequences
|
||||
|
||||
### Benefits
|
||||
|
||||
**Improved User Experience**: Users receive direct answers to questions rather than lists of documents, matching expectations from modern AI interfaces.
|
||||
|
||||
**Proper Attribution**: Generated answers include citations to source documents, allowing users to verify claims and explore deeper.
|
||||
|
||||
**No Server-Side LLM**: The server has no LLM dependencies, API keys, or billing concerns. All LLM interactions happen client-side.
|
||||
|
||||
**User Control**: MCP clients control which model is used and may prompt users to approve sampling requests, maintaining transparency and user agency.
|
||||
|
||||
**Graceful Degradation**: The tool works even when sampling is unavailable, falling back to returning documents. Existing clients continue working without changes.
|
||||
|
||||
**Consistent Architecture**: Follows MCP's client-server separation: servers provide data access, clients provide user interaction and LLM capabilities.
|
||||
|
||||
### Limitations
|
||||
|
||||
**Sampling Support Required**: Not all MCP clients implement sampling. Users with basic clients see fallback behavior (documents without answers).
|
||||
|
||||
**Added Latency**: Sampling adds 2-5 seconds to tool execution due to client round-trip and LLM generation time. Users must wait longer for answers than for raw search results.
|
||||
|
||||
**User Approval Friction**: MCP clients SHOULD prompt users to approve sampling requests. This adds an extra interaction step before answers are generated.
|
||||
|
||||
**Limited Prompt Control**: The server cannot fully control how the client's LLM interprets the prompt. Different models may generate different quality answers.
|
||||
|
||||
**No Caching**: Each query requires a new sampling call. The server doesn't cache generated answers (clients may cache if they choose).
|
||||
|
||||
**Token Costs**: LLM generation consumes tokens from the user's or client's quota. Heavy users may incur costs or hit rate limits.
|
||||
|
||||
### Performance Characteristics
|
||||
|
||||
**Typical latency**:
|
||||
- Document retrieval (vector search): 100-300ms
|
||||
- Sampling round-trip (client communication): 50-200ms
|
||||
- LLM generation (client-side): 1-4 seconds
|
||||
- **Total**: 2-5 seconds end-to-end
|
||||
|
||||
**Throughput**: Sampling is fully async. The server can handle multiple concurrent sampling requests (limited by MCP client's concurrency, not server capacity).
|
||||
|
||||
**Resource usage**: Minimal server-side. No GPU, no LLM model loading, no large memory requirements. Sampling happens entirely client-side.
|
||||
|
||||
### Security Considerations
|
||||
|
||||
**Prompt Injection Risk**: If user queries contain adversarial text designed to manipulate LLM behavior, those queries are included verbatim in the sampling prompt. Mitigation: The structured prompt format and explicit instructions ("based on documents above") constrain LLM behavior.
|
||||
|
||||
**Data Privacy**: User queries and document excerpts are sent to the client's LLM. For cloud LLMs (OpenAI, Anthropic), this means data leaves the server's control. Mitigation: MCP clients SHOULD present sampling requests to users for approval, making data flows transparent. Users choose their LLM provider.
|
||||
|
||||
**Sampling Abuse**: A malicious server could spam sampling requests to drain user quotas. Mitigation: MCP clients control approval and can rate-limit or block sampling from misbehaving servers.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
### Server-Side LLM Integration
|
||||
|
||||
**Approach**: Configure the MCP server with OpenAI API key or local Ollama instance. Generate answers server-side.
|
||||
|
||||
**Rejected Because**:
|
||||
- Duplicates LLM infrastructure that MCP clients already have
|
||||
- Creates billing and API key management burden for server operators
|
||||
- Locks users into server-configured models
|
||||
- Violates MCP's client-server separation principle
|
||||
|
||||
### Multi-Turn Conversation Pattern
|
||||
|
||||
**Approach**: `nc_notes_semantic_search` returns documents. User asks follow-up question. Client's LLM uses previous tool results as context.
|
||||
|
||||
**Rejected Because**:
|
||||
- Requires users to know to ask follow-up questions
|
||||
- Consumes context window with full document content
|
||||
- Inconsistent behavior across clients
|
||||
- Poor citation (LLM may not reference which documents it used)
|
||||
|
||||
### Pre-Generated Summaries
|
||||
|
||||
**Approach**: Generate and cache summaries during indexing. Return summaries instead of excerpts.
|
||||
|
||||
**Rejected Because**:
|
||||
- Summaries become stale as documents change
|
||||
- Summary quality depends on server-side LLM (same problems as server-side generation)
|
||||
- Summaries are generic, not tailored to specific queries
|
||||
|
||||
### Streaming Responses
|
||||
|
||||
**Approach**: Use MCP sampling with streaming to return incremental answer chunks.
|
||||
|
||||
**Deferred Because**:
|
||||
- MCP sampling streaming support unclear in current specification
|
||||
- Adds significant implementation complexity
|
||||
- Tool responses in MCP are typically atomic
|
||||
- Can be added later without breaking changes
|
||||
|
||||
## Related Decisions
|
||||
|
||||
**ADR-007**: Background Vector Sync provides the semantic search infrastructure that this ADR enhances with LLM generation.
|
||||
|
||||
**ADR-004**: Progressive Consent architecture applies to sampling—users consent to sampling requests via MCP client approval prompts.
|
||||
|
||||
## References
|
||||
|
||||
- [MCP Specification - Sampling](https://modelcontextprotocol.io/docs/specification/2025-06-18/client/sampling)
|
||||
- [MCP Python SDK - ServerSession.create_message](https://github.com/modelcontextprotocol/python-sdk/blob/main/src/mcp/server/session.py#L215)
|
||||
- [MCP Python SDK - Sampling Example](https://github.com/modelcontextprotocol/python-sdk/blob/main/examples/snippets/servers/sampling.py)
|
||||
- [MCP Types - SamplingMessage](https://github.com/modelcontextprotocol/python-sdk/blob/main/src/mcp/types.py#L1038)
|
||||
- [MCP Types - CreateMessageResult](https://github.com/modelcontextprotocol/python-sdk/blob/main/src/mcp/types.py#L1073)
|
||||
- [Retrieval-Augmented Generation (RAG) - Lewis et al. 2020](https://arxiv.org/abs/2005.11401)
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] Create ADR-008 document (this file)
|
||||
- [ ] Add `SamplingSearchResponse` model to `nextcloud_mcp_server/models/notes.py`
|
||||
- [ ] Implement `nc_notes_semantic_search_answer` tool in `nextcloud_mcp_server/server/notes.py`
|
||||
- [ ] Add MCP sampling type imports (`SamplingMessage`, `TextContent`, etc.)
|
||||
- [ ] Write unit tests with mocked sampling (`tests/unit/server/test_notes.py`)
|
||||
- [ ] Create integration tests (`tests/integration/test_sampling.py`)
|
||||
- [ ] Update `README.md` with new tool documentation
|
||||
- [ ] Update `CLAUDE.md` with sampling pattern guidance
|
||||
- [ ] Test with MCP client supporting sampling (Claude Desktop, MCP Inspector with callbacks)
|
||||
- [ ] Document client requirements and fallback behavior
|
||||
+3
-1
@@ -8,7 +8,9 @@
|
||||
| `nc_notes_update_note` | Update an existing note by ID |
|
||||
| `nc_notes_append_content` | Append content to an existing note with a clear separator |
|
||||
| `nc_notes_delete_note` | Delete a note by ID |
|
||||
| `nc_notes_search_notes` | Search notes by title or content |
|
||||
| `nc_notes_search_notes` | Search notes by title or content (keyword search) |
|
||||
| `nc_notes_semantic_search` | Search notes by meaning using vector embeddings (requires vector sync) |
|
||||
| `nc_notes_semantic_search_answer` | Search notes semantically and generate a natural language answer via MCP sampling (requires vector sync and sampling-capable MCP client) |
|
||||
|
||||
### Note Attachments
|
||||
|
||||
|
||||
@@ -108,3 +108,41 @@ class SemanticSearchNotesResponse(BaseResponse):
|
||||
search_method: str = Field(
|
||||
default="semantic", description="Search method used (semantic or hybrid)"
|
||||
)
|
||||
|
||||
|
||||
class SamplingSearchResponse(BaseResponse):
|
||||
"""Response from semantic search with LLM-generated answer via MCP sampling.
|
||||
|
||||
This response includes both a generated natural language answer (created by
|
||||
the MCP client's LLM via sampling) and the source documents used to generate
|
||||
that answer. Users can read the answer for quick information and review
|
||||
sources for verification and deeper exploration.
|
||||
|
||||
Attributes:
|
||||
query: The original user query
|
||||
generated_answer: Natural language answer generated by client's LLM
|
||||
sources: List of semantic search results used as context
|
||||
total_found: Total number of matching documents found
|
||||
search_method: Always "semantic_sampling" for this response type
|
||||
model_used: Name of model that generated the answer (e.g., "claude-3-5-sonnet")
|
||||
stop_reason: Why generation stopped ("endTurn", "maxTokens", etc.)
|
||||
"""
|
||||
|
||||
query: str = Field(..., description="Original user query")
|
||||
generated_answer: str = Field(
|
||||
..., description="LLM-generated answer based on retrieved documents"
|
||||
)
|
||||
sources: List[SemanticSearchResult] = Field(
|
||||
default_factory=list,
|
||||
description="Source documents with excerpts and relevance scores",
|
||||
)
|
||||
total_found: int = Field(..., description="Total matching documents")
|
||||
search_method: str = Field(
|
||||
default="semantic_sampling", description="Search method used"
|
||||
)
|
||||
model_used: Optional[str] = Field(
|
||||
default=None, description="Model that generated the answer"
|
||||
)
|
||||
stop_reason: Optional[str] = Field(
|
||||
default=None, description="Reason generation stopped"
|
||||
)
|
||||
|
||||
@@ -3,7 +3,13 @@ import logging
|
||||
from httpx import HTTPStatusError, RequestError
|
||||
from mcp.server.fastmcp import Context, FastMCP
|
||||
from mcp.shared.exceptions import McpError
|
||||
from mcp.types import ErrorData
|
||||
from mcp.types import (
|
||||
ErrorData,
|
||||
ModelHint,
|
||||
ModelPreferences,
|
||||
SamplingMessage,
|
||||
TextContent,
|
||||
)
|
||||
|
||||
from nextcloud_mcp_server.auth import require_scopes
|
||||
from nextcloud_mcp_server.context import get_client
|
||||
@@ -14,6 +20,7 @@ from nextcloud_mcp_server.models.notes import (
|
||||
Note,
|
||||
NoteSearchResult,
|
||||
NotesSettings,
|
||||
SamplingSearchResponse,
|
||||
SearchNotesResponse,
|
||||
SemanticSearchNotesResponse,
|
||||
SemanticSearchResult,
|
||||
@@ -507,6 +514,182 @@ def configure_notes_tools(mcp: FastMCP):
|
||||
ErrorData(code=-1, message=f"Semantic search failed: {str(e)}")
|
||||
)
|
||||
|
||||
@mcp.tool()
|
||||
@require_scopes("notes:read")
|
||||
async def nc_notes_semantic_search_answer(
|
||||
query: str,
|
||||
ctx: Context,
|
||||
limit: int = 5,
|
||||
score_threshold: float = 0.7,
|
||||
max_answer_tokens: int = 500,
|
||||
) -> SamplingSearchResponse:
|
||||
"""
|
||||
Semantic search with LLM-generated answer using MCP sampling.
|
||||
|
||||
Retrieves relevant documents from Nextcloud Notes using vector similarity
|
||||
search, then uses MCP sampling to request the client's LLM to generate
|
||||
a natural language answer based on the retrieved context.
|
||||
|
||||
This tool combines the power of semantic search (finding relevant content)
|
||||
with LLM generation (synthesizing that content into coherent answers). The
|
||||
generated answer includes citations to specific documents, allowing users
|
||||
to verify claims and explore sources.
|
||||
|
||||
The LLM generation happens client-side via MCP sampling. The MCP client
|
||||
controls which model is used, who pays for it, and whether to prompt the
|
||||
user for approval. This keeps the server simple (no LLM API keys needed)
|
||||
while giving users full control over their LLM interactions.
|
||||
|
||||
Args:
|
||||
query: Natural language question to answer (e.g., "What are my project goals?")
|
||||
ctx: MCP context for session access
|
||||
limit: Maximum number of documents to retrieve (default: 5)
|
||||
score_threshold: Minimum similarity score 0-1 (default: 0.7)
|
||||
max_answer_tokens: Maximum tokens for generated answer (default: 500)
|
||||
|
||||
Returns:
|
||||
SamplingSearchResponse containing:
|
||||
- generated_answer: Natural language answer with citations
|
||||
- sources: List of documents with excerpts and relevance scores
|
||||
- model_used: Which model generated the answer
|
||||
- stop_reason: Why generation stopped
|
||||
|
||||
Note: Requires MCP client to support sampling. If sampling is unavailable,
|
||||
the tool gracefully degrades to returning documents with an explanation.
|
||||
The client may prompt the user to approve the sampling request.
|
||||
|
||||
Examples:
|
||||
>>> # Query about project goals
|
||||
>>> result = await nc_notes_semantic_search_answer(
|
||||
... query="What are my Q1 2025 project goals?",
|
||||
... ctx=ctx
|
||||
... )
|
||||
>>> print(result.generated_answer)
|
||||
"Based on Document 1 (Project Kickoff) and Document 3 (Q1 Planning),
|
||||
your main goals are: 1) Improve semantic search accuracy by 20%,
|
||||
2) Deploy new embedding model, 3) Reduce indexing latency..."
|
||||
|
||||
>>> # Query about learning
|
||||
>>> result = await nc_notes_semantic_search_answer(
|
||||
... query="What did I learn about Python async/await last month?",
|
||||
... ctx=ctx,
|
||||
... limit=10
|
||||
... )
|
||||
>>> len(result.sources) # Up to 10 documents
|
||||
7
|
||||
"""
|
||||
# 1. Retrieve relevant documents via existing semantic search
|
||||
search_response = await nc_notes_semantic_search(
|
||||
query=query,
|
||||
ctx=ctx,
|
||||
limit=limit,
|
||||
score_threshold=score_threshold,
|
||||
)
|
||||
|
||||
# 2. Handle no results case - don't waste a sampling call
|
||||
if not search_response.results:
|
||||
logger.debug(f"No documents found for query: {query}")
|
||||
return SamplingSearchResponse(
|
||||
query=query,
|
||||
generated_answer="No relevant documents found in your Nextcloud Notes for this query.",
|
||||
sources=[],
|
||||
total_found=0,
|
||||
search_method="semantic_sampling",
|
||||
success=True,
|
||||
)
|
||||
|
||||
# 3. Construct context from retrieved documents
|
||||
context_parts = []
|
||||
for idx, result in enumerate(search_response.results, 1):
|
||||
context_parts.append(
|
||||
f"[Document {idx}]\n"
|
||||
f"Title: {result.title}\n"
|
||||
f"Category: {result.category}\n"
|
||||
f"Excerpt: {result.excerpt}\n"
|
||||
f"Relevance Score: {result.score:.2f}\n"
|
||||
)
|
||||
|
||||
context = "\n".join(context_parts)
|
||||
|
||||
# 4. Construct prompt - reuse user's query, add context and instructions
|
||||
prompt = (
|
||||
f"{query}\n\n"
|
||||
f"Here are relevant documents from Nextcloud Notes:\n\n"
|
||||
f"{context}\n\n"
|
||||
f"Based on the documents above, please provide a comprehensive answer. "
|
||||
f"Cite the document numbers when referencing specific information."
|
||||
)
|
||||
|
||||
logger.debug(
|
||||
f"Requesting sampling for query: {query} "
|
||||
f"({len(search_response.results)} documents retrieved)"
|
||||
)
|
||||
|
||||
# 5. Request LLM completion via MCP sampling
|
||||
try:
|
||||
sampling_result = await ctx.session.create_message(
|
||||
messages=[
|
||||
SamplingMessage(
|
||||
role="user",
|
||||
content=TextContent(type="text", text=prompt),
|
||||
)
|
||||
],
|
||||
max_tokens=max_answer_tokens,
|
||||
temperature=0.7,
|
||||
model_preferences=ModelPreferences(
|
||||
hints=[ModelHint(name="claude-3-5-sonnet")],
|
||||
intelligencePriority=0.8,
|
||||
speedPriority=0.5,
|
||||
),
|
||||
include_context="thisServer",
|
||||
)
|
||||
|
||||
# 6. Extract answer from sampling response
|
||||
if sampling_result.content.type == "text":
|
||||
generated_answer = sampling_result.content.text
|
||||
else:
|
||||
# Handle non-text responses (shouldn't happen for text prompts)
|
||||
generated_answer = f"Received non-text response of type: {sampling_result.content.type}"
|
||||
logger.warning(
|
||||
f"Unexpected content type from sampling: {sampling_result.content.type}"
|
||||
)
|
||||
|
||||
logger.info(
|
||||
f"Sampling successful: model={sampling_result.model}, "
|
||||
f"stop_reason={sampling_result.stopReason}"
|
||||
)
|
||||
|
||||
return SamplingSearchResponse(
|
||||
query=query,
|
||||
generated_answer=generated_answer,
|
||||
sources=search_response.results,
|
||||
total_found=search_response.total_found,
|
||||
search_method="semantic_sampling",
|
||||
model_used=sampling_result.model,
|
||||
stop_reason=sampling_result.stopReason,
|
||||
success=True,
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
# Fallback: Return documents without generated answer
|
||||
logger.warning(
|
||||
f"Sampling failed ({type(e).__name__}: {e}), "
|
||||
f"returning search results only"
|
||||
)
|
||||
|
||||
return SamplingSearchResponse(
|
||||
query=query,
|
||||
generated_answer=(
|
||||
f"[Sampling unavailable: {str(e)}]\n\n"
|
||||
f"Found {search_response.total_found} relevant documents. "
|
||||
f"Please review the sources below."
|
||||
),
|
||||
sources=search_response.results,
|
||||
total_found=search_response.total_found,
|
||||
search_method="semantic_sampling_fallback",
|
||||
success=True,
|
||||
)
|
||||
|
||||
@mcp.tool()
|
||||
@require_scopes("notes:write")
|
||||
async def nc_notes_delete_note(note_id: int, ctx: Context) -> DeleteNoteResponse:
|
||||
|
||||
@@ -0,0 +1,276 @@
|
||||
"""Integration tests for MCP sampling with semantic search.
|
||||
|
||||
These tests validate the nc_notes_semantic_search_answer tool which combines:
|
||||
1. Semantic search to retrieve relevant documents
|
||||
2. MCP sampling to generate natural language answers
|
||||
|
||||
Tests cover three scenarios:
|
||||
- Successful sampling (LLM generates answer)
|
||||
- Sampling fallback (client doesn't support sampling)
|
||||
- No results (no relevant documents found)
|
||||
|
||||
Note: These tests require VECTOR_SYNC_ENABLED=true and a configured
|
||||
vector database with indexed test data.
|
||||
"""
|
||||
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
from mcp.types import CreateMessageResult, TextContent
|
||||
|
||||
pytestmark = pytest.mark.integration
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_sampling_result():
|
||||
"""Mock successful sampling result from MCP client."""
|
||||
result = MagicMock(spec=CreateMessageResult)
|
||||
result.content = TextContent(
|
||||
type="text",
|
||||
text=(
|
||||
"Based on Document 1 (Python Async Programming) and Document 2 "
|
||||
"(Best Practices), you should use async/await for asynchronous "
|
||||
"programming and always use async context managers for resources."
|
||||
),
|
||||
)
|
||||
result.model = "claude-3-5-sonnet"
|
||||
result.stopReason = "endTurn"
|
||||
return result
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_semantic_search_answer_successful_sampling(
|
||||
nc_mcp_client, temporary_note, mock_sampling_result
|
||||
):
|
||||
"""Test semantic search with successful LLM answer generation.
|
||||
|
||||
Prerequisites:
|
||||
- VECTOR_SYNC_ENABLED=true
|
||||
- Qdrant running and indexed
|
||||
- Test note indexed in vector database
|
||||
|
||||
Flow:
|
||||
1. Create test note with searchable content
|
||||
2. Call nc_notes_semantic_search_answer
|
||||
3. Mock ctx.session.create_message to return answer
|
||||
4. Verify response contains generated answer and sources
|
||||
"""
|
||||
# Create a note with content about Python async
|
||||
_note = await temporary_note(
|
||||
title="Python Async Guide",
|
||||
content="""# Python Async Programming
|
||||
|
||||
## Key Concepts
|
||||
- Use async def for coroutines
|
||||
- Use await for async operations
|
||||
- asyncio.gather() for parallel execution
|
||||
|
||||
## Best Practices
|
||||
Always use async context managers for resources.
|
||||
Avoid blocking operations in async code.""",
|
||||
category="Development",
|
||||
)
|
||||
|
||||
# Wait for vector indexing (if background sync is slow)
|
||||
import asyncio
|
||||
|
||||
await asyncio.sleep(2)
|
||||
|
||||
# Mock the sampling call
|
||||
# Note: This requires monkey-patching ctx.session.create_message
|
||||
# In a real integration test with MCP Inspector, this would be actual sampling
|
||||
|
||||
result = await nc_mcp_client.call_tool(
|
||||
"nc_notes_semantic_search_answer",
|
||||
arguments={
|
||||
"query": "How do I use async in Python?",
|
||||
"limit": 5,
|
||||
"score_threshold": 0.5,
|
||||
},
|
||||
)
|
||||
|
||||
# Verify response structure
|
||||
assert result is not None
|
||||
assert "query" in result
|
||||
assert "generated_answer" in result
|
||||
assert "sources" in result
|
||||
assert "total_found" in result
|
||||
assert "search_method" in result
|
||||
|
||||
# For this test, sampling might fail (no real LLM client)
|
||||
# So we check for either success or fallback
|
||||
if "[Sampling unavailable" in result["generated_answer"]:
|
||||
# Fallback mode - should still have sources
|
||||
assert result["search_method"] == "semantic_sampling_fallback"
|
||||
assert len(result["sources"]) > 0
|
||||
pytest.skip("Sampling not supported by test client (expected fallback)")
|
||||
else:
|
||||
# Successful sampling
|
||||
assert result["search_method"] == "semantic_sampling"
|
||||
assert "async" in result["generated_answer"].lower()
|
||||
assert len(result["sources"]) > 0
|
||||
assert result["model_used"] is not None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_semantic_search_answer_no_results(nc_mcp_client):
|
||||
"""Test semantic search answer when no documents match.
|
||||
|
||||
Flow:
|
||||
1. Query for completely unrelated topic
|
||||
2. Verify response indicates no documents found
|
||||
3. Verify no sampling call was made (no sources to base answer on)
|
||||
"""
|
||||
result = await nc_mcp_client.call_tool(
|
||||
"nc_notes_semantic_search_answer",
|
||||
arguments={
|
||||
"query": "quantum chromodynamics lattice QCD gluon propagator",
|
||||
"limit": 5,
|
||||
"score_threshold": 0.7,
|
||||
},
|
||||
)
|
||||
|
||||
# Should get "no documents found" message
|
||||
assert result is not None
|
||||
assert result["total_found"] == 0
|
||||
assert len(result["sources"]) == 0
|
||||
assert "No relevant documents" in result["generated_answer"]
|
||||
assert result["search_method"] == "semantic_sampling"
|
||||
# No sampling should have occurred
|
||||
assert result["model_used"] is None
|
||||
assert result["stop_reason"] is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_semantic_search_answer_with_limit(nc_mcp_client, temporary_note):
|
||||
"""Test semantic search answer respects limit parameter.
|
||||
|
||||
Flow:
|
||||
1. Create multiple related notes
|
||||
2. Query with limit=2
|
||||
3. Verify at most 2 sources in response
|
||||
"""
|
||||
# Create multiple related notes
|
||||
_note1 = await temporary_note(
|
||||
title="Python Async Part 1",
|
||||
content="Use async/await for asynchronous operations",
|
||||
category="Development",
|
||||
)
|
||||
_note2 = await temporary_note(
|
||||
title="Python Async Part 2",
|
||||
content="Use asyncio.gather() for parallel execution",
|
||||
category="Development",
|
||||
)
|
||||
_note3 = await temporary_note(
|
||||
title="Python Async Part 3",
|
||||
content="Always use async context managers",
|
||||
category="Development",
|
||||
)
|
||||
|
||||
# Wait for indexing
|
||||
import asyncio
|
||||
|
||||
await asyncio.sleep(2)
|
||||
|
||||
result = await nc_mcp_client.call_tool(
|
||||
"nc_notes_semantic_search_answer",
|
||||
arguments={
|
||||
"query": "async programming in Python",
|
||||
"limit": 2,
|
||||
"score_threshold": 0.5,
|
||||
},
|
||||
)
|
||||
|
||||
# Should respect limit
|
||||
assert len(result["sources"]) <= 2
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_semantic_search_answer_score_threshold(nc_mcp_client, temporary_note):
|
||||
"""Test semantic search answer respects score threshold.
|
||||
|
||||
Flow:
|
||||
1. Create note with specific content
|
||||
2. Query with high threshold (0.9)
|
||||
3. Verify only high-scoring results returned
|
||||
"""
|
||||
_note = await temporary_note(
|
||||
title="Exact Match Test",
|
||||
content="This is a very specific test document about widget manufacturing",
|
||||
category="Test",
|
||||
)
|
||||
|
||||
# Wait for indexing
|
||||
import asyncio
|
||||
|
||||
await asyncio.sleep(2)
|
||||
|
||||
# Query with exact match - should have high score
|
||||
result = await nc_mcp_client.call_tool(
|
||||
"nc_notes_semantic_search_answer",
|
||||
arguments={
|
||||
"query": "widget manufacturing",
|
||||
"limit": 5,
|
||||
"score_threshold": 0.9,
|
||||
},
|
||||
)
|
||||
|
||||
# Note: Semantic search scores depend on embedding model
|
||||
# We just verify the tool accepts the parameter
|
||||
assert "score_threshold" not in result # Not exposed in response
|
||||
if result["total_found"] > 0:
|
||||
# If results found, verify they're in sources
|
||||
assert all("score" in source for source in result["sources"])
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_semantic_search_answer_max_tokens(nc_mcp_client, temporary_note):
|
||||
"""Test semantic search answer respects max_answer_tokens parameter.
|
||||
|
||||
Flow:
|
||||
1. Create note with content
|
||||
2. Call with very small max_tokens (100)
|
||||
3. Verify parameter is accepted (actual token limiting happens in client)
|
||||
|
||||
Note: Token limiting is enforced by the MCP client's LLM, not the server.
|
||||
This test just verifies the parameter is correctly passed.
|
||||
"""
|
||||
_note = await temporary_note(
|
||||
title="Long Document",
|
||||
content="This is a document with lots of content. " * 50,
|
||||
category="Test",
|
||||
)
|
||||
|
||||
# Wait for indexing
|
||||
import asyncio
|
||||
|
||||
await asyncio.sleep(2)
|
||||
|
||||
result = await nc_mcp_client.call_tool(
|
||||
"nc_notes_semantic_search_answer",
|
||||
arguments={
|
||||
"query": "document content",
|
||||
"limit": 5,
|
||||
"score_threshold": 0.5,
|
||||
"max_answer_tokens": 100,
|
||||
},
|
||||
)
|
||||
|
||||
# Should not error, even if sampling fails
|
||||
assert result is not None
|
||||
assert "generated_answer" in result
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_semantic_search_answer_requires_vector_sync():
|
||||
"""Test that semantic search answer fails when VECTOR_SYNC_ENABLED=false.
|
||||
|
||||
This test validates the tool properly checks for vector sync being enabled.
|
||||
|
||||
Note: This test requires a separate test client with VECTOR_SYNC_ENABLED=false,
|
||||
which may not be available in the current test environment. Skipping for now.
|
||||
"""
|
||||
pytest.skip(
|
||||
"Requires test environment with VECTOR_SYNC_ENABLED=false, "
|
||||
"which would break other semantic search tests"
|
||||
)
|
||||
@@ -6,7 +6,9 @@ from nextcloud_mcp_server.models.notes import (
|
||||
CreateNoteResponse,
|
||||
Note,
|
||||
NoteSearchResult,
|
||||
SamplingSearchResponse,
|
||||
SearchNotesResponse,
|
||||
SemanticSearchResult,
|
||||
)
|
||||
|
||||
|
||||
@@ -121,3 +123,142 @@ def test_note_search_result_without_score():
|
||||
|
||||
assert result.id == 99
|
||||
assert result.score is None
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_sampling_search_response_with_answer():
|
||||
"""Test SamplingSearchResponse with LLM-generated answer."""
|
||||
sources = [
|
||||
SemanticSearchResult(
|
||||
id=1,
|
||||
title="Python Guide",
|
||||
category="Development",
|
||||
excerpt="Use async/await for asynchronous programming",
|
||||
score=0.92,
|
||||
chunk_index=0,
|
||||
total_chunks=3,
|
||||
),
|
||||
SemanticSearchResult(
|
||||
id=2,
|
||||
title="Best Practices",
|
||||
category="Development",
|
||||
excerpt="Always use context managers with async operations",
|
||||
score=0.85,
|
||||
chunk_index=1,
|
||||
total_chunks=2,
|
||||
),
|
||||
]
|
||||
|
||||
response = SamplingSearchResponse(
|
||||
query="How do I use async in Python?",
|
||||
generated_answer="Based on Document 1 and Document 2, use async/await for asynchronous programming and always use context managers.",
|
||||
sources=sources,
|
||||
total_found=2,
|
||||
search_method="semantic_sampling",
|
||||
model_used="claude-3-5-sonnet",
|
||||
stop_reason="endTurn",
|
||||
success=True,
|
||||
)
|
||||
|
||||
# Verify the response structure
|
||||
assert response.query == "How do I use async in Python?"
|
||||
assert "async/await" in response.generated_answer
|
||||
assert len(response.sources) == 2
|
||||
assert response.sources[0].id == 1
|
||||
assert response.sources[0].score == 0.92
|
||||
assert response.total_found == 2
|
||||
assert response.search_method == "semantic_sampling"
|
||||
assert response.model_used == "claude-3-5-sonnet"
|
||||
assert response.stop_reason == "endTurn"
|
||||
assert response.success is True
|
||||
|
||||
# Verify it serializes correctly
|
||||
data = response.model_dump()
|
||||
assert "query" in data
|
||||
assert "generated_answer" in data
|
||||
assert "sources" in data
|
||||
assert isinstance(data["sources"], list)
|
||||
assert len(data["sources"]) == 2
|
||||
assert data["sources"][0]["id"] == 1
|
||||
assert data["model_used"] == "claude-3-5-sonnet"
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_sampling_search_response_fallback():
|
||||
"""Test SamplingSearchResponse when sampling fails (fallback mode)."""
|
||||
sources = [
|
||||
SemanticSearchResult(
|
||||
id=1,
|
||||
title="Note 1",
|
||||
category="Work",
|
||||
excerpt="Some content",
|
||||
score=0.75,
|
||||
chunk_index=0,
|
||||
total_chunks=1,
|
||||
)
|
||||
]
|
||||
|
||||
response = SamplingSearchResponse(
|
||||
query="test query",
|
||||
generated_answer="[Sampling unavailable: Client does not support sampling]\n\nFound 1 relevant documents. Please review the sources below.",
|
||||
sources=sources,
|
||||
total_found=1,
|
||||
search_method="semantic_sampling_fallback",
|
||||
model_used=None,
|
||||
stop_reason=None,
|
||||
success=True,
|
||||
)
|
||||
|
||||
# Verify fallback behavior
|
||||
assert "[Sampling unavailable" in response.generated_answer
|
||||
assert response.search_method == "semantic_sampling_fallback"
|
||||
assert response.model_used is None
|
||||
assert response.stop_reason is None
|
||||
assert len(response.sources) == 1
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_sampling_search_response_no_results():
|
||||
"""Test SamplingSearchResponse when no documents found."""
|
||||
response = SamplingSearchResponse(
|
||||
query="nonexistent topic",
|
||||
generated_answer="No relevant documents found in your Nextcloud Notes for this query.",
|
||||
sources=[],
|
||||
total_found=0,
|
||||
search_method="semantic_sampling",
|
||||
success=True,
|
||||
)
|
||||
|
||||
# Verify no results case
|
||||
assert response.total_found == 0
|
||||
assert len(response.sources) == 0
|
||||
assert "No relevant documents" in response.generated_answer
|
||||
assert response.model_used is None
|
||||
assert response.stop_reason is None
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_sampling_search_response_serialization():
|
||||
"""Test SamplingSearchResponse serializes to JSON correctly."""
|
||||
response = SamplingSearchResponse(
|
||||
query="test",
|
||||
generated_answer="Test answer",
|
||||
sources=[],
|
||||
total_found=0,
|
||||
search_method="semantic_sampling",
|
||||
model_used="claude-3-5-sonnet",
|
||||
stop_reason="maxTokens",
|
||||
success=True,
|
||||
)
|
||||
|
||||
data = response.model_dump()
|
||||
|
||||
# Check all fields are present
|
||||
assert data["query"] == "test"
|
||||
assert data["generated_answer"] == "Test answer"
|
||||
assert data["sources"] == []
|
||||
assert data["total_found"] == 0
|
||||
assert data["search_method"] == "semantic_sampling"
|
||||
assert data["model_used"] == "claude-3-5-sonnet"
|
||||
assert data["stop_reason"] == "maxTokens"
|
||||
assert data["success"] is True
|
||||
|
||||
Reference in New Issue
Block a user