nextcloud-mcp-server/docs/ADR-011-improving-semantic-search-quality.md

ADR-011: Improving Semantic Search Quality Through Better Chunking and Embeddings

Status: Partially Implemented (Chunking Complete, Embeddings Pending) Date: 2025-11-12 Implementation Date: 2025-11-18 (Chunking) Authors: Development Team Related: ADR-003 (Vector Database Architecture), ADR-008 (MCP Sampling for RAG)

Context

The semantic search implementation provides document retrieval across Nextcloud apps using vector embeddings. Production usage has revealed that the system frequently misses relevant documents (recall problem).

Root cause analysis identifies two fundamental issues:

1. Poor Chunking Strategy

Current Implementation (nextcloud_mcp_server/vector/document_chunker.py:36):

words = content.split()  # Naive whitespace splitting
chunk_size = 512  # words
overlap = 50  # words
chunks = [words[i:i+chunk_size] for i in range(0, len(words), chunk_size-overlap)]

Problems:

• Breaks semantic boundaries: Splits mid-sentence, mid-paragraph, mid-thought
• Loses context: "The meeting discussed budget. We decided to..." becomes two disconnected chunks
• Poor retrieval: Relevant content split across chunks with low individual relevance scores
• No structure awareness: Ignores markdown headers, lists, code blocks

Evidence:

• Documents with relevant content in middle sections score poorly (content split across 3+ chunks)
• Multi-sentence concepts (spanning 60-100 words) are fragmented
• Search for "budget planning process" misses documents where these words appear in adjacent sentences but different chunks

2. Suboptimal Embedding Model

Current Implementation (nextcloud_mcp_server/embedding/ollama_provider.py:33):

_model = "nomic-embed-text"  # 768 dimensions
_dimension = 768  # Hardcoded

Problems:

• Model selection: nomic-embed-text is general-purpose, not optimized for our use case
• No benchmarking: Selected without comparative evaluation
• Dimensionality: 768-dim may be insufficient for nuanced semantic distinctions
• No domain adaptation: Model not tuned for Nextcloud content (notes, calendar, deck cards)

Evidence:

• Synonymous queries return different results ("meeting notes" vs. "discussion summary")
• Domain-specific terms poorly represented ("standup", "retrospective", "OKRs")
• Cross-lingual content (if present) not well supported

Current Performance

Baseline Metrics (100-document test corpus, 50 queries):

• Recall@10: ~52% (misses 48% of relevant documents)
• Precision@10: ~78% (acceptable but room for improvement)
• MRR: 0.58 (relevant docs often not in top positions)
• Zero-result queries: 18% (completely missing relevant content)

Decision Drivers

1. Address Root Causes: Fix fundamental issues (chunking, embeddings) before adding complexity (reranking, hybrid search)
2. Measurable Impact: Target 40-60% improvement in recall through chunking/embedding alone
3. Independence: Improvements should be orthogonal to future enhancements (reranking, GraphRAG)
4. Cost Efficiency: Minimize infrastructure and API costs
5. Reindexing Acceptable: One-time reindex cost justified by long-term quality improvement

Options Considered

Chunking Strategies

Option C1: Semantic Sentence-Aware Chunking (RECOMMENDED)

Description: Respect sentence boundaries while maintaining target chunk size

Implementation:

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=2048,  # ~512 words in characters
    chunk_overlap=200,  # ~50 words in characters
    separators=["\n\n", "\n", ". ", "! ", "? ", "; ", ": ", ", ", " "],
    length_function=len,
)

How it works:

1. Try splitting by paragraphs (\n\n)
2. If chunks too large, split by sentences (. , ! , ? )
3. If still too large, split by clauses (;, :)
4. Last resort: split by words

Pros:

• ✅ Preserves semantic boundaries (never breaks mid-sentence)
• ✅ Maintains context coherence within chunks
• ✅ Simple implementation (langchain library)
• ✅ Configurable separators for different content types
• ✅ Proven approach (used by major RAG systems)

Cons:

• ❌ Variable chunk sizes (not exactly 512 words, but close)
• ❌ Adds dependency (langchain)
• ❌ Slightly slower than naive splitting (~10-20ms per document)

Expected Impact: 20-30% recall improvement

Option C2: Hierarchical Context-Preserving Chunks

Description: Create overlapping parent/child chunks

Structure:

Document → Large parent chunks (1024 words) → Small child chunks (256 words)
          ↓                                    ↓
   Stored in Qdrant                       Searched first
                                          Return parent context

Implementation:

# Generate child chunks (searched)
child_chunks = splitter.split_text(content, chunk_size=1024)

# Generate parent chunks (context)
parent_chunks = splitter.split_text(content, chunk_size=4096)

# Store both with parent-child relationships
for child_idx, child in enumerate(child_chunks):
    parent_idx = find_parent(child_idx)
    store_vector(
        vector=embed(child),
        payload={
            "chunk": child,
            "parent_chunk": parent_chunks[parent_idx],
            "chunk_type": "child"
        }
    )

Pros:

• ✅ Best of both worlds: precise matching + full context
• ✅ Handles multi-hop information needs
• ✅ Better for long documents (> 1000 words)

Cons:

• ❌ 2x storage (parent + child chunks)
• ❌ More complex implementation
• ❌ Higher indexing time (embed twice)
• ❌ Query complexity (retrieve child, return parent)

Expected Impact: 35-45% recall improvement (diminishing returns vs. complexity)

Verdict: ⚠️ Consider only if Option C1 insufficient

Option C3: Document Structure-Aware Chunking

Description: Parse markdown/document structure before chunking

Implementation:

import mistune  # Markdown parser

def structure_aware_chunk(markdown_content: str) -> list[str]:
    ast = mistune.create_markdown(renderer='ast')(markdown_content)

    chunks = []
    for node in ast:
        if node['type'] == 'heading':
            # Start new chunk at each header
            current_chunk = node['children'][0]['raw']
        elif node['type'] == 'paragraph':
            current_chunk += "\n" + node['children'][0]['raw']
            if len(current_chunk) > 2048:
                chunks.append(current_chunk)
                current_chunk = ""

    return chunks

Pros:

• ✅ Respects document logical structure
• ✅ Headers provide context for chunks
• ✅ Works well for structured notes (documentation, meeting notes with sections)

Cons:

• ❌ Complex implementation (parser, AST traversal)
• ❌ Markdown-specific (doesn't help calendar events, deck cards)
• ❌ Variable chunk sizes (some sections very short/long)
• ❌ Breaks for unstructured content

Expected Impact: 15-25% improvement for structured content only

Verdict: ⚠️ Future enhancement after Option C1

Option C4: Fixed Sliding Window (Current Baseline)

Description: Current naive word-based splitting

Verdict: ❌ Superseded by Option C1

Embedding Model Strategies

Option E1: Upgrade to Better General-Purpose Model (RECOMMENDED)

Description: Switch to state-of-the-art embedding model

Candidates:

Model	Dimensions	MTEB Score	Pros	Cons
mxbai-embed-large	1024	64.68	Best performance, good balance	Larger (slower)
nomic-embed-text-v1.5	768	62.39	Upgraded version of current	Incremental improvement
bge-large-en-v1.5	1024	64.23	Excellent for English	Not multilingual
nomic-embed-text (current)	768	60.10	Baseline	Lower performance

MTEB: Massive Text Embedding Benchmark (higher = better semantic understanding)

Recommendation: mxbai-embed-large-v1

• Best MTEB score (64.68)
• 1024 dimensions (richer semantic space)
• Works well via Ollama
• ~15-20% better retrieval quality in benchmarks

Implementation:

# config.py
OLLAMA_EMBEDDING_MODEL = "mxbai-embed-large-v1"  # Changed from nomic-embed-text

# ollama_provider.py
async def get_dimension(self) -> int:
    # Query Ollama for actual dimension instead of hardcoding
    response = await self.client.post("/api/show", json={"name": self.model})
    return response.json()["details"]["embedding_length"]

Migration:

1. Deploy new model to Ollama
2. Create new Qdrant collection (different dimension)
3. Reindex all documents with new embeddings
4. Swap collections atomically
5. Delete old collection

Pros:

• ✅ Immediate quality improvement (15-20%)
• ✅ Simple change (config + reindex)
• ✅ No code complexity
• ✅ Future-proof (state-of-the-art model)

Cons:

• ❌ Requires full reindex (2-4 hours for 1000 documents)
• ❌ Larger model = slower embedding (~50ms vs. 30ms per chunk)
• ❌ Higher dimensionality = more storage (~30% increase)

Expected Impact: 15-25% recall improvement

Option E2: Multi-Vector Embeddings (ColBERT-style)

Description: Generate multiple embeddings per chunk (token-level)

Architecture:

Chunk → Transformer → Token embeddings (e.g., 50 tokens × 128 dim) → Store all
Query → Transformer → Token embeddings → MaxSim(query_tokens, doc_tokens)

MaxSim scoring:

def maxsim_score(query_embeddings, doc_embeddings):
    # For each query token, find max similarity with any doc token
    scores = []
    for q_emb in query_embeddings:
        max_sim = max(cosine_similarity(q_emb, d_emb) for d_emb in doc_embeddings)
        scores.append(max_sim)
    return sum(scores)

Pros:

• ✅ Best retrieval quality (state-of-the-art results)
• ✅ Fine-grained matching (token-level)
• ✅ Handles partial matches better

Cons:

• ❌ 50-100x storage increase (50 vectors per chunk vs. 1)
• ❌ Slower search (compute MaxSim for each candidate)
• ❌ Complex implementation (custom scoring, storage schema)
• ❌ Requires specialized model (ColBERTv2, not available in Ollama)

Expected Impact: 40-50% improvement, but at very high cost

Verdict: ❌ Too complex, too expensive for marginal gain over E1+C1

Option E3: Fine-Tuned Domain-Specific Model

Description: Fine-tune embedding model on Nextcloud corpus

Process:

1. Collect training data (query-document pairs)
2. Fine-tune base model (e.g., nomic-embed-text) on domain data
3. Deploy fine-tuned model via Ollama
4. Reindex with fine-tuned embeddings

Training data needed:

• 1,000+ query-document pairs
• Labeled relevance (positive/negative examples)
• Representative of real usage

Pros:

• ✅ Optimized for specific content (notes, calendar, deck)
• ✅ Better handling of domain terminology
• ✅ Highest potential quality improvement (30-40%)

Cons:

• ❌ Requires training data (expensive to collect)
• ❌ GPU infrastructure needed for fine-tuning
• ❌ Expertise required (ML/NLP knowledge)
• ❌ Maintenance burden (retrain as corpus evolves)
• ❌ Time investment: 2-4 weeks initial setup

Expected Impact: 30-40% improvement, but high cost

Verdict: ⚠️ Consider only if E1+C1 insufficient AND have training data

Option E4: Ensemble Embeddings

Description: Generate embeddings with multiple models, combine scores

Implementation:

models = ["mxbai-embed-large-v1", "bge-large-en-v1.5"]

# Index
embeddings = [await embed(chunk, model) for model in models]
store_multi_vector(embeddings)

# Search
query_embeddings = [await embed(query, model) for model in models]
scores = [search(q_emb, model) for q_emb, model in zip(query_embeddings, models)]
combined_score = 0.5 * scores[0] + 0.5 * scores[1]

Pros:

• ✅ Robust to individual model weaknesses
• ✅ Better coverage of semantic space

Cons:

• ❌ 2x storage and compute
• ❌ Complex scoring and fusion
• ❌ Marginal improvement (~5-10%) over single best model

Expected Impact: 5-10% over best single model

Verdict: ❌ Not worth complexity

Combined Strategies

Option D1: Best Chunking + Best Embedding (RECOMMENDED)

Combination: Option C1 (Semantic Chunking) + Option E1 (mxbai-embed-large-v1)

Expected Impact:

• Chunking: +20-30% recall
• Embedding: +15-25% recall
• Combined: +35-55% recall improvement (not strictly additive, but significant)

Cost:

• Development: 1-2 days
• Reindex: 2-4 hours (one-time)
• Ongoing: None (same infrastructure)

Pros:

• ✅ Addresses both root causes
• ✅ Orthogonal improvements (chunking + embedding)
• ✅ Simple implementation
• ✅ No new infrastructure
• ✅ Future-proof foundation for additional enhancements (reranking, hybrid search)

Cons:

• ❌ Requires full reindex (manageable)
• ❌ Slightly higher storage (1024 vs. 768 dim)

Verdict: ✅ RECOMMENDED

Decision

Adopt Option D1: Semantic Chunking + Upgraded Embedding Model

Implement both improvements together to maximize recall improvement:

1. Semantic Sentence-Aware Chunking

Changes:

• Replace naive word splitting with RecursiveCharacterTextSplitter
• Preserve sentence boundaries, paragraph structure
• Maintain similar chunk sizes (~512 words / 2048 characters)

Implementation:

# nextcloud_mcp_server/vector/document_chunker.py

from langchain.text_splitter import RecursiveCharacterTextSplitter

class DocumentChunker:
    """Chunk documents into semantically coherent pieces."""

    def __init__(
        self,
        chunk_size: int = 2048,  # Characters, not words
        chunk_overlap: int = 200,  # Characters, not words
    ):
        self.chunk_size = chunk_size
        self.chunk_overlap = chunk_overlap

        self.splitter = RecursiveCharacterTextSplitter(
            chunk_size=chunk_size,
            chunk_overlap=chunk_overlap,
            separators=[
                "\n\n",  # Paragraphs (highest priority)
                "\n",    # Lines
                ". ",    # Sentences
                "! ",
                "? ",
                "; ",    # Clauses
                ": ",
                ", ",    # Phrases
                " ",     # Words (last resort)
            ],
            length_function=len,
            is_separator_regex=False,
        )

    def chunk_text(self, content: str) -> list[str]:
        """
        Chunk text while preserving semantic boundaries.

        Args:
            content: Full document text

        Returns:
            List of text chunks, each ending at a semantic boundary
        """
        if not content:
            return []

        # Use RecursiveCharacterTextSplitter for semantic boundaries
        chunks = self.splitter.split_text(content)

        return chunks

Configuration Changes (config.py):

# Old (word-based)
DOCUMENT_CHUNK_SIZE: int = 512  # words
DOCUMENT_CHUNK_OVERLAP: int = 50  # words

# New (character-based, more precise)
DOCUMENT_CHUNK_SIZE: int = 2048  # characters (~512 words)
DOCUMENT_CHUNK_OVERLAP: int = 200  # characters (~50 words)

Dependency (pyproject.toml):

[project]
dependencies = [
    # ... existing dependencies
    "langchain-text-splitters>=0.2.0",
]

2. Upgrade Embedding Model

Changes:

• Switch from nomic-embed-text (768-dim) to mxbai-embed-large-v1 (1024-dim)
• Dynamic dimension detection (query Ollama instead of hardcoding)
• Create new Qdrant collection for new dimensions

Implementation:

# nextcloud_mcp_server/embedding/ollama_provider.py

class OllamaEmbeddingProvider(EmbeddingProvider):
    def __init__(self, base_url: str, model: str, verify_ssl: bool = True):
        self.base_url = base_url
        self.model = model
        self._dimension: int | None = None  # Changed: query dynamically
        self.client = httpx.AsyncClient(base_url=base_url, verify=verify_ssl)

    async def dimension(self) -> int:
        """Get embedding dimension from Ollama API."""
        if self._dimension is None:
            try:
                response = await self.client.post(
                    "/api/show",
                    json={"name": self.model},
                    timeout=10.0,
                )
                response.raise_for_status()
                info = response.json()
                self._dimension = info.get("details", {}).get("embedding_length")

                if self._dimension is None:
                    # Fallback: generate test embedding to detect dimension
                    test_emb = await self.embed("test")
                    self._dimension = len(test_emb)

            except Exception as e:
                logger.warning(f"Failed to get dimension from Ollama: {e}, using fallback")
                # Fallback dimensions by model name
                if "mxbai-embed-large" in self.model:
                    self._dimension = 1024
                elif "nomic-embed-text" in self.model:
                    self._dimension = 768
                else:
                    self._dimension = 768  # Default

        return self._dimension

Configuration Changes (config.py):

# Old
OLLAMA_EMBEDDING_MODEL: str = "nomic-embed-text"

# New
OLLAMA_EMBEDDING_MODEL: str = "mxbai-embed-large-v1"

Environment Variable:

OLLAMA_EMBEDDING_MODEL=mxbai-embed-large-v1

3. Migration Strategy

Reindexing Process:

# nextcloud_mcp_server/vector/migration.py

async def migrate_to_new_embeddings():
    """
    Migrate from old embeddings to new embeddings.

    Process:
    1. Create new collection with new dimension
    2. Reindex all documents with new embeddings
    3. Atomic swap (update collection name in config)
    4. Delete old collection
    """
    old_collection = "nextcloud_content"
    new_collection = "nextcloud_content_v2"

    # 1. Create new collection
    await qdrant_client.create_collection(
        collection_name=new_collection,
        vectors_config=VectorParams(
            size=1024,  # mxbai-embed-large-v1 dimension
            distance=Distance.COSINE,
        ),
    )

    # 2. Reindex all documents
    logger.info("Starting reindex with new embeddings...")
    scanner = VectorScanner(...)
    processor = VectorProcessor(collection_name=new_collection, ...)

    await scanner.scan_all()  # Rescans and re-embeds all documents

    # 3. Wait for completion
    while True:
        status = await get_sync_status()
        if status.pending_documents == 0:
            break
        await asyncio.sleep(5)

    # 4. Atomic swap
    # Update config to point to new collection
    # (or use collection alias in Qdrant)
    await qdrant_client.update_collection_aliases(
        change_aliases_operations=[
            CreateAliasOperation(
                create_alias=CreateAlias(
                    collection_name=new_collection,
                    alias_name="nextcloud_content"
                )
            )
        ]
    )

    # 5. Verify new collection works
    test_results = await run_benchmark_queries()
    if test_results.recall < baseline_recall:
        # Rollback
        logger.error("New embeddings worse than baseline, rolling back")
        await rollback_migration()
        return False

    # 6. Delete old collection
    await qdrant_client.delete_collection(old_collection)
    logger.info("Migration complete!")
    return True

Downtime Mitigation:

• Use Qdrant collection aliases for atomic swap
• Reindex can happen in background
• Only brief downtime during alias swap (~1s)

Rollback Plan:

• Keep old collection until validation complete
• If new embeddings worse, swap alias back to old collection
• No data loss

4. Validation & Benchmarking

Before/After Comparison:

# tests/benchmarks/chunking_embedding_comparison.py

async def benchmark_chunking_embeddings():
    """
    Compare old vs. new chunking and embeddings on test queries.
    """
    test_queries = load_benchmark_queries()  # 100 queries with known relevant docs

    # Baseline (current)
    baseline_results = await run_queries(
        queries=test_queries,
        collection="nextcloud_content",  # Old: nomic-embed-text, word chunks
    )

    # New implementation
    new_results = await run_queries(
        queries=test_queries,
        collection="nextcloud_content_v2",  # New: mxbai-embed-large-v1, semantic chunks
    )

    # Compare metrics
    comparison = {
        "baseline": {
            "recall@10": calculate_recall(baseline_results, k=10),
            "precision@10": calculate_precision(baseline_results, k=10),
            "mrr": calculate_mrr(baseline_results),
            "zero_result_rate": calculate_zero_result_rate(baseline_results),
        },
        "new": {
            "recall@10": calculate_recall(new_results, k=10),
            "precision@10": calculate_precision(new_results, k=10),
            "mrr": calculate_mrr(new_results),
            "zero_result_rate": calculate_zero_result_rate(new_results),
        },
        "improvement": {
            "recall_improvement": (new_recall - baseline_recall) / baseline_recall,
            "precision_improvement": (new_precision - baseline_precision) / baseline_precision,
        }
    }

    return comparison

Success Criteria:

• Recall@10: Improve from ~52% to ≥75% (+40% improvement)
• Precision@10: Maintain ≥75% (no degradation)
• MRR: Improve from 0.58 to ≥0.70
• Zero-result rate: Reduce from 18% to ≤10%
• Indexing time: Maintain ≤10s per document

Validation Process:

1. Run benchmark on baseline (current implementation)
2. Implement changes
3. Run benchmark on new implementation
4. Compare metrics
5. If improvement ≥40%, proceed to production
6. If improvement <40%, investigate and iterate

Implementation Timeline

Week 1: Development & Testing

Day 1-2: Chunking Implementation

• Add langchain-text-splitters dependency
• Refactor document_chunker.py
• Update configuration (character-based chunk sizes)
• Write unit tests for semantic boundaries
• Validate: Chunks never break mid-sentence

Day 3-4: Embedding Implementation

• Update ollama_provider.py with dynamic dimension detection
• Update configuration (new model name)
• Deploy mxbai-embed-large-v1 to Ollama
• Test embedding generation with new model
• Validate: Embeddings are 1024-dim

Day 5: Migration Script

• Write migration script (collection creation, reindexing, alias swap)
• Test migration on staging environment
• Validate: No data loss, atomic swap works

Week 2: Reindexing & Validation

Day 1-2: Staging Reindex

• Run full reindex on staging environment
• Monitor indexing performance
• Validate: All documents indexed correctly

Day 3: Benchmarking

• Run benchmark queries on old collection (baseline)
• Run benchmark queries on new collection
• Compare metrics (recall, precision, MRR)
• Validate: ≥40% recall improvement

Day 4: Production Reindex

• Schedule maintenance window (optional, can run in background)
• Run migration script on production
• Monitor reindexing progress
• Atomic swap when complete

Day 5: Production Validation

• Monitor search quality metrics
• Collect user feedback
• Compare production metrics to staging
• Rollback if issues detected

Cost Analysis

Development Cost

• Time: 1-2 weeks (implementation + validation)
• Effort: 40-60 hours @ $100/hour = $4,000 - $6,000

Infrastructure Cost

• Storage: +30% (1024-dim vs. 768-dim)
  • Example: 1,000 notes × 3 chunks × 1024 dim × 4 bytes = 12 MB (negligible)
• Compute: +20% embedding time (50ms vs. 30ms per chunk)
  • Amortized over batch indexing, minimal impact
• No new infrastructure: Uses existing Ollama + Qdrant

Reindexing Cost (One-Time)

• Time: 2-4 hours for 1,000 documents
  • 1,000 docs × 3 chunks × 50ms = 150 seconds (~2.5 minutes embedding)
  • • Ollama processing time + Qdrant insertion
• Downtime: ~1 second (atomic alias swap)

Total Cost

• Initial: $4,000 - $6,000 (development + testing)
• Ongoing: $0 (no new infrastructure or API costs)

ROI

• Recall improvement: +40-60% (finding relevant documents)
• User satisfaction: Reduced zero-result queries (18% → 10%)
• Foundation: Enables future enhancements (reranking, hybrid search)
• Cost per % improvement: $100 - $150 (excellent ROI)

Consequences

Positive

1. Addresses Root Causes: Fixes fundamental issues (chunking, embeddings) not symptoms
2. High Impact: Expected 40-60% recall improvement from foundational changes
3. Future-Proof: Creates solid foundation for future enhancements (reranking, hybrid search, GraphRAG)
4. Simple: No architectural changes, no new infrastructure
5. Orthogonal: Improvements are independent, can be validated separately
6. Low Risk: Proven techniques (RecursiveCharacterTextSplitter, mxbai-embed-large-v1)
7. Maintainable: Standard libraries and models, easy to debug

Negative

1. Reindexing Required: 2-4 hours one-time cost (manageable, can run in background)
2. Storage Increase: +30% for higher-dimensional embeddings (12 MB vs. 9 MB for 1K docs)
3. Slower Indexing: +20% embedding time (50ms vs. 30ms per chunk)
4. Dependency: Adds langchain-text-splitters (minimal, well-maintained library)
5. Not a Complete Solution: May still need reranking/hybrid search for optimal recall (but solid foundation)

Neutral

1. Model Lock-In: Committed to mxbai-embed-large-v1, but can change later (another reindex)
2. Chunk Size Trade-offs: ~512 words is heuristic, may need tuning for specific content types

Monitoring & Success Metrics

Real-Time Metrics (Grafana)

Search Quality:

• semantic_search_recall_at_10 (target: ≥75%)
• semantic_search_precision_at_10 (target: ≥75%)
• semantic_search_mrr (target: ≥0.70)
• semantic_search_zero_result_rate (target: ≤10%)

Performance:

• semantic_search_latency_ms (p50, p95, p99)
• embedding_generation_time_ms
• indexing_throughput_docs_per_sec

Indexing:

• documents_indexed_total
• documents_pending
• indexing_errors_total

Weekly Validation

A/B Testing (if gradual rollout):

• 50% users: New embeddings
• 50% users: Old embeddings
• Compare metrics for 1 week
• Full rollout if new embeddings superior

User Feedback:

• Survey: "How satisfied are you with search results?" (1-5 scale)
• Track: Number of "search not working" support tickets
• Monitor: User-reported false negatives ("I know this doc exists")

Rollback Criteria

Automatic Rollback if:

• Recall decreases by >10% from baseline
• Error rate increases by >50%
• Query latency increases by >100%

Manual Rollback if:

• User complaints increase significantly
• Zero-result queries increase instead of decrease

Future Enhancements

These improvements create a solid foundation. Future enhancements (in order of priority):

1. Cross-Encoder Reranking (ADR-012)

   • Two-stage retrieval: broad recall (50 candidates) → precise reranking (top 10)
   • Expected: +15-20% additional recall improvement
   • Builds on: Better embeddings retrieve better candidates to rerank

2. Hybrid Search (ADR-013)

   • Combine vector search + BM25 keyword search
   • Expected: +10-15% additional recall (especially for exact matches)
   • Builds on: Semantic chunks provide better keyword match context

3. Multi-App Indexing (ADR-014)

   • Index calendar, deck, files (currently notes-only)
   • Expected: Expands searchable corpus 3-5x
   • Builds on: Proven chunking and embedding strategy

4. GraphRAG (ADR-015, conditional)

   • Only if: Global thematic queries needed OR corpus >10K documents
   • Expected: Relationship discovery, multi-hop reasoning
   • Builds on: High-quality embeddings improve graph construction

References

Research Papers

1. RecursiveCharacterTextSplitter

   • LangChain Documentation: https://python.langchain.com/docs/modules/data_connection/document_transformers/text_splitters/recursive_text_splitter
   • Proven technique used by major RAG systems

2. MTEB Leaderboard (Massive Text Embedding Benchmark)

   • https://huggingface.co/spaces/mteb/leaderboard
   • Comprehensive embedding model comparison

3. mxbai-embed-large

   • Model: https://huggingface.co/mixedbread-ai/mxbai-embed-large-v1
   • Best general-purpose embedding model (MTEB: 64.68)

Related ADRs

• ADR-003: Vector Database and Semantic Search Architecture (original implementation)
• ADR-008: MCP Sampling for Multi-App Semantic Search with RAG (answer generation)

Tools & Libraries

• LangChain Text Splitters: https://python.langchain.com/docs/modules/data_connection/document_transformers/
• Ollama Embedding Models: https://ollama.ai/library
• Qdrant Collections: https://qdrant.tech/documentation/concepts/collections/

Summary

This ADR addresses the root causes of poor semantic search recall:

1. Better Chunking: Semantic sentence-aware splitting (preserves context)
2. Better Embeddings: Upgrade to mxbai-embed-large-v1 (richer semantic space)

Expected Impact: 40-60% recall improvement with minimal cost and complexity.

Why This Approach:

• Fixes fundamentals before adding complexity
• Proven techniques (not experimental)
• Simple implementation (1-2 weeks)
• Creates foundation for future enhancements
• No new infrastructure or ongoing costs

Next Steps: Approve ADR → Implement changes → Reindex → Validate → Production rollout

Implementation Status

Completed (2025-11-18)

✅ Semantic Markdown-Aware Chunking (Option C1 + C3 Hybrid)

Implementation details:

• Replaced custom word-based chunking with MarkdownTextSplitter from LangChain
• Optimized for Nextcloud Notes markdown content with special handling for:
  • Headers (#, ##, ###, etc.)
  • Code blocks (```)
  • Lists (-, *, 1.)
  • Horizontal rules (---)
  • Paragraphs and sentences
• Maintained ChunkWithPosition interface for backward compatibility
• Updated configuration defaults:
  • DOCUMENT_CHUNK_SIZE: 512 words → 2048 characters
  • DOCUMENT_CHUNK_OVERLAP: 50 words → 200 characters
• Updated unit tests to verify position tracking and boundary preservation
• All tests passing with markdown-aware character-based chunking

Files Modified:

• nextcloud_mcp_server/vector/document_chunker.py - LangChain integration
• nextcloud_mcp_server/config.py - Character-based defaults
• tests/unit/test_document_chunker.py - Updated test suite

Dependencies Added:

• langchain-text-splitters>=1.0.0 (already present in pyproject.toml)

Migration Required:

• ⚠️ Full reindex required to apply new chunking strategy
• Existing documents in vector database use old word-based chunks
• See "Migration Strategy" section above for reindexing process

Pending

⏳ Embedding Model Upgrade (Option E1)

Still to be implemented:

• Switch from nomic-embed-text (768-dim) to mxbai-embed-large-v1 (1024-dim)
• Implement dynamic dimension detection in ollama_provider.py
• Create migration script for collection reindexing
• Run benchmarking to validate improvement
• Deploy to production with atomic collection swap

Estimated Timeline: 1-2 weeks for implementation and validation