Replace direct os.getenv() calls with get_settings().vector_sync_enabled
to ensure consistent behavior with both VECTOR_SYNC_ENABLED (deprecated)
and ENABLE_SEMANTIC_SEARCH environment variables.
Also add webhook management documentation guide.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Previously, the multi-user BasicAuth mode attempted to retrieve app passwords
via OAuth client_credentials grant, which Nextcloud OIDC doesn't support.
This fix implements local storage for app passwords:
- Add app_passwords table via Alembic migration (002)
- Add store/get/delete methods to RefreshTokenStorage
- Add management API endpoints for app password provisioning:
- POST /api/v1/users/{user_id}/app-password
- GET /api/v1/users/{user_id}/app-password
- DELETE /api/v1/users/{user_id}/app-password
- Update oauth_sync.py to read from local storage
- Update Astrolabe to send app passwords to MCP server after validation
- Add app-hook to configure mcp_server_url in Nextcloud
The flow is now:
1. User creates app password in Nextcloud Security settings
2. User enters it in Astrolabe Personal Settings
3. Astrolabe validates against Nextcloud, then sends to MCP server
4. MCP server stores encrypted app password locally
5. Background sync uses locally stored password
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Fixes NC PHP app (Astrolabe) OAuth integration by making token validation
more lenient for management API access.
Problem:
- Astrolabe calls Nextcloud OIDC token endpoint via internal URL (http://localhost)
- Tokens are issued with iss: http://localhost (internal)
- MCP server expects iss: http://localhost:8080 (external)
- Token validation failed with "Invalid issuer"
Solution:
- Add skip_issuer_check parameter to _verify_jwt_signature()
- verify_token_for_management_api() now skips both audience and issuer checks
- Security maintained: signature still verified, authorization checked by API
Also includes related fixes from previous session:
- Update test selectors for Vue 3 UI ("Enable Semantic Search")
- Fix OIDC discovery URL transformation in OAuthController.php
- Add overwrite.cli.url to setup hook for proper external URLs
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Migrate all direct ENABLE_OFFLINE_ACCESS environment variable checks to
use settings.enable_offline_access, which handles both the new
ENABLE_BACKGROUND_OPERATIONS and deprecated ENABLE_OFFLINE_ACCESS vars.
Also fixes JWT issuer validation in Docker by using NEXTCLOUD_PUBLIC_ISSUER_URL
when set, resolving 401 errors caused by internal/external URL mismatch.
Changes:
- app.py: Use settings for offline access checks in setup_oauth_config,
register_oauth_client, and tool registration
- oauth_tools.py: Use settings in provision_nextcloud_access and check_logged_in
- management.py: Use settings in get_user_session
- scope_authorization.py: Use settings in require_scopes decorator
- Remove unused os imports after migration
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Remove URL rewriting logic from MCP server that was converting
public URLs to internal Docker URLs. This was a workaround for
Nextcloud's overwritehost setting forcing URLs to localhost:8080.
Changes:
- Remove OIDC endpoint rewriting in app.py (setup_oauth_config)
- Remove OIDC_JWKS_URI override support (no longer needed)
- Remove URL rewriting in browser_oauth_routes.py
- Remove URL rewriting in token_broker.py
- Update Helm chart values and README
- Add hybrid auth setup unit tests
- Update Astrolabe admin UI for Vue 3
The proper fix is in the previous commit which removes the
overwritehost setting from Nextcloud, allowing it to respect
the Host header from incoming requests.
Implement multi-user BasicAuth pass-through mode (ADR-020) where each
request includes BasicAuth credentials that are forwarded to Nextcloud
APIs without persistent storage.
Changes:
- Add _get_client_from_basic_auth() in context.py to extract credentials
from Authorization header (set by BasicAuthMiddleware)
- Add AstrolabeClient for app password provisioning via Astrolabe API
- Update oauth_sync.py with dual credential support (app passwords first,
then refresh tokens as fallback)
- Simplify oauth_tools.py provisioning logic
- Add integration tests for app password provisioning and multi-user BasicAuth
Features:
- Stateless multi-user mode: credentials passed per-request
- Optional background sync via app passwords (stored in Astrolabe)
- Falls back to refresh tokens if app password not available
- Test coverage for provisioning flow and pass-through mode
Related: ADR-019 (Multi-user BasicAuth), ADR-020 (Deployment Modes)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Add type casts for Starlette app state access
- Add assertions for cipher, card, board, stack after initialization
- Add None checks for XML element text attributes
- Handle __package__ being None in tracing setup
- Fix TokenBrokerService initialization to use storage credentials
Resolves 42 type warnings from ty-check, enabling CI linting to pass.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Implements Alembic for managing token storage database schema versions.
Migrations run automatically on startup with full backward compatibility.
**Changes:**
- Add Alembic dependency (1.14.0+) and SQLAlchemy (auto-installed)
- Create migration infrastructure in alembic/ directory
- Add initial migration (001) capturing current schema
- Modify RefreshTokenStorage.initialize() to run migrations via anyio
- Add CLI commands: db upgrade, current, history, downgrade, migrate
- Add comprehensive migration documentation
**Backward Compatibility:**
- Pre-Alembic databases automatically stamped with revision 001
- No schema changes for existing databases
- Automatic upgrade on first startup after update
**Migration Strategy:**
Three scenarios handled:
1. New database → Run migrations from scratch
2. Pre-Alembic database → Stamp with 001 (no changes)
3. Alembic-managed → Upgrade to latest
**Architecture:**
- Uses anyio.to_thread.run_sync() for structured concurrency
- Alembic env.py runs with anyio.run() in worker thread
- SQLite-friendly migration patterns documented
- No ThreadPoolExecutor needed (anyio handles it)
**CLI Usage:**
```bash
nextcloud-mcp-server db upgrade # Upgrade to latest
nextcloud-mcp-server db current # Show version
nextcloud-mcp-server db history # View changelog
nextcloud-mcp-server db downgrade # Rollback (with confirmation)
nextcloud-mcp-server db migrate "description" # Create migration
```
**Testing:**
- All 13 webhook storage tests pass
- New/pre-Alembic database scenarios validated
- anyio integration tested
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit addresses 4 critical issues identified in code review:
1. **Token Rotation Race Condition** (token_broker.py)
- Added per-user locking mechanism to prevent concurrent refresh token corruption
- Implemented double-check pattern for cache after acquiring lock
- Users can now safely refresh concurrently without token desync
2. **Hardcoded OAuth Client ID** (PHP files)
- Made client ID configurable via `astroglobe_client_id` in system config
- Updated McpServerClient to provide getClientId() method
- Injected McpServerClient into IdpTokenRefresher and OAuthController
- Updated admin settings UI to display client ID configuration status
- App gracefully handles missing client ID with warnings in admin UI
3. **Missing Cache Invalidation** (management.py:revoke_user_access)
- Added cache.invalidate() call when revoking user access
- Ensures both storage AND cache are cleared atomically
- Prevents stale cached tokens from being used after revocation
4. **Error Message Exposure** (management.py)
- Created _sanitize_error_for_client() helper function
- Updated all error handlers to log detailed errors internally
- Returns generic messages to clients to prevent information leakage
- Protects against exposing database paths, API URLs, tokens, etc.
All changes are backward compatible and preserve existing functionality.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Adds comprehensive vector search support for Nextcloud Deck cards,
including semantic search indexing, chunk preview in the vector viz UI,
and proper deep linking to cards.
**Vector Search Indexing**
- Add deck_card scanning in scanner.py (scan_deck_cards function)
- Index cards from non-archived, non-deleted boards
- Store metadata: board_id, board_title, stack_id, stack_title, card_type, duedate, owner
- Content structure: title + "\n\n" + description (matches indexing format)
- Incremental sync based on lastModified timestamp
- Deletion tracking with grace period
**Vector Visualization Support**
- Add deck_card handler in context.py for chunk preview expansion
- Include board_id in search result metadata (bm25_hybrid.py, semantic.py)
- Expose metadata in viz_routes.py JSON responses
- Update vector-viz.js to construct proper Deck URLs: /apps/deck/board/{board_id}/card/{card_id}
- Update vector_viz.html filter label from "Deck" to "Deck Cards"
**Bug Fixes**
- Skip soft-deleted boards (deletedAt > 0) to prevent 403 Forbidden errors
- Applies to scanner, processor, and context expansion code paths
- Deck API returns deleted boards but rejects stack access with 403
**Testing**
- Add integration tests in test_deck_vector_search.py:
- test_deck_card_semantic_search: Filtered search with doc_type="deck_card"
- test_deck_card_appears_in_cross_app_search: Cross-app search includes deck cards
- test_deck_card_chunk_context: Chunk context fetching for viz preview
**Documentation**
- Update README.md: Add Deck cards to semantic search feature list
- Update semantic-search-architecture.md: Document deck_card support
- Update nc_semantic_search tool documentation
**Type Safety**
- Fix type narrowing for page_boundaries (could be None) using cast()
- Fix scanner.py payload None check for type safety
Resolves vector search for Deck cards across indexing, search, and visualization.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add support for news_item document type in the vector visualization page:
- Add "News" checkbox to document type filter options
- Add URL handler to link news items to /apps/news/item/{id}
- Add content fetching for news items in chunk context expansion
This enables users to search and view news articles in the vector
visualization, with clickable links back to Nextcloud News and the
ability to expand chunks to see full article context.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Fixed 8 type checker errors across the codebase:
- vector/scanner.py: Handle None scroll results with null-safe iteration
- search/{bm25_hybrid,semantic}.py: Add None checks for result.payload
- auth/{unified_verifier,webhook_routes}.py: Assert non-None auth credentials
- client/webdav.py: Add None checks before int() conversions
- providers/openai.py: Assert embedding_model is not None
- search/algorithms.py: Explicitly type doc_types set and cast values
- observability/logging_config.py: Match parent class signature (log_data)
Also fixed test_create_tag_creates_system_tag to match WebDAV implementation
(was testing OCS API endpoint, now tests correct WebDAV endpoint with
Content-Location header).
Type checker: 0 errors (down from 8), 20 warnings (ignored)
Tests: All 192 unit tests passing
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Move scanner/processor tasks from FastMCP session lifespan to Starlette
server lifespan (correct architecture: background tasks run once at
server level, not per-session)
- Change default CLI transport from SSE to streamable-http
- Remove SSE transport option from CLI (SSE is deprecated)
- Remove SSE client session factory from test fixtures
- Add tracing instrumentation to BM25 hybrid search operations for
better observability
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Replace sequential Qdrant scroll calls with batch retrieve
(50 HTTP requests → 1 request, ~50x faster vector fetch)
- Add point_id to SearchResult to enable batch retrieval by Qdrant point ID
- Reuse query embedding from search algorithm in viz_routes
(eliminates redundant embedding call, saves ~30ms)
- Make BM25 encode() async with thread pool to avoid blocking event loop
(~4.4s was blocking, now properly async)
- Run PCA computation in thread pool to avoid blocking event loop
(~1.2s was blocking, now properly async)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Previously, pymupdf4llm.to_markdown() was called twice - once in
PyMuPDFProcessor during indexing and again in PDFHighlighter during
visualization. Different image path lengths caused different character
offsets, leading to highlighted pages not matching their chunks.
Also fixed issue where all chunks on the same page showed all highlights
instead of just their own highlight. Now restores original page contents
between chunks using xref stream caching.
Changes:
- Add PDFHighlighter class requiring pre-computed page_boundaries and
full_text from document processor (no fallback extraction)
- Pass pre-computed data from processor to highlighter
- Extract page-relative portion of chunk text for cross-page chunks
- Add bounding box highlighting using text anchor search
- Run highlight generation in parallel with embedding/BM25
- Cache and restore page contents to isolate highlights per chunk
Results: Highlighting success rate improved from 51% to 95% (121/128).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Implements optional context expansion for semantic search results that
fetches adjacent chunks (N-1 and N+1) from Qdrant to provide before/after
context. Removes configurable chunk overlap (default 200 chars) to avoid
duplicate text appearing in both context and excerpt.
Key changes:
- Add include_context and context_chars parameters to nc_semantic_search
and nc_semantic_search_answer tools
- Implement Qdrant cache fast path for chunk retrieval (avoids re-fetching
and re-parsing documents, especially important for PDFs)
- Add _get_chunk_by_index_from_qdrant() to fetch adjacent chunks
- Remove chunk overlap from before_context (last N chars) and after_context
(first N chars) to prevent duplicate text
- Fetch context in parallel with anyio.Semaphore (max 20 concurrent)
- Pass through page_number from SearchResult to SemanticSearchResult
- Remove document-level deduplication (keep chunk-level dedup from algorithm)
Context expansion is opt-in via include_context=true parameter. When enabled:
- Populates has_context_expansion, marked_text, before_context, after_context
- Adds truncation flags when context exceeds context_chars limit
- Falls back to document fetch for legacy data with truncated excerpts
Related: nextcloud_mcp_server/search/context.py:87-382,
nextcloud_mcp_server/server/semantic.py:161-255
Introduces a placeholder-based state tracking system to prevent duplicate
document processing during the gap between scanner queuing and processor
completion.
**Key Changes:**
1. **Placeholder Helper Functions** (`vector/placeholder.py`):
- `write_placeholder_point()` - Creates zero-vector placeholder when queuing
- `query_document_metadata()` - Queries for existing entry (placeholder or real)
- `delete_placeholder_point()` - Removes placeholder before writing real vectors
- `get_placeholder_filter()` - Filters placeholders from user-facing queries
2. **Scanner Updates** (`vector/scanner.py`):
- Replace `indexed_at` comparison with `modified_at` comparison
- Write placeholder before queuing each document
- Query per-document metadata instead of bulk-querying indexed_at
- Fixes bug where files were resubmitted every scan cycle
3. **Processor Updates** (`vector/processor.py`):
- Delete placeholder before upserting real vectors
- Ensures no duplicate points in Qdrant
4. **Query Filters** (all search files):
- Add `get_placeholder_filter()` to all user-facing queries
- Ensures placeholders never appear in search results or visualizations
- Applied to: bm25_hybrid.py, semantic.py, viz_routes.py, algorithms.py
**Architecture:**
- Placeholders use zero vectors with dimension from embedding service
- Payload includes `is_placeholder: True` flag for filtering
- Status field tracks: "pending", "processing", "completed", "failed"
- Deterministic UUIDs using uuid5 for consistent point IDs
**Impact:**
- Eliminates duplicate processing of same documents
- Fixes race condition where long-running documents get queued multiple times
- Prevents scanner from resubmitting files every scan cycle
- Maintains clean separation between in-flight and indexed documents
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
When vector visualization search returns zero results, the code was returning
query_coords: null, which caused JavaScript error "can't access property 0,
queryCoords is null" when the frontend tried to access the array.
Changed to return empty array [] to match expected type and prevent crash.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- algorithms.py: Revert SearchResult.id to int (all docs use int IDs now)
- semantic.py: Revert SemanticSearchResult.id to int, remove Union import
- viz_routes.py: Remove str() conversion when querying doc_id from Qdrant
- viz_routes.py: Convert doc_id from query param to int in chunk context
Fixes vector visualization which was collapsing all chunks to a single
point because Qdrant queries were failing to match doc_id (string vs int).
Major improvements to vector visualization page:
- Refactor PCA to display individual chunks instead of averaged documents
- Add context expansion module for fetching surrounding text from notes and PDFs
- Update deduplication to use (doc_id, doc_type, chunk_start, chunk_end) keys
- Fix Alpine.js rendering with chunk-specific keys including offsets
- Refactor authentication helper to return NextcloudClient for better reuse
- Add async context manager support to NextcloudClient
Technical details:
- viz_routes.py: Fetch specific chunk vectors instead of averaging per document
- context.py: New module supporting both notes and PDF text extraction via PyMuPDF
- search algorithms: Extract page_number, chunk_index, total_chunks from Qdrant
- vector-viz.js/html: Use chunk positions in expansion tracking keys
This enables users to see which specific chunks match their query
and view them with surrounding context in the PCA visualization.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
This commit addresses multiple issues with async operations, PDF metadata
extraction, and type safety in document processing and search.
## Async/Await Fixes
- processor.py:259 - Added await for chunker.chunk_text(content)
- processor.py:270 - Added await for bm25_service.encode_batch(chunk_texts)
- tests/unit/test_document_chunker.py - Converted all 12 test methods to async
## PDF Metadata Enhancement
- pymupdf.py:143 - Added file_size metadata extraction
- pymupdf.py:145-206 - Refactored to extract text page-by-page
- Manually loop through pages instead of using page_chunks=True
- Generate page_boundaries metadata for precise page tracking
- Works around pymupdf.layout.activate() breaking page_chunks=True
- processor.py:32-66 - Added assign_page_numbers() helper function
- Assigns page numbers to chunks based on overlap with page boundaries
- Handles chunks spanning multiple pages
- processor.py:298-300 - Call assign_page_numbers() for PDF files
## Type Safety Fixes
- bm25_hybrid.py:184 - Removed int() conversion of doc_id
- semantic.py:131 - Removed int() conversion of doc_id
- viz_routes.py:275 - Removed int() conversion of doc_id
- Added comments documenting that doc_id can be int (notes) or str (file paths)
## Testing
- All 18 tests passing (12 unit + 6 integration)
- No type errors in modified files
- Container logs show successful processing
- Vector viz searches working correctly
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Get container dimensions before creating Plotly layout to render at correct size immediately
- Add init() method with window resize listener for responsive plot sizing
- Remove post-render resize call (no longer needed with explicit dimensions)
- Improve colorbar positioning and scene domain configuration
This eliminates the visual "jump" during initial render and ensures the plot resizes smoothly when the browser window changes size.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Two fixes for the vector visualization page:
1. **CSS Loading Fix**: Moved CSS <link> from vector_viz.html fragment
to user_info.html <head> block. HTMX fragments don't process <link>
tags in <head>, causing unstyled page. Now CSS loads correctly.
2. **Camera Preservation**: Modified renderPlot() to preserve camera
position when toggling query point visibility. Previously, toggling
the "Show Query Point" checkbox would reset zoom/rotation to default.
Now reads existing camera settings from plot before updating.
Related: nextcloud_mcp_server/auth/static/vector-viz.js:123-130
Related: nextcloud_mcp_server/auth/templates/user_info.html:12
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Extract CSS and JavaScript into separate static files
- Created nextcloud_mcp_server/auth/static/vector-viz.css
- Created nextcloud_mcp_server/auth/static/vector-viz.js
- Updated templates to reference external assets
- Fix vector visualization issues:
- Normalize vectors before PCA to match Qdrant's cosine distance
- Add zero-norm and NaN detection/handling for large datasets
- Enable responsive Plotly sizing (autosize + responsive config)
- Widen plot area to full viewport width with minimized margins
- Improve visualization accuracy:
- Query point now positioned correctly relative to documents
- Handles 200+ points without JSON serialization errors
- Full-width plot maximizes screen space utilization
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
This commit updates the web interface to better align with Nextcloud's
design system and improve the Vector Viz layout.
Changes:
- Replace emoji icons with Material Design SVG icons for better
consistency with Nextcloud apps
- Simplify navigation styling with minimal padding and subtle active
states (250px width)
- Update CSS variables to match Nextcloud design system
- Restructure Vector Viz from two-column to single-column vertical
layout for better plot visibility
- Move search controls to compact horizontal grid at top
- Make navigation toggle always visible (not just on mobile)
- Fix plot container sizing with overflow:visible to prevent colorbar
clipping
- Remove heavy shadows and custom card styling for cleaner aesthetic
- Add error and success page templates with consistent styling
Technical details:
- Preserve Alpine.js for reactive functionality
- Use CSS Grid for responsive horizontal controls layout
- Add smooth transitions for navigation collapse/expand
- Maintain HTMX for dynamic content loading
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
This commit enhances the vector visualization interface with better score
transparency and improved UX:
**Dual-Score Display:**
- Store original algorithm scores before normalization (viz_routes.py:203)
- Display both raw and normalized scores: "Raw Score: 0.842 (89% relative)"
- Update plot hover text with dual scores (userinfo_routes.py:740)
- Fixes issue where all queries showed at least one 100% match regardless
of actual relevance (normalization artifact)
**UI Improvements:**
1. Fusion Method dropdown: Changed from x-show to :disabled
- Prevents jarring layout shift when switching algorithms
- Dropdown stays visible but grayed out when Semantic is selected
- Better UX with opacity: 0.5 and cursor: not-allowed
2. Score Threshold: Changed step from 0.1 to "any"
- Allows arbitrary float precision (0.7, 0.85, 0.123)
- Users can now fine-tune threshold values
3. Document Types: Converted multi-select to checkbox grid
- Replaced clunky Ctrl/Cmd multi-select listbox
- Checkbox grid with cleaner layout
- Positioned left of Score Threshold and Result Limit inputs
- More intuitive UX
**Technical Details:**
- Raw score ranges vary by algorithm:
- Semantic: 0.0-1.0 (cosine similarity)
- BM25 RRF: ~0.001-0.033 (Reciprocal Rank Fusion)
- BM25 DBSF: Can exceed 1.0 (Distribution-Based Score Fusion)
- Normalized scores (0-1) used for visual encoding (marker size, color)
- Original scores preserved in API response via getattr fallback
Files modified:
- nextcloud_mcp_server/auth/viz_routes.py (store original_score)
- nextcloud_mcp_server/auth/templates/vector_viz.html (UI controls)
- nextcloud_mcp_server/auth/userinfo_routes.py (plot hover text)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Fix false-positive validation error where DBSF (Distribution-Based Score
Fusion) correctly produces scores > 1.0 but SearchResult validation
incorrectly rejected them.
**Root Cause**: SearchResult.__post_init__() enforced scores in [0.0, 1.0]
range, but DBSF sums normalized scores from multiple retrieval systems
(dense semantic + sparse BM25), resulting in scores like 1.55 when both
systems strongly agree a document is relevant.
**Changes**:
- Relaxed validation to allow any score ≥ 0.0 (algorithms.py:147-157)
- Updated SearchResult and SemanticSearchResult documentation to explain
score ranges for RRF ([0.0, 1.0]) vs DBSF (unbounded)
- Added comprehensive test coverage for both fusion methods
- Added DBSF fusion option to vector visualization UI
- Updated viz routes and vizApp() to support fusion parameter selection
**Testing**: All 157 unit tests pass, type checking passes, ruff passes
Fixes error: "Configuration error: Score must be between 0.0 and 1.0, got 1.1528953"
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- viz_routes.py: Extract "dense" vector from named vector dict
- semantic.py: Specify using="dense" for BM25 hybrid collections
- Fixes "X must be 2D array" error in hybrid search
- Fixes "Dense vector is not found" error in semantic search
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
The visualization UI was still using the old 'hybrid' algorithm name and
weight parameters that were replaced by the BM25 hybrid search refactor.
This caused "Unknown algorithm: hybrid" errors when using the search
& visualize feature.
Changes:
- Update default algorithm from 'hybrid' to 'bm25_hybrid'
- Update default scoreThreshold from 0.7 to 0.0 to match backend
- Remove deprecated semanticWeight, keywordWeight, fuzzyWeight parameters
- Remove weight parameters from search request
Fixes the visualization search functionality after BM25 hybrid refactor.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Remove obsolete search algorithm imports (Fuzzy, Keyword, Hybrid)
- Update UI to only show Semantic and BM25 Hybrid algorithms
- Replace manual weight controls with RRF fusion info message
- Update default algorithm from "hybrid" to "bm25_hybrid"
- Remove weight parameters (semantic_weight, keyword_weight, fuzzy_weight)
- Update score_threshold default from 0.7 to 0.0 for RRF scoring
- Document ty type checker in CLAUDE.md
Fixes unresolved-import type errors after BM25 refactor.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
This commit addresses critical performance issues with vector visualization
search (reducing time from 40s to ~2s) and improves result visualization
through better visual encoding.
## Performance Fixes
### 1. Fix blocking sleep in retry decorator (base.py:51)
- Changed `time.sleep(5)` to `await anyio.sleep(5)` in @retry_on_429
- Prevents entire event loop from freezing during rate limit retries
- Impact: Reduced search time from 22s to 16s initially
### 2. Add concurrency limiting for verification (verification.py:77-93)
- Added `anyio.Semaphore(20)` to limit concurrent HTTP requests
- Prevents connection pool exhaustion (RequestError) from 90+ simultaneous requests
- Fixes false filtering (was filtering 77/90 results incorrectly)
- Note: Semaphore still in code but verification removed from viz endpoint
### 3. Remove unnecessary verification from viz endpoint (viz_routes.py:483-486)
- Visualization only needs Qdrant metadata (title, excerpt), not full content
- Verification only required for sampling (LLM needs full note content)
- Impact: Reduced search time from 43.7s to ~2s (final fix)
### 4. Restore streaming scanner pattern (scanner.py)
- Process notes one-at-a-time using async generator
- Avoids loading all notes into memory
## Visualization Improvements
### 5. Result-relative score normalization (viz_routes.py:489-504)
- Normalize scores within result set: best=1.0, worst=0.0
- Removes arbitrary RRF normalization (theoretical max didn't make sense)
- Makes visual encoding meaningful regardless of algorithm scores
### 6. Power scaling for marker sizes (userinfo_routes.py:743)
- Changed from linear `8 + (score * 12)` to power `6 + (score² * 14)`
- Creates dramatic visual contrast: 0.0→6px, 0.5→9.5px, 1.0→20px
- Combined with opacity (0.2-1.0) for clear visual hierarchy
### 7. Multi-channel visual encoding (userinfo_routes.py:740-745)
- Size: Exponentially scaled with score²
- Opacity: Linear 0.2-1.0 (keeps all points visible)
- Color: Viridis gradient (blue→yellow)
- Effect: Top results are large/bright/opaque, context results small/dim/transparent
## Result
- Search time: 40s → ~2s (20x faster)
- Visual contrast: Subtle → dramatic (clear result hierarchy)
- No arbitrary cutoffs: All results visible, best naturally highlighted
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Replace asyncio primitives with anyio equivalents throughout the codebase
to establish a single async pattern. This provides better structured
concurrency with automatic cancellation on errors and aligns with the
pytest anyio configuration.
Changes:
- hybrid.py: Replace asyncio.gather() with anyio task groups
- token_broker.py: Replace asyncio.Lock() with anyio.Lock()
- storage.py: Replace asyncio.run() with anyio.run()
- app.py: Replace tg.start_soon() with await tg.start() for task status
- processor.py: Add task_status parameter for structured startup
- scanner.py: Add task_status parameter for structured startup
- CLAUDE.md: Update async/await patterns guidance
The change from start_soon() to await tg.start() enables proper task
initialization signaling, ensuring background tasks are ready before
proceeding. This follows anyio best practices for structured concurrency.
All 118 unit tests pass with the new implementation.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Move access verification from individual search algorithms to final output
stage, eliminating redundant API calls and improving performance.
## Changes
**New:**
- `search/verification.py`: Centralized verification using anyio task groups
- Deduplicates results by (doc_id, doc_type) before verification
- Verifies all unique documents in parallel using structured concurrency
- Filters out inaccessible documents in single pass
**Modified Search Algorithms:**
- `search/semantic.py`: Removed _deduplicate_and_verify() and _verify_document_access()
- `search/keyword.py`: Removed _verify_access() and parallel verification
- `search/fuzzy.py`: Removed _verify_access() and parallel verification
- `search/hybrid.py`: Removed nextcloud_client parameter passing
All algorithms now return unverified results from Qdrant payload.
**Modified Output Stages:**
- `server/semantic.py`: Added verify_search_results() call after search
- `auth/viz_routes.py`: Added verify_search_results() call after search
Both endpoints now verify access once at final stage with deduplication.
## Performance Impact
**Before:**
- Hybrid mode (limit=10): 30 API calls (10 per algorithm × 3 algorithms)
- Single algorithm: 10-20 API calls (with verification buffer)
**After:**
- Hybrid mode (limit=10): 10 API calls (deduplicated verification)
- Single algorithm: 10 API calls (deduplicated verification)
**Performance Gain:** 3x reduction in API calls for hybrid search
## Architecture Benefits
- **Separation of concerns**: Algorithms handle scoring, output stage handles security
- **Deduplication**: Each document verified exactly once
- **Parallel execution**: All verifications run concurrently via anyio task groups
- **Consistency**: Same verification logic across MCP tools and viz endpoints
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Vector Visualization Improvements:
- Add interactive vector viz tab with Alpine.js and Plotly.js to user info page
- Refactor viz route CSS for better scoping and maintainability
- Remove unused nextcloud_host variable
Performance Optimizations:
- Parallelize access verification in fuzzy and keyword search algorithms
- Use asyncio.gather() to verify multiple documents concurrently
- Add exception handling with return_exceptions=True for resilience
Dependencies:
- Update third_party/oidc submodule to include RFC 9728 resource_url support
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Skip tracing for /app/vector-sync/status to reduce noise from HTMX polling.
Metrics collection continues for this endpoint.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Move Webhooks tab to the right (User Info | Vector Sync | Vector Viz | Webhooks)
- Use request.user.display_name instead of session for viz routes
- Fixes session middleware error when accessing via iframe
- Add /app/vector-viz endpoint for interactive search testing
- Implement server-side PCA dimensionality reduction (768-dim → 2D)
- Support multi-select document type filter for cross-app search
- Support all search algorithms: semantic, keyword, fuzzy, hybrid
- Display 2D scatter plot of vector embeddings using Plotly
- Show search results with scores and document types
- Register viz routes in app.py
- Add custom PCA implementation using numpy eigendecomposition
- Replace sklearn.decomposition.PCA with custom implementation
- Maintains same API (fit, transform, fit_transform)
- Supports explained_variance_ratio_ for variance analysis
- Removes scikit-learn dependency from project
- Add type hints and assertion for type safety
Fixes layout issues on the webhooks admin tab:
- Add min-height to container to fill viewport consistently
- Use CSS Grid to overlay tab panes without jumpiness
- Add smooth htmx fade transitions for content swaps
- Adjust vector sync polling interval from 3s to 10s
- Add .playwright-mcp/ to gitignore for test screenshots
The CSS Grid approach allows tabs to overlay without absolute positioning,
preventing content cutoff while maintaining smooth transitions without
container resizing jumps.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Implement real-time vector sync status updates in the /app UI without
requiring page refreshes. The status (indexed documents, pending
documents, sync state) now updates automatically every 3 seconds.
Changes:
- Add vector_sync_status_fragment() endpoint that returns HTML fragment
with current vector sync status
- Modify user_info_html() to use htmx loading for vector sync section
with hx-trigger="load" on initial render
- Status fragment includes hx-trigger="every 3s" for continuous polling
- Add /app/vector-sync/status route to browser_routes
The implementation uses htmx (already loaded on page) to poll the status
endpoint, providing near real-time updates with minimal overhead. The
endpoint queries Qdrant for indexed count and reads from memory streams
for pending count, returning only the status HTML fragment.
Pattern follows existing webhook management UI which also uses htmx
for dynamic loading.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Chris Coutinho