108 Commits

Author SHA1 Message Date
Chris Coutinho 9d1a84af5a feat(auth): implement OAuth AS proxy to fix audience mismatch (ADR-023)
MCP clients like Claude Code were unable to use tools because tokens
obtained directly from Nextcloud had the wrong audience claim. The MCP
server now acts as its own OAuth Authorization Server, proxying auth
to Nextcloud with its own client_id so tokens have the correct audience.

New endpoints: /.well-known/oauth-authorization-server, /oauth/token,
/oauth/register. Modified /oauth/authorize from pass-through to
intermediary pattern. PRM now points authorization_servers to the MCP
server instead of Nextcloud.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 11:25:54 +01:00
Chris Coutinho 0d259d2dfd docs(ADR-022): concrete Smithery rationale + app password lifecycle
Address reviewer feedback on two fronts:

- Replace vague privacy-only Smithery deprecation rationale with concrete
  justification: free tier sunsetting March 2026 (primary), privacy as
  secondary. Updated in context, migration table, and Alternative 5.

- Add App Password Lifecycle Management section covering stale/revoked
  password detection (401 handling), login flow session cleanup (background
  task), and optional password rotation (APP_PASSWORD_MAX_AGE_DAYS).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-18 10:19:13 +01:00
Chris Coutinho dae2f276ae docs(ADR-022): address reviewer feedback
Changes based on review:

1. Add Nextcloud platform limitation section documenting OAuth/scope
   support by endpoint type (WebDAV supports OAuth, others don't)

2. Update MCP elicitation to show capability negotiation and graceful
   fallback - URL in error message when elicitation not supported

3. Simplify Smithery section - recommend self-hosted for privacy,
   don't detail platform changes

4. Expand re-auth section with scope merging behavior, scenarios table,
   and explicit design choice for tool-based re-auth over auto-elicitation

5. Make rate limiting configurable with environment variables and
   admin guidance by deployment size

6. Clarify OAuth alternative - keep simple now, revisit if Nextcloud
   adds scoped OAuth support

7. Expand verification steps with required tests, add recommended
   Nextcloud configuration, add required README security notice

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-18 09:23:32 +01:00
Chris Coutinho d94610d0ec docs: add ADR-022 for deployment mode consolidation via Login Flow v2
Proposes consolidating five deployment modes into two:
- Single-User: App password in env vars (trusted environment)
- Multi-User: Login Flow v2 for per-user app password acquisition

Key changes:
- Use Nextcloud Login Flow v2 (NC 16+) for delegated authentication
- Application-level scope enforcement (app passwords have no native scopes)
- MCP elicitation for seamless authorization prompting
- Astrolabe front-end integration for scope management UI
- Clear security posture documentation for administrators

This removes the need for upstream Nextcloud OAuth patches and simplifies
deployment while maintaining security through defense-in-depth.

Related: #521

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-18 09:23:32 +01:00
Chris Coutinho 1707b2e6e1 feat: add self-signed SSL certificate support for Nextcloud connections
Add NEXTCLOUD_VERIFY_SSL and NEXTCLOUD_CA_BUNDLE env vars to configure
TLS certificate verification for all outbound Nextcloud connections.
Centralizes SSL config via a new HTTP client factory (http.py) used by
all 27 Nextcloud-bound call sites, including API clients, OIDC endpoints,
OAuth flows, and health checks.

Closes #560

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-16 09:21:21 +01:00
Chris Coutinho 08d37a6597 docs: clean up astrolabe references after extraction
Remove astrolabe-specific docs and sections that belong in the
astrolabe repo. Update remaining references to point to the
astrolabe repo where appropriate.

- Fix .gitmodules SSH → HTTPS URL for astrolabe submodule
- Remove bump-version.yml stale "astrolabe" scope comment
- Delete blog-introducing-astrolabe.md (moved to astrolabe repo)
- Remove "Astrolabe Background Token Refresh" section from auth-flows.md
- Replace "Astrolabe User Setup" section in authentication.md with link
- Remove "Astrolabe Internal URL" section from configuration.md
- Remove "Webhook Presets (via Astrolabe UI)" from webhook guide

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 23:16:50 +01:00
Chris Coutinho c97ffe8e47 docs(astrolabe): Add initial blog post 2026-01-30 19:17:23 +00:00
Chris Coutinho c7882adb24 docs: add authentication flows reference by deployment mode
Create unified documentation covering authentication flows across all five
deployment modes. Documents three communication patterns (MCP Client → MCP
Server → Nextcloud, background sync, Astrolabe → MCP Server) with ASCII
sequence diagrams and implementation references.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-28 08:38:29 +01:00
Chris Coutinho c018268681 docs(astrolabe): add config docs and unit tests for internal URL
Address PR #487 reviewer feedback:

- Add documentation for `astrolabe_internal_url` config option
- Add unit tests for `IdpTokenRefresher::getNextcloudBaseUrl()`
- Fix CI workflow paths (astroglobe -> astrolabe)
- Add PHPUnit job to CI workflow for PHP 8.1, 8.2, 8.3
- Remove obsolete ApiTest that tested non-existent method

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-16 22:24:43 +01:00
Chris Coutinho 104a2ec9e3 test: Add unit tests for status endpoint OIDC config
Add unit tests for /api/v1/status endpoint focusing on OIDC config:
- Test hybrid mode (multi_user_basic + enable_offline_access) returns OIDC
- Test pure multi_user_basic mode without offline_access omits OIDC
- Test OAuth mode returns OIDC config
- Test single-user BasicAuth mode omits OIDC config
- Test partial OIDC config (only discovery_url or only issuer)

Also updates docs/authentication.md with Astrolabe hybrid mode setup:
- Two-step credential setup (OAuth + app password)
- Technical details for each credential type
- Request direction table explaining why two credentials needed

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-16 10:43:59 +01:00
Chris Coutinho 01ad2b3d21 refactor: Use get_settings() for vector sync enabled check
Replace direct os.getenv() calls with get_settings().vector_sync_enabled
to ensure consistent behavior with both VECTOR_SYNC_ENABLED (deprecated)
and ENABLE_SEMANTIC_SEARCH environment variables.

Also add webhook management documentation guide.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-14 20:30:51 +01:00
Chris Coutinho 4248b67b2e feat: Migrate to vue 3 2025-12-23 05:46:49 +01:00
Chris Coutinho 1a5bb10cd0 feat(config): consolidate configuration with smart dependency resolution (ADR-021)
Simplifies configuration by consolidating overlapping settings and adding
automatic dependency resolution. This makes semantic search configuration
significantly easier for users while maintaining 100% backward compatibility.

## Key Changes

### Variable Renaming (Backward Compatible)
- `VECTOR_SYNC_ENABLED` → `ENABLE_SEMANTIC_SEARCH` (old name still works)
- `ENABLE_OFFLINE_ACCESS` → `ENABLE_BACKGROUND_OPERATIONS` (old name still works)
- Deprecation warnings logged when old names used
- Old names will be removed in v1.0.0

### Smart Dependency Resolution
- `ENABLE_SEMANTIC_SEARCH` automatically enables background operations in multi-user modes
- No need to set both `ENABLE_OFFLINE_ACCESS` and `VECTOR_SYNC_ENABLED` anymore
- Single-user mode doesn't auto-enable background ops (not needed)

### Explicit Mode Selection (Optional)
- New `MCP_DEPLOYMENT_MODE` environment variable
- Valid values: single_user_basic, multi_user_basic, oauth_single_audience,
  oauth_token_exchange, smithery
- Removes ambiguity about which deployment mode is active
- Falls back to auto-detection if not set (existing behavior)

### Configuration Templates
- Reorganized `env.sample` by deployment mode with clear sections
- Added mode-specific quick-start templates:
  - `env.sample.single-user` - Simplest configuration
  - `env.sample.oauth-multi-user` - Recommended multi-user
  - `env.sample.oauth-advanced` - Token exchange mode

## Implementation Details

### Files Modified
- `nextcloud_mcp_server/config.py` - Smart dependency resolution helpers
- `nextcloud_mcp_server/config_validators.py` - Simplified validation, explicit mode
- `tests/unit/test_config_validators.py` - 19 new tests (60 total, all passing)
- `env.sample` - Reorganized by deployment mode
- `docs/configuration.md` - Complete rewrite with consolidated approach
- `docs/troubleshooting.md` - New consolidation troubleshooting section
- `README.md` - Updated variable references

### New Files
- `docs/ADR-021-configuration-consolidation.md` - Architecture decision record
- `docs/configuration-migration-v2.md` - Comprehensive migration guide
- `env.sample.single-user` - Single-user quick-start template
- `env.sample.oauth-multi-user` - OAuth multi-user quick-start template
- `env.sample.oauth-advanced` - Token exchange quick-start template

## User Impact

### Before (Confusing)
```bash
ENABLE_OFFLINE_ACCESS=true      # Why both?
VECTOR_SYNC_ENABLED=true        # What's the relationship?
```

### After (Simplified)
```bash
MCP_DEPLOYMENT_MODE=oauth_single_audience  # Explicit (optional)
ENABLE_SEMANTIC_SEARCH=true                # Auto-enables background ops!
```

### Benefits
- 📉 2 fewer variables to understand for semantic search
- 📋 Clear intent ("I want semantic search")
- 🎯 Explicit mode declaration available
- 🔄 100% backward compatible
-  All 265 unit tests passing

## Testing
- All 60 config validation tests passing
- 10 new tests for configuration consolidation
- 9 new tests for explicit mode selection
- Full unit test suite: 265 tests passing
- Backward compatibility verified

## Migration
Users can migrate at their own pace. Old variable names continue working
with deprecation warnings. See docs/configuration-migration-v2.md for
detailed migration instructions.

Related: ADR-021

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-21 20:36:36 +01:00
Chris Coutinho 4507359760 refactor(config): centralize configuration validation and simplify startup
Implement centralized configuration validation (ADR-020) to simplify
deployment mode detection and improve error messages.

Changes:
- Create ADR-020 documenting 5 deployment modes with required/optional config
- Add config_validators.py with validate_configuration() and mode detection
- Simplify app.py startup with single validation point at get_app()
- Remove duplicate is_oauth_mode() function (43 lines)
- Fix DeploymentMode mapping (only SELF_HOSTED and SMITHERY_STATELESS exist)
- Add comprehensive unit tests (41 tests covering all modes and edge cases)
- Add enable_multi_user_basic_auth to Settings and BasicAuthMiddleware

Docker Compose:
- Remove conflicting ENABLE_MULTI_USER_BASIC_AUTH from mcp-oauth service
- Add dedicated mcp-multi-user-basic service on port 8003

Test Results:
- 237/237 integration tests PASSED
- All deployment modes verified: single-user BasicAuth, multi-user BasicAuth,
  OAuth single-audience, OAuth token exchange (Keycloak), Smithery stateless

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-20 20:49:28 +01:00
Chris Coutinho d4c0da85da docs: update running guide to prioritize Docker usage
Updated docs/running.md to use Docker container examples instead of
direct Python commands. This aligns with the CLI change to require
explicit 'run' subcommand while maintaining backward compatibility
for Docker users (ENTRYPOINT includes 'run').

Key changes:
- Quick Start: Use Docker commands instead of uv run
- Running Locally → Running with Docker: All examples use Docker
- Development Mode: Added CLI subcommands documentation (run/db)
- Database Migrations: Documented Alembic integration for developers
- Server Options: Docker port mapping instead of --host/--port flags
- Process Management: Simplified to Docker Compose only (removed systemd)
- Performance Tuning: Production Docker Compose with resource limits
- Troubleshooting: Docker logs and debug commands

Updated Dockerfile ENTRYPOINT:
- Changed from: ["/app/.venv/bin/nextcloud-mcp-server", "--host", "0.0.0.0"]
- Changed to: ["/app/.venv/bin/nextcloud-mcp-server", "run", "--host", "0.0.0.0"]

No breaking changes for Docker/Helm users - container interface unchanged.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-18 00:02:09 +01:00
Chris Coutinho 3fa376905c feat: add Alembic database migration system
Implements Alembic for managing token storage database schema versions.
Migrations run automatically on startup with full backward compatibility.

**Changes:**
- Add Alembic dependency (1.14.0+) and SQLAlchemy (auto-installed)
- Create migration infrastructure in alembic/ directory
- Add initial migration (001) capturing current schema
- Modify RefreshTokenStorage.initialize() to run migrations via anyio
- Add CLI commands: db upgrade, current, history, downgrade, migrate
- Add comprehensive migration documentation

**Backward Compatibility:**
- Pre-Alembic databases automatically stamped with revision 001
- No schema changes for existing databases
- Automatic upgrade on first startup after update

**Migration Strategy:**
Three scenarios handled:
1. New database → Run migrations from scratch
2. Pre-Alembic database → Stamp with 001 (no changes)
3. Alembic-managed → Upgrade to latest

**Architecture:**
- Uses anyio.to_thread.run_sync() for structured concurrency
- Alembic env.py runs with anyio.run() in worker thread
- SQLite-friendly migration patterns documented
- No ThreadPoolExecutor needed (anyio handles it)

**CLI Usage:**
```bash
nextcloud-mcp-server db upgrade    # Upgrade to latest
nextcloud-mcp-server db current    # Show version
nextcloud-mcp-server db history    # View changelog
nextcloud-mcp-server db downgrade  # Rollback (with confirmation)
nextcloud-mcp-server db migrate "description"  # Create migration
```

**Testing:**
- All 13 webhook storage tests pass
- New/pre-Alembic database scenarios validated
- anyio integration tested

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-18 00:02:09 +01:00
Chris Coutinho d235dfa023 chore: Rename Astroglobe -> Astrolabe 2025-12-18 00:02:08 +01:00
Chris Coutinho be7f512244 docs: document deployment modes and Nextcloud log querying
Update ADR-018 with comprehensive deployment architecture and add
Nextcloud application log querying patterns to CLAUDE.md.

Changes:
- ADR-018 deployment modes documentation:
  - Mode 1: Basic single-user (development/simple)
  - Mode 2: Basic multi-user pass-through (no OIDC)
  - Mode 3: OAuth multi-user with progressive consent
  - Authentication flows for each mode
  - Communication path diagrams
  - Implementation examples
  - Use cases and limitations
- CLAUDE.md additions:
  - Nextcloud application log querying patterns
  - Common jq filters for debugging
  - Log structure documentation
  - App-specific filtering examples

Benefits:
- Clear guidance on deployment architecture selection
- Documented authentication flows for all scenarios
- Easier debugging with log query patterns
- Complete reference for mode-specific configurations

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-18 00:01:53 +01:00
Chris Coutinho 21817543ad feat(astrolabe): add Nextcloud PHP app for MCP server management
Adds a native Nextcloud app "Astroglobe" that provides:
- Personal settings: OAuth authorization for background MCP access
- Admin settings: Server status and vector sync monitoring
- API endpoints for MCP server communication

The app uses PKCE OAuth flow to obtain tokens for the MCP server,
enabling features like background vector sync per ADR-018.

Includes:
- PHP app structure (controllers, services, settings)
- Vue.js frontend components
- Docker compose mount configuration
- Installation hook for development testing
- ADR-018 documentation

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-18 00:00:40 +01:00
Chris Coutinho 20404cf3f2 feat(vector): add Deck card vector search with visualization support
Adds comprehensive vector search support for Nextcloud Deck cards,
including semantic search indexing, chunk preview in the vector viz UI,
and proper deep linking to cards.

**Vector Search Indexing**
- Add deck_card scanning in scanner.py (scan_deck_cards function)
- Index cards from non-archived, non-deleted boards
- Store metadata: board_id, board_title, stack_id, stack_title, card_type, duedate, owner
- Content structure: title + "\n\n" + description (matches indexing format)
- Incremental sync based on lastModified timestamp
- Deletion tracking with grace period

**Vector Visualization Support**
- Add deck_card handler in context.py for chunk preview expansion
- Include board_id in search result metadata (bm25_hybrid.py, semantic.py)
- Expose metadata in viz_routes.py JSON responses
- Update vector-viz.js to construct proper Deck URLs: /apps/deck/board/{board_id}/card/{card_id}
- Update vector_viz.html filter label from "Deck" to "Deck Cards"

**Bug Fixes**
- Skip soft-deleted boards (deletedAt > 0) to prevent 403 Forbidden errors
- Applies to scanner, processor, and context expansion code paths
- Deck API returns deleted boards but rejects stack access with 403

**Testing**
- Add integration tests in test_deck_vector_search.py:
  - test_deck_card_semantic_search: Filtered search with doc_type="deck_card"
  - test_deck_card_appears_in_cross_app_search: Cross-app search includes deck cards
  - test_deck_card_chunk_context: Chunk context fetching for viz preview

**Documentation**
- Update README.md: Add Deck cards to semantic search feature list
- Update semantic-search-architecture.md: Document deck_card support
- Update nc_semantic_search tool documentation

**Type Safety**
- Fix type narrowing for page_boundaries (could be None) using cast()
- Fix scanner.py payload None check for type safety

Resolves vector search for Deck cards across indexing, search, and visualization.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-13 23:51:18 +01:00
Chris Coutinho c8da826ef7 Merge pull request #382 from cbcoutinho/renovate/mcp-1.x
fix(deps): update dependency mcp to >=1.23,<1.24
2025-12-12 18:00:04 +01:00
Chris Coutinho ec70e70a5d fix: Disable DNS rebinding protection for containerized deployments
MCP Python SDK 1.23.0 introduced automatic DNS rebinding protection that
auto-enables when host="127.0.0.1" (the default). This breaks containerized
deployments (Kubernetes, Docker) because the protection rejects requests
with Host headers like "nextcloud-mcp-server.default.svc.cluster.local:8000".

Root cause:
- FastMCP defaults to host="127.0.0.1"
- SDK auto-enables DNS rebinding protection with allowed_hosts=["127.0.0.1:*", "localhost:*", "[::1]:*"]
- K8s/Docker requests use service DNS names or proxied hostnames
- Protection middleware rejects these requests (421 Misdirected Request)

Solution:
- Explicitly pass transport_security=TransportSecuritySettings(enable_dns_rebinding_protection=False)
- Applied to all three FastMCP initializations (OAuth, Smithery, BasicAuth)
- DNS rebinding attacks mitigated by OAuth authentication and network isolation

This fixes issue #373 and enables MCP 1.23.x upgrade in PR #382.

For detailed analysis, see docs/MCP-1.23-DNS-REBINDING-FIX.md
2025-12-12 17:30:22 +01:00
Chris Coutinho 19183ad14a fix: address PR review feedback
Address all reviewer comments from PR #387:

1.  Add unit tests for annotations (tests/server/test_annotations.py)
   - 10 comprehensive test functions validating all annotation patterns
   - Tests for titles, read-only, destructive, idempotent operations
   - Validates specific ADR-017 decisions (webdav write, semantic search)
   - Cross-category consistency checks

2.  Fix nc_webdav_write_file idempotency classification
   - Changed from idempotentHint=False to idempotentHint=True
   - Rationale: Uses HTTP PUT without version control
   - Writing same content to same path = same end state (idempotent)

3.  Fix semantic search openWorldHint inconsistency
   - Changed from openWorldHint=False to openWorldHint=True
   - Rationale: Consistent with other Nextcloud tools
   - Nextcloud is external to MCP server (indexed data is implementation detail)

4.  Update ADR-017 with resolved decisions
   - Converted Open Questions to Resolved Questions
   - Added detailed rationale for webdav write and semantic search
   - Updated status from Proposed to Implemented
   - Added decision timeline with dates

5.  Add MCP Tool Annotations guidelines to CLAUDE.md
   - Comprehensive section with code examples for all patterns
   - Key principles documented (idempotency, destructive, open world)
   - References ADR-017 for detailed rationale

All OAuth tools verified to have proper annotations (oauth_tools.py lines 686-751).
2025-12-11 13:50:55 +01:00
Chris Coutinho e1412320a7 feat: add MCP tool annotations for enhanced UX
Add ToolAnnotations to all 105+ MCP tools across 13 modules to enable
better client-side UX with human-readable titles and behavioral hints.

Changes:
- Add title and ToolAnnotations to all @mcp.tool() decorators
- Apply correct idempotency classification per ADR-017
- Add destructiveHint for delete operations
- Set openWorldHint=False for semantic search (internal data only)

Modules updated:
- OAuth (4 tools): Authentication and provisioning
- Notes (7 tools): Note management
- WebDAV (11 tools): File operations
- Semantic (3 tools): Semantic search and RAG
- Calendar (16 tools): Events and todos
- Contacts (7 tools): Address book management
- Sharing (5 tools): File/folder sharing
- Tables (6 tools): Structured data
- Deck (25 tools): Kanban board management
- Cookbook (13 tools): Recipe management
- News (8 tools): RSS feed reader

Annotation patterns:
- Read operations: readOnlyHint=True, openWorldHint=True
- Create operations: idempotentHint=False, openWorldHint=True
- Update operations: idempotentHint=False, openWorldHint=True
- Delete operations: destructiveHint=True, idempotentHint=True, openWorldHint=True

See docs/ADR-017-mcp-tool-annotations.md for rationale and implementation details.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-11 12:45:02 +01:00
Chris Coutinho 482ef89a73 docs: Add ADR-016 for Smithery stateless deployment
Add architecture decision record for supporting Smithery-hosted MCP
server in a stateless mode for multi-user public Nextcloud instances.

Key decisions:
- New SMITHERY_STATELESS deployment mode alongside SELF_HOSTED
- Session-based configuration (nextcloud_url, username, app_password)
- Feature subset excluding semantic search and background sync
- Admin UI (/app) excluded in Smithery mode
- Per-request client creation from session config

This enables users to try the MCP server without self-hosting
infrastructure while supporting multiple Nextcloud instances.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-22 17:13:18 +01:00
Chris Coutinho c126c3ec03 fix: Preserve 3D plot camera and improve documentation
This commit addresses PR feedback and fixes plot camera behavior.

## JavaScript Fix - Camera Preservation
- Changed plot update strategy from recreating layout to using Plotly.restyle()
- Query point visibility now toggles via restyle() which only modifies trace visibility
- Camera position/zoom naturally preserved since layout remains untouched
- Resolves jumpy plot behavior when toggling "Show Query Point" checkbox

Related: nextcloud_mcp_server/auth/static/vector-viz.js:58-73

## Documentation Improvements
- Condensed vector-sync-ui.md from 316 to 94 lines (~70% reduction)
- Removed redundant FAQ section (content merged into main sections)
- Simplified use cases from 4 detailed sections to 3 focused paragraphs
- Streamlined troubleshooting to 3 common issues
- Merged technical details into overview section
- Retained all essential information while improving readability

## Screenshot Updates
Removed old/outdated images (5 files):
- rag-workflow-bidirectional-final.png
- rag-workflow-prominent-llm.png
- rag-workflow-simple-final.png
- vector-viz-interface.png
- welcome-page.png

Replaced with current screenshots (3 files):
- vector-viz-document-types-2col.png - Now shows plot + results
- vector-viz-chunk-context.png - Centered content view
- vector-viz-results.png - Updated results list

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-19 14:10:53 +01:00
Chris Coutinho 53689d076b feat: Improve vector visualization with static assets and fixes
- Extract CSS and JavaScript into separate static files
  - Created nextcloud_mcp_server/auth/static/vector-viz.css
  - Created nextcloud_mcp_server/auth/static/vector-viz.js
  - Updated templates to reference external assets

- Fix vector visualization issues:
  - Normalize vectors before PCA to match Qdrant's cosine distance
  - Add zero-norm and NaN detection/handling for large datasets
  - Enable responsive Plotly sizing (autosize + responsive config)
  - Widen plot area to full viewport width with minimized margins

- Improve visualization accuracy:
  - Query point now positioned correctly relative to documents
  - Handles 200+ points without JSON serialization errors
  - Full-width plot maximizes screen space utilization

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-19 04:10:44 +01:00
Chris Coutinho eec923eff5 feat: Replace custom document chunker with LangChain MarkdownTextSplitter
Migrates from custom word-based chunking to LangChain's MarkdownTextSplitter
for better semantic search quality. This implements the chunking portion of
ADR-011.

Changes:
- Replace custom regex word chunker with MarkdownTextSplitter
- Optimized for Markdown content (headers, code blocks, lists)
- Convert from word-based (512 words) to character-based (2048 chars) chunking
- Maintain backward-compatible ChunkWithPosition interface
- Update configuration defaults and validation
- Update all unit tests (12/12 passing)

Benefits:
- Respects markdown structure boundaries
- Never breaks code blocks or headers mid-chunk
- Preserves semantic coherence within chunks
- Expected 20-30% improvement in recall quality
- Industry-standard approach (used by production RAG systems)

Note: Full reindex required to apply new chunking to existing documents.
Current vector database still contains old word-based chunks.

Related: ADR-011 (Improving Semantic Search Quality)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-18 12:17:23 +01:00
Chris Coutinho 6cfd7e2729 feat: add configurable fusion algorithms for BM25 hybrid search
Added support for two fusion algorithms (RRF and DBSF) to combine dense
semantic and sparse BM25 search results, with comprehensive documentation
and unit tests.

Changes:
- Added fusion parameter to nc_semantic_search and nc_semantic_search_answer tools
- Updated ADR-014 with detailed comparison of RRF vs DBSF fusion algorithms
- Added unit tests for fusion algorithm initialization and validation
- Updated search_method in responses to include fusion type (e.g., "bm25_hybrid_rrf")

Fusion Algorithms:
- RRF (Reciprocal Rank Fusion): Default, rank-based, general-purpose
- DBSF (Distribution-Based Score Fusion): Score normalization using statistics

RRF is recommended for most use cases due to its robustness and established
track record. DBSF may provide better results when retrieval systems have
very different score distributions.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-17 06:48:43 +01:00
Chris Coutinho ed33b39062 docs: fix ADR-014 template text and numbering
- Remove template instruction text from line 1
- Fix ADR numbering from 007 to 014 to match filename
2025-11-16 12:08:37 +01:00
Chris Coutinho 1504df6fb5 Merge branch 'master' into feature/bedrock 2025-11-16 12:08:23 +01:00
Chris Coutinho c28fc955ca Merge origin/master into feature/bm25
Resolved conflicts:
- viz_routes.py: Kept bm25's extract_dense_vector() function for robust vector handling
- hybrid.py: Removed (bm25 uses native Qdrant RRF fusion instead)
- uv.lock: Regenerated after accepting master's dependencies

This merge brings in:
- RAG evaluation framework (ADR-013)
- Performance optimizations (double-fetch elimination)
- Migration from asyncio to anyio
- OpenTelemetry tracing improvements
- Notes app enhancements

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-16 11:52:40 +01:00
Chris Coutinho 5b484c9226 feat: add unified provider architecture with Amazon Bedrock support
Refactored LLM provider infrastructure to support sustainable additions of new providers with both embedding and text generation capabilities.

## Major Changes

### Unified Provider Architecture (ADR-015)
- Created `nextcloud_mcp_server/providers/` with unified Provider ABC
- Providers now support optional capabilities (embeddings and/or generation)
- Auto-detection registry with priority: Bedrock → Ollama → Simple
- Backward compatible - existing code continues to work

### New Providers
- **BedrockProvider**: Full Amazon Bedrock integration
  - Embeddings: Titan Embed, Cohere Embed models
  - Generation: Claude, Llama, Titan Text, Mistral models
  - Model-specific request/response handling
  - AWS credential chain integration
- **OllamaProvider**: Migrated with both capabilities support
- **AnthropicProvider**: Moved from test code to production providers
- **SimpleProvider**: Migrated in-memory fallback provider

### Breaking Changes
None - full backward compatibility maintained:
- `embedding.get_embedding_service()` still works
- RAG evaluation tests updated to use unified providers
- All existing tests pass (127 unit tests)

### Testing
- Added 9 comprehensive Bedrock unit tests with mocked boto3
- All existing unit tests pass
- Type checking (ty) and linting (ruff) pass
- Verified backward compatibility

### Documentation
- `docs/ADR-015-unified-provider-architecture.md`: Comprehensive ADR
- `docs/bedrock-setup.md`: AWS setup guide with IAM permissions
- `CLAUDE.md`: Updated with provider architecture section

### Dependencies
- Added `boto3>=1.35.0` to dev dependencies (optional)

## Environment Variables

### Bedrock
- `AWS_REGION`: AWS region (e.g., "us-east-1")
- `BEDROCK_EMBEDDING_MODEL`: Model ID for embeddings
- `BEDROCK_GENERATION_MODEL`: Model ID for generation
- `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`: Optional credentials

### Ollama
- `OLLAMA_BASE_URL`: API URL
- `OLLAMA_EMBEDDING_MODEL`: Embedding model (default: "nomic-embed-text")
- `OLLAMA_GENERATION_MODEL`: Generation model

## AWS Bedrock Permissions Required

Minimal IAM policy:
```json
{
  "Effect": "Allow",
  "Action": ["bedrock:InvokeModel"],
  "Resource": ["arn:aws:bedrock:*::foundation-model/*"]
}
```

See `docs/bedrock-setup.md` for detailed setup instructions.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-16 11:36:58 +01:00
Chris Coutinho 8799450c7d Merge pull request #306 from cbcoutinho/rag-evaluation
feat: RAG evaluation framework with performance improvements
2025-11-16 11:17:41 +01:00
Chris Coutinho c4bf077050 feat: Add OpenTelemetry tracing to @instrument_tool decorator
Enhances the @instrument_tool decorator to create distributed traces
for all MCP tool executions, improving observability and debugging.

Changes:
- Modified @instrument_tool to wrap tool execution in trace_operation
- Added automatic span creation with mcp.tool.* span names
- Sanitized tool arguments before adding to span attributes
  (excludes password, token, secret, api_key, etag, ctx)
- Limited argument strings to 500 characters to prevent huge spans
- Maintained existing Prometheus metrics functionality
- Updated docs/observability.md to reflect correct decorator name
- Added comprehensive unit tests

All ~50+ MCP tools now emit traces automatically without code changes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-16 11:16:05 +01:00
Chris Coutinho 2aa82d849c Merge branch 'feature/bm25' 2025-11-16 07:57:36 +01:00
Chris Coutinho 6fe5596c13 feat: Implement BM25 hybrid search with native Qdrant RRF fusion
Replace custom keyword/fuzzy search algorithms with industry-standard BM25
sparse vectors, combined with dense semantic vectors using Qdrant's native
Reciprocal Rank Fusion (RRF). This consolidates search architecture and
improves relevance for both semantic and keyword queries.

Key changes:
- Add fastembed dependency for BM25 sparse vector generation
- Update Qdrant collection schema to support named vectors (dense + sparse)
- Create BM25SparseEmbeddingProvider using FastEmbed's Qdrant/bm25 model
- Implement BM25HybridSearchAlgorithm with native Qdrant RRF prefetch
- Update document processor to generate both dense and sparse embeddings
- Simplify nc_semantic_search() tool to use BM25 hybrid only
- Remove legacy keyword.py, fuzzy.py, and custom hybrid.py (736 lines)
- Update ADR-014 with implementation notes and test results

Benefits:
- Consolidated architecture (single Qdrant database)
- Native database-level RRF fusion (more efficient)
- Industry-standard BM25 (replaces brittle custom keyword search)
- Better relevance across semantic and keyword queries
- Simplified codebase (-285 net lines)

Tests: All 125 tests passing (118 unit, 7 integration)

Implements ADR-014: Replace Custom Keyword Search with BM25 Hybrid Search

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-16 06:59:44 +01:00
Chris Coutinho f5bc3e3bc3 docs: init ADR 2025-11-16 06:24:25 +01:00
Chris Coutinho fca8ab0cfd Merge remote-tracking branch 'origin/master' into rag-evaluation 2025-11-16 00:32:59 +01:00
Chris Coutinho c272ddd82d feat: implement RAG evaluation framework with CLI tooling
- Add ADR-013 documenting RAG evaluation architecture
- Implement two-part evaluation: Context Recall (retrieval) + Answer Correctness (generation)
- Create Click CLI for ground truth generation and corpus upload
- Add pytest fixtures and tests for retrieval/generation quality
- Use BeIR/nfcorpus dataset with 5 selected test queries
- Support Ollama and Anthropic LLM providers
- Generate synthetic ground truth answers offline
- Add comprehensive documentation in tests/rag_evaluation/README.md

The framework separates one-time setup (generate/upload) from test execution,
making tests much faster (~6-12 min vs ~15-25 min per run).

Tests are manual only (not in CI) and require external LLM access.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-15 23:11:21 +01:00
Chris Coutinho 56bd85c0f7 docs: Emphasize server-side processing in ADR-012 viz pane
Updates ADR-012 to clarify that all search and filtering operations
must happen server-side, not in the browser.

Key changes:
- Enhanced viz pane data flow showing server-side processing
- Added performance benefits section (384x bandwidth reduction)
- Detailed server-side filtering approach:
  * Query execution via search/algorithms.py
  * User ID filtering (multi-tenant security)
  * Document type filtering
  * PCA reduction (768-dim → 2D) on server
  * Only 2D coordinates + metadata sent to client
- Updated Phase 3 implementation plan:
  * Remove ALL client-side search logic
  * Implement /app/vector-viz server endpoint
  * htmx form submission for queries
  * Performance optimizations (caching, streaming)

This ensures:
- Minimal bandwidth usage (only 2 floats per doc vs 768)
- Client handles only visualization, not computation
- Can visualize 10,000+ documents without client lag
- Raw vectors never leave server (security)
- Same search logic as MCP tool (consistency)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-15 00:02:54 +01:00
Chris Coutinho 5e67277049 docs: Add architecture diagrams and viz pane UI to ADR-012
Enhances ADR-012 with detailed architecture visualization and UI mockup
for the vector visualization pane.

Added sections:
- Architecture diagram showing MCP tool and viz pane integration
- Data flow diagrams for both MCP requests and viz pane interactions
- Detailed UI mockup with ASCII art showing:
  * Search configuration controls
  * Algorithm selector with weight sliders
  * Interactive 2D scatter plot (Plotly.js)
  * Results panel with scores
  * Performance comparison table
- Technology stack details (htmx, Alpine.js, Plotly.js, Tailwind CSS)

The diagrams illustrate how the viz pane and MCP tool share the same
search algorithm implementations from search/algorithms.py, ensuring
consistency between user testing interface and programmatic API.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-15 00:00:40 +01:00
Chris Coutinho 66a7109130 docs: Add ADR-012 for unified multi-algorithm search
Proposes unified search architecture with client-configurable algorithm
selection and weighting. Addresses the need for flexible search options
beyond pure semantic search.

Key features:
- Four algorithms: semantic, keyword, fuzzy, hybrid
- Client-configurable weights for hybrid search
- Shared implementation between viz pane and MCP tools
- Reciprocal Rank Fusion (RRF) for result combination
- Backward compatible with existing nc_semantic_search()

Implements designs from:
- ADR-003: Hybrid search with RRF (previously unimplemented)
- ADR-001: Token-based keyword search (previously unimplemented)

Supersedes ADR-011's placeholder for "ADR-013: Hybrid Search"

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-14 23:56:09 +01:00
Chris Coutinho 73b3d80026 Merge pull request #294 from cbcoutinho/feature/app_api
docs: Add ADR-011 for hybrid OAuth + AppAPI deployment architecture
2025-11-13 23:43:25 +01:00
Chris Coutinho 26099d643d docs: Update ADR-011 to rejected status with Context Agent validation
After comprehensive research, the hybrid OAuth + AppAPI architecture is NOT
being implemented due to fundamental architectural incompatibilities.

Key updates:
- Status: Proposed → Not Planned
- Added validation from Nextcloud Context Agent project
- Context Agent (official NC ExApp with MCP) faces IDENTICAL limitations
- Proves constraints are architectural, not implementation-specific

Context Agent findings:
- ExApp with MCP server endpoint (~28 tools exposed)
- Uses Task Processing API for confirmations (NOT MCP elicitation)
- Works around AppAPI proxy limitations by changing protocol
- MCP endpoint is secondary feature with documented constraints
- Primary use: In-app Assistant integration, not external MCP clients

Critical features impossible through AppAPI proxy:
-  MCP sampling (eliminates RAG/LLM features)
-  MCP elicitation (user prompts)
-  Real-time progress updates
-  Bidirectional streaming
- Validated by Context Agent facing same limitations

Decision rationale:
- MCP requires multi-turn nested interactions
- AppAPI provides stateless request/response proxy only
- No implementation effort can bridge this fundamental gap
- Would require complete AppAPI redesign (WebSocket, message routing)
- Even official Nextcloud projects work around these limitations

Alternative considered for future:
- Register as Task Processing provider (different product)
- Use Nextcloud Assistant UI (not external MCP clients)
- Accept different capabilities (no sampling, custom flows)

OAuth mode remains sole solution for external MCP client integration.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-13 23:30:14 +01:00
Chris Coutinho c3023d2cc3 feat: Complete Phase 5 - Instrument all 93 MCP tools
Applied @instrument_tool decorator to all 86 remaining tools
across 8 server files.

Instrumented files:
- calendar.py: 16 tools
- contacts.py: 7 tools
- deck.py: 25 tools
- webdav.py: 11 tools
- tables.py: 6 tools
- sharing.py: 5 tools
- cookbook.py: 13 tools
- semantic.py: 3 tools

Total: 93 tools instrumented (7 in notes.py + 86 in other files)

These metrics populate:
- MCP Tool Calls panel (by tool name and status)
- MCP Tool Duration panel (histogram)
- MCP Tool Errors panel (by tool name and error type)

This completes PR #295 - All 5 phases of metrics instrumentation done:
 Phase 1: Queue size metrics (2 locations)
 Phase 2: Health checks (1 location)
 Phase 3: Database operations (3 methods)
 Phase 4: OAuth token metrics (3 locations)
 Phase 5: MCP tool metrics (93 tools)

All 34 dashboard panels now have data sources.
2025-11-13 16:58:44 +01:00
Chris Coutinho ff3123a190 docs: Add ADR-011 for hybrid OAuth + AppAPI deployment architecture
This ADR documents the architectural decision to support both OAuth and
AppAPI (ExApp) deployment modes in a single codebase with 90%+ code sharing.

Key additions:
- Comprehensive analysis of AppAPI limitations and challenges
- Feature parity matrix comparing OAuth vs AppAPI modes
- Resolution of critical open questions via research:
  * Non-browser client authentication (app passwords/OAuth)
  * Streaming transport compatibility (buffered, not real-time)
  * Callbacks/webhooks (MCP notifications not possible in AppAPI)
- Detailed implementation plan with 4 phases (10 days)
- Mode-aware architecture with abstraction layer

Critical findings:
- AppAPI mode does NOT support MCP sampling (RAG features)
- No real-time progress updates (use Nextcloud notifications)
- Buffered streaming only (Streamable HTTP works, WebSocket doesn't)
- Requires app password support in AppAPI proxy

Deployment mode selection:
- OAuth: Multi-tenant, external clients, sampling/RAG, real-time updates
- AppAPI: Single-tenant, simplified install, native UI, admin-controlled

Related to investigation of ~/Software/app_api/ and ~/Software/nc_py_api/
for AppAPI integration patterns.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-13 13:10:21 +01:00
Chris Coutinho d86a185e04 refactor: move webapp from /user/page to /app
Simplified the webapp routing structure by consolidating the admin UI
to a single clean endpoint.

Changes:
- Moved webapp from /user/page to /app (root of mount)
- Removed /user JSON endpoint (no longer needed)
- Updated mount point from /user to /app in app.py
- Updated all route path checks (3 locations)
- Updated OAuth redirects to point to /app
- Updated all HTMX endpoint references
- Updated documentation (ADR-007, CHANGELOG)
- Added redirect from /app to /app/ for trailing slash handling

New Route Structure:
- /app - Main webapp (HTML UI with tabs)
- /app/revoke - Revoke background access
- /app/webhooks - Webhook management UI
- /app/webhooks/enable/{preset_id} - Enable webhook preset
- /app/webhooks/disable/{preset_id} - Disable webhook preset

Breaking Change: Existing bookmarks to /user or /user/page will no longer work.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-11 20:53:43 +01:00
Chris Coutinho f4759e424d feat: add webhook management UI and BeforeNodeDeletedEvent support
Added comprehensive webhook management capabilities including:

Webhook Client & API:
- Added WebhooksClient for Nextcloud webhooks API integration
- Create, list, update, and delete webhooks programmatically
- Support for event filters in webhook registration

Webhook Presets:
- Added preset system for common webhook configurations
- notes_sync: BeforeNodeDeletedEvent for Notes file operations
- calendar_sync: Calendar events (create, update, delete)
- deck_sync: Deck card operations
- files_sync: File system changes
- forms_sync: Form submissions (conditional)
- Filter presets by installed apps

Admin UI:
- Added multi-pane app view with tabs (User Info, Vector Sync, Webhooks)
- Webhooks tab for admin users only
- Enable/disable preset webhooks via UI
- View currently registered webhooks
- Uses htmx for dynamic loading and Alpine.js for tab state
- Admin permission checking via OCS API

CLI Improvements:
- Refactored CLI to separate module (cli.py)
- Updated entry point in pyproject.toml

BeforeNodeDeletedEvent Fix:
- Updated ADR-010 to document NodeDeletedEvent issue
- BeforeNodeDeletedEvent includes node.id before deletion
- NodeDeletedEvent lacks node.id (file already deleted)
- Implemented per Nextcloud maintainer recommendation

Testing:
- Added comprehensive webhook client tests
- Added webhook preset filtering tests
- Added admin permission tests

Configuration:
- Updated docker-compose.yml Qdrant settings

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-11 20:35:08 +01:00
Chris Coutinho b58e7238ae feat: validate Nextcloud webhook schemas and document findings
Manual testing of Nextcloud webhook_listeners app to validate webhook
payloads against ADR-010 expected schemas and document implementation
requirements for webhook-based vector synchronization.

## Changes

- Add test webhook endpoint at /webhooks/nextcloud in app.py
  - Captures and logs webhook payloads for analysis
  - Returns 200 OK immediately for webhook delivery confirmation

- Create webhook-testing-findings.md with comprehensive test results
  - Captured payloads for 5/6 webhook event types
  - Critical findings: missing node.id in deletions, type mismatches
  - Implementation recommendations with code examples

- Update ADR-010 with Appendix A: Manual Webhook Testing Results
  - Document actual vs expected webhook behavior
  - Update event mapping table with tested webhook status
  - Add 6 specific implementation recommendations
  - Include testing implications for future development

## Testing Results

 NodeCreatedEvent - fires correctly, includes node.id (integer)
 NodeWrittenEvent - fires correctly, includes node.id (integer)
 NodeDeletedEvent - fires but missing node.id field (path only)
 CalendarObjectCreatedEvent - fires correctly with full iCal
 CalendarObjectUpdatedEvent - fires correctly with full iCal
 CalendarObjectDeletedEvent - does not fire (potential NC bug)

## Key Findings

1. NodeDeletedEvent missing node.id field - requires path-based fallback
2. node.id returns integer not string - needs casting for consistency
3. Multiple webhooks fire per operation - needs deduplication logic
4. Calendar deletion webhooks don't fire - reported as issue #53497
5. Calendar webhooks include full iCal content - enables rich parsing

## GitHub Issues

- Created issue #56371: NodeDeletedEvent missing node.id field
- Commented on issue #53497: CalendarObjectDeletedEvent not firing

Closes #283

---

_This commit was generated with the help of AI, and reviewed by a Human_
2025-11-11 12:13:20 +01:00