Commit Graph

201 Commits

Author SHA1 Message Date
Chris Coutinho 4a5766b84e feat(config): enable DCR for multi-user BasicAuth with offline access
Allows multi-user BasicAuth mode to use Dynamic Client Registration (DCR)
for OAuth credentials when ENABLE_OFFLINE_ACCESS is enabled, making it
consistent with OAuth modes and reducing configuration burden.

**Changes:**

Configuration Validation:
- Relaxed OAuth credential requirements for multi-user BasicAuth
- OAuth credentials now optional when offline access enabled
- Will use DCR as fallback if NEXTCLOUD_OIDC_CLIENT_ID/SECRET not set
- Updated validation to log info instead of error when DCR will be used

Startup Logic (app.py):
- Added DCR workflow for multi-user BasicAuth before uvicorn starts
- Creates oauth_context for management APIs when offline access enabled
- Allows Astrolabe to authenticate management API calls with OAuth
- DCR runs synchronously at same lifecycle point as OAuth modes
- Added traceback import for better error logging
- Fixed type assertions for nextcloud_host
- Fixed undefined variable references in vector sync logging

Management API:
- Improved auth mode detection using proper detect_auth_mode()
- Added auth_mode field to /status endpoint:
  * "basic" - Single-user BasicAuth
  * "multi_user_basic" - Multi-user BasicAuth
  * "oauth" - OAuth modes
  * "smithery" - Smithery stateless
- Added supports_app_passwords indicator for multi-user BasicAuth

Docker Compose:
- Updated mcp-multi-user-basic service configuration:
  * Enabled vector sync (VECTOR_SYNC_ENABLED=true)
  * Added ENABLE_OFFLINE_ACCESS=true for app password support
  * Added NEXTCLOUD_MCP_SERVER_URL for Astrolabe integration
  * Documented optional static OAuth credentials

Testing:
- Updated test_config_validators.py to expect DCR fallback
- Enhanced configure_astrolabe_for_mcp_server fixture with verification
- Added debug logging to test_users_setup fixture

**Workflow:**
1. User configures ENABLE_OFFLINE_ACCESS=true
2. Server checks for static NEXTCLOUD_OIDC_CLIENT_ID/SECRET
3. If not found, performs DCR before uvicorn starts
4. DCR registers client with Nextcloud OIDC provider
5. OAuth credentials used for Astrolabe management API auth
6. Background sync can retrieve user app passwords via Astrolabe

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-22 19:43:24 +01:00
Chris Coutinho 4507359760 refactor(config): centralize configuration validation and simplify startup
Implement centralized configuration validation (ADR-020) to simplify
deployment mode detection and improve error messages.

Changes:
- Create ADR-020 documenting 5 deployment modes with required/optional config
- Add config_validators.py with validate_configuration() and mode detection
- Simplify app.py startup with single validation point at get_app()
- Remove duplicate is_oauth_mode() function (43 lines)
- Fix DeploymentMode mapping (only SELF_HOSTED and SMITHERY_STATELESS exist)
- Add comprehensive unit tests (41 tests covering all modes and edge cases)
- Add enable_multi_user_basic_auth to Settings and BasicAuthMiddleware

Docker Compose:
- Remove conflicting ENABLE_MULTI_USER_BASIC_AUTH from mcp-oauth service
- Add dedicated mcp-multi-user-basic service on port 8003

Test Results:
- 237/237 integration tests PASSED
- All deployment modes verified: single-user BasicAuth, multi-user BasicAuth,
  OAuth single-audience, OAuth token exchange (Keycloak), Smithery stateless

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-20 20:49:28 +01:00
Chris Coutinho d4d1a332fb fix(astrolabe): Update screenshots 2025-12-20 00:52:21 +01:00
Chris Coutinho 60d80970a4 Merge pull request #401 from cbcoutinho/feature/nc-app-ui
feat(astrolabe): Nextcloud app UI with PDF viewer, webhooks, and OAuth refresh
2025-12-19 14:22:42 +01:00
Chris Coutinho d235dfa023 chore: Rename Astroglobe -> Astrolabe 2025-12-18 00:02:08 +01:00
Chris Coutinho 21817543ad feat(astrolabe): add Nextcloud PHP app for MCP server management
Adds a native Nextcloud app "Astroglobe" that provides:
- Personal settings: OAuth authorization for background MCP access
- Admin settings: Server status and vector sync monitoring
- API endpoints for MCP server communication

The app uses PKCE OAuth flow to obtain tokens for the MCP server,
enabling features like background vector sync per ADR-018.

Includes:
- PHP app structure (controllers, services, settings)
- Vue.js frontend components
- Docker compose mount configuration
- Installation hook for development testing
- ADR-018 documentation

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-18 00:00:40 +01:00
renovate-bot-cbcoutinho[bot] 76251e935e chore(deps): update docker.io/library/nextcloud:32.0.3 docker digest to 53231a9 2025-12-15 11:06:31 +00:00
renovate-bot-cbcoutinho[bot] 4741d60e4c chore(deps): update docker.io/library/nginx:alpine docker digest to 052b75a 2025-12-14 11:06:52 +00:00
renovate-bot-cbcoutinho[bot] 76ae1c3603 chore(deps): update docker.io/library/nextcloud docker tag to v32.0.3 2025-12-12 11:09:06 +00:00
renovate-bot-cbcoutinho[bot] f1247817d3 chore(deps): update docker.io/library/nginx:alpine docker digest to 289deca 2025-12-10 11:10:31 +00:00
renovate-bot-cbcoutinho[bot] 0561b55af5 chore(deps): update docker.io/library/nextcloud:32.0.2 docker digest to 04cc195 2025-12-09 11:07:29 +00:00
renovate-bot-cbcoutinho[bot] 0aa307f0b6 chore(deps): update qdrant/qdrant:v1.16.2 docker digest to dab6de3 2025-12-05 11:11:18 +00:00
renovate-bot-cbcoutinho[bot] c05beb66e9 chore(deps): update qdrant/qdrant docker tag to v1.16.2 2025-12-04 11:09:16 +00:00
renovate-bot-cbcoutinho[bot] ce6bbff389 chore(deps): update quay.io/keycloak/keycloak docker tag to v26.4.7 2025-12-01 11:07:45 +00:00
renovate-bot-cbcoutinho[bot] f8e67519e1 chore(deps): update quay.io/keycloak/keycloak docker tag to v26.4.6 2025-11-25 23:06:05 +00:00
renovate-bot-cbcoutinho[bot] 41e128190b chore(deps): update qdrant/qdrant docker tag to v1.16.1 2025-11-25 17:06:22 +00:00
renovate-bot-cbcoutinho[bot] e94b8ff714 chore(deps): update docker.io/library/nextcloud:32.0.2 docker digest to 8cb1dc8 2025-11-23 17:04:03 +00:00
Chris Coutinho d881373dce ci: Remove third_party from app mounts 2025-11-23 05:48:17 +01:00
Chris Coutinho 3b41776110 Merge pull request #342 from cbcoutinho/feature/smithery
feat: Add Smithery stateless deployment support (ADR-016)
2025-11-22 19:39:53 +01:00
Chris Coutinho 7b22e5be0f build: add smithery image to docker compose 2025-11-22 19:06:25 +01:00
Chris Coutinho 8c79993280 Merge pull request #334 from cbcoutinho/renovate/docker.io-library-redis-alpine
chore(deps): update docker.io/library/redis:alpine docker digest to 6cbef35
2025-11-21 14:24:54 +01:00
renovate-bot-cbcoutinho[bot] 03495d901d chore(deps): update docker.io/library/nextcloud docker tag to v32.0.2 2025-11-21 05:14:28 +00:00
renovate-bot-cbcoutinho[bot] 0b6a02075c chore(deps): update docker.io/library/redis:alpine docker digest to 6cbef35 2025-11-20 23:10:43 +00:00
renovate-bot-cbcoutinho[bot] b0735bae85 chore(deps): update docker.io/library/nextcloud:32.0.1 docker digest to d572839 2025-11-19 05:08:00 +00:00
renovate-bot-cbcoutinho[bot] 7ddf8370e6 chore(deps): update docker.io/library/redis:alpine docker digest to 5013e94 2025-11-18 23:10:41 +00:00
Chris Coutinho c2fd87a5d3 Merge pull request #324 from cbcoutinho/renovate/docker.io-library-nextcloud-32.0.1
chore(deps): update docker.io/library/nextcloud:32.0.1 docker digest to f6232ea
2025-11-18 14:03:38 +01:00
Chris Coutinho b488d69939 Merge pull request #326 from cbcoutinho/feature/notes2
feat: Replace custom document chunker with LangChain MarkdownTextSplitter
2025-11-18 13:55:34 +01:00
renovate-bot-cbcoutinho[bot] 3b1cd96722 chore(deps): update docker.io/library/nextcloud:32.0.1 docker digest to f6232ea 2025-11-18 11:08:03 +00:00
renovate-bot-cbcoutinho[bot] 4288814ff4 chore(deps): update docker.io/library/mariadb:lts docker digest to 1cac849 2025-11-17 23:11:14 +00:00
Chris Coutinho b72aeca55f test: Add custom notes app 2025-11-17 22:14:01 +01:00
renovate-bot-cbcoutinho[bot] 839cf159b8 chore(deps): update qdrant/qdrant docker tag to v1.16.0 2025-11-17 17:09:02 +00:00
Chris Coutinho 259d33b41d Revert "Feature/notes" 2025-11-16 11:17:59 +01:00
Chris Coutinho 32d8eaaab6 Merge pull request #305 from cbcoutinho/feature/notes
Feature/notes
2025-11-16 11:17:51 +01:00
Chris Coutinho 529daf2b48 ci: temp disable sse in ci 2025-11-16 07:03:18 +01:00
Chris Coutinho b96657c935 ci: Add open-webui to docker-compose 2025-11-16 07:00:20 +01:00
Chris Coutinho fca8ab0cfd Merge remote-tracking branch 'origin/master' into rag-evaluation 2025-11-16 00:32:59 +01:00
Chris Coutinho 7e7d861797 Merge pull request #302 from cbcoutinho/feature/viz
feat: Vector visualization enhancements and search optimizations
2025-11-16 00:32:31 +01:00
Chris Coutinho 4fa2edf4c7 ci: Set default scan interval to 5min 2025-11-16 00:10:12 +01:00
Chris Coutinho e3153822f7 perf: Exclude vector-sync status polling from distributed tracing
Skip tracing for /app/vector-sync/status to reduce noise from HTMX polling.
Metrics collection continues for this endpoint.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-15 05:19:35 +01:00
Chris Coutinho 00e72d24a6 feat: Enable SSE transport for mcp service and update test fixtures
Changes:
- Remove streamable-http transport override from mcp service in docker-compose.yml
- Service now uses CLI default SSE transport on /sse endpoint
- Add create_mcp_client_session_sse() helper for SSE connections
- Update nc_mcp_client fixture to use SSE transport
- Fix unpacking for SSE client (yields 2 values vs 3 for streamable-http)

Testing:
- All 4 smoke tests pass with SSE transport
- 32/34 affected tests pass (2 skipped for vector sync)
- OAuth services remain on streamable-http (unchanged)

Note: SSE transport is being deprecated in favor of streamable-http.
This enables minimal validation testing before deprecation.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-14 19:20:30 +01:00
renovate-bot-cbcoutinho[bot] 86891173b2 chore(deps): update docker.io/library/mariadb:lts docker digest to 6b848cb 2025-11-14 05:07:34 +00:00
renovate-bot-cbcoutinho[bot] 2c37ad165e chore(deps): update quay.io/keycloak/keycloak docker tag to v26.4.5 2025-11-12 17:09:23 +00:00
Chris Coutinho f6656fee06 Merge pull request #288 from cbcoutinho/feat/webhook-testing-validation
feat: webhook-based vector sync with management UI and validation
2025-11-12 00:54:20 +01:00
Chris Coutinho 3430b2409d build: Set default logging to text 2025-11-11 23:19:37 +01:00
Chris Coutinho f4759e424d feat: add webhook management UI and BeforeNodeDeletedEvent support
Added comprehensive webhook management capabilities including:

Webhook Client & API:
- Added WebhooksClient for Nextcloud webhooks API integration
- Create, list, update, and delete webhooks programmatically
- Support for event filters in webhook registration

Webhook Presets:
- Added preset system for common webhook configurations
- notes_sync: BeforeNodeDeletedEvent for Notes file operations
- calendar_sync: Calendar events (create, update, delete)
- deck_sync: Deck card operations
- files_sync: File system changes
- forms_sync: Form submissions (conditional)
- Filter presets by installed apps

Admin UI:
- Added multi-pane app view with tabs (User Info, Vector Sync, Webhooks)
- Webhooks tab for admin users only
- Enable/disable preset webhooks via UI
- View currently registered webhooks
- Uses htmx for dynamic loading and Alpine.js for tab state
- Admin permission checking via OCS API

CLI Improvements:
- Refactored CLI to separate module (cli.py)
- Updated entry point in pyproject.toml

BeforeNodeDeletedEvent Fix:
- Updated ADR-010 to document NodeDeletedEvent issue
- BeforeNodeDeletedEvent includes node.id before deletion
- NodeDeletedEvent lacks node.id (file already deleted)
- Implemented per Nextcloud maintainer recommendation

Testing:
- Added comprehensive webhook client tests
- Added webhook preset filtering tests
- Added admin permission tests

Configuration:
- Updated docker-compose.yml Qdrant settings

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-11 20:35:08 +01:00
renovate-bot-cbcoutinho[bot] b52f482a51 chore(deps): update docker.io/library/mariadb:lts docker digest to 404ebf2 2025-11-10 23:10:04 +00:00
Chris Coutinho a6e5f3d8ff refactor: simplify OpenTelemetry tracing configuration
Simplifies the OpenTelemetry tracing setup by removing the redundant
OTEL_ENABLED flag and using the presence of OTEL_EXPORTER_OTLP_ENDPOINT
to determine if tracing should be enabled. This follows the standard
OpenTelemetry environment variable conventions more closely.

Changes:
- Remove OTEL_ENABLED/tracing_enabled flag in favor of checking if
  OTEL_EXPORTER_OTLP_ENDPOINT is set
- Add OTEL_EXPORTER_VERIFY_SSL configuration option for OTLP endpoints
  with self-signed certificates (defaults to false for development)
- Move HTTPXClientInstrumentor initialization to module level to ensure
  httpx calls are traced across all Nextcloud API requests
- Add tracing spans to vector sync operations (scan_user_documents)
- Fix authorization header logging to only warn about missing headers
  in OAuth mode (BasicAuth mode doesn't use Authorization headers)
- Update observability documentation to reflect simplified configuration
- Refactor Dockerfile to use --no-editable flag for uv sync

Breaking changes:
- OTEL_ENABLED environment variable is removed
- Tracing is now automatically enabled when OTEL_EXPORTER_OTLP_ENDPOINT
  is set

Migration guide:
- Remove OTEL_ENABLED=true from environment configuration
- Tracing will be enabled automatically if OTEL_EXPORTER_OTLP_ENDPOINT
  is configured

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-10 22:48:37 +01:00
renovate-bot-cbcoutinho[bot] bf4eed6007 chore(deps): pin qdrant/qdrant docker tag to 0fb8897 2025-11-10 05:12:36 +00:00
Chris Coutinho 157e433d65 fix: Support in-memory Qdrant for CI testing
Changes to make tests work without external qdrant/ollama dependencies:

1. docker-compose.yml (mcp service):
   - Switch from QDRANT_URL (network mode) to QDRANT_LOCATION=":memory:"
   - Comment out QDRANT_URL and QDRANT_API_KEY (not needed for in-memory)
   - Keep OLLAMA_BASE_URL commented out (use SimpleEmbeddingProvider fallback)

2. nextcloud_mcp_server/vector/qdrant_client.py:
   - Fix collection creation bug in in-memory mode
   - Previously: All ValueError exceptions were re-raised
   - Now: Only dimension mismatch ValueError is re-raised
   - Allows "Collection not found" ValueError to trigger auto-creation

3. tests/integration/test_sampling.py:
   - Update test to handle all sampling unsupported cases
   - Check for multiple fallback search_method values
   - Skip test gracefully when sampling unavailable

This configuration enables:
- CI testing without external services (qdrant, ollama)
- In-memory vector database (ephemeral but sufficient for tests)
- SimpleEmbeddingProvider for embeddings (feature hashing, 384 dims)
- Automatic collection creation on first use

Test result: test_semantic_search_answer_successful_sampling now passes
(skipped with appropriate message when sampling unsupported)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-10 03:21:27 +01:00
Chris Coutinho cb39b3fca4 feat(vector): Add configurable chunk size and overlap for document embedding
Enable users to tune document chunking parameters to match their embedding
model and content type by adding DOCUMENT_CHUNK_SIZE and DOCUMENT_CHUNK_OVERLAP
environment variables.

- **config.py**: Added `document_chunk_size` (default: 512) and
  `document_chunk_overlap` (default: 50) configuration fields with validation:
  - Ensures overlap < chunk_size
  - Warns if chunk_size < 100 words
  - Prevents negative overlap values

- **processor.py**: Updated DocumentChunker instantiation to use config
  settings instead of hardcoded values (line 174-177)

- **tests/unit/test_config.py**: Added TestChunkConfigValidation class with
  9 tests covering:
  - Default values
  - Valid configurations
  - Validation errors (overlap >= chunk_size, negative overlap)
  - Warning for small chunk sizes
  - Environment variable loading

- **docs/configuration.md**: Added comprehensive "Document Chunking
  Configuration" section with:
  - Chunk size selection guidance (256-384 vs 512 vs 768-1024 words)
  - Overlap recommendations (10-20% of chunk size)
  - Configuration examples for different use cases
  - Added env vars to reference table

- **docs/semantic-search-architecture.md**: Added "Document Chunking Strategy"
  section with:
  - Chunking process explanation
  - Example showing sliding window behavior
  - Search behavior with chunks
  - Tuning recommendations

- **env.sample**: Added complete "Semantic Search & Vector Sync Configuration"
  section with:
  - Vector sync settings
  - Qdrant configuration (3 modes)
  - Ollama embedding service
  - Document chunking configuration

- **docker-compose.yml**: Added commented examples for DOCUMENT_CHUNK_SIZE and
  DOCUMENT_CHUNK_OVERLAP with usage notes

\`\`\`bash
DOCUMENT_CHUNK_SIZE=512

DOCUMENT_CHUNK_OVERLAP=50
\`\`\`

1. \`overlap\` must be less than \`chunk_size\`
2. \`overlap\` cannot be negative
3. Warning issued if \`chunk_size\` < 100 words

**Precise matching** (small notes, specific queries):
\`\`\`bash
DOCUMENT_CHUNK_SIZE=256
DOCUMENT_CHUNK_OVERLAP=25
\`\`\`

**Balanced** (default, general purpose):
\`\`\`bash
DOCUMENT_CHUNK_SIZE=512
DOCUMENT_CHUNK_OVERLAP=50
\`\`\`

**Contextual** (long documents, broader topics):
\`\`\`bash
DOCUMENT_CHUNK_SIZE=1024
DOCUMENT_CHUNK_OVERLAP=100
\`\`\`

 **User control** - Tune chunking to match embedding model capabilities
 **Experimentation** - Test different chunk sizes for optimal results
 **Model alignment** - Match chunk size to embedding context window
 **Backward compatible** - Defaults maintain existing behavior
 **Well validated** - Comprehensive tests prevent misconfiguration

All 22 config validation tests pass (9 new tests for chunking):
- Default values work correctly
- Validation prevents invalid configurations
- Environment variables load properly
- Warning system works as expected

With configurable chunk sizes, users can now experiment with different Ollama
embedding models and tune chunk parameters for optimal semantic search quality.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-10 02:47:57 +01:00