849c67c32a
fix: Complete Keycloak external IdP integration with all tests passing
...
This commit completes the Keycloak external IdP integration for the MCP
server, implementing ADR-002 Tier 2 (External Identity Provider) with
full Bearer token authentication support.
Key Changes:
1. **Keycloak backchannel-dynamic configuration**
- Added --hostname-strict=false and --hostname-backchannel-dynamic=true
- Allows external issuer (localhost:8888) with internal endpoints (keycloak:8080)
- Solves Docker networking issue where containers can't reach localhost
2. **CORSMiddleware Bearer token patch**
- Created app-hooks/patches/cors-bearer-token.patch from upstream commit 8fb5e77db82
- Allows Bearer tokens to bypass CORS/CSRF checks (stateless authentication)
- Applied via post-installation hook 20-apply-cors-bearer-token-patch.sh
- Enables app-specific APIs (Notes, Calendar, etc.) to work with Bearer tokens
3. **Patch organization**
- Moved patches to app-hooks/patches/ directory
- Updated docker-compose.yml to mount entire app-hooks directory
- Consolidated patch management for better maintainability
4. **Test improvements**
- All 11 Keycloak integration tests passing
- Tests validate OAuth token acquisition, MCP connectivity, token validation,
tool execution, token persistence, user provisioning, scope filtering,
and error handling
Architecture:
- Keycloak acts as external OAuth/OIDC identity provider
- MCP server uses Keycloak tokens to access Nextcloud APIs
- Nextcloud user_oidc app validates Bearer tokens from Keycloak
- No admin credentials needed - all API access uses user's OAuth tokens
Cache Note:
- Discovery and JWKS caches must be cleared when switching Keycloak configurations
- Use: docker compose exec redis redis-cli DEL "<cache-key>"
- Or: docker compose exec app php occ user_oidc:provider keycloak --clientid nextcloud
Related:
- ADR-002: Vector sync background jobs authentication
- Validates external IdP integration pattern
- Demonstrates offline_access with refresh tokens (Tier 1 & 2)
🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-Authored-By: Claude <noreply@anthropic.com >
2025-11-02 22:03:20 +01:00
6117aaaed3
fix: Complete Keycloak external IdP integration with all tests passing
...
This commit completes the Keycloak external identity provider integration,
implementing the ADR-002 architecture where Keycloak acts as an external
OAuth/OIDC provider and Nextcloud validates tokens via the user_oidc app.
Architecture:
MCP Client → Keycloak (OAuth) → MCP Server → Nextcloud user_oidc → APIs
Key Fixes:
1. Keycloak JWT token configuration
- Added 'sub' claim protocol mapper to realm-export.json
- Updated token_verifier.py to accept both 'sub' and 'preferred_username'
- Ensures tokens contain required OIDC claims
2. Keycloak hostname configuration for Docker networking
- Implemented --hostname-backchannel-dynamic=true in docker-compose.yml
- External clients use localhost:8888 (public)
- Internal services use keycloak:8080 (Docker network)
- Same issuer (localhost:8888) everywhere for token consistency
- Restored frontendUrl in realm attributes
3. MCP server provider mode detection
- Fixed URL normalization to handle port differences (http://app vs http://app:80 )
- Correctly distinguishes integrated mode vs external IdP mode
- Removes explicit default ports (80 for HTTP, 443 for HTTPS)
4. Nextcloud SSRF protection configuration
- Added allow_local_remote_servers=true to user_oidc install script
- Enables Nextcloud to fetch JWKS from internal Keycloak container
- Required for external IdP token validation
5. OAuth lifespan cleanup
- Fixed RefreshTokenStorage close() error (uses context managers)
- Added safe cleanup for oauth_client with hasattr check
- Prevents session crash on shutdown
6. Test suite fixes
- Fixed test_user_auto_provisioning to reflect actual behavior
- Fixed test_scope_filtering_with_keycloak tool name (nc_webdav_write_file)
- Updated test_keycloak_oauth_client_credentials_discovery for hostname config
- All 11 Keycloak external IdP tests now passing
Testing:
✅ All 11 tests in test_keycloak_external_idp.py passing
✅ OAuth token acquisition via Playwright automation
✅ Token validation through Nextcloud user_oidc app
✅ Write operations (Notes create, Calendar create, File upload)
✅ Read operations (search, list, get)
✅ Token persistence across multiple operations
✅ User authentication and bearer token validation
✅ Scope-based tool filtering
✅ Error handling for invalid operations
Implementation validates:
- ADR-002 external identity provider architecture
- No admin credentials needed in MCP server
- Centralized identity management via Keycloak
- Standards-based OAuth 2.0 / OIDC integration
- User auto-provisioning from IdP claims
🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-Authored-By: Claude <noreply@anthropic.com >
2025-11-02 22:03:20 +01:00
403f8be429
feat: Add Keycloak external IdP integration with custom scopes
...
Add comprehensive support for using Keycloak as an external identity
provider with Nextcloud custom scopes. This enables testing of ADR-002
external IdP integration patterns.
**Keycloak Realm Configuration:**
- Add frontendUrl attribute to issue tokens with public issuer URL
- Define 18 Nextcloud custom client scopes (notes:read/write,
calendar:read/write, contacts:read/write, cookbook:read/write,
deck:read/write, tables:read/write, files:read/write,
sharing:read/write, todo:read/write)
- Add all custom scopes to nextcloud-mcp-server client optional scopes
- Scopes include consent screen text for user-friendly OAuth flow
**MCP Server Configuration:**
- Add OIDC_JWKS_URI environment variable support
- Implement JWKS URI override logic for Docker networking
- Update NEXTCLOUD_PUBLIC_ISSUER_URL to include full realm path
- Enable MCP server to fetch JWKS from internal Docker network
**Test Infrastructure:**
- Add keycloak_oauth_client_credentials fixture (session-scoped)
- Add keycloak_oauth_token fixture with Playwright automation
- Implement PKCE (S256) support for Keycloak OAuth flow
- Add nc_mcp_keycloak_client fixture for MCP testing
- Create comprehensive test suite in test_keycloak_external_idp.py
**Tests Created:**
- test_keycloak_oauth_token_acquisition: Token acquisition via Playwright
- test_keycloak_oauth_client_credentials_discovery: OIDC discovery
- test_mcp_client_connects_to_keycloak_server: MCP connectivity
- test_external_idp_server_initialization: Server auto-detection
- test_external_idp_token_validation: Token validation flow
- test_tools_work_with_keycloak_token: End-to-end tool execution
- test_keycloak_token_persistence: Multi-operation token reuse
- test_user_auto_provisioning: Nextcloud user provisioning
- test_scope_filtering_with_keycloak: Scope-based tool filtering
- test_keycloak_error_handling: Error handling
- test_external_idp_architecture: Architecture documentation
**Current Status:**
- ✅ Keycloak realm configuration complete
- ✅ Custom scopes defined and available
- ✅ OAuth token acquisition working (1 test passing)
- ⚠️ Token validation needs additional work (external IdP userinfo)
**Files Modified:**
- keycloak/realm-export.json: Realm configuration with scopes
- tests/conftest.py: Keycloak OAuth fixtures (+285 lines)
- tests/server/oauth/test_keycloak_external_idp.py: New test suite
- docker-compose.yml: OIDC_JWKS_URI and issuer configuration
- nextcloud_mcp_server/app.py: JWKS URI override logic
🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-Authored-By: Claude <noreply@anthropic.com >
2025-11-02 22:03:20 +01:00
2a1274d8a8
refactor: Unify OAuth configuration to be provider-agnostic
...
Replace provider-specific environment variables (OAUTH_PROVIDER, KEYCLOAK_*)
with generic OIDC_* variables that work with any OIDC-compliant provider.
**Key Changes:**
- Auto-detect provider mode from OIDC_DISCOVERY_URL issuer
- External IdP mode: issuer ≠ NEXTCLOUD_HOST (Keycloak, Auth0, Okta, etc.)
- Integrated mode: issuer = NEXTCLOUD_HOST (Nextcloud OIDC app)
- Unified OIDC discovery flow (single code path)
- Generic client credential loading (static or DCR)
- Simplified docker-compose.yml environment variables
**Environment Variables:**
BEFORE:
OAUTH_PROVIDER=keycloak
KEYCLOAK_URL=http://keycloak:8080
KEYCLOAK_REALM=nextcloud-mcp
KEYCLOAK_CLIENT_ID=...
KEYCLOAK_DISCOVERY_URL=...
AFTER:
OIDC_DISCOVERY_URL=http://keycloak:8080/realms/nextcloud-mcp/.well-known/ ...
OIDC_CLIENT_ID=nextcloud-mcp-server
OIDC_CLIENT_SECRET=...
**Benefits:**
- Works with any OIDC provider without code changes
- No manual provider selection needed
- Cleaner environment variable naming
- Reduced code duplication (~150 lines removed)
**Testing:**
✅ mcp-keycloak auto-detects external IdP mode
✅ Token exchange test passes with generic config
✅ Backward compatible - integrated mode still works
🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-Authored-By: Claude <noreply@anthropic.com >
2025-11-02 22:03:20 +01:00
e331544cee
feat: Implement RFC 8693 token exchange for Keycloak (ADR-002 Tier 2)
...
Implements OAuth 2.0 Token Exchange (RFC 8693) enabling the MCP server to
exchange service account tokens for user-scoped tokens. This provides an
alternative to refresh tokens for background operations.
**Core Implementation:**
- Added `get_service_account_token()` method to KeycloakOAuthClient for
client_credentials grant
- Added `exchange_token_for_user()` method implementing RFC 8693 token exchange
- Fixed Fernet encryption key handling in RefreshTokenStorage (was incorrectly
base64 decoding already-encoded keys)
- Updated OAuth configuration to support offline_access scope and refresh token
storage infrastructure
**Keycloak Configuration:**
- Enabled `serviceAccountsEnabled` in realm-export.json
- Added `token.exchange.grant.enabled` attribute
- Added `client.token.exchange.standard.enabled` attribute (required for
Keycloak 26.2+ Standard Token Exchange V2)
- Fresh Keycloak imports now correctly enable token exchange
**Docker Compose:**
- Added TOKEN_ENCRYPTION_KEY and ENABLE_OFFLINE_ACCESS environment variables
- Created oauth-tokens volume for refresh token storage
- Configured both mcp-oauth and mcp-keycloak services
**Testing & Documentation:**
- Added tests/manual/test_token_exchange.py - Validates complete RFC 8693 flow
- Added tests/manual/test_nextcloud_impersonate.py - Documents session-based
impersonation limitations
- Added docs/oauth-impersonation-findings.md - Comprehensive investigation
findings and resolution documentation
**Verified Working:**
✅ Service account token acquisition (client_credentials grant)
✅ RFC 8693 token exchange for internal-to-internal tokens
✅ Exchanged tokens validate with Nextcloud APIs
✅ Keycloak 26.4.2 Standard Token Exchange V2 support
**Known Limitations:**
- User impersonation (requested_subject) requires Keycloak Legacy V1 with
preview features
- Cross-client token exchange limited to same realm
- Refresh token storage infrastructure ready but unused (MCP protocol limitation)
Dependencies: aiosqlite>=0.20.0
🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-Authored-By: Claude <noreply@anthropic.com >
2025-11-02 22:03:19 +01:00
f34366a260
feat: Add Keycloak OAuth provider support with refresh token storage
...
Implements Keycloak as an external OIDC provider following ADR-002
architecture for background job authentication using offline_access.
## Features
- Keycloak OAuth provider with PKCE and offline_access support
- Refresh token storage with Fernet encryption
- Token verifier for both JWT and opaque tokens
- Multi-client validation (realm-level trust)
- Sample configuration for Keycloak integration
## Implementation
### OAuth Provider (keycloak_oauth.py)
- Authorization Code Flow with PKCE
- Refresh token exchange
- OIDC discovery endpoint support
- Token validation with JWKS
### Token Storage (refresh_token_storage.py)
- Encrypted storage using Fernet symmetric encryption
- SQLite backend for persistence
- Token rotation support
- Per-user token management
### Token Verifier Updates
- Support both JWT (self-encoded) and opaque tokens
- JWKS-based JWT signature verification
- Introspection endpoint fallback for opaque tokens
- Scope extraction from both token types
### Configuration
- .env.keycloak.sample: Example configuration with Keycloak URLs
- docs/keycloak-multi-client-validation.md: Realm-level validation documentation
- app-hooks/post-installation/10-install-user_oidc-app.sh: Updated dependencies
## Architecture Notes
- MCP Server is a protected resource (requires OAuth)
- MCP Client initiates OAuth flow and shares refresh tokens
- Refresh tokens enable background operations without admin credentials
- Supports future token exchange delegation when Keycloak implements it
## References
- ADR-002: Vector Database Background Sync Authentication
- RFC 6749: OAuth 2.0 (offline_access, refresh tokens)
- RFC 7517: JSON Web Key (JWK)
🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-Authored-By: Claude <noreply@anthropic.com >
2025-11-02 22:03:19 +01:00
562c102711
feat(server): Add /live & /health endpoints
2025-10-29 10:29:30 +01:00
415b1c901b
docs: Parse available scopes from registered tools and update docs
2025-10-25 21:16:40 +02:00
a36038422b
feat: Add text processing background worker for telling client about progress
2025-10-25 19:52:45 +02:00
2147fc1696
refactor: Transform document parsing into pluggable processor architecture
...
Refactors PR #190 's hardcoded Unstructured.io integration into a flexible,
extensible plugin system supporting multiple text extraction engines.
- **`DocumentProcessor` ABC**: Abstract interface for all processors
- **`ProcessorRegistry`**: Central registry for discovery and routing
- **`ProcessingResult`**: Standardized output format across processors
- **`UnstructuredProcessor`**: Refactored from `UnstructuredClient`
- **`TesseractProcessor`**: Local OCR for images (lightweight alternative)
- **`CustomHTTPProcessor`**: Generic wrapper for custom HTTP APIs
- New `get_document_processor_config()` returns structured config
- Supports enabling/disabling individual processors
- Per-processor configuration via environment variables
- **Breaking Change**: `ENABLE_UNSTRUCTURED_PARSING` replaced with:
- `ENABLE_DOCUMENT_PROCESSING=true/false` (master switch)
- `ENABLE_UNSTRUCTURED=true/false` (per-processor)
- `ENABLE_TESSERACT=true/false`
- `ENABLE_CUSTOM_PROCESSOR=true/false`
- `parse_document()` now uses `ProcessorRegistry`
- Auto-selects appropriate processor based on MIME type
- Processor priority system (Unstructured=10, Tesseract=5, Custom=1)
- `initialize_document_processors()` registers processors at startup
- Integrated into both BasicAuth and OAuth lifespans
- Graceful degradation if processors fail to initialize
```env
ENABLE_DOCUMENT_PROCESSING=false
ENABLE_UNSTRUCTURED=false
UNSTRUCTURED_API_URL=http://unstructured:8000
UNSTRUCTURED_STRATEGY=auto # auto|fast|hi_res
UNSTRUCTURED_LANGUAGES=eng,deu
ENABLE_TESSERACT=false
TESSERACT_LANG=eng
ENABLE_CUSTOM_PROCESSOR=false
CUSTOM_PROCESSOR_URL=http://localhost:9000/process
CUSTOM_PROCESSOR_TYPES=application/pdf,image/jpeg
```
- **Removed**: `tests/test_unstructured_config.py` (legacy tests)
- **Added**: `tests/unit/test_document_processor_config.py`
- 7 unit tests for new config system
- Tests individual and multi-processor configurations
- **Added**:
- `nextcloud_mcp_server/document_processors/__init__.py`
- `nextcloud_mcp_server/document_processors/base.py`
- `nextcloud_mcp_server/document_processors/registry.py`
- `nextcloud_mcp_server/document_processors/unstructured.py`
- `nextcloud_mcp_server/document_processors/tesseract.py`
- `nextcloud_mcp_server/document_processors/custom_http.py`
- `tests/unit/test_document_processor_config.py`
- **Modified**:
- `nextcloud_mcp_server/config.py` - New plugin config system
- `nextcloud_mcp_server/app.py` - Processor initialization
- `nextcloud_mcp_server/utils/document_parser.py` - Uses registry
- `nextcloud_mcp_server/server/webdav.py` - Import updates
- `env.sample` - New configuration format
- `docker-compose.yml` - (profile changes from previous work)
- **Removed**:
- `nextcloud_mcp_server/client/unstructured_client.py` - Replaced by UnstructuredProcessor
- `tests/test_unstructured_config.py` - Replaced with new tests
✅ **Extensible**: Add processors without modifying core code
✅ **Testable**: Mock processors for unit tests
✅ **Configurable**: Enable only needed processors
✅ **Flexible**: Choose fast (Tesseract) vs accurate (Unstructured)
✅ **Opt-in**: Disabled by default, no mandatory dependencies
Users upgrading from PR #190 need to update environment variables:
```bash
ENABLE_UNSTRUCTURED_PARSING=true
ENABLE_DOCUMENT_PROCESSING=true
ENABLE_UNSTRUCTURED=true
```
🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-Authored-By: Claude <noreply@anthropic.com >
2025-10-25 19:28:35 +02:00
f0e5333e43
Merge branch 'master' into feature/introduce_files_parsing_with_unstructured_service_for_webdav_files_retrieval
2025-10-25 17:23:38 +02:00
50b69a2531
fix: Add support for RFC 7592 client registration and deletion
2025-10-24 19:19:27 +02:00
8e0a4d8ce5
feat(auth): Add support for client registration deletion
2025-10-24 18:54:24 +02:00
1e877f17f7
test: Replace persistent OAuth client cache with session-scoped fixtures
...
Remove file-based caching of OAuth client credentials and implement automatic
client lifecycle management for test fixtures.
Changes:
- Add RFC 7592 client deletion function in auth/client_registration.py
- Remove cache_file parameter from _create_oauth_client_with_scopes helper
- Update all OAuth credential fixtures to use yield/finalizer pattern
- Add automatic client cleanup at end of test session (best-effort)
- Remove persistent .nextcloud_oauth_*.json cache files
Benefits:
- No persistent cache files cluttering repository
- Fresh OAuth clients created for each test session via DCR
- Automatic cleanup attempts (RFC 7592 DELETE endpoint)
- Cleaner test environment with proper fixture lifecycle
Note: Client deletion may fail due to Nextcloud authentication middleware
(logged as warning). The key improvement is removing persistent cache files.
OAuth clients may accumulate in Nextcloud but can be cleaned manually.
2025-10-24 08:11:22 +02:00
13f76a7734
chore: Upgrade pydantic Config to ConfigDict
2025-10-24 06:18:13 +02:00
81ca799410
fix: Update webdav models for proper serialization
2025-10-24 06:01:02 +02:00
d452684535
feat: Split read/write scopes into app:read/write scopes
2025-10-24 04:38:49 +02:00
29df645d53
Merge branch 'master' into feature/introduce_files_parsing_with_unstructured_service_for_webdav_files_retrieval
2025-10-23 21:30:09 +02:00
bdb0e17401
chore: Add logging to token introspection
2025-10-23 21:18:14 +02:00
f2d2dd8068
feat: Enable token introspection for opaque tokens
2025-10-23 15:51:27 +02:00
053cf7798b
fix: Add CORS middleware to allow browser-based clients like MCP Inspector
2025-10-23 15:23:41 +02:00
737780b417
chore: Make all env vars available to be overriden as cli options
2025-10-23 11:48:01 +02:00
e9a16c43b5
refactor: Update JWT client to use DCR, re-enable tool filtering
2025-10-23 09:33:06 +02:00
e48f5f3f30
feat(server): Add support for custom OIDC scopes and permissions via JWTs
2025-10-23 08:37:36 +02:00
c069d78f80
feat: Initialize JWT-scoped tools
2025-10-22 06:21:16 +02:00
64649c902d
Merge branch 'master' into feature/introduce_files_parsing_with_unstructured_service_for_webdav_files_retrieval
2025-10-21 20:37:00 +02:00
63b898c0e3
chore: Update logs
2025-10-20 22:57:18 +02:00
e8f1340133
fix(caldav): Fix caldav search() due to missing todos
2025-10-20 22:18:46 +02:00
71f09a47ca
docs: Update CalendarClient docstrings [skip ci]
2025-10-20 00:54:35 +02:00
f4dd68735c
test: Fix how categories are handled in calendar
2025-10-20 00:04:38 +02:00
a143123acc
fix(caldav): Check that calendar exists after creation to avoid race condition
...
Verify that field preservation tests still operate
2025-10-19 23:44:39 +02:00
1dc2ddfdb7
fix(caldav): Properly parse datetimes as vDDDTypes
2025-10-19 20:13:05 +02:00
92e18825bc
feat(caldav): Add support for tasks
2025-10-19 18:02:43 +02:00
c2f6c6ce0d
ci: Set cookbook recipe import timeout to 5min
2025-10-19 01:49:21 +02:00
31ffeba69b
chore: Move timeout to recipe import
2025-10-18 23:12:31 +02:00
6158a890af
feat(webdav): Add search and list favorite response tools
2025-10-18 22:02:26 +02:00
37164dbdbc
chore: sort imports
2025-10-18 22:02:25 +02:00
83917b3786
perf(notes): Improve notes search performance using async iterators
2025-10-18 22:02:19 +02:00
5e829fc7e7
refactor: Unify logging & remove factory deployment
2025-10-18 01:15:06 +02:00
8e7191e0ea
fix: Increase HTTP client timeout to 30s
...
The default 5s timeout was too short for Nextcloud Cookbook app to fetch and process recipes from external URLs, causing intermittent test failures with ReadTimeout errors.
Fixes intermittent CI failures in cookbook import tests.
2025-10-17 04:41:28 +02:00
dbcf9d93ca
chore: Improve RequestError message details
...
Show exception type and cause when str(e) is empty for better debugging
2025-10-17 04:37:31 +02:00
2999d4b65e
fix: Handle RequestError in mcp tools
2025-10-17 04:17:41 +02:00
9de59db718
feat(cookbook): Add full Cookbook app support with 13 tools and 2 resources
...
- Import recipes from URLs using schema.org metadata
- Full CRUD operations for recipes
- Search, categorize, and organize recipes
- Manage keywords/tags and categories
- Configure app settings and trigger reindexing
2025-10-17 03:08:16 +02:00
3ad9198f36
fix(oauth): Remove the option to force_register new clients
2025-10-15 16:27:22 +02:00
5db02313a1
test: Update share client to fix test, update passwords
2025-10-15 10:35:22 +02:00
85f8522085
feat: Add Groups API client
2025-10-15 03:43:25 +02:00
a38c795124
feat: add sharing API client and server tools
2025-10-15 02:59:26 +02:00
7004104873
test: Fix multi-user tests
2025-10-15 02:11:17 +02:00
7a4a31b52d
fix: Update user/groups API to OCS v2
2025-10-15 00:05:22 +02:00
898c2e72ae
Merge remote-tracking branch 'origin/master' into feature/user-api
2025-10-14 23:43:03 +02:00
Chris Coutinho