brandon/nextcloud-mcp-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
807 Commits 193 Branches 339 Tags
2ca6725fc6f85ef4ec7737b1dee802be473f14cd
Commit Graph

128 Commits

Author SHA1 Message Date
Chris Coutinho 849c67c32a fix: Complete Keycloak external IdP integration with all tests passing 
This commit completes the Keycloak external IdP integration for the MCP
server, implementing ADR-002 Tier 2 (External Identity Provider) with
full Bearer token authentication support.

Key Changes:
1. **Keycloak backchannel-dynamic configuration**
   - Added --hostname-strict=false and --hostname-backchannel-dynamic=true
   - Allows external issuer (localhost:8888) with internal endpoints (keycloak:8080)
   - Solves Docker networking issue where containers can't reach localhost

2. **CORSMiddleware Bearer token patch**
   - Created app-hooks/patches/cors-bearer-token.patch from upstream commit 8fb5e77db82
   - Allows Bearer tokens to bypass CORS/CSRF checks (stateless authentication)
   - Applied via post-installation hook 20-apply-cors-bearer-token-patch.sh
   - Enables app-specific APIs (Notes, Calendar, etc.) to work with Bearer tokens

3. **Patch organization**
   - Moved patches to app-hooks/patches/ directory
   - Updated docker-compose.yml to mount entire app-hooks directory
   - Consolidated patch management for better maintainability

4. **Test improvements**
   - All 11 Keycloak integration tests passing
   - Tests validate OAuth token acquisition, MCP connectivity, token validation,
     tool execution, token persistence, user provisioning, scope filtering,
     and error handling

Architecture:
- Keycloak acts as external OAuth/OIDC identity provider
- MCP server uses Keycloak tokens to access Nextcloud APIs
- Nextcloud user_oidc app validates Bearer tokens from Keycloak
- No admin credentials needed - all API access uses user's OAuth tokens

Cache Note:
- Discovery and JWKS caches must be cleared when switching Keycloak configurations
- Use: docker compose exec redis redis-cli DEL "<cache-key>"
- Or: docker compose exec app php occ user_oidc:provider keycloak --clientid nextcloud

Related:
- ADR-002: Vector sync background jobs authentication
- Validates external IdP integration pattern
- Demonstrates offline_access with refresh tokens (Tier 1 & 2)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-11-02 22:03:20 +01:00
Chris Coutinho 6117aaaed3 fix: Complete Keycloak external IdP integration with all tests passing 
This commit completes the Keycloak external identity provider integration,
implementing the ADR-002 architecture where Keycloak acts as an external
OAuth/OIDC provider and Nextcloud validates tokens via the user_oidc app.

Architecture:
  MCP Client → Keycloak (OAuth) → MCP Server → Nextcloud user_oidc → APIs

Key Fixes:

1. Keycloak JWT token configuration
   - Added 'sub' claim protocol mapper to realm-export.json
   - Updated token_verifier.py to accept both 'sub' and 'preferred_username'
   - Ensures tokens contain required OIDC claims

2. Keycloak hostname configuration for Docker networking
   - Implemented --hostname-backchannel-dynamic=true in docker-compose.yml
   - External clients use localhost:8888 (public)
   - Internal services use keycloak:8080 (Docker network)
   - Same issuer (localhost:8888) everywhere for token consistency
   - Restored frontendUrl in realm attributes

3. MCP server provider mode detection
   - Fixed URL normalization to handle port differences (http://app vs http://app:80)
   - Correctly distinguishes integrated mode vs external IdP mode
   - Removes explicit default ports (80 for HTTP, 443 for HTTPS)

4. Nextcloud SSRF protection configuration
   - Added allow_local_remote_servers=true to user_oidc install script
   - Enables Nextcloud to fetch JWKS from internal Keycloak container
   - Required for external IdP token validation

5. OAuth lifespan cleanup
   - Fixed RefreshTokenStorage close() error (uses context managers)
   - Added safe cleanup for oauth_client with hasattr check
   - Prevents session crash on shutdown

6. Test suite fixes
   - Fixed test_user_auto_provisioning to reflect actual behavior
   - Fixed test_scope_filtering_with_keycloak tool name (nc_webdav_write_file)
   - Updated test_keycloak_oauth_client_credentials_discovery for hostname config
   - All 11 Keycloak external IdP tests now passing

Testing:
  ✅ All 11 tests in test_keycloak_external_idp.py passing
  ✅ OAuth token acquisition via Playwright automation
  ✅ Token validation through Nextcloud user_oidc app
  ✅ Write operations (Notes create, Calendar create, File upload)
  ✅ Read operations (search, list, get)
  ✅ Token persistence across multiple operations
  ✅ User authentication and bearer token validation
  ✅ Scope-based tool filtering
  ✅ Error handling for invalid operations

Implementation validates:
  - ADR-002 external identity provider architecture
  - No admin credentials needed in MCP server
  - Centralized identity management via Keycloak
  - Standards-based OAuth 2.0 / OIDC integration
  - User auto-provisioning from IdP claims

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-11-02 22:03:20 +01:00
Chris Coutinho 403f8be429 feat: Add Keycloak external IdP integration with custom scopes 
Add comprehensive support for using Keycloak as an external identity
provider with Nextcloud custom scopes. This enables testing of ADR-002
external IdP integration patterns.

**Keycloak Realm Configuration:**
- Add frontendUrl attribute to issue tokens with public issuer URL
- Define 18 Nextcloud custom client scopes (notes:read/write,
  calendar:read/write, contacts:read/write, cookbook:read/write,
  deck:read/write, tables:read/write, files:read/write,
  sharing:read/write, todo:read/write)
- Add all custom scopes to nextcloud-mcp-server client optional scopes
- Scopes include consent screen text for user-friendly OAuth flow

**MCP Server Configuration:**
- Add OIDC_JWKS_URI environment variable support
- Implement JWKS URI override logic for Docker networking
- Update NEXTCLOUD_PUBLIC_ISSUER_URL to include full realm path
- Enable MCP server to fetch JWKS from internal Docker network

**Test Infrastructure:**
- Add keycloak_oauth_client_credentials fixture (session-scoped)
- Add keycloak_oauth_token fixture with Playwright automation
- Implement PKCE (S256) support for Keycloak OAuth flow
- Add nc_mcp_keycloak_client fixture for MCP testing
- Create comprehensive test suite in test_keycloak_external_idp.py

**Tests Created:**
- test_keycloak_oauth_token_acquisition: Token acquisition via Playwright
- test_keycloak_oauth_client_credentials_discovery: OIDC discovery
- test_mcp_client_connects_to_keycloak_server: MCP connectivity
- test_external_idp_server_initialization: Server auto-detection
- test_external_idp_token_validation: Token validation flow
- test_tools_work_with_keycloak_token: End-to-end tool execution
- test_keycloak_token_persistence: Multi-operation token reuse
- test_user_auto_provisioning: Nextcloud user provisioning
- test_scope_filtering_with_keycloak: Scope-based tool filtering
- test_keycloak_error_handling: Error handling
- test_external_idp_architecture: Architecture documentation

**Current Status:**
- ✅ Keycloak realm configuration complete
- ✅ Custom scopes defined and available
- ✅ OAuth token acquisition working (1 test passing)
- ⚠️  Token validation needs additional work (external IdP userinfo)

**Files Modified:**
- keycloak/realm-export.json: Realm configuration with scopes
- tests/conftest.py: Keycloak OAuth fixtures (+285 lines)
- tests/server/oauth/test_keycloak_external_idp.py: New test suite
- docker-compose.yml: OIDC_JWKS_URI and issuer configuration
- nextcloud_mcp_server/app.py: JWKS URI override logic

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-11-02 22:03:20 +01:00
Chris Coutinho 2a1274d8a8 refactor: Unify OAuth configuration to be provider-agnostic 
Replace provider-specific environment variables (OAUTH_PROVIDER, KEYCLOAK_*)
with generic OIDC_* variables that work with any OIDC-compliant provider.

**Key Changes:**
- Auto-detect provider mode from OIDC_DISCOVERY_URL issuer
  - External IdP mode: issuer ≠ NEXTCLOUD_HOST (Keycloak, Auth0, Okta, etc.)
  - Integrated mode: issuer = NEXTCLOUD_HOST (Nextcloud OIDC app)
- Unified OIDC discovery flow (single code path)
- Generic client credential loading (static or DCR)
- Simplified docker-compose.yml environment variables

**Environment Variables:**
BEFORE:
  OAUTH_PROVIDER=keycloak
  KEYCLOAK_URL=http://keycloak:8080
  KEYCLOAK_REALM=nextcloud-mcp
  KEYCLOAK_CLIENT_ID=...
  KEYCLOAK_DISCOVERY_URL=...

AFTER:
  OIDC_DISCOVERY_URL=http://keycloak:8080/realms/nextcloud-mcp/.well-known/...
  OIDC_CLIENT_ID=nextcloud-mcp-server
  OIDC_CLIENT_SECRET=...

**Benefits:**
- Works with any OIDC provider without code changes
- No manual provider selection needed
- Cleaner environment variable naming
- Reduced code duplication (~150 lines removed)

**Testing:**
✅ mcp-keycloak auto-detects external IdP mode
✅ Token exchange test passes with generic config
✅ Backward compatible - integrated mode still works

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-11-02 22:03:20 +01:00
Chris Coutinho e331544cee feat: Implement RFC 8693 token exchange for Keycloak (ADR-002 Tier 2) 
Implements OAuth 2.0 Token Exchange (RFC 8693) enabling the MCP server to
exchange service account tokens for user-scoped tokens. This provides an
alternative to refresh tokens for background operations.

**Core Implementation:**
- Added `get_service_account_token()` method to KeycloakOAuthClient for
  client_credentials grant
- Added `exchange_token_for_user()` method implementing RFC 8693 token exchange
- Fixed Fernet encryption key handling in RefreshTokenStorage (was incorrectly
  base64 decoding already-encoded keys)
- Updated OAuth configuration to support offline_access scope and refresh token
  storage infrastructure

**Keycloak Configuration:**
- Enabled `serviceAccountsEnabled` in realm-export.json
- Added `token.exchange.grant.enabled` attribute
- Added `client.token.exchange.standard.enabled` attribute (required for
  Keycloak 26.2+ Standard Token Exchange V2)
- Fresh Keycloak imports now correctly enable token exchange

**Docker Compose:**
- Added TOKEN_ENCRYPTION_KEY and ENABLE_OFFLINE_ACCESS environment variables
- Created oauth-tokens volume for refresh token storage
- Configured both mcp-oauth and mcp-keycloak services

**Testing & Documentation:**
- Added tests/manual/test_token_exchange.py - Validates complete RFC 8693 flow
- Added tests/manual/test_nextcloud_impersonate.py - Documents session-based
  impersonation limitations
- Added docs/oauth-impersonation-findings.md - Comprehensive investigation
  findings and resolution documentation

**Verified Working:**
✅ Service account token acquisition (client_credentials grant)
✅ RFC 8693 token exchange for internal-to-internal tokens
✅ Exchanged tokens validate with Nextcloud APIs
✅ Keycloak 26.4.2 Standard Token Exchange V2 support

**Known Limitations:**
- User impersonation (requested_subject) requires Keycloak Legacy V1 with
  preview features
- Cross-client token exchange limited to same realm
- Refresh token storage infrastructure ready but unused (MCP protocol limitation)

Dependencies: aiosqlite>=0.20.0

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-11-02 22:03:19 +01:00
Chris Coutinho 529dc4616b docs: Implement separate clients architecture for Keycloak integration 
Implements proper OAuth 2.0 separation following RFC 8707 best practices
with distinct resource server and OAuth client configurations.

## Architecture Changes

- Create separate "nextcloud" bearer-only client (resource server)
- Configure "nextcloud-mcp-server" OAuth client with audience mapper
- Audience mapper targets "nextcloud" resource server
- Token flow: aud="nextcloud", azp="nextcloud-mcp-server"

## Benefits

- Proper OAuth client vs resource server separation
- Support for future multi-resource tokens: aud=["nextcloud", "other-service"]
- RFC 8707 Resource Indicators compliance
- Clear requester identification via azp claim

## Documentation Updates

- Correct OAuth flow: MCP Client initiates, handles redirect, shares tokens
- Explain MCP Server as protected resource architecture
- Document offline_access with refresh tokens (Tier 1, current)
- Document token exchange with delegation (Tier 2, future when Keycloak adds support)
- Reference Keycloak issue #38279 for delegation status

## Files

- keycloak/realm-export.json: Add separate clients configuration
- app-hooks/post-installation/15-setup-keycloak-provider.sh: Setup user_oidc with "nextcloud" client
- docs/audience-validation-setup.md: Comprehensive documentation with corrected OAuth flow and delegation comparison
- docker-compose.yml: Fix Keycloak healthcheck (bash TCP instead of curl)
- scripts/test_separate_clients.sh: Verification script for architecture

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-11-02 22:03:19 +01:00
Chris Coutinho f739330341 ci: fix typo 2025-11-02 22:03:19 +01:00
Chris Coutinho 136df2422b build: Add keykloak to docker-compose.yml 2025-11-02 22:03:19 +01:00
renovate-bot-cbcoutinho[bot] 72e4eb3d19 chore(deps): update docker.io/library/nginx:alpine docker digest to b3c656d 2025-10-30 17:06:28 +00:00
renovate-bot-cbcoutinho[bot] dd636e6a08 chore(deps): update docker.io/library/nginx:alpine docker digest to 9dacca6 2025-10-29 05:07:08 +00:00
renovate-bot-cbcoutinho[bot] 4fc0cb5a41 chore(deps): update docker.io/library/nextcloud:32.0.1 docker digest to 1e4eae5 2025-10-27 23:10:34 +00:00
renovate-bot-cbcoutinho[bot] 7db2a5c586 chore(deps): pin downloads.unstructured.io/unstructured-io/unstructured-api docker tag to a43ab55 2025-10-25 22:05:59 +00:00
Chris Coutinho a36038422b feat: Add text processing background worker for telling client about progress 2025-10-25 19:52:45 +02:00
Chris Coutinho 2147fc1696 refactor: Transform document parsing into pluggable processor architecture 
Refactors PR #190's hardcoded Unstructured.io integration into a flexible,
extensible plugin system supporting multiple text extraction engines.

- **`DocumentProcessor` ABC**: Abstract interface for all processors
- **`ProcessorRegistry`**: Central registry for discovery and routing
- **`ProcessingResult`**: Standardized output format across processors

- **`UnstructuredProcessor`**: Refactored from `UnstructuredClient`
- **`TesseractProcessor`**: Local OCR for images (lightweight alternative)
- **`CustomHTTPProcessor`**: Generic wrapper for custom HTTP APIs

- New `get_document_processor_config()` returns structured config
- Supports enabling/disabling individual processors
- Per-processor configuration via environment variables
- **Breaking Change**: `ENABLE_UNSTRUCTURED_PARSING` replaced with:
  - `ENABLE_DOCUMENT_PROCESSING=true/false` (master switch)
  - `ENABLE_UNSTRUCTURED=true/false` (per-processor)
  - `ENABLE_TESSERACT=true/false`
  - `ENABLE_CUSTOM_PROCESSOR=true/false`

- `parse_document()` now uses `ProcessorRegistry`
- Auto-selects appropriate processor based on MIME type
- Processor priority system (Unstructured=10, Tesseract=5, Custom=1)

- `initialize_document_processors()` registers processors at startup
- Integrated into both BasicAuth and OAuth lifespans
- Graceful degradation if processors fail to initialize

```env
ENABLE_DOCUMENT_PROCESSING=false

ENABLE_UNSTRUCTURED=false
UNSTRUCTURED_API_URL=http://unstructured:8000
UNSTRUCTURED_STRATEGY=auto  # auto|fast|hi_res
UNSTRUCTURED_LANGUAGES=eng,deu

ENABLE_TESSERACT=false
TESSERACT_LANG=eng

ENABLE_CUSTOM_PROCESSOR=false
CUSTOM_PROCESSOR_URL=http://localhost:9000/process
CUSTOM_PROCESSOR_TYPES=application/pdf,image/jpeg
```

- **Removed**: `tests/test_unstructured_config.py` (legacy tests)
- **Added**: `tests/unit/test_document_processor_config.py`
  - 7 unit tests for new config system
  - Tests individual and multi-processor configurations

- **Added**:
  - `nextcloud_mcp_server/document_processors/__init__.py`
  - `nextcloud_mcp_server/document_processors/base.py`
  - `nextcloud_mcp_server/document_processors/registry.py`
  - `nextcloud_mcp_server/document_processors/unstructured.py`
  - `nextcloud_mcp_server/document_processors/tesseract.py`
  - `nextcloud_mcp_server/document_processors/custom_http.py`
  - `tests/unit/test_document_processor_config.py`

- **Modified**:
  - `nextcloud_mcp_server/config.py` - New plugin config system
  - `nextcloud_mcp_server/app.py` - Processor initialization
  - `nextcloud_mcp_server/utils/document_parser.py` - Uses registry
  - `nextcloud_mcp_server/server/webdav.py` - Import updates
  - `env.sample` - New configuration format
  - `docker-compose.yml` - (profile changes from previous work)

- **Removed**:
  - `nextcloud_mcp_server/client/unstructured_client.py` - Replaced by UnstructuredProcessor
  - `tests/test_unstructured_config.py` - Replaced with new tests

✅ **Extensible**: Add processors without modifying core code
✅ **Testable**: Mock processors for unit tests
✅ **Configurable**: Enable only needed processors
✅ **Flexible**: Choose fast (Tesseract) vs accurate (Unstructured)
✅ **Opt-in**: Disabled by default, no mandatory dependencies

Users upgrading from PR #190 need to update environment variables:
```bash
ENABLE_UNSTRUCTURED_PARSING=true

ENABLE_DOCUMENT_PROCESSING=true
ENABLE_UNSTRUCTURED=true
```

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-10-25 19:28:35 +02:00
yuisheaven f0e5333e43 Merge branch 'master' into feature/introduce_files_parsing_with_unstructured_service_for_webdav_files_retrieval 2025-10-25 17:23:38 +02:00
renovate-bot-cbcoutinho[bot] ff20031601 chore(deps): update docker.io/library/nextcloud docker tag to v32.0.1 2025-10-25 10:06:16 +00:00
Chris Coutinho d452684535 feat: Split read/write scopes into app:read/write scopes 2025-10-24 04:38:49 +02:00
yuisheaven 29df645d53 Merge branch 'master' into feature/introduce_files_parsing_with_unstructured_service_for_webdav_files_retrieval 2025-10-23 21:30:09 +02:00
Chris Coutinho 38e12db46a Merge pull request #233 from cbcoutinho/feature/jwt-scopes 
feat: Initialize JWT-scoped tools
 2025-10-23 12:23:12 +02:00
Chris Coutinho 54e975198f test: Update all test network hosts to respect iss claims from JWTs 2025-10-23 11:09:51 +02:00
Chris Coutinho e9a16c43b5 refactor: Update JWT client to use DCR, re-enable tool filtering 2025-10-23 09:33:06 +02:00
Chris Coutinho e48f5f3f30 feat(server): Add support for custom OIDC scopes and permissions via JWTs 2025-10-23 08:37:36 +02:00
Chris Coutinho c069d78f80 feat: Initialize JWT-scoped tools 2025-10-22 06:21:16 +02:00
renovate-bot-cbcoutinho[bot] e3436fecc0 chore(deps): update docker.io/library/nextcloud:32.0.0 docker digest to f9bec5c 2025-10-22 04:06:24 +00:00
yuisheaven 98627593d5 corrected smaller merge issues 2025-10-21 20:55:33 +02:00
yuisheaven 64649c902d Merge branch 'master' into feature/introduce_files_parsing_with_unstructured_service_for_webdav_files_retrieval 2025-10-21 20:37:00 +02:00
renovate-bot-cbcoutinho[bot] 08ebab9f48 chore(deps): update docker.io/library/nextcloud:32.0.0 docker digest to d3d8b9d 2025-10-21 16:06:08 +00:00
renovate-bot-cbcoutinho[bot] 27bb0a4b56 chore(deps): update docker.io/library/nextcloud:32.0.0 docker digest to 4fbd72f 2025-10-21 10:06:57 +00:00
Chris Coutinho 92e18825bc feat(caldav): Add support for tasks 2025-10-19 18:02:43 +02:00
Chris Coutinho 2f805e54b7 test: Migrate load test benchmark scripts to anyio 
Remove unused redis container
 2025-10-18 22:40:50 +02:00
renovate-bot-cbcoutinho[bot] 9b29eabfaa chore(deps): pin docker.io/library/nginx docker tag to 61e0128 2025-10-17 04:07:05 +00:00
Chris Coutinho 27519d0f62 test: Replace http server for recipes with nginx container 2025-10-17 04:30:03 +02:00
Chris Coutinho 0fd32ecd34 test: Fix test networking 2025-10-17 03:58:36 +02:00
Chris Coutinho 7004104873 test: Fix multi-user tests 2025-10-15 02:11:17 +02:00
Chris Coutinho 1023a7d9c7 chore: Remove comments 2025-10-14 01:23:49 +02:00
Chris Coutinho 057e25b653 chore: Add support for overriding public issuer URL 
test: Add patch for PKCE support
 2025-10-14 01:23:41 +02:00
Chris Coutinho f48d3714d2 test: Add restart to mcp containers in docker-compose.yml 2025-10-14 01:23:39 +02:00
Chris Coutinho 4d7e4b9a4b feat(server): Experimental support for OAuth2/OIDC authentication 2025-10-14 01:22:15 +02:00
renovate-bot-cbcoutinho[bot] f16af39b97 chore(deps): update docker.io/library/redis:alpine docker digest to 59b6e69 2025-10-10 22:09:45 +00:00
renovate-bot-cbcoutinho[bot] 391f418934 chore(deps): update docker.io/library/mariadb:lts docker digest to ae61197 2025-10-10 04:06:36 +00:00
renovate-bot-cbcoutinho[bot] 900d1bb462 chore(deps): update docker.io/library/redis:alpine docker digest to b4ab73c 2025-10-09 10:13:48 +00:00
renovate-bot-cbcoutinho[bot] 1402da0ac0 chore(deps): update docker.io/library/redis:alpine docker digest to 0ea5184 2025-10-09 04:06:45 +00:00
renovate-bot-cbcoutinho[bot] 5f3ff60531 chore(deps): update docker.io/library/nextcloud:32.0.0 docker digest to 3e70e4d 2025-10-04 16:07:30 +00:00
yuisheaven c9a687171a added envs for unstructured to control OCR quality and OCR languages 2025-10-04 05:21:02 +02:00
yuisheaven 642108ee91 added new "unstructured" docker service to compose stack and introduced new envs 2025-10-04 04:27:31 +02:00
renovate-bot-cbcoutinho[bot] 7463234ccb chore(deps): replace redis docker tag with docker.io/library/redis alpine 2025-10-03 22:07:16 +00:00
renovate-bot-cbcoutinho[bot] 0faa32fd10 chore(deps): replace mariadb docker tag with docker.io/library/mariadb lts 2025-10-02 10:04:52 +00:00
Chris Coutinho 8a9fa2a3c4 Merge pull request #180 from cbcoutinho/renovate/docker.io-library-nextcloud-32.0.0 
chore(deps): update docker.io/library/nextcloud:32.0.0 docker digest to f4d0a4a
 2025-10-02 08:08:04 +02:00
renovate-bot-cbcoutinho[bot] 9be03ef0de chore(deps): update mariadb:lts docker digest to 24264e9 2025-10-02 04:04:12 +00:00
renovate-bot-cbcoutinho[bot] eda6753253 chore(deps): update docker.io/library/nextcloud:32.0.0 docker digest to f4d0a4a 2025-10-01 22:06:55 +00:00