4c7d1cfc8d
test: Add scope-based authorization tests for Keycloak external IdP
...
This enhances the Keycloak integration test suite with comprehensive
scope-based authorization validation, matching the OIDC test structure.
Changes:
- Add 3 test users to Keycloak realm (read-only, write-only, no-custom-scopes)
- Create OAuth token fixtures with different scope combinations
- Create MCP client fixtures for each scope configuration
- Add 4 new tests validating scope-based tool filtering:
* Read-only tokens filter out write tools
* Write-only tokens filter out read tools
* Full access tokens show all 90+ tools
* No custom scopes result in zero tools
Test Results:
- All 15 Keycloak integration tests pass (11 existing + 4 new)
- Validates proper JWT scope enforcement in external IdP architecture
- Confirms security isolation when users decline custom scopes
This completes ADR-002 scope authorization testing for the Keycloak
external identity provider integration.
🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-Authored-By: Claude <noreply@anthropic.com >
2025-11-02 22:03:21 +01:00
849c67c32a
fix: Complete Keycloak external IdP integration with all tests passing
...
This commit completes the Keycloak external IdP integration for the MCP
server, implementing ADR-002 Tier 2 (External Identity Provider) with
full Bearer token authentication support.
Key Changes:
1. **Keycloak backchannel-dynamic configuration**
- Added --hostname-strict=false and --hostname-backchannel-dynamic=true
- Allows external issuer (localhost:8888) with internal endpoints (keycloak:8080)
- Solves Docker networking issue where containers can't reach localhost
2. **CORSMiddleware Bearer token patch**
- Created app-hooks/patches/cors-bearer-token.patch from upstream commit 8fb5e77db82
- Allows Bearer tokens to bypass CORS/CSRF checks (stateless authentication)
- Applied via post-installation hook 20-apply-cors-bearer-token-patch.sh
- Enables app-specific APIs (Notes, Calendar, etc.) to work with Bearer tokens
3. **Patch organization**
- Moved patches to app-hooks/patches/ directory
- Updated docker-compose.yml to mount entire app-hooks directory
- Consolidated patch management for better maintainability
4. **Test improvements**
- All 11 Keycloak integration tests passing
- Tests validate OAuth token acquisition, MCP connectivity, token validation,
tool execution, token persistence, user provisioning, scope filtering,
and error handling
Architecture:
- Keycloak acts as external OAuth/OIDC identity provider
- MCP server uses Keycloak tokens to access Nextcloud APIs
- Nextcloud user_oidc app validates Bearer tokens from Keycloak
- No admin credentials needed - all API access uses user's OAuth tokens
Cache Note:
- Discovery and JWKS caches must be cleared when switching Keycloak configurations
- Use: docker compose exec redis redis-cli DEL "<cache-key>"
- Or: docker compose exec app php occ user_oidc:provider keycloak --clientid nextcloud
Related:
- ADR-002: Vector sync background jobs authentication
- Validates external IdP integration pattern
- Demonstrates offline_access with refresh tokens (Tier 1 & 2)
🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-Authored-By: Claude <noreply@anthropic.com >
2025-11-02 22:03:20 +01:00
6117aaaed3
fix: Complete Keycloak external IdP integration with all tests passing
...
This commit completes the Keycloak external identity provider integration,
implementing the ADR-002 architecture where Keycloak acts as an external
OAuth/OIDC provider and Nextcloud validates tokens via the user_oidc app.
Architecture:
MCP Client → Keycloak (OAuth) → MCP Server → Nextcloud user_oidc → APIs
Key Fixes:
1. Keycloak JWT token configuration
- Added 'sub' claim protocol mapper to realm-export.json
- Updated token_verifier.py to accept both 'sub' and 'preferred_username'
- Ensures tokens contain required OIDC claims
2. Keycloak hostname configuration for Docker networking
- Implemented --hostname-backchannel-dynamic=true in docker-compose.yml
- External clients use localhost:8888 (public)
- Internal services use keycloak:8080 (Docker network)
- Same issuer (localhost:8888) everywhere for token consistency
- Restored frontendUrl in realm attributes
3. MCP server provider mode detection
- Fixed URL normalization to handle port differences (http://app vs http://app:80 )
- Correctly distinguishes integrated mode vs external IdP mode
- Removes explicit default ports (80 for HTTP, 443 for HTTPS)
4. Nextcloud SSRF protection configuration
- Added allow_local_remote_servers=true to user_oidc install script
- Enables Nextcloud to fetch JWKS from internal Keycloak container
- Required for external IdP token validation
5. OAuth lifespan cleanup
- Fixed RefreshTokenStorage close() error (uses context managers)
- Added safe cleanup for oauth_client with hasattr check
- Prevents session crash on shutdown
6. Test suite fixes
- Fixed test_user_auto_provisioning to reflect actual behavior
- Fixed test_scope_filtering_with_keycloak tool name (nc_webdav_write_file)
- Updated test_keycloak_oauth_client_credentials_discovery for hostname config
- All 11 Keycloak external IdP tests now passing
Testing:
✅ All 11 tests in test_keycloak_external_idp.py passing
✅ OAuth token acquisition via Playwright automation
✅ Token validation through Nextcloud user_oidc app
✅ Write operations (Notes create, Calendar create, File upload)
✅ Read operations (search, list, get)
✅ Token persistence across multiple operations
✅ User authentication and bearer token validation
✅ Scope-based tool filtering
✅ Error handling for invalid operations
Implementation validates:
- ADR-002 external identity provider architecture
- No admin credentials needed in MCP server
- Centralized identity management via Keycloak
- Standards-based OAuth 2.0 / OIDC integration
- User auto-provisioning from IdP claims
🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-Authored-By: Claude <noreply@anthropic.com >
2025-11-02 22:03:20 +01:00
403f8be429
feat: Add Keycloak external IdP integration with custom scopes
...
Add comprehensive support for using Keycloak as an external identity
provider with Nextcloud custom scopes. This enables testing of ADR-002
external IdP integration patterns.
**Keycloak Realm Configuration:**
- Add frontendUrl attribute to issue tokens with public issuer URL
- Define 18 Nextcloud custom client scopes (notes:read/write,
calendar:read/write, contacts:read/write, cookbook:read/write,
deck:read/write, tables:read/write, files:read/write,
sharing:read/write, todo:read/write)
- Add all custom scopes to nextcloud-mcp-server client optional scopes
- Scopes include consent screen text for user-friendly OAuth flow
**MCP Server Configuration:**
- Add OIDC_JWKS_URI environment variable support
- Implement JWKS URI override logic for Docker networking
- Update NEXTCLOUD_PUBLIC_ISSUER_URL to include full realm path
- Enable MCP server to fetch JWKS from internal Docker network
**Test Infrastructure:**
- Add keycloak_oauth_client_credentials fixture (session-scoped)
- Add keycloak_oauth_token fixture with Playwright automation
- Implement PKCE (S256) support for Keycloak OAuth flow
- Add nc_mcp_keycloak_client fixture for MCP testing
- Create comprehensive test suite in test_keycloak_external_idp.py
**Tests Created:**
- test_keycloak_oauth_token_acquisition: Token acquisition via Playwright
- test_keycloak_oauth_client_credentials_discovery: OIDC discovery
- test_mcp_client_connects_to_keycloak_server: MCP connectivity
- test_external_idp_server_initialization: Server auto-detection
- test_external_idp_token_validation: Token validation flow
- test_tools_work_with_keycloak_token: End-to-end tool execution
- test_keycloak_token_persistence: Multi-operation token reuse
- test_user_auto_provisioning: Nextcloud user provisioning
- test_scope_filtering_with_keycloak: Scope-based tool filtering
- test_keycloak_error_handling: Error handling
- test_external_idp_architecture: Architecture documentation
**Current Status:**
- ✅ Keycloak realm configuration complete
- ✅ Custom scopes defined and available
- ✅ OAuth token acquisition working (1 test passing)
- ⚠️ Token validation needs additional work (external IdP userinfo)
**Files Modified:**
- keycloak/realm-export.json: Realm configuration with scopes
- tests/conftest.py: Keycloak OAuth fixtures (+285 lines)
- tests/server/oauth/test_keycloak_external_idp.py: New test suite
- docker-compose.yml: OIDC_JWKS_URI and issuer configuration
- nextcloud_mcp_server/app.py: JWKS URI override logic
🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-Authored-By: Claude <noreply@anthropic.com >
2025-11-02 22:03:20 +01:00
e331544cee
feat: Implement RFC 8693 token exchange for Keycloak (ADR-002 Tier 2)
...
Implements OAuth 2.0 Token Exchange (RFC 8693) enabling the MCP server to
exchange service account tokens for user-scoped tokens. This provides an
alternative to refresh tokens for background operations.
**Core Implementation:**
- Added `get_service_account_token()` method to KeycloakOAuthClient for
client_credentials grant
- Added `exchange_token_for_user()` method implementing RFC 8693 token exchange
- Fixed Fernet encryption key handling in RefreshTokenStorage (was incorrectly
base64 decoding already-encoded keys)
- Updated OAuth configuration to support offline_access scope and refresh token
storage infrastructure
**Keycloak Configuration:**
- Enabled `serviceAccountsEnabled` in realm-export.json
- Added `token.exchange.grant.enabled` attribute
- Added `client.token.exchange.standard.enabled` attribute (required for
Keycloak 26.2+ Standard Token Exchange V2)
- Fresh Keycloak imports now correctly enable token exchange
**Docker Compose:**
- Added TOKEN_ENCRYPTION_KEY and ENABLE_OFFLINE_ACCESS environment variables
- Created oauth-tokens volume for refresh token storage
- Configured both mcp-oauth and mcp-keycloak services
**Testing & Documentation:**
- Added tests/manual/test_token_exchange.py - Validates complete RFC 8693 flow
- Added tests/manual/test_nextcloud_impersonate.py - Documents session-based
impersonation limitations
- Added docs/oauth-impersonation-findings.md - Comprehensive investigation
findings and resolution documentation
**Verified Working:**
✅ Service account token acquisition (client_credentials grant)
✅ RFC 8693 token exchange for internal-to-internal tokens
✅ Exchanged tokens validate with Nextcloud APIs
✅ Keycloak 26.4.2 Standard Token Exchange V2 support
**Known Limitations:**
- User impersonation (requested_subject) requires Keycloak Legacy V1 with
preview features
- Cross-client token exchange limited to same realm
- Refresh token storage infrastructure ready but unused (MCP protocol limitation)
Dependencies: aiosqlite>=0.20.0
🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-Authored-By: Claude <noreply@anthropic.com >
2025-11-02 22:03:19 +01:00
37b0b4a281
fix: Update DCR token_type tests for OIDC app changes
...
The Nextcloud OIDC app has updated token_type parameter values:
- Changed from "Bearer" → "opaque" for opaque tokens
- Changed from "JWT" → "jwt" for JWT tokens
Updated test_dcr_token_type.py to use lowercase token_type values:
- token_type="jwt" for JWT-formatted tokens
- token_type="opaque" for opaque/bearer tokens
This fixes test failures where tests were using the old "Bearer" and
"JWT" (uppercase) values which are no longer recognized by the OIDC app.
Fixes test: test_dcr_respects_bearer_token_type
🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-Authored-By: Claude <noreply@anthropic.com >
2025-11-02 22:03:19 +01:00
ab7411d9fd
test: Fix tests
2025-10-25 22:07:46 +02:00
9414d9c9c3
test: Add integration marker to user/group tests
2025-10-25 20:16:14 +02:00
8a52df4a8e
test: Skip unstructured tests if not enabled
2025-10-25 20:13:41 +02:00
a36038422b
feat: Add text processing background worker for telling client about progress
2025-10-25 19:52:45 +02:00
2147fc1696
refactor: Transform document parsing into pluggable processor architecture
...
Refactors PR #190 's hardcoded Unstructured.io integration into a flexible,
extensible plugin system supporting multiple text extraction engines.
- **`DocumentProcessor` ABC**: Abstract interface for all processors
- **`ProcessorRegistry`**: Central registry for discovery and routing
- **`ProcessingResult`**: Standardized output format across processors
- **`UnstructuredProcessor`**: Refactored from `UnstructuredClient`
- **`TesseractProcessor`**: Local OCR for images (lightweight alternative)
- **`CustomHTTPProcessor`**: Generic wrapper for custom HTTP APIs
- New `get_document_processor_config()` returns structured config
- Supports enabling/disabling individual processors
- Per-processor configuration via environment variables
- **Breaking Change**: `ENABLE_UNSTRUCTURED_PARSING` replaced with:
- `ENABLE_DOCUMENT_PROCESSING=true/false` (master switch)
- `ENABLE_UNSTRUCTURED=true/false` (per-processor)
- `ENABLE_TESSERACT=true/false`
- `ENABLE_CUSTOM_PROCESSOR=true/false`
- `parse_document()` now uses `ProcessorRegistry`
- Auto-selects appropriate processor based on MIME type
- Processor priority system (Unstructured=10, Tesseract=5, Custom=1)
- `initialize_document_processors()` registers processors at startup
- Integrated into both BasicAuth and OAuth lifespans
- Graceful degradation if processors fail to initialize
```env
ENABLE_DOCUMENT_PROCESSING=false
ENABLE_UNSTRUCTURED=false
UNSTRUCTURED_API_URL=http://unstructured:8000
UNSTRUCTURED_STRATEGY=auto # auto|fast|hi_res
UNSTRUCTURED_LANGUAGES=eng,deu
ENABLE_TESSERACT=false
TESSERACT_LANG=eng
ENABLE_CUSTOM_PROCESSOR=false
CUSTOM_PROCESSOR_URL=http://localhost:9000/process
CUSTOM_PROCESSOR_TYPES=application/pdf,image/jpeg
```
- **Removed**: `tests/test_unstructured_config.py` (legacy tests)
- **Added**: `tests/unit/test_document_processor_config.py`
- 7 unit tests for new config system
- Tests individual and multi-processor configurations
- **Added**:
- `nextcloud_mcp_server/document_processors/__init__.py`
- `nextcloud_mcp_server/document_processors/base.py`
- `nextcloud_mcp_server/document_processors/registry.py`
- `nextcloud_mcp_server/document_processors/unstructured.py`
- `nextcloud_mcp_server/document_processors/tesseract.py`
- `nextcloud_mcp_server/document_processors/custom_http.py`
- `tests/unit/test_document_processor_config.py`
- **Modified**:
- `nextcloud_mcp_server/config.py` - New plugin config system
- `nextcloud_mcp_server/app.py` - Processor initialization
- `nextcloud_mcp_server/utils/document_parser.py` - Uses registry
- `nextcloud_mcp_server/server/webdav.py` - Import updates
- `env.sample` - New configuration format
- `docker-compose.yml` - (profile changes from previous work)
- **Removed**:
- `nextcloud_mcp_server/client/unstructured_client.py` - Replaced by UnstructuredProcessor
- `tests/test_unstructured_config.py` - Replaced with new tests
✅ **Extensible**: Add processors without modifying core code
✅ **Testable**: Mock processors for unit tests
✅ **Configurable**: Enable only needed processors
✅ **Flexible**: Choose fast (Tesseract) vs accurate (Unstructured)
✅ **Opt-in**: Disabled by default, no mandatory dependencies
Users upgrading from PR #190 need to update environment variables:
```bash
ENABLE_UNSTRUCTURED_PARSING=true
ENABLE_DOCUMENT_PROCESSING=true
ENABLE_UNSTRUCTURED=true
```
🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-Authored-By: Claude <noreply@anthropic.com >
2025-10-25 19:28:35 +02:00
f0e5333e43
Merge branch 'master' into feature/introduce_files_parsing_with_unstructured_service_for_webdav_files_retrieval
2025-10-25 17:23:38 +02:00
01b43c96ba
test: Update client id/secret -> client_info
2025-10-24 19:47:49 +02:00
50b69a2531
fix: Add support for RFC 7592 client registration and deletion
2025-10-24 19:19:27 +02:00
8e0a4d8ce5
feat(auth): Add support for client registration deletion
2025-10-24 18:54:24 +02:00
72fce189d2
test: Add tests for dcr endpoint and update oidc app
2025-10-24 18:48:05 +02:00
1e877f17f7
test: Replace persistent OAuth client cache with session-scoped fixtures
...
Remove file-based caching of OAuth client credentials and implement automatic
client lifecycle management for test fixtures.
Changes:
- Add RFC 7592 client deletion function in auth/client_registration.py
- Remove cache_file parameter from _create_oauth_client_with_scopes helper
- Update all OAuth credential fixtures to use yield/finalizer pattern
- Add automatic client cleanup at end of test session (best-effort)
- Remove persistent .nextcloud_oauth_*.json cache files
Benefits:
- No persistent cache files cluttering repository
- Fresh OAuth clients created for each test session via DCR
- Automatic cleanup attempts (RFC 7592 DELETE endpoint)
- Cleaner test environment with proper fixture lifecycle
Note: Client deletion may fail due to Nextcloud authentication middleware
(logged as warning). The key improvement is removing persistent cache files.
OAuth clients may accumulate in Nextcloud but can be cleaned manually.
2025-10-24 08:11:22 +02:00
13f76a7734
chore: Upgrade pydantic Config to ConfigDict
2025-10-24 06:18:13 +02:00
2f1bd1bbe9
test: Move client integration tests to mocked unit tests
2025-10-24 05:50:25 +02:00
d452684535
feat: Split read/write scopes into app:read/write scopes
2025-10-24 04:38:49 +02:00
d55e5708c7
ci: fix imports
2025-10-24 01:04:30 +02:00
d4ee5a74c2
test: Update default tokens to JWT, add to introspection tests
2025-10-24 00:51:50 +02:00
db79afacb9
improved tests - fixing the linting
2025-10-23 22:56:25 +02:00
6730dd4a4b
added new tests for unstructured api (pdf and docx workflow)
2025-10-23 22:38:27 +02:00
8734c4b292
add new tests for unstructured config
2025-10-23 22:37:52 +02:00
053cf7798b
fix: Add CORS middleware to allow browser-based clients like MCP Inspector
2025-10-23 15:23:41 +02:00
737780b417
chore: Make all env vars available to be overriden as cli options
2025-10-23 11:48:01 +02:00
54e975198f
test: Update all test network hosts to respect iss claims from JWTs
2025-10-23 11:09:51 +02:00
e48f5f3f30
feat(server): Add support for custom OIDC scopes and permissions via JWTs
2025-10-23 08:37:36 +02:00
1aecb099e6
fix: Use occ-created OAuth clients with allowed_scopes for all tests
...
The shared_oauth_client_credentials fixture was using Dynamic Client
Registration which doesn't support Nextcloud's allowed_scopes parameter.
This caused tokens to lack proper scope configuration, resulting in empty
tool lists when the server validated scopes.
Changes:
1. Updated shared_oauth_client_credentials to use occ oidc:create with
allowed_scopes="openid profile email nc:read nc:write"
2. Created opaque token client (not JWT) for port 8001 compatibility
3. Enhanced _create_oauth_client_with_scopes to support both JWT and
opaque token types via token_type parameter
This ensures:
- Regular OAuth tests (port 8001) get opaque tokens with proper scopes
- JWT OAuth tests (port 8002) get JWT tokens with embedded scopes
- Both token types have allowed_scopes configured on the OAuth client
Fixes test_mcp_oauth_server_connection which was getting empty tool list
🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-Authored-By: Claude <noreply@anthropic.com >
2025-10-22 07:38:16 +02:00
2c35e07675
fix: Separate OAuth fixtures for opaque vs JWT tokens
...
Previous fix created a JWT OAuth client for all tests, which broke the
regular OAuth server (port 8001) that expects opaque tokens.
This commit:
1. Reverts shared_oauth_client_credentials to use regular OAuth (opaque tokens)
2. Creates new shared_jwt_oauth_client_credentials for JWT OAuth clients
3. Creates new playwright_oauth_token_jwt fixture using JWT credentials
4. Updates nc_mcp_oauth_jwt_client to use JWT token fixture
This ensures:
- Regular OAuth tests (port 8001) use opaque tokens
- JWT OAuth tests (port 8002) use JWT tokens with embedded scopes
Fixes remaining CI failure in test_mcp_oauth_server_connection
2025-10-22 07:17:43 +02:00
5cfdff0faf
test: Create JWT OAuth client with explicit scopes for shared test fixture
...
The shared_oauth_client_credentials fixture was creating an OAuth client
without explicit allowed_scopes configuration. This caused JWT tokens to
lack nc:read and nc:write scope claims, resulting in the JWT MCP server
filtering out ALL tools when list_tools() was called.
Changed the fixture to use _create_oauth_client_with_scopes() helper to
create a JWT client with explicit allowed_scopes="openid profile email
nc:read nc:write", matching the scopes requested in the authorization
URL and the behavior of other scoped test fixtures.
This fixes CI test failures in:
- test_mcp_oauth.py::test_mcp_oauth_server_connection
- test_mcp_oauth_jwt.py::test_jwt_mcp_server_connection
- test_mcp_oauth_jwt.py::test_jwt_tool_list_operations
- test_mcp_oauth_jwt.py::test_jwt_automation_worked
All were failing with: assert len(result.tools) > 0 (result.tools was empty)
2025-10-22 07:02:40 +02:00
8a3269f366
test: Use separate docker compose command
2025-10-22 06:38:05 +02:00
c069d78f80
feat: Initialize JWT-scoped tools
2025-10-22 06:21:16 +02:00
63b898c0e3
chore: Update logs
2025-10-20 22:57:18 +02:00
a143123acc
fix(caldav): Check that calendar exists after creation to avoid race condition
...
Verify that field preservation tests still operate
2025-10-19 23:44:39 +02:00
1dc2ddfdb7
fix(caldav): Properly parse datetimes as vDDDTypes
2025-10-19 20:13:05 +02:00
92e18825bc
feat(caldav): Add support for tasks
2025-10-19 18:02:43 +02:00
5757f2582b
ci: Run oauth tests
2025-10-19 00:49:55 +02:00
b72514bb32
ci: Add pytest-timeout to dev deps
2025-10-19 00:27:19 +02:00
95da43ea0f
ci: Increase playwright timeout to 60s
2025-10-18 23:26:50 +02:00
963a504ae2
ci: Replace 0.5 stagger with 10s in CI
2025-10-18 22:57:47 +02:00
ead298c132
chore: revert conftest.py
2025-10-18 22:44:51 +02:00
2f805e54b7
test: Migrate load test benchmark scripts to anyio
...
Remove unused redis container
2025-10-18 22:40:50 +02:00
6158a890af
feat(webdav): Add search and list favorite response tools
2025-10-18 22:02:26 +02:00
240ceb3808
test: Migrate load test framework to anyio as well
2025-10-18 22:02:25 +02:00
1459fe9bc8
test: Replace pytest-asyncio plugin fixtures with anyio fixtures
2025-10-18 22:02:25 +02:00
37164dbdbc
chore: sort imports
2025-10-18 22:02:25 +02:00
c3ff92a8c1
test: Cleanup testing fixtures regarding canceled scopes
2025-10-18 22:02:25 +02:00
371d0c93a5
test: Update oauth benchmark tests
2025-10-18 22:02:25 +02:00
Chris Coutinho