Commit Graph

133 Commits

Author SHA1 Message Date
Chris Coutinho ab7411d9fd test: Fix tests 2025-10-25 22:07:46 +02:00
Chris Coutinho 9414d9c9c3 test: Add integration marker to user/group tests 2025-10-25 20:16:14 +02:00
Chris Coutinho 8a52df4a8e test: Skip unstructured tests if not enabled 2025-10-25 20:13:41 +02:00
Chris Coutinho a36038422b feat: Add text processing background worker for telling client about progress 2025-10-25 19:52:45 +02:00
Chris Coutinho 2147fc1696 refactor: Transform document parsing into pluggable processor architecture
Refactors PR #190's hardcoded Unstructured.io integration into a flexible,
extensible plugin system supporting multiple text extraction engines.

- **`DocumentProcessor` ABC**: Abstract interface for all processors
- **`ProcessorRegistry`**: Central registry for discovery and routing
- **`ProcessingResult`**: Standardized output format across processors

- **`UnstructuredProcessor`**: Refactored from `UnstructuredClient`
- **`TesseractProcessor`**: Local OCR for images (lightweight alternative)
- **`CustomHTTPProcessor`**: Generic wrapper for custom HTTP APIs

- New `get_document_processor_config()` returns structured config
- Supports enabling/disabling individual processors
- Per-processor configuration via environment variables
- **Breaking Change**: `ENABLE_UNSTRUCTURED_PARSING` replaced with:
  - `ENABLE_DOCUMENT_PROCESSING=true/false` (master switch)
  - `ENABLE_UNSTRUCTURED=true/false` (per-processor)
  - `ENABLE_TESSERACT=true/false`
  - `ENABLE_CUSTOM_PROCESSOR=true/false`

- `parse_document()` now uses `ProcessorRegistry`
- Auto-selects appropriate processor based on MIME type
- Processor priority system (Unstructured=10, Tesseract=5, Custom=1)

- `initialize_document_processors()` registers processors at startup
- Integrated into both BasicAuth and OAuth lifespans
- Graceful degradation if processors fail to initialize

```env
ENABLE_DOCUMENT_PROCESSING=false

ENABLE_UNSTRUCTURED=false
UNSTRUCTURED_API_URL=http://unstructured:8000
UNSTRUCTURED_STRATEGY=auto  # auto|fast|hi_res
UNSTRUCTURED_LANGUAGES=eng,deu

ENABLE_TESSERACT=false
TESSERACT_LANG=eng

ENABLE_CUSTOM_PROCESSOR=false
CUSTOM_PROCESSOR_URL=http://localhost:9000/process
CUSTOM_PROCESSOR_TYPES=application/pdf,image/jpeg
```

- **Removed**: `tests/test_unstructured_config.py` (legacy tests)
- **Added**: `tests/unit/test_document_processor_config.py`
  - 7 unit tests for new config system
  - Tests individual and multi-processor configurations

- **Added**:
  - `nextcloud_mcp_server/document_processors/__init__.py`
  - `nextcloud_mcp_server/document_processors/base.py`
  - `nextcloud_mcp_server/document_processors/registry.py`
  - `nextcloud_mcp_server/document_processors/unstructured.py`
  - `nextcloud_mcp_server/document_processors/tesseract.py`
  - `nextcloud_mcp_server/document_processors/custom_http.py`
  - `tests/unit/test_document_processor_config.py`

- **Modified**:
  - `nextcloud_mcp_server/config.py` - New plugin config system
  - `nextcloud_mcp_server/app.py` - Processor initialization
  - `nextcloud_mcp_server/utils/document_parser.py` - Uses registry
  - `nextcloud_mcp_server/server/webdav.py` - Import updates
  - `env.sample` - New configuration format
  - `docker-compose.yml` - (profile changes from previous work)

- **Removed**:
  - `nextcloud_mcp_server/client/unstructured_client.py` - Replaced by UnstructuredProcessor
  - `tests/test_unstructured_config.py` - Replaced with new tests

 **Extensible**: Add processors without modifying core code
 **Testable**: Mock processors for unit tests
 **Configurable**: Enable only needed processors
 **Flexible**: Choose fast (Tesseract) vs accurate (Unstructured)
 **Opt-in**: Disabled by default, no mandatory dependencies

Users upgrading from PR #190 need to update environment variables:
```bash
ENABLE_UNSTRUCTURED_PARSING=true

ENABLE_DOCUMENT_PROCESSING=true
ENABLE_UNSTRUCTURED=true
```

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-25 19:28:35 +02:00
yuisheaven f0e5333e43 Merge branch 'master' into feature/introduce_files_parsing_with_unstructured_service_for_webdav_files_retrieval 2025-10-25 17:23:38 +02:00
Chris Coutinho 01b43c96ba test: Update client id/secret -> client_info 2025-10-24 19:47:49 +02:00
Chris Coutinho 50b69a2531 fix: Add support for RFC 7592 client registration and deletion 2025-10-24 19:19:27 +02:00
Chris Coutinho 8e0a4d8ce5 feat(auth): Add support for client registration deletion 2025-10-24 18:54:24 +02:00
Chris Coutinho 72fce189d2 test: Add tests for dcr endpoint and update oidc app 2025-10-24 18:48:05 +02:00
Chris Coutinho 1e877f17f7 test: Replace persistent OAuth client cache with session-scoped fixtures
Remove file-based caching of OAuth client credentials and implement automatic
client lifecycle management for test fixtures.

Changes:
- Add RFC 7592 client deletion function in auth/client_registration.py
- Remove cache_file parameter from _create_oauth_client_with_scopes helper
- Update all OAuth credential fixtures to use yield/finalizer pattern
- Add automatic client cleanup at end of test session (best-effort)
- Remove persistent .nextcloud_oauth_*.json cache files

Benefits:
- No persistent cache files cluttering repository
- Fresh OAuth clients created for each test session via DCR
- Automatic cleanup attempts (RFC 7592 DELETE endpoint)
- Cleaner test environment with proper fixture lifecycle

Note: Client deletion may fail due to Nextcloud authentication middleware
(logged as warning). The key improvement is removing persistent cache files.
OAuth clients may accumulate in Nextcloud but can be cleaned manually.
2025-10-24 08:11:22 +02:00
Chris Coutinho 13f76a7734 chore: Upgrade pydantic Config to ConfigDict 2025-10-24 06:18:13 +02:00
Chris Coutinho 2f1bd1bbe9 test: Move client integration tests to mocked unit tests 2025-10-24 05:50:25 +02:00
Chris Coutinho d452684535 feat: Split read/write scopes into app:read/write scopes 2025-10-24 04:38:49 +02:00
Chris Coutinho d55e5708c7 ci: fix imports 2025-10-24 01:04:30 +02:00
Chris Coutinho d4ee5a74c2 test: Update default tokens to JWT, add to introspection tests 2025-10-24 00:51:50 +02:00
yuisheaven db79afacb9 improved tests - fixing the linting 2025-10-23 22:56:25 +02:00
yuisheaven 6730dd4a4b added new tests for unstructured api (pdf and docx workflow) 2025-10-23 22:38:27 +02:00
yuisheaven 8734c4b292 add new tests for unstructured config 2025-10-23 22:37:52 +02:00
Chris Coutinho 053cf7798b fix: Add CORS middleware to allow browser-based clients like MCP Inspector 2025-10-23 15:23:41 +02:00
Chris Coutinho 737780b417 chore: Make all env vars available to be overriden as cli options 2025-10-23 11:48:01 +02:00
Chris Coutinho 54e975198f test: Update all test network hosts to respect iss claims from JWTs 2025-10-23 11:09:51 +02:00
Chris Coutinho e48f5f3f30 feat(server): Add support for custom OIDC scopes and permissions via JWTs 2025-10-23 08:37:36 +02:00
Chris Coutinho 1aecb099e6 fix: Use occ-created OAuth clients with allowed_scopes for all tests
The shared_oauth_client_credentials fixture was using Dynamic Client
Registration which doesn't support Nextcloud's allowed_scopes parameter.
This caused tokens to lack proper scope configuration, resulting in empty
tool lists when the server validated scopes.

Changes:
1. Updated shared_oauth_client_credentials to use occ oidc:create with
   allowed_scopes="openid profile email nc:read nc:write"
2. Created opaque token client (not JWT) for port 8001 compatibility
3. Enhanced _create_oauth_client_with_scopes to support both JWT and
   opaque token types via token_type parameter

This ensures:
- Regular OAuth tests (port 8001) get opaque tokens with proper scopes
- JWT OAuth tests (port 8002) get JWT tokens with embedded scopes
- Both token types have allowed_scopes configured on the OAuth client

Fixes test_mcp_oauth_server_connection which was getting empty tool list

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-22 07:38:16 +02:00
Chris Coutinho 2c35e07675 fix: Separate OAuth fixtures for opaque vs JWT tokens
Previous fix created a JWT OAuth client for all tests, which broke the
regular OAuth server (port 8001) that expects opaque tokens.

This commit:
1. Reverts shared_oauth_client_credentials to use regular OAuth (opaque tokens)
2. Creates new shared_jwt_oauth_client_credentials for JWT OAuth clients
3. Creates new playwright_oauth_token_jwt fixture using JWT credentials
4. Updates nc_mcp_oauth_jwt_client to use JWT token fixture

This ensures:
- Regular OAuth tests (port 8001) use opaque tokens
- JWT OAuth tests (port 8002) use JWT tokens with embedded scopes

Fixes remaining CI failure in test_mcp_oauth_server_connection
2025-10-22 07:17:43 +02:00
Chris Coutinho 5cfdff0faf test: Create JWT OAuth client with explicit scopes for shared test fixture
The shared_oauth_client_credentials fixture was creating an OAuth client
without explicit allowed_scopes configuration. This caused JWT tokens to
lack nc:read and nc:write scope claims, resulting in the JWT MCP server
filtering out ALL tools when list_tools() was called.

Changed the fixture to use _create_oauth_client_with_scopes() helper to
create a JWT client with explicit allowed_scopes="openid profile email
nc:read nc:write", matching the scopes requested in the authorization
URL and the behavior of other scoped test fixtures.

This fixes CI test failures in:
- test_mcp_oauth.py::test_mcp_oauth_server_connection
- test_mcp_oauth_jwt.py::test_jwt_mcp_server_connection
- test_mcp_oauth_jwt.py::test_jwt_tool_list_operations
- test_mcp_oauth_jwt.py::test_jwt_automation_worked

All were failing with: assert len(result.tools) > 0 (result.tools was empty)
2025-10-22 07:02:40 +02:00
Chris Coutinho 8a3269f366 test: Use separate docker compose command 2025-10-22 06:38:05 +02:00
Chris Coutinho c069d78f80 feat: Initialize JWT-scoped tools 2025-10-22 06:21:16 +02:00
Chris Coutinho 63b898c0e3 chore: Update logs 2025-10-20 22:57:18 +02:00
Chris Coutinho a143123acc fix(caldav): Check that calendar exists after creation to avoid race condition
Verify that field preservation tests still operate
2025-10-19 23:44:39 +02:00
Chris Coutinho 1dc2ddfdb7 fix(caldav): Properly parse datetimes as vDDDTypes 2025-10-19 20:13:05 +02:00
Chris Coutinho 92e18825bc feat(caldav): Add support for tasks 2025-10-19 18:02:43 +02:00
Chris Coutinho 5757f2582b ci: Run oauth tests 2025-10-19 00:49:55 +02:00
Chris Coutinho b72514bb32 ci: Add pytest-timeout to dev deps 2025-10-19 00:27:19 +02:00
Chris Coutinho 95da43ea0f ci: Increase playwright timeout to 60s 2025-10-18 23:26:50 +02:00
Chris Coutinho 963a504ae2 ci: Replace 0.5 stagger with 10s in CI 2025-10-18 22:57:47 +02:00
Chris Coutinho ead298c132 chore: revert conftest.py 2025-10-18 22:44:51 +02:00
Chris Coutinho 2f805e54b7 test: Migrate load test benchmark scripts to anyio
Remove unused redis container
2025-10-18 22:40:50 +02:00
Chris Coutinho 6158a890af feat(webdav): Add search and list favorite response tools 2025-10-18 22:02:26 +02:00
Chris Coutinho 240ceb3808 test: Migrate load test framework to anyio as well 2025-10-18 22:02:25 +02:00
Chris Coutinho 1459fe9bc8 test: Replace pytest-asyncio plugin fixtures with anyio fixtures 2025-10-18 22:02:25 +02:00
Chris Coutinho 37164dbdbc chore: sort imports 2025-10-18 22:02:25 +02:00
Chris Coutinho c3ff92a8c1 test: Cleanup testing fixtures regarding canceled scopes 2025-10-18 22:02:25 +02:00
Chris Coutinho 371d0c93a5 test: Update oauth benchmark tests 2025-10-18 22:02:25 +02:00
Chris Coutinho 644c59bf78 docs: remove old docs 2025-10-18 22:02:25 +02:00
Chris Coutinho 056b6fc9d6 test: Initialize load testing framework 2025-10-18 22:02:24 +02:00
Chris Coutinho 83917b3786 perf(notes): Improve notes search performance using async iterators 2025-10-18 22:02:19 +02:00
Chris Coutinho 955ad78f13 test: Add load testing framework 2025-10-18 22:02:19 +02:00
Chris Coutinho 27519d0f62 test: Replace http server for recipes with nginx container 2025-10-17 04:30:03 +02:00
Chris Coutinho 0fd32ecd34 test: Fix test networking 2025-10-17 03:58:36 +02:00