feat: add Alembic database migration system

Implements Alembic for managing token storage database schema versions. Migrations run automatically on startup with full backward compatibility. **Changes:** - Add Alembic dependency (1.14.0+) and SQLAlchemy (auto-installed) - Create migration infrastructure in alembic/ directory - Add initial migration (001) capturing current schema - Modify RefreshTokenStorage.initialize() to run migrations via anyio - Add CLI commands: db upgrade, current, history, downgrade, migrate - Add comprehensive migration documentation **Backward Compatibility:** - Pre-Alembic databases automatically stamped with revision 001 - No schema changes for existing databases - Automatic upgrade on first startup after update **Migration Strategy:** Three scenarios handled: 1. New database → Run migrations from scratch 2. Pre-Alembic database → Stamp with 001 (no changes) 3. Alembic-managed → Upgrade to latest **Architecture:** - Uses anyio.to_thread.run_sync() for structured concurrency - Alembic env.py runs with anyio.run() in worker thread - SQLite-friendly migration patterns documented - No ThreadPoolExecutor needed (anyio handles it) **CLI Usage:** ```bash nextcloud-mcp-server db upgrade # Upgrade to latest nextcloud-mcp-server db current # Show version nextcloud-mcp-server db history # View changelog nextcloud-mcp-server db downgrade # Rollback (with confirmation) nextcloud-mcp-server db migrate "description" # Create migration ``` **Testing:** - All 13 webhook storage tests pass - New/pre-Alembic database scenarios validated - anyio integration tested 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-17 23:19:32 +01:00
parent a4a34e46a8
commit 3fa376905c
17 changed files with 1436 additions and 152 deletions
@@ -0,0 +1,71 @@
+Database Migrations for nextcloud-mcp-server
+============================================
+
+This directory contains Alembic database migrations for the token storage database.
+
+Structure
+---------
+- env.py: Alembic environment configuration
+- script.py.mako: Template for generating new migration files
+- versions/: Directory containing migration scripts
+
+Usage
+-----
+Migrations are managed via the CLI:
+
+    # Upgrade database to latest version
+    uv run nextcloud-mcp-server db upgrade
+
+    # Show current database version
+    uv run nextcloud-mcp-server db current
+
+    # Show migration history
+    uv run nextcloud-mcp-server db history
+
+    # Create a new migration (developers only)
+    uv run nextcloud-mcp-server db migrate "description of changes"
+
+    # Downgrade database by one version (emergency use only)
+    uv run nextcloud-mcp-server db downgrade
+
+Direct Alembic Usage
+--------------------
+You can also use Alembic commands directly:
+
+    # Specify database URL via -x flag
+    uv run alembic -x database_url=sqlite+aiosqlite:////path/to/tokens.db upgrade head
+
+    # Or set in alembic.ini and run
+    uv run alembic upgrade head
+    uv run alembic current
+    uv run alembic history
+
+Writing Migrations
+------------------
+Since we don't use SQLAlchemy models, migrations are written with raw SQL:
+
+    def upgrade() -> None:
+        op.execute("""
+            ALTER TABLE refresh_tokens
+            ADD COLUMN new_field TEXT
+        """)
+
+    def downgrade() -> None:
+        # SQLite doesn't support DROP COLUMN, use table recreation
+        op.execute("""
+            CREATE TABLE refresh_tokens_new AS
+            SELECT user_id, encrypted_token, ... FROM refresh_tokens
+        """)
+        op.execute("DROP TABLE refresh_tokens")
+        op.execute("ALTER TABLE refresh_tokens_new RENAME TO refresh_tokens")
+
+Migration File Naming
+---------------------
+Format: YYYYMMDD_HHMM_<revision>_<slug>.py
+Example: 20251217_2200_001_initial_schema.py
+
+Notes
+-----
+- Migrations run automatically when RefreshTokenStorage.initialize() is called
+- Existing databases are automatically stamped with the initial version
+- SQLite has limited ALTER TABLE support - complex changes require table recreation
@@ -0,0 +1,128 @@
+"""Alembic environment configuration for nextcloud-mcp-server.
+
+This module configures how Alembic runs database migrations for the
+token storage database. It supports both online and offline migration modes.
+
+Uses anyio for async operations, consistent with the project's async patterns.
+"""
+
+import logging
+from pathlib import Path
+
+import anyio
+from sqlalchemy import pool
+from sqlalchemy.engine import Connection
+from sqlalchemy.ext.asyncio import async_engine_from_config
+
+from alembic import context
+
+# Configure logging
+logger = logging.getLogger("alembic.env")
+
+# This is the Alembic Config object, which provides
+# access to the values within the .ini file in use.
+config = context.config
+
+# We don't use SQLAlchemy models, so target_metadata is None
+# Migrations will be written manually using op.execute() for raw SQL
+target_metadata = None
+
+
+def get_database_url() -> str:
+    """
+    Get the database URL from Alembic config or environment.
+
+    The URL can be set in alembic.ini or passed via -x database_url=...
+    when running Alembic commands.
+
+    Returns:
+        Database URL (SQLite URL format)
+    """
+    # Check if URL is passed via -x database_url=...
+    url = context.get_x_argument(as_dictionary=True).get("database_url")
+
+    if not url:
+        # Fall back to alembic.ini configuration
+        url = config.get_main_option("sqlalchemy.url")
+
+    if not url:
+        # Default to /app/data/tokens.db for Docker deployments
+        db_path = Path("/app/data/tokens.db")
+        url = f"sqlite+aiosqlite:///{db_path}"
+        logger.warning(
+            f"No database URL configured, using default: {url}. "
+            "Set sqlalchemy.url in alembic.ini or pass -x database_url=..."
+        )
+
+    return url
+
+
+def run_migrations_offline() -> None:
+    """Run migrations in 'offline' mode.
+
+    This configures the context with just a URL and not an Engine,
+    though an Engine is acceptable here as well. By skipping the
+    Engine creation we don't even need a DBAPI to be available.
+
+    Calls to context.execute() here emit the given string to the
+    script output.
+
+    This mode is useful for generating SQL scripts without database access.
+    """
+    url = get_database_url()
+    context.configure(
+        url=url,
+        target_metadata=target_metadata,
+        literal_binds=True,
+        dialect_opts={"paramstyle": "named"},
+    )
+
+    with context.begin_transaction():
+        context.run_migrations()
+
+
+def do_run_migrations(connection: Connection) -> None:
+    """Execute migrations within a database connection."""
+    context.configure(connection=connection, target_metadata=target_metadata)
+
+    with context.begin_transaction():
+        context.run_migrations()
+
+
+async def run_async_migrations() -> None:
+    """Run migrations in 'online' mode with async support.
+
+    In this scenario we create an async Engine and associate
+    a connection with the context.
+    """
+    # Get database URL and update config
+    url = get_database_url()
+    config.set_main_option("sqlalchemy.url", url)
+
+    # Create async engine
+    connectable = async_engine_from_config(
+        config.get_section(config.config_ini_section, {}),
+        prefix="sqlalchemy.",
+        poolclass=pool.NullPool,  # Don't pool connections for migrations
+    )
+
+    async with connectable.connect() as connection:
+        await connection.run_sync(do_run_migrations)
+
+    await connectable.dispose()
+
+
+def run_migrations_online() -> None:
+    """Run migrations in 'online' mode.
+
+    This function is called from storage.py's initialize() method via
+    anyio.to_thread.run_sync(), so it always runs in a worker thread
+    with its own event loop. We can safely use anyio.run() here.
+    """
+    anyio.run(run_async_migrations)
+
+
+if context.is_offline_mode():
+    run_migrations_offline()
+else:
+    run_migrations_online()
@@ -0,0 +1,26 @@
+"""${message}
+
+Revision ID: ${up_revision}
+Revises: ${down_revision | comma,n}
+Create Date: ${create_date}
+
+"""
+from alembic import op
+import sqlalchemy as sa
+${imports if imports else ""}
+
+# revision identifiers, used by Alembic.
+revision = ${repr(up_revision)}
+down_revision = ${repr(down_revision)}
+branch_labels = ${repr(branch_labels)}
+depends_on = ${repr(depends_on)}
+
+
+def upgrade() -> None:
+    """Apply migration changes to upgrade the database schema."""
+    ${upgrades if upgrades else "pass"}
+
+
+def downgrade() -> None:
+    """Revert migration changes to downgrade the database schema."""
+    ${downgrades if downgrades else "pass"}
@@ -0,0 +1,185 @@
+"""Initial schema for token storage database
+
+This migration creates the initial database schema including:
+- refresh_tokens: OAuth refresh tokens and user profiles
+- audit_logs: Audit trail for security events
+- oauth_clients: OAuth client credentials (DCR)
+- oauth_sessions: OAuth flow session state (ADR-004 Progressive Consent)
+- registered_webhooks: Webhook registration tracking (both OAuth and BasicAuth)
+- schema_version: Legacy schema version tracking (deprecated, use alembic_version)
+
+Revision ID: 001
+Revises:
+Create Date: 2025-12-17 22:00:00.000000
+
+"""
+
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision = "001"
+down_revision = None
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    """Create initial database schema."""
+
+    # Refresh tokens table (OAuth mode only, for background jobs)
+    op.execute(
+        """
+        CREATE TABLE IF NOT EXISTS refresh_tokens (
+            user_id TEXT PRIMARY KEY,
+            encrypted_token BLOB NOT NULL,
+            expires_at INTEGER,
+            created_at INTEGER NOT NULL,
+            updated_at INTEGER NOT NULL,
+            -- ADR-004 Progressive Consent fields
+            flow_type TEXT DEFAULT 'hybrid',
+            token_audience TEXT DEFAULT 'nextcloud',
+            provisioned_at INTEGER,
+            provisioning_client_id TEXT,
+            scopes TEXT,
+            -- Browser session profile cache
+            user_profile TEXT,
+            profile_cached_at INTEGER
+        )
+        """
+    )
+
+    # Audit logs table (both OAuth and BasicAuth modes)
+    op.execute(
+        """
+        CREATE TABLE IF NOT EXISTS audit_logs (
+            id INTEGER PRIMARY KEY AUTOINCREMENT,
+            timestamp INTEGER NOT NULL,
+            event TEXT NOT NULL,
+            user_id TEXT NOT NULL,
+            resource_type TEXT,
+            resource_id TEXT,
+            auth_method TEXT,
+            hostname TEXT
+        )
+        """
+    )
+
+    # Index on audit logs for efficient queries
+    op.execute(
+        """
+        CREATE INDEX IF NOT EXISTS idx_audit_user_timestamp
+        ON audit_logs(user_id, timestamp)
+        """
+    )
+
+    # OAuth client credentials storage (OAuth mode only)
+    op.execute(
+        """
+        CREATE TABLE IF NOT EXISTS oauth_clients (
+            id INTEGER PRIMARY KEY,
+            client_id TEXT UNIQUE NOT NULL,
+            encrypted_client_secret BLOB NOT NULL,
+            client_id_issued_at INTEGER NOT NULL,
+            client_secret_expires_at INTEGER NOT NULL,
+            redirect_uris TEXT NOT NULL,
+            encrypted_registration_access_token BLOB,
+            registration_client_uri TEXT,
+            created_at INTEGER NOT NULL,
+            updated_at INTEGER NOT NULL
+        )
+        """
+    )
+
+    # OAuth flow sessions (ADR-004 Progressive Consent)
+    op.execute(
+        """
+        CREATE TABLE IF NOT EXISTS oauth_sessions (
+            session_id TEXT PRIMARY KEY,
+            client_id TEXT,
+            client_redirect_uri TEXT NOT NULL,
+            state TEXT,
+            code_challenge TEXT,
+            code_challenge_method TEXT,
+            mcp_authorization_code TEXT UNIQUE,
+            idp_access_token TEXT,
+            idp_refresh_token TEXT,
+            user_id TEXT,
+            created_at INTEGER NOT NULL,
+            expires_at INTEGER NOT NULL,
+            -- ADR-004 Progressive Consent fields
+            flow_type TEXT DEFAULT 'hybrid',
+            requested_scopes TEXT,
+            granted_scopes TEXT,
+            is_provisioning BOOLEAN DEFAULT FALSE
+        )
+        """
+    )
+
+    # Index for MCP authorization code lookups
+    op.execute(
+        """
+        CREATE INDEX IF NOT EXISTS idx_oauth_sessions_mcp_code
+        ON oauth_sessions(mcp_authorization_code)
+        """
+    )
+
+    # Legacy schema version tracking table
+    # NOTE: This is deprecated in favor of Alembic's alembic_version table
+    # Kept for backward compatibility with pre-Alembic databases
+    op.execute(
+        """
+        CREATE TABLE IF NOT EXISTS schema_version (
+            version INTEGER PRIMARY KEY,
+            applied_at REAL NOT NULL
+        )
+        """
+    )
+
+    # Registered webhooks tracking (both BasicAuth and OAuth modes)
+    op.execute(
+        """
+        CREATE TABLE IF NOT EXISTS registered_webhooks (
+            id INTEGER PRIMARY KEY AUTOINCREMENT,
+            webhook_id INTEGER NOT NULL UNIQUE,
+            preset_id TEXT NOT NULL,
+            created_at REAL NOT NULL
+        )
+        """
+    )
+
+    # Indexes for efficient webhook queries
+    op.execute(
+        """
+        CREATE INDEX IF NOT EXISTS idx_webhooks_preset
+        ON registered_webhooks(preset_id)
+        """
+    )
+
+    op.execute(
+        """
+        CREATE INDEX IF NOT EXISTS idx_webhooks_created
+        ON registered_webhooks(created_at)
+        """
+    )
+
+
+def downgrade() -> None:
+    """Drop all tables and indexes.
+
+    WARNING: This will destroy all data in the database!
+    Use with extreme caution.
+    """
+
+    # Drop indexes first
+    op.execute("DROP INDEX IF EXISTS idx_webhooks_created")
+    op.execute("DROP INDEX IF EXISTS idx_webhooks_preset")
+    op.execute("DROP INDEX IF EXISTS idx_oauth_sessions_mcp_code")
+    op.execute("DROP INDEX IF EXISTS idx_audit_user_timestamp")
+
+    # Drop tables
+    op.execute("DROP TABLE IF EXISTS registered_webhooks")
+    op.execute("DROP TABLE IF EXISTS schema_version")
+    op.execute("DROP TABLE IF EXISTS oauth_sessions")
+    op.execute("DROP TABLE IF EXISTS oauth_clients")
+    op.execute("DROP TABLE IF EXISTS audit_logs")
+    op.execute("DROP TABLE IF EXISTS refresh_tokens")