Backup Restoration & Recovery

Comprehensive backup restoration system providing point-in-time recovery, disaster recovery, and cross-environment/cross-server restore capabilities for Odoo instances.

Overview

The OEC.SH platform provides enterprise-grade backup restoration capabilities:

• Point-in-Time Recovery: Restore any environment to any previous backup snapshot
• Cross-Environment Restore: Restore backups between development, staging, and production
• Cross-Server Restore: Restore backups across different servers in the same organization
• Selective Restoration: Choose to restore database-only or filestore-only
• Pre-Restore Safety Backup: Automatic backup before restore for rollback capability
• Migration Restore Safeguards: Protection against accidental data loss during redeployment
• Background Processing: All restore operations run asynchronously via ARQ task queue
• Real-Time Progress: Track restore progress via Server-Sent Events (SSE)

Restore Workflow

Restore to Same Environment (In-Place)

Use Case

Recover from data corruption, restore accidentally deleted data, or roll back to a known-good state.

API Endpoint

POST /api/v1/backups/restore
Content-Type: application/json
Authorization: Bearer <token>
 
{
  "backup_id": "uuid-of-backup",
  "create_pre_restore_backup": true,
  "include_filestore": true
}

Process Flow

1. Pre-Restore Safety Backup

   • Creates automatic backup of current state (type: pre_restore)
   • Stored in configured storage provider
   • Allows rollback if restore fails or produces unexpected results

2. Environment Preparation

   • Stop Odoo container to prevent active connections
   • Verify environment is in valid state for restore

3. Download Backup

   • Download backup ZIP from storage provider (S3, R2, B2, MinIO, FTP, SFTP)
   • Extract dump.sql and filestore.tar.gz from ZIP package
   • Local temporary storage during restore process

4. Database Restoration

   • Get PostgreSQL container name: {env_id}_db
   • Force disconnect all active database connections:

     SELECT pg_terminate_backend(pid)
     FROM pg_stat_activity
     WHERE datname = 'db_name' AND pid <> pg_backend_pid();
   • Drop existing database:

     DROP DATABASE IF EXISTS "db_name";
   • Create fresh database:

     CREATE DATABASE "db_name" OWNER odoo;
   • Copy dump file into PostgreSQL container: docker cp dump.sql {container}:/tmp/restore.dump
   • Execute pg_restore:

     docker exec {db_container} pg_restore -U odoo -d db_name -v /tmp/restore.dump
   • Timeout: 1 hour (3600 seconds)
   • Note: pg_restore may return non-zero exit code for warnings; check for FATAL in stderr

5. Filestore Restoration (if include_filestore=true)

   • Remove existing filestore directory: rm -rf /var/lib/odoo/filestore/{db_name}
   • Copy archive into Odoo container: docker cp filestore.tar.gz {container}:/tmp/filestore.tar.gz
   • Extract archive: tar -xzf /tmp/filestore.tar.gz -C /var/lib/odoo/filestore
   • Handle database name changes (cross-env restore): rename directory if source/target db names differ
   • Fix ownership: chown -R odoo:odoo /var/lib/odoo/filestore/{db_name}

6. Start Odoo Container

   • Start Odoo container: docker start {env_id}_odoo
   • Container uses restored database and filestore

7. Verification

   • Wait for container health check (max 30 attempts, 2-second intervals)
   • Verify database connection from Odoo:

     SELECT COUNT(*) FROM res_users;
   • Ensure container is in healthy or running state

8. Cleanup

   • Remove temporary files from server
   • Update restore point status to COMPLETED

Downtime Considerations

Expected Downtime: 2-15 minutes (depends on database size)

• Small database (<1 GB): ~2-5 minutes
• Medium database (1-10 GB): ~5-10 minutes
• Large database (>10 GB): ~10-15+ minutes

The environment is unavailable during the entire restore process. Plan restores during maintenance windows for production environments.

Data Overwrite Warnings

Best Practices:

• Always enable pre-restore safety backup (default)
• Verify backup timestamp before restoring
• Test restore on staging environment first
• Notify users before production restore
• Document restore reason for audit trail

Restore to Different Environment

Use Case

• Copy production data to staging for testing
• Refresh development environment with production data
• Clone environment state to different environment type
• Test restore without affecting source environment

API Endpoint

POST /api/v1/backups/restore
Content-Type: application/json
Authorization: Bearer <token>
 
{
  "backup_id": "uuid-of-backup",
  "target_environment_id": "uuid-of-target-env",
  "create_pre_restore_backup": true,
  "include_filestore": true
}

Requirements

• Same Project: Target environment must belong to same project as backup source
• Server Assignment: Target environment must have a VM assigned
• Environment Status: Target should be in valid state (not deleted)
• Permissions: User must have project.backups.restore permission for both source and target

Compatibility Validation

The system validates compatibility before restore:

# Project validation
if source_env.project_id != target_env.project_id:
    raise RestoreError("Cross-project restore not supported")
 
# VM validation
if not target_env.vm_id:
    raise RestoreError("Target environment has no server assigned")
 
# Version compatibility (warning only)
if backup_odoo_version != target_env.odoo_version:
    logger.warning("Odoo version mismatch - may cause compatibility issues")

Database Name Handling

When restoring to a different environment:

1. Extract backup using original database name from environment_snapshot
2. Restore database to target environment's database name
3. If filestore database directory differs, rename after extraction:

   mv /var/lib/odoo/filestore/{original_db} /var/lib/odoo/filestore/{target_db}

Configuration Differences

Target environment retains its own configuration:

• Domain/Subdomain: Target environment's domain settings (not overwritten)
• Resource Limits: Target environment's CPU/RAM/disk quotas
• Environment Variables: Target environment's custom env vars
• Odoo Config Overrides: Target environment's odoo.conf customizations

Only database and filestore content are restored from backup.

Restore to New Environment

Use Case

• Create clone of production for isolated testing
• Spin up environment from backup on different server
• Disaster recovery with fresh infrastructure
• Migration to new server hardware

API Endpoint (v2)

POST /api/v1/backups/restore/v2
Content-Type: application/json
Authorization: Bearer <token>
 
{
  "backup_id": "uuid-of-backup",
  "target": {
    "mode": "new",
    "new_environment": {
      "name": "Production Clone (Jan 2025)",
      "environment_type": "staging",
      "server_id": "uuid-of-target-server"
    }
  },
  "create_pre_restore_backup": false,
  "include_filestore": true
}

New Environment Creation

The restore service creates a complete new environment:

new_env = ProjectEnvironment(
    id=uuid4(),
    project_id=source_environment.project_id,
    name="Restored Environment Name",
    environment_type=EnvironmentType.STAGING,
    vm_id=target_server_id,
    # Copy resource settings from source
    cpu_cores=source_environment.cpu_cores,
    ram_mb=source_environment.ram_mb,
    disk_gb=source_environment.disk_gb,
    status="pending",  # Activated after restore completes
    created_by=user_id,
)

Infrastructure Deployment

Before restoring data, the system deploys complete container infrastructure:

docker network create paasportal_net_{env_uuid}
docker network create traefik-public  # If not exists

docker run -d \
  --name {env_uuid}_db \
  --network paasportal_net_{env_uuid} \
  --restart unless-stopped \
  -e POSTGRES_USER=odoo \
  -e POSTGRES_PASSWORD={generated_password} \
  -e POSTGRES_DB={env_uuid} \
  -v paasportal_pgdata_{env_uuid}:/var/lib/postgresql/data \
  postgres:15-alpine

[options]
db_host = {env_uuid}_db
db_port = 5432
db_user = odoo
db_password = {generated_password}
db_name = {env_uuid}
dbfilter = ^{env_uuid}$
data_dir = /var/lib/odoo
addons_path = /mnt/extra-addons,/usr/lib/python3/dist-packages/odoo/addons
http_interface = 0.0.0.0
http_port = 8069
proxy_mode = True

docker run -d \
  --name {env_uuid}_odoo \
  --network traefik-public \
  --restart unless-stopped \
  -v {config_dir}/odoo.conf:/etc/odoo/odoo.conf:ro \
  -v {addons_dir}:/mnt/extra-addons:ro \
  -v paasportal_odoo_{env_uuid}_data:/var/lib/odoo \
  {traefik_labels} \
  {odoo_image}

Odoo Image Selection

The Odoo Docker image is determined from the source backup's environment snapshot:

odoo_image = "odoo:17.0"  # Default fallback
if backup.environment_snapshot:
    odoo_image = backup.environment_snapshot.get("odoo_image", odoo_image)
    # Also check nested structure
    if not odoo_image or odoo_image == "odoo:17.0":
        env_snapshot = backup.environment_snapshot.get("environment", {})
        odoo_image = env_snapshot.get("odoo_image", "odoo:17.0")

Cleanup on Failure

If restore fails, the system automatically cleans up resources:

• Stop and remove Odoo container
• Stop and remove PostgreSQL container
• Remove Docker volumes (pgdata and Odoo data)
• Remove Docker network
• Remove configuration directory: /opt/paasportal/{env_uuid}
• Delete environment record from database

This prevents orphaned resources and quota consumption.

Cross-Server Restore

Use Case

• Migrate environment to server with more resources
• Balance load across multiple servers
• Disaster recovery to different datacenter/region
• Server decommissioning with environment migration

API Endpoint (v2)

POST /api/v1/backups/restore/v2
Content-Type: application/json
Authorization: Bearer <token>
 
{
  "backup_id": "uuid-of-backup",
  "target": {
    "mode": "cross_server",
    "new_environment": {
      "name": "Migrated Production",
      "environment_type": "production",
      "server_id": "uuid-of-different-server"
    }
  },
  "create_pre_restore_backup": false,
  "include_filestore": true
}

Target Server Validation

The system validates target server before initiating restore:

# Get target server
target_vm = await db.get(VM, target_server_id)
if not target_vm:
    raise RestoreError("Target server not found")
 
# Validate same organization
if target_vm.organization_id != source_org_id:
    raise RestoreError(
        "Cross-organization restore not supported. "
        "Target server must belong to same organization."
    )
 
# Validate server status
if vm_status not in ['online', 'running', 'active']:
    logger.warning(f"Target server status is {vm_status}, may not be available")

Resource Availability Check

The system calculates available resources on target server:

# Calculate used resources (sum of all environments on server)
used_cpu = sum(env.cpu_cores or 0 for env in vm_environments)
used_ram_mb = sum(env.ram_mb or 0 for env in vm_environments)
used_disk_gb = sum(env.disk_gb or 0 for env in vm_environments)
 
# Calculate available resources
total_cpu = vm.cpu_cores or 0
total_ram_mb = (vm.memory_gb or 0) * 1024  # Convert GB to MB
total_disk_gb = vm.disk_gb or 0
 
available_cpu = max(0, total_cpu - used_cpu)
available_ram_mb = max(0, total_ram_mb - used_ram_mb)
available_disk_gb = max(0, total_disk_gb - used_disk_gb)
 
# Check if sufficient resources for restore
required_cpu = source_env.cpu_cores or 2
required_ram_mb = source_env.ram_mb or 2048
required_disk_gb = source_env.disk_gb or 10
 
if available_cpu < required_cpu:
    incompatibility_reason = "Insufficient CPU"

Compatible Servers List

Get list of servers available for cross-server restore:

GET /api/v1/backups/{backup_id}/compatible-servers
Authorization: Bearer <token>

Response:

{
  "servers": [
    {
      "id": "uuid-of-server",
      "name": "Production Server 2",
      "ip_address": "192.168.1.100",
      "available_cpu": 8,
      "available_ram_mb": 16384,
      "available_disk_gb": 500,
      "current_environments": 3,
      "recommended": true,
      "has_sufficient_resources": true,
      "incompatibility_reason": null
    },
    {
      "id": "uuid-of-server-2",
      "name": "Development Server 1",
      "ip_address": "192.168.1.101",
      "available_cpu": 2,
      "available_ram_mb": 2048,
      "available_disk_gb": 100,
      "current_environments": 8,
      "recommended": false,
      "has_sufficient_resources": false,
      "incompatibility_reason": "Insufficient CPU (available: 2, required: 4)"
    }
  ]
}

Network Considerations

Cross-server restore involves network transfer of backup data:

• Backup downloaded from storage provider to target server
• Not transferred between servers directly
• Storage provider bandwidth and latency affect restore speed
• Large backups (>10 GB) may take significant time to download

Selective Restoration

Database-Only Restore

Restore only the PostgreSQL database without touching filestore.

Use Case:

• Recover from database corruption
• Restore database schema/data after development changes
• Keep existing uploaded files intact

API Request:

{
  "backup_id": "uuid-of-backup",
  "include_filestore": false
}

Process:

1. Download backup and extract dump.sql only
2. Restore database via pg_restore
3. Skip filestore extraction and restoration
4. Existing filestore remains unchanged

Filestore-Only Restore

Restore only Odoo filestore without touching database.

Manual Process:

1. Download backup ZIP from storage
2. Extract filestore.tar.gz from ZIP package
3. Copy to server and extract into Odoo container:

   docker cp filestore.tar.gz {env_id}_odoo:/tmp/
   docker exec {env_id}_odoo tar -xzf /tmp/filestore.tar.gz -C /var/lib/odoo/filestore
   docker exec {env_id}_odoo chown -R odoo:odoo /var/lib/odoo/filestore/{db_name}

Restore Validation

Post-Restore Health Checks

After restore completes, the system performs comprehensive validation:

docker inspect --format='{{.State.Health.Status}}' {container_name}

-- Execute from PostgreSQL container
docker exec {db_container} psql -U odoo -d "db_name" -c "SELECT COUNT(*) FROM res_users;"

-- Verify Odoo modules are present
SELECT COUNT(*) FROM ir_module_module;

Data Integrity Verification

The restore process validates:

• Database Restore: Check for FATAL errors in pg_restore output
• Filestore Extraction: Verify tar archive extracted without errors
• File Permissions: Ensure odoo:odoo ownership of all filestore files
• Environment Snapshot Match: Compare restored database name with backup metadata

Automatic Rollback on Failure

If restore fails during any step:

1. Mark Restore Point as FAILED

   • Set status = RestoreStatus.FAILED
   • Record error_message and error_step
   • Calculate duration_seconds

2. Preserve Pre-Restore Backup (if created)

   • Safety backup remains in storage
   • Can be manually restored to recover original state

3. Clean Up Resources (new environment only)

   • Remove containers, volumes, networks
   • Delete environment record from database
   • Release quota allocation

4. Log Failure Details

   • Full error message and stack trace in logs
   • Deployment log entries for each step
   • SSE event to notify frontend of failure

Cross-Version Restoration

Odoo Version Compatibility

Restoring backups across different Odoo versions has compatibility considerations:

Migration Considerations

When restoring to a different Odoo version:

Validation Code:

if backup.environment_snapshot:
    backup_odoo_version = backup.environment_snapshot.get("odoo_version")
    if backup_odoo_version and target_env.odoo_version_id:
        logger.warning(
            f"Backup Odoo version ({backup_odoo_version}) may differ from "
            f"target environment configuration"
        )

Upgrade Path

To restore a backup to a newer Odoo version:

docker exec {env_id}_odoo odoo -d {db_name} -u all --stop-after-init

Schema Compatibility Issues

Common issues when restoring across versions:

• Missing Columns: New Odoo version expects columns not in backup database
• Removed Tables: Old tables in backup not used in new version
• Foreign Key Constraints: Schema changes may violate existing constraints
• Module Dependencies: Modules in backup may not be compatible with new version

Resolution: Use Odoo's built-in upgrade mechanism (-u all flag) after restore to migrate schema.

Restore from Migration (Sprint 2E7)

Migration Restore Feature

When environments are created from Odoo.sh migration backups, special safeguards prevent data loss on redeployment.

migration_restore_completed Flag

Purpose: Track whether migration backup has been restored to prevent accidental re-restore on redeploy.

Database Fields (on project_environments table):

• migration_id (UUID, FK to migrations.id): Reference to migration record
• migration_restore_completed (Boolean): Whether restore completed successfully
• migration_restore_completed_at (Timestamp): When restore completed

Restore Logic

First Deployment (after migration environment created):

if environment.migration_id and not environment.migration_restore_completed:
    # Download migration backup from R2
    # Restore database and filestore
    # Set migration_restore_completed = True
    logger.info("Migration backup restored successfully")

Subsequent Redeployments:

elif environment.migration_id and environment.migration_restore_completed:
    logger.info(
        f"Migration restore already completed at {environment.migration_restore_completed_at}, "
        "skipping restore to preserve data"
    )
    # Skip restore - use existing database

Data Loss Prevention

Example scenario:

1. Migrate environment from Odoo.sh → creates environment with migration_id
2. First deployment → restores migration backup, sets migration_restore_completed = True
3. User makes changes to database (creates sales orders, products, etc.)
4. Redeploy environment (e.g., git push or manual deploy)
5. ✅ Existing database preserved - no re-restore occurs

Force Re-Restore

To intentionally re-restore the migration backup (destroying current data):

API Request:

POST /api/v1/environments/{env_id}/deploy?force_restore=true
Authorization: Bearer <token>

Process:

if force_restore and environment.migration_id and environment.migration_restore_completed:
    logger.warning(
        f"force_restore=true: Clearing migration_restore_completed flag for environment {env_id}. "
        "This WILL destroy current database and re-restore migration backup!"
    )
    environment.migration_restore_completed = False
    environment.migration_restore_completed_at = None
    await db.commit()

Frontend Confirmation (required):

// User must type "RESTORE" to confirm
const [confirmText, setConfirmText] = useState("");
const handleForceRestore = async () => {
  if (confirmText !== "RESTORE") {
    alert("Type 'RESTORE' to confirm");
    return;
  }
  await api.post(`/environments/${envId}/deploy?force_restore=true`);
};

Migration Restore Steps

When restoring migration backup during deployment:

# Migration backup stored in Cloudflare R2
r2_key = config.migration_r2_key  # e.g., "migrations/uuid/backup.zip"
r2_client = boto3.client('s3', endpoint_url=CLOUDFLARE_R2_ENDPOINT, ...)
r2_client.download_file(bucket, r2_key, local_zip_path)

unzip backup.zip -d /tmp/migration_restore_{uuid}
# Extract contains:
#   - dump.sql (or dump.dump)
#   - filestore/ (directory or filestore.tar.gz)
#   - manifest.json (metadata)

# Copy dump to PostgreSQL container
docker cp /tmp/dump.sql {db_container}:/tmp/dump.sql
 
# Drop and recreate database
docker exec {db_container} psql -U odoo -d postgres -c "DROP DATABASE IF EXISTS \"{db_name}\";"
docker exec {db_container} psql -U odoo -d postgres -c "CREATE DATABASE \"{db_name}\" OWNER odoo;"
 
# Restore database
docker exec {db_container} psql -U odoo -d {db_name} -f /tmp/dump.sql
# OR if .dump format:
docker exec {db_container} pg_restore -U odoo -d {db_name} /tmp/dump.sql

# Find filestore directory or archive
filestore_dir=$(find /tmp/migration_restore_{uuid} -type d -name 'filestore' | head -1)
# If archive, extract first
if [ -f "/tmp/filestore.tar.gz" ]; then
  tar -xzf /tmp/filestore.tar.gz
fi
 
# Copy to Odoo container
docker exec --user root {odoo_container} mkdir -p /var/lib/odoo/filestore/{db_name}
docker cp $filestore_dir/. {odoo_container}:/var/lib/odoo/filestore/{db_name}/
docker exec --user root {odoo_container} chown -R odoo:odoo /var/lib/odoo/filestore/{db_name}

-- Check database has Odoo tables
docker exec {db_container} psql -U odoo -d {db_name} -c "SELECT COUNT(*) FROM ir_module_module"

# Update migration status
migration.status = MigrationStatus.COMPLETED
migration.completed_at = datetime.now()
 
# Set restore completed flag
environment.migration_restore_completed = True
environment.migration_restore_completed_at = datetime.now()
await db.commit()

docker restart {env_id}_odoo
# Container restarts with restored database and filestore

Restore Performance

Estimated Restoration Time

Restore duration depends on multiple factors:

Performance Benchmarks

Test Environment: 4 CPU, 8GB RAM, 100Mbps network, S3 storage (us-east-1)

Large Database Optimization

For databases >10 GB, consider:

1. Parallel Restore (not currently supported, future enhancement)

   • Use pg_restore --jobs=4 for parallel restore threads
   • Requires custom format dump (-Fc flag in pg_dump)

2. Storage Provider Selection

   • Use MinIO on same datacenter for fastest transfers
   • Cloudflare R2 with regional buckets for distributed teams
   • Avoid FTP/SFTP for large backups (slower than S3-compatible)

3. Network Optimization

   • Upgrade server network bandwidth (1Gbps or 10Gbps)
   • Use storage provider with CDN acceleration

4. Compression Level

   • Backups use gzip compression for filestore (level 6 by default)
   • Higher compression saves storage but increases CPU time during restore

Progress Tracking via SSE

Monitor restore progress in real-time using Server-Sent Events:

SSE Endpoint: GET /api/v1/events/stream

Event Stream:

event: restore.started
data: {"restore_id": "uuid", "backup_id": "uuid", "environment_id": "uuid"}

event: restore.progress
data: {"restore_id": "uuid", "status": "downloading", "progress": 25}

event: restore.progress
data: {"restore_id": "uuid", "status": "restoring_db", "progress": 50}

event: restore.progress
data: {"restore_id": "uuid", "status": "restoring_files", "progress": 75}

event: restore.completed
data: {"restore_id": "uuid", "duration_seconds": 180}

Frontend Integration:

const useRestoreProgress = (restoreId: string) => {
  const [status, setStatus] = useState<RestoreStatus>("pending");
  const [progress, setProgress] = useState(0);
 
  useEffect(() => {
    const eventSource = new EventSource(`/api/v1/events/stream`);
 
    eventSource.addEventListener("restore.progress", (event) => {
      const data = JSON.parse(event.data);
      if (data.restore_id === restoreId) {
        setStatus(data.status);
        setProgress(data.progress);
      }
    });
 
    return () => eventSource.close();
  }, [restoreId]);
 
  return { status, progress };
};

Disaster Recovery

Complete Environment Rebuild

Recover entire environment from backup after catastrophic failure (server crash, data center outage, etc.).

Disaster Recovery Procedure

{
  "backup_id": "uuid-of-latest-backup",
  "target": {
    "mode": "cross_server",
    "new_environment": {
      "name": "Production (Recovered)",
      "environment_type": "production",
      "server_id": "uuid-of-new-server"
    }
  },
  "include_filestore": true
}

Multi-Environment Restoration

Restore multiple environments after widespread failure:

Strategy: Prioritize environments by business criticality

1. Production First: Restore revenue-critical production environments
2. Staging Second: Restore staging for testing and QA
3. Development Last: Restore dev environments (or recreate from git)

Parallel Restores: The system supports multiple concurrent restores (limited by server resources and ARQ worker capacity).

Recovery Time Objective (RTO)

Expected recovery time based on disaster scope:

Backup Download

Manual Backup Download

Download backup files for external restore, archival, or transfer to another system.

API Endpoint

GET /api/v1/backups/{backup_id}/download
Authorization: Bearer <token>

Response:

{
  "backup_id": "uuid-of-backup",
  "expires_in": 3600,
  "urls": {
    "database": "https://s3.amazonaws.com/bucket/path/database.dump?signature=...",
    "filestore": "https://s3.amazonaws.com/bucket/path/filestore.tar.gz?signature=...",
    "zip": "https://r2.cloudflarestorage.com/bucket/path/backup.zip?signature=..."
  }
}

Pre-Signed URL Generation

The system generates temporary signed URLs for secure download:

S3-Compatible Providers (AWS S3, R2, B2, MinIO):

provider = get_storage_provider(config.provider, **config.to_provider_config())
urls = await provider.generate_download_urls(backup_id, expires_in=3600)
# URLs valid for 1 hour (3600 seconds)

FTP/SFTP Providers:

• Direct download not supported (no URL-based access)
• API returns error: "Direct download not supported for FTP/SFTP"
• Use sftp or ftp client to retrieve files directly from server

Download Workflow

curl -X GET "https://portal.oec.sh/api/v1/backups/{backup_id}/download" \
  -H "Authorization: Bearer {token}"

# Download ZIP (contains both database and filestore)
curl -o backup.zip "{zip_url}"
 
# OR download separately
curl -o database.dump "{database_url}"
curl -o filestore.tar.gz "{filestore_url}"

# Extract ZIP
unzip backup.zip
# Contains:
#   - dump.sql
#   - filestore.tar.gz
#   - manifest.json
 
# Verify integrity
md5sum dump.sql filestore.tar.gz
# Compare checksums with manifest.json

# Restore database
psql -U odoo -d target_database -f dump.sql
 
# Restore filestore
tar -xzf filestore.tar.gz -C /var/lib/odoo/filestore/
chown -R odoo:odoo /var/lib/odoo/filestore/{db_name}

Download Permissions

Required Permission: project.backups.view or org.backups.list

Users must have backup view permission to download backup files. This prevents unauthorized access to sensitive production data.

Permissions

Required Permissions for Restore Operations

The platform uses granular RBAC permissions for restore operations:

Production Environment Restrictions

Recommended Role Configuration:

Approval Workflows for Critical Restores

For production environment restores, implement manual approval workflow:

Permission Validation in API

The API enforces permission checks at multiple levels:

# Check permission to restore backups
has_permission = await check_permission(
    db=db,
    user=current_user,
    permission_code="project.backups.restore",
    organization_id=backup.environment.project.organization_id,
)
if not has_permission:
    raise HTTPException(
        status_code=403,
        detail="You don't have permission to restore backups. "
               "Required permission: project.backups.restore"
    )
 
# If restoring to different environment, verify access there too
if target_environment_id != backup.environment_id:
    target_has_permission = await check_permission(
        db=db,
        user=current_user,
        permission_code="project.backups.restore",
        organization_id=target_env.project.organization_id,
    )
    if not target_has_permission:
        raise HTTPException(
            status_code=403,
            detail="You don't have permission to restore to target environment"
        )

API Reference

Restore Endpoints

POST /api/v1/backups/restore

Restore backup to original or existing environment (simple mode).

Request Body:

{
  "backup_id": "uuid",
  "target_environment_id": "uuid",  // Optional, defaults to backup's environment
  "create_pre_restore_backup": true,  // Default: true
  "include_filestore": true  // Default: true
}

Response: RestorePointResponse

{
  "id": "uuid",
  "backup_id": "uuid",
  "target_environment_id": "uuid",
  "status": "pending",
  "task_id": "uuid",
  "pre_restore_backup_id": null,
  "is_new_environment": false,
  "is_cross_server": false,
  "started_at": null,
  "completed_at": null,
  "duration_seconds": null,
  "error_message": null,
  "error_step": null,
  "create_date": "2025-01-15T10:30:00Z",
  "created_by": "uuid"
}

Status Codes:

• 201 Created: Restore queued successfully
• 400 Bad Request: Backup not in COMPLETED status
• 403 Forbidden: Missing restore permission
• 404 Not Found: Backup or target environment not found
• 500 Internal Server Error: Failed to queue restore

POST /api/v1/backups/restore/v2

Restore with advanced options (new environment, cross-server).

Request Body:

{
  "backup_id": "uuid",
  "target": {
    "mode": "new",  // "original", "existing", "new", "cross_server"
    "environment_id": "uuid",  // Required for "existing" mode
    "new_environment": {  // Required for "new" or "cross_server" mode
      "name": "Restored Environment",
      "environment_type": "staging",  // "development", "staging", "production"
      "server_id": "uuid"  // Required for cross_server
    }
  },
  "create_pre_restore_backup": true,
  "include_filestore": true
}

Response: RestoreResponseV2

{
  "restore_point": {
    "id": "uuid",
    "backup_id": "uuid",
    "target_environment_id": "uuid",
    "status": "pending",
    "is_new_environment": true,
    "is_cross_server": false,
    "target_vm_id": "uuid",
    ...
  },
  "task": {
    "id": "uuid",
    "task_type": "restore",
    "status": "queued",
    "arq_job_id": "job-uuid",
    ...
  },
  "target_environment": {
    "id": "uuid",
    "name": "Restored Environment",
    "environment_type": "staging",
    "status": "pending",
    ...
  }
}

GET /api/v1/backups/{backup_id}/compatible-environments

Get environments compatible with backup for restore.

Response:

{
  "environments": [
    {
      "id": "uuid",
      "name": "Staging",
      "environment_type": "staging",
      "status": "running",
      "server_name": "Production Server 1",
      "odoo_version": "17.0"
    }
  ]
}

GET /api/v1/backups/{backup_id}/compatible-servers

Get servers compatible for cross-server restore.

Response:

{
  "servers": [
    {
      "id": "uuid",
      "name": "Production Server 2",
      "ip_address": "192.168.1.100",
      "available_cpu": 8,
      "available_ram_mb": 16384,
      "available_disk_gb": 500,
      "current_environments": 3,
      "recommended": true,
      "has_sufficient_resources": true,
      "incompatibility_reason": null
    }
  ]
}

GET /api/v1/restore/{restore_id}

Get restore point details by ID.

Response: RestorePointResponse

GET /api/v1/environments/{environment_id}/restore-points

List restore points for an environment.

Query Parameters:

• status: Filter by restore status (optional)
• page: Page number (default: 1)
• page_size: Items per page (default: 20, max: 100)

Response:

[
  {
    "id": "uuid",
    "backup_id": "uuid",
    "target_environment_id": "uuid",
    "status": "completed",
    "duration_seconds": 180,
    ...
  }
]

POST /api/v1/restore/{restore_id}/cancel

Cancel a pending or in-progress restore.

Response: RestorePointResponse with status: "cancelled"

Status Codes:

• 200 OK: Restore cancelled
• 400 Bad Request: Restore cannot be cancelled (already completed/failed)
• 403 Forbidden: Missing restore permission
• 404 Not Found: Restore point not found

GET /api/v1/backups/{backup_id}/download

Get pre-signed download URLs for backup files.

Query Parameters:

• expires_in: URL expiration time in seconds (default: 3600, max: 86400)

Response: BackupDownloadResponse

{
  "backup_id": "uuid",
  "expires_in": 3600,
  "urls": {
    "database": "https://...",
    "filestore": "https://...",
    "zip": "https://..."
  }
}

Restore Options and Flags

Best Practices

Testing Restore Procedures

Restore Testing Schedule:

Test Restore Checklist:

• Download latest backup
• Restore to staging environment
• Verify database connection
• Test user login
• Check critical business flows (sales orders, invoicing, etc.)
• Verify filestore attachments load correctly
• Review restore logs for warnings/errors
• Document restore duration and any issues

Restore to Staging Before Production

Never restore directly to production without testing on staging first.

Workflow:

{
  "backup_id": "uuid-of-production-backup",
  "target": {
    "mode": "existing",
    "environment_id": "uuid-of-staging-env"
  }
}

{
  "backup_id": "uuid-of-production-backup",
  "create_pre_restore_backup": true  // CRITICAL - enables rollback
}

Backup Verification Before Critical Operations

Before any risky operation (Odoo upgrade, major module installation, schema changes), verify backup:

Pre-Operation Checklist:

• Latest automatic backup completed successfully (check status)
• Backup size reasonable (not suspiciously small)
• Test restore to staging and verify (for critical operations)
• Create manual on-demand backup immediately before operation
• Document backup IDs for quick rollback reference

Example:

# Before Odoo 17 → 18 upgrade:
 
# 1. Create manual backup
curl -X POST "https://portal.oec.sh/api/v1/environments/{env_id}/backups" \
  -H "Authorization: Bearer {token}" \
  -d '{"backup_type": "manual", "description": "Pre-upgrade safety backup"}'
 
# 2. Wait for backup to complete (monitor SSE events)
 
# 3. Test restore to staging
curl -X POST "https://portal.oec.sh/api/v1/backups/restore" \
  -d '{"backup_id": "{backup_id}", "target_environment_id": "{staging_id}"}'
 
# 4. Verify staging works with Odoo 18
 
# 5. Proceed with production upgrade (with backup ID documented for rollback)

Disaster Recovery Preparedness

Quarterly Disaster Recovery Drill:

Simulate complete disaster scenario and measure recovery:

Example Recovery Metrics (target vs actual):

Troubleshooting

Restore Failures

Symptom: Restore stuck in "downloading" status

Possible Causes:

• Storage provider connection issues (network timeout, auth failure)
• Backup file corrupted or missing in storage
• ARQ worker not processing tasks

Resolution:

1. Check storage provider connectivity:

   # Test storage connection
   curl -X POST "/api/v1/backups/storage/test" \
     -d '{"provider": "aws_s3", "access_key": "...", "bucket": "..."}'

2. Verify backup file exists in storage:

   # List files in storage bucket
   aws s3 ls s3://bucket-name/backups/{org_id}/{env_id}/

3. Check ARQ worker logs:

   docker logs paasportal_worker
   # Look for "execute_restore" task errors

4. Cancel and retry restore:

   curl -X POST "/api/v1/restore/{restore_id}/cancel"
   # Wait for cancellation, then retry

Symptom: "pg_restore failed" error

Possible Causes:

• PostgreSQL version mismatch (backup from newer version)
• Corrupted backup file
• Insufficient disk space on server
• Database already exists with conflicting data

Resolution:

1. Check PostgreSQL versions:

   # Source backup version (from environment_snapshot)
   curl "/api/v1/backups/{backup_id}" | jq '.environment_snapshot.postgres_version'
    
   # Target environment version
   docker exec {env_id}_db psql -V

2. Verify disk space on server:

   ssh root@{server_ip} df -h
   # Ensure sufficient space in /var/lib/docker and /tmp

3. Manually verify backup integrity:

   # Download backup
   curl -o backup.zip "{download_url}"
    
   # Extract and test
   unzip backup.zip
   pg_restore --list dump.sql
   # Should list database objects without errors

4. If version mismatch, restore to environment with matching PostgreSQL version first, then upgrade.

Symptom: Filestore restore failed, files missing

Possible Causes:

• Backup was database-only (no filestore)
• Filestore archive corrupted
• Insufficient permissions in Odoo container

Resolution:

1. Check if backup includes filestore:

   curl "/api/v1/backups/{backup_id}" | jq '.filestore_size_bytes'
   # If null or 0, backup has no filestore

2. Manually extract and verify filestore:

   unzip backup.zip
   tar -tzf filestore.tar.gz | head -20
   # Should list filestore directory structure

3. Check container permissions:

   docker exec {env_id}_odoo ls -la /var/lib/odoo/filestore/{db_name}
   # Should be owned by odoo:odoo

4. Manually fix permissions if needed:

   docker exec --user root {env_id}_odoo chown -R odoo:odoo /var/lib/odoo/filestore/{db_name}

Database Compatibility Issues

Symptom: Database restored but Odoo won't start

Possible Causes:

• Odoo version incompatible with database schema
• Missing Python dependencies for custom modules
• Database connection configuration incorrect

Resolution:

1. Check Odoo logs for specific error:

   docker logs {env_id}_odoo
   # Look for error messages about missing columns, modules, or dependencies

2. Verify database connection from Odoo:

   docker exec {env_id}_odoo odoo shell -d {db_name} --stop-after-init
   # Should connect without errors

3. If version mismatch, run Odoo upgrade:

   docker exec {env_id}_odoo odoo -d {db_name} -u all --stop-after-init
   # Updates database schema to match Odoo version

4. If missing dependencies, install them:

   docker exec {env_id}_odoo pip install -r /mnt/extra-addons/requirements.txt
   docker restart {env_id}_odoo

Symptom: Cross-version restore causes "column does not exist" errors

Cause: Database schema from older Odoo version missing columns expected by newer version.

Resolution:

1. Do NOT attempt to fix manually - Odoo upgrade mechanism handles this.

2. Restore to environment with same Odoo version as backup first:

   {
     "backup_id": "uuid",
     "target": {
       "mode": "new",
       "new_environment": {
         "name": "Restore Staging (v17)",
         "environment_type": "staging",
         "server_id": "uuid"
       }
     }
   }

   Ensure new environment uses same Odoo version (e.g., 17.0).

3. Once restored successfully, upgrade the environment to target version:

   # Update environment to use Odoo 18 image
   # Then redeploy with upgrade flag
   docker exec {env_id}_odoo odoo -d {db_name} -u all --stop-after-init

4. Verify upgrade succeeded:

   # Check Odoo version
   docker exec {env_id}_odoo odoo --version
    
   # Test application
   curl https://{domain}

5. If upgrade succeeds on staging, repeat for production.

Insufficient Disk Space

Symptom: "No space left on device" during restore

Possible Causes:

• Server disk full
• Docker volume space exhausted
• Temporary directory /tmp full during restore

Resolution:

1. Check disk usage on server:

   ssh root@{server_ip} df -h
   # Identify which mount point is full

2. Clean up Docker resources:

   # Remove unused containers
   docker container prune -f
    
   # Remove unused images
   docker image prune -a -f
    
   # Remove unused volumes
   docker volume prune -f

3. Clean up temporary files:

   rm -rf /tmp/restore_*
   rm -rf /tmp/backup_*

4. Free up space in Docker volumes:

   # Check volume usage
   docker system df -v
    
   # Remove old backups from environment
   curl -X DELETE "/api/v1/backups/{old_backup_id}"

5. If server consistently out of space:

   • Upgrade server disk size (add more storage)
   • Enable backup rotation (delete old backups automatically)
   • Use external storage (NFS/S3 for filestore)

Corrupt Backup Files

Symptom: "Backup file corrupted" or "ZIP extraction failed"

Possible Causes:

• Incomplete backup upload (network interruption)
• Storage provider data corruption
• Backup created with errors but marked as completed

Resolution:

1. Download backup and verify integrity:

   curl -o backup.zip "{download_url}"
    
   # Test ZIP integrity
   unzip -t backup.zip
   # Should report "No errors detected"

2. If ZIP corrupted, check backup status:

   curl "/api/v1/backups/{backup_id}" | jq '.status, .error_message'

3. Try previous backup:

   # List recent backups
   curl "/api/v1/environments/{env_id}/backups?page_size=10"
    
   # Restore from earlier backup
   curl -X POST "/api/v1/backups/restore" \
     -d '{"backup_id": "{previous_backup_id}"}'

4. If all recent backups corrupted, investigate storage provider:

   # Test storage connection
   curl -X POST "/api/v1/backups/storage/test" -d '{...}'
    
   # Check storage provider status (S3, R2, etc.)
   # May be regional outage or authentication issue

5. Prevention: Enable backup verification after creation:

   • Download backup after creation
   • Verify ZIP integrity
   • Test restore to dev environment monthly

Related Documentation

• Backup Creation & Scheduling - Creating manual and automatic backups
• Backup Retention & Lifecycle - GFS retention policies and lifecycle management
• Storage Providers - Configuring S3, R2, B2, MinIO, FTP, SFTP storage
• Disaster Recovery Planning - Comprehensive disaster recovery strategies
• Environment Management - Creating and managing Odoo environments
• Deployment Guide - Deploying Odoo environments to servers
• Permissions & RBAC - Configuring role-based access control
• ARQ Task Queue - Background task processing architecture

Summary

OEC.SH provides enterprise-grade backup restoration capabilities with:

• Flexible Restore Targets: Original, existing, new, and cross-server restore modes
• Pre-Restore Safety: Automatic backup before restore for rollback capability
• Selective Restoration: Database-only or full restore (database + filestore)
• Migration Safeguards: Protection against accidental data loss during redeployment
• Real-Time Monitoring: Track restore progress via SSE events
• Disaster Recovery: Complete environment rebuild from backup
• Permission Controls: Granular RBAC for restore operations
• Performance Optimized: Efficient restore process with progress tracking

Regular restore testing, proper permission configuration, and disaster recovery preparedness ensure your Odoo environments can recover quickly from any failure scenario.