django-dbbackup

repository·master·Indexed 22 days ago

https://github.com/archmonger/django-dbbackup

A Django application providing management commands to backup and restore project databases and media files. It utilizes native database tools and supports various storage backends such as Amazon S3, Dropbox, and local storage. Key features include GPG encryption, compression, remote archiving, and backup retention cleanup. It includes commands such as dbbackup, dbrestore, mediabackup, mediarestore, and listbackups.

Tokens
15.3K
Snippets
53
Records
68
Agent score
77%

What's inside django-dbbackup

  1. Overview of Django DBBackup

    master

    Django DBBackup is a Django application that provides management commands for backing up and restoring your project's database and media files. It acts as a bridge between your Django project and various storage backends (such as Amazon S3, Dropbox, or local storage).

    Key capabilities include:

    • Native Backups: Uses your database's native/standard procedures instead of relying on Django's dumpdata/loaddata commands, making it more efficient.
    • Security: Supports GPG signatures and encryption for backups.
    • Archiving: Supports compression and remote archiving.
    • Automation: Can be integrated with Crontab or Celery for automated backup schedules.
    • Storage Versatility: Works with any Django-supported storage backend.
  2. Configure MySQL backup connector

    master

    The default connector for MySQL is MysqlDumpConnector (dbbackup.db.mysql.MysqlDumpConnector). It creates a SQL dump using mysqldump and restores it by piping the dump into mysql.

    Note on EXCLUDE: When using the EXCLUDE setting, provide plain table names. The connector will pass them to mysqldump using the --ignore-table=<database>.<table> flag.

  3. Configure SQLite backup connectors

    master

    DBBackup provides three distinct connectors for SQLite:

    1. SqliteBackupConnector (dbbackup.db.sqlite.SqliteBackupConnector): The default connector. It uses the SQLite .backup command, making it safe to run even with active connections.
    2. SqliteConnector (dbbackup.db.sqlite.SqliteConnector): A pure Python implementation that creates a SQL dump. It can be used to restore to a 'dirty' database, though this is generally not recommended.
    3. SqliteCPConnector (dbbackup.db.sqlite.SqliteCPConnector): Performs a raw file copy (snapshot). Warning: This is not suitable for databases with ongoing connections and does not support in-memory databases.

    Only the SqliteConnector supports the common EXCLUDE setting.

  4. Use the DjangoConnector for universal database support

    master

    The DjangoConnector (dbbackup.db.django.DjangoConnector) provides database-agnostic backup and restore functionality using Django's built-in dumpdata and loaddata commands. It is automatically used for any unmapped database engines.

    Use this connector when:

    • Using Oracle databases (default behavior).
    • Using custom or third-party database backends.
    • Simplicity is preferred in development environments.
    • External database tools are unavailable.

    Limitations:

    • Performance: Slower than native tools for large datasets.
    • Scope: Only backs up data; it does not back up database schema, indices, or procedures.

    Explicit Configuration:

    DBBACKUP_CONNECTORS = {
        'default': {
            'CONNECTOR': 'dbbackup.db.django.DjangoConnector',
        }
    }
  5. Configure MongoDB backup connector

    master

    The default connector for MongoDB is MongoDumpConnector (dbbackup.db.mongodb.MongoDumpConnector), which utilizes mongodump and mongorestore.

    Settings:

    • OBJECT_CHECK: Validate documents before inserting (Default: True).
    • DROP: Replace existing objects during restore (Default: True).
    • AUTH_SOURCE: For authenticated deployments, specify the database used to verify credentials.

    Example Configuration:

    DBBACKUP_CONNECTORS = {
        'default': {
            'AUTH_SOURCE': 'admin',
        }
    }
  6. Submit a patch to django-dbbackup

    master

    To propose changes, follow these steps:

    1. Fork the project and create a new branch.
    2. Implement changes with tests and documentation.
    3. Verify changes by running hatch test and hatch run functional:all.
    4. Ensure code quality by running hatch fmt.
    5. Push changes to your fork and verify via GitHub Actions.
    6. Open a pull request.

    Note: It is highly recommended to run hatch test --all to ensure compatibility across multiple Python and Django versions before pushing, as file operations can behave differently between versions.

  7. Schedule backups with Django-Cron

    master

    To run recurring backups using django-cron, create a class that inherits from CronJobBase. Inside the do method, use django.core.management.call_command to invoke the dbbackup command.

    import os
    from django.core import management
    from django.conf import settings
    from django_cron import CronJobBase, Schedule
    
    
    class Backup(CronJobBase):
        RUN_AT_TIMES = ['6:00', '18:00']
        schedule = Schedule(run_at_times=RUN_AT_TIMES)
        code = 'my_app.Backup'
    
        def do(self):
            management.call_command('dbbackup')
  8. Schedule backups with Django-Crontab

    master

    To use django-crontab for backups, define a wrapper function that calls the dbbackup and mediabackup management commands (optionally using the --clean flag to remove old backups) and then register this function in your settings.py CRONJOBS list. Finally, run the command to add the jobs to your system crontab.

    # In settings.py
    CRONTAB_COMMAND_SUFFIX = '2>&1'
    CRONJOBS = [
        ('0 5 * * *', 'core.backup.backup_job', '>> ' + os.path.join(CORE_DIR, 'backup/backup.log'))
    ]
    
    # In backup.py
    from datetime import datetime
    from django.core import management
    
    def backup_job():
        print("[{}] Backing up database and media files...".format(datetime.now()))
        management.call_command('dbbackup', '--clean')
        management.call_command('mediabackup', '--clean')
        print("[{}] Backup done!".format(datetime.now()))

    Add the cron job

    python manage.py crontab add
  9. Configure Google Cloud Storage

    master

    Use django-storages[google] to store backups in a Google Cloud bucket.

    Setup:

    1. Install dependencies: pip install django-storages[google]
    2. Configure STORAGES with the storages.backends.gcloud.GoogleCloudStorage backend.

    Required/Common Options:

    • bucket_name: The name of your GCS bucket.
    • project_id: Your Google Cloud project ID.
    • blob_chunk_size: Size of chunks for uploads (e.g., 1024 * 1024).
    STORAGES = {
        "dbbackup": {
            "BACKEND": "storages.backends.gcloud.GoogleCloudStorage",
            "OPTIONS": {
                "bucket_name": "your_bucket_name",
                "project_id": "your_project_id",
                "blob_chunk_size": 1024 * 1024,
            },
        },
    }
  10. Enable GPG encryption for backups

    master

    To protect sensitive data, you can encrypt backups using GPG.

    Requirements:

    1. Install python-gnupg: pip install python-gnupg>=0.5.0.
    2. Have a GPG key installed.
    3. On Windows, the gpg executable must be in your PATH.

    Configuration:

    • DBBACKUP_GPG_RECIPIENT: The name, key ID, fingerprint, or email of the GPG key to use. Required for encryption and automatic decryption.
    • DBBACKUP_GPG_ALWAYS_TRUST: Set to True to add --trust-model always to GPG commands, bypassing trust checks (use only in controlled environments).

    Usage:

    • To encrypt during backup: python manage.py dbbackup --encrypt
    • To decrypt during restore: python manage.py dbrestore --decrypt
    # Encrypting a backup
    python manage.py dbbackup --encrypt
    
    # Restoring from an encrypted backup
    python manage.py dbrestore --decrypt