django-elasticsearch-dsl

repository·master·Indexed 21 days ago

https://github.com/django-es/django-elasticsearch-dsl

A Django integration for Elasticsearch DSL that allows developers to define Elasticsearch document mappings and search queries using Django models. It provides tools for managing indices via the search_index management command, support for complex field mappings using ObjectField and NestedField, and utilities for data transformation via prepare_field methods.

Tokens
7.6K
Snippets
30
Records
36
Agent score
74%

What's inside django-elasticsearch-dsl

  1. Overview of Django Elasticsearch DSL

    master

    Django Elasticsearch DSL is a package designed to index Django models in Elasticsearch. It acts as a thin wrapper around elasticsearch-py, allowing developers to leverage the full feature set of the official Elasticsearch Python client while integrating seamlessly with Django.

    Key features include:

    • Search Capabilities: Uses the Search class from elasticsearch-dsl for querying.
    • Automatic Synchronization: Uses Django signal receivers on save and delete to keep Elasticsearch indices in sync with your database.
    • Management Commands: Provides CLI commands for creating, deleting, rebuilding, and populating indices.
    • Auto-mapping: Automatically maps Django model fields to Elasticsearch mappings.
    • Complex Types: Supports complex field types like ObjectField and NestedField.
    • Performance: Supports parallel indexing for faster operations.
  2. Check Elasticsearch and Library Compatibility

    master

    The library is compatible with Elasticsearch versions 5.x and later. However, you must use the major version of django-elasticsearch-dsl that matches your Elasticsearch major version.

    Ensure your elasticsearch or elasticsearch-dsl dependency requirements match your target Elasticsearch version as shown below.

    # Elasticsearch 9.x
    elasticsearch>=9.0.0,<10.0.0
    
    # Elasticsearch 8.x
    elasticsearch-dsl>=8.0.0,<9.0.0
    
    # Elasticsearch 7.x
    elasticsearch-dsl>=7.0.0,<8.0.0
    
    # Elasticsearch 6.x
    elasticsearch-dsl>=6.0.0,<7.0.0
  3. Initialize the test app database and data

    master

    Follow these steps to prepare the application for testing:

    1. Run migrations: Create the necessary database tables.
    2. Create a superuser: Set up an admin account.
    3. Populate data: Use django-autofixture to populate models with fake data for testing purposes.
    4. Build the index: Rebuild the Elasticsearch index to include the newly created data.
    5. Start the server: Run the Django development server.
    # Run migrations
    $ ./manage.py migrate
    
    # Create superuser
    $ ./manage.py createsuperuser
    
    # Populate models with fake data (requires django-autofixture)
    $ ./manage.py loadtestdata test_app.Manufacturer:10 test_app.Car:100 test_app.Ad:500
    
    # Build the Elasticsearch index
    $ ./manage.py search_index --rebuild
    
    # Run the server
    $ ./manage.py runserver
  4. Pull Request Guidelines

    master

    When submitting a pull request for django-elasticsearch-dsl, ensure the following requirements are met:

    • Testing: Every pull request must include tests.
    • Documentation: If adding new functionality, update the documentation. New functionality should be encapsulated in a function with a docstring, and the feature should be added to the list in README.rst.
    • Compatibility: The code must work for Python 2.7, Python 3.4, and PyPy. Verify that tests pass for all supported Python versions by checking the GitHub Actions status.
  5. Run tests for django-elasticsearch-dsl

    master

    To run the test suite, first create a Python virtual environment and install the necessary dependencies using requirements_test.txt:

    pip install -r requirements_test.txt

    Then, execute the tests using the runtests.py script.

    For standard unit tests:

    python runtests.py

    For integration testing that requires a live Elasticsearch server, use the --elasticsearch flag to specify the server address:

    python runtests.py --elasticsearch [localhost:9200]
    python runtests.py --elasticsearch [localhost:9200]
  6. Install and configure django-elasticsearch-dsl

    master

    To get started, install the package via pip and add django_elasticsearch_dsl to your Django INSTALLED_APPS.

    You must define the ELASTICSEARCH_DSL dictionary in your Django settings to specify connection details like hosts and authentication. This configuration is passed to elasticsearch-dsl-py.connections.configure.

    pip install django-elasticsearch-dsl
    # settings.py
    
    INSTALLED_APPS = [
        # ...
        'django_elasticsearch_dsl',
    ]
    
    ELASTICSEARCH_DSL = {
        'default': {
            'hosts': 'localhost:9200',
            'http_auth': ('username', 'password')
        }
    }
  7. How to report bugs and propose features

    master

    Feedback and issues should be managed via GitHub issues at https://github.com/sabricot/django-elasticsearch-dsl/issues.

    Reporting a Bug: Include your operating system name/version, details about your local setup, and detailed steps to reproduce the bug.

    Proposing a Feature:

    • Explain the proposed functionality in detail.
    • Keep the scope as narrow as possible to facilitate implementation.
  8. Handle relationships with `ObjectField` and `NestedField`

    master

    To index related models (e.g., via ForeignKey), use ObjectField or NestedField.

    • ObjectField: Used for standard object relationships.
    • NestedField: Used when you need to maintain the relationship between sub-documents in Elasticsearch.

    When using these, you should:

    1. Define the properties dictionary mapping sub-field names to field instances.
    2. Add the related models to the related_models list in the Django inner class to ensure the document is re-indexed when the related object changes.
    3. Implement get_instances_from_related(self, related_instance) to tell the library how to find the parent document(s) from the related object.
    4. (Optional) Override get_queryset() to use select_related for better performance.
    @registry.register_document
    class CarDocument(Document):
        manufacturer = fields.ObjectField(properties={
            'name': fields.TextField(),
            'country_code': fields.TextField(),
        })
        ads = fields.NestedField(properties={
            'description': fields.TextField(),
            'title': fields.TextField(),
            'pk': fields.IntegerField(),
        })
    
        class Django:
            model = Car
            fields = ['name', 'color']
            related_models = [Manufacturer, Ad]
    
        def get_instances_from_related(self, related_instance):
            if isinstance(related_instance, Manufacturer):
                return related_instance.car_set.all()
            elif isinstance(related_instance, Ad):
                return related_instance.car
  9. Set up django-elasticsearch-dsl for local development

    master

    To contribute to the project, follow these steps to set up a local development environment:

    1. Fork and Clone: Fork the repository on GitHub and clone your fork locally.
    2. Virtual Environment: Create a virtual environment and install your local copy in development mode using setup.py develop.
    3. Branching: Create a new branch for your specific bugfix or feature.
    4. Verification: Before submitting, ensure your changes pass linting with flake8, pass the local test suite, and pass compatibility tests via tox.
    5. Submission: Commit your changes, push to your GitHub fork, and submit a pull request.
    # 1. Clone your fork
    $ git clone git@github.com:your_name_here/django-elasticsearch-dsl.git
    
    # 2. Set up virtualenv and install in development mode
    $ mkvirtualenv django-elasticsearch-dsl
    $ cd django-elasticsearch-dsl/
    $ python setup.py develop
    
    # 3. Create a development branch
    $ git checkout -b name-of-your-bugfix-or-feature
    
    # 4. Run linting and tests
    $ flake8 django_elasticsearch_dsl tests
    $ python setup.py test
    $ tox
    
    # 5. Commit and push
    $ git add .
    $ git commit -m "Your detailed description of your changes."
    $ git push origin name-of-your-bugfix-or-feature
  10. Define an Elasticsearch index using the Index class

    master

    While using class Index inside a Document class is the standard approach, you can also define an index by instantiating elasticsearch.dsl.Index directly.

    To use this method:

    1. Instantiate Index with the desired name.
    2. Configure settings using .settings().
    3. Use the @car.document decorator on your Document class to associate it with that index.
    4. Use the @registry.register_document decorator to register the document with django-elasticsearch-dsl.

    Alternatively, you can define index settings and names within an inner class Index inside your Document class.

    # Method 1: Direct Index instantiation
    from elasticsearch.dsl import Index
    from django_elasticsearch_dsl import Document, registry
    from .models import Car
    
    car_index = Index('cars')
    car_index.settings(number_of_shards=1, number_of_replicas=0)
    
    @registry.register_document
    @car_index.document
    class CarDocument(Document):
        class Django:
            model = Car
            fields = ['name', 'color']
    
    # Method 2: Inner Index class
    @registry.register_document
    class ManufacturerDocument(Document):
        class Index:
            name = 'manufacture'
            settings = {'number_of_shards': 1, 'number_of_replicas': 0}
    
        class Django:
            model = Manufacturer
            fields = ['name', 'country_code']
  11. Declare data to index using Document classes

    master

    To index a Django model, create a subclass of django_elasticsearch_dsl.Document in a documents.py file within your app directory. Register this class using the @registry.register_document decorator.

    Inside the Document class, you define two inner classes:

    1. class Index: Defines Elasticsearch-specific settings like the index name and settings (shards, replicas, etc.).
    2. class Django: Links the document to a specific Django model and defines which fields to index. It also allows for advanced configuration like signal handling, refresh policies, and queryset pagination.
    # documents.py
    
    from django_elasticsearch_dsl import Document
    from django_elasticsearch_dsl.registries import registry
    from .models import Car
    
    @registry.register_document
    class CarDocument(Document):
        class Index:
            name = 'cars'
            settings = {'number_of_shards': 1, 'number_of_replicas': 0}
    
        class Django:
            model = Car
            fields = ['name', 'color', 'description', 'type']
            # ignore_signals = True
            # auto_refresh = False
            # queryset_pagination = 5000