Elixir Driver for MongoDB

repository·master·Indexed 42 days ago

https://github.com/zookzook/elixir-mongodb-driver

A performant, feature-rich Elixir interface for MongoDB supporting replica sets, transactions, streaming cursors, and change streams. Includes support for connection pooling via DBConnection, BSON encoding/decoding, and mapping documents to Elixir structs via Mongo.Collection and Mongo.Repo. Version 1.5.0 supports network compression using zlib and zstd.

Tokens
15.8K
Snippets
69
Records
77
Agent score
54%

What's inside mongodb_driver

  1. How Mongo.Stream works and how to use it

    master

    Most query functions return a %Mongo.Stream{} struct, which implements the Enumerable protocol. The driver uses this to manage MongoDB sessions automatically: it checks out a session, streams batches from the server, and checks the session back in once the stream is exhausted.

    Warning: Do not attempt to access the internal :docs field of the %Mongo.Stream{} struct directly. Doing so will prevent the session from being checked back in and will only retrieve the first batch of documents.

    Correct Usage: Always process the result using an Enum or Stream function to ensure proper session lifecycle management.

    # CORRECT: Use Enum to ensure session management
    @topology
    |> Mongo.aggregate(collection, pipeline, opts)
    |> Enum.to_list()
    
    # INCORRECT: Accessing internal docs directly leaves session open
    %Mongo.Stream{docs: docs} = Mongo.aggregate(@topology, collection, pipeline, opts)
    Enum.map(docs, fn elem -> elem end)
  2. Compare unpooled vs pooled MongoDB connections

    master

    The CrudExample project demonstrates two different ways to interact with MongoDB:

    1. CrudExample.example_1(): Uses a single, unpooled connection to the database without utilizing the application supervisor. This is useful for simple, one-off tasks where connection management overhead is not a concern.
    2. CrudExample.example_2(): Uses connection pooling and the application supervisor. This is the recommended approach for production applications to manage multiple concurrent connections and ensure connection resilience via the supervisor.
  3. How the ChangeStream module works

    master

    The ChangeStream module implements change stream observation using a GenServer. It spawns a separate process to consume documents returned by the change stream cursor.

    As the process runs, it handles two primary types of data:

    1. Tokens: Periodically, you will receive a token. This token can be used to reconnect to the change stream, allowing you to resume observation from that specific point without re-processing old change documents.
    2. Documents: When data in the monitored collection changes, the process receives a document describing the change (e.g., which collection and database were affected).
    pid = spawn(fn -> Enum.each(get_cursor(state), fn doc -> new_doc(doc) end) end)
  4. Configure SSL ciphers for cloud providers (e.g. AWS)

    master

    Some cloud providers require specific TLS ciphers that are not enabled by default in the Erlang SSL module. You can specify these in the ssl_opts key of Mongo.start_link/1.

    Example for enabling AES256-GCM-SHA384:

    {:ok, pid} = Mongo.start_link(database: "test",
          ssl_opts: [
            ciphers: ['AES256-GCM-SHA384'],
            cacertfile: "...",
            certfile: "..."
          ]
    )
  5. Authenticate using X.509 certificates

    master

    To use MONGODB-X509 authentication, you must provide specific SSL options in Mongo.start_link/1.

    Requirements:

    1. Set auth_mechanism: :x509.
    2. Set username to the subject entry of the user certificate (e.g., "CN=cert-user").
    3. Provide an empty password (required for current implementation).
    4. Configure ssl_opts with verify: :verify_peer, a cacertfile, and a certfile.

    Note: If using MongoDB Atlas with OTP 26, you may need to force TLS 1.2 using versions: [:"tlsv1.2"] in your SSL options to avoid compatibility issues.

    opts = [
        url: "mongodb+srv://cluster0.xxx.mongodb.net/myFirstDatabase?authSource=%24external&retryWrites=true&w=majority",
        ssl: true,
        username: "CN=cert-user",
        password: "",
        auth_mechanism: :x509,
        ssl_opts: [
          verify: :verify_peer,
          cacertfile: to_charlist(CAStore.file_path()),
          certfile: '/path-to-cert/X509-cert.pem',
          customize_hostname_check: [
            match_fun: :public_key.pkix_verify_hostname_match_fun(:https)
          ]
        ]
    ]
    
    Mongo.start_link(opts)
  6. Prerequisites for using Change Streams

    master

    To use MongoDB Change Streams with this driver, your MongoDB instance must be configured as a replica set or a sharded cluster. Change streams are not available on standalone MongoDB instances.

    If you are setting up a new replica set for testing, ensure you create the target database (e.g., db-1) before starting the change stream observer to avoid errors caused by observing a non-existent database.

  7. Configure connection pooling

    master

    The driver supports connection pooling via DBConnection. By default, it starts a single connection. To enable pooling, use the pool_size option in Mongo.start_link/1 and in all subsequent Mongo function calls.

    It is recommended to add the pooled connection to your application's supervision tree using a named instance (e.g., :mongo_db) so you can call Mongo functions using that name throughout your application.

    # Starts a pooled connection
    {:ok, top} = Mongo.start_link(url: "mongodb://localhost:27017/db-name", pool_size: 3)
    
    # In your supervision tree
    def start(_type, _args) do
      children = [
        {Mongo, [name: :mongo_db, url: "mongodb://localhost:27017/test", pool_size: 3]}
      ]
      Supervisor.start_link(children, strategy: :one_for_one)
    end
    
    # Call using the name
    Mongo.find(:mongo_db, "collection", %{})
  8. Enable network compression (zlib or zstd)

    master

    The driver supports zlib and zstd compression to reduce network payload size at the cost of increased CPU usage. Compression is supported for major operations like find, insert_one, update, aggregate, etc.

    zlib

    Append compressors=zlib to your connection URL.

    zstd

    1. If using Erlang/OTP 28+, no extra dependency is needed.
    2. For older OTP releases, add {:ezstd, "~> 1.1"} to your mix.exs dependencies.
    3. Append compressors=zstd to your connection URL.

    Disabling compression

    You can disable compression for a specific function call using the compression: false option.

    # zlib example
    {:ok, top} = Mongo.start_link(url: "mongodb://localhost:27017/db?compressors=zlib")
    
    # zstd example
    {:ok, top} = Mongo.start_link(url: "mongodb://localhost:27017/db?compressors=zstd")
    
    # Disable for one call
    Mongo.find(top, "tasks", %{}, compression: false)
  9. Manage database migrations

    master

    The driver provides a migration workflow similar to Ecto to maintain indexes and manage collections.

    Workflow

    1. Generate a migration: Run mix mongo.gen.migration <name> [topology] to create a new script in priv/mongo/migrations/ (or priv/<topology>/migrations/).
    2. Define logic: Implement the up/1 and down/1 functions in the generated Elixir script.
    3. Run migrations: Use mix mongo.migrate or call Mongo.Migration.migrate() within your application startup logic.

    Migration Locking

    To prevent multiple instances from running migrations simultaneously, the driver uses a software lock in the migration collection. If a migration fails and leaves the system locked, you can:

    • Call Mongo.Migration.unlock() or mix mongo.unlock.
    • Manually delete the document with {_id: "lock"} from the migration collection.

    Supporting Multiple Topologies

    You can organize migrations for different databases or clusters by using the topology option. This creates separate sub-folders and module namespaces.

    • Generate: mix mongo.gen.migration add_indexes topology_2 creates scripts in priv/topology_2/migrations/.
    • Run: Mongo.Migration.migrate([topology: :topology_2]) executes migrations for that specific topology.
    # Example migration script
    defmodule Mongo.Migrations.AddIndexes do
      def up(opts) do
        indexes = [
          [key: [email: 1], name: "email_index", unique: true]
        ]
        Mongo.create_indexes(:my_db, "my_collection", indexes)
      end
    
      def down(opts) do
        Mongo.drop_index(:my_db, "my_collection", "email_index")
      end
    end
    
    # Running migration in an application release
    defmodule MyApp.Release do
      def migrate() do
        # ... setup code ...
        Mongo.start_link(name: :mongo_db, url: "mongodb://localhost:27017/my-database", timeout: 60_000)
        Mongo.Migration.migrate()
      end
    end
  10. Preserve BSON document key order when encoding

    master

    Because Elixir maps are unordered, using them for operations where key order matters (like multi-field sorting) can lead to incorrect results. To preserve order when encoding from Elixir to BSON, use a list of tuples or a keyword list instead of a map.

    # CORRECT: Use keyword list for ordered sorting
    @topology
    |> Mongo.find("users", %{}, sort: [last_name: 1, first_name: 1, _id: 1])
    |> Enum.to_list()
    
    # INCORRECT: Map keys are unordered
    @topology
    |> Mongo.find("users", %{}, sort: %{last_name: 1, first_name: 1, _id: 1})
    |> Enum.to_list()
  11. Start the Insights Phoenix server

    master

    To run the Insights project locally, follow these steps to install dependencies, prepare the database, and launch the Phoenix server:

    1. Install project dependencies: mix deps.get
    2. Create and migrate the database: mix ecto.setup
    3. Start the Phoenix endpoint:
      • Using Mix: mix phx.server
      • Inside an interactive Elixir shell (IEx): iex -S mix phx.server

    Once started, the application is accessible at http://localhost:4000.

    mix deps.get
    mix ecto.setup
    mix phx.server
  12. Use Mongo.Collection to map MongoDB documents to Elixir structs

    master

    While the driver uses maps and keyword lists for documents, you can use Mongo.Collection to automate the conversion between MongoDB maps and Elixir structs. This provides support for default values, type specifications, and nested structures.

    To use it, include use Mongo.Collection and define your attributes using the document macro. This automatically generates functions like load/2 (map to struct) and dump/1 (struct to map).

    Note on Querying: Mongo.Collection does not automatically rewrite attribute names in queries or updates. You must use the actual MongoDB attribute names (e.g., if you aliased :uuid to :u, you must query using :u) when performing database operations.

    defmodule Label do
        use Mongo.Collection
    
        document do
          attribute :name, String.t(), default: "warning"
          attribute :color, String.t(), default: :red
        end
    end
    
    # Usage:
    x = Label.new()
    m = Label.dump(x)
    Label.load(m)