SurrealDB Python SDK

repository·main·Indexed 42 days ago

https://github.com/surrealdb/surrealdb.py

The official SurrealDB SDK for Python (version 3.0.0-beta.8), providing a high-level interface to interact with SurrealDB databases using Pythonic methods and raw SurrealQL queries. It supports both asynchronous and synchronous operations, remote connections via WebSockets and HTTP, and embedded database support (via the surrealdb-embedded extension) for in-memory or file-based storage.

Tokens
42K
Snippets
145
Records
183
Agent score
50%

What's inside SurrealDB

  1. Handle errors in the SurrealDB Python SDK

    main

    All SDK errors derive from SurrealError. Errors are categorized into two main branches based on whether the request reached the server and whether the server rejected it:

    • ServerError: The server received and processed the request but rejected it (e.g., syntax errors, permission issues). These are generally not retryable without changes. ServerError includes specific subclasses like NotFoundError that mirror SurrealDB's structured error format, allowing you to match on kind and details.
    • TransportError: The request failed to produce a structured response (e.g., network failure, timeout). These may be retryable.

    Common TransportError subclasses include:

    • ConnectionUnavailableError: Host unreachable or socket closed.
    • TransportTimeoutError: Request timed out.
    • HttpStatusError: A non-2xx HTTP response (contains .status, .body, and .url).

    Invalid Python inputs (like wrong types) will raise standard ValueError or TypeError before reaching the server.

    from surrealdb import Surreal
    from surrealdb.errors import NotFoundError, ServerError, SurrealError, TransportError
    
    with Surreal("ws://localhost:8000/rpc") as db:
        db.signin({"username": "root", "password": "root"})
        db.use("ns", "db")
    
        try:
            db.query("SELECT * FROM nonexistent:1").execute()
        except NotFoundError as error:
            print(error.kind, error.table_name)   # NotFound nonexistent
        except ServerError as error:
            print(error.kind, error.details)
        except TransportError as error:
            print("could not reach the server:", error)
  2. Understand CBOR serialization in SurrealDB

    main
    SurrealDB uses CBOR (Concise Binary Object Representation) for efficient, compact binary data serialization. Because SurrealDB uses custom data types that are not natively supported by the standard CBOR specification, this implementation provides custom encoding and decoding logic to ensure domain-specific types can be safely exchanged between systems.
  3. Handle SurrealDB Sets with SurrealSet

    main

    SurrealDB's set<T> is a deduplicated sequence. In Python, these are returned as SurrealSet objects.

    Characteristics

    • Type: SurrealSet is a subclass of list. It supports indexing and comparison like a list.
    • Compatibility: Unlike a standard Python set, a SurrealSet can hold non-hashable members like dict or list (e.g., set<object>).
    • Round-tripping: Writing a SurrealSet back to the database preserves the set type. Writing a plain Python set also works and is sent as a set.
    • Normalization: SurrealDB normalizes sets, so the order of elements in a SurrealSet may differ from the input order.

    Note: SurrealSet support requires SurrealDB 3.x. SurrealDB 2.x does not have a CBOR set representation.

    # SurrealSet behaves like a list
    row = db.select(rec)                   # {"tags": SurrealSet(['a', 'b'])}
    row["name"] = "new name"
    db.update(rec, row)              # tags is still a set
    
    # Comparison with list works
    db.select(rec)["tags"] == ["a", "b"]   # True
  4. How File storage works in SurrealDB

    main

    SurrealDB can store files in a bucket (in-memory, on disk, or via object storage like S3).

    Key Concepts:

    • A File is a reference consisting of a bucket name and a key. It is not a Python file object and holds no bytes itself.
    • Experimental Feature: To use buckets, the server must be started with SURREAL_CAPS_ALLOW_EXPERIMENTAL=files and a bucket must be defined (e.g., DEFINE BUCKET images BACKEND "memory";).
    • Non-Transactional Writes: File writes (db.files.put) are not transactional. They write directly to the bucket backend. This means a file write is visible to other connections immediately and will survive a txn.cancel(), even if a record write in the same transaction is rolled back.
    • Key Normalization: Keys are normalized to start with /. File("b", "a.txt") and File("b", "/a.txt") refer to the same file.
    • Data Types: put accepts bytes, bytearray, or memoryview. get returns bytes. You must manually encode strings (e.g., .encode("utf-8")) before putting them, as the server stores raw bytes.
    from surrealdb import File, Surreal
    
    with Surreal("ws://localhost:8000/rpc") as db:
        db.signin({"username": "root", "password": "root"})
        db.use("ns", "db")
    
        avatar = File("images", "/photos/avatar.png")
    
        with open("avatar.png", "rb") as handle:
            db.files.put(avatar, handle.read())
    
        png_bytes = db.files.get(avatar)         # -> bytes, or None if absent
        meta = db.files.head(avatar)             # -> FileMetadata | None
  5. Performance and persistence of embedded engines

    main

    Embedded SurrealDB engines differ in how they handle data storage and performance:

    In-Memory (memory or mem://)

    • Performance: Fastest (all data resides in RAM).
    • Persistence: Non-persistent; all data is lost when the connection closes.
    • Best for: Unit tests, caches, and temporary data.

    File-Based (file://, surrealkv://, or surrealkv+versioned://)

    • Performance: Good performance using the SurrealKV storage engine.
    • Persistence: Persistent; data is saved to disk.
    • Best for: Local applications, development, and single-node deployments.
  6. How DataLoader and Subscriptions work in the GraphQL example

    main

    This example implements two key patterns for high-performance GraphQL servers:

    DataLoader for N+1 Prevention

    To prevent the N+1 query problem (where fetching a list of items triggers a separate database query for every item's relation), the example uses a DataLoader. This batches and caches database queries.

    # Efficiently loads multiple users in a single query
    user_loader = UserDataLoader(db)
    users = await asyncio.gather(*[user_loader.load(id) for id in user_ids])

    Subscriptions via SurrealDB Live Queries

    Real-time updates are achieved by leveraging SurrealDB's live query capabilities through WebSockets.

    # Subscribe to user changes
    async for user in subscribe_to_users():
        yield user  # Pushed to client in real-time
  7. Implement SurrealDB connection management in Django middleware

    main

    To manage SurrealDB connections efficiently, use a custom middleware that provides a connection per request. This ensures the connection is opened when a request starts and closed when the response is sent.

    class SurrealDBMiddleware:
        """Manages SurrealDB connection per request."""
        
        def __init__(self, get_response):
            self.get_response = get_response
        
        async def __call__(self, request):
            request.db = await get_connection()
            response = await self.get_response(request)
            await close_connection(request.db)
            return response
    class SurrealDBMiddleware:
        """Manages SurrealDB connection per request."""
        
        def __init__(self, get_response):
            self.get_response = get_response
        
        async def __call__(self, request):
            request.db = await get_connection()
            response = await self.get_response(request)
            await close_connection(request.db)
            return response
  8. Use client-side transactions and sessions (WebSocket only)

    main

    Client-side transactions and multi-session support are available only for WebSocket connections (ws:// or wss://). They are not supported for HTTP or embedded connections.

    To use them, you must:

    1. Create a session using await db.new_session().
    2. Use the session object to perform operations.
    3. Start a transaction on the session using await session.begin_transaction().
    4. Commit the transaction with await txn.commit() or roll it back with await txn.cancel().

    Both AsyncSurrealSession/BlockingSurrealSession and AsyncSurrealTransaction/BlockingSurrealTransaction provide the same CRUD builder, query(), run(), query_raw(), info(), and version() APIs. Operations are scoped to the session or transaction instance, so you do not need to pass session or transaction IDs manually.

    async with AsyncSurreal("ws://localhost:8000/rpc") as db:
        await db.signin({"username": "root", "password": "root"})
        await db.use("ns", "db")
    
        # Create a session
        session = await db.new_session()
        await session.use("ns", "db")
    
        # Start a transaction on the session
        txn = await session.begin_transaction()
        await txn.create(RecordID("account", "alice"), {"balance": 100})
        await txn.update(RecordID("account", "bob")).merge({"balance": 50})
    
        # Commit (or call `await txn.cancel()` to roll back)
        await txn.commit()
    
        await session.close_session()
  9. Compare Embedded vs Remote SurrealDB usage

    main

    Choosing between an embedded database and a remote database depends on your architecture and persistence requirements.

    Use Embedded Database

    Use connection strings like memory, mem://, file://, surrealkv://, or surrealkv+versioned:// when:

    • Desktop Applications: No server setup required.
    • Testing: In-memory databases are extremely fast for unit tests.
    • Local Development: Quick prototyping without managing infrastructure.
    • Edge Computing: Running on devices without network access.
    • Single-App Storage: When the data is specific to one application.

    Use Remote Database

    Use connection strings like ws:// or http:// when:

    • Multi-App Architecture: Multiple applications need to share the same data.
    • Distributed Systems: Microservices or distributed architectures.
    • Cloud Deployments: Scalable cloud infrastructure.
    • Horizontal Scaling: When you need to scale across multiple instances.
    • Team Collaboration: Centralized data for multiple developers.
  10. Manage Sessions and Transactions

    main

    The SDK provides mechanisms for managing stateful connections and atomic operations, primarily via WebSockets:

    • Sessions: Use attach() on a WebSocket connection to create a new session (returns a UUID). You can use new_session() to obtain a scoped AsyncSurrealSession or BlockingSurrealSession. To end a session, call close_session() or detach(session_id).
    • Transactions: To perform atomic operations, call begin_transaction() on a session (or the default connection). This returns a Transaction object. Use .commit() to apply changes or .cancel() to roll back.

    Important: attach(), detach(), begin(), commit(), cancel(), and new_session() are not supported on HTTP or embedded connections and will raise UnsupportedFeatureError.

  11. How RecordID and Table work

    main

    The RecordID and Table constructors perform client-side validation to catch errors before they reach the server.

    • Table: The name must be a str. Passing None will raise a TypeError.
    • RecordID: The ID must be one of str, int, uuid.UUID, list, tuple, dict, or a Range.
      • Rejected types: None, bool, float, and bytes are rejected by the server and will cause issues if used.
      • Composite IDs: Supported via list or dict types.

    Note: While constructors validate types, records decoded from the server bypass these checks to ensure compatibility with future server versions.

    RecordID("person", "tobie")     # ok
    RecordID("person", ["a", 1])    # ok (composite id)
    RecordID(1, "tobie")            # TypeError: arguments swapped
    Table(None)                     # TypeError: name must be a str
  12. Use Django REST Framework ViewSets with SurrealDB

    main

    When using SurrealDB with Django, you can bypass the standard Django ORM and use DRF ViewSets to interact with SurrealDB directly via the connection attached to the request.

    class UserViewSet(viewsets.ModelViewSet):
        serializer_class = UserSerializer
        
        def get_queryset(self):
            # Fetch from SurrealDB
            return self.request.db.select("users")
    class UserViewSet(viewsets.ModelViewSet):
        serializer_class = UserSerializer
        
        def get_queryset(self):
            # Fetch from SurrealDB
            return self.request.db.select("users")