Dgraph Distributed GraphQL Database

repository·main·Indexed 12 days ago

https://github.com/dgraph-io/dgraph

A distributed, horizontally scalable GraphQL database designed for high-performance graph queries. Dgraph provides ACID transactions, low-latency retrieval of complex interconnected data, and native support for Full-Text Search, Regular Expressions, and Geo Search. It communicates over gRPC and HTTP and supports Linux/amd64 and Linux/arm64 architectures.

Tokens
40.8K
Snippets
121
Records
183
Agent score
97%

What's inside Dgraph

  1. Dgraph Core Concepts and Capabilities

    main

    Dgraph is a horizontally scalable, distributed GraphQL database with a graph backend.

    Key Features:

    • Transactions: Provides ACID transactions, consistent replication, and linearizable reads.
    • Query Languages: Supports GraphQL query syntax and responds via JSON or Protocol Buffers.
    • Protocols: Communicates over gRPC and HTTP.
    • Scalability: Designed for Google-scale throughput and low latency over terabytes of data.
    • Native Features: Includes native support for Full-Text Search, Regular Expressions, and Geo Search.
  2. Manage integration2 test versions and binaries

    main

    The integration2 testing framework handles versioning automatically. On the first run, it clones the Dgraph repository, checks out the requested version, and builds the binary (GOOS=linux), caching it in dgraphtest/binaries/dgraph_<version>. Subsequent runs reuse this cache.

    Supported Version Formats:

    • "local": Uses the binary at $GOPATH/bin/dgraph (default).
    • "v23.0.1": A specific git tag.
    • "4fc9cfd": A specific commit hash.
  3. Use testify/suite for Integration and Upgrade tests

    main

    Dgraph uses testify/suite for complex testing scenarios that require shared setup/teardown logic across multiple methods. This is particularly useful for running the same test logic in two different modes:

    1. Integration Mode (//go:build integration): Creates a cluster once and tests the current version's behavior.
    2. Upgrade Mode (//go:build upgrade): Creates a cluster with an older version, runs tests, performs an Upgrade(), and runs the same tests again to ensure data remains valid.

    Available Hooks

    • SetupSuite(): Once before all tests.
    • SetupTest(): Before each test method.
    • SetupSubTest(): Before each subtest.
    • TearDownTest(): After each test method.
    • TearDownSuite(): Once after all tests.

    Running Suites

    Using go test:

    # Run integration suite
    go test -v --tags=integration ./systest/plugin/
    
    # Run specific method in integration suite
    go test -v --tags=integration --run 'TestPluginTestSuite/TestPasswordReturn' ./systest/plugin/
    
    # Run same tests in upgrade mode
    go test -v --tags=upgrade --run 'TestPluginTestSuite/TestPasswordReturn' ./systest/plugin/

    Using make:

    # Run plugin systest via t/ runner
    make test SUITE=systest PKG=systest/plugin
    
    # Run specific test
    make test SUITE=systest PKG=systest/plugin TEST=TestPluginTestSuite/TestPasswordReturn
    
    # Run in upgrade mode
    make test TAGS=upgrade PKG=systest/plugin TEST=TestPluginTestSuite/TestPasswordReturn
  4. Avoid using package z in production code

    main
    The z package is strictly intended for use within test files (e.g., *_test.go). It is not designed for production or non-test code because its error handling is minimal and often results in fatal errors. Using this package in application logic can lead to unexpected crashes.
  5. How to choose between Unit and Integration tests

    main

    Dgraph uses a layered testing approach. Choosing the right test type depends on whether the code requires a running Dgraph cluster.

    Unit Tests

    • Purpose: Test pure logic, parsing, data conversions, and algorithms in isolation.
    • Requirement: Does not require a Dgraph cluster.
    • Placement: Place *_test.go files in the same package as the source code.
    • Build Tags: No build tag is required. Files without a //go:build tag are treated as unit tests.

    Integration Tests

    • Purpose: Test component interactions, full system workflows, and cluster-dependent behavior.
    • Requirement: Requires a running Dgraph cluster.
    • Placement: Often placed in the systest/ directory or specific functional directories (e.g., systest/backup/).
    • Build Tags: Requires specific Go build tags to be included in the compilation (e.g., //go:build integration).

    Summary Table

    FeatureTest TypeBuild TagTypical Location
    Query/Mutation logicIntegrationintegrationExisting package or systest/
    Backup / RestoreIntegrationintegrationsystest/backup/
    ExportIntegrationintegrationsystest/export/
    Vector / EmbeddingsIntegrationintegrationsystest/vector/
    GraphQL schema/endpointsIntegrationintegrationgraphql/e2e/
    ACL / AuthIntegrationintegrationacl/ or systest/acl/
    Version UpgradesUpgradeupgradeSame package as code
    Fine-grained cluster controlIntegration2integration2systest/integration2/
  6. Docker Compose conventions for integration tests

    main

    The testing framework uses a hierarchical discovery system for docker-compose.yml files during integration tests:

    1. It first looks in the immediate package directory of the test.
    2. If not found, it searches parent directories recursively.
    3. The default root-level configuration is located at ../dgraph/docker-compose.yml.

    If you are implementing a new test that requires a unique Docker configuration, create a new directory for your tests and include a custom docker-compose.yml file within that directory.

  7. Understand Dgraph test types and build tags

    main

    Dgraph uses Go build tags to categorize tests by their cost and purpose. Use these tags to run specific subsets of the test suite.

    Test TypePurposeBuild TagRequirements
    Unit TestsTest functions/components in isolationNoneNone (fast)
    Integration TestsTest component interactions and workflowsintegrationRunning Dgraph cluster (Docker)
    Upgrade TestsTest database migrations/upgradesupgradeRunning Dgraph cluster (Docker)
    Benchmark TestsPerformance testingNone (uses Benchmark prefix)High performance requirements
    Cloud TestsCloud-specific functionalitycloudDeprecated

    Note on Test Runners: Newer integration and upgrade tests use the dgraphtest package for programmatic cluster control. Older tests may use the t/ runner. For new tests, prefer dgraphtest for cluster management and dgraphapi for client operations.

  8. Dgraph Supported Platforms

    main

    Dgraph officially supports the following architectures:

    • Linux/amd64
    • Linux/arm64

    Note: Official support for macOS and Windows was dropped in 2021 to optimize for Linux-specific memory performance and advancements. While you can build Dgraph on other platforms (e.g., for bulk loading), official support is limited to the Linux architectures listed above.

  9. Understand the query benchmark data structure

    main

    The benchmark directory contains gob-encoded data representing processed SubGraphs. The files are categorized by the type of query they represent and the scale of the result set. The suffix in the filename (e.g., 10, 100, 1000) indicates the number of entities returned by that specific query.

    Supported query types in these benchmarks include:

    • Actors query: Retrieves actor names and their associated films.
    • Directors query: Retrieves director names and the genres of their films.
    // Example Actors query used in benchmarks
    {
        me(_xid_:m.08624h) {
            type.object.name.en
            film.actor.film {
                film.performance.film {
                    type.object.name.en
                }
            }
        }
    }
  10. Use AllowedOptions to parse index directive parameters

    main

    When implementing or interacting with index factories, you use the AllowedOptions class to define which named options are valid and how their values should be parsed. This prevents type mismatch errors (e.g., treating an integer as a string).

    To populate an Options instance from key-value pairs extracted from a schema directive:

    1. Retrieve the allowed options from the factory using AllowedOpts().
    2. Create a new Options instance using NewOptions().
    3. Use GetParsedOption(name, value) to validate and convert a specific value.
    4. Use PopulateOptions(pairs, optionsInstance) to bulk-process a collection of key-value pairs into an Options object.
    // 1. Get allowed options from the factory
    allowedOpts := hnswFactory.AllowedOpts()
    
    // 2. Initialize new options instance
    myAttributeIndexOpts := NewOptions()
    
    // 3. Parse a specific option manually
    val, err := allowedOpts.GetParsedOption("exponent", "6")
    if err != nil {
        return ErrBadOptionNoBiscuit
    }
    myAttributeIndexOpts.SetOpt("exponent", val)
    
    // 4. Or, populate all options at once from a map of pairs
    // pairs := map[string]string{"metric": "euclidean", "exponent": "6"}
    err = allowedOpts.PopulateOptions(pairs, myAttributeIndexOpts)
  11. Understand the Group Delete Test scenario

    main

    The Group Delete Test validates the system's ability to handle the removal of nodes from groups. The goal is to ensure that when a group contains zero nodes, the system correctly identifies and deletes the empty group. The test follows this lifecycle:

    1. Initialization: Start a cluster containing 3 groups, with 1 node assigned to each group.
    2. Node Removal: Remove a node from a specific group (e.g., Group 3).
    3. Zero-State Verification: Verify that the system's zero state reflects the deletion of the now-empty group.
    4. Cluster Viability Check: Execute a query to ensure the cluster remains operational after the group deletion.
    5. Iterative Deletion: Repeat the process for remaining groups (e.g., Group 2) to ensure continuous stability.
  12. Avoid common testing anti-patterns in Dgraph

    main

    To ensure reliable and maintainable tests, avoid these common mistakes:

    • ❌ Don't use time.Sleep for synchronization: This leads to flaky tests. Instead, use condition checks like require.NoError(t, c.HealthCheck(false)) to wait for an actual state change.
    • ❌ Don't share mutable state between tests: Avoid global variables like var sharedClient *Client. Each test should instantiate its own resources to prevent interference.
    • ❌ Don't depend on test execution order: Every test should be independent. If TestQuery needs data, call a setupData(t) helper within that test rather than assuming TestInsertData ran previously.
    • ❌ Don't ignore errors: Always check errors using require.NoError(t, err) or similar. Ignoring errors in tests can mask critical bugs.
    • ❌ Use t.Parallel() with caution: Only use it for truly independent tests. Do not use it for integration tests that share clusters, modify global state, or use the same network ports/resources.