DOCSEARCHbyAlgolia
Submit Docs

QLever RDF/SPARQL Graph Database

repository·master·Indexed 21 days ago

https://github.com/ad-freiburg/qlever

A high-performance RDF/SPARQL 1.1 graph database designed for massive datasets on single commodity servers. It supports federated queries, named graphs, text search, and spatial queries. QLever can be deployed as a standalone server, an embedded C++ library, or via QLever.js—a WebAssembly-based engine for running SPARQL queries directly in the browser or Node.js.

Tokens
4.1K
Snippets
9
Records
16
Agent score
75%

What's inside QLever

  1. Overview of QLever graph database

    master

    QLever is a high-performance graph database that implements the RDF and SPARQL 1.1 standards. It is designed to efficiently load and query massive datasets (hundreds of billions of triples) on single commodity hardware.

    Key capabilities include:

    • Full SPARQL 1.1 Support: Includes federated queries, named graphs, updates, and the Graph Store HTTP Protocol.
    • Advanced Features: Materialized views, advanced text-search, context-sensitive SPARQL autocompletion, live query analysis, and efficient spatial queries.
    • Deployment Modes: Can be used in a standard client-server setup or as an embedded database running in-process within C++ code.
    • Visualization: Supports interactive visualization of large numbers of geometric objects on a map.

  2. Quickstart with QLever.js

    master

    QLever.js is a WebAssembly-based SPARQL engine that allows you to build RDF indexes and run queries directly in the browser or Node.js.

    Important Usage Notes:

    • Threading: QLever uses threads, making index building and querying blocking calls. You must run them inside a Web Worker (browser) or a worker thread (Node.js).
    • Memory Management: Objects created with new allocate memory in the WebAssembly heap that is NOT managed by the JavaScript garbage collector. You must release them explicitly using the using declaration (available in modern environments) or by calling .delete() in a finally block.
    • Module Type: The package is an ES module; its default export is a module factory function.
    import createQleverModule from "@ad-freiburg/qlever";

const qlever = await createQleverModule();

// Put the input data into the in-memory filesystem.
const turtleData = `
<http://example.org/a> <http://example.org/p> <http://example.org/b> .
<http://example.org/a> <http://example.org/p> "some literal" .
<http://example.org/b> <http://example.org/q> <http://example.org/c> .
`;
qlever.FS.writeFile("/input.ttl", turtleData);

// Build an index.
using config = new qlever.IndexBuilderConfig();
config.baseName = "/index";
using file = new qlever.InputFileSpecification();
file.filename = "/input.ttl";
file.filetype = qlever.Filetype.Turtle;
using files = new qlever.InputFileSpecificationVector();
files.push_back(file);
config.inputFiles = files;
qlever.Qlever.buildIndex(config);

// Load the index and query it.
using engineConfig = new qlever.EngineConfig(config);
using engine = new qlever.Qlever(engineConfig);
const result = engine.query(
    "SELECT * WHERE { ?s ?p ?o }", qlever.MediaType.sparqlJson);
const bindings = JSON.parse(result).results.bindings;
console.log(`Number of result rows: ${bindings.length}`);

  3. Pass configuration values at runtime via JSON or CLI shorthand

    master

    Once options are defined, you can set their values at runtime using two methods:

    1. JSON File

    Provide a JSON file containing the configuration paths and values, then pass the file location via the CLI. Example path structure: {"tableSizes": {"someNumber": 20}}

    2. CLI Shorthand

    You can use a simplified JSON-like syntax directly in the CLI. The shorthand has three specific rules:

    • No line breaks: It must be a single line.
    • Implicit Braces: The string is automatically treated as if it were wrapped in {}.
    • Unquoted Keys: Keys do not need double quotes (e.g., key: value instead of "key": value).

    Note: If both a JSON file and CLI shorthand are used, the CLI shorthand takes precedence and overwrites the JSON value.

    Error Handling: If a provided value cannot be interpreted as the type defined for the configuration option, an exception will be thrown.

  5. Configure browser deployment for QLever.js

    master

    To deploy QLever.js in a browser, you must satisfy the following requirements:

    1. Cross-Origin Isolation: Because QLever uses pthreads and SharedArrayBuffer, your web page must be served with these HTTP headers:

      • Cross-Origin-Opener-Policy: same-origin
      • Cross-Origin-Embedder-Policy: require-corp

    2. WASM File Location: Ship qlever.wasm in the same directory as qlever.mjs. The loader automatically resolves the WASM file relative to the module's URL. If your file structure is different, use the locateFile option in the module factory to point to the correct location.

    3. Runtime Requirements: The module is built for wasm64, which requires a recent browser or Node.js ≥ 24.

  6. Run performance comparison scripts for SparqlEngineMain

    master

    Use the compare_performance_only_own.py script to performance test different QLever SparqlEngineMain binaries (for example, after making performance-relevant changes).

    Note that the script does not currently perform checks for valid execution, and query sets may still contain legacy <in-text> tags without reporting errors.

    ./compare_performance_only_own.py \
   --index <path-to-wikipedia-freebase-easy>/wikipedia-freebase-easy \
   --queryfile query-sets/query-file-notext.txt \
   qlever-version-name ../build/SparqlEngineMain cost_factors.default.tsv \
   | column -t -s $'\t' | less -S

  7. How to write a basic benchmark class

    master

    To implement a benchmark, create a class that inherits from BenchmarkInterface (defined in benchmark/infrastructure/Benchmark.h) within the ad_benchmark namespace.

    Your class must implement the following interface:

    • name(): Returns a std::string identifying the benchmark class.
    • runAllBenchmarks(): The core logic where you perform measurements using BenchmarkResults.
    • getGeneralMetadata(): Returns metadata for the entire class.
    • getConfigManager(): Returns the configuration manager for runtime options.
    • updateDefaultGeneralMetadata(): Infrastructure method (ignore).

    Important Note on Compiler Optimizations: The infrastructure does not currently prevent compiler optimizations. Ensure that measured functions have side effects or that their return values are used, otherwise, the compiler might optimize the code away, resulting in inaccurate execution times.

    #include "benchmark/infrastructure/Benchmark.h"

namespace ad_benchmark {
class MyBenchmark : public BenchmarkInterface {
public:
  std::string name() const override { return "MyBenchmark"; }

  BenchmarkResults runAllBenchmarks() override {
    BenchmarkResults results{};

    auto dummyFunctionToMeasure = []() {
      // Code to measure
    };

    const std::string identifier = "Some identifier";
    results.addMeasurement(identifier, dummyFunctionToMeasure);

    return results;
  }

  // Implement other required interface methods...
};
}

  8. Register a benchmark class with AD_REGISTER_BENCHMARK and CMake

    master

    After defining your benchmark class, you must register it to make it part of the benchmark suite.

    1. Register in C++: Use the AD_REGISTER_BENCHMARK macro within the ad_benchmark namespace.

      AD_REGISTER_BENCHMARK(MyClass, ConstructorArgument1, ConstructorArgument2, ...);

    2. Register in CMake: Add the file to benchmark/CMakeLists.txt using the addAndLinkBenchmark function. Do not include the .cpp extension.

      addAndLinkBenchmark(MyBenchmarkClassFile)

    Compiled benchmarks are located in the benchmark folder within your build directory.

    # In benchmark/CMakeLists.txt
addAndLinkBenchmark(MyBenchmarkClassFile)

  9. Install and get started with QLever

    master

    To use QLever, you can install native packages or use a containerized version. All operations are managed through the qlever command-line tool.

    Installation Options:

    • Debian/Ubuntu: Use native packages released for these distributions.
    • macOS: Use native packages (including Apple Silicon support).
    • Docker/Podman: Use the platform-independent image available on Docker Hub: adfreiburg/qlever.

    For detailed installation steps, refer to the official Quickstart documentation.

  10. Initialize and run the QLever Server

    master

    The Server class is the primary entrypoint for running the QLever engine as an HTTP service. To use it, instantiate the Server with a port, thread count, access token, and engine configuration, then call run(). Note that run() is a blocking call that handles request processing and only returns if an exception is thrown.

    Constructor Parameters

    • port: The port number to listen on.
    • numThreads: The number of threads for the query thread pool.
    • accessToken: A string used for authentication.
    • config: A qlever::EngineConfig object defining engine behavior.
    • noAccessCheck (optional): If true, bypasses access token validation.
    • metricsReader (optional): A pointer to a MetricsReader to enable the /metrics endpoint.
    // Conceptual usage pattern
Server server(
    8080, 
    16, 
    "your-access-token", 
    engineConfig
);
server.run();

  11. Add runtime configuration options to a benchmark

    master

    You can define configuration options that can be modified at runtime. This is typically done in the constructor of your benchmark class using the ConfigManager::addOption method on the manager_ member variable.

    Supported Types

    • bool
    • std::string
    • int
    • size_t
    • float
    • std::vector<T> (where T is one of the above)

    How it works

    When adding an option, you provide a name, a description, an optional default value, and a type. You must also pass a pointer to a variable of the target type. The ConfigManager will update this variable whenever the option is set at runtime.

    Options are identified using JSON-like string paths (e.g., tableSizes/someNumber).

  12. Measure functions and organize results with BenchmarkResults

    master

    Inside your runAllBenchmarks() implementation, use the BenchmarkResults object to capture and organize measurements.

    Single Measurements

    Use results.addMeasurement(identifier, lambda) to measure the execution time of a lambda function. The lambda must take no arguments.

    Tables

    Use results.addTable(identifier, rowNames, columnNames) to create a table.

    • Rows: The first column is used to store row names.
    • Entries: Use table.addMeasurement(row, col, lambda) to add a timed measurement to a specific cell, or table.setEntry(row, col, value) to set a custom value.
    • Types: table.setEntry accepts any type in ad_benchmark::ResultTable::EntryType except std::monostate.

    Groups

    Use results.addGroup(identifier) to create a logical grouping. You can then add measurements or tables to this group using group.addMeasurement(...) or group.addTable(...).

    BenchmarkResults runAllBenchmarks() {
  BenchmarkResults results{};

  auto& dummyFunctionToMeasure = []() {
    // Do work
  };
  const std::string identifier = "Some identifier";

  // 1. Single measurement
  results.addMeasurement(identifier, dummyFunctionToMeasure);

  // 2. Table
  auto& table = results.addTable(identifier, {"row1", "row2"}, {"Col1", "Col2"});
  table.addMeasurement(0, 1, dummyFunctionToMeasure); // Row 0, Col 1
  table.setEntry(0, 1, "Custom string entry");
  table.setEntry(0, 0, "new_row_name"); // Replacing a row name

  // 3. Group
  auto& group = results.addGroup(identifier);
  group.addMeasurement(identifier, dummyFunctionToMeasure);
  group.addTable(identifier, {"r1"}, {"c1"});

  return results;
}