DOCSEARCHbyAlgolia
Submit Docs

Kinesalite Documentation

repository·master·Indexed 21 days ago

https://github.com/mhart/kinesalite

A local implementation of Amazon Kinesis built on LevelDB, providing a mock HTTP server for testing Kinesis-dependent applications. It supports a subset of the Kinesis API, including CreateStream, PutRecord, and GetRecords, and can be used via a CLI tool or programmatically as a Node.js module.

Tokens
1.4K
Snippets
7
Records
10
Agent score
74%

What's inside kinesalite

  2. Supported Content Types and Serialization

    master

    Kinesalite handles requests using AWS-compatible serialization formats. It detects the format based on the Content-Type header. Supported types include:

    • application/x-amz-json-1.1 (Amazon JSON)
    • application/x-amz-cbor-1.1 (Amazon CBOR)
    • application/json (Standard JSON)

    Note: While application/json is accepted for parsing, the server logic specifically checks for the Amazon-specific headers to progress certain operations.

  3. Connect to Kinesalite using the `kinesis` module

    master

    You can use the kinesis module to connect to Kinesalite. Note that this module currently only works in HTTPS mode, which requires starting Kinesalite with the --ssl flag.

    var kinesis = require('kinesis')

kinesis.listStreams({host: 'localhost', port: 4567}, console.log)

  4. Connect to Kinesalite using the AWS SDK

    master

    To interact with Kinesalite using the official aws-sdk, point the Kinesis client to the Kinesalite endpoint (e.g., http://localhost:4567). Ensure the SDK is configured before initializing the client.

    var AWS = require('aws-sdk')

var kinesis = new AWS.Kinesis({endpoint: 'http://localhost:4567'})

kinesis.listStreams(console.log.bind(console))

  5. Use Kinesalite programmatically in Node.js

    master

    You can require kinesalite as a module to get a standard Node.js HTTP server instance. This allows you to configure the LevelDB path and stream state durations programmatically.

    // Returns a standard Node.js HTTP server
var kinesalite = require('kinesalite'),
    kinesaliteServer = kinesalite({path: './mydb', createStreamMs: 50})

// Listen on port 4567
kinesaliteServer.listen(4567, function(err) {
  if (err) throw err
  console.log('Kinesalite started on port 4567')
})

  6. Kinesalite CLI Options Reference

    master

    The following options are available when running the kinesalite command:

    OptionDescription
    --helpDisplay this help message and exit
    --port <port>The port to listen on (default: 4567)
    --path <path>The path to use for the LevelDB store (in-memory by default)
    --sslEnable SSL for the web server (default: false)
    --createStreamMs <ms>Amount of time streams stay in CREATING state (default: 500)
    --deleteStreamMs <ms>Amount of time streams stay in DELETING state (default: 500)
    --updateStreamMs <ms>Amount of time streams stay in UPDATING state (default: 500)
    --shardLimit <limit>Shard limit for error reporting (default: 10)

  7. Run Kinesalite via CLI

    master

    Kinesalite can be run as a standalone HTTP server from the command line. It is an implementation of Amazon's Kinesis API backed by LevelDB.

    $ kinesalite --help

Usage: kinesalite [--port <port>] [--path <path>] [--ssl] [options]

  8. Initialize Kinesalite server

    master

    The kinesalite(options) function is the primary entrypoint. It returns a standard Node.js http.Server or https.Server instance configured to emulate the Amazon Kinesis API.

    To enable HTTPS, provide an options object containing SSL credentials. If options.ssl is true, the server will attempt to load default certificates from the ssl/ directory relative to the module if they are not explicitly provided in the options object.

    When calling server.close(), Kinesalite ensures that the underlying database connection is also closed before completing the callback.

    const kinesalite = require('kinesalite');

// Standard HTTP server
const server = kinesalite();
server.listen(4567);

// HTTPS server with custom certificates
const httpsServer = kinesalite({
  ssl: true,
  key: fs.readFileSync('path/to/server-key.pem'),
  cert: fs.readFileSync('path/to/server-crt.pem'),
  ca: fs.readFileSync('path/to/ca-crt.pem')
});
httpsServer.listen(4567);

  9. Supported Kinesis Operations

    master

    Kinesalite supports a specific subset of the Kinesis API. The following operations are implemented:

    • AddTagsToStream
    • CreateStream
    • DeleteStream
    • DescribeStream
    • DescribeStreamSummary
    • GetRecords
    • GetShardIterator
    • ListShards
    • ListStreams
    • ListTagsForStream
    • MergeShards
    • PutRecord
    • PutRecords
    • RemoveTagsFromStream
    • SplitShard
    • IncreaseStreamRetentionPeriod
    • DecreaseStreamRetentionPeriod

  10. Use the kinesalite CLI

    master

    Kinesalite is a Kinesis HTTP server that can be optionally backed by LevelDB. You can run it from the command line to mock Kinesis functionality. By default, it listens on port 4567 and uses an in-memory store unless a --path is provided.

    kinesalite [--port <port>] [--path <path>] [--ssl] [options]