DOCSEARCHbyAlgolia
Submit Docs

dukascopy-node

repository·master·Indexed 21 days ago

https://github.com/leo4815162342/dukascopy-node

A Node.js library and CLI tool for downloading free historical and real-time market price tick data for Crypto, Stocks, ETFs, CFDs, Forex, Commodities, and Bonds. It supports multiple output formats including JSON, CSV, and arrays, and includes features for disk caching, custom request batching to avoid rate-limiting, and full TypeScript support.

Tokens
35.8K
Snippets
55
Records
78
Agent score
73%

What's inside dukascopy-node

  3. Enable caching for historical price data

    master

    By default, caching is disabled in dukascopy-node. Enabling it allows completed-period JSON responses to be stored on disk, enabling faster subsequent requests for the same instrument and date range by serving data from the local file system instead of the network.

    To enable caching, set the useCache flag to true. You can optionally specify a custom directory using cacheFolderPath (the default is .dukascopy-cache).

    Caching Limitations

    Buckets containing the current time are not cached because the data is still changing. This applies to:

    • tick and s1: current hour
    • m1, m5, m15, and m30: current day
    • h1 and h4: current month
    • d1 and mn1: current year

    Note that requests for an earlier range inside one of these active buckets will still be fetched from the network.

    const { getHistoricalRates } = require("dukascopy-node");

(async () => {
  try {
    const data = await getHistoricalRates({
      instrument: "eurusd",
      dates: {
        from: new Date("2021-02-01"),
        to: new Date("2021-03-01"),
      },
      timeframe: "d1",
      format: "json",
      useCache: true, // Enables caching
    });

    console.log(data);
  } catch (error) {
    console.log("error", error);
  }
})();

  4. Quick start guide for dukascopy-node

    master

    The dukascopy-node project provides several ways to interact with financial data. You can explore the following topics to get started:

    • Basic usage: Understanding output formats and core logic.
    • Downloading tick data: How to retrieve historical tick-level data.
    • Date formatting and timezones: Managing time conversions for data requests.
    • Error handling: Dealing with empty data sets and common errors.
    • Caching and Batching: Optimizing downloads using cache or custom batching strategies.
    • TypeScript integration: Using the library in TypeScript environments.

  6. Enable debugging for dukascopy-node

    master

    The library uses the debug module to provide detailed execution logs. You can enable debugging by setting the DEBUG environment variable or using a CLI flag.

    Using Environment Variables

    To enable all debugging logs, set the environment variable to DEBUG=dukascopy-node:*.

    If you are specifically debugging the CLI interface, you can use DEBUG=dukascopy-node:cli:* to target CLI-specific logs.

    Using the CLI Flag

    When using the command line interface, you can simply append the -d flag to your command to enable debugging output.

    # Enable all debugging via environment variable
DEBUG=dukascopy-node:* npx dukascopy-node -i usdjpy -from 2022-03-21 -to 2022-03-22 -t m1 -d

# Enable only CLI debugging via environment variable
DEBUG=dukascopy-node:cli:* npx dukascopy-node -i usdjpy -from 2022-03-21 -to 2022-03-22 -t m1

# Enable debugging using the -d flag
npx dukascopy-node -i usdjpy -from 2022-03-21 -to 2022-03-22 -t m1 -d

  7. Configure custom batching for historical price downloads

    master

    When downloading historical price data, dukascopy-node generates multiple URLs (one per day) to fetch data from Dukascopy's JSON API. To avoid rate-limiting and overwhelming the servers, you can control how these requests are grouped and spaced using batchSize and pauseBetweenBatchesMs.

    By default, the library uses a batchSize of 10 and a pauseBetweenBatchesMs of 1000 ms. Note that cache hits do not trigger the pause.

    For large tick ranges, it is recommended to use a smaller batchSize and a longer pauseBetweenBatchesMs to reduce request pressure.

    const { getHistoricalRates } = require('dukascopy-node');

(async () => {
  const data = await getHistoricalRates({
    instrument: 'eurusd',
    dates: {
      from: new Date('2019-06-01'),
      to: new Date('2019-07-01')
    },
    timeframe: 'm1',
    batchSize: 15,
    pauseBetweenBatchesMs: 2000
  });

  console.log(data);
})();

  8. Configure RealTimeRatesConfig for different output formats

    master

    The getRealTimeRates function uses a discriminated union for its configuration object to ensure type safety based on the requested format.

    • For Arrays: Set format: 'array'. If timeframe is 'tick', it uses RealTimeRatesConfigArrayTickItem. Otherwise, it uses RealTimeRatesConfigArrayItem.
    • For JSON: Set format: 'json'. If timeframe is 'tick', it uses RealTimeRatesConfigJsonTickItem. Otherwise, it uses RealTimeRatesConfigJsonItem.
    • For CSV: Set format: 'csv'. This uses RealTimeRatesConfigCsv and allows any timeframe.

  9. Header mapping for different timeframes

    master

    The stream-writer uses different header sets depending on the TimeframeType provided. This ensures the output columns match the data structure.

    Standard Timeframes (e.g., m1, h1): ['timestamp', 'open', 'high', 'low', 'close', 'volume']

    Tick Timeframe: ['timestamp', 'askPrice', 'bidPrice', 'askVolume', 'bidVolume']

    Note: If volumes is set to false in BatchStreamWriter, the last column (volume/bidVolume) is removed from the headers and the output.

  10. How BufferFetcher handles caching

    master

    When a cacheManager is provided to BufferFetcher, the following logic is applied:

    1. Cache Lookup: Before fetching a URL, the fetcher checks if the URL is "mutable" using isMutableDataUrl. If the URL is not mutable, it attempts to read the buffer from the cacheManager using readItemFromCache(url).
    2. Cache Hit: If found in cache, isCacheHit is set to true for that item, and the network request is skipped.
    3. Cache Write: After a batch is processed, all successfully fetched items that are not mutable are written to the cache via writeItemsToCache(items[]).

    Note on Mutability: URLs that start with the URL_ROOT and contain a from query parameter are considered mutable and are not cached to prevent stale data issues.

  11. Understand the Config union types

    master

    The Config type is a union of several specialized interfaces that enforce specific combinations of timeframe and format. This ensures that your configuration is valid for the requested output type.

    InterfaceAllowed TimeframesAllowed Formats
    ConfigArrayTickItem'tick''array'
    ConfigArrayItemAny except 'tick''array'
    ConfigJsonTickItem'tick''json'
    ConfigJsonItemAny except 'tick''json'
    ConfigCsvItemAny'csv'

  12. Enable debug logging for the CLI

    master

    You can enable detailed debugging information for the CLI in two ways:

    1. Via CLI argument: Pass the debug option in your configuration.
    2. Via Environment Variable: Set the DEBUG environment variable. The CLI uses the namespace dukascopy-node:cli.

    When debug mode is active, the CLI will output internal details such as generated URLs, fetcher status (cache vs network), and configuration state.

    # Using environment variable
DEBUG=dukascopy-node:cli* node path/to/cli-entrypoint.js