csvtojson

repository·master·Indexed 24 days ago

https://github.com/keyang/node-csvtojson

A high-performance Node.js library and CLI tool for converting large CSV files into complex or nested JSON objects. Version 2.0.14 supports streaming, multi-core processing, and custom transformations via hooks like .preRawData() and .preFileLine(). It provides methods to parse from files, strings, and streams, and can be used in both Node.js and browser environments. The CLI supports extensive configuration for delimiters, quoting, header handling, and output formats including JSON, CSV row arrays, and line strings.

Tokens
15.7K
Snippets
38
Records
86
Agent score
83%

What's inside csvtojson

  1. Performance characteristics of csvtojson

    master
    csvtojson is optimized for Node.js applications and is designed for high-speed CSV parsing. Benchmarks indicate that it can be approximately 4 to 6 times faster than other popular CSV parsing libraries. Performance is significantly improved in version 1.1.0 and later compared to older versions.
  2. Configure field types: Implicit and Explicit

    master

    You can control how values are typed in the resulting JSON.

    Implicit Typing

    When checkType is enabled, the parser automatically converts values to their inferred type (e.g., true becomes a boolean, 12 becomes a number, and JSON strings become objects).

    Explicit Typing

    You can force a type by prefixing the column name with #! in the CSV header. Supported types include:

    • string
    • number

    Example:

    string#!appNumber, string#!finished
    201401010002, true

    Result:

    {
      "appNumber": "201401010002",
      "finished": "true"
    }
  3. Convert CSV to nested JSON structures

    master

    By default, csvtojson supports converting CSV headers into nested JSON objects and arrays using dot notation (e.g., fieldA.children.0.name).

    Example Header Mapping:

    • fieldA.title $\rightarrow$ { "fieldA": { "title": "..." } }
    • fieldA.address.0 $\rightarrow$ { "fieldA": { "address": ["..."] } }

    To disable this behavior and keep keys flat (e.g., {"a.b": 1}), set flatKeys: true in the configuration object.

    /**
    csvStr: 
    a.b,a.c
    1,2
    */
    csv({flatKeys:true})
    .fromString(csvStr)
    .on('json',(jsonObj)=>{
    	//{"a.b":1,"a.c":2}  rather than  {"a":{"b":1,"c":2}}
    });
  4. Create complex nested JSON structures from CSV headers

    master

    The library can replicate complex JSON graphs using specific header naming conventions:

    • Dot notation (.): Represents nested objects. field1.field2 becomes { "field1": { "field2": value } }.
    • Square brackets ([]): Represents arrays. field1.field2[0].name creates an array of objects.
    • Implicit Arrays: Different columns with the same header name are automatically added to the same array.

    Note: To prevent information loss in arrays containing objects with multiple fields, you should explicitly include the index (e.g., field[0].name, field[1].name).

    To prevent the parser from interpreting these headers as literal keys, use the --flatKeys=true flag.

  5. Enable Multi-CPU Core support

    master

    To prevent the main Node.js process from blocking during large CSV parsing, you can enable experimental multi-core support. This spawns worker processes to handle the parsing work.

    Usage:

    • Library: Pass workerNum (integer $\ge 1$) in the options object.
    • CLI: Use the --workerNum flag.

    Limitations:

    • Does not support columns containing line breaks.
    • You cannot use custom functions in the colParser parameter because worker processes cannot access them.
  6. Convert CSV to Nested JSON

    master
    By default, csvtojson supports converting CSV headers into nested JSON structures using dot notation (e.g., fieldA.title or fieldA.children.0.name). This allows for complex objects and arrays to be represented directly from a flat CSV file.
  7. Transform JSON results asynchronously

    master

    Asynchronous transformations (e.g., fetching data from a database for each row) can be achieved in two ways:

    1. Using the record_parsed event with an Async Queue

    This is recommended when each row requires external data. You can use a library like async to manage concurrency.

    Note: This approach will not change the JSON object that is pushed downstream by the converter itself. It is used for side effects or external processing.

    2. Using a Writable/Transform Stream

    Create a custom Node.js Writable or Transform stream to process the data asynchronously as it flows through the pipeline.

    var Conv=require("csvtojson").Converter;
    var async=require("async");
    var rs=require("fs").createReadStream("path/to/csv");
    var q=async.queue(function(json,callback){
      require("request").get("http://myserver/user/"+json.userId,function(err,user){
        json.user=user;
        callback();
      });
    },10);
    
    q.saturated=function(){
      rs.pause();
    }
    q.empty=function(){
      rs.resume();
    }
    
    var conv=new Conv({construct:false});
    conv.transform=function(json){
      q.push(json);
    };
    conv.on("end_parsed",function(){
      q.drain=function(){}
    })
    rs.pipe(conv);
  8. Upgrade from csvtojson v1 to v2

    master

    Upgrading to v2 involves several breaking changes. Ensure your code accounts for the following:

    • Node.js Version: Requires Node.js >= 4.0.0.
    • Event Replacement: The events 'csv', 'json', 'record_parsed', and 'end_parsed' are removed. Use .subscribe() for line-by-line processing and .then() for the final result.
    • Callback Removal: fromFile, fromStream, and fromString no longer accept callbacks. Use .then() or await.
    • Column Filtering: ignoreColumns and includeColumns now accept only RegExp (previously they accepted arrays).
    • Transformation: The .transf method is removed. Use .subscribe() instead for result transformation.
    • Line Numbering: Line numbers now start at 0 instead of 1.
    • End Event: The 'end' event may not emit if there is no downstream. Use the 'done' event to detect when parsing is finished.
    • Worker Removal: The Worker feature has been removed to simplify the architecture.
  9. Convert CSV from a file using the Library API

    master

    You can convert CSV files using the Converter class in two ways: via a File stream or the fromFile convenience method.

    Using File Stream

    Pipe a fs.createReadStream into a Converter instance. The end_parsed event is emitted once the entire file is processed.

    Using fromFile

    A more direct method that accepts a file path and a callback function.

    // Using File Stream
    var Converter = require("csvtojson").Converter;
    var converter = new Converter({});
    
    // end_parsed will be emitted once parsing finished
    converter.on("end_parsed", function (jsonArray) {
       console.log(jsonArray); // result jsonarray
    });
    
    require("fs").createReadStream("./file.csv").pipe(converter);
    
    // Using fromFile
    var Converter = require("csvtojson").Converter;
    var converter = new Converter({});
    converter.fromFile("./file.csv", function(err, result) {
      // handle result
    });
    var Converter = require("csvtojson").Converter;
    var converter = new Converter({});
    
    //end_parsed will be emitted once parsing finished
    converter.on("end_parsed", function (jsonArray) {
       console.log(jsonArray); //here is your result jsonarray
    });
    
    //read from file
    require("fs").createReadStream("./file.csv").pipe(converter);
    
    // OR
    
    var Converter = require("csvtojson").Converter;
    var converter = new Converter({});
    converter.fromFile("./file.csv",function(err,result){
    
    });
  10. Convert a CSV file using Streams

    master

    Since version 0.3, the Converter class inherits from Node.js stream.Transform. To convert a file, create a file read stream and pipe it into a new Converter instance.

    Use the end_parsed event to receive the final parsed JSON object once the stream has finished processing.

    var fs = require("fs");
    var Converter = require("csvtojson").Converter;
    var fileStream = fs.createReadStream("./file.csv");
    
    // new converter instance
    var converter = new Converter({constructResult:true});
    
    // end_parsed will be emitted once parsing finished
    converter.on("end_parsed", function (jsonObj) {
       console.log(jsonObj); // your result json object
    });
    
    // read from file
    fileStream.pipe(converter);
  11. Use Promise and Async/Await with csvtojson

    master

    In v2, csvtojson supports native Promises and async/await syntax. Methods like fromFile, fromStream, and fromString no longer accept callbacks; instead, you should use .then() or await to handle the resulting JSON array.

    // Using Promises
    csv()
    .fromFile(myCSVFilePath)
    .then((jsonArray) => {
      // handle success
    }, errorHandle);
    
    // Using async/await
    const jsonArray = await csv().fromFile(myCSVFilePath);
    
    // Chaining with other promises
    request.get(csvUrl)
    .then((csvdata) => {
      return csv().fromString(csvdata);
    })
    .then((jsonArray) => {
      // handle final json array
    });
    // Promise
    csv()
    .fromFile(myCSVFilePath)
    .then((jsonArray)=>{
    
    }, errorHandle);
    
    // async / await
    const jsonArray= await csv().fromFile(myCSVFilePath);
    
    // Promise chain
    request.get(csvUrl)
    .then((csvdata)=>{
      return csv().fromString(csvdata)
    })
    .then((jsonArray)=>{
    
    })
  12. Install csvtojson

    master

    You can install csvtojson either as a global command-line tool or as a local dependency for your Node.js project.

    To install globally:

    npm install -g csvtojson

    To install as a project dependency:

    npm install csvtojson --save
    npm install csvtojson --save