EVL Tool Documentation

EVL is a high-performance, lightweight ETL (Extract-Transform-Load) tool written in C++ and designed exclusively for Linux. It follows Unix philosophies of interoperability and modularity, utilizing a strict command-line approach for efficient development and offering a Graphical User Interface via the EVL Manager. Key capabilities include the use of templates for high-level abstraction and the ability to dynamically create jobs.

EVL is a high-performance, lightweight ETL (Extract–Transform–Load) tool written in C++ for Linux. It follows Unix philosophies of interoperability and modularity, utilizing templates and dynamic job creation to handle complex data workflows. It is managed via a graphical user interface called EVL Manager and is distributed as rpm, deb, or tgz packages.

EVL Data Anonymization is a microservice designed for the fast, automated anonymization and pseudonymization of production data to ensure GDPR compliance and protect commercially sensitive information. It supports various data sources including CSV files, databases (Oracle, Teradata, PostgreSQL, etc.), and Hadoop streaming data such as Kafka. The tool uses a metadata-driven approach and supports parallelization and custom functions.

The data anonymization process follows a specific sequence of steps to move from a raw data source to anonymized output. The general workflow is: create a project, add a data source, configure the anonymization preferences in CSV files, validate the configuration, build the jobs, and finally execute the anonymization jobs or a combined workflow.

# 1. Create project
evl datahub project new <project_dir>

# 2. Add source (folder or database)
evl datahub source new <source_name> --guess-from-csv <path_to_folder>

# 3. (Manual step) Edit the generated config CSV files

# 4. Validate config
evl datahub check <config_file>

# 5. Generate jobs and workflow
evl datahub build <config_file>

# 6. Run specific job
evl run/datahub/<table_or_file>.evl

# OR run the entire workflow for a source
evl run workflow/datahub/<source_name>.ewf

EVL provides a unified ecosystem for metadata-driven data automation designed to standardize data engineering. It transforms complex requirements—ranging from ASN.1 ingestion to automated orchestration—into repeatable and governed data pipelines.

The EVL Data Anonymization Microservice provides automated and cost-effective anonymization of data sets. It is primarily used to ensure production data complies with GDPR requirements and to protect commercially sensitive data when providing access to developers, testers, and outside contractors.

The EVL Data Generation Microservice is used to create simulated test data that mimics real-life patterns and volumes for application development, ETL process implementation, and stress testing. It supports reading from and writing to various sources, including CSV files, databases (Oracle, Teradata, SQL Server), and Hadoop streaming data (Kafka, Flume). Key features include automatic random data generation based on data types, customization of data patterns (min-max intervals, null probability), and parallel job execution using low-level IO techniques for high performance.

EVL is a high-performance, lightweight ETL (Extract-Transform-Load) tool written in C++ for Linux. It follows Unix philosophies of interoperability and modularity, utilizing templates and dynamic job creation to manage data workflows. It is available as an rpm/deb package or a tgz archive and includes a graphical user interface called EVL Manager.

EVL is a high-performance, lightweight ETL (Extract–Transform–Load) tool written in C++ for Linux. It follows Unix philosophies of interoperability and modularity, utilizing templates and dynamic job creation to handle complex data workflows. It includes a Graphical User Interface called EVL Manager.

EVL Workflow is a code-based orchestration tool that manages workflows as Directed Acyclic Graphs (DAGs). Workflows are defined in text files resolved as Bash scripts, allowing tasks to be executed in a specific order, in parallel, or on specific target hosts (local, SSH, or Kubernetes). The orchestrator manages tasks based on priorities, maximum parallel execution limits, CPU/memory consumption history, maximum run times, and retry counts.

EVL is a high-performance, lightweight ETL (Extract-Transform-Load) tool written in C++ for Linux. It follows Unix philosophies of interoperability and modularity, utilizing templates and dynamic job creation to handle complex data workflows. It is available as rpm/deb packages or tgz archives and includes a graphical user interface called EVL Manager.