website·Indexed 33 days ago
https://docs.shadowtraffic.ioDocumentation for ShadowTraffic, a synthetic data generation tool. It covers the ShadowTraffic API, including Generators, Functions (Scalar, Math, Selection, Date/time, and Transformation), and Runtime configuration. It also provides guides for establishing connections to Apache Kafka, MotherDuck, Google Cloud Storage, and Amazon S3.
The serialize function modifier is used to convert a generated value into its final type before it is written to an output target. This is necessary when ShadowTraffic cannot implicitly determine the correct serialization format or when a specific target type is required.
Important: Once a value is serialized, it is converted to its final representation (e.g., a byte sequence) and cannot be further manipulated or operated on by other ShadowTraffic functions.
Shadow Traffic allows the generation of strings using a templating syntax #{}. The system resolves expressions within these brackets in the following order of precedence:
This mechanism is used to create dynamic data for traffic generation.
fork generator dynamically creates new instances of a generator, each identified by a unique key. Only one instance of a generator runs at a time for each unique fork key; if a duplicate key is produced while an instance is already running, it is ignored. The current fork key is accessible within the generator using the variable forkKey. Note that key cannot be null._gen: "ai" function generates free text by calling an LLM provider. It automatically assembles a preprompt containing all variables generated prior to the function call to provide the LLM with additional context. Supported services include Ollama, AWS Bedrock, and Google Vertex.The profile function generates demographically realistic profiles of people, including names, emails, and company details. It uses name distributions based on US voter registrations and company data from the Open Data 500.
Note: The first invocation per ShadowTraffic instance may have a delay of a few seconds while backing data is loaded into memory; subsequent calls are cached and fast.
easingChain stateful function runs a series of easing functions in succession. It is used to model complex value changes over time. By default, each stage runs sequentially until completion. Once the final stage finishes, the value remains pegged at the last easing's from value unless looping is enabled.profile() function was introduced in version 1.15.0 to generate realistic data about people.Shadow Traffic has added support for the following connections across several versions:
subdir key in bucketConfigs. This is typically used with a variable (e.g., from a fork). Each unique subdir value maintains its own independent blob-roll milestones (time/size/events).{
"generators": [
{
"bucket": "sandbox",
"fork": {
"key": ["a", "b", "c"]
},
"bucketConfigs": {
"subdir": {
"_gen": "var",
"var": "forkKey"
},
"keyPrefix": "foo-",
"format": "jsonl"
},
"data": {
"a": { "_gen": "uuid" },
"b": { "_gen": "boolean" }
},
"localConfigs": {
"throttleMs": 200
}
}
],
"connections": {
"s3": {
"kind": "s3"
}
}
}geoWander stateful function generates an infinite, randomized path of coordinates within a specified geospatial bounding box. It is used to simulate movement within a defined area. The function outputs the current latitude, longitude, the duration (in milliseconds) between points for speed-realism throttling, and the navigational heading/bearing in degrees.{
"_gen": "geoWander",
"boundingBox": [
[45, -125],
[50, -70]
],
"speed": 500,
"scale": 0.025
}format to parquet in bucketConfigs. ShadowTraffic attempts to guess the Parquet schema automatically. If the automatic guess fails, you must manually specify the schema using avroSchemaHint within localConfigs.{
"generators": [
{
"bucket": "sandbox",
"bucketConfigs": {
"format": "parquet",
"blobPrefix": "part-"
},
"data": {
"luckyNumber": { "_gen": "oneOf", "choices": [1, 2, 3] }
},
"localConfigs": {
"avroSchemaHint": {
"data": {
"type": "record",
"name": "MyRecordName",
"fields": [
{ "name": "luckyNumber", "type": "int" }
]
}
}
}
}
],
"connections": {
"gcs": {
"kind": "googleCloudStorage",
"connectionConfigs": {
"projectId": "myProject"
}
}
}
}