Snowfakery

Snowfakery is a tool designed to generate fake data with complex relationships between tables. It produces unique and random data for every row (similar to a snowflake).

Key Characteristics:

• Configuration: Data generation logic is defined in YAML recipe files.
• Outputs: Snowfakery can output data to stdout or any database supported by SQLAlchemy.
• Salesforce Integration: When used within CumulusCI, Snowfakery can output data directly to a Salesforce org.

You can create values for computations that do not appear in the final output (CSV, database, or Salesforce) by prefixing the field or object name with two underscores (__). This is useful for intermediate variables or temporary object templates used to calculate complex fields.

- object: Dates
  fields:
    __total_months: 48
    __first_month: ${{today - relativedelta(months=__total_months)}}
    ProgramStartDate: ${{__first_month}}

Snowfakery operates on three distinct levels of data lifetime, which is critical when designing recipes for large-scale data generation:

1. Iterations: A single top-to-bottom run-through of the recipe. Data that only survives an iteration is called a Transient.
2. Continuations: A pattern used to generate data in chunks (portions) and hand them off to a parent application (like CumulusCI). This uses a 'start-stop-start-stop' pattern. When you start/stop, a new Interpreter object is created. Data that survives a single continuation (like FakerData objects) is Interpreter Managed.
3. Globals: Data that is saved to a file and loaded back in to survive across different continuations (different processes). This includes just_once objects and specific persistent data.

The random_choice function allows you to assign weights to different options. These weights do not need to sum to 100; Snowfakery calculates the probability based on the relative sum of all provided weights.

StageName:
  random_choice:
    Closed Won: 5
    In Progress: 3
    New: 4

The friends feature allows you to create a rule where creating one object automatically triggers the creation of others. This is useful for establishing an implicit relationship in quantity (e.g., "For every Person, create two Animals").

Note: While friends creates the objects, it does not automatically create a formal relational link (like a foreign key) unless combined with other relationship features.

- object: Person
  count: 3
  fields:
    name:
      fake: name
  friends:
    - object: Animal
      count: 2
      fields:
        name:
          fake: FirstName

An Object Template instructs Snowfakery how to create rows (records). The main properties are:

• object: The type declaration that determines the target table or file.
• count: The number of rows to create. This can be a static integer, a randomized value, or a computed value using Function Blocks or the Formula Language.
• fields: The data values to populate in the row.

You can define multiple Object Templates for the same row type to generate different subsets of data (e.g., two different Person templates to create different age groups).

- object: Person
  count: 10
  fields:
    name:
      fake: name
    age:
      random_number:
        min: 12
        max: 95

Snowfakery uses RowHistory to enable randomized access to objects generated within a single continuation. This is primarily used for the random_reference feature, allowing you to pick a random object or nickname from previously generated rows.

Key constraints:

• Scope: Row History only survives for the duration of a single continuation. You cannot use random_reference to link to objects created in a previous continuation run.
• Implementation: It is backed by a SQLite database. If the dataset is too large for RAM, it spills to disk.
• Performance: Large datasets with many fields can consume significant disk space. For example, a recipe generating 100M rows with large string fields will result in a large SQLite file on disk.

Snowfakery supports creating Person Accounts. Because Person Accounts involve a relationship between an Account and a Contact, you use the Salesforce.SpecialObject function to generate a placeholder PersonContact object.

CumulusCI handles the heavy lifting of resolving these references during the data load process. Ensure the Account object is configured to load before any other objects that reference the PersonContactId (you may need a .load.yml file to control this order).

- plugin: snowfakery.standard_plugins.Salesforce
- object: Account
  fields:
    FirstName: 
      fake: FirstName
    LastName: 
      fake: LastName
    PersonContactId:
      Salesforce.SpecialObject: PersonContact

- object: User
  fields:
    ContactId:
      reference: Account.PersonContactId

Snowfakery features a special 'update mode' that allows you to loop over every row of an input file (like a CSV), apply transformations or additions via a recipe, and output the results.

Requirements for an update recipe:

• Must have a single top-level object.
• The top-level object must not have a count attribute.
• The recipe can optionally take options.
• The output will contain the same number of rows as the input file.

To access data from the input file within your recipe, use the ${{input.FIELD_NAME}} syntax.

CLI Command: Use the --update-input-file flag to specify the source data and --output-format to define the result format.

$ snowfakery examples/updates/update_contacts.recipe.yml --update-input-file examples/datasets/addresses.csv --output-format csv

Macros allow you to define a reusable group of fields that can be included in multiple objects. This prevents repetition and allows for easy updates to common field sets.

Key behaviors:

• Use the include key within an object to apply a macro.
• You can include multiple macros by separating them with commas (e.g., include: macro1, macro2).
• Macros can include other macros.
• Overriding: If a field or friends list is declared in a macro, it will be overridden by fields/friends declared in subsequent macros or by fields/friends declared directly in the object definition.

To create a macro, define it with the macro key and specify its fields.

- macro: canine
  fields:
    sound: barks
    legs: 4
    family: Caninae

- object: Animal
  include: canine
  fields:
    species: dog
    home: inside

Snowfakery can be extended using Python code via two mechanisms:

• Plugins: Add new functions to Snowfakery.
• Providers: Add new capabilities to the fake: keyword (the Faker library).

To include them in a recipe, use the plugin key under a list item at the top level of your YAML file.

- plugin: package.module.classname

You can parameterize your recipes by declaring option blocks. This allows you to pass values (numbers, strings, booleans) from the CLI into your formulas.

1. Declare the option in the YAML: - option: my_var default: 10.
2. Reference it in formulas: ${{my_var / 2}}.
3. Pass it via CLI: --option my_var 20.

- option: num_accounts
  default: 10

- object: Account
  count: ${{num_accounts}}

--option num_accounts 10