NeuralAmpModelerCore Documentation

A single Layer in the NAM architecture performs a multi-stage computation involving dilated convolutions, conditioning, and residual/skip connections.

Computation Stages

1. Input Convolution: The input undergoes a dilated 1D convolution. This can be bookended by optional Pre-FiLM and Post-FiLM modules that modulate the signal using the conditioning signal.
2. Input Mixin: The conditioning input is processed via a 1x1 convolution and optionally modulated by Pre-FiLM and Post-FiLM modules before being added to the main convolution output.
3. Sum and Pre-Activation FiLM: The convolution output and mixin output are summed. An optional Pre-Activation FiLM can then modulate this sum.
4. Activation: The summed signal passes through an activation stage. The behavior depends on the GatingMode:
   • GatingMode::NONE: A simple activation function is applied.
   • GatingMode::GATED: The output channels are doubled (2 * bottleneck). The top half uses a primary activation, and the bottom half uses a secondary activation (e.g., sigmoid). The two are multiplied element-wise.
   • GatingMode::BLENDED: A weighted blend is performed: output = alpha * activated_input + (1 - alpha) * pre_activation_input, where alpha is derived from the secondary activation. An optional Post-Activation FiLM can be applied after this stage.
5. 1x1 Convolution: A 1x1 convolution reduces the bottleneck channels back to the layer's channel count. This can also be followed by an optional Post-1x1 FiLM.
6. Head 1x1 (Optional): If configured, a head1x1 convolution processes the activated output for the skip connection, allowing the skip path to project to an arbitrary dimension.
7. Residual and Skip Connections:
   • Residual Connection: The input is added to the 1x1 convolution output to form the signal for the next layer: output_next_layer = input + 1x1_output.
   • Skip Connection: The activated output (or the head1x1 output if present) is sent to the model head: output_head = activated_output (or head1x1 output).

Data Dimensions

• Input/Output: (channels, frames)
• Bottleneck: b
• Head Output: dh (equals b if no head1x1 is used; otherwise determined by head1x1 output channels).
• Gating factor g: 2 if GATED or BLENDED modes are used, otherwise 1.

The NeuralAmpModeler (NAM) WaveNet is a feedforward, dilated convolutional neural network used for audio regression. While inspired by the original WaveNet, it differs in several key ways:

• Feedforward vs. Autoregressive: NAM is feedforward (used for regression), whereas the original WaveNet is autoregressive (used for generation).
• Stacked Structure: The model is composed of multiple LayerArray objects, effectively creating a "stacked WaveNet."
• Conditioning: It uses optional DSP processing of the input to generate conditioning signals that modulate the layers via FiLM (Feature-wise Linear Modulation).
• Modified Layers: The layers include additional skip connections (like input mixin) and optional gated activations. In many standard configurations (e.g., A1 standard/lite/feather/nano), gated activation is frequently omitted.
• Advanced Modules: Recent versions include FiLMs, bottlenecks, and arbitrary conditioning DSP modules to embed input signals more effectively.

The WaveNet processing pipeline manages the flow of audio through conditioning signals and multiple LayerArray stages. The pipeline consists of three main phases:

1. Condition Processing: If a _condition_dsp is provided, the input is processed through it to generate a conditioning signal. If no DSP is provided, the input is used directly as the condition.
2. LayerArray Processing:
   • The first LayerArray processes the input with zeroed head inputs.
   • Subsequent LayerArrays process the output of the previous array (GetLayerOutputs()) and incorporate the head outputs from the previous array (GetHeadOutputs()) as part of their input.
3. Head Scaling and Output: The final head output from the last LayerArray is scaled and written to the output buffers.

This architecture allows for complex conditioning (e.g., using a convolution or RNN as a condition module) to influence the audio processing at every stage of the hierarchy.

// Step 1: Condition processing
void WaveNet::_process_condition(const int num_frames) {
    if (this->_condition_dsp != nullptr) {
        // Process input through condition DSP
        this->_condition_dsp->process(/* input */, /* output */, num_frames);
        // Copy output to condition buffer
    } else {
        // Use input directly as condition
        this->_condition_output = this->_condition_input;
    }
}

// Step 2: LayerArray processing
// First layer array
this->_layer_arrays[0].Process(input, condition, num_frames);

// Subsequent layer arrays
for (size_t i = 1; i < this->_layer_arrays.size(); i++) {
    Eigen::MatrixXf& prev_output = this->_layer_arrays[i-1].GetLayerOutputs();
    Eigen::MatrixXf& prev_head = this->_layer_arrays[i-1].GetHeadOutputs();
    this->_layer_arrays[i].Process(prev_output, condition, prev_head, num_frames);
}

// Step 3: Head scaling and output
Eigen::MatrixXf& final_head = this->_layer_arrays.back().GetHeadOutputs();
// Apply head scale and write to output buffers

A LayerArray is a sequential chain of Layer objects that processes data through multiple stages while accumulating "head outputs" via skip-out connections. The computation follows three distinct steps:

1. Rechanneling: The input is projected to match the specific channel count required by the layers using _rechannel.process_.
2. Layer Processing Loop:
   • The first layer processes the rechanneled input.
   • Subsequent layers process the residual output from the previous layer (GetOutputNextLayer()).
   • Every layer's "head output" (GetOutputHead()) is accumulated into a central head buffer.
3. Head Rechanneling: The accumulated head outputs are projected to the final output dimension for the LayerArray using _head_rechannel.process_.

This structure allows for deep residual processing where information from every layer is preserved and combined at the end.

// Step 1: Rechanneling
this->_rechannel.process_(layer_inputs, num_frames);
Eigen::MatrixXf& rechannel_output = _rechannel.GetOutput();

// Step 2: Layer processing loop
for (size_t i = 0; i < this->_layers.size(); i++) {
    if (i == 0) {
        // First layer consumes the rechannel output buffer
        this->_layers[i].Process(rechannel_output, condition, num_frames);
    } else {
        // Subsequent layers consume the previous layer's output
        Eigen::MatrixXf& prev_output = this->_layers[i - 1].GetOutputNextLayer();
        this->_layers[i].Process(prev_output, condition, num_frames);
    }
    
    // Accumulate head output from this layer
    this->_head_inputs.leftCols(num_frames).noalias() += 
        this->_layers[i].GetOutputHead().leftCols(num_frames);
}

// Step 3: Head rechanneling
_head_rechannel.process_(this->_head_inputs, num_frames);