micrograd

Install the micrograd library using pip to get access to the autograd engine and neural network library.

pip install micrograd

To visualize the computational graph in micrograd, you must install the graphviz system library and the Python wrapper.

On macOS using Homebrew:

brew install graphviz

Using pip:

pip install graphviz

When defining a custom loss function, you can incorporate regularization and specific loss types using Value operations:

• SVM Max-Margin Loss: Calculated as (1 + -yi * scorei).relu() for each sample.
• L2 Regularization: Sum the squares of all parameters in the model and multiply by an alpha coefficient: alpha * sum((p*p for p in model.parameters())).
• Total Loss: The sum of the data loss and the regularization loss.

# svm "max-margin" loss
losses = [(1 + -yi*scorei).relu() for yi, scorei in zip(yb, scores)]
data_loss = sum(losses) * (1.0 / len(losses))

# L2 regularization
alpha = 1e-4
reg_loss = alpha * sum((p*p for p in model.parameters()))

total_loss = data_loss + reg_loss

To train a model in MicroGrad, follow the standard optimization loop:

1. Forward Pass: Compute the total_loss (which must be a Value object).
2. Zero Gradients: Call model.zero_grad() to clear previous gradients.
3. Backward Pass: Call total_loss.backward() to compute gradients via autograd.
4. Update Parameters: Iterate through model.parameters() and manually update the .data attribute using the calculated .grad and a learning rate.

Note: MicroGrad does not provide a built-in optimizer like Adam; you implement the update step manually.

for k in range(100):
    # 1. Forward
    total_loss, acc = loss()

    # 2. Backward
    model.zero_grad()
    total_loss.backward()

    # 3. Update (Stochastic Gradient Descent)
    learning_rate = 0.01
    for p in model.parameters():
        p.data -= learning_rate * p.grad

You can visualize the internal structure of a micrograd.nn.Neuron by passing its output through draw_dot after calling .backward().

import random
from micrograd import nn
from micrograd.engine import Value

random.seed(1337)
n = nn.Neuron(2)
x = [Value(1.0), Value(-2.0)]
y = n(x)
y.backward()

dot = draw_dot(y)
dot.render('neuron_graph')

The core of micrograd is the Value class from micrograd.engine. It implements reverse-mode autodiff over scalar values. You can perform standard arithmetic operations and activation functions (like .relu()) on Value objects. After a forward pass, calling .backward() on the output node computes the gradients for all nodes in the computational graph, which can be accessed via the .grad attribute.

from micrograd.engine import Value

a = Value(-4.0)
b = Value(2.0)
c = a + b
d = a * b + b**3
c += c + 1
c += 1 + c + (-a)
d += d * 2 + (b + a).relu()
d += 3 * d + (b - a).relu()
e = c - d
f = e**2
g = f / 2.0
g += 10.0 / f

print(f'{g.data:.4f}') # prints the outcome of the forward pass
g.backward()
print(f'{a.grad:.4f}') # prints the numerical value of dg/da
print(f'{b.grad:.4f}') # prints the numerical value of dg/db

To verify the correctness of the calculated gradients, you can run the unit tests. Note that you must have PyTorch installed, as the tests use it as a reference implementation for verification.

python -m pytest

MicroGrad operates on Value objects. When passing data through a model, you must convert your raw numerical inputs (e.g., from a NumPy array) into a list of Value objects.

Because MLP is a container of layers, you can iterate over your inputs and call the model instance directly on each input list to get the corresponding Value scores.

from micrograd.engine import Value

# Assuming Xb is a batch of numpy arrays
inputs = [list(map(Value, xrow)) for xrow in Xb]

# Forward pass to get scores
scores = list(map(model, inputs))

You can create a neural network using the MLP class from micrograd.nn. The constructor takes the input dimension as the first argument, followed by a list representing the number of neurons in each hidden layer and the output layer.

To inspect the model or count the total number of trainable parameters, use the print() function on the model instance and call the .parameters() method.

from micrograd.nn import MLP

# Create a 2-layer neural network with 2 inputs, two hidden layers of 16 neurons, and 1 output
model = MLP(2, [16, 16, 1])

print(model)
print("number of parameters", len(model.parameters()))