Neural Networks and Deep Learning Implementation

This repository contains code samples for the book "Neural Networks and Deep Learning".

Important Compatibility Notes:

• Python Version: The code in this repository is written for Python 2.6 or 2.7. It is not compatible with Python 3.
• Python 3 Alternative: If you require Python 3.8-3.10 compatibility, use the version hosted at https://github.com/unexploredtest/neural-networks-and-deep-learning.
• Theano Dependency: The program src/network3.py requires Theano version 0.6 or 0.7. It will not work with later versions of Theano without manual modification.

The main() function provides a complete execution flow for the naive classifier:

1. Loads MNIST data using mnist_loader.load_data().
2. Computes average darknesses for each digit using the training set.
3. Tests the classifier against the test set by comparing the predicted digit (based on darkness) to the actual digit.
4. Prints the number of correct classifications to the console.

python src/mnist_average_darkness.py

The mnist_svm.py script provides a baseline implementation for recognizing handwritten digits from the MNIST dataset using a Support Vector Machine (SVM) classifier from sklearn.

When executed as a script, it performs the following steps:

1. Loads the MNIST dataset using mnist_loader.load_data().
2. Initializes and trains an sklearn.svm.SVC classifier on the training data.
3. Predicts labels for the test dataset.
4. Calculates and prints the number of correct predictions out of the total test samples.

python src/mnist_svm.py

Return the number of test inputs for which the neural network outputs the correct result. The output is determined by the index of the neuron in the final layer with the highest activation (np.argmax).

# test_data is a list of (input_array, label_index)
correct_count = n.evaluate(test_data)

Use plot_errors to create a visual grid of images from the test set that the ensemble model failed to classify correctly.

Parameters:

• error_locations: The list of indices of misclassified images (typically returned by ensemble()).
• erroneous_predictions (optional): A list of the incorrect predictions made by the ensemble. If provided, these are displayed as text on the images.

Returns a matplotlib figure object.

# Plotting errors from an ensemble
error_locations, erroneous_predictions = ensemble(nets)
plt = plot_errors(error_locations, erroneous_predictions)
plt.savefig("ensemble_errors.png")

The Network class accepts a cost function class that implements two static methods:

1. fn(a, y): Returns the cost for output a and desired output y.
2. delta(z, a, y): Returns the error $\delta$ from the output layer.

Available classes:

• QuadraticCost: Uses the squared Euclidean norm.
• CrossEntropyCost: Uses the cross-entropy cost function, which is generally better for classification tasks. It includes numerical stability handling via np.nan_to_num.

from network2 import QuadraticCost, CrossEntropyCost

# Example of how these are passed to the Network
net_quad = Network([784, 30, 10], cost=QuadraticCost)
net_ce = Network([784, 30, 10], cost=CrossEntropyCost)

The Network class implements a feedforward neural network using stochastic gradient descent (SGD). To use it, provide a list of layer sizes (e.g., [784, 30, 10] for a network with 784 inputs, 30 hidden neurons, and 10 outputs) and an optional cost function class.

Training is performed using the SGD method, which supports mini-batch learning, regularization via lmbda, and monitoring of cost and accuracy for both training and evaluation datasets.

from network2 import Network, CrossEntropyCost

# Define network architecture: 784 inputs, 30 hidden, 10 outputs
sizes = [784, 30, 10]
# Initialize with CrossEntropyCost (default)
net = Network(sizes, cost=CrossEntropyCost)

# training_data is a list of (x, y) tuples
# eta is the learning rate, lmbda is regularization
et.SGD(training_data, epochs=30, mini_batch_size=10, eta=3.0, lmbda=0.1)

The run_experiments function is a convenience wrapper that executes the full suite of experiments described in Chapter 6 of 'Neural Networks and Deep Learning'.

Requirements:

• Access to expanded MNIST data (can be generated via expand_mnist.py).

Experiments included:

• Shallow networks.
• Basic Conv + FC architectures.
• Conv-only architectures (no FC).
• Double Conv + FC architectures (with Sigmoid and ReLU).
• Regularized double convolution experiments.
• Experiments using expanded MNIST data with varying FC layer sizes.
• Dropout-based networks with multiple FC layers.
• Ensemble voting and error visualization.

import conv
conv.run_experiments()

Use load_data_wrapper() to obtain MNIST datasets in a format optimized for neural network training. It returns a tuple of (training_data, validation_data, test_data) with the following structures:

• training_data: A list of 50,000 2-tuples (x, y).
  • x: A 784-dimensional numpy.ndarray (input image).
  • y: A 10-dimensional numpy.ndarray representing a unit vector (one-hot encoded) for the correct digit.
• validation_data: A list of 10,000 2-tuples (x, y).
  • x: A 784-dimensional numpy.ndarray (input image).
  • y: An integer representing the digit value (0-9).
• test_data: A list of 10,000 2-tuples (x, y).
  • x: A 784-dimensional numpy.ndarray (input image).
  • y: An integer representing the digit value (0-9).

from mnist_loader import load_data_wrapper

training_data, validation_data, test_data = load_data_wrapper()

Train the neural network using mini-batch stochastic gradient descent.

Parameters:

• training_data: A list of tuples (x, y) representing training inputs and desired outputs.
• epochs: The number of training epochs.
• mini_batch_size: The size of each mini-batch.
• eta: The learning rate.
• test_data (optional): A list of tuples (x, y) used to evaluate the network after each epoch. If provided, the network prints progress (accuracy/test size) to the console.

from network import Network

n = Network([784, 30, 10])
# training_data and test_data should be lists of (numpy_array, numpy_array)
n.SGD(training_data, epochs=30, mini_batch_size=10, eta=3.0, test_data=test_data)

The avg_darknesses function calculates the average darkness for each digit (0-9) based on a training dataset. Darkness for an individual image is defined as the sum of the darkness values of all its pixels.

Returns a defaultdict where keys are the digits (0-9) and values are the corresponding average darknesses.

from collections import defaultdict

# training_data is expected to be a tuple (images, labels)
# where images is a list/array of pixel values
avgs = avg_darknesses(training_data)