C++ Neural Core
A deep learning framework core built from scratch in pure C++ — tensor engine, reverse-mode autograd, and convolutional networks, with no external AI frameworks.
Overview
Neural Core is a self-contained deep learning core written in C++20. It implements the full stack of a modern framework — an N-dimensional tensor type, a dynamic reverse-mode autodiff engine, layers, losses, and an optimizer — entirely from first principles, with no PyTorch, TensorFlow, or any other ML library linked.
The framework is PyTorch-flavored by design: a define-by-run computational graph is built during the forward pass, parameters carry their own accumulated gradients, and a scalar loss is turned into gradients by calling backward(). The public API deliberately mirrors torch.nn and torch.optim (Module, Sequential, Linear, Conv2D, CrossEntropyLoss, SGD), so code reads almost like its PyTorch equivalent while every operation underneath is hand-written.
At a glance
Motivation
Understand deep learning from the inside out
The goal was to stop treating deep learning as a black box. Frameworks like PyTorch are wonderfully convenient, but that convenience hides the mechanics that make them work: how tensors are laid out in memory, how broadcasting interacts with gradients, how a computation graph is recorded and traversed in reverse, and where the real performance costs live.
Building each piece by hand — and proving it correct against finite differences and a PyTorch reference — turned those abstractions into concrete, debuggable code. Once the feature set worked, the focus shifted from “does it learn?” to “how fast?”
Architecture
Layers, built from scratch
Tensor engine. N-dimensional (rank ≥ 1) row-major storage with precomputed strides and double precision. NumPy-style broadcasting aligns operands from the right and inserts stride-0 dimensions; gradients are un-broadcast during the backward pass by summing over the dimensions that were expanded, so shapes always match the original inputs.
Autograd. Each differentiable op returns a tensor that stores its creator node, so the graph is defined by running the forward pass. Tensor::backward() seeds a gradient of 1.0 on the scalar loss and walks the graph in reverse, applying the chain rule and accumulating gradients into leaf tensors. Edges use std::shared_ptr, so intermediate activations stay alive for the whole backward pass.
Modules. A Module base class mirrors torch.nn.Module; Sequential chains modules and aggregates parameters. Conv2D is implemented as im2col → batched matmul → bias add, so its gradients reuse the generic matmul/reshape/add autodiff nodes.
Loss & optimizer. CrossEntropyLoss fuses softmax + negative log-likelihood into one graph node (it takes raw logits, matching nn.CrossEntropyLoss), and SGD reads gradients straight off the parameter tensors.
// The public API mirrors torch.nn / torch.optim.
auto net = Sequential{
Linear{784, 256}, ReLU{}, Linear{256, 10}
};
Tensor logits = net.forward(x);
Tensor loss = CrossEntropyLoss{}(logits, y);
loss.backward();
SGD optimizer{ net.parameters(), /*lr=*/0.01 };
optimizer.step();Validation
Prove it, don't hope it
Correctness is checked at two levels: Catch2 unit tests per component, and integration tests that gradient-check the analytic backward pass through the full chain against finite differences (tolerance 1e-5). Forward and backward are also validated numerically against a PyTorch reference script.
- →Gradient-check Linear → Sigmoid → MSE and Linear → CrossEntropyLoss
- →SGD provably minimizes a convex regression and learns XOR
- →Full CNN pipeline trains end-to-end; Conv2D is batch-independent
- →CI runs configure, build, and ctest on every pull request
Training results
It learns
The XOR MLP drops MSE from ~0.250 to ~0.001 over 5,000 epochs, matching the PyTorch reference almost digit-for-digit. On MNIST, both the MLP (784 → 256 → 128 → 10, 235,146 parameters) and a small convolutional network train from scratch on a 1,000/250 subset — the MLP reaching 92.5% test accuracy in 10 epochs, the CNN 97.6%.
MNIST · train loss
MNIST · test accuracy
Performance
Then it got fast
Once correctness was established, the focus shifted from “does it learn?” to “how fast?” Profiling the CNN training step surfaced a one-line fix (see Challenges) that cut Conv2D forward by ~38% and the full CNN forward pass by ~21% — with no change in results.
Per-layer timing (v1.1.1, Apple M3, Release, 10 reps)
| Layer | Forward (ms) | Backward (ms) |
|---|---|---|
| Conv2D | 11.32 | 15.44 |
| Linear1 | 12.42 | 22.54 |
| MaxPool2D | 2.59 | 3.07 |
| Linear2 | 0.042 | 0.096 |
| Flatten | 0.027 | 0.083 |
Challenges
What building it taught me
Broadcasting + autograd
Elementwise gradients needed a dedicated un-broadcast step: any dimension stretched from size 1 to N must have its gradient summed back down. Invisible until a shape mismatch or a subtly wrong gradient appears.
A counter-intuitive benchmark that turned out to be real
Conv2D's forward pass was slower than its backward pass — the opposite of what the matmul cost model predicts. Profiling revealed repeated heap allocations of a coordinate vector inside the elementwise operators. Hoisting one std::vector<size_t> out of the hot loop cut Conv2D forward by ~38%.
A “bug” that was actually correct behavior
Some XOR runs failed to train because every ReLU unit had died at initialization — the classic Dying ReLU problem. The zero gradient was exactly what the ReLU backward was supposed to produce, which doubled as a sanity check on the implementation.
Validate incrementally
Each capability was unit- and gradient-checked in isolation, then proven on a toy problem, and only then pointed at real data — keeping failures attributable to a single component.
Takeaway