Forget Python: How I built my own LLM in C++ and everything I learned by failing

What I built: embeddings and a small network

I started by turning words into identifiers with a tokenizer and giving each one a 12-dimensional embedding. The model then combines a question summary with the last eight reply words, passes those features through a 48-unit tanh layer, and calculates a distribution over the vocabulary. One of my first challenges was understanding how to connect those representations without losing track of what information the network received at each step.

This is a contextual multilayer perceptron. It uses neither pretrained weights nor attention. Its scale makes it possible to follow an entire prediction without confusing the experiment with the architecture or capabilities of a modern GPT.

Stable softmax: avoiding overflow

When I reached the network output, I needed to turn scores into probabilities. Softmax applies exponentials and normalizes the result, but I ran into an easy-to-miss detail: computing exp(x) directly can overflow when a score is large. To avoid that, the implementation subtracts the maximum before exponentiation.

Subtracting the same constant from every score does not change the mathematical distribution, but keeps the exponents in a safer range. The function also rejects empty inputs and non-finite scores. This is the complete method used by Dani:

vector<double> Model::softmax(const vector<double>& scores) const {
    if (scores.empty()) return {};
    const double maximum = *max_element(scores.begin(), scores.end());
    vector<double> probabilities(scores.size());
    double total = 0.0;
    for (size_t index = 0; index < scores.size(); ++index) {
        if (!isfinite(scores[index])) return {};
        probabilities[index] = exp(scores[index] - maximum);
        total += probabilities[index];
    }
    for (double& value : probabilities) value /= total;
    return probabilities;
}

How the model retains the question while generating

For each prediction, the model keeps an average of the question embeddings available. The following excerpt comes from context_embedding: it identifies the words before the assistant marker and adds their contribution to the summary.

This approach helped me keep the question available, but it also forced me to accept a limitation: averaging loses its word order. It is not attention and does not guarantee correct interpretation of syntax or negation. Recent reply words, by contrast, are concatenated in order elsewhere in the same function.

    if (separator >= 0) {
        // Pool the complete question separately so it survives long replies and small word shifts.
        const size_t prompt_size = static_cast<size_t>(marker - token_ids.begin() - 1);
        for (auto token = token_ids.begin() + 1; token != marker; ++token) {
            const auto& embedding = embeddings.at(*token);
            for (int dimension = 0; dimension < embedding_size; ++dimension) {
                features[dimension] += embedding[dimension] / static_cast<double>(prompt_size);
            }
        }
        reply_start = static_cast<size_t>(marker - token_ids.begin() + 1);
    }

Backpropagation before updating weights

Training uses cross-entropy and gradient descent. Backpropagation was one of the hardest parts for me to reason through: calculating derivatives was not enough; I also had to respect the order of operations. In train_step, the code calculates gradients for the hidden layer and input features before changing the weights. Mixing updated weights with gradients from the previous forward pass would make the update inconsistent.

The expression 1.0 - hidden[unit] * hidden[unit] applies the tanh derivative to its output. This excerpt shows only that stage; the full function, linked below, also updates the matrices, biases, and embeddings.

    // Compute all upstream gradients before updating any weights.
    vector<double> gradient_hidden(hidden.size(), 0.0);
    for (size_t unit = 0; unit < hidden.size(); ++unit) {
        for (size_t token = 0; token < gradient_scores.size(); ++token) {
            gradient_hidden[unit] += output_weights[unit][token] * gradient_scores[token];
        }
        gradient_hidden[unit] *= 1.0 - hidden[unit] * hidden[unit];
    }
    vector<double> gradient_features(features.size(), 0.0);
    for (size_t input = 0; input < features.size(); ++input) {
        for (size_t unit = 0; unit < hidden.size(); ++unit) {
            gradient_features[input] += hidden_weights[input][unit] * gradient_hidden[unit];
        }
    }

How to compile and reproduce training

After cloning the repository, run these commands from its root directory with a C++17-compatible compiler. The first compiles the program; the second forces training with seed 42 and runs evaluation. These are repository instructions, not a new training run performed for this article.

The documentation describes 284 Spanish training exchanges, with synthetic examples and a small reviewed OpenAssistant sample. Validation and test files are separate. The seed helps reproduce a run, but alone does not guarantee identical results across every platform.

clang++ -std=c++17 -O2 -Wall -Wextra -pedantic -Iinclude/llm \
  src/main.cpp src/model.cpp src/tokenizer.cpp src/dataset.cpp -o llm_test

./llm_test --train --epochs 180 --seed 42 --evaluate

What the results mean and what Dani cannot do

When reviewing the results, I had to learn not to get carried away by a single number. In the run documented in the README, the vocabulary contains 868 tokens and test next-token accuracy is 86.29%. These are project-reported results on a small local set, not an independent benchmark or a measure of correct conversations.

Evaluation uses teacher forcing: it receives the previous words of the reference answer. Generating a whole reply is harder because one mistake can affect every subsequent step. The README illustrates that limitation: when asked about a cat, the model responded about a dog.

Dani retains no history between messages and does not learn during chat. Some known personal questions also use a deterministic catalog separate from the network. The --neural-only mode exposes neural output without that layer. Distinguishing these parts is more useful than treating every correct reply as language understanding.