Symbolic Transformer

Symbolic Transformer

A next-token predictor that replaces floating-point weights with hierarchical label pattern matching. Instead of learned matrices, the model uses a DAG of semantic labels, counting bloom filters, and explicit pattern-matching neurons — all programmed iteratively by a Claude-powered debug agent.

Why?

Transformers work, but their internals are opaque. This project asks: *what if a transformer's weights were human-readable programs?* Every "neuron" is a named pattern with a description. Every "attention head" has an explicit query ("look for verbs in the last 3 tokens") and value ("extract the tense label"). The hierarchy of labels mirrors how we believe transformers organize information internally.

The model is trained by a Claude agent that inspects activation traces, diagnoses high-loss positions, and proposes specific mutations — adding neurons, hierarchy edges, and attention circuits. It's autoresearch: an AI system iteratively programming a symbolic version of itself.

Architecture

Text → GPT-2 tokenizer → Embedding → [Attention → MLP] × N layers → Unembed → softmax → loss

Each token carries a **TokenState**: positive and negative label sets. Labels live in a hierarchy DAG — if a token has "dog" and the hierarchy says dog→mammal→animal, the token implicitly has "mammal" and "animal" too. Pattern matching uses this hierarchy for generalization: a neuron matching "mammal" fires on any mammal.

Components

Component	Traditional Transformer	Symbolic Transformer
Weights	Float matrices	Named neurons with label patterns
Activation	Vector of floats	Set of string labels
Dot product	Matrix multiply	Count matched labels via hierarchy walk
MLP	Linear → ReLU → Linear	Pattern match → threshold → merge output labels
Attention	QKV projections + softmax	Query MLP → score keys with position labels → top-k → value MLP
Embeddin