GPT saka Nol — Panduan Teknik Lengkap

Proyek iki mbangun model basa GPT saka dhasar, dimiwiti saka neuron siji nganti model transformer lengkap sing bisa nggawe teks. Kode ditulis nggunakake Python lan PyTorch, dikelompokke dadi pirang-pirang modul sing jelas lan gampang diwaca.

---

Struktur Direktori

text-embedding/
├── gpt-py/                  # Source code utama
│   ├── foundations/         # Dhasar-dhasar deep learning
│   ├── data/                # Tokenisasi lan dataset
│   ├── model/               # Komponen model GPT
│   ├── train.py             # Loop latihan
│   └── generate.py          # Generasi teks
└── gpt-from-scratch.ipynb   # Notebook interaktif

---

Isi Modul

foundations/ — Dhasar Jaringan Saraf

File	Isi
neuron.py	Neuron siji karo forward lan backward pass
backprop.py	Backpropagation rong layer saka nol
multi_layer_backprop.py	MLP backprop nggunakake NumPy
activations.py	ReLU, GELU, Sigmoid, SiLU, SwiGLU
softmax.py	Softmax stabil, log-softmax, top-k
loss.py	Cross-entropy, MSE, BCE lan gradien-e
gradient_descent.py	SGD, Momentum, RMSProp, Adam saka nol
linear_regression.py	Forward pass lan gradien analitik
linear_regression_training.py	Latihan regresi karo SGD batch
mlp.py	MLP PyTorch karo skip connection
weight_init.py	Xavier, He, lan GPT-style initialization
pytorch_basics.py	Autograd, checkpoint, mixed precision
training_loop.py	Loop latihan lan evaluasi standar
training_diagnostics.py	Monitor gradien, aktivasi, NaN
dead_relu_detector.py	Deteksi neuron ReLU sing mati
digit_classifier.py	CNN lan MLP kanggo klasifikasi angka
sentiment.py	RNN lan Transformer kanggo analisis sentimen

data/ — Tokenisasi lan Dataset

File	Isi
vocab.py	Kamus karakter: encode, decode, simpan, muat
tokenizer.py	BPE tokenizer saka nol (GPT-2 style)
tokenizer_utils.py	Padding, truncation, chunking, masking
nlp_preprocessing.py	Bersih-bersih teks: HTML, URL, kontraksi
loader.py	DataLoader karo split stratifikasi
dataset.py	GPTDataset lan PackedDataset

model/ — Komponen GPT

File	Isi
normalization.py	LayerNorm saka nol
batch_normalization.py	BatchNorm karo running stats
rms_normalization.py	RMSNorm (LLaMA-style, tanpa bias)
embeddings.py	Token embedding + weight tying
positional_encoding.py	Sinusoidal PE lan RoPE
attention.py	Scaled dot-product attention + FlashAttention
multi_head_attention.py	MHA lan CrossAttention
grouped_query_attention.py	GQA kanggo ngurangi memori KV
kv_cache.py	KV Cache kanggo inferensi cepet
transformer.py	TransformerBlock (pre-norm + FFN)
gpt.py	Model GPT lengkap karo GPTConfig

---

Cara Nggunakake

1. Pasang Dependensi

pip install -r gpt-py/requirements.txt

Dependensi sing dibutuhake:

torch>=2.1.0
numpy>=1.24.0
regex>=2023.6.3
matplotlib>=3.7.0
tqdm>=4.65.0
torchvision>=0.16.0

2. Latihan Model

cd gpt-py
python train.py path/menyang/file.txt

Parameter latihan bisa diganti langsung ing fungsi train():

train(
    text_path  = 'data/input.txt',
    out_dir    = 'checkpoints',
    max_iters  = 5000,       # cacah langkah
    lr_max     = 3e-4,       # learning rate maksimum
    batch_size = 16,
    block_size = 128,        # dawa konteks
    warmup     = 200,        # langkah warmup
)

Jadwal learning rate nggunakake cosine decay karo linear warmup:

$$\eta_t = \eta_{\min} + \tfrac{1}{2}(\eta_{\max}-\eta_{\min})\left(1+\cos\frac{\pi(t-t_w)}{T-t_w}\right)$$

3. Nggawe Teks

python generate.py checkpoints/gpt_final.pt "Sakwise iku"

Utawa saka Python:

from generate import load_and_generate

teks = load_and_generate(
    ckpt_path   = 'checkpoints/gpt_final.pt',
    prompt      = 'Transformer iku',
    max_new     = 200,
    temperature = 0.8,   # 0.1 = greedy, 2.0 = acak
    top_k       = 40,    # sampling top-k
)
print(teks)

4. Nggunakake Notebook

jupyter notebook gpt-from-scratch.ipynb

Notebook iki njelasake saben konsep kanthi:

• Rumus matematika sing jelas
• Kode Python sing bisa langsung dijalanake
• Visualisasi hasil (grafik, heatmap, peta atensi)

---

Arsitektur Model

GPT sing diimplementasike ing kene nggunakake desain decoder-only pre-norm:

Input Token IDs
       ↓
Token Embedding + Positional Embedding
       ↓
  ┌─────────────────────────┐
  │  TransformerBlock × N   │
  │  ┌───────────────────┐  │
  │  │ LayerNorm         │  │
  │  │ MultiHeadAttention│  │
  │  │ + Residual        │  │
  │  ├───────────────────┤  │
  │  │ LayerNorm         │  │
  │  │ FeedForward(4×)   │  │
  │  │ + Residual        │  │
  │  └───────────────────┘  │
  └─────────────────────────┘
       ↓
Final LayerNorm
       ↓
LM Head (weight-tied karo embedding)
       ↓
Logits → Softmax → Token

Konfigurasi Preset

from model.gpt import GPT, GPTConfig

# Cilik — kanggo eksperimen
cfg = GPTConfig.small()   # 256d, 8h, 6L  ≈ 10M params

# Sedeng
cfg = GPTConfig.medium()  # 512d, 8h, 8L  ≈ 85M params

# Gedhe
cfg = GPTConfig.large()   # 1024d, 16h, 24L ≈ 800M params

model = GPT(cfg)
print(f'Params: {model.count_params():,}')

---

Fitur Teknik

Weight Tying

Bobot output head (lm_head.weight) padha karo bobot token embedding (tok_emb.weight). Iki ngurangi parameter lan nambah performa:

self.lm_head.weight = self.tok_emb.weight

Grouped Query Attention (GQA)

Ngurangi memori KV cache ing inferensi kanthi nuduhake head K/V ing antarane kelompok head Q:

from model.grouped_query_attention import GroupedQueryAttention
gqa = GroupedQueryAttention(d_model=512, n_heads=8, n_kv_heads=2)
# KV cache 4x luwih cilik tinimbang MHA biasa

KV Cache

Nyepetake generasi teks kanthi nyimpen komputasi K/V saka token sadurunge:

• Tanpa cache: $O(T^2)$ saben langkah
• Karo cache: $O(T)$ saben langkah

from model.kv_cache import KVCache, CachedAttention
cache = KVCache()

BPE Tokenizer

Tokenizer Byte-Pair Encoding saka nol, kompatibel karo pola GPT-2:

from data.tokenizer import BPETokenizer

tok = BPETokenizer()
tok.train(teks_latihan, vocab_size=8000)
ids = tok.encode("hello world")
print(tok.decode(ids))

---

Teknik Latihan

Teknik	Implementasi	Lokasi
AdamW optimizer	torch.optim.AdamW	train.py
Cosine LR + warmup	cosine_lr()	train.py
Gradient clipping	clip_grad_norm_(..., 1.0)	train.py
Mixed precision	GradScaler	foundations/training_loop.py
Grad accumulation	grad_accum_step()	foundations/pytorch_basics.py
Dead ReLU monitor	DeadReLUDetector	foundations/dead_relu_detector.py
Gradient stats	grad_stats()	foundations/training_diagnostics.py

---

Strategi Generasi Teks

Logits
  ↓
÷ Temperature          # ngatur kerandoman
  ↓
Top-k masking          # batesi pilihan token
  ↓
Top-p (nucleus)        # filter kumulatif prob
  ↓
Softmax → Multinomial  # sampling
  ↓
Token anyar

Parameter	Efek
temperature=0.1	Deterministik, teks konsisten
temperature=0.8	Seimbang (rekomendasi)
temperature=1.5	Kreatif, luwih acak
top_k=1	Greedy decoding
top_k=40	Sampling standar

---

Syarat Sistem

• Python 3.9 utawa luwih anyar
• PyTorch 2.1 utawa luwih anyar
• GPU NVIDIA (opsional, nanging disaranake kanggo latihan)
• RAM minimal 8 GB

---

Referensi

• Vaswani et al. (2017) — Attention Is All You Need
• Radford et al. (2019) — Language Models are Unsupervised Multitask Learners (GPT-2)
• Su et al. (2021) — RoFormer: Enhanced Transformer with Rotary Position Embedding
• Ainslie et al. (2023) — GQA: Training Generalized Multi-Query Transformer Models
• Ba et al. (2016) — Layer Normalization