Large Language Models as Emerging Operating Systems

A large language model is just two files: parameters and a small run-time.

Llama 2 70B is 140 GB of float16 weights plus ~500 lines of C code; no internet required.

Why: Once trained, inference is cheap; the magic is in the compressed knowledge inside the weights.

"a large language model is just two files right... you can take these two files and you can take your MacBook and this is a fully self-contained package"

Pre-training is lossy compression of the internet into neural weights.

10 TB of text → 140 GB weights ≈ 100× compression; $2 M, 6 000 GPUs, 12 days for Llama 2 70B.

Why: Next-word prediction forces the model to internalize world knowledge to minimize prediction error.

"what we're doing can best be sort of understood as kind of a compression of a good chunk of Internet... it's kind of like a lossy compression"

Next-word prediction is a deceptively powerful objective that captures world knowledge.

To predict the next token about Ruth Handler the model must encode facts of her life.

Why: Statistical correlation between prediction accuracy and compression implies knowledge must be stored.

"the next word prediction task... forces you to learn a lot about the world inside the parameters of the neural network"

We know how to optimize parameters but not what each parameter does.

100 B parameters collaborate in ways we can measure (loss ↓) but not mechanistically explain.

Why: Neural nets are empirical artifacts produced by gradient descent, unlike engineered systems.

"we don't actually really know what these 100 billion parameters are doing... we kind of treat them mostly as empirical artifacts"

Two-stage recipe: pre-training for knowledge, fine-tuning for alignment.

Stage 1: internet text → base model. Stage 2: curated Q&A dialogs → assistant model.

Why: Base models dream documents; fine-tuning reshapes format while preserving prior knowledge.

"pre-training stage... it's about knowledge... fine-tuning stage is about what we call alignment"

Performance scales smoothly and predictably with parameters N and data D.

Accuracy = f(N, D); trends show no sign of saturation, enabling compute-driven roadmaps.

Why: Empirical regularity implies bigger clusters + more data → better models without new algorithms.

"the performance of these large language models... is a remarkably smooth well behaved and predictable function of only two variables"

LLMs become more capable by orchestrating external tools rather than working in-head.

Browse, calculator, Python, DALL-E invoked via special tokens; mirrors human problem solving.

Why: Extends model beyond context-window limits and leverages existing software infrastructure.

"tool use is a major aspect in how these models are becoming a lot more capable"

Vision, audio, and other modalities plug into the same transformer backbone.

Images, speech, music handled via tokenization; enables seeing sketches, voice chat, etc.

Why: Unified architecture allows cross-modal reasoning and richer user interfaces.

"multi modality is actually like a major axis along which large language models are getting better"