Two Frameworks Rule Them All: PyTorch vs JAX Inside the AI Factory
Almost every frontier model is written in one of two frameworks. PyTorch, born at Facebook and now governed by an open foundation, runs your code line by line and compiles it when asked. JAX, built at Google, treats your program as a pure function and hands it whole to a compiler. That difference shapes how labs scale to thousands of chips, which hardware they can use and whom they can hire. This part explains both in depth, with runnable-looking code, an interactive sharding playground and a guide to choosing.
1. Case file
2. Timeline
Deep learning frameworks, 2007 โ 2026
Tap a milestone. Dashed borders are widely reported years rather than a primary source.
The history explains the present. Theano and Lua Torch proved the idea of a framework that computes gradients automatically. Google's TensorFlow (2015) brought industrial scale but asked researchers to build a static graph first, which made debugging painful. PyTorch (2016) let them write ordinary Python that runs immediately, and won research within a few years. JAX (2018) took a different route: keep the program pure and functional, and let a compiler, XLA, do the heavy lifting, which suits Google's TPUs. In 2023 PyTorch 2.0 added a compiler of its own, and the two approaches have been converging ever since.
3. What a framework actually does
A framework is the language researchers write in. It stores numbers as tensors on accelerators, records the operations applied to them so it can compute gradients by the chain rule (automatic differentiation), sends each operation to a fast GPU or TPU routine, and increasingly compiles whole blocks of code into fused programs. At AI-factory scale it must also describe how arrays are split across thousands of chips. Everything above it in the stack, from parallelism libraries to inference engines, is built on these four jobs.
4. Two philosophies
Eager, compiled, or both?
Pick a mode and step through how the same three lines of model code execute. Simplified.
Bars show where time goes in each mode, illustratively: Python overhead, kernel launches and actual compute. Not measured data.
grad (gradients), jit (compile), vmap (vectorise) and sharding annotations turn them into new functions. jit traces the function once and XLA compiles the whole thing into one optimised program.5. Same model, two ways
The clearest way to see the difference is a tiny two-layer network and one training step, written idiomatically in each. Toggle between them.
PyTorch keeps state inside objects: the model owns its parameters, the optimizer owns its momentum, and loss.backward() fills in gradients as a side effect. JAX keeps state outside: parameters are a plain tree of arrays passed in and returned, gradients come from transforming the loss function, and nothing changes in place. The PyTorch version feels like ordinary programming; the JAX version is easier for a compiler to reason about, and easier to split across devices, because every input and output is explicit.
6. Inside the compilers
6.1 torch.compile
PyTorch 2.0 introduced torch.compile in 2023. Three pieces do the work: TorchDynamo hooks into Python's bytecode to capture graphs of tensor operations from ordinary code; AOTAutograd traces the backward pass ahead of time so it can be optimised too; and TorchInductor generates fast kernels, mostly in OpenAI's Triton language for GPUs. Where Dynamo meets something it cannot capture, it breaks the graph and runs that part in plain Python, which is why compiled PyTorch keeps working on messy code.
torch.compile at launch
PyTorch's own benchmark of 163 open-source models (Hugging Face, TIMM, TorchBench) on an NVIDIA A100, March 2023.
The headline 43% is a weighted average (0.75 ร AMP + 0.25 ร FP32). PyTorch noted lower speedups on desktop GPUs such as the RTX 3090.
6.2 XLA and jax.jit
JAX never runs eagerly in production. jax.jit traces a function with abstract values, producing an intermediate program (expressed in StableHLO), which Google's XLA compiler optimises as a whole, fusing operations, planning memory and inserting communication between devices, before emitting code for TPUs or GPUs. The cost is compile time and less flexibility (Python control flow must be expressed with JAX's own constructs); the benefit is predictable, whole-program optimisation. JAX also offers Pallas for writing custom kernels when XLA's output is not fast enough, the counterpart of Triton (Part 4).
7. Scaling to thousands of chips
A frontier model does not fit on one chip, so the framework must describe how arrays are split. Both frameworks now do this with a device mesh: a logical grid of chips with named axes (for example "data" and "model"), plus a rule for how each array is laid out across it.
DeviceMesh names the grid; DTensor is a tensor that knows how it is sharded or replicated across it. FSDP2 (fully_shard) converts a model's parameters into DTensors sharded per parameter, gathering them just in time for each layer, and composes with tensor and pipeline parallelism. TorchTitan is PyTorch's reference for combining them for LLM pre-training; Megatron-Core and DeepSpeed are the main alternatives (Part 3).NamedSharding over a Mesh; you annotate how inputs and parameters are split and XLA's partitioner (GSPMD, now Shardy) works out the rest, inserting collectives automatically. shard_map lets you write per-device code by hand when you need control. Pathways runs one program across many TPU pods; MaxText is Google's reference LLM training code.The philosophical difference shows up again: PyTorch tends to make parallelism explicit in library calls you compose, while JAX tends to let you annotate shardings and have the compiler propagate them. Both are used at the largest scales: Meta trained Llama 3 405B on 16,384 GPUs with PyTorch (Producing Intelligence Part 4), and Google trains Gemini with JAX on TPU pods of up to 9,216 chips (Case Study 6).
8. Sharding playground
Split a model across 16 chips
An illustration. Pick a model size and a strategy for a 4 ร 4 mesh of 80 GB accelerators. Memory assumes ~16 bytes per parameter for mixed-precision training with Adam (weights, gradients, optimizer state), ignoring activations.
9. Hardware reach
Framework ร chip
How each framework reaches each accelerator family, as described in project documentation. Tap a cell.
Hardware reach is where the framework choice becomes a business decision. PyTorch's breadth (NVIDIA through CUDA, AMD through ROCm, Amazon's Trainium through the Neuron SDK, Intel and others) is one reason it dominates outside Google. JAX's tight coupling to XLA makes it the natural choice on TPUs, which is why Anthropic's large TPU commitments (Case Study 6) matter for software as well as hardware; JAX also runs on NVIDIA GPUs, where its share is small.
10. Ecosystem
Signs of adoption
Different measures from different sources; none is a perfect proxy for use in AI factories.
The ecosystem has tipped decisively. Hugging Face's Transformers library, the most widely used collection of model code, dropped TensorFlow and Flax in version 5 and now supports only PyTorch, pointing JAX users to partner libraries. Most open-weight models are released as PyTorch checkpoints, most inference engines (vLLM, SGLang, Part 7) are built on PyTorch, and the PyTorch Foundation now gathers many of these projects under one roof. JAX's ecosystem is smaller but deep where it matters to Google: Flax for neural networks, Optax for optimisers, Orbax for checkpoints, MaxText and Pathways for scale.
11. Who uses what
| Lab | Framework | Scaling stack | Source |
|---|---|---|---|
| Meta | PyTorch | In-house 4D parallelism; FSDP, TorchTitan lineage | Llama 3 paper; Meta created PyTorch |
| Google DeepMind | JAX | XLA sharding, Pathways, TPUs | Google publications |
| xAI | JAX | Custom, with Rust and Kubernetes | xAI Grok announcement, 2023 |
| DeepSeek | PyTorch-based โ ๏ธ | In-house: DualPipe, DeepEP, DeepGEMM | Inferred from open-sourced code |
| OpenAI, Anthropic | Not disclosed | Not disclosed | โ |
| Most startups & open-weight labs | PyTorch | Megatron-Core, DeepSpeed, FSDP2, TorchTitan | Public model releases |
12. The rest of the field
- TensorFlow: still running in many production systems, but no longer where new frontier models are written; Hugging Face's removal of TensorFlow classes in Transformers v5 is a sign of the shift.
- Keras 3: a high-level API that can run on JAX, TensorFlow or PyTorch, chosen with a backend setting. Convenient for portability, with some overhead and not every low-level optimisation exposed.
- Apple MLX: an array framework for Apple silicon, used for running and fine-tuning models on Macs rather than in AI factories.
- Mojo and MAX (Modular): a newer language and inference stack aiming to unify programming across hardware; interesting, but not yet a frontier training framework.
13. Choosing a framework
Which framework fits?
Four quick questions. An editorial guide, not a rule.
14. Reading the numbers
- PyTorch
- 2.13: 3,328 commits, 526 contributors (PyTorch newsletter); torch.compile works on 93% of 163 models, +43% training on A100 (21% FP32, 51% AMP; weighted 0.75 AMP + 0.25 FP32) per the PyTorch 2.0 announcement; FSDP2 converts parameters to DTensors (docs).
- JAX
- 0.9.x in 2026 with the Shardy partitioner maturing (release coverage); Pathways and MaxText per Google materials.
- Ecosystem โ ๏ธ
- Hugging Face Hub ~248k PyTorch vs ~12k JAX models (Sep 2026 snapshot cited by a blog); research share ~55โ85% PyTorch, ~10% JAX (conflicting blogs); Transformers v5 PyTorch-only (Hugging Face and coverage).
- Illustrations
- Execution-mode time bars, the sharding playground (16 bytes/parameter, activations ignored) and the chooser are simplified.
- PyTorch
- PyTorch 2.0: torch.compile and benchmark results
- PyTorch docs: FSDP2 (fully_shard)
- PyTorch newsletter, July 2026
- InfoQ: PyTorch 2.0 release and compile
- JAX and comparisons
- PyTorch vs JAX overview
- ML framework comparison 2026
- JAX vs PyTorch 2026 benchmarks
- arXiv: CNN performance in Keras, PyTorch and JAX
- Ecosystem
- InfoQ: Hugging Face Transformers v5
- AssemblyAI: Transformers v5 changes
- Keras 3 review
- Part 1: the ML layer map