Home โ€บ Blog โ€บ PyTorch vs JAX
Inside the ML Layer ยท Part 2 ยท Frameworks

Two Frameworks Rule Them All: PyTorch vs JAX Inside the AI Factory

Almost every frontier model is written in one of two frameworks. PyTorch, born at Facebook and now governed by an open foundation, runs your code line by line and compiles it when asked. JAX, built at Google, treats your program as a pure function and hands it whole to a compiler. That difference shapes how labs scale to thousands of chips, which hardware they can use and whom they can hire. This part explains both in depth, with runnable-looking code, an interactive sharding playground and a guide to choosing.

FL
FrontierAGI Team
โš ๏ธ How this was researched (as of 26 Oct 2026) Sources are the PyTorch 2.0 announcement and PyTorch documentation (torch.compile, DTensor, FSDP2), the PyTorch Foundation's newsletter, JAX documentation and release notes, Google's material on XLA, Pathways and MaxText, Hugging Face's Transformers v5 notes, and framework comparisons. Usage-share figures come from blogs that disagree and are marked โš ๏ธ. Code samples are simplified for reading and omit imports and error handling. The sharding playground is our own illustration. See Reading the numbers.

1. Case file

๐Ÿงฑ
The framework layer, October 2026Layer 6 of 8 in the ML layer map
PyTorch2.13 ยท 3,328 commits, 526 contributors
PyTorch governancePyTorch Foundation (Linux Foundation) since 2022
JAX0.9.x in 2026 ยท Google-led
torch.compile (PyTorch 2.0)+43% training speed on 163 models (A100)
Hugging Face Transformersv5 supports PyTorch only
Research share (est.)PyTorch ~55โ€“85% โš ๏ธ ยท JAX ~10% โš ๏ธ

2. Timeline

Deep learning frameworks, 2007 โ†’ 2026

Tap a milestone. Dashed borders are widely reported years rather than a primary source.

Tap a milestone.

The history explains the present. Theano and Lua Torch proved the idea of a framework that computes gradients automatically. Google's TensorFlow (2015) brought industrial scale but asked researchers to build a static graph first, which made debugging painful. PyTorch (2016) let them write ordinary Python that runs immediately, and won research within a few years. JAX (2018) took a different route: keep the program pure and functional, and let a compiler, XLA, do the heavy lifting, which suits Google's TPUs. In 2023 PyTorch 2.0 added a compiler of its own, and the two approaches have been converging ever since.

3. What a framework actually does

TensorsMulti-dimensional arrays on GPU or TPU memory
AutodiffGradients computed automatically from your code
Dispatch & compileEach operation sent to a fast kernel, or fused by a compiler
ShardingSplitting arrays and work across many devices

A framework is the language researchers write in. It stores numbers as tensors on accelerators, records the operations applied to them so it can compute gradients by the chain rule (automatic differentiation), sends each operation to a fast GPU or TPU routine, and increasingly compiles whole blocks of code into fused programs. At AI-factory scale it must also describe how arrays are split across thousands of chips. Everything above it in the stack, from parallelism libraries to inference engines, is built on these four jobs.

4. Two philosophies

Eager, compiled, or both?

Pick a mode and step through how the same three lines of model code execute. Simplified.


        

Bars show where time goes in each mode, illustratively: Python overhead, kernel launches and actual compute. Not measured data.

PyTorch: define-by-runCode runs as Python executes it. You can print, branch and debug anywhere. Speed comes from fast individual kernels and, since 2.0, from torch.compile, which captures graphs from your Python and fuses them, falling back to normal Python where it cannot ("graph breaks").
JAX: functions in, programs outYou write pure functions over arrays. Transformations such as grad (gradients), jit (compile), vmap (vectorise) and sharding annotations turn them into new functions. jit traces the function once and XLA compiles the whole thing into one optimised program.

5. Same model, two ways

The clearest way to see the difference is a tiny two-layer network and one training step, written idiomatically in each. Toggle between them.


      

PyTorch keeps state inside objects: the model owns its parameters, the optimizer owns its momentum, and loss.backward() fills in gradients as a side effect. JAX keeps state outside: parameters are a plain tree of arrays passed in and returned, gradients come from transforming the loss function, and nothing changes in place. The PyTorch version feels like ordinary programming; the JAX version is easier for a compiler to reason about, and easier to split across devices, because every input and output is explicit.

6. Inside the compilers

6.1 torch.compile

PyTorch 2.0 introduced torch.compile in 2023. Three pieces do the work: TorchDynamo hooks into Python's bytecode to capture graphs of tensor operations from ordinary code; AOTAutograd traces the backward pass ahead of time so it can be optimised too; and TorchInductor generates fast kernels, mostly in OpenAI's Triton language for GPUs. Where Dynamo meets something it cannot capture, it breaks the graph and runs that part in plain Python, which is why compiled PyTorch keeps working on messy code.

torch.compile at launch

PyTorch's own benchmark of 163 open-source models (Hugging Face, TIMM, TorchBench) on an NVIDIA A100, March 2023.

The headline 43% is a weighted average (0.75 ร— AMP + 0.25 ร— FP32). PyTorch noted lower speedups on desktop GPUs such as the RTX 3090.

6.2 XLA and jax.jit

JAX never runs eagerly in production. jax.jit traces a function with abstract values, producing an intermediate program (expressed in StableHLO), which Google's XLA compiler optimises as a whole, fusing operations, planning memory and inserting communication between devices, before emitting code for TPUs or GPUs. The cost is compile time and less flexibility (Python control flow must be expressed with JAX's own constructs); the benefit is predictable, whole-program optimisation. JAX also offers Pallas for writing custom kernels when XLA's output is not fast enough, the counterpart of Triton (Part 4).

๐Ÿ’ก ConvergencePyTorch added a compiler; JAX added more Python-friendly debugging. Both now lower to similar intermediate forms, and projects such as PyTorch/XLA let PyTorch code run through XLA on TPUs. The difference today is less "eager vs compiled" than "which ecosystem and hardware you are committing to".

7. Scaling to thousands of chips

A frontier model does not fit on one chip, so the framework must describe how arrays are split. Both frameworks now do this with a device mesh: a logical grid of chips with named axes (for example "data" and "model"), plus a rule for how each array is laid out across it.

PyTorch: DTensor, DeviceMesh, FSDP2DeviceMesh names the grid; DTensor is a tensor that knows how it is sharded or replicated across it. FSDP2 (fully_shard) converts a model's parameters into DTensors sharded per parameter, gathering them just in time for each layer, and composes with tensor and pipeline parallelism. TorchTitan is PyTorch's reference for combining them for LLM pre-training; Megatron-Core and DeepSpeed are the main alternatives (Part 3).
JAX: shardings, shard_map, PathwaysEach array carries a NamedSharding over a Mesh; you annotate how inputs and parameters are split and XLA's partitioner (GSPMD, now Shardy) works out the rest, inserting collectives automatically. shard_map lets you write per-device code by hand when you need control. Pathways runs one program across many TPU pods; MaxText is Google's reference LLM training code.

The philosophical difference shows up again: PyTorch tends to make parallelism explicit in library calls you compose, while JAX tends to let you annotate shardings and have the compiler propagate them. Both are used at the largest scales: Meta trained Llama 3 405B on 16,384 GPUs with PyTorch (Producing Intelligence Part 4), and Google trains Gemini with JAX on TPU pods of up to 9,216 chips (Case Study 6).

8. Sharding playground

Split a model across 16 chips

An illustration. Pick a model size and a strategy for a 4 ร— 4 mesh of 80 GB accelerators. Memory assumes ~16 bytes per parameter for mixed-precision training with Adam (weights, gradients, optimizer state), ignoring activations.

9. Hardware reach

Framework ร— chip

How each framework reaches each accelerator family, as described in project documentation. Tap a cell.

Tap a cell.

Hardware reach is where the framework choice becomes a business decision. PyTorch's breadth (NVIDIA through CUDA, AMD through ROCm, Amazon's Trainium through the Neuron SDK, Intel and others) is one reason it dominates outside Google. JAX's tight coupling to XLA makes it the natural choice on TPUs, which is why Anthropic's large TPU commitments (Case Study 6) matter for software as well as hardware; JAX also runs on NVIDIA GPUs, where its share is small.

10. Ecosystem

Signs of adoption

Different measures from different sources; none is a perfect proxy for use in AI factories.

The ecosystem has tipped decisively. Hugging Face's Transformers library, the most widely used collection of model code, dropped TensorFlow and Flax in version 5 and now supports only PyTorch, pointing JAX users to partner libraries. Most open-weight models are released as PyTorch checkpoints, most inference engines (vLLM, SGLang, Part 7) are built on PyTorch, and the PyTorch Foundation now gathers many of these projects under one roof. JAX's ecosystem is smaller but deep where it matters to Google: Flax for neural networks, Optax for optimisers, Orbax for checkpoints, MaxText and Pathways for scale.

11. Who uses what

LabFrameworkScaling stackSource
MetaPyTorchIn-house 4D parallelism; FSDP, TorchTitan lineageLlama 3 paper; Meta created PyTorch
Google DeepMindJAXXLA sharding, Pathways, TPUsGoogle publications
xAIJAXCustom, with Rust and KubernetesxAI Grok announcement, 2023
DeepSeekPyTorch-based โš ๏ธIn-house: DualPipe, DeepEP, DeepGEMMInferred from open-sourced code
OpenAI, AnthropicNot disclosedNot disclosedโ€”
Most startups & open-weight labsPyTorchMegatron-Core, DeepSpeed, FSDP2, TorchTitanPublic model releases

12. The rest of the field

13. Choosing a framework

Which framework fits?

Four quick questions. An editorial guide, not a rule.

VerdictPyTorch has won breadth: most researchers, most open models, most hardware and most of the open serving stack. JAX has won depth where Google's TPUs are the platform, and remains the choice of some of the most capable labs. For a new lab the default is PyTorch, unless your compute comes from TPUs or your team already thinks in JAX. Either way, the real performance comes from the layers below and beside the framework: parallelism libraries (Part 3) and kernels (Part 4).

14. Reading the numbers

PyTorch
2.13: 3,328 commits, 526 contributors (PyTorch newsletter); torch.compile works on 93% of 163 models, +43% training on A100 (21% FP32, 51% AMP; weighted 0.75 AMP + 0.25 FP32) per the PyTorch 2.0 announcement; FSDP2 converts parameters to DTensors (docs).
JAX
0.9.x in 2026 with the Shardy partitioner maturing (release coverage); Pathways and MaxText per Google materials.
Ecosystem โš ๏ธ
Hugging Face Hub ~248k PyTorch vs ~12k JAX models (Sep 2026 snapshot cited by a blog); research share ~55โ€“85% PyTorch, ~10% JAX (conflicting blogs); Transformers v5 PyTorch-only (Hugging Face and coverage).
Illustrations
Execution-mode time bars, the sharding playground (16 bytes/parameter, activations ignored) and the chooser are simplified.