Is It Ready? How Meta Tested Llama 3 405B
A finished model is just a large file of weights until someone decides it is good enough, and safe enough, to release. For Llama 3 that decision rested on four kinds of evidence: academic benchmarks, checks that the model had not simply memorised the tests, side-by-side judgments by human annotators, and red teams trying to make it do harm. This part looks at each one, what Meta reported, and why every score needs reading with care.
1. Evaluation is a stage, not a step
Evaluation is easy to picture as an exam at the end. In practice it runs through the whole project, and each earlier part of this series depended on it.
That makes the evaluation harness real infrastructure: thousands of prompts, many benchmarks, many model variants, run repeatedly on GPU inference clusters, with results tracked like any other experiment.
2. The launch benchmarks
On 23 July 2024 Meta published a table comparing Llama 3.1 405B Instruct with the leading closed models of the time. The headline was that an open-weights model was, for the first time, roughly level with GPT-4o and Claude 3.5 Sonnet on many standard tests, ahead on some and behind on others.
Llama 3.1 405B Instruct vs the closed leaders
Scores as published in Meta's launch table, reproduced from secondary sources โ ๏ธ. Pick a benchmark group.
Shots, chain-of-thought and scoring differ by benchmark and by lab; scores within about a point should be treated as a tie. Empty bars mean we found no figure.
- MMLU / MMLU-Pro
- Multiple-choice knowledge across 57 subjects; Pro has 10 options and harder questions.
- GPQA
- Graduate-level science questions written to be hard to look up.
- HumanEval
- Write Python functions that pass hidden unit tests.
- GSM8K / MATH
- Grade-school word problems; competition maths.
- IFEval
- Follow checkable instructions ("answer in under 50 words, in JSON").
- BFCL
- Berkeley Function Calling Leaderboard: call tools with the right arguments.
- MGSM
- GSM8K-style maths in multiple languages.
3. How fragile multiple-choice scores are
A model's benchmark score depends on the exact way questions are put to it. Meta's paper tests the robustness of its pre-trained models on multiple-choice benchmarks against changes in the answer labels, the order of the answer options and the prompt format. The demo below shows the kinds of variation tested; it does not reproduce Meta's results, which we could not retrieve.
One question, four presentations
The knowledge required is identical; a robust model should answer all four the same way.
4. Contamination: did the model see the test?
Benchmarks live on the public web, and the web is the training data. If test questions appear in the pre-training corpus, a high score may measure memory, not skill. Meta's method (paper section 5.1.4) marks an evaluation example as contaminated when enough of its tokens fall inside 8-grams (runs of eight tokens) that also appear in the training data. The threshold for "enough" is chosen separately for each benchmark, as the one that shows the largest estimated performance gain on contaminated examples. Meta notes that such methods produce both false positives and false negatives.
8-gram overlap highlighter
An illustrative test question checked against an illustrative training snippet. Highlighted words are inside a shared 8-word run.
5. Human judges
Benchmarks reward exact answers; users care about whether a reply is helpful. Meta therefore ran human evaluations: annotators saw responses from two models to the same prompt and rated them on a seven-point scale. A rating of "better" or "much better" counts as a win. Prompts covered six single-turn categories (English, reasoning, coding, Hindi, Spanish, Portuguese) and three multi-turn ones (English, reasoning, coding).
What the human comparisons found
Qualitative results as described in the paper. Exact win rates are in the paper's figures and are not reproduced here.
6. Safety metrics: two numbers in tension
Meta's safety evaluation reports two rates for each model on internal benchmarks: the violation rate, how often a response breaks safety policy, and the false refusal rate, how often the model refuses a harmless request. Llama 3 was compared with anonymised competitor systems. Meta's stated aim is to keep both low, and it notes the benchmarks are internal, so outsiders cannot reproduce the figures.
The trade-off
Illustrative curve. Making a model refuse more lowers violations but raises false refusals; better training moves the whole curve towards the corner.
7. Red teams and uplift studies
Benchmarks test known questions. Red teams look for unknown failures: people (and automated tools) who try to make the model produce harmful content, leak instructions or misbehave with tools. For the most serious risks, Meta also ran uplift studies, which ask a narrower question: does access to the model make a would-be attacker meaningfully more capable than they would be with the internet alone?
Red-team areas
8. Open evaluation
Because the weights are public, anyone can re-run Meta's tests. Meta published its evaluation configurations (shots, metrics, prompts) on GitHub and the generated evaluation data on Hugging Face. Outsiders found differences, as they do for every model, and public leaderboards such as Chatbot Arena added their own rankings based on user votes.
Benchmarks also wear out. MMLU and GSM8K were nearly saturated by mid-2024, which is why harder tests such as MMLU-Pro and GPQA appeared in the launch table, and why later models are judged on agentic tests like SWE-bench and on new, held-out question sets.
๐๏ธ Llama 3 case file: evaluation
- Benchmarks
- Level with GPT-4o and Claude 3.5 Sonnet on MMLU, GSM8K and IFEval; behind on coding, MATH, GPQA and MMLU-Pro โ ๏ธ secondary
- Robustness
- Tested against label, order and format changes in multiple-choice
- Contamination
- 8-gram overlap, threshold chosen per benchmark
- Human
- 7-point pairwise scale; 6 single-turn + 3 multi-turn categories; beats GPT-4 (0125) on multi-turn reasoning and coding, trails Claude 3.5 Sonnet on coding and reasoning
- Safety
- Violation and false refusal rates on internal benchmarks vs anonymised competitors
- Uplift
- Cyber: no significant uplift (CyberSecEval 3). Chem/bio: six-hour planning study with expert grading
- Openness
- Eval configs on GitHub, generated eval data on Hugging Face
๐ What to watch
- Held-out and continuously refreshed benchmarks that resist contamination.
- Agentic evaluations (multi-step tasks with tools) replacing single-question tests.
- Third-party pre-release testing by government AI safety institutes.
- Next in this series: serving, how a checkpoint becomes an endpoint.
๐ข Reading the numbers
- Evaluation structure, the 8-gram contamination method, human-evaluation design and qualitative results, safety metric definitions and the uplift study design come from the Llama 3 paper as quoted in search results.
- Benchmark scores come from secondary reproductions of Meta's launch table โ ๏ธ.
- Cyber findings come from CyberSecEval 3.
- The chemical/biological uplift outcome, exact human win rates, violation and false refusal rates and the robustness results are not reproduced because we could not verify them.
- The format demo, contamination highlighter and trade-off curve are illustrations.
๐ References
- Llama 3
- Llama Team, Meta: "The Llama 3 Herd of Models" (2024), section 5
- Wan et al., Meta: "CyberSecEval 3" (2024)
- Meta: Introducing Llama 3.1 (launch post and benchmark table)
- PromptLayer: LLM benchmarks (secondary reproduction of the launch table)
- Vellum: LLM benchmarks overview (secondary)
- LessWrong: Llama Llama-3-405B? (discussion)
- Benchmarks and methods
- Hendrycks et al.: MMLU (2020) ยท MMLU-Pro (2024) ยท GPQA (2023)
- Chen et al.: HumanEval (2021) ยท GSM8K (2021) ยท IFEval (2023)
- A Comprehensive Survey of Contamination Detection Methods in LLMs (2024)
- Chiang et al.: Chatbot Arena (2024)
- Related on this site
- Part 6: Post-Training ยท Part 3: Planning the Run ยท Part 2: The Data Refinery