Home โ€บ Blog โ€บ Evaluation and Red-Teaming
Producing Intelligence ยท Part 7 ยท Evaluation

Is It Ready? How Meta Tested Llama 3 405B

A finished model is just a large file of weights until someone decides it is good enough, and safe enough, to release. For Llama 3 that decision rested on four kinds of evidence: academic benchmarks, checks that the model had not simply memorised the tests, side-by-side judgments by human annotators, and red teams trying to make it do harm. This part looks at each one, what Meta reported, and why every score needs reading with care.

FL
FrontierAGI Team
โš ๏ธ How this was researched Sources are section 5 of Meta's paper "The Llama 3 Herd of Models" (arXiv 2407.21783) as quoted in search results, Meta's CyberSecEval 3 paper (arXiv 2408.01605), and the benchmark table Meta published at launch. We could not open Meta's launch post directly; its numbers are taken from secondary reproductions and marked โš ๏ธ secondary. Exact human-evaluation win rates and safety violation rates are not reproduced because we could not verify them. The format demo, contamination highlighter and safety trade-off slider are illustrations. See Reading the numbers.
88.6MMLU, level with GPT-4o (88.7) โš ๏ธ secondary
8-gramoverlap test used to detect contaminated test items
7-pointscale for human side-by-side comparisons
No significantcyber-attack uplift vs search engines (CyberSecEval 3)

1. Evaluation is a stage, not a step

Evaluation is easy to picture as an exam at the end. In practice it runs through the whole project, and each earlier part of this series depended on it.

Data (Part 2). Small models trained on candidate data mixes are scored to choose the mix.
Planning (Part 3). Scaling laws predicted the 405B model's benchmark scores before training began.
Running (Part 5). Checkpoints are evaluated as training goes, to catch regressions.
Post-training (Part 6). Every round, and every averaged variant, is scored to pick the best model.
Release (this part). Benchmarks, human judgments and red-team findings decide whether and how to ship.

That makes the evaluation harness real infrastructure: thousands of prompts, many benchmarks, many model variants, run repeatedly on GPU inference clusters, with results tracked like any other experiment.

2. The launch benchmarks

On 23 July 2024 Meta published a table comparing Llama 3.1 405B Instruct with the leading closed models of the time. The headline was that an open-weights model was, for the first time, roughly level with GPT-4o and Claude 3.5 Sonnet on many standard tests, ahead on some and behind on others.

Llama 3.1 405B Instruct vs the closed leaders

Scores as published in Meta's launch table, reproduced from secondary sources โš ๏ธ. Pick a benchmark group.

Shots, chain-of-thought and scoring differ by benchmark and by lab; scores within about a point should be treated as a tie. Empty bars mean we found no figure.

๐Ÿ’ก What each benchmark tests
MMLU / MMLU-Pro
Multiple-choice knowledge across 57 subjects; Pro has 10 options and harder questions.
GPQA
Graduate-level science questions written to be hard to look up.
HumanEval
Write Python functions that pass hidden unit tests.
GSM8K / MATH
Grade-school word problems; competition maths.
IFEval
Follow checkable instructions ("answer in under 50 words, in JSON").
BFCL
Berkeley Function Calling Leaderboard: call tools with the right arguments.
MGSM
GSM8K-style maths in multiple languages.

3. How fragile multiple-choice scores are

A model's benchmark score depends on the exact way questions are put to it. Meta's paper tests the robustness of its pre-trained models on multiple-choice benchmarks against changes in the answer labels, the order of the answer options and the prompt format. The demo below shows the kinds of variation tested; it does not reproduce Meta's results, which we could not retrieve.

One question, four presentations

The knowledge required is identical; a robust model should answer all four the same way.


      

4. Contamination: did the model see the test?

Benchmarks live on the public web, and the web is the training data. If test questions appear in the pre-training corpus, a high score may measure memory, not skill. Meta's method (paper section 5.1.4) marks an evaluation example as contaminated when enough of its tokens fall inside 8-grams (runs of eight tokens) that also appear in the training data. The threshold for "enough" is chosen separately for each benchmark, as the one that shows the largest estimated performance gain on contaminated examples. Meta notes that such methods produce both false positives and false negatives.

8-gram overlap highlighter

An illustrative test question checked against an illustrative training snippet. Highlighted words are inside a shared 8-word run.

GPT-3 used 13-grams, Llama 2 used 10-grams (tokens; here we use words for readability).
Test question

Found in training data

5. Human judges

Benchmarks reward exact answers; users care about whether a reply is helpful. Meta therefore ran human evaluations: annotators saw responses from two models to the same prompt and rated them on a seven-point scale. A rating of "better" or "much better" counts as a win. Prompts covered six single-turn categories (English, reasoning, coding, Hindi, Spanish, Portuguese) and three multi-turn ones (English, reasoning, coding).

What the human comparisons found

Qualitative results as described in the paper. Exact win rates are in the paper's figures and are not reproduced here.

โš ๏ธ Human evaluation has its own biases Annotators tend to prefer longer, confident and well-formatted answers even when they are not more correct. Results also depend on the prompt set and the annotator pool, which makes them hard to compare across labs. Public leaderboards like Chatbot Arena have the same issue at larger scale.

6. Safety metrics: two numbers in tension

Meta's safety evaluation reports two rates for each model on internal benchmarks: the violation rate, how often a response breaks safety policy, and the false refusal rate, how often the model refuses a harmless request. Llama 3 was compared with anonymised competitor systems. Meta's stated aim is to keep both low, and it notes the benchmarks are internal, so outsiders cannot reproduce the figures.

The trade-off

Illustrative curve. Making a model refuse more lowers violations but raises false refusals; better training moves the whole curve towards the corner.

7. Red teams and uplift studies

Benchmarks test known questions. Red teams look for unknown failures: people (and automated tools) who try to make the model produce harmful content, leak instructions or misbehave with tools. For the most serious risks, Meta also ran uplift studies, which ask a narrower question: does access to the model make a would-be attacker meaningfully more capable than they would be with the internet alone?

Red-team areas

8. Open evaluation

Because the weights are public, anyone can re-run Meta's tests. Meta published its evaluation configurations (shots, metrics, prompts) on GitHub and the generated evaluation data on Hugging Face. Outsiders found differences, as they do for every model, and public leaderboards such as Chatbot Arena added their own rankings based on user votes.

Benchmarks also wear out. MMLU and GSM8K were nearly saturated by mid-2024, which is why harder tests such as MMLU-Pro and GPQA appeared in the launch table, and why later models are judged on agentic tests like SWE-bench and on new, held-out question sets.

๐Ÿ—‚๏ธ Llama 3 case file: evaluation

Benchmarks
Level with GPT-4o and Claude 3.5 Sonnet on MMLU, GSM8K and IFEval; behind on coding, MATH, GPQA and MMLU-Pro โš ๏ธ secondary
Robustness
Tested against label, order and format changes in multiple-choice
Contamination
8-gram overlap, threshold chosen per benchmark
Human
7-point pairwise scale; 6 single-turn + 3 multi-turn categories; beats GPT-4 (0125) on multi-turn reasoning and coding, trails Claude 3.5 Sonnet on coding and reasoning
Safety
Violation and false refusal rates on internal benchmarks vs anonymised competitors
Uplift
Cyber: no significant uplift (CyberSecEval 3). Chem/bio: six-hour planning study with expert grading
Openness
Eval configs on GitHub, generated eval data on Hugging Face
Verdict Llama 3's evaluation is notable less for any single score than for its breadth and openness: benchmarks with published configs, a stated contamination method, human comparisons, paired safety metrics and uplift studies. Each kind of evidence has gaps (benchmarks saturate and leak, humans are biased, safety benchmarks are private), so the release decision rests on the combination. For readers, the lesson is to treat a one-point benchmark lead as a tie and to ask how a score was produced before trusting it.

๐Ÿ‘€ What to watch

๐Ÿ”ข Reading the numbers

๐Ÿ“š References