Frontier Lab Engineering Practicum · Article 1 of 9🧑💻
Day One at a Frontier Lab: The Engineering Onboarding Nobody Writes Down
The AGI Researcher Foundations series covered what to know before you arrive. This new series covers what actually happens once you're inside, as a practicing engineer — the repo, the code review culture, the tools, and the realistic first task, explained visually and in plain language, not as theory but as a job.
Knowing the Stack Isn't the Same as Knowing the Job
This site's AGI Researcher Foundations series answered "what do I need to know" — Python, PyTorch, distributed training theory, the math underneath it. It never answered a different, equally important question: what does the actual first week of doing this job, inside a real engineering organization, look like? This new series answers that question directly, starting here, with the parts of onboarding that don't show up in any job posting.
A framing note before starting: "frontier lab" here means any organization training large models at real scale — the specifics below (tools named, culture described) are patterns observed across the industry, not a claim about one specific employer's exact setup, which will always vary. Treat this as the shape of the job, not a verbatim script.
BillionsLines of code Google reported storing in a single shared monorepo, per Potvin & Levenberg's 2016 paper
7,500Kubernetes nodes OpenAI reported scaling to for research infrastructure, per their engineering blog
10xRough ratio of code read to code written in a typical engineer's first month, per common industry onboarding guidance
1Model you will probably NOT train in week one — see Part 3
Part 1 — The Repo Before You Write a Line
Monorepo vs. Multi-Repo: The First Thing That Shapes Your Day
Before any code review or task assignment, the single structural fact that shapes how your first weeks feel is how the organization's code is split up. Potvin & Levenberg's 2016 paper on Google's monorepo — a single shared repository holding effectively the entire company's code — remains the clearest real-world case study of why large ML organizations often favor one giant, shared repo over many small ones: it makes cross-team dependencies visible and enforces that everyone is always building against the current version of shared code, at the cost of needing serious tooling (custom build systems, code search) just to make a repo that large navigable at all.
Monorepo: one shared source of truth, powerful but requires heavy tooling to navigate. Multi-repo: independent teams move faster locally, at the cost of version-mismatch bugs between repos.
Whichever structure you land in, your first practical task is rarely "understand the whole repo" — it's learning where the boundary is between code you're expected to modify freely and shared internal libraries you should treat as (mostly) fixed dependencies, the same distinction Article 1 of the Foundations series drew between "your code" and open-source frameworks, just applied internally.
Part 2 — Code Review Culture, For Real
What Actually Gets Blocked vs. What Gets a Shrug
A pattern that surprises almost everyone arriving from an academic or solo-project background: a five-line change to a shared training configuration can get more scrutiny than a five-hundred-line new feature in an isolated area of the codebase. This isn't bureaucracy for its own sake — it's a direct, rational response to blast radius: a config change affecting every future training run has a much larger, harder-to-undo impact than an addition nobody else depends on yet.
Review intensity tracks blast radius, not lines of code — a small change to something everyone depends on is treated more carefully than a large change nobody does yet.
1
Reading dominates writing, especially early on. A newcomer typically spends far more time reading existing PRs and internal documentation than writing new code in the first weeks — treat this as the actual onboarding task, not a delay before "real work" starts.
2
Review comments are usually about the "why," not the syntax. Automated linting and formatting (often enforced by CI, not humans) handle style; human reviewers focus on whether the change's reasoning holds up — very similar in spirit to this site's "How to Read a Paper Like a Researcher" framework, applied to code instead of papers.
3
A blocked PR is rarely personal. Given Part 1's blast-radius logic, a reviewer blocking a shared-code change is applying a consistent standard, not making a judgment about you specifically — internalizing this early avoids a common, unnecessary source of new-hire anxiety.
Part 3 — Your First Assigned Task Is Probably Not a Model
The Iceberg Nobody Warns You About
The public-facing output of a frontier lab — a released model — is the visible tip of a much larger structure. A realistic first task looks far more like: fix a data-loading bug that silently drops 2% of a dataset, add a missing metric to an existing evaluation pipeline, or reproduce a colleague's ablation to confirm a reported result — precisely the reproduction-first instinct Article 2 of the Foundations series argued is the real skill frontier labs test for, now showing up as literally your first ticket.
The model is the visible tip; almost everything an engineer actually spends time on lives in the much larger, unglamorous layer underneath it.
Being assigned a data-pipeline bug in week one isn't a sign you're not trusted with "real" work yet — the pipeline is the real work, for most of an engineer's career at this scale.
Part 4 — The Daily Tool Stack
What You'll Actually Have Open Every Day
Distinct from the Foundations series' framework-level stack (PyTorch, JAX), this is the operational layer you'll touch daily regardless of which model architecture your team works on.
A layered daily stack — job scheduling at the base, experiment tracking above it, then monitoring and on-call tooling for when things go wrong (Article 10 of the Foundations series covers the theory the scheduler layer is managing).
Slurm or similar workload managerWeights & Biases (already cited in Article 2)Internal monitoring dashboardsIncident/paging tools
Part 5 — A Realistic First Week
Day by Day, Roughly
Day
Realistic Focus
Day 1–2
Environment setup, repo access, reading onboarding docs, getting a "hello world" job to actually run on the cluster
Day 2–3
Reading existing code in your team's area — Part 2's "10x more reading than writing" in practice
Day 3–4
First small ticket assigned — likely a data pipeline, eval, or tooling fix (Part 3)
Day 4–5
First PR opened, first round of review feedback (Part 2), learning the team's specific conventions
Part 6 — Real Scenarios
Real Scenario Walkthroughs
🐛Scenario A — The Data Loader That Silently Drops Samples
A common realistic first bug: a data-loading pipeline is quietly filtering out a small percentage of training examples due to an edge case in a parsing function — no crash, no error, just slightly less data than intended reaching the model. Finding this requires exactly the data-quality intuition Article 1 of the Foundations series flagged as an underrated skill, plus patience: the bug is invisible unless you specifically instrument the pipeline to count inputs versus outputs at each stage.
The lesson: the most valuable early wins are often invisible-until-found bugs like this — unglamorous, but exactly the kind of work that builds trust fast.
🔁Scenario B — Your First PR Gets Three Rounds of Review Comments
A newcomer's instinct is often to read multiple review rounds as a bad sign. In practice, per Part 2's blast-radius logic, three rounds on a first PR touching shared code is a completely normal outcome — reviewers are calibrating both the specific change and how much context you have, and each round of feedback is effectively free, fast-turnaround mentorship on the codebase's actual conventions.
The lesson: review round count says more about a change's blast radius and your unfamiliarity with local conventions than it does about code quality in isolation.
Readiness Checklist
1
Have you ever submitted a pull request to a codebase you didn't fully understand yet, and incorporated review feedback into a revision?
2
Have you used a job scheduler (even a simple one) to queue work rather than running everything interactively?
3
Have you debugged a "silent" data issue — no crash, just wrong or missing output — rather than only crash-based bugs?
4
Can you read someone else's training configuration file and explain what each major setting controls?
⚠️ What's Missing or Uncertain in This Article
This article describes general, observed industry patterns, not any single employer's exact process. Specific tools, review norms, and onboarding structure vary significantly between organizations and even between teams within the same organization — treat this as a realistic shape to expect, not a literal script, and calibrate against what you actually observe in your first week.
Where This Series Goes Next
Article 2 moves from onboarding to the actual daily work: reading and writing production training code at a scale well beyond Article 2 of the Foundations series' nanoGPT-style reproductions — real configs, launchers, and experiment naming conventions used across a team, not a solo project.
🧭 Closing — The Job Is Mostly the Iceberg, Not the Tip
🎯 The Bottom Line
A frontier lab engineering job's daily reality — reading far more code than you write, having a small config change reviewed more heavily than a large isolated one, being handed a data-pipeline bug instead of a new model to train — isn't a watered-down version of "real" AI work. It is the real work, for the large majority of engineers, the large majority of the time. Arriving with that expectation set correctly, rather than expecting to train a frontier model in week one, is itself the single most useful thing this article can hand you before day one actually starts.