If this pattern holds, then current work on loop training pipelines might be in the "pre-ResNet" phase. The field is solving the hard problem of how to make destructive surgery work, when the real breakthrough will be discovering a non-destructive surgery that doesn't need the hard solutions.

The reason ResNet worked at 100+ layers when previous architectures failed at 20 layers is almost entirely the additive structure. Everything else — batch norm, careful initialization — was supporting machinery. The core insight was: make the default computation be the identity, and let the network learn deviations from identity.

The retrofit playbook is incomplete without surgery recipes. Compare to the MoE retrofit story: Mixtral didn't just succeed because the training for upcycled MoE was figured out. It succeeded because the surgery — how to initialize expert weights from a dense base, how to insert routing layers, how to handle the load balancing transition — was also figured out. Those surgery techniques were as important as the training techniques, and they took multiple papers and years to mature.

So DeltaFormer is recovering some of the inductive bias of RNNs (state-tracking via recurrent memory updates) without abandoning the transformer's parallelizability — because the delta-rule recurrence can still be computed efficiently in a chunked/parallel form.

shorten CoT chains while preserving (or improving) reasoning quality, by moving the parts of reasoning that don't benefit from verbalization into latent space.

If looping-retrofit really does have LoRA-style adoption potential, then the research path that maximizes impact isn't "build the best looped model" — it's "build the cleanest retrofit recipe."

Retrofit-for-looping needs a specific pitch. "Better reasoning on multi-hop tasks at 5% of pretraining cost" is concrete and testable. "Generally better models" is not. The community adopts capabilities tied to specific problems they have.

Retrofit-for-looping needs an equivalent: a clean prescription that says "take base model X, apply procedure Y on data Z, train for N tokens, you get a looped version with capability gain Q." Retrofitted Recurrence is the closest thing today, but the recipe is still rough. The research opportunity is to refine it.

If d is too small, beyond some loop count you're overwriting useful information faster than you're adding it.

The key insight: answer-level diagnostics are cheap but coarse. They help the reflection LM form hypotheses but rarely confirm them.

If you ask about coding now, she doesn't judge you; she gives you "inside info" to help you pass because she wants her candidate to win.

Your updated, final belief after considering the evidence.

and, this is why it is tempting for us, scientists, to think that this is the right and perhaps only way to tackle challenging problems. first, figure out precise mechanism by which phenomena behind these challenging problems happen, and then based on the uncovered mechanism, come up with solutions to these problems. surely once we know why and who problems manifest themselves, we can fix the problems.

The Limitation: Its greatest weakness is its inability to perform extrapolation.

It is fundamentally an exploratory approach.

It is fundamentally an exploitative approach, seeking the "best" answer according to the flawed map it's given.

The system's potential is demonstrated through concrete validation in three biomedical areas: drug repurposing, novel target discovery, and explaining mechanisms of antimicrobial resistance. For instance, it proposed drug candidates for acute myeloid leukemia that showed tumor inhibition in vitro, showcasing its ability to produce genuinely valuable and original scientific insights. This work frames a new vision for AI: a system that can navigate the high-dimensional space of existing scientific knowledge to discover the unknown.

Foundational Models and Scientific Discovery: The AI Co-Scientist

Proposes a multi-agent system based on Gemini 2.0 to automate hypothesis generation in scientific discovery, using a "generate, debate, and evolve" approach.

A crucial operational maxim is to "be stubborn on the vision and flexible on the details," acknowledging that this flexibility is necessary because the world is changing [1].

change very slowly

points of stability

safety mechanism — it builds understanding rather than surface mimicry.

To understand, we need to find an internal model that can generate those patterns.

It becomes more honest as it becomes more powerful.

Formally, it minimizes something like log loss or cross-entropy between predicted probabilities and actual observed outcomes — a metric that rewards calibrated truthfulness.

In standard RL or reward-maximizing setups, an agent can over-optimize the reward proxy (Goodhart’s law) and exploit loopholes. A Bayesian, inference-only system doesn’t optimize for outcomes—it merely models them. The ensemble of hypotheses provides natural regularization against runaway single-metric optimization.

Toronto team’s output is on general AI capabilities (LLMs, generative models, agent learning), rather than domain-specific science tools.

the prevailing corporate governance models are fundamentally ill-suited to the unique, long-term safety requirements of AGI development.

Institutionalizing the Vision: The Role of LawZero

The "Guardrail" Function: Using Safe AI to Monitor Unsafe AI

non-agentic. Instead of being trained to take actions to achieve goals, it is designed to explain the world from observations.

graph Laplacian

boundary value problem on a graph

discrete Poisson equation

Adapting Geometry for Directedness

The GFlowNet "flow matching" loss condition can be interpreted using this language, relating the flow into a state to the flow out of it.

dynamics

The production rules of a grammar are the constraints that force the data to a low-dimensional manifold. A molecule's grammar, for instance, prevents the vast majority of random atom-and-bond combinations from ever being formed, concentrating all valid molecules onto a specific, structured surface.

improving predictive models of the world

Where MuZero actually sits

diffusion needs extra machinery to cope with discreteness

Native fit for combinatorial/structured objects.

Discrete/combinatorial data is awkward.

these biases might over-constrain search and prevent discovery of novel molecules

Effect: credit from terminal rewards can reach all ancestors in one step through the balancing equations — no need to wait for step-by-step temporal backups.

So instead of waiting for a single reward at the end of a trajectory, each local edge is trained to satisfy a conservation law consistent with terminal rewards.

Aggregating Multiple Trajectories

Local Signal

RL would need repeated training or MCMC-like sampling for each new query.

Efficient Exploration of Combinatorial Spaces

GFlowNets use flow-matching (inflow = outflow) as a local conservation law — mathematically simpler, more stable, better credit assignment.

Produces a distribution over solutions — crucial for scientific discovery, Bayesian inference, or multi-modal problems.

Inefficient Exploration

Single Trajectory Credit Assignment

interpolates/extrapolates reward structure: If many molecules with a substructure A had high reward, The policy will bias flow toward other molecules that also contain substructure A — even if it never explicitly saw them.

Generalization across modes

Every new sample requires running a potentially long Markov chain. No benefit is carried over between runs.

The above two equations are forced to be consistent (i.e. there is an FFF that gives rise to both PBP_BPB​ and PFP_FPF​) when FFF satisfies the flow-matching constraint (the amount of entering flow equals the amount of outgoing flow), which is not necessarily true when FFF is estimated by a neural network that is being trained (and we have not fully completed training and brought the training loss to 0 everywhere).

System 1 (policy net) learns to approximate those judgments instantly.

They don’t use explicit grammar rules. Instead, the grammar is implicitly encoded in the model’s weights.

The world is combinatorially large (endless possible combinations). Humans generalize well by reusing parts without needing to see every possible combo in training.

AlphaZero → trained on massive self-play, now plays “intuitively” via forward neural evaluation (System 1).

A small model might need symbolic rules to parse and generate. A large LLM trained on trillions of tokens can produce fluent sentences directly, implicitly encoding the grammar.

Represented symbolically, compositionally, or in step-by-step logic. Conscious and accessible: you can bring it into working memory, verbalize it, and explain it.

Drug discovery models (like GFlowNets, graph diffusion, VAEs) exploit this structure: they learn to navigate and generate only chemically valid molecules rather than arbitrary graphs.

Sequential Construction

2. Initially introduced for active learning in discrete spaces

Instead of treating a design as one huge blob (like a voxel grid), CGID represents objects as a composition of functional parts.

1. Does a GFlowNet need training data points? Not necessarily. If you have a reward function that you can compute for any candidate object, the GFlowNet can train purely by sampling objects (even ones it has never seen before) and scoring them. This is why they’re attractive for drug/material design: you don’t need a huge dataset of known good molecules, just a scoring function (simulator, energy model, ML predictor) to evaluate new candidates.

Data-efficiency problem: to represent rare but valuable cases, you need a lot of data containing those cases.

By itself: can sample realistic shapes, but functionality is random (not aligned with tasks).

Rather than finetuning the whole diffusion model, it optimizes the conditioning embeddings (e.g., CLIP embeddings) with feedback from simulation performance.

soft robots

LLM priors keep evolving

But if they have an energy landscape (a surface with valleys and peaks corresponding to good vs bad configurations), then you don’t need to brute force. You just need to learn the gradient — the direction downhill toward stable/valid solutions.

This assumption is empirical: generative models (autoencoders, GANs, diffusion models, LLMs) succeed precisely because such structure exists and is learnable.

In mathematical terms, if the total search space is size NNN, but the effective dimensionality of “plausible” or “stable” solutions is only size M≪NM \ll NM≪N, then we say the domain has exploitable structure.

stable, reusable patterns (games with coherent strategy, protein physics shaped by evolution).

What’s strong about P

i'm not sure if this is blackpill or whitepill, but my there are a heap of new papers along with my own experiences that are showing "best of N is all you need" for most problems as long as: - sufficient core knowledge was included in the training data - the model is sufficiently large / you use more than 1 model to promote reasonable idea diversity - N is sufficiently large for the complexity of the problem at hand - you have some reasonable discrimination process at the end to determine / approximate "best" result we really haven't come close to leveraging the full potential of existing models, and the antiquated sampling process / approaches are the single biggest culprit

This is already happening with smaller labs: open-source models are trained partly on outputs from frontier APIs.

What Culture Does for Humans

Aidan hints at this: culture is to humans as wrappers are to models

If they achieve network effects (large user base, ecosystem of integrations, proprietary data flows), they could become the “operating system” for interacting with AI.

Consumers

5. Cultural and Human Analogies

Both groups are in a race to the bottom of commoditization, where only brand, UX, or network effects (e.g., Perplexity’s model-switching) can provide some edge

Wrappers that add UX, workflow, and integration value may persist, but rarely become billion-dollar companies.

Even if scaling is more generally efficient than search, search allows for quicker intelligence in narrow domains. Training larger foundation models is slow. With search, you don’t have to wait.

I like research topics that are simple, general, and stand the test of time, and I try to avoid projects that are complicated, task-specific, or short-lived.

Most people (including me) would benefit greatly by spending more time on idea selection, since doing this well is a huge multiplier on research impact. Conversely, working on a narrow topic with little headroom caps the impact of the project, no matter how well it is executed.

A good suggestion from a friend is to either (1) work on a hot topic and do it better than everyone else, or (2) work on something that might become the next hot topic. Strategy 1 is lower risk and requires working very hard. Strategy 2 is higher risk but has potentially very high reward.

🔹 General Research Topic → Broad Benchmark Suite

They scale with the frontier: as foundation models improve, broad topics grow in importance, while narrow ones fade.

It is an open question why exactly scaling works, but here are two hand-wavy reasons. One is that small language models can’t memorize as much knowledge in their parameters, whereas large language models can memorize a huge amount of factual information about the world. A second guess is that while small language models are capacity-constrained, they might only learn first-order correlations in data. Large language models on the other hand, can learn complex heuristics in data.

Intuition 3. Tokens can have very different information density, so give language models time to think.

The solution to this is to give language models more compute by allowing them to perform natural language reasoning before giving the final answer.

You can imagine that if you’re ChatGPT, and as soon as you have to see the prompt you have to immediately start typing, it would be pretty hard to get that question right.

Intuition 2. Learning input-output relationships can be cast as next-word prediction. This is known as in-context learning.

This is an interesting example of how a simple objective, when combined with complex data can lead to highly intelligent behavior (assuming you agree that language models are intelligent).

It is important to understand that this does not mean LLMs will be gods producing 100x code, because virtually no domain that software engineering is useful has a perfect oracle. A perfect oracle is a type of feedback where you are given a “correct/incorrect” answer every single time, and they almost only appear in games as real world typically doesn’t have perfect models of correctness. Winning or losing a game is a perfect oracle, as well as creating a program that can pass the judge in a competitive programming contest.

It is the limit that tells us what we cannot implement via LLMs, and it cannot be solved with agentic approaches.

However, in scientific innovation, we are in a totally different realm where we only care about solving a single problem (train=test!) because it’s an unsolved problem and potentially extremely valuable.

overfitting

any solvable problem that fits those five properties will be solved in the next few years

Speed of iteration

If you consider the history of deep learning, we have seen that virtually anything that can be measured can be optimized

If you consider the history of deep learning, we have seen that virtually anything that can be measured can be optimized

Prefer methods that pass a scaling test: their delta is flat or increasing as you go from S→M→L models.

Extra compute/search

Hard constraints:

If a method is just a shortcut for weak models (e.g., brittle augmentations, dataset-specific features), it will fade.

If a method is just a shortcut for weak models (e.g., brittle augmentations, dataset-specific features), it will fade.

Method: anything you wrap around or plug into the core model: data curation, training tricks, inference procedures, retrieval, tools, constraints, rewards, routing, etc.

Why methods often fade away

🔹 What does “method” mean here?

4. Broader principle

Build Small Internal Tools

summaries of what worked, not the nuanced landscape of the problem

you’re not competing — you’re complementing.

adjacent signals: lots of startups are already building agent frameworks (LangChain, AutoGPT, CrewAI). Their bottlenecks hint at where research could contribute

2020: you can cast any language task as sequence prediction and learn it via pretrain + finetune 2021: scaling to GPT-3 size enables doing arbitrary tasks specified via instructions 2022: scaling to GPT-3.5/PaLM size unlocks reasoning via chain of thought 2023: LLMs themselves can be a product a lot of people will use 2024: to push capabilities past GPT-4, scale test-time compute

Old habit: Just grab a standard benchmark (e.g., GLUE, ImageNet, MMLU) and test your method there. Problem today: Those benchmarks might not stress the thing your method is designed for. You’ll conclude your idea “doesn’t work,” when in fact you just used the wrong test.

Now: Large language models are so capable and multi-task that whether a method works depends a lot on which dataset you test it on.

AlphaEvolve

The field rewards researchers who can translate expensive experimentation into deep, portable ideas.

AcknowledgementsThis blog post features contributions from Gabriel Ilharco. I would like to thank Hattie Zhou, Nelson Liu, Noah Smith, Gabriel Ilharco, Mitchell Wortsman, Luke Zettlemoyer, Aditya Kusupati, Jungo Kasai, and Ofir Press for their valuable feedback on drafts of this blog post.

It is important to note that there is neither right or wrong nor good or bad research style.

He builds hacks, understands the deep relationships of how his hack affects the system, and then extracts this insight in the most minimalistic and well-formulated way possible along with his practical hack.

Navigating this uncertainty is best done through fast iterations and balancing multiple projects to maximize the chances of a big success.

The learning from one doesn’t always transfer, because the ideas are not anchored to a common goal.

✅ Correct in spirit: Solving “make X work for the first time” usually looks like jumping from essentially no working method to a viable solution (say 0% → 70%).

X = real-time object detection on a single GPU.

If they join a crowded subfield late, the low-hanging fruit is gone.

Pick a meaningful dimension of stress (tokens, latency, noise, compositional depth, generalization to new domains, etc.).

Relevance to the field’s trajectory The capability should connect to active conversations in the community. Example: In 2020–2022, instruction-following was “in the air” because GPT-3 showed emergent abilities, but not controllability. So InstructGPT’s capability (follow human instructions) was both new and natural.

Do you want me to break down how to tell, when reading a new paper, whether it’s a “first-time X” paper or a “make X better” paper? That skill will help you classify work quickly.

Alternatively, armed with the knowledge gained from working on the first idea, you might move on to a different idea aligned with the same goal, with a higher chance of success.

If you are thinking in terms of ideas, you’d be easily frustrated and might give the idea a few more attempts before finally giving up and moving on to another, possibly unrelated idea, repeating the same process

John Schulman argues that goals have more longevity than ideas

The main issue is that ideas have a very short lifespan; an idea is unlikely to work at first, might not be novel enough, might be easily scooped

When you’re starting in a new area without a full understanding of the challenges or limitations, It is very tempting to run after a sole idea that you think will work.

What Yang was talking about is reading enough papers to cover most of the literature in your area. Needless to say, it is not only about the paper count you read, although the paper count can serve as a good indicator of how well you are engaged with the literature in your research area.

enabling you to make larger leaps of progress.

new AI capabilities

showing how to do X

choosing a different problem from the rest of the community can lead you to explore different ideas.

initial exploration

Goals also make it possible for a team of researchers to work together and attack different aspects of the problem, whereas idea-driven research is most effectively carried out by “teams” of 1-2 people.

On the other hand, with goal-driven research, your goal will give you a perspective that’s differentiated from the rest of the community. It will lead you to ask questions that no one else is asking,

To make breakthroughs with idea-driven research, you need to develop an exceptionally deep understanding of your subject, and a perspective that diverges from the rest of the community—some can do it, but it’s difficult.

you test a variety of existing methods from the literature, and then you develop your own methods that improve on them

I’ll take goal-driven research to mean that your goal is more specific than your whole subfield’s goal, and it’s more like make X work for the first time than make X work better.

solve problems that bring you closer to that goal.

Follow some sector of the literature.

This AI-driven auto-scheduling is a massive time-saver, eliminating the tedious manual process of trying to Tetris tasks into your day.

A 40 hour time-blocked work week, I estimate, produces the same amount of output as a 60+ hour work week pursued without structure.

Combine with MVO first → secure a skeleton first, then apply Pareto logic for depth.

5. Pareto 80/20 Why #5: Still powerful — often a few papers/experiments/figures give most of the insight. But research is exploratory, so it’s easy to misjudge which 20% matters most until later. Works best if combined with checkpoints. Example: In experiment runs → 2–3 baseline setups cover 80% of insight; no need to sweep every hyperparameter.

6. Progressive Layering Idea: Build the output in layers: skeleton → basic fill → deeper detail → polish. How to apply: Do a quick pass that touches everything at a shallow level, then loop back to deepen. Good for: Ensuring balanced coverage and avoiding “holes.” Limits: Requires resisting the urge to perfect one section before moving on.

4. Greedy Value-per-Time Idea: Always pick the next piece of work that gives the highest value for the time spent. How to apply: For each possible action, estimate: “How much value will this add?” / “How long will it take?” → Do the one with the best ratio. Good for: Gradually building up value in the most efficient order. Limits: Estimation can be rough; might overlook long-term gains.

3. Time-Boxed Checkpoints Idea: Divide your time budget into checkpoints (e.g., 25%, 50%, 75% of T). How to apply: At each checkpoint, stop and take stock. Freeze the current delivery so it’s already usable, then decide whether to add or polish. Good for: Avoiding the trap of chasing “better” until the very last minute. Limits: Requires discipline to actually stop and review.

Lesson: the hidden blocker (“taxonomy dimension unclear”) surfaced cheaply at L0.

Without early reality checks: you drift → big risks discovered late (wasted days). With L0/L1 reality checks: you “fail fast, small, and cheap,” so every later step builds on a working scaffold.

You fix it before scaling up.

utility proxies (does this draft/pipeline already work in some small way?)

8) Common pitfalls (and the fix)

You’ll increase 1–2 knobs per level, never all.

usability-focused.

Converts one far-off, sparse reward into 5–6 immediate feedback signals.

Break your open-ended milestone into layers, each with a feedback proxy:

Solution: tie every proxy to usability: “Can this artifact unblock the next step?” If no → proxy doesn’t count.

Very sharp connection, Dat 👍 — you’re right: open-ended research tasks suffer from the same two problems as reinforcement learning (RL):

Get to “usable” fast → then refine only if it proves valuable.

👉 Philosophy: The goal of early outputs isn’t to be complete — it’s to learn faster.

Philosophy: Closure is manufactured, not discovered. If you don’t impose an end, the task never ends.

If you ship something usable by the deadline → ✅ pipeline works. If not → ❌ pipeline broke (e.g., stuck in collection or polish).

This is the essence of layered refinement: start thin, add depth only if needed.

Momentum

Closes the loop faster → You don’t wait months for evaluation. Prevents drift → Every day/week forces a checkpoint. Shapes behavior → You optimize for progress under time instead of perfect completeness.

Each week is a reward event. It shapes your trajectory forward, instead of leaving you wandering until the “final boss” (PhD defense).

🔹 2. Daily Micro-Outputs as Rewards Reward Rule: +1 if by end of day you capture something tangible (a new cluster, a new theme, or one updated paragraph of outline). Penalty Rule: 0 if you only consumed/collected without producing. 👉 This ensures every day has a checkpoint. Even if small, it signals whether the pipeline is alive.

That is feedback about the weak link in your pipeline: maybe your clustering method is too slow, or you’re over-expanding the input set. Without the deadline, you’d never notice the bottleneck — you’d just keep drifting.

If you don’t ship → ❌ that’s feedback that your process got stuck in over-collection, over-polishing, or paralysis.

Iterate in Layers → Each week is one “thin slice” of output → feedback → refinement.

Manufacture Feedback → Use proxy tests, peer review, or time deadlines instead of waiting for perfect signals.

Force Output → Every cycle must produce something tangible (outline, synthesis, test).

Cap Input → Don’t endlessly collect; pre-decide how much you’ll allow yourself.

The goal is closure and forward motion, not exhaustiveness.

MVO: a plain input box with keyword search that returns results by exact match. → This lets you confirm: Do users even use the search bar? If yes, you iterate.

No natural stopping point Completeness instinct: A dev team building a new chat feature decides they must include typing indicators, message reactions, file sharing, and push notifications before launch. Result: Months pass before users even test the basic messaging experience. MVO version: Ship a bare-bones text-only chat. Once people actually use it, you’ll know if features like reactions are worth adding.

Layered refinement → MVO outputs become scaffolds. You can always enrich them later, but at least you have something concrete.

3. Reinforcement Learning (Sparse, Delayed Feedback)

If you reach the deadline and you have something coherent → ✔ that’s feedback that your process worked. If you reach the deadline and you’re still collecting without synthesis → ❌ that’s feedback that your process got stuck, and you must force closure.

Your first synthesis attempt is the feedback resource. If it produces something coherent, that’s your signal to stop collecting and move forward. If it produces obvious holes, those holes tell you precisely what to collect next.

Instead of waiting for someone else to tell you “you have enough,” you let the process of synthesis tell you.

Force yourself to draft a 1-page proto-framework: Section 1: How I choose problems Section 2: How I generate ideas Section 3: How I design experiments Section 4: How I reflect and adapt Even if rough, you’ll quickly feel whether your collected advices are enough to fill these slots.

If your 1-page proto-style already helps you reason, structure, or discuss, that’s feedback enough. You don’t need 100% collection before synthesis.

🔹 D. Time-Bound Feedback Impose a milestone deadline: “If I can’t cluster 15 advices into a draft framework by end of this week, I must ship an MVO draft anyway.” Here, the deadline itself is the feedback resource → it forces closure, telling you that completeness is no longer the metric; progress is.