I like research topics that are simple, general, and stand the test of time, and I try to avoid projects that are complicated, task-specific, or short-lived.

Most people (including me) would benefit greatly by spending more time on idea selection, since doing this well is a huge multiplier on research impact. Conversely, working on a narrow topic with little headroom caps the impact of the project, no matter how well it is executed.

A good suggestion from a friend is to either (1) work on a hot topic and do it better than everyone else, or (2) work on something that might become the next hot topic. Strategy 1 is lower risk and requires working very hard. Strategy 2 is higher risk but has potentially very high reward.

🔹 General Research Topic → Broad Benchmark Suite

They scale with the frontier: as foundation models improve, broad topics grow in importance, while narrow ones fade.

It is an open question why exactly scaling works, but here are two hand-wavy reasons. One is that small language models can’t memorize as much knowledge in their parameters, whereas large language models can memorize a huge amount of factual information about the world. A second guess is that while small language models are capacity-constrained, they might only learn first-order correlations in data. Large language models on the other hand, can learn complex heuristics in data.

Intuition 3. Tokens can have very different information density, so give language models time to think.

The solution to this is to give language models more compute by allowing them to perform natural language reasoning before giving the final answer.

You can imagine that if you’re ChatGPT, and as soon as you have to see the prompt you have to immediately start typing, it would be pretty hard to get that question right.

Intuition 2. Learning input-output relationships can be cast as next-word prediction. This is known as in-context learning.

This is an interesting example of how a simple objective, when combined with complex data can lead to highly intelligent behavior (assuming you agree that language models are intelligent).

It is important to understand that this does not mean LLMs will be gods producing 100x code, because virtually no domain that software engineering is useful has a perfect oracle. A perfect oracle is a type of feedback where you are given a “correct/incorrect” answer every single time, and they almost only appear in games as real world typically doesn’t have perfect models of correctness. Winning or losing a game is a perfect oracle, as well as creating a program that can pass the judge in a competitive programming contest.

It is the limit that tells us what we cannot implement via LLMs, and it cannot be solved with agentic approaches.

However, in scientific innovation, we are in a totally different realm where we only care about solving a single problem (train=test!) because it’s an unsolved problem and potentially extremely valuable.

overfitting

any solvable problem that fits those five properties will be solved in the next few years

Speed of iteration

If you consider the history of deep learning, we have seen that virtually anything that can be measured can be optimized

If you consider the history of deep learning, we have seen that virtually anything that can be measured can be optimized

Prefer methods that pass a scaling test: their delta is flat or increasing as you go from S→M→L models.

Extra compute/search

Hard constraints:

If a method is just a shortcut for weak models (e.g., brittle augmentations, dataset-specific features), it will fade.

If a method is just a shortcut for weak models (e.g., brittle augmentations, dataset-specific features), it will fade.

Method: anything you wrap around or plug into the core model: data curation, training tricks, inference procedures, retrieval, tools, constraints, rewards, routing, etc.

Why methods often fade away

🔹 What does “method” mean here?

4. Broader principle

Build Small Internal Tools

summaries of what worked, not the nuanced landscape of the problem

you’re not competing — you’re complementing.

adjacent signals: lots of startups are already building agent frameworks (LangChain, AutoGPT, CrewAI). Their bottlenecks hint at where research could contribute

2020: you can cast any language task as sequence prediction and learn it via pretrain + finetune 2021: scaling to GPT-3 size enables doing arbitrary tasks specified via instructions 2022: scaling to GPT-3.5/PaLM size unlocks reasoning via chain of thought 2023: LLMs themselves can be a product a lot of people will use 2024: to push capabilities past GPT-4, scale test-time compute

Old habit: Just grab a standard benchmark (e.g., GLUE, ImageNet, MMLU) and test your method there. Problem today: Those benchmarks might not stress the thing your method is designed for. You’ll conclude your idea “doesn’t work,” when in fact you just used the wrong test.

Now: Large language models are so capable and multi-task that whether a method works depends a lot on which dataset you test it on.

AlphaEvolve

The field rewards researchers who can translate expensive experimentation into deep, portable ideas.

AcknowledgementsThis blog post features contributions from Gabriel Ilharco. I would like to thank Hattie Zhou, Nelson Liu, Noah Smith, Gabriel Ilharco, Mitchell Wortsman, Luke Zettlemoyer, Aditya Kusupati, Jungo Kasai, and Ofir Press for their valuable feedback on drafts of this blog post.

It is important to note that there is neither right or wrong nor good or bad research style.

He builds hacks, understands the deep relationships of how his hack affects the system, and then extracts this insight in the most minimalistic and well-formulated way possible along with his practical hack.

Navigating this uncertainty is best done through fast iterations and balancing multiple projects to maximize the chances of a big success.

The learning from one doesn’t always transfer, because the ideas are not anchored to a common goal.

✅ Correct in spirit: Solving “make X work for the first time” usually looks like jumping from essentially no working method to a viable solution (say 0% → 70%).

X = real-time object detection on a single GPU.

If they join a crowded subfield late, the low-hanging fruit is gone.

Pick a meaningful dimension of stress (tokens, latency, noise, compositional depth, generalization to new domains, etc.).

Relevance to the field’s trajectory The capability should connect to active conversations in the community. Example: In 2020–2022, instruction-following was “in the air” because GPT-3 showed emergent abilities, but not controllability. So InstructGPT’s capability (follow human instructions) was both new and natural.

Do you want me to break down how to tell, when reading a new paper, whether it’s a “first-time X” paper or a “make X better” paper? That skill will help you classify work quickly.

Alternatively, armed with the knowledge gained from working on the first idea, you might move on to a different idea aligned with the same goal, with a higher chance of success.

If you are thinking in terms of ideas, you’d be easily frustrated and might give the idea a few more attempts before finally giving up and moving on to another, possibly unrelated idea, repeating the same process

John Schulman argues that goals have more longevity than ideas

The main issue is that ideas have a very short lifespan; an idea is unlikely to work at first, might not be novel enough, might be easily scooped

When you’re starting in a new area without a full understanding of the challenges or limitations, It is very tempting to run after a sole idea that you think will work.

What Yang was talking about is reading enough papers to cover most of the literature in your area. Needless to say, it is not only about the paper count you read, although the paper count can serve as a good indicator of how well you are engaged with the literature in your research area.

enabling you to make larger leaps of progress.

new AI capabilities

showing how to do X

choosing a different problem from the rest of the community can lead you to explore different ideas.

initial exploration

Goals also make it possible for a team of researchers to work together and attack different aspects of the problem, whereas idea-driven research is most effectively carried out by “teams” of 1-2 people.

On the other hand, with goal-driven research, your goal will give you a perspective that’s differentiated from the rest of the community. It will lead you to ask questions that no one else is asking,

To make breakthroughs with idea-driven research, you need to develop an exceptionally deep understanding of your subject, and a perspective that diverges from the rest of the community—some can do it, but it’s difficult.

you test a variety of existing methods from the literature, and then you develop your own methods that improve on them

I’ll take goal-driven research to mean that your goal is more specific than your whole subfield’s goal, and it’s more like make X work for the first time than make X work better.

solve problems that bring you closer to that goal.

Follow some sector of the literature.

This AI-driven auto-scheduling is a massive time-saver, eliminating the tedious manual process of trying to Tetris tasks into your day.

A 40 hour time-blocked work week, I estimate, produces the same amount of output as a 60+ hour work week pursued without structure.

Combine with MVO first → secure a skeleton first, then apply Pareto logic for depth.

5. Pareto 80/20 Why #5: Still powerful — often a few papers/experiments/figures give most of the insight. But research is exploratory, so it’s easy to misjudge which 20% matters most until later. Works best if combined with checkpoints. Example: In experiment runs → 2–3 baseline setups cover 80% of insight; no need to sweep every hyperparameter.

6. Progressive Layering Idea: Build the output in layers: skeleton → basic fill → deeper detail → polish. How to apply: Do a quick pass that touches everything at a shallow level, then loop back to deepen. Good for: Ensuring balanced coverage and avoiding “holes.” Limits: Requires resisting the urge to perfect one section before moving on.

4. Greedy Value-per-Time Idea: Always pick the next piece of work that gives the highest value for the time spent. How to apply: For each possible action, estimate: “How much value will this add?” / “How long will it take?” → Do the one with the best ratio. Good for: Gradually building up value in the most efficient order. Limits: Estimation can be rough; might overlook long-term gains.

3. Time-Boxed Checkpoints Idea: Divide your time budget into checkpoints (e.g., 25%, 50%, 75% of T). How to apply: At each checkpoint, stop and take stock. Freeze the current delivery so it’s already usable, then decide whether to add or polish. Good for: Avoiding the trap of chasing “better” until the very last minute. Limits: Requires discipline to actually stop and review.

Lesson: the hidden blocker (“taxonomy dimension unclear”) surfaced cheaply at L0.

Without early reality checks: you drift → big risks discovered late (wasted days). With L0/L1 reality checks: you “fail fast, small, and cheap,” so every later step builds on a working scaffold.

You fix it before scaling up.

utility proxies (does this draft/pipeline already work in some small way?)

8) Common pitfalls (and the fix)

You’ll increase 1–2 knobs per level, never all.

usability-focused.

Converts one far-off, sparse reward into 5–6 immediate feedback signals.

Break your open-ended milestone into layers, each with a feedback proxy:

Solution: tie every proxy to usability: “Can this artifact unblock the next step?” If no → proxy doesn’t count.

Very sharp connection, Dat 👍 — you’re right: open-ended research tasks suffer from the same two problems as reinforcement learning (RL):

Get to “usable” fast → then refine only if it proves valuable.

👉 Philosophy: The goal of early outputs isn’t to be complete — it’s to learn faster.

Philosophy: Closure is manufactured, not discovered. If you don’t impose an end, the task never ends.

If you ship something usable by the deadline → ✅ pipeline works. If not → ❌ pipeline broke (e.g., stuck in collection or polish).

This is the essence of layered refinement: start thin, add depth only if needed.

Momentum

Closes the loop faster → You don’t wait months for evaluation. Prevents drift → Every day/week forces a checkpoint. Shapes behavior → You optimize for progress under time instead of perfect completeness.

Each week is a reward event. It shapes your trajectory forward, instead of leaving you wandering until the “final boss” (PhD defense).

🔹 2. Daily Micro-Outputs as Rewards Reward Rule: +1 if by end of day you capture something tangible (a new cluster, a new theme, or one updated paragraph of outline). Penalty Rule: 0 if you only consumed/collected without producing. 👉 This ensures every day has a checkpoint. Even if small, it signals whether the pipeline is alive.

That is feedback about the weak link in your pipeline: maybe your clustering method is too slow, or you’re over-expanding the input set. Without the deadline, you’d never notice the bottleneck — you’d just keep drifting.

If you don’t ship → ❌ that’s feedback that your process got stuck in over-collection, over-polishing, or paralysis.

Iterate in Layers → Each week is one “thin slice” of output → feedback → refinement.

Manufacture Feedback → Use proxy tests, peer review, or time deadlines instead of waiting for perfect signals.

Force Output → Every cycle must produce something tangible (outline, synthesis, test).

Cap Input → Don’t endlessly collect; pre-decide how much you’ll allow yourself.

The goal is closure and forward motion, not exhaustiveness.

MVO: a plain input box with keyword search that returns results by exact match. → This lets you confirm: Do users even use the search bar? If yes, you iterate.

No natural stopping point Completeness instinct: A dev team building a new chat feature decides they must include typing indicators, message reactions, file sharing, and push notifications before launch. Result: Months pass before users even test the basic messaging experience. MVO version: Ship a bare-bones text-only chat. Once people actually use it, you’ll know if features like reactions are worth adding.

Layered refinement → MVO outputs become scaffolds. You can always enrich them later, but at least you have something concrete.

Progress > Perfection → momentum compounds over time, while perfectionism stalls.

Barely acceptable ≠ low quality. It means the smallest unit of work that is valid enough to be tested, evaluated, or built upon. The spirit is: cross the acceptance line quickly, then refine if it proves worthwhile.

3. Reinforcement Learning (Sparse, Delayed Feedback)

If you reach the deadline and you have something coherent → ✔ that’s feedback that your process worked. If you reach the deadline and you’re still collecting without synthesis → ❌ that’s feedback that your process got stuck, and you must force closure.

Your first synthesis attempt is the feedback resource. If it produces something coherent, that’s your signal to stop collecting and move forward. If it produces obvious holes, those holes tell you precisely what to collect next.

Instead of waiting for someone else to tell you “you have enough,” you let the process of synthesis tell you.

Force yourself to draft a 1-page proto-framework: Section 1: How I choose problems Section 2: How I generate ideas Section 3: How I design experiments Section 4: How I reflect and adapt Even if rough, you’ll quickly feel whether your collected advices are enough to fill these slots.

If your 1-page proto-style already helps you reason, structure, or discuss, that’s feedback enough. You don’t need 100% collection before synthesis.

🔹 D. Time-Bound Feedback Impose a milestone deadline: “If I can’t cluster 15 advices into a draft framework by end of this week, I must ship an MVO draft anyway.” Here, the deadline itself is the feedback resource → it forces closure, telling you that completeness is no longer the metric; progress is.

It closes the milestone with evidence and prevents “analysis paralysis.”

Without this cap, “just one more run” can spiral into weeks of GPU drain.

A single config eliminates “rabbit holes” of hyperparameter tuning. One config = just enough to generate a signal.

You don’t need the entire dataset to test whether your pipeline works.

Evidence-First Approach → once you have a minimal version, you can test if it holds water before investing more. Layered Refinement → each MVO creates a scaffold; if time allows, you can enrich it later.

Instead of “complete everything,” you aim for the leanest but valid version that lets you move forward.

Balance Depth vs. Breadth Early milestones should prioritize breadth (collect many inputs). Later milestones should prioritize depth (synthesize into frameworks).

Set Sprint-Style Boundaries Treat milestones as 1–2 week sprints (like in software dev). End each sprint with a “demo” artifact (outline, table, draft).

Work Backwards from Deadline Decide: “I want a working draft in 1 month”. Break backwards into weekly checkpoints (Week 1 = Collect, Week 2 = Cluster, Week 3 = Synthesize, Week 4 = Draft).

Planning everything down to the day months in advance is unrealistic in research. But if you don’t plan at all, you drift. The solution: think in layers of stability.

Research = uncertainty. Unlike building a house, you don’t know all the materials or obstacles ahead.

Visible progress tracker Have a simple Kanban board (Notion/Trello) with columns: Month → Week → Day → Done. This makes milestones not just a plan, but a visual progress bar.

Define Minimum Viable Output (MVO) Ask: What’s the smallest version of this milestone that proves progress? Example: For Collect → don’t wait until you’ve read 50 blog posts. Ten high-quality ones may be enough to proceed.

Without milestones → work naturally keeps inflating.

A milestone introduces artificial scarcity: “I only have 2 weeks to gather advice → better prioritize the most important sources.”

“Keep reading until I know enough” → “By end of Week 2, I’ll have 20 summarized advices.” “Keep synthesizing until it feels right” → “By end of Week 4, I’ll have a draft, even if messy.”

Now, even though the universe of possible advice is infinite, you’ve forced closure at each step.

Creates time pressure → reduces endless wandering. Builds momentum by giving you small wins. Forces trade-offs: if time is almost up, you’ll cut distractions more aggressively.

derailing

Exploration Log → Any recurring themes worth folding into (1) later?

prioritization

Step 4: Tight Execution Loops for (1)

Example: Friday afternoons = serendipity time.

Rapid

Step 1: Define a Sharp Focus Box Write down your main goal in one crisp sentence: “Collect and synthesize research advice on problem-driven research to develop my own meta research style.” Break this into milestones (e.g. “Collect 20 pieces of advice → Cluster key themes → Draft my meta approach”). Keep this Focus Box visible in your workspace (sticky note, Notion top bar). This serves as your anchor whenever FOMO tries to pull you away.

A good “bit to flip” has 3 properties: Universality: shared by many current methods. Constraining: it actually limits progress toward capabilities. Replaceability: you can imagine an alternative (not just removing, but replacing with something better). Bad flips: assumptions that were helpful approximations without being real bottlenecks → flipping them just adds complexity.

3. Build Trust in Your Evaluation Harness If you never rerun baselines, how do you know: your code, logging, and evaluation are sound? improvements aren’t due to hidden differences in implementation? By reproducing others’ results in your own harness, you ensure a fair playing field for comparing your new method. 👉 Otherwise, reviewers (and yourself) can’t be sure whether gains are real or artifacts.

Papers tend to report best-case or average numbers, not the messy reality: instability across seeds, catastrophic drops mid-training, reproducibility quirks. By rerunning, you see the variance, fragility, and edge cases that are invisible in benchmark tables. 👉 Schulman’s insight about instability in policy gradients (leading to TRPO) only came from seeing the failures himself, not from reported scores.

Different methods break in different ways. Running several gives you a menu of failure patterns: instability, sample inefficiency, poor generalization, etc. These failures help surface the real subproblems you need to tackle.

5. De-Risk Your Research Jumping straight into your own method is high-risk — it may fail for reasons you could’ve predicted by simply running known baselines. Testing others first is a low-risk learning stage that grounds your work in reality before you spend months on a new algorithm. 👉 It’s like scouting the terrain before building your own path.

Running others’ methods forces you to internalize their assumptions, strengths, and weaknesses. Often, you notice small tweaks that make a huge difference, or you realize why a method doesn’t scale to your problem. Schulman explicitly said: testing many methods was a way of discovering what properties mattered most (stability, variance reduction, etc.). 👉 This hands-on process sharpens your taste and informs your own design choices.

Lets you add comments anywhere on a webpage, including ChatGPT.

Highlight text on ChatGPT responses.

Add sticky notes/comments