Lilly Tech Systems
Research Taste & Vision
Research taste is the most difficult skill to evaluate and the most valued by hiring committees. It is the ability to identify which problems are important, which approaches are promising, and which questions, if answered, would unlock broad progress. This lesson covers how interviewers assess research taste and how to demonstrate it.
What Is Research Taste?
Research taste is the intuition that allows a researcher to distinguish between problems that are merely publishable and problems that are genuinely important. Richard Hamming famously asked: "What are the important problems in your field? And why aren't you working on them?"
Problem Selection
Taste means choosing to work on problems where progress would have outsized impact. The transformer paper mattered not because attention was novel (it was not), but because the authors identified the right combination of ideas at the right time to unlock a new paradigm.
Method Selection
Taste means knowing which tools to apply to a problem. When should you use a theoretical approach versus an empirical one? When is scaling the answer versus architectural innovation? Researchers with taste make these decisions correctly more often than chance.
Knowing What to Ignore
Taste means knowing which trends are fads and which represent genuine progress. Researchers with taste do not chase every benchmark or follow every hype cycle. They maintain focus on problems that matter, even when the field's attention is elsewhere.
Connecting Ideas
Taste often manifests as the ability to see connections between disparate areas. The best research often comes from applying ideas from one field to problems in another: optimal transport to generative modeling, information theory to representation learning, control theory to alignment.
Interview Questions That Test Research Taste
These are real questions asked at Google DeepMind, Meta FAIR, OpenAI, and Anthropic. Practice answering each one thoughtfully.
Q1: "What is the most important open problem in AI right now?"
How to answer: Pick a problem you genuinely care about and can defend. The interviewer is not testing whether you pick the "right" problem — they are testing whether you can articulate why it matters and what progress would look like.
Example strong answer: "I believe the most important open problem is understanding and controlling what large language models actually learn during training — mechanistic interpretability at scale. We are deploying models with capabilities we do not understand, and the gap between what models can do and what we understand about how they do it is growing. If we could reliably reverse-engineer the algorithms learned by transformers, we could verify alignment claims, predict emergent capabilities before they appear, and build safer systems. The challenge is that current interpretability methods (probing, activation patching) do not scale to the largest models, and we lack a theoretical framework for what 'understanding a neural network' even means."
What makes this answer strong: It identifies a specific problem, explains why it matters broadly (safety, prediction, control), acknowledges the current state of the art, and identifies what makes the problem hard.
Q2: "If you had unlimited compute for 1 year, what would you research?"
How to answer: This tests whether you think about research at a strategic level, not just at the level of individual experiments. The "unlimited compute" framing removes practical constraints to see what you really care about.
Example strong answer: "I would run a systematic study of how reasoning capabilities emerge and change across model scales, architectures, and training data compositions. Specifically, I would train hundreds of models from 100M to 100B parameters on carefully controlled data mixtures, evaluating on a comprehensive reasoning benchmark suite at regular checkpoints. The goal is to build a predictive theory of when and why specific reasoning capabilities emerge, so we can predict what a 10x larger model will be able to do before we train it. This would transform AI development from 'train and hope' to 'predict and verify,' which has massive implications for both capability forecasting and safety."
Q3: "What currently popular research direction do you think is overrated, and why?"
How to answer: This is a trap question if answered carelessly. Do not be dismissive or arrogant. Acknowledge the value of the direction while explaining your concerns. Show that your skepticism comes from deep understanding, not ignorance.
Example strong answer: "I think the current emphasis on benchmark-driven evaluation of language models is overrated, though I understand why it exists. We optimize for MMLU, HumanEval, and GSM8K as if they are ground truth, but these benchmarks have known contamination issues, they measure narrow capabilities, and improving on them does not necessarily mean the model is more generally capable or safer. I think the field would benefit from investing more in evaluation methodology — developing benchmarks that resist contamination, measure transfer to novel tasks, and evaluate failure modes rather than just success cases. The challenge is that good evaluation is less glamorous than new architectures, so it gets less attention."
What makes this answer strong: It is nuanced (acknowledges why benchmarks exist), specific (names concrete problems), and constructive (proposes an alternative direction).
Q4: "Name a paper from outside your subfield that influenced your thinking."
Why they ask this: Researchers who read only within their subfield produce incremental work. Breakthrough ideas often come from cross-pollination. This question tests the breadth of your intellectual curiosity.
How to answer: Choose a paper from a genuinely different field (not just a neighboring subfield) and explain how it changed how you think about your own research. Be specific about the connection.
Example: "Kauffman's work on autocatalytic sets in theoretical biology influenced how I think about emergent capabilities in neural networks. Kauffman showed that when a chemical system reaches a critical complexity threshold, self-sustaining reaction networks spontaneously emerge. I think something analogous happens in neural networks: when a model reaches a critical size and training compute, previously absent capabilities suddenly appear not because they were gradually learned, but because the internal representations crossed a phase transition. This lens has made me think about emergence in AI more carefully and question whether the scaling laws view (smooth power-law improvement) captures the full picture."
Developing Long-Term Research Vision
Interviewers often ask: "Where do you see your research program in 5 years?" or "What is your research vision?" This tests whether you think strategically about research, not just tactically.
How to Build a Research Vision
| Component | What It Answers | Example |
|---|---|---|
| North star question | What is the big question driving your research? | "How do we build AI systems whose behavior we can reliably predict and verify?" |
| Research pillars | What are 2–3 concrete research directions that advance toward the north star? | "(1) Mechanistic interpretability of reasoning, (2) Formal verification of neural network properties, (3) Scaling laws for safety-relevant behaviors" |
| Near-term projects | What can you do in the next 6–12 months? | "Develop a circuit-level understanding of how transformers implement multi-step arithmetic" |
| Success criteria | How will you know if you are making progress? | "If our interpretability tools can predict a model's behavior on held-out tasks from its internal representations alone" |
Navigating Uncertainty in Research
Research is inherently uncertain. Most experiments fail. Most hypotheses are wrong. Interviewers want to see that you can navigate this uncertainty productively.
Q5: "Tell me about a time your research hypothesis was wrong. What did you do?"
How to answer: Be honest about a real failure. Then explain: (1) How you discovered the hypothesis was wrong, (2) What you learned from the failure, (3) How you pivoted. The best researchers extract value from negative results.
What interviewers look for: Intellectual honesty (not blaming tools or data), systematic debugging (not random changes), and the ability to pivot (not stubbornly pursuing a dead end). A researcher who has never been wrong has never tried anything interesting.
Q6: "How do you decide when to abandon a research direction?"
Strong answer elements:
- Evidence-based criteria: "I set concrete milestones at the start. If after 3 months of focused effort, the approach has not shown any signal on the simplest version of the problem, I reassess."
- Distinguish between 'hard' and 'wrong': "Some problems are hard but tractable — slow progress is still progress. Others are fundamentally misframed. I try to distinguish between these by asking: 'If this approach worked perfectly, would it actually solve the problem I care about?'"
- Sunk cost awareness: "I actively guard against the sunk cost fallacy. Three months of work on a dead approach does not justify three more months."
- Pivot, do not just quit: "Abandoning a direction does not mean throwing away everything. Failed approaches often reveal constraints or insights that inform the next attempt."
Connecting Research Fields
The most impactful AI research often happens at the intersection of fields. Interviewers value candidates who can draw connections across areas. Here are examples of productive cross-field connections:
| Connection | Fields | Impact |
|---|---|---|
| Diffusion models | Statistical physics + generative modeling | Score-based generative models revolutionized image synthesis by importing ideas from Langevin dynamics |
| Optimal transport for domain adaptation | Pure mathematics + transfer learning | Wasserstein distance provides principled ways to measure and minimize domain shift |
| Attention as kernel regression | Kernel methods + transformers | Understanding attention through the lens of kernel theory (Performers, Random Feature Attention) enabled linear-time attention approximations |
| Mechanistic interpretability | Neuroscience + deep learning | Techniques from computational neuroscience (ablation studies, causal interventions) applied to understanding neural network internals |
| Constitutional AI | Ethics/philosophy + alignment | Importing structured ethical reasoning into AI training through constitutional principles |
Key Takeaways
- Research taste is the ability to identify important problems and promising approaches — it is the most valued and hardest-to-evaluate skill
- When asked about important problems, pick one you genuinely care about and explain why it matters, what progress looks like, and what makes it hard
- Build a coherent research vision with a north star question, 2–3 research pillars, near-term projects, and success criteria
- Demonstrate intellectual honesty: discuss failures, acknowledge uncertainty, and show that you can distinguish between hard problems and wrong approaches
- Read broadly across fields — the best research often comes from connecting ideas across disciplines