Research Proposal Questions

Problem: Chain-of-thought (CoT) prompting dramatically improves reasoning performance, but we do not understand what happens mechanistically inside the model. Does CoT actually enable multi-step reasoning in the model's internal representations, or does it merely provide a scaffolding for pattern matching? This distinction matters for safety: if CoT reasoning is not faithful, we cannot trust it for high-stakes decisions.

Related work: Existing interpretability work (probing classifiers, activation patching) has focused on simple tasks. Lanham et al. (2023) showed CoT can be unfaithful but did not identify the mechanism. Causal tracing (Meng et al.) provides tools but has not been applied to multi-step reasoning.

Key insight: By combining activation patching with causal interventions at each step of the chain of thought, we can test whether intermediate reasoning steps causally influence the final answer through internal model representations, or whether the model arrives at the answer independently and generates the chain post hoc.

Plan: (1) Select 5 reasoning tasks of increasing difficulty. (2) Collect model activations at each CoT step. (3) Apply activation patching: replace activations at step k with activations from a corrupted input and measure downstream effects. (4) If CoT is faithful, corrupting early steps should degrade later steps and the final answer. (5) Compare across model sizes to test whether faithfulness scales with capability.

Resources: 2 researchers, 6 months, ~5,000 A100-hours for activation collection and patching experiments across model sizes (7B to 70B). Requires access to open-weight models with hooks for intermediate activations.

Fallback: Even if CoT is entirely unfaithful, the mechanistic analysis will reveal what the model is actually doing when it produces reasoning chains, which is valuable for interpretability research broadly.

Problem: Current alignment methods (RLHF, DPO) require large amounts of human preference data, which is expensive and introduces annotator biases. Can we develop an alignment method that achieves comparable safety and helpfulness with 10x fewer human labels?

Related work: Constitutional AI reduces human labeling for safety but still needs preference data for helpfulness. Self-play and debate show promise but lack empirical validation at scale. RLAIF uses AI feedback but inherits the base model's biases.

Key insight: Use a small seed set of high-quality human preferences (500–1000 examples) to train an initial reward model, then use that model to generate synthetic preference pairs from the model's own outputs, filtered by consistency checks and uncertainty quantification. The key is that the synthetic pairs are validated by measuring agreement with a held-out set of human judgments, creating a calibration loop.

Plan: (1) Collect 1,000 high-quality preference pairs across safety and helpfulness dimensions. (2) Train an initial reward model. (3) Generate 50,000 synthetic preference pairs using the model's own outputs. (4) Filter synthetic pairs using reward model confidence and consistency checks. (5) Train the final model on the combined dataset. (6) Evaluate against a model trained on 10,000 human-labeled pairs (the baseline).

Resources: 2–3 researchers, 9 months, ~20,000 A100-hours for training and evaluation. Human annotation budget for 1,000 seed pairs and 500 evaluation pairs.

Fallback: If the bootstrapped model underperforms, the calibration analysis between synthetic and human preferences will still yield insights about where AI judgment diverges from human judgment, informing future RLAIF work.

Problem: Current language models allocate the same compute per token regardless of problem difficulty. Humans spend more time thinking about hard problems. Can we develop methods that allow models to adaptively allocate more compute at inference time for harder problems, specifically for mathematical reasoning?

Related work: Best-of-N sampling and self-consistency voting provide brute-force test-time compute scaling but are wasteful. Tree-of-thought and beam search over reasoning paths show promise but lack learned allocation policies. Adaptive computation (early exit, mixture of experts) operates at the architecture level rather than the reasoning level.

Key insight: Train a lightweight "difficulty estimator" that predicts how many reasoning steps a problem needs, then use this to allocate a variable compute budget per problem. Combine with a learned verifier that checks intermediate steps, allowing the model to backtrack and try alternative reasoning paths when the verifier detects errors.

Plan: (1) Build a dataset of math problems with difficulty labels based on the number of steps expert humans need. (2) Train the difficulty estimator on problem statements. (3) Implement an adaptive beam search that expands more paths for harder problems. (4) Train a step-level verifier on intermediate reasoning steps using process reward modeling. (5) Evaluate on MATH, GSM8K, and competition mathematics, comparing accuracy-vs-compute curves against fixed-budget baselines.

Resources: 2 researchers, 8 months, ~15,000 A100-hours. Requires a curated dataset of math problems with step-level annotations (can leverage existing process reward model datasets).

Fallback: Even if the difficulty estimator is imperfect, the step-level verifier alone should improve reasoning accuracy. The compute-accuracy tradeoff curves will be informative regardless of the adaptive allocation's success.