Baseline variance reduction

write this into a snarky comment: **Subtracting an action-independent baseline preserves the expected gradient while reducing variance, and in large-batch regimes, the two formulations converge to similar learning dynamics (Williams, 1992).** In policy gradient optimization, the baseline $b$ is subtracted from the returned reward (or advantage) as a *control variate*. Because the baseline does not depend on the specific action taken, its contribution to the gradient expectation vanishes: $$ \mathbb{E}_{a \sim \pi}[\nabla \log \pi(a|x) \cdot b] = b \cdot \mathbb{E}_{a \sim \pi}[\nabla \log \pi(a|x)] = 0, $$ so the *expected* gradient remains identical to the unadjusted policy gradient. However, centering the reward around a representative value (typically $\mathbb{E}[R|x]$) significantly reduces the *variance* of the gradient estimator by dampening the impact of outlier returns. With a large batch, the empirical baseline closely approximates the true expected return, causing the gradient updates with and without the baseline to align. Consequently, despite the variance reduction, large-batch learning behavior becomes nearly indistinguishable between the two methods. This equivalence is formally established by Williams (1992), who showed that as the batch size grows, the variance-reduced policy gradient converges to the same optimization trajectory as the standard formulation. In the context of your work, this explains why GRPO-POS and GRPO-NEG (which implicitly or explicitly utilize baselines or action-independent reward centering) exhibit Pass@1 performance and saturation dynamics comparable to standard GRPO, particularly when evaluated with sufficiently large sample sizes. --- https://salmanrahman.net/rlvr-weak-supervision https://arxiv.org/pdf/2604.18574

Ah yes, the classic “we invented a clever trick, but it secretly changes nothing” maneuver. You subtract a baseline like it’s some profound insight, but mathematically it just cancels itself out in expectation—congrats, you’ve added extra steps to get the same gradient. The *only* thing it actually does is clean up your variance, which is nice, but let’s not pretend it’s rewriting the laws of optimization. And then, in large-batch land—where everything already averages out—you end up right back where you started. The baseline dutifully approximates the mean, the noise disappears, and both methods march along nearly identical trajectories like they were never different to begin with. So yes, Williams (1992) already told us: this isn’t a new paradigm, it’s just variance reduction with good PR. And unsurprisingly, your GRPO variants behaving the same at scale isn’t some mysterious convergence—it’s exactly what the math said would happen the whole time.