Yuhan Chi

Choose the Noise, Keep the Flow

2026-10-18T00:00:00+08:00

A recent survey by Nahim, Khan, and Ju, A Survey of Flow Matching in Reinforcement Learning (TMLR, 2026), organizes the field along two axes: what the flow models (actions, critics, dynamics) and where the RL signal enters (energy-weighted regression, policy gradients through the ODE, GRPO-style objectives). It is a useful map. I wanted to look more closely at one family that is not brought together as a group in that survey.

At the center are methods that leave the velocity field untouched. They keep a pretrained flow or diffusion policy frozen and train a small network that decides which noise to feed it. Given a state $s$, a proposer picks a point $w$ in noise space, and the frozen generator turns $w$ into an action. Imagine a robot policy that sometimes reaches around an obstacle on the left and sometimes on the right. Steering the noise changes how often it chooses each route, while reusing the generator that produces the movements.

The survey already covers an important relative. Its offline section discusses ReFORM (Zhang et al., 2026), which manipulates noise inside a bounded support so that the composed policy stays within the learned behavior model’s support. Its discussion of “freezing the flow and learning a correction” covers policies that edit the action after it is generated. DSRL (Wagenmaker et al., 2025) and related methods that choose the input instead of editing the output are absent. This post follows that thread, including a few neighbors that also train the decoder or learn from corrections rather than rewards.

The post makes two points and then checks them on a small example.

A VAE-style inference view connects the methods. The noise proposer plays the role of an encoder and the frozen generator that of a decoder. For a KL-regularized objective, this becomes an exact variational identity, with value taking the place of reconstruction quality. Other methods share the architecture without optimizing that same objective.
The most useful question to ask of each method is where RL takes effect. Action-space data can train a critic directly, but the proposer lives in noise space. Each method is defined largely by how it carries value from one space to the other.
A two-dimensional experiment (Section 4) trains the same noise proposer with five different ways of carrying value, so one can watch the noise move and the actions follow.

This post is also available as a PDF, with numbered equations, stated propositions, and all tables expanded.

Figure 1. The shared encoder–decoder pattern. (a) A VAE trains an encoder and a decoder jointly to reconstruct its input. (b) In noise-space RL, a proposer $q_\phi$ chooses the noise $w$ for a state $s$, and a pretrained, frozen flow $\psi_\theta$ decodes it into an action. Both latents are pulled toward the same prior $\mathcal{N}(0, I)$ by a KL term. The decoder in (b) produces an action rather than a reconstruction, and value $Q/\beta$ takes the place of the reconstruction term. The correspondence is between objectives (Section 2), not between generative models.

1. Setup and notation

A flow-matching policy learns a velocity field $v_\theta(x, t, s)$ that transports Gaussian noise at time $t = 0$ to actions at $t = 1$. To act, one draws $w \sim \mathcal{N}(0, I)$ and integrates $\dot x_t = v_\theta(x_t, t, s)$ from $x_0 = w$, usually with a few Euler steps,

\[x_{k+1} = x_k + \Delta t\, v_\theta(x_k, t_k, s), \qquad t_k = k\,\Delta t, \quad \Delta t = 1/K, \quad k = 0, \dots, K-1,\]

and outputs $a = x_K$. Run with any deterministic sampler (this Euler scheme, the probability-flow ODE of a diffusion policy, DDIM with zero variance, or a one-step distilled model), the policy becomes a fixed map

\[a = \psi_\theta(s, w), \qquad w \sim \mathcal{N}(0, I),\]

and all of its randomness enters through $w$. Table 1 (click to expand) collects the notation used in the rest of the post; the decoder parameters $\theta$ are pretrained and stay frozen, and only the proposer parameters $\phi$ are learned.

Table 1. Notation.

Symbol	Meaning
$s,\ a$	state and action (for robot policies, often a chunk of future actions)
$w \in \mathbb{R}^{d}$	noise fed to the generator; for flow policies, $d$ equals the action dimension
$v_\theta,\ K$	velocity field of the pretrained flow, and the number of sampler steps
$\psi_\theta(s, w)$	the decoder: the deterministic sampler that maps noise $w$ to action $a$
$f_{\#}\mu$	pushforward: the distribution of $f(x)$ when $x \sim \mu$
$\pi_{\text{base}}(\cdot \mid s)$	the pretrained policy, $\psi_\theta(s, \cdot)_{\#}\, \mathcal{N}(0, I)$
$q_\phi(w \mid s)$	the noise proposer (encoder), usually a state-conditioned Gaussian
$\pi_\phi(\cdot \mid s)$	the steered policy, $\psi_\theta(s, \cdot)_{\#}\, q_\phi(\cdot \mid s)$
$Q(s, a)$	an action-value estimate (critic)
$\beta > 0$	temperature: weight of the KL penalty that keeps $q_\phi$ near $\mathcal{N}(0, I)$
$q^\star,\ \pi^\star$	the optimal tilted noise distribution and policy, defined in Section 2

A noise-space method keeps $\theta$ fixed and replaces only the distribution of the input, from $\mathcal{N}(0, I)$ to $q_\phi(\cdot \mid s)$. The proposer can be a Gaussian with a state-dependent mean and variance, a deterministic map, a residual added to fresh noise, or even a single constant vector. It is usually smaller than the generator, though not necessarily tiny: Noise PPO uses a roughly 187M-parameter proposer. The decoder can be very large (DSRL steers $\pi_0$, a 3.3B-parameter generalist policy) and is never updated.

DSRL states the construction cleanly. Define a new MDP whose actions are noises:

\[P^{W}(\cdot \mid s, w) = P(\cdot \mid s, \psi_\theta(s, w)), \qquad r^{W}(s, w) = r(s, \psi_\theta(s, w)).\]

The diffusion policy becomes part of the environment. Any RL algorithm can now run over $w$, using only forward passes of the generator, without its weights or its intermediate denoising steps.

The idea is older than diffusion policies. PLAS (Zhou et al., 2020) learned a policy in the latent space of a conditional VAE for offline RL. PARROT (Singh et al., 2021) trained a normalizing-flow behavioral prior and ran RL on its noise. It compared directly against a conditional-VAE latent space and argued that invertibility lets the agent keep full control over the final action. DSRL singles out PARROT as the precedent in control and makes the same point from the other side: diffusion models are not invertible by construction, so it is “much less clear” that steering their noise is expressive enough. In practice, it turned out to be.

2. Reading it as a VAE

Recall the evidence lower bound of an ordinary VAE (Kingma & Welling, 2014); a conditional VAE additionally conditions its distributions on context:

\[\log p(x) \;\ge\; \mathbb{E}_{q_\phi(z \mid x)}\big[\log p_\theta(x \mid z)\big] \;-\; \mathrm{KL}\big(q_\phi(z \mid x) \,\Vert\, p(z)\big).\]

To make the connection precise, consider the following KL-regularized policy-improvement objective:

\[J(\phi) = \mathbb{E}_{s}\Big[\, \mathbb{E}_{w \sim q_\phi(\cdot \mid s)}\, Q\big(s, \psi_\theta(s, w)\big) \;-\; \beta\, \mathrm{KL}\big(q_\phi(\cdot \mid s) \,\Vert\, \mathcal{N}(0, I)\big) \Big].\]

Here $Q$ is held fixed during the proposer update, and states are sampled from a fixed training distribution. This is a local improvement objective, not by itself a complete solution to sequential RL. Table 2 lines up the two expressions term by term.

Table 2. VAE terms and their noise-space counterparts.

VAE	Noise-space RL
observed input $x$	desired optimality, with state $s$ as context
encoder $q_\phi(z \mid x)$	noise proposer $q_\phi(w \mid s)$
prior $p(z) = \mathcal{N}(0, I)$	the Gaussian the flow was trained from
decoder $p_\theta(x \mid z)$	frozen flow $\psi_\theta(s, w)$
reconstruction $\log p_\theta(x \mid z)$	value $Q(s, \psi_\theta(s, w)) / \beta$
KL to the prior	KL to $\mathcal{N}(0, I)$: penalize departure from the training prior
decoder trained jointly	decoder pretrained by imitation, usually frozen

The exact connection is a variational identity. Define $Z(s) = \mathbb{E}_{w \sim \mathcal{N}(0,I)}[\exp(Q(s, \psi_\theta(s,w))/\beta)]$ and assume it is finite. Then, for each state,

\[\log Z(s) = \underbrace{\mathbb{E}_{q_\phi}[Q(s, \psi_\theta(s,w))/\beta] - \mathrm{KL}(q_\phi \Vert \mathcal{N}(0,I))}_{J(\phi;\, s)/\beta} + \mathrm{KL}(q_\phi \Vert q^\star).\]

The last term is nonnegative, so the first two terms form a lower bound. In the control-as-inference view (Levine, 2018), this can be read as an ELBO for a desired “optimality” observation. When $Q$ is bounded above, a state-dependent scaling makes the exponential a valid binary likelihood; the variational identity itself needs only a finite $Z(s)$. The unrestricted best proposer is

\[q^\star(w \mid s) = \frac{\mathcal{N}(w; 0, I)\, \exp\big(Q(s, \psi_\theta(s, w))/\beta\big)}{Z(s)}.\]

Writing $J(\phi; s)$ for the term inside the outer expectation of $J$, the identity also gives a ruler: for any proposer, $\log Z(s) - J(\phi; s)/\beta = \mathrm{KL}(q_\phi \Vert q^\star)$ measures how far it is from the best one. Section 4 uses it.

The difference from an ordinary VAE is what counts as the observation. The encoder is not asked to explain an action that was seen. It is asked to infer which noise would have produced a good action. The evidence is desired success, represented here by the value estimate. This is why “VAE-style” is more accurate than calling every method a conditional VAE. A deterministic proposer, for example, has infinite KL to a continuous Gaussian; box constraints, spherical latents, and supervised inversion require their own objectives.

Two facts make this reading useful rather than decorative. Eyring et al. (2025) state both for distilled text-to-image generators, and both carry over to policies without change.

Tilting the noise tilts the policy. Pushing the noise posterior through the frozen decoder gives exactly the reward-tilted base policy:

\[\psi_\theta(s, \cdot)_{\#}\, q^\star(\cdot \mid s) \;=\; \pi^\star(\cdot \mid s) \;\propto\; \pi_{\text{base}}(\cdot \mid s)\, \exp\big(Q(s, \cdot)/\beta\big).\]

The reason is that the tilt depends on $w$ only through $a = \psi_\theta(s, w)$.

Why the tilt commutes with decoding

For a bounded test function $f$, suppressing the fixed state $s$,

\[\begin{aligned} \mathbb{E}_{q^\star}[f(\psi_\theta(w))] &= Z^{-1}\mathbb{E}_{w\sim\mathcal{N}(0,I)} [f(\psi_\theta(w))e^{Q(\psi_\theta(w))/\beta}] \\ &= Z^{-1}\mathbb{E}_{a\sim\pi_{\text{base}}} [f(a)e^{Q(a)/\beta}]. \end{aligned}\]

No invertibility is needed. This is the same target that KL-regularized fine-tuning of the whole flow aims for, but a small proposer can only approximate it as well as its distribution family and optimization allow. A single Gaussian, for example, need not represent a multimodal noise posterior. Figure 2 shows the target on a 2D example.

Figure 2. Tilting the noise tilts the policy (exact, no learning). The base policy is a four-cluster mixture in a 2D action space, and $\psi$ is the exact flow-matching ODE map onto it. Each column is one value of $\beta$. Top: the tilted noise distribution $q^\star(w) \propto \mathcal{N}(w; 0, I)\, e^{Q(\psi(w))/\beta}$ (shading) and samples from it (dots); grey lines split the noise plane into the four regions that decode to the four clusters. Bottom: the same samples after the frozen flow, over the tilted policy $\pi_{\text{base}}\, e^{Q/\beta}$ (shading); dashed circles are contours of $Q$, and the star marks its peak. At $\beta = \infty$ we recover the base policy. As $\beta$ shrinks, probability moves to the cluster nearest the peak, but no new cluster appears.

The noise-space KL is a conservative proxy. By the data processing inequality,

\[\mathrm{KL}\big(\pi_\phi(\cdot \mid s) \,\Vert\, \pi_{\text{base}}(\cdot \mid s)\big) \;\le\; \mathrm{KL}\big(q_\phi(\cdot \mid s) \,\Vert\, \mathcal{N}(0, I)\big),\]

with equality when $\psi_\theta(s, \cdot)$ is invertible. The left side can be expensive to evaluate: a continuous normalizing flow can compute densities through divergence integration, while a generic distilled decoder need not offer a tractable density. For a Gaussian proposer $\mathcal{N}(\mu, \mathrm{diag}\,\sigma^2)$, the right side is the familiar VAE term $\tfrac12 \sum_j (\mu_j^2 + \sigma_j^2 - 1 - \log \sigma_j^2)$. Noise PPO (Miao et al., 2025) uses exactly this KL penalty, together with an entropy bonus. Noise Hypernetworks and LP-DS (Simsir & Oguz, 2026) write the steered noise as a residual $w = \epsilon + \Delta$ and approximate the KL by $\tfrac12\, \mathbb{E}\Vert\Delta\Vert^2$, a surrogate justified under regularity and small-Jacobian assumptions. Bounding the residual’s magnitude alone does not establish those assumptions.

3. Where RL takes effect: carrying value into noise space

The decision variable for RL is now the noise. Much of the available supervision, however, lives in the other space. An action-space critic $Q(s, a)$ is ordinary temporal-difference learning on $(s, a, r, s')$ transitions, and offline datasets, demonstrations, and human corrections all come labeled with actions, not noises. The proposer needs a value of $w$, or a gradient with respect to $w$, of the composition

\[w \;\longmapsto\; Q\big(s, \psi_\theta(s, w)\big).\]

This simplifies the policy interface but not necessarily the optimization: nearby noises may have very different values. So the central design question of the family is how to carry an action-space value back into noise space. The papers suggest five routes, summarized in Figure 3. They are not disjoint categories: Route 4 is a special case of Route 3, and a method can combine inversion with a noise critic. What separates them is what each needs from the decoder: forward calls only, an inverse, or derivatives.

Figure 3. Five routes for training the noise proposer. In each row, noise space is on the left, the frozen decoder in the middle, and action space on the right. Grey dashed arrows are forward decoding; orange arrows show what crosses back into noise space. Route 1 is drawn in its two-critic form; Route 2 uses action supervision rather than a critic; Route 5 uses returns directly. Representative methods are listed at the right of each row.

Route 1: learn a critic in noise space. One option is to learn a noise-space critic directly from rollouts, as in DSRL-SAC. DSRL-NA (“noise-aliased”) instead keeps two critics. It learns $Q^{A}(s, a)$ by temporal-difference learning in action space, then distills it into a noise-space critic by regressing $Q^{W}(s, w)$ onto $Q^{A}(s, \psi_\theta(s, w))$ with $w \sim \mathcal{N}(0, I)$. The decoder is only ever run forward. Because many noises decode to similar actions, a noise that was never played can still receive an accurate value. On Robomimic Square, plain SAC in noise space (DSRL-SAC) needs about twice as many samples. On real robots, DSRL reports cases where success rose from 20% to 90% within fewer than 50 online episodes. LP-DS keeps the same two-critic design. A related forward-only idea appears in GoldenStart (Zhang et al., 2026), which picks the highest-value noise for each state by decoding candidates and scoring them with the critic, then fits a conditional VAE to these noises as a learned prior. GoldenStart is not a pure frozen-decoder method, though: its flow teacher keeps training, and the deployed policy is a distilled one-step student. Cost: a second approximation. LPS argues that a distilled critic can match values well and still point its gradient in the wrong direction.

Route 2: invert the flow and move the data. Sometimes the signal is an action rather than a reward: a human correction, or a coarse suggestion from a vision-language model. A flow can be integrated backwards to recover the noise that would have produced that action, $w^\star \approx \psi_\theta^{-1}(s, a^\star)$, and the corrected actions become noise-space training targets. A generic VAE decoder has no inverse, so it relies on a learned encoder for inference. An exact ODE flow is invertible under suitable regularity assumptions; a coarse numerical sampler or a distilled decoder need not be. FlowDAgger (Murray et al., 2026) regresses the proposer onto inverted corrections. UniSteer (Lu et al., 2026) stores inverted transitions $(s, \hat w, r, s')$ in both a demonstration buffer and an RL buffer, and trains a noise-space critic on them by ordinary TD learning. Flow Reversal Steering (Tang et al., 2026) maps coarse reference actions to nearby modes of a generalist policy, then distills them or uses them to bootstrap DSRL. Inversion supplies a noise that reconstructs the correction, not a sample from the reward-tilted posterior above. Cost: inversion error. With the short ODE schedules of robot action heads, a naive reverse Euler step is not accurate enough, which is why FlowDAgger and UniSteer both use per-step fixed-point refinement for their flow action heads. Convergence requires conditions such as a step size small relative to the velocity field’s Lipschitz constant; it is not automatic.

Route 3: backpropagate through the sampling iterations. The chain rule gives the noise gradient exactly:

\[\nabla_w\, Q\big(s, \psi_\theta(s, w)\big) = \Big(\frac{\partial \psi_\theta}{\partial w}\Big)^{\!\top} \nabla_a Q(s, a),\] \[\frac{\partial \psi_\theta}{\partial w} = \prod_{k=K-1}^{0} \Big(I + \Delta t\, \frac{\partial v_\theta}{\partial x}(x_k, t_k, s)\Big)\]

for the $K$-step Euler sampler of Section 1, with later-step Jacobians multiplying on the left. “Exact” here means the derivative of the implemented sampler and learned critic, not of the true environment return. With a reparameterized proposer $w = \mu_\phi(s) + \sigma_\phi(s) \odot \epsilon$, this is the VAE reparameterization trick with a frozen decoder, and no second critic is needed. D-Flow (Ben-Hamu et al., 2024) takes this route for per-sample noise optimization in image, audio, and molecule generation. Cost: the gradient has to pass through every step of the solver. DSRL calls this “computationally intensive and numerically unstable”, and LPS calls it “expensive and unstable”.

Route 4: make the decoder one step. If $\psi_\theta$ is a single network evaluation, the product above has a single factor, and the critic gradient reaches the noise in one backward pass. LPS (Im et al., 2026) uses a one-step MeanFlow base policy for exactly this reason, and backpropagates $\nabla_a Q$ straight into the latent actor without a noise-space critic. It is a useful architectural neighbor rather than a strictly frozen-decoder method: its Algorithm 1 also updates the base policy by imitation during training. ReNO does the same with reward models on one-step image generators, and Noise Hypernetworks on distilled one- or few-step generators (SD-Turbo, SANA-Sprint, FLUX-Schnell). With a few steps the chain in Route 3 is short rather than trivial, which is still cheap. Cost: the base policy must be one-step, which usually means it was trained or distilled for that purpose.

Route 5: skip the transfer. The likelihood-ratio gradient of the return term, $\mathbb{E}\big[\nabla_\phi \log q_\phi(w \mid s)\, A(s, w)\big]$ with an advantage estimate $A$, needs only $\log q_\phi$, which is exact for a Gaussian proposer: no ODE divergence, no SDE conversion, no surrogate likelihood. Much of the survey’s online-RL section is about recovering a usable likelihood for flow policies; in noise space the problem does not arise. Noise PPO treats text-to-image generation as a one-step MDP whose action is the initial noise, and runs PPO with sampled rewards; KL and entropy regularizers contribute additional gradient terms. Cost: the usual variance of score-function estimates.

4. A two-dimensional experiment

To see the five routes at work, I trained the same noise proposer with each of them on a toy problem small enough to plot and to evaluate exactly. The code for this section and all figures is on GitHub.

Setup. Actions are points $a \in \mathbb{R}^2$. The base policy imitates the four-cluster mixture of Figure 2: a small velocity network (an MLP) is trained once by conditional flow matching and then frozen, and the decoder $\psi_\theta$ is its $K = 10$-step Euler sampler. The pushforward of $\mathcal{N}(0, I)$ through it puts between 24.9% and 25.1% of its mass on each cluster. A state is a goal angle $s \in [0, 2\pi)$, and the critic is known exactly: $Q(s, a) = -\tfrac12 \Vert a - g(s) \Vert^2$, where the goal $g(s)$ lies on the circle through the cluster centers. With $\beta = 1$, a goal between two clusters makes $q^\star$ bimodal. The proposer is a Gaussian $\mathcal{N}(\mu_\phi(s), \mathrm{diag}\,\sigma_\phi(s)^2)$ whose mean and log standard deviation come from a small MLP, initialized at the prior $\mathcal{N}(0, I)$.

Because $Q$ is given, the experiment isolates the transfer step: there is no TD learning and no exploration. The ruler from Section 2 measures the result: $\mathrm{KL}(q_\phi \Vert q^\star) = \log Z(s) - J(\phi; s)/\beta$, estimated by Monte Carlo and averaged over 64 evenly spaced states.

How the noise moves. Figure 4 follows one state during training with Route 3. The proposer starts as the prior, so the actions are the base policy. Within a few dozen updates, the noise cloud shrinks and slides into the region of noise space that decodes to the cluster nearest the goal; the actions follow it without any change to the decoder.

In this state the goal lies between two clusters, so the target policy is bimodal. One Gaussian is nonetheless enough to reach both: the two clusters decode from adjacent regions of noise space, and the proposer settles across their shared boundary. A distribution that is bimodal in action space can be unimodal in noise space. What the Gaussian cannot match is the shape of $q^\star$, which is cut off sharply along the grey lines; a few of its samples also still decode to the two far clusters. The remaining $\mathrm{KL}(q_\phi \Vert q^\star)$ at this state is about 0.17 nats.

Figure 4. Learning which noise to choose, for one state (goal angle 67.5°) with Route 3. Columns are snapshots during training. Top: samples from the proposer $q_\phi(w \mid s)$ (dots) and its two-standard-deviation ellipse (orange), over the exact tilted posterior $q^\star$ for the learned decoder (shading); grey lines separate the noise regions that decode to different clusters. Bottom: the same samples after the frozen 10-step decoder, over the tilted target policy $\pi_{\text{base}}\, e^{Q/\beta}$ (shading, computed from the mixture the flow imitates); the star is the goal and dashed circles are contours of $Q$. The decoder never changes; only the noise does.

Comparing the routes. Each route trains the same proposer for 2,000 updates, each on 1,024 decoded noises (256 states with 4 noises each), and is run with three seeds. Routes 1, 3, 4, and 5 maximize the objective $J$ of Section 2, using a distilled noise critic, gradients through the 10 Euler steps, gradients through a one-step decoder distilled from the 10-step sampler, and a likelihood-ratio gradient with a leave-one-out baseline. Route 2 never sees $Q$. It receives “corrected” actions sampled from the tilted target policy, a stand-in for an expert, inverts them through the decoder by fixed-point iteration, and fits the proposer to the recovered noises by maximum likelihood. Table 3 and Figure 5 summarize the outcome.

Table 3. Five routes on the 2D toy. $\mathrm{KL}(q_\phi \Vert q^\star)$ is averaged over 64 states (mean ± standard deviation over three seeds; lower is better). Time is the median wall-clock time per update on a laptop GPU (RTX 5060), including Python overhead. Route 4 is measured against the posterior of its own one-step decoder.

Route	Needs from the decoder	$\mathrm{KL}(q_\phi \Vert q^\star)$	ms / update
1. noise critic	forward calls	0.207 ± 0.004	8.6
2. inversion	an inverse (20 fixed-point iterations per step)	0.312 ± 0.007	60.2
3. backprop through 10 steps	derivatives	0.211 ± 0.002	7.2
4. one-step decoder	derivatives of a distilled one-step student	0.206 ± 0.002	2.9
5. likelihood ratio	forward calls	0.211 ± 0.003	5.6
reference: best Gaussian for each state separately	derivatives	0.198	—

Figure 5. Five routes, one proposer. (a) The final proposer of each route for the same state as Figure 4, drawn as in Figure 4 (seed 0). (b) $\mathrm{KL}(q_\phi \Vert q^\star)$ averaged over 64 states during training; lines are means and bands span three seeds. The dashed line is the best single Gaussian fitted to each state separately.

Three things stand out.

The four routes that maximize $J$ end in the same place. Routes 1, 3, 4, and 5 reach 0.206–0.211 nats, against 0.198 for the best single Gaussian fitted to each state separately. In this toy the remaining error comes from the Gaussian family, not from the transfer route, and amortizing over states costs little.
They differ in how they get there. Route 1 starts slowly, because its noise critic has to be learned before its gradient means anything (Figure 5b). The cost per update spans a factor of twenty: 2.9 ms through the one-step decoder, 7.2 ms through ten Euler steps, and 60 ms for inversion.
Route 2 settles higher, and not because inversion fails. Fixed-point inversion reconstructs the corrected actions to a median error of about $10^{-7}$; a single naive reverse Euler pass per step leaves a median error of 0.05, about an eighth of a cluster’s standard deviation (0.38). The difference lies in the objective. Maximum likelihood on samples from the target minimizes $\mathrm{KL}(q^\star \Vert q_\phi)$, which favors a wider Gaussian that covers the posterior (Figure 5a), while the ruler measures $\mathrm{KL}(q_\phi \Vert q^\star)$. Corrections teach the proposer to imitate the target, not to optimize the objective.

These numbers come from a deliberately easy setting: two dimensions, an exact critic, and a smooth decoder. They say nothing about critic error, about the variance of Route 5 in high dimensions, or about the instability that DSRL and LPS report for Route 3 with large decoders. What the toy does show is the shape of the problem: the decoder fixes which actions are reachable, the proposer only decides how much noise to send to each region, and a single Gaussian must choose between regions when the target has several.

5. A map of the family

Table 4 groups the methods by their main learning route. It includes related designs with changing decoders, labeled explicitly. “Prior control” distinguishes actual constraints from initialization or supervision alone.

Table 4. The family at a glance, grouped by route.

Route	Method	Setting	Proposer $q_\phi(w \mid s)$	Prior control
1. noise critic	PARROT (2021; preprint 2020)	robot manipulation	RL policy over the noise of an invertible normalizing flow	prior initialization; no lasting proximity guarantee
1. noise critic	DSRL (2025)	online and offline control; $\pi_0$	state-conditioned noise actor; SAC, or an action critic distilled into a noise critic	noise box $[-b, b]^d$
1. noise critic	LP-DS (2026)	online control; $\pi_0$	residual $\epsilon + \Delta_\phi(s, \epsilon)$ with noise-aliased critics	Lagrangian trust region on $\mathbb{E}\Vert\Delta\Vert^2$
1. forward scoring	GoldenStart (2026)	distilled one-step flow policies; teacher not frozen	conditional VAE fit to the highest-value noise per state	CVAE fitting; entropy control is on the action student
2. inversion	FlowDAgger, UniSteer (2026)	human-in-the-loop VLA adaptation	noise policy regressed onto inverted corrections; UniSteer adds a noise critic	inverted targets; inversion alone does not ensure typical noise
2. inversion	Flow Reversal Steering (2026)	generalist policies	auxiliary noise policy trained on inverted coarse actions	coarse references guide reversal; no generic KL bound
3 / 4. backpropagation	ReFORM (2026)	offline RL; BC model and distilled decoder also trained	reflected flow generating bounded input noise	projected steps keep noise inside the source ball
4. one-step	Noise Hypernetworks (2025)	distilled text-to-image	residual on fresh noise (LoRA), reward gradient through the generator	$\tfrac12\Vert\Delta\Vert^2$ as a KL surrogate
4. one-step	LPS (2026)	offline RL; real robots; base also trained by imitation	latent actor on a sphere, $\nabla_a Q$ through a one-step MeanFlow	sphere of radius $\sqrt{d}$
5. likelihood ratio	Noise PPO (2025)	text-to-image	prompt-conditioned Gaussian trained by PPO	KL to $\mathcal{N}(0, I)$, entropy bonus
5. likelihood ratio	GoRL (2025)	online control	Gaussian encoder; decoder refit in alternating phases	KL to $\mathcal{N}(0, I)$
search	Golden Ticket (2026)	generative robot policies	one constant noise vector found by search	—

ReFORM spans Routes 3 and 4: its original objective differentiates through the BC flow, and its practical design introduces one-step BC distillation to shorten this path. Its Algorithm 1 also continues fitting the BC model. D-Flow is a per-sample example of Route 3.

Three neighbors are worth knowing. PDP (Zhang et al., 2026) argues that Gaussian noise is a poorly conditioned control interface. Instead of steering it, PDP trains a literal VAE over whole demonstrations, with a geometry-alignment term, and modulates the denoiser with the resulting low-dimensional code. Condition-dependent source flow matching (Kim et al., 2026) and LAFM (Machado et al., 2026) learn the prior itself together with the flow at training time, which in VAE terms is a learned prior. D-Flow (Ben-Hamu et al., 2024) and ReNO (Eyring et al., 2024) optimize the noise separately for each sample instead of training a proposer.

6. What the VAE view explains

The decoder fixes the map, not a safety guarantee. With a frozen decoder, every steered action lies in the image of $\psi_\theta(s, \cdot)$. For finite $Q$ and positive $\beta$, the ideal tilted policy has the same support as the base policy. But support is not the same as likely or reliable behavior: a Gaussian-driven invertible flow can have full support, including actions far from the demonstrations. ReFORM makes a stronger construction by bounding the source support; relating its learned model’s support to the dataset still depends on how well that model fits. A truly unreachable action cannot be recovered by steering, while a very unlikely action might be. GoRL (Zhang et al., 2025) changes the map itself by training the decoder again. Its decoder starts as an identity-like map. Training then alternates between two phases: optimizing a Gaussian encoder with a KL penalty toward $\mathcal{N}(0, I)$ while the decoder is frozen, and refitting the decoder by flow matching on recent rollouts while the encoder is frozen.

$\beta$ is a dial between the base policy and high-value actions. For a fixed, well-behaved $Q$, as $\beta \to \infty$ the ideal proposer approaches the prior and the system returns to the base policy. This resembles posterior collapse in a VAE, though here returning to the base can be intentional. As $\beta \to 0$, mass concentrates near maximizers, when they exist. Multiple equally good actions—or multiple noises for one action—can remain; a single point mass is not inevitable. LP-DS reports that prior-replacing steering can lose modes, and uses a residual trust region whose multiplier plays a role similar to $\beta$, for its surrogate penalty. Golden Ticket (Patil et al., 2026) imposes a different restriction: one noise vector shared across states. It is not the general low-temperature limit of a state-conditioned proposer.

Holes in the latent space. A VAE can decode poorly in regions its training rarely visits. A Gaussian-driven flow can have the same problem: most training draws come from typical regions of the Gaussian. The family uses several ways to discourage atypical queries, with different strengths. DSRL bounds each noise coordinate in $[-b,b]$ and reports that $b$ between 1 and 3 works well online. A box bounds coordinates; it does not by itself enforce the Gaussian’s typical radius. LPS trains both its base policy and its latent actor on the sphere of radius $\sqrt{d}$, where high-dimensional Gaussian mass concentrates, to prevent what it calls “norm explosion”; strictly, its prior is uniform on that sphere rather than $\mathcal{N}(0, I)$. LP-DS bounds the squared norm of its residual, its stand-in for the KL.

Aliasing makes some noise changes redundant. Many noises can decode to similar actions. This is related to latent ambiguity, but distinct from the statistical non-identifiability of a VAE’s latent coordinates. DSRL-NA turns this weakness into data sharing across noises. The same fact explains why the noise KL is only an upper bound: when the decoder is genuinely many-to-one, part of the KL budget can be spent on redistributing noise among identical outputs. For an invertible flow the KLs remain equal, even if different noises produce very similar actions.

Amortization has a gap. Optimizing noise separately for each sample (D-Flow, ReNO) and training a proposer that predicts it (Noise Hypernetworks, whose subtitle is “Amortizing Test-Time Compute”) are the policy-side versions of per-datapoint variational inference and an amortized encoder. The cost of amortization has a name in the VAE literature, the amortization gap (Cremer et al., 2018). Noise Hypernetworks report recovering a substantial portion of the gains of explicit test-time optimization at a fraction of its cost, which illustrates the quality–compute tradeoff. That comparison is not, by itself, a measurement of the formal variational amortization gap.

7. Open questions

When is a policy steerable? DSRL notes that the base policy must be “steerable” for the method to work and studies when this holds empirically, but does not give a formal definition. Nothing in the construction guarantees useful steering in advance. It helps to separate three questions: can the decoder produce a good action, does the allowed proposer put enough probability on its preimages, and can the learning signal find them? Invertibility addresses only the first.
How does the number of sampling steps matter? Noise PPO finds the largest gains at few inference steps; the benefit shrinks, but does not disappear, as the number of steps grows. Several robot papers rely on one-step or few-step decoders for different reasons: cheap Jacobians for LPS, low latency for robot control. Few steps are not free, however: FlowDAgger finds that coarse steps make naive inversion inaccurate.
When should the decoder be trained? Freezing the decoder preserves the learned map and can improve sample efficiency; training it, as in GoRL, can change which useful behaviors are easy to represent. Where a pretrained policy should sit between the two is not settled.

A practical reading rule. If the decoder is available only through forward calls, start by considering a noise critic or a likelihood-ratio update. If corrections are available, inversion may turn them into direct supervision. If gradients are available and the sampler is short, direct backpropagation avoids critic distillation. Then check the paper’s actual prior constraint and whether the decoder stays frozen; the shared diagram does not settle either question.

The survey’s second axis asks where the RL signal enters a flow-matching method. This family adds one more answer to that question: the RL signal can enter through the input-noise distribution.

References

Nahim, N. Z., Khan, F. N., & Ju, P. (2026). A Survey of Flow Matching in Reinforcement Learning. TMLR.
Wagenmaker, A., Nakamoto, M., Zhang, Y., et al. (2025). Steering Your Diffusion Policy with Latent Space Reinforcement Learning. (DSRL)
Singh, A., Liu, H., Zhou, G., Yu, A., Rhinehart, N., & Levine, S. (2021). PARROT: Data-Driven Behavioral Priors for Reinforcement Learning. ICLR.
Zhou, W., Bajracharya, S., & Held, D. (2020). PLAS: Latent Action Space for Offline Reinforcement Learning. CoRL.
Miao, Y., Loh, W., Poupart, P., & Kothawade, S. (2025). A Minimalist Method for Fine-tuning Text-to-Image Diffusion Models. (Noise PPO)
Eyring, L., Karthik, S., Dosovitskiy, A., Ruiz, N., & Akata, Z. (2025). Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Models.
Eyring, L., Karthik, S., Roth, K., Dosovitskiy, A., & Akata, Z. (2024). ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimization. NeurIPS.
Ben-Hamu, H., Puny, O., et al. (2024). D-Flow: Differentiating through Flows for Controlled Generation. ICML.
Zhang, C., Wan, Z., Chen, F., Yu, X., Tsang, I., & An, B. (2025). GoRL: An Algorithm-Agnostic Framework for Online Reinforcement Learning with Generative Policies. The current arXiv version is titled Training Diffusion Policies via Prior-Mapping Co-Evolution; this post follows v1.
Zhang, H., Sun, Y., & Xiong, H. (2026). GoldenStart: Q-Guided Priors and Entropy Control for Distilling Flow Policies. ICLR.
Simsir, H., & Oguz, O. S. (2026). Lagrangian Perturbation Diffusion Steering: Latent Reinforcement Learning for Generative Policies. ICML. (LP-DS)
Im, H., Kolobov, A., Fu, J., & Lee, Y. (2026). Latent Policy Steering through One-Step Flow Policies. RSS. (LPS)
Zhang, S., So, O., Ahmad, H. M. S., Yu, E. Y., Cleaveland, M., Black, M., & Fan, C. (2026). ReFORM: Reflected Flows for On-Support Offline RL via Noise Manipulation. ICLR.
Murray, M., Chen, D., Bagaria, S., et al. (2026). FlowDAgger: Human-in-the-Loop Adaptation of Generative Robot Policies in Latent Space.
Lu, J., Qin, X., et al. (2026). UniSteer: Unified Noise Steering for Efficient Human-Guided VLA Adaptation.
Tang, A., Chen, W., Wagenmaker, A., Finn, C., & Levine, S. (2026). Improving Robotic Generalist Policies via Flow Reversal Steering.
Patil, O., Biza, O., Weng, T., et al. (2026). You’ve Got a Golden Ticket: Improving Generative Robot Policies With A Single Noise Vector.
Zhang, R., Fu, H., Jia, M., Konidaris, G., Du, Y., & da Silva, B. C. (2026). From Noise to Control: Parameterized Diffusion Policies. ICML. (PDP)
Kim, J., Park, J., Jeon, S., & Kim, S. (2026). Better Source, Better Flow: Learning Condition-Dependent Source Distribution for Flow Matching.
Machado, B., Chapin, A., Dellandréa, E., & Chen, L. (2026). Flowing With Purpose: Latent Action Guided Flow Matching Policies for Robotic Manipulation. (LAFM)
Kingma, D. P., & Welling, M. (2014). Auto-Encoding Variational Bayes. ICLR.
Levine, S. (2018). Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review.
Cremer, C., Li, X., & Duvenaud, D. (2018). Inference Suboptimality in Variational Autoencoders. ICML.

Cite this post

If you find this post useful in your work, you can cite it as:

Chi, Yuhan. “Choose the Noise, Keep the Flow.” October 18, 2026. https://yuhanchi.github.io/writing/choose-the-noise/

Or use the following BibTeX entry:

@misc{chi2026choosethenoise,
  author = {Chi, Yuhan},
  title = {{Choose the Noise, Keep the Flow}},
  year = {2026},
  month = oct,
  day = {18},
  howpublished = {Blog post},
  url = {https://yuhanchi.github.io/writing/choose-the-noise/}
}

This citation refers to the blog’s explanation and synthesis. Please cite the original papers above for their methods and results.

What Is Luck?

2026-10-15T00:00:00+08:00

What is luck? It is really a good question. With the wind, even pigs can fly.

I always admire some friends’ luck, or destiny. Many golden opportunities fall into their laps! They meet visionary people by chance, or receive unexpected invitations to study or intern.

I think I at least work hard, but lack some luck. However, today’s lecture changed my view. A professor gave a lecture. Although it was a little difficult and I got lost in the second half, a senior of mine mentioned a paper, “DiffusionNFT,” and was curious about its relationship to the method in the lecture. “Fortunately,” I had read that paper! She had other things to attend to and had to leave early, so she couldn’t continue the discussion. I got an “opportunity” to follow up on her question and explain the paper’s algorithm in more detail on her behalf, carrying the discussion forward (though I could only explain the algorithm and couldn’t fully follow the professor’s analysis of it). The professor was interested in that topic.

Did I gain any rewards from that process? Nothing, exxcept acknowledgement—acknowledgement of the knowledge and understanding I had accumulated over time. If I hadn’t read enough papers, I couldn’t have known what they were referring to or explained the algorithm intuitively to the professor.Similarly, my friend is a bold, almost reckless person and never hesitates to connect with anyone. His character is like that, so he does it naturally.

This motivates me to keep working on deep thinking and accumulation. This time, the chance encounter was a positive one, but I believe the same applies to setbacks: the knowledge and understanding I accumulate can help me turn those into opportunities too.

When enough effort has been devoted, odds become opportunities. Luck, in turn, is part of your calibre.

Knowing Is Not Enough

2026-10-11T00:00:00+08:00

Today, a senior asked me, “You say you read many papers, but how do you know whether a paper has taken effect after you read it?” I was tongue-tied.

I have run into this kind of problem a lot lately. In fact, I have felt stuck in a kind of stagnation since this summer. I seemed to have done and experienced many things, but in the end I had only “known” them. This discouraged me a lot: my meta-abilities and my understanding had barely improved, despite my apparent diligence.

That is a serious problem. After all, I devoted my time, my attention and my energy. Ideally, whether an experiment or a study succeeds or fails, I should come away with some understanding from it.

There is a common temptation. New techniques, trends and other fancy things are attractive, and there is some bar even to “knowing” them. So we spend a certain amount of time and energy getting to know them, and then we are easily satisfied. That is terrible.

Reading papers has a similar trap: we are awed by novel ideas and forget to think critically. The figures and graphs may be impressive, but a series of questions should still be asked:

What is the baseline or benchmark? The authors claim strong results, but strong compared with what? If I have read the related papers, I should know the common baselines. Is a common baseline missing? Does the benchmark truly match the problem?
Why this method? If they use PPO, was that forced by a constraint (e.g. a continuous action space rules out plain Q-learning), or were there real alternatives? Did they justify the choice? If they say an abandoned method gave unstable training losses, did they actually find and explain the reason?
Where does the gain come from? How do the authors explain their edge over other methods? The ablation experiments may prove something. But does the reasoning part, such as arguments about the manifold or about multiple modes, really convince me?
What obstacles did I run into while reading? If I ran into some, I deal with them. But what if I have no questions at all? There are only two possibilities: either I truly understand it, or I am trusting my instinct, which says everything is fine. Then I always need to ask one more question: is my instinct trustworthy? Many pitfalls hide beneath instinct.
What am I reading this paper for? Is it to push back the frontier of my knowledge, or just to learn a technique more deeply?

This approach is not only for reading papers; I believe I can apply it to many other things. As a freshman, I might have been proud of “having dabbled in many areas.” Now that I am a sophomore, things should change. Harder, deeper and more critical thinking should become part of my daily study.

Besides, I need to output. Output forces me to understand and then regenerate. When I explain something again, I automatically rethink the problem. Chats, seminars, talks: I will try them all.

So here is the answer I did not have that day: a paper has taken effect if I can rebuild its argument without looking, say where it would fail, and use it in my own work. If all I can say is “I’ve read it,” then I only know it.

Proofs as Observations

2026-09-30T00:00:00+08:00

What happens after an AI produces a proof?

Much of the discussion concerns whether the proof is correct, elegant, or genuinely useful. I have been wondering about a further possibility:

Mathematics could remain deductive in justification while becoming more empirical in how we discover structures and form theories.

Discussions about AI proofs often turn to elegance and understanding. A machine might produce a correct argument that is long, awkward, or difficult to learn from. That seems like a limitation. But perhaps such a proof could also be a beginning.

Proofs as observations

In the past, we could only observe small examples or special cases that humans can handle by hand. With AI, observation can be extended to a far larger scale. The intermediate objects, reductions, and constructions of complex counterexamples become material to study. They are the phenomena of mathematical principles, and observations that can scale up.

flowchart TD
    A["Mathematician: ask related questions<br/>and propose conjectures from partial knowledge"]
    B["AI assistant: seek proofs or counterexamples<br/>Verify results before treating them as observations"]
    C["Mathematician: synthesize observations<br/>Find shared constructions and limits of validity"]
    D["Distill concepts and prove general theorems<br/>Deepen understanding of the mathematical object"]
    A --> B --> C --> D
    C -->|Design further mathematical experiments| A
    D -->|Ask new questions| A

Like observations that helped physicists develop thermodynamics, verified proofs and counterexamples can guide mathematicians toward underlying structures. Summarizing these observations and testing further conjectures creates a cycle of discovery; advances in AI can make this cycle easier to repeat at scale.

Terence Tao’s brief tour of the Equational Theories Project gives a concrete example of exploring a large family of mathematical questions with machine assistance.

Verification and understanding are different achievements

Verification settles truth, but understanding still lives in the mind rather than the compiler. Mathematicians have long turned opaque arguments into understanding by reworking them, and a collection of machine-generated proofs could sometimes invite the same process. An awkward proof may contain a useful idea that has not yet been expressed clearly.

Some concerns

What about failures? Failures are ambiguous. Failing to find a proof or counterexample may reflect the model’s limitations or the nature of the conjecture itself. We need to distinguish the two.
Will lock-in take place and we stop delving into foundations? I believe mathematicians can notice repeated friction in AI’s output. That is where we should revisit our foundations.

Cite this post

If you find this post useful in your work, you can cite it as:

Chi, Yuhan. “Proofs as Observations.” September 30, 2026. https://yuhanchi.github.io/writing/proofs-as-observations/

Or use the following BibTeX entry:

@misc{chi2026proofs,
  author = {Chi, Yuhan},
  title = {{Proofs as Observations}},
  year = {2026},
  month = sep,
  howpublished = {Blog post},
  url = {https://yuhanchi.github.io/writing/proofs-as-observations/}
}

Choice and Alignment

2026-08-14T00:00:00+08:00

Is it possible to please everyone?

In a society, we all have our own preferences, yet we often need to reach a collective decision or ranking. This is the basic question of social choice theory. It also resembles a central problem in AI alignment: a system may need to act on behalf of many people whose values are neither identical nor easy to compare.

Arrow’s framework makes this question precise.

Alternatives and individuals

Let

\[X = \{x_1, x_2, \ldots, x_m\}\]

be a finite set of alternatives. These may be candidates in an election, public policies, or possible actions of an AI system. Let

\[N = \{1, 2, \ldots, n\}\]

be the set of individuals in the society.

Each individual $i \in N$ has a preference relation $\succeq_i$ over $X$. The statement $x \succeq_i y$ means that individual $i$ considers $x$ at least as good as $y$. We normally require this relation to be:

complete: for any $x,y \in X$, either $x \succeq_i y$ or $y \succeq_i x$ (or both); and
transitive: if $x \succeq_i y$ and $y \succeq_i z$, then $x \succeq_i z$.

A complete and transitive preference relation is called a weak order. Its strict part is written $\succ_i$: $x \succ_i y$ means that $i$ strictly prefers $x$ to $y$. Indifference is written $x \sim_i y$.

Let $\mathcal{R}(X)$ denote the set of all weak orders over $X$.

Preference profiles and society

A preference profile records the preferences of everyone in the society:

\[R = (\succeq_1, \succeq_2, \ldots, \succeq_n) \in \mathcal{R}(X)^N.\]

The set $\mathcal{R}(X)^N$ is called the preference-profile space.

Thus, a preference profile is not itself a social ranking. It is the complete input to an aggregation rule: one preference relation for each individual. In this formal model, $N$ is the society, while $R$ is the society’s preference profile over the set of alternatives $X$.

For example, suppose three people rank three alternatives as follows:

Individual	Ranking
1	$x \succ_1 y \succ_1 z$
2	$y \succ_2 z \succ_2 x$
3	$z \succ_3 x \succ_3 y$

Pairwise majority voting gives $x \succ y$, $y \succ z$, and $z \succ x$. Although every individual ranking is transitive, the majority relation is cyclic. This is the Condorcet paradox, and it shows why aggregation is not simply a matter of counting pairwise votes.

A social welfare function is a rule

\[F: \mathcal{R}(X)^N \longrightarrow \mathcal{R}(X)\]

that maps every preference profile $R$ to a complete and transitive social preference relation, or social ranking, $F(R)$. It does not merely select a winner; it ranks all alternatives for society as a whole.

This distinction matters. A social choice function instead maps a profile to one chosen alternative (or a set of chosen alternatives). Arrow’s theorem is usually stated for social welfare functions.

Arrow’s requirements

As we learn in geometry, we should build from the most basic elements.

After these primitive definitions, what kinds of axioms come to mind?

Should it satisfy universality? It seems natural that every preference profile should be mapped to a social ranking.
Should it follow unanimity? If everyone strictly prefers $x$ to $y$—that is, $x \succ_i y$ for every $i \in N$—then shouldn’t the social ranking also have $x \succ y$?
What about independence of irrelevant alternatives? Should the relative ranking of $x$ and $y$ depend only on how people compare $x$ with $y$, regardless of how good or bad another alternative $z$ is?
Should it satisfy monotonicity? Suppose the current social ranking has $x \succ y$. If an individual $k$ changes their preference from $x \prec_k y$ to $x \succ_k y$, shouldn’t the social ranking still have $x \succ y$? This idea also relies on condition 3: reversing the relation between $x$ and $y$ may affect how they compare with other alternatives, but those changes should not affect the collective comparison between $x$ and $y$, right?

More importantly, just as in Euclidean geometry, we want as few axioms as possible. It turns out that condition 4 can be derived from the first three.

Lemma (Monotonicity). Suppose a social welfare function satisfies universality, unanimity, and IIA. If its social ranking has $x \succ y$ under a preference profile $R$, and one individual changes their preference from $x \prec_k y$ to $x \succ_k y$, then the social ranking under the new profile $R’$ still has $x \succ y$.

Proof.

Suppose that, under a preference profile $R$, the individuals in a group $G$ prefer $x$ to $y$, while everyone outside $G$ prefers $y$ to $x$. Nevertheless, the resulting social ranking has $x \succ y$.

Now choose an individual $k \notin G$ and change only their comparison of $x$ and $y$, from $x \prec_k y$ to $x \succ_k y$. Call the resulting preference profile $R’$. We want to show that its social ranking still has $x \succ y$.

Introduce a third alternative $z$ and construct two auxiliary profiles, $R_1$ and $R_2$. The relevant preferences in all four profiles are:

Individuals	$R$	$R_1$	$R_2$	$R’$
$i \in G$	$x \succ_i y$	$x \succ_i y \succ_i z$	$x \succ_i z \succ_i y$	$x \succ_i y$
$k$	$y \succ_k x$	$y \succ_k x \succ_k z$	$x \succ_k z \succ_k y$	$x \succ_k y$
$j \notin G \cup {k}$	$y \succ_j x$	$y \succ_j z \succ_j x$	$z \succ_j y \succ_j x$	$y \succ_j x$

Can you see how to prove it? Think boldly!

First consider $R_1$. Every individual compares $x$ with $y$ exactly as they do under $R$. By IIA, the social comparison is unchanged, so $x \succ y$ under $R_1$. Moreover, everyone prefers $y$ to $z$. By unanimity, the social ranking has $y \succ z$. Transitivity then gives $x \succ z$.

Next consider $R_2$. Every individual compares $x$ with $z$ exactly as they do under $R_1$. By IIA, we retain $x \succ z$. Everyone also prefers $z$ to $y$, so unanimity gives $z \succ y$. By transitivity, $x \succ y$ under $R_2$.

Finally, every individual compares $x$ with $y$ in the same way under $R_2$ and $R’$. Applying IIA once more gives $x \succ y$ under $R’$. Thus, moving one individual’s preference in favor of $x$ cannot cause $x$ to fall below $y$ in the social ranking. This is exactly monotonicity.

\[\square\]

A instinctive idea is whether we can apply this theory to LLM, especially alignment.

We can easily formulate a very simple one-to-one corresponding between AI alignment and Social choice theory.

AI alignment	Social choice theory	Notation
Model responses, actions, or trajectories	Alternatives	$x \in X$
Annotators, users, or stakeholders	Individuals	$i \in N$
One person’s comparison of model behaviors	Individual preference relation	$\succeq_i$
All human feedback	Preference profile	$R=(\succeq_1,\ldots,\succeq_n)$
Feedback-aggregation procedure	Social welfare function	$F$
Collective ordering learned from feedback	Social ranking	$F(R)$

The crucial step is still the same: many individual rankings are compressed into one collective target.
If alignment is meant to incorporate everyone’s ordering, then the model must somehow turn a profile of diverse human preferences into one coherent ranking of its possible behaviors.

And now an uncomfortable concern arises. Will the model’s values be dominated by a small group of people who hold the power to define them? More sharply, could there be a single dominator—one voice whose preferences always become the model’s preferences?

Actually, Arrow gives this concern a precise name.

Definition (Dictator). An individual $d \in N$ is a dictator if, for every preference profile and every pair $x,y \in X$, $x \succ_d y$ always implies $x \succ y$ in the social ranking, regardless of what everyone else prefers.

Surely we would also want non-dictatorship: no such individual should exist. Yet this final wish conflicts with the apparently natural requirements we already imposed.

A dictator always exists

Let $G \subseteq N$ and let $x,y \in X$.

Definition (Weakly decisive set). The set $G$ is weakly decisive (or almost decisive) for $x$ over $y$ if

\[\left. \begin{aligned} x &\succ_i y && \text{for every } i\in G,\\ y &\succ_j x && \text{for every } j\notin G \end{aligned} \right\} \quad\Longrightarrow\quad x\succ y.\]

Thus, $G$ wins even when everyone outside $G$ takes the opposite side.

Definition (Strongly decisive set). The set $G$ is strongly decisive for $x$ over $y$ if

\[x\succ_i y \text{ for every } i\in G \quad\Longrightarrow\quad x\succ y,\]

regardless of the preferences outside $G$. A strongly decisive set is usually called simply a decisive set.

The monotonicity lemma immediately connects the two definitions:

\[\text{weakly decisive for }(x,y) \quad\Longrightarrow\quad \text{strongly decisive for }(x,y).\]

IIA first removes all comparisons unrelated to $x$ and $y$. Starting from the worst case, in which every outsider opposes $G$, monotonicity then tells us that moving any outsider toward $x$ cannot reverse $x\succ y$.

Lemma (Field expansion). If $G$ is decisive for one ordered pair, then it is decisive for every ordered pair.

Proof. Suppose $G$ is decisive for $x$ over $y$. Introduce $z$ and construct

\[\begin{array}{c|cc} & G & N\setminus G \\ \hline P_1 & x\succ y\succ z & y\succ z\succ x \end{array}\]

Then

\[\begin{aligned} x&\succ y,\\ y&\succ z,\\ \therefore\quad x&\succ z. \end{aligned}\]

The first relation follows from decisiveness, the second from unanimity, and the third from transitivity. Exactly $G$ supports $x$ over $z$; by IIA, the same social comparison holds in every profile with this pairwise pattern. Thus $G$ is weakly—and hence strongly—decisive for $x$ over $z$.

Now construct

\[\begin{array}{c|cc} & G & N\setminus G \\ \hline P_2 & z\succ x\succ y & y\succ z\succ x \end{array}\]

This time,

\[\begin{aligned} z&\succ x,\\ x&\succ y,\\ \therefore\quad z&\succ y. \end{aligned}\]

Exactly $G$ supports $z$ over $y$, so IIA and monotonicity make $G$ decisive for this pair as well. Repeating these two extensions reaches every ordered pair. $\square$

Theorem (Arrow). If there are at least three alternatives, every social welfare function satisfying universality, unanimity, and IIA has a dictator.

Proof. By unanimity, $N$ is decisive. Choose a decisive set $G$ of minimum size. Suppose $|G|>1$ and split it into two nonempty sets:

\[G=G_1\mathbin{\dot\cup}G_2, \qquad H=N\setminus G.\]

By universality, the following profile is admissible:

Individuals	Preference
$G_1$	$x \succ y \succ z$
$G_2$	$y \succ z \succ x$
$H$	$z \succ x \succ y$

Every member of $G$ prefers $y$ to $z$, while everyone in $H$ prefers $z$ to $y$. Since $G$ is decisive,

\[y\succ z.\]

By completeness, either

\[x\succ z \qquad\text{or}\qquad z\succeq x.\]

Case 1: $x\succ z$. Exactly the members of $G_1$ prefer $x$ to $z$. By IIA, $G_1$ is weakly decisive for $(x,z)$, hence strongly decisive for every pair. But

\[\varnothing\neq G_1\subsetneq G,\]

contradicting the minimality of $G$.

Case 2: $z\succeq x$. From transitivity,

\[\begin{aligned} y&\succ z,\\ z&\succeq x,\\ \therefore\quad y&\succ x. \end{aligned}\]

Exactly the members of $G_2$ prefer $y$ to $x$. Thus $G_2$ is weakly decisive for $(y,x)$ by IIA, hence strongly decisive for every pair. Again,

\[\varnothing\neq G_2\subsetneq G,\]

contradicting minimality.

Therefore $G$ cannot contain more than one individual. Hence

\[G=\{d\}\]

for some $d\in N$. Since $G$ is decisive for every ordered pair, $d$ is a dictator.

\[\square\]

Therefore universality, unanimity, IIA, and non-dictatorship cannot all hold at once.

Back to real alignment

Before drawing any conclusion about deployed systems, it is worth being precise about what the theorem is.

Arrow’s theorem is a conditional. It says:

if a rule accepts every profile,
if its input is ordinal and interpersonally non-comparable,
if it obeys IIA, and
if it returns one complete transitive ranking over at least three alternatives,
$\rightarrow$ then it has a dictator.

But actually, every real alignment method escapes the conclusion, because every one of them violates a hypothesis. So the interesting question is never “does this system have a dictator?” It is:

Which hypothesis is the system paying to avoid one, and what is the bill?

Three answers follow, and then one that says the bill cannot be avoided by refusing to aggregate at all.

What RLHF gives up: independence

The standard reward model is Bradley–Terry:

\[\Pr(y \succ y') = \sigma\big(r(y) - r(y')\big), \qquad \sigma(t) = \frac{1}{1 + e^{-t}}.\]

Here $r$ is a single scalar reward fitted by maximum likelihood to comparisons pooled across annotators. Two of Arrow’s assumptions fail at once. The reward is cardinal rather than merely ordinal, and annotator identity is usually discarded. The rule therefore never sees a profile $R=(\succeq_1,\ldots,\succeq_n)$, only an unlabeled pool of comparisons.

Cardinality is the classical way around Arrow’s theorem: once interpersonal comparison is allowed, utilitarian-style rules become available. In this case, we can also identify the aggregation rule more precisely.

Recall the Borda count. With $m$ alternatives, a candidate receives a point for each alternative ranked below it, and those points are summed across voters. Siththaranjan, Laidlaw, and Hadfield-Menell show that preference learning implicitly implements this rule when disagreement comes from hidden context, such as annotator identity, values, or labeling criteria.

Once identity is discarded, each comparison is averaged over the context that produced it. Counting wins across voters becomes the probability of winning against a random opponent:

\[\mathrm{BC}(y) = \Pr_{y'\sim\rho}(y\succ y'),\]

where $\rho$ is the distribution of training comparisons. Thus the aggregation performed by RLHF is not unspecified: under these assumptions, it behaves like the Borda rule. This is a consequence of how the data are pooled, not an explicit design choice.

Two of Borda’s known defects transfer directly.

IIA fails operationally. Suppose

\[\Pr(y_1\succ y_2)=0.6,\qquad \Pr(y_1\succ y_3)=0.4,\qquad \Pr(y_2\succ y_3)=0.7.\]

	Pair weights	$\mathrm{BC}(y_1)$	$\mathrm{BC}(y_2)$	Winner
Uniform	$(y_1 y_2)=\tfrac13,\;(y_1 y_3)=\tfrac13,\;(y_2 y_3)=\tfrac13$	$0.50$	$\mathbf{0.55}$	$y_2$
Oversample $y_1 y_2$	$(y_1 y_2)=\tfrac{9}{10},\;(y_1 y_3)=\tfrac{1}{20},\;(y_2 y_3)=\tfrac{1}{20}$	$\mathbf{0.59}$	$0.42$	$y_1$

$\Pr(y_1\succ y_2)=0.6$ in both rows. Only $y_3$’s sampling frequency changed; the winner flipped.

The rule is manipulable. Borda rewards strategic ranking. The same paper argues that annotators may therefore benefit from misreporting their preferences to shift the learned model, with possible downstream effects on jailbreak robustness.

What Constitutional AI gives up: universality

Constitutional AI begins with a fixed set of principles rather than aggregating an unrestricted set of preferences. In Arrow’s language, this restricts the domain: $F$ is defined not on all of $\mathcal{R}(X)^N$, but on a subset $\mathcal{D}\subsetneq\mathcal{R}(X)^N$. Profiles that conflict with the constitution are not reconciled within the rule; they are treated as inadmissible.

Restricting the domain is sufficient to avoid Arrow’s conclusion. Black’s theorem gives a standard example.

Definition (Single-peaked). Preferences are single-peaked with respect to a common ordering of $X$ if each individual has an ideal point, and satisfaction falls monotonically when moving away from it in either direction.

Theorem (Black, 1948). If every individual’s preference is single-peaked with respect to one common ordering, pairwise majority rule yields a transitive social ranking topped by the median voter’s ideal point.

Under these conditions, majority rule has neither a cycle nor a dictator. The result depends on a substantive restriction: a common axis must be specified, and preferences must be single-peaked along it. The median voter’s ideal point then determines the top-ranked alternative.

Writing a constitution is analogous to choosing that axis. It does not remove the aggregation problem; it moves the problem up one level, from deciding which response is best to deciding which principles should govern responses.

The question can recur: who decides how opinions about principles are combined? This regress is not fatal; an institution can choose a stopping point and justify it. But the stopping point remains a choice rather than a neutral consequence of the theory.

Personalization and minimal liberalism

A natural response is to give each user a separate policy. If no collective ranking is required, perhaps dictatorship can be avoided.

Sen’s theorem shows why this move has limits. It concerns rights rather than rankings.

Definition (Minimal liberalism). A rule satisfies minimal liberalism if there exist $i \neq j$ and pairs ${x,y}, {z,w} \subseteq X$ such that $i$ is decisive over ${x,y}$, and $j$ is decisive over ${z,w}$.

Theorem (Sen, 1970). No rule on the unrestricted domain satisfies both weak Pareto and minimal liberalism while always returning an acyclic social preference.

Proof.

\[a: \text{model answers user 1}, \quad b: \text{model answers user 2}, \quad c: \text{refuses both}\]

User 1’s sphere: ${a,c}$. User 2’s sphere: ${b,c}$.

	Ranking
1	$c \succ_1 a \succ_1 b$
2	$a \succ_2 b \succ_2 c$

\[c \succ a \ (\text{sphere}_1), \qquad b \succ c \ (\text{sphere}_2), \qquad a \succ b \ (\text{Pareto})\] \[\Rightarrow a \succ b \succ c \succ a. \qquad \square\]

Personalization is a form of minimal liberalism: allowing user $i$ to set their own policy grants them decisiveness over alternatives that differ only in $i$’s affairs. The cycle requires external preferences, meaning that people care about choices in others’ spheres as well as their own. Without such preferences, Pareto cannot produce the final edge of the cycle. In practice, however, disputes over model policy often include external preferences. Excluding them would itself be a value judgment.

Randomization and strategyproofness

A decision scheme maps profiles to distributions over $X$: $d: \mathcal{R}(X)^N \to \Delta(X)$.

Definition. Strategyproof: truthful reporting is always weakly optimal. Ex post Pareto optimal: never assigns positive probability to a unanimously dominated alternative.

Theorem (Gibbard, 1977). With $\geq 3$ alternatives, any strategyproof, ex post Pareto-optimal scheme is a probability mixture of dictatorships: fix weights $(w_1,\ldots,w_n)$, draw a person, and implement that person’s top choice.

If probabilities are determined by a score, such as a Borda count or weighted average, misreporting can shift both the score and the resulting odds. Under random dictatorship, the weights are fixed before anyone reports. A person cannot change their chance of being selected, and truthful reporting is optimal once they are selected.

The mechanism must still specify who enters the lottery and with what weight. Using a smoother aggregation rule requires giving up strategyproofness and accepting incentives to misreport.

Summary

Arrow’s hypothesis	Given up by	What replaces it	Trade-off
Universality	Constitutional AI, hard refusals	Restricted domain; Black-style possibility	The constitution must specify both principles and a common axis
Ordinal, non-comparable input	Bradley–Terry reward models	Cardinal aggregation — in fact Borda count	IIA violations and incentives for strategic annotation
One transitive social ranking	Stochastic policies	Decision schemes	Strategyproofness and efficiency jointly permit only random dictatorship
One ranking for everyone	Personalization	Per-user policies	External preferences can conflict with individual decisiveness

Every column on the right represents a choice about whose preferences count, how they are compared, and which decisions they may govern. None of these choices is forced by the mathematics, and none is neutral.

There is no neutral aggregation rule under these requirements. In practice, the rule’s assumptions and trade-offs should be explicit, revisable, and open to scrutiny by the people whose preferences it aggregates.

No rule can satisfy every preference under all conditions. What remains possible is to state the trade-offs clearly and make the resulting decisions accountable.

Cite this post

If you find this post useful in your work, you can cite it as:

Chi, Yuhan. “Choice and Alignment.” August 14, 2026. https://yuhanchi.github.io/writing/choice-and-alignment/

Or use the following BibTeX entry:

@misc{chi2026choice,
  author = {Chi, Yuhan},
  title = {{Choice and Alignment}},
  year = {2026},
  month = aug,
  howpublished = {Blog post},
  url = {https://yuhanchi.github.io/writing/choice-and-alignment/}
}

My Visit to Professor Ziad Obermeyer at UC Berkeley

2026-07-15T00:00:00+08:00

Meeting with Professor Ziad Obermeyer (School of Public Health)

Takeaways

Applied Mathematics?
Cannot be the top in one field? Try to be in the top 5% in several fields!
In serious scenarios, like medical systems, how much do we value interpretability?
Very much. Interpretability in this area means that we want our models to perform well on data they haven’t seen before. Besides, we need to know if it really has a sense of “science”. For example, if I edit the symptom a little, what kind of change will happen in the output, and is the change aligned with the biology?
Why do we still need mathematical models? “The bitter lesson” has said: the data and compute will definitely outperform engineered architecture.
We need them. First, “Garbage in, garbage out”. We need mathematical models to make some data more “high-quality”. Secondly, sometimes we cannot directly investigate the things we want. Hence, we should use mathematical models to find proxies.

Floating Point Notes in Computers

2026-07-11T00:00:00+08:00

Let’s build the concept of floating-point numbers in computers from first principles.

1. The Core Conflict: The Finite Cannot Represent the Infinite

The real number line is perfectly continuous and stretches to infinity. However, computers use fixed-size memory registers (e.g., 32 bits or 64 bits). A 32-bit sequence can only ever represent $2^{32}$ distinct states.

First Principle: You cannot map an infinite, continuous space into a finite, discrete space without losing information. We must accept that computer mathematics will inherently involve approximations and rounding errors.

2. Binary and the Need for Exponents

Computers operate in binary. Just as we use base-10 decimals, we can use binary fractions (e.g., $1.01_2 = 1.25_{10}$).

If we used a fixed-point system, we couldn’t represent both cosmic scales ($10^{20}$) and quantum scales ($10^{-10}$) simultaneously. To cover both within just 32 bits, we borrow from scientific notation: $a \times 10^b$. In binary, this becomes $F \times 2^E$. The radix point “floats” depending on the exponent $E$.

3. Floating-Point Memory Layout (Bit Distribution)

In the IEEE 754 standard, a floating-point number is stored in three parts: Sign (S), Exponent (E), and Fraction/Mantissa (F).

Single Precision (float): 1 sign bit + 8 exponent bits + 23 fraction bits = 32 bits
Double Precision (double): 1 sign bit + 11 exponent bits + 52 fraction bits = 64 bits

Single Precision (32-bit) Memory Layout Diagram:

 31  30           23  22                                      0
 +--+---------------+-----------------------------------------+
 | S|    Exponent   |                Fraction                 |
 +--+---------------+-----------------------------------------+
 1-bit   8-bit                         23-bit

Double Precision (64-bit) Memory Layout Diagram:

 63  62                     52  51                            0
 +--+-------------------------+-------------------------------+
 | S|        Exponent         |           Fraction            |
 +--+-------------------------+-------------------------------+
 1-bit        11-bit                         52-bit

4. Engineering Trade-offs: Implicit Bit, Gradual Underflow, and Biased Exponent

When designing the standard, engineers made several major trade-offs to optimize for efficiency, speed, and mathematical continuity:

Trade-off 1: The Implicit Leading Bit (Efficiency)

In binary scientific notation, a normalized number always starts with 1 (e.g., $1.011_2 \times 2^3$). Since the digit before the radix point is always 1, we don’t store it; we simply assume its presence. This provides one free bit of precision.

BUT, there is a problem with the implicit 1. If every number starts with 1.xxx, how do we represent numbers very close to 0?

If the smallest exponent is $-126$, the smallest normalized positive number is $1.0 \times 2^{-126}$. Below this, the value would abruptly jump straight to 0. This huge gap breaks the mathematical property that $x - y = 0$ if and only if $x = y$.

The Solution: Denormalization (Gradual Underflow)

To fix this, the standard introduces a rule: if the Exponent field is all 0s, the implicit leading bit changes from 1. to 0..

These are called Denormalized (or Subnormal) numbers.
Denormalized numbers do not have higher precision. They actually have lower precision, but they can represent numbers much closer to 0.
As the value gets smaller, the number of leading zeros increases, and the significant bits drop from 24 down to 1. This gradual underflow bridges the gap to 0 and prevents abrupt jumps.

Trade-off 2: Biased Exponent (Fast Comparison)

To rapidly compare two floating-point numbers, we want to feed their raw 32-bit patterns directly into standard integer comparison circuits.

If we used two’s complement for negative exponents, a negative exponent (like 11111111) would appear as a massive unsigned integer, breaking our fast integer comparison trick.

Solution: We use a bias (offset). By adding a constant (e.g., 127 for single precision) to the actual exponent, all stored exponents become strictly positive. This preserves the integer sorting order!

Single precision: $\text{Stored } E = \text{True exponent} + 127$
To decode: $\text{True exponent} = \text{Stored } E - 127$
Double precision: bias is 1023.

5. Read the Bits Directly: Extreme Values in 32-bit Single Precision

This is the place to stop talking in the abstract and read real 0/1 patterns. Single precision rule recap:

Layout: S (1 bit) | E (8 bits) | F (23 bits), Bias = 127.
If E != 00000000 and E != 11111111: normalized, value = $(-1)^S \times 1.F_2 \times 2^{E_{unsigned}-127}$.
If E == 00000000: denormalized / zero, value = $(-1)^S \times 0.F_2 \times 2^{-126}$. Note the exponent is fixed at $-126$ here, not $0-127=-127$.

Overview of the six extremes:

#	Name	32-bit pattern `S EEEEEEEE F...`	Hex	Approx. value
A	Largest finite positive	`0 11111110 11111111111111111111111`	`0x7F7FFFFF`	$\approx +3.40 \times 10^{38}$
B	Smallest positive normalized	`0 00000001 00000000000000000000000`	`0x00800000`	$\approx +1.18 \times 10^{-38}$
C	Smallest positive denormalized	`0 00000000 00000000000000000000001`	`0x00000001`	$\approx +1.40 \times 10^{-45}$
D	Most negative (largest magnitude)	`1 11111110 11111111111111111111111`	`0xFF7FFFFF`	$\approx -3.40 \times 10^{38}$
E	Negative closest to zero, normalized	`1 00000001 00000000000000000000000`	`0x80800000`	$\approx -1.18 \times 10^{-38}$
F	Negative closest to zero, denormalized	`1 00000000 00000000000000000000001`	`0x80000001`	$\approx -1.40 \times 10^{-45}$

Case A — Largest finite positive: 0 11111110 11111111111111111111111

S = 0  (+)
E = 11111110
F = 11111111111111111111111  (23 ones)
Hex = 0x7F7FFFFF

Exponent bits to unsigned: $11111110_2 = 128+64+32+16+8+4+2+0 = 254_{10}$.
Minus bias: $E_{true} = 254 - 127 = +127$.
Mantissa: $1.F_2 = 1.111\dots111_2 = 2 - 2^{-23} \approx 1.99999988$.
Value: $(2-2^{-23}) \times 2^{127} = 2^{128} - 2^{104} \approx 3.4028235 \times 10^{38}$.

Why not E = 11111111? That pattern is reserved for Inf / NaN, so 11111110 is the largest finite exponent.

Case B — Smallest positive normalized: 0 00000001 00000000000000000000000

S = 0  (+)
E = 00000001
F = 00000000000000000000000  (23 zeros)
Hex = 0x00800000

Exponent bits to unsigned: $00000001_2 = 1_{10}$.
Minus bias: $E_{true} = 1 - 127 = -126$.
Mantissa: $1.000\dots000_2 = 1.0$.
Value: $1.0 \times 2^{-126} \approx 1.17549435 \times 10^{-38}$.

Anything smaller in magnitude cannot stay normalized; it must become denormalized.

Case C — Smallest positive denormalized: 0 00000000 00000000000000000000001

S = 0  (+)
E = 00000000  (special: denormalized)
F = 00000000000000000000001  (only LSB = 1)
Hex = 0x00000001

Exponent field is all zeros, so the rule changes: leading bit is 0., true exponent is fixed at $-126$ (do not compute $0-127=-127$).
Mantissa: $0.F_2 = 0.000\dots001_2 = 2^{-23}$.
Value: $2^{-23} \times 2^{-126} = 2^{-149} \approx 1.40129846 \times 10^{-45}$.

This is the closest positive number to zero. One step smaller is exactly +0.

Case D — Most negative (mirror of Case A): 1 11111110 11111111111111111111111

S = 1  (-)
E = 11111110
F = 11111111111111111111111
Hex = 0xFF7FFFFF

Exponent: $11111110_2 = 254_{10}$, $254-127 = +127$, identical to Case A.
Value: $-(2-2^{-23}) \times 2^{127} \approx -3.4028235 \times 10^{38}$.

Only the sign bit flips; exponent and fraction decoding are unchanged.

Case E — Negative closest to zero, normalized: 1 00000001 00000000000000000000000

S = 1  (-)
E = 00000001
F = 00000000000000000000000
Hex = 0x80800000

Exponent: $00000001_2 = 1_{10}$, $1-127 = -126$.
Value: $-1.0 \times 2^{-126} \approx -1.17549435 \times 10^{-38}$.

Case F — Negative closest to zero, denormalized: 1 00000000 00000000000000000000001

S = 1  (-)
E = 00000000  (special: denormalized)
F = 00000000000000000000001
Hex = 0x80000001

Same denormalized rule: $0.F_2 \times 2^{-126} = 2^{-23} \times 2^{-126} = 2^{-149}$.
Value: $\approx -1.40129846 \times 10^{-45}$.

Reference for intuition: the largest denormalized number 0 00000000 11111111111111111111111 is $(1-2^{-23}) \times 2^{-126} \approx 1.17549421 \times 10^{-38}$, just one step below the smallest normalized number in Case B. That is gradual underflow in action.

6. Special Numbers: 0, Inf, NaN

Because our system is finite, operations can overflow or yield mathematically undefined results. To preserve our fast integer sorting trick, where should infinity live? At the very top (the largest possible exponent).

Exponent = All 1s:
- If Fraction == 0: Infinity (+Inf or -Inf).
- If Fraction != 0: NaN (Not a Number, e.g., for $\sqrt{-1}$).
Exponent = All 0s: Reserved for Zero (if the fraction is 0) and Subnormals (if the fraction is non-zero, see Cases C/F above).

7. Key Examples to Test Your Understanding

Example 1: What is the smallest single-precision number strictly greater than 2?

To find this, we examine how $2.0$ is represented.
$2.0 = 1.0 \times 2^1$.
In Single Precision: Sign = 0, Exponent = $1 + 127 = 128$, Fraction = $000\dots000$ (23 zeros).
The next representable number simply increments the fraction by the smallest possible amount (setting the least significant bit, LSB, to 1).
The Fraction becomes $000\dots001$.
The value is $1.000\dots001_2 \times 2^1 = (1 + 2^{-23}) \times 2 = 2 + 2^{-22}$.
Therefore, the smallest number greater than 2 is $2 + 2^{-22}$ (approximately $2 + 2.38 \times 10^{-7}$).

Example 2: What is the largest exact odd integer that can be represented?

To represent an exact integer, the fractional part must completely cover the integer bits without any fractional remainder.
Single precision has 24 bits of precision (1 implicit + 23 explicit).
The largest integer where the spacing between representable numbers is exactly 1 occurs when the exponent pushes the radix point perfectly to the end of the 23-bit fraction.
This happens when $E - 127 = 23 \implies \text{Exponent} = 150$.
The form is $1.\text{Fraction} \times 2^{23}$.
The maximum value of this form is $1.111\dots111_2 \times 2^{23} = 2^{24} - 1 = 16,777,215$.
What if we go larger? At $E - 127 = 24$, the value is $1.F \times 2^{24}$. The least significant bit now represents $2^{-23} \times 2^{24} = 2^1 = 2$. This means numbers in this range can only represent EVEN integers (the spacing is 2).
So, the largest odd integer in single precision is $2^{24} - 1 = 16,777,215$. (For double precision, it is $2^{53} - 1 = 9,007,199,254,740,991$).

8. How Does the Computer Actually Calculate?

The underlying philosophy for calculations is to reduce complex operations to integer math and polynomial approximations.

Addition (`+`) and Subtraction (`-`)

Alignment: The CPU right-shifts the fraction of the number with the smaller exponent until both exponents match.
Integer Math: The fractions are fed into the exact same two’s complement adder circuit used for integers.
Normalization: The CPU shifts the fraction back into the $1.\text{xxx}$ format and adjusts the exponent accordingly.

Warning: Subtracting two very close numbers causes catastrophic cancellation, where significant bits cancel out, instantly destroying precision.

Multiplication (`*`) and Division (`/`)

Multiply: Multiply the fractions (using an integer multiplier) and add the exponents (subtracting one bias, since both exponents inherently included a bias).
Divide: Divide the fractions and subtract the exponents (adding back one bias).

Exponentiation ($e^x$) and Logarithm ($\ln x$)

The floating-point format ($1.F \times 2^{E-\text{Bias}}$) is already an exponential function!

Exponent ($e^x$): Using the identity $e^x = 2^{x / \ln 2}$, the integer part of the result is directly added to the exponent ($E$) field (fast integer addition!). The tiny fractional part is calculated using a minimax polynomial.
Logarithm ($\ln x$): Since $\ln(x) = \ln(1.F) + (E-\text{Bias})\ln(2)$, the $(E-\text{Bias})$ term is extracted via simple integer arithmetic. The CPU only needs to approximate $\ln(1.F)$ using polynomials.

Power ($x^y$)

This is calculated as $x^y = e^{y \cdot \ln(x)}$ by calling the highly optimized hardware or software log and exp functions.

Trigonometric Functions ($\sin$, $\cos$)

Range Reduction: The input is reduced (e.g., $x \bmod (\pi/2)$). This requires an incredibly precise stored value of $\pi$ to avoid massive errors for large inputs (e.g., using the Payne-Hanek algorithm).
Polynomial Approximation: The reduced input is evaluated using a minimax polynomial (like a Chebyshev series) via hardware FMA (Fused Multiply-Add) instructions, which calculate $(A \times B) + C$ in a single cycle without intermediate rounding errors.

9. Epilogue: Two’s Complement and Hexadecimal

To fully appreciate floating-point design, we must contrast it with how standard integers are stored.

Why do Integers use Two’s Complement?

While floating-point exponents use a Bias to preserve their raw sorting order, standard signed integers use Two’s Complement. Why?

Unified Adder/Subtractor: The biggest advantage is that $A - B$ literally becomes $A + (\text{Two's Complement of } B)$. The CPU only needs to build a single fast addition circuit (ALU) to handle both addition and subtraction.
What about Multipliers and Dividers? Do they share circuits too? Yes! Integer multipliers use arrays of adders (partial products). Two’s complement allows the exact same core multiplier hardware to be used for signed numbers with only minor sign-extension tweaks (like Booth’s multiplication algorithm). Dividers (like non-restoring division) also seamlessly leverage the same add/subtract ALU.
Single Zero: Unlike “Sign-and-Magnitude” encoding, Two’s complement has only one representation for $0$ (no $+0$ and $-0$), preventing bugs in software if (x == 0) checks.

Why do we always read memory in Hexadecimal (Hex)?

You will often see 32-bit floats or integers printed as 0x40490FDB instead of a massive 32-digit binary string.

Perfect mapping: Binary is too long for humans to read, and converting back and forth between Base-10 and Base-2 is computationally annoying. But Base-16 (Hexadecimal) is the perfect middle ground because $16 = 2^4$.
Exactly 4 binary bits map directly to exactly 1 Hex character ($0000_2 = \text{0}_{16}$ through $1111_2 = \text{F}_{16}$).
We can instantly break an 8-bit byte into two 4-bit “nibbles”, represented cleanly by 2 Hex digits. Hex is not a new number format; it is simply a highly readable, compact way for humans to look at pure binary!

CARLA Simulator on WSL2: Resolving the Docker Vulkan Trap

2026-05-31T00:00:00+08:00

1. System & Hardware Specifications

The configuration and environment details used:

Host OS: Windows 11
WSL2 Distro: Ubuntu 24.04 LTS (Kernel: 6.6.87.2-microsoft-standard-WSL2)
GPU: NVIDIA GeForce RTX 5060 Laptop (8GB VRAM, Host Driver: 592.27, CUDA 13.1)
Docker: Docker Desktop 4.63.0 (Engine 29.2.1)
CARLA Version: 0.9.16

2. The Working Solution: Windows Server + WSL2 Client Hybrid Architecture

Instead of fighting virtualization layers and containerized Vulkan driver mapping, the most performant and stable approach for CARLA on WSL2 is a hybrid architecture:

CARLA Server runs natively on Windows: Gives Unreal Engine 4 direct, high-performance access to the host GPU (RTX 5060) for 3D rendering and physics.
CARLA Client runs inside WSL2: Keeps the Python development, training pipelines, and model libraries in the native Linux environment.

┌─────────────────────────────────────────────┐
│  Windows Host (Direct GPU Access)           │
│  CARLA Simulator Server (CarlaUE4.exe)      │
│  Listens on Port 2000                       │
│  Handles: 3D Rendering, Physics, Autopilot  │
└──────────────────┬──────────────────────────┘
                   │
                   │ Network (127.0.0.1:2000)
                   │
┌──────────────────▼──────────────────────────┐
│  WSL2 (Ubuntu 24.04 Dev Environment)        │
│  Python Client (carenv Conda Environment)   │
│  Handles: Agent Logic, RL training, Sensors │
└─────────────────────────────────────────────┘

Installation Steps

Step 1: Windows Side Setup

Download CARLA_0.9.16Windows.zip from the CARLA GitHub Releases.
Extract the zip file to your preferred directory (referred to as <CARLA_PATH> here, e.g., C:\CARLA_0.9.16).
Install the required dependencies:
- DirectX End-User Runtime: Download and run dxwebsetup.exe.
- Visual C++ Redistributable 2015-2022 (x64): Download and run VC_redist.x64.exe.

Step 2: WSL2 Side Setup

Create and activate a conda environment for the CARLA client:

conda create -n carenv python=3.10 -y
conda activate carenv
pip install carla

Verify the installation:

python -c "import carla; print('CARLA client package successfully installed!')"

Step 3: Network Configuration (Mirrored Mode)

To ensure seamless network communication between WSL2 and Windows via 127.0.0.1, enable Mirrored mode in WSL2.

Create or edit your Windows user profile configuration file %USERPROFILE%\.wslconfig:

[wsl2]
networkingMode=mirrored

Then, restart WSL2 from a Windows PowerShell terminal to apply the changes:

wsl --shutdown

3. Running the Simulation

Step 1: Start the CARLA Server (Windows)

Open a Windows PowerShell terminal and start the server:

cd <CARLA_PATH>

# Options:
# 1. With visual window (default)
.\CarlaUE4.exe

# 2. Headless mode (no visual output, lower VRAM overhead)
.\CarlaUE4.exe -RenderOffScreen -nosound

# 3. Custom window resolution
.\CarlaUE4.exe -windowed -ResX=1280 -ResY=720

Step 2: Run Python Client Scripts (WSL2)

Once the server is running on the Windows side, launch the client scripts inside WSL2:

1. Connection Test (`test_connection.py`)

"""CARLA 连接测试脚本 - 从 WSL2 连接 Windows CARLA 服务端"""
import carla

client = carla.Client('127.0.0.1', 2000)
client.set_timeout(10.0)

print(f"Server version: {client.get_server_version()}")
print(f"Client version: {client.get_client_version()}")
print(f"可用地图: {client.get_available_maps()}")

world = client.get_world()
print(f"当前地图: {world.get_map().name}")
print(f"当前 actors 数量: {len(world.get_actors())}")
print("\n连接测试通过!")

2. Spawn and Control Vehicle (`spawn_vehicle.py`)

This script spawns a Tesla Model 3, attaches a front-facing camera, enables autopilot, and saves camera frames to a local folder for 10 seconds:

"""CARLA 生成车辆 + 摄像头截图"""
import carla
import random
import time
import os

OUTPUT_DIR = "output"
os.makedirs(OUTPUT_DIR, exist_ok=True)

# 连接
client = carla.Client('127.0.0.1', 2000)
client.set_timeout(10.0)
world = client.get_world()

# 获取蓝图库
blueprint_library = world.get_blueprint_library()

# 找一个车辆蓝图
vehicle_bp = blueprint_library.find('vehicle.tesla.model3')
spawn_points = world.get_map().get_spawn_points()
spawn_point = random.choice(spawn_points)

# 生成车辆
vehicle = world.spawn_actor(vehicle_bp, spawn_point)
print(f"车辆已生成: {vehicle.type_id} at {spawn_point.location}")

# 添加摄像头传感器
camera_bp = blueprint_library.find('sensor.camera.rgb')
camera_bp.set_attribute('image_size_x', '800')
camera_bp.set_attribute('image_size_y', '600')
camera_bp.set_attribute('fov', '90')

# 摄像头放在车顶上方
camera_transform = carla.Transform(carla.Location(x=1.5, z=2.4))
camera = world.spawn_actor(camera_bp, camera_transform, attach_to=vehicle)
print(f"摄像头已挂载")

# 保存截图计数
frame_count = [0]

def save_image(image):
    frame_count[0] += 1
    filepath = os.path.join(OUTPUT_DIR, f"frame_{frame_count[0]:04d}.png")
    image.save_to_disk(filepath)
    print(f"截图保存: {filepath}")

camera.listen(save_image)

# 让车辆自动驾驶
vehicle.set_autopilot(True)
print("自动驾驶已开启")

# 运行 10 秒，采集截图
print("开始采集，运行 10 秒...")
time.sleep(10)

# 清理
camera.stop()
camera.destroy()
vehicle.destroy()
print(f"完成! 共采集 {frame_count[0]} 帧截图，保存在 {OUTPUT_DIR}")

4. The Failed Path: Running CARLA inside WSL2 Docker (Troubleshooting Record)

For completeness, below is a record of our attempts to run CARLA directly inside a WSL2 Docker container, and why it consistently failed.

Attempted Run Configurations

Attempt 1: Basic GPU-enabled Container

docker run -d \
    --name carla \
    --gpus all \
    --net=host \
    -e NVIDIA_VISIBLE_DEVICES=all \
    -e NVIDIA_DRIVER_CAPABILITIES=all \
    carlasim/carla:0.9.16 \
    bash CarlaUE4.sh -RenderOffScreen -nosound

Result: The engine complains: WARNING: lavapipe is not a conformant vulkan implementation. After 60 seconds, it crashes with GameThread timed out waiting for RenderThread.

Attempt 2: Mount Host Vulkan ICD File

docker run -d \
    --name carla \
    --gpus all \
    --net=host \
    -e NVIDIA_VISIBLE_DEVICES=all \
    -e NVIDIA_DRIVER_CAPABILITIES=all \
    -e VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/nvidia_icd.json \
    -v /usr/share/vulkan/icd.d:/usr/share/vulkan/icd.d:ro \
    carlasim/carla:0.9.16 \
    bash CarlaUE4.sh -RenderOffScreen -nosound

Result: Immediate Segmentation Fault.

Attempt 3: Using the Official `--runtime=nvidia` Flag

docker run -d \
    --name carla \
    --runtime=nvidia \
    --net=host \
    -e NVIDIA_VISIBLE_DEVICES=all \
    -e NVIDIA_DRIVER_CAPABILITIES=all \
    carlasim/carla:0.9.16 \
    bash CarlaUE4.sh -RenderOffScreen -nosound

Result: The same lavapipe warning, followed by a render thread timeout crash after 60 seconds.

Attempt 4: Forcing OpenGL Mode

docker run -d \
    --name carla \
    --gpus all \
    --net=host \
    -e NVIDIA_VISIBLE_DEVICES=all \
    -e NVIDIA_DRIVER_CAPABILITIES=all \
    -e DISPLAY=:0 \
    carlasim/carla:0.9.16 \
    bash CarlaUE4.sh -RenderOffScreen -nosound -opengl

Result: Crash. WSL2 does not expose a /dev/dri/ device inside the container, making OpenGL hardware acceleration unavailable.

Root Cause Analysis

The rendering failure in the container is caused by three layered issues:

1. The `nvidia_icd.json` Directory Bug

In Ubuntu 24.04:

$ file /usr/share/vulkan/icd.d/nvidia_icd.json
/usr/share/vulkan/icd.d/nvidia_icd.json: directory

A post-installation package manager bug mistakenly runs mkdir -p on the path, converting the configuration JSON file into an empty folder. This breaks standard Vulkan loader lookups.

Fix: sudo rm -rf /usr/share/vulkan/icd.d/nvidia_icd.json

2. Missing Vulkan Mapping Libraries in WSL2

Even with the JSON path fixed, WSL2 is missing NVIDIA’s Vulkan mapping libraries. Under /usr/lib/wsl/lib/ (and its Windows host counterpart C:\Windows\System32\lxss\lib\), we can see:

libcuda.so (CUDA computation works)
libd3d12.so (DirectX 12 works)
libnvidia-ml.so.1 (nvidia-smi works)
libvulkan_nvidia.so (❌ Missing!)
libGLX_nvidia.so (❌ Missing!)

NVIDIA’s host drivers do not map the Vulkan/OpenGL shared libraries into the WSL2 environment for Blackwell-based Laptop GPUs (like the RTX 5060) on host driver version 592.27.

3. Software Renderer Fallback

Because no hardware Vulkan/OpenGL drivers are visible to WSL2, the Vulkan loader falls back to llvmpipe / lavapipe (CPU-based software rendering):

$ VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/gfxstream_vk_icd.x86_64.json vulkaninfo --summary
ERROR: setup_loader_term_phys_devs: Failed to detect any valid GPUs in the current config

CARLA (based on Unreal Engine 4.26) requires hardware rasterization. Forcing the simulation to render on the CPU leads to massive lag, eventually triggering the GameThread timed out waiting for RenderThread watchdog mechanism.

5. Summary of Tips & Troubleshooting

VRAM Optimization: The RTX 5060 Laptop has 8GB VRAM. Running CARLA with a visual window in standard mode occupies around 7.7GB of VRAM. If you run out of memory (OOM), run the server with -quality-level=Low or run it offscreen (-RenderOffScreen).
Networking Issues: If the client fails to connect to 127.0.0.1, check your Windows Firewall settings to ensure port 2000 is open, and verify that networkingMode=mirrored is active in WSL2.
Docker Cleanup: Since Dockerized CARLA is unusable in this setup, free up disk space by removing the unused container image:
```
docker rmi carlasim/carla:0.9.16
```

6. Useful Resources

When CoT Surface Features Transfer

2026-05-04T00:00:00+08:00

This note summarizes my project code-not-text. The interactive version is here: demo. The full technical report is here: PDF report.

Same model, same feature family, two very different traces. Left: a correct AIME24 solution whose reasoning converges step by step. Right: an incorrect LiveCodeBench-v5 trace that reads fluently but keeps hedging while the logic goes wrong. The AUROC in each title is the domain-level AUROC@100% (math 0.982, coding 0.407). From the interactive demo; click the image for full size.

Question

Can cheap summaries of a chain-of-thought trace predict whether a solution is correct?

I study one deliberately narrow feature family: hand-crafted CoT-surface and token-trajectory descriptors. The point is not to test every possible verifier. The point is to ask whether this cheap, interpretable feature family transfers across domains.

Setup

The experiments use DeepSeek-R1-0528-Qwen3-8B at temperature 1.0 across three domains:

Math: AIME24, AIME25, BRUMO25, and HMMT25; 7,680 aligned runs
Science: GPQA-style tasks; 12,672 runs
Coding: LiveCodeBench-v5 tasks; 10,688 runs

The feature family includes token-confidence summaries, token-trajectory statistics, trajectory continuity, novelty, reflection count, and a small activation-derived descriptor. Evaluation uses problem-grouped splits. I report AoA (AUC-of-AUROC across trace anchors), AUROC@100%, and best-of-N reranking.

Main Result

The same feature family behaves very differently across domains.

Domain	AoA	AUROC@100%	Best-of-N=64 pass@1
Math	0.958	0.982	+10.0 pp
Science	0.799	0.841	+8.0 pp
Coding	0.434	0.407	-0.6 pp

This is the central finding: the feature family is strong in math, narrower in science, and weak for coding correctness on unseen problems.

Interpretation

The same feature names do not necessarily measure the same latent construct.

In math, features such as reflection count and trajectory continuity can track whether reasoning is converging. In science, the signal is more confidence-heavy. In coding, executable correctness is farther away from the surface form of the reasoning trace. A trace can look organized and confident while still implementing the wrong program logic.

I view this as a measurement non-invariance result: a feature family can be useful in one domain without being a general-purpose correctness proxy.

Robustness Checks

The coding result is not just one failed probe. I tested several possible explanations:

an 83-scalar coding-specific feature sweep,
grouped feature ablations,
a coding-specific CoT-only judge,
nonlinear MLP classifiers,
SSL pre-training on 42K unlabeled traces,
token-level de-knotting.

None produced a strong generalizable coding verifier from this feature family.

Semantic Knots

I also ran a smaller semantic analysis around what I call semantic knots: local places in the visible reasoning prose where the trace appears to lose state control and does not immediately repair itself. Ordinary uncertainty, normal exploration, and clean one-step self-correction do not count.

This analysis supports the same cross-domain story:

Math: knots are common and strongly associated with incorrect answers.
Science: knots appear less often, and the correctness relationship is weaker.
Coding: knots are rare and do not meaningfully separate correct from incorrect solutions.

The de-knotting check is especially useful. Removing knot spans reduces math signal, which suggests that those spans carry failure information. In coding, de-knotting does not reveal a hidden verifier; it leaves the weak signal essentially weak. That is why I read the result as measurement non-invariance rather than a missing preprocessing trick.

Boundary

This does not claim that all text-based verifiers fail. It also does not rule out code-aware models, execution-aware selectors, hidden-state probes, raw-text classifiers, or LLM judges.

The narrower conclusion is: cheap CoT-surface features are domain-specific measurement instruments, not general-purpose correctness proxies.

Cite this post

If you find this post useful in your work, you can cite it as:

Chi, Yuhan. “When CoT Surface Features Transfer.” May 4, 2026. https://yuhanchi.github.io/writing/code-not-text/

Or use the following BibTeX entry:

@misc{chi2026cot,
  author = {Chi, Yuhan},
  title = {{When CoT Surface Features Transfer}},
  year = {2026},
  month = may,
  howpublished = {Blog post},
  url = {https://yuhanchi.github.io/writing/code-not-text/}
}

Symbol	Meaning
\(s,\ a\)	state and action (for robot policies, often a chunk of future actions)
\(w \in \mathbb{R}^{d}\)	noise fed to the generator; for flow policies, \(d\) equals the action dimension
\(v_\theta,\ K\)	velocity field of the pretrained flow, and the number of sampler steps
\(\psi_\theta(s, w)\)	the decoder: the deterministic sampler that maps noise \(w\) to action \(a\)
\(f_{\#}\mu\)	pushforward: the distribution of \(f(x)\) when \(x \sim \mu\)
\(\pi_{\text{base}}(\cdot \mid s)\)	the pretrained policy, \(\psi_\theta(s, \cdot)_{\#}\, \mathcal{N}(0, I)\)
\(q_\phi(w \mid s)\)	the noise proposer (encoder), usually a state-conditioned Gaussian
\(\pi_\phi(\cdot \mid s)\)	the steered policy, \(\psi_\theta(s, \cdot)_{\#}\, q_\phi(\cdot \mid s)\)
\(Q(s, a)\)	an action-value estimate (critic)
\(\beta > 0\)	temperature: weight of the KL penalty that keeps \(q_\phi\) near \(\mathcal{N}(0, I)\)
\(q^\star,\ \pi^\star\)	the optimal tilted noise distribution and policy, defined in Section 2

VAE	Noise-space RL
observed input \(x\)	desired optimality, with state \(s\) as context
encoder \(q_\phi(z \mid x)\)	noise proposer \(q_\phi(w \mid s)\)
prior \(p(z) = \mathcal{N}(0, I)\)	the Gaussian the flow was trained from
decoder \(p_\theta(x \mid z)\)	frozen flow \(\psi_\theta(s, w)\)
reconstruction \(\log p_\theta(x \mid z)\)	value \(Q(s, \psi_\theta(s, w)) / \beta\)
KL to the prior	KL to \(\mathcal{N}(0, I)\): penalize departure from the training prior
decoder trained jointly	decoder pretrained by imitation, usually frozen

Yuhan Chi

Choose the Noise, Keep the Flow

1. Setup and notation

2. Reading it as a VAE

3. Where RL takes effect: carrying value into noise space

4. A two-dimensional experiment

5. A map of the family

6. What the VAE view explains

7. Open questions

References

What Is Luck?

Knowing Is Not Enough

Proofs as Observations

What happens after an AI produces a proof?

Proofs as observations

Verification and understanding are different achievements

Some concerns

Choice and Alignment

Alternatives and individuals

Preference profiles and society

From a profile to a social ranking

Arrow’s requirements

When social choice becomes alignment

A dictator always exists

Back to real alignment

What RLHF gives up: independence

What Constitutional AI gives up: universality

Personalization and minimal liberalism

Randomization and strategyproofness

Summary

My Visit to Professor Ziad Obermeyer at UC Berkeley

Meeting with Professor Ziad Obermeyer (School of Public Health)

Takeaways

Floating Point Notes in Computers

1. The Core Conflict: The Finite Cannot Represent the Infinite

2. Binary and the Need for Exponents

3. Floating-Point Memory Layout (Bit Distribution)

4. Engineering Trade-offs: Implicit Bit, Gradual Underflow, and Biased Exponent

Trade-off 1: The Implicit Leading Bit (Efficiency)

Trade-off 2: Biased Exponent (Fast Comparison)

5. Read the Bits Directly: Extreme Values in 32-bit Single Precision

6. Special Numbers: 0, Inf, NaN

7. Key Examples to Test Your Understanding

8. How Does the Computer Actually Calculate?

Addition (+) and Subtraction (-)

Multiplication (*) and Division (/)

Exponentiation (\(e^x\)) and Logarithm (\(\ln x\))

Power (\(x^y\))

Trigonometric Functions (\(\sin\), \(\cos\))

9. Epilogue: Two’s Complement and Hexadecimal

Why do Integers use Two’s Complement?

Why do we always read memory in Hexadecimal (Hex)?

CARLA Simulator on WSL2: Resolving the Docker Vulkan Trap

1. System & Hardware Specifications

2. The Working Solution: Windows Server + WSL2 Client Hybrid Architecture

Installation Steps

Step 1: Windows Side Setup

Step 2: WSL2 Side Setup

Step 3: Network Configuration (Mirrored Mode)

3. Running the Simulation

Step 1: Start the CARLA Server (Windows)

Step 2: Run Python Client Scripts (WSL2)

1. Connection Test (test_connection.py)

2. Spawn and Control Vehicle (spawn_vehicle.py)

4. The Failed Path: Running CARLA inside WSL2 Docker (Troubleshooting Record)

Attempted Run Configurations

Attempt 1: Basic GPU-enabled Container

Attempt 2: Mount Host Vulkan ICD File

Attempt 3: Using the Official --runtime=nvidia Flag

Attempt 4: Forcing OpenGL Mode

Root Cause Analysis

1. The nvidia_icd.json Directory Bug

2. Missing Vulkan Mapping Libraries in WSL2

3. Software Renderer Fallback

5. Summary of Tips & Troubleshooting

6. Useful Resources

When CoT Surface Features Transfer

Question

Setup

Main Result

Addition (`+`) and Subtraction (`-`)

Multiplication (`*`) and Division (`/`)

1. Connection Test (`test_connection.py`)

2. Spawn and Control Vehicle (`spawn_vehicle.py`)

Attempt 3: Using the Official `--runtime=nvidia` Flag

1. The `nvidia_icd.json` Directory Bug