Apparent Cross-Language Invariance Reflects Shared Identifiers in Parallel Programs

Cross-language probe transfer is often taken as evidence that a code model represents variable roles independently of programming language. We show that this conclusion depends on whether the readout includes the identifier tokens, which parallel programs systematically preserve. A span-pooled residual probe on Qwen2.5-Coder-1.5B looks flat across the language contrast (−0.012, 95% problem-clustered interval [−0.034, 0.020]), while a model-free surface classifier that masks the identifier loses 0.126 macro-F1 on the same contrast. Excluding the occurrence from the probe readout changes the boundary to −0.060 [−0.088, −0.030], a shift of −0.048 [−0.088, −0.016], in line with context-matched untrained controls. The surface baseline outperforms the context-pooled probe in 14 of 18 matched cells. StarCoder2-7B reproduces the reversal more sharply, moving from −0.014 span-pooled to −0.147 [−0.186, −0.092] occurrence-excluded, with a trained-minus-untrained boundary difference of −0.068 [−0.114, −0.009] that excludes zero. The original invariance is therefore not identified independently of the readout, while the restricted readout still contains training-dependent role information.

A variable can play the same role in programs written in different languages. An accumulator collects a running result, an iterator advances through a sequence, and an index key selects an element. If a probe trained to recognize such roles in one language transfers to another, it may appear to have recovered a language-independent property of the model's representations.

Parallel programs introduce a complication. They often preserve variable names as well as behavior. A probe that reads the hidden states at a variable's own tokens may therefore benefit from information that survives translation before any claim about abstract role representations is warranted.

Our study examines this measurement problem in Qwen2.5-Coder-1.5B and StarCoder2-7B. We compare probes with a classifier that uses only surface text, then change which token positions supply the probe's input. This follows the principle behind control tasks for interpreting probes and model-free baselines in code probing. High predictive accuracy alone does not establish what information supports the prediction or whether the model uses that information during generation.

Parallel programs

We use XLCoST, restricting the corpus to 869 problems with implementations in Python, JavaScript, and PHP. These are the only three languages in the dataset with at least 500 shared problems for every pair. A fixed split by problem keeps occurrences from the same problem together. Extracting the three variable roles then determines which problems contribute usable examples to each comparison.

The central contrast compares transfers involving Python with transfers between JavaScript and PHP. We define the cross-language boundary as mean macro-F1 on the four directed Python transfers minus mean macro-F1 on the two JavaScript–PHP transfers. A negative value indicates lower performance on transfers involving Python. Macro-F1 averages the class-specific F1 scores. Reported uncertainty resamples whole problems rather than treating their occurrences as independent observations.

Readouts and controls

The span-pooled probe fits logistic regression to residual activations averaged over the target occurrence's own tokens. These tokens include the identifier. The occurrence-excluded probe instead averages the 16 tokens on either side of the occurrence, leaving out its own tokens. The source program and model weights remain unchanged. Only the positions read by the probe change.

The surface classifier uses character n-grams from a local window after replacing every occurrence of the target variable's name with a placeholder. Whitespace is removed to avoid a formatting difference between the language corpora. The same fixed feature set is used across comparisons.

We also evaluate randomly initialized versions of each model architecture with the same tokenizer and readout. Prior probing work shows that untrained encoders can preserve information useful to a probe. Our controls estimate how much transfer is available without trained weights. Probe layers are selected using source-language validation data.

The name-masked surface classifier reaches 0.929 macro-F1 on JavaScript–PHP transfers and 0.804 on transfers involving Python. The mean difference across cells is −0.126. A separate pooled estimate with problem-clustered uncertainty is −0.114, with a 95% interval of [−0.147, −0.081]. These are different summaries of the same comparison.

Most of the surface contrast comes from the iterator role. Its difference is −0.249, compared with −0.064 for accumulator and index-key roles together. This is also the role with the weakest cross-language label comparability. Python uses an abstract syntax tree extractor, while JavaScript and PHP use regular expressions, and some cells contain only ten minority examples.

The result establishes a descriptive contrast in this extracted sample. It does not isolate an effect of language family. It also shows that substantial role information can be recovered without model activations or access to the target identifier.

Including or excluding the occurrence

For Qwen2.5-Coder-1.5B, the span-pooled probe has a nearly flat cross-language boundary of −0.012. Excluding the occurrence changes it to −0.060. Mean cross-language transfer falls from 0.844 to 0.589, while the context-matched untrained model reaches 0.485.

StarCoder2-7B shows the same qualitative change, with a larger shift. Its boundary moves from −0.014 with span pooling to −0.147 when the occurrence is excluded.

The graph below compares both models with their untrained controls. In both models, the appearance of invariance depends on the readout.

What training contributes

The occurrence-excluded Qwen boundary is similar to its matched untrained control. The trained-minus-untrained boundary difference is +0.003 [−0.027, 0.036]. Training nonetheless raises transfer performance on both sides of the contrast, by +0.102 on JavaScript–PHP transfers and +0.106 on Python transfers against the first untrained initialization.

StarCoder2 differs. Its occurrence-excluded boundary is reliably steeper than the untrained boundary, with a difference of −0.068 [−0.114, −0.009]. This limits the claim that the results can be explained entirely by an untrained representation. The restricted readout still contains training-dependent role information, but that information does not produce a language-independent transfer boundary.

The surface classifier also outperforms the Qwen context-pooled probe in 14 of 18 matched cells. Because the surface classifier masks names while the model processes the original program, this comparison demonstrates surface sufficiency on the prediction task. It does not decompose the model's signal into lexical and semantic components.

The evidence concerns how a representation is measured. A readout over an occurrence exposes information concentrated at those positions, including lexical regularities preserved by parallel implementations. An untrained control measures the contribution of training under that readout. It does not establish that the readout isolates the intended concept.

Prior work on programming-language alignment finds Python the least aligned of these three languages on parallel programs. Our span-pooled result initially appears different, but the studies measure different quantities. Excluding the occurrence moves the probe contrast toward that earlier finding without establishing a shared mechanism.

A stronger test of the identifier explanation would replace identifiers while holding the readout positions fixed, or repeat the comparison on parallel programs without shared naming conventions. If either preserved the original flat transfer pattern, it would weaken that explanation.

The practical implication is to report what the probe can read alongside what it predicts. A model-free comparator, a readout-matched untrained control, and a direct readout ablation help distinguish evidence of transferable labels from evidence of a language-independent representation.

The study covers two models, three languages, three roles, and one context width. Python supplies the only broad language contrast. Role coverage varies after extraction, iterator extraction differs across languages, and no multilingual manual audit quantifies the resulting label error.

The untrained controls use two probe seeds. Qwen has two weight initializations, while StarCoder2 has one. Qwen's untrained boundary varies across initializations even when its overall transfer level remains similar, so the cross-model comparison should not be treated as an exhaustive account of initialization effects.

Finally, occurrence exclusion changes pooling support and token positions together. Nearby uses can still carry correlated information, and the model continues to process unmasked source code. The results therefore establish readout dependence in role decodability. They do not isolate identifier identity as the sole cause or show how either model uses role information to generate code.