What did the Virtual Cell predict? Target genes and evaluation shortcuts

Daiki Kumakura · 2026-10-11 · Blog · 1.0

A Japanese review of perturbation forecasting, target-gene masking and recent benchmarks, with a reproducible synthetic example distinguishing evaluation shortcuts from downstream prediction. The example does not establish real-data performance or the superiority of pairwise masking.

Literature checked on October 6, 2026. Reported research is distinguished from the synthetic calculations created for this article.

A model predicts cells after a gene is suppressed and correctly predicts that gene's lower expression. Does that alone establish understanding of the cellular response?

Eric Kernfeld's December 2025 blog post starts from this question to reconsider evaluation of Virtual Cell models, which computationally predict post-intervention responses. Here I read recent perturbation studies and evaluation specifications to determine what their conclusions support. The focus is CRISPRi transcriptional responses and retrieval of candidates from mean responses, rather than every aspect of Virtual Cell evaluation.

My judgment is that predicting the target gene's direct change should be evaluated separately from predicting other genes' responses. Retrieval metrics should also avoid removing only the correct target's direct effect while retaining direct effects of incorrect candidates' targets. These concerns do not invalidate all single-cell foundation models.

Cell representations and post-intervention prediction

Perturb-seq combines genetic intervention with single-cell RNA-seq to measure post-intervention transcriptional states. Extending predictions to untested targets or backgrounds could help choose experiments, one attraction of the Virtual Cell idea.

Single-cell foundation models (scFMs) pretrain on large cellular datasets and reuse representations across analyses. Geneformer reported transfer learning from transcriptional data; scFoundation reported large-scale pretraining and multiple downstream tasks. Good state representations and accurate expression prediction after unseen interventions nevertheless require separate validation (Geneformer; scFoundation).

Evaluation first needs to define what remains unseen.

This is my organizational framework; each paper's actual split still needs checking. Wei and colleagues' broad benchmark likewise separates unseen perturbations and unseen backgrounds. It appeared online in December 2025 and in a 2026 volume (Wei et al.).

Randomly splitting cells so that the same target and experiment appear on both sides does not test unseen-target generalization. Where possible, evaluation experiments should also be audited for overlap with pretraining. That differs from masking a target gene.

What target downregulation establishes

CRISPRi is designed to suppress target transcription. If that direct effect dominates, evaluation emphasizing strongly changed genes can reward a prediction that merely lowers the target.

Csendes and colleagues compared scGPT and scFoundation with simple references and also excluded CRISPR targets from Top-20 DE evaluation, the top 20 genes selected by differential-expression testing. With targets included, scGPT outperformed Train Mean, which supplies the same training-perturbation average to every test perturbation. But the target's own change contributes to that evaluation. The question is which genes account for the score improvement (Csendes et al.).

Wong and colleagues also report the importance of suitable simple baselines and corrections to existing benchmark data. I use only what was verifiable from their abstract (Wong et al.).

Calling the target response nonbiological would go too far. Knockdown efficiency, residual expression and guide differences can themselves be legitimate prediction objectives. The distinction is predicting intervention efficacy versus predicting the response under that efficacy.

CRISPR knockout also does not equate loss of function with disappearance of mRNA. Setting measured expression uniformly to zero when mRNA can remain is a modeling assumption tied to a purpose, rather than an observation. The PEREGGRN paper notes that knockout mRNA can persist or increase (Kernfeld et al.).

Different problems should not all be called leakage

Let I be the intervention label, Z the post-intervention target expression, X other expression and c the cellular background. Predicting the full response from the label could target

\[ p(Z,X\mid I,c) \]

If achieved target suppression is already known and downstream prediction is wanted, the target becomes

\[ p(X\mid Z,I,c) \]

The latter conditions on more information.

Under PEREGGRN's closest-control setting, prediction replaces target expression with its measured post-intervention value. This illustrates conditioning on target change, but that measured value may not be available when prospectively predicting an unknown experiment (PEREGGRN methods).

Supplying the target's name is normally a legitimate input, rather than leakage in perturbation prediction. When using “target-gene leakage,” specify whether the issue is unavailable input information or an evaluation shortcut.

A shortcut from removing only the query target

Retrieval evaluation matches a prediction to observed candidates and ranks the correct match. For perturbation i, prediction \(\hat x_i\) is compared with true \(x_i\) and other perturbations \(x_j\), rewarding a closer true match.

The problematic row mask removes only query target i from every comparison. Coordinates are shared within a query, yet correct and competing targets are treated asymmetrically:

\[ d_{-i}(\hat x_i,x_i),\qquad d_{-i}(\hat x_i,x_j). \]

Candidate j retains its strong decrease at target j. Artificially raising other targets in the prediction can push incorrect candidates farther away.

Arc's public specification reports pds_cosine values of 0.798, 0.757 and 0.761 for such shortcut predictions across three validation contexts, against baselines of 0.530, 0.528 and 0.510. These are Arc's reported results, not my recalculation from real data (cell-eval2 metrics reference).

The competition brief dated August 19, 2026 defines pds_cosine using a fixed mask of all panel targets, so surviving coordinates do not depend on the candidate. This does not apply to all six metrics: the other five remove each perturbation's own target (VCC2026 metrics brief).

The document date should not be assumed to be an operational change date. A raw PDS reference of 0.5 also differs from zero on a competition score normalized with baseline and replicate controls.

A three-perturbation example

Before examining large datasets, I constructed a minimal example: suppress A, B or C and measure downstream genes D and E. These synthetic changes relative to nontargeting control are not experimental data.

A shortcut-only prediction sets D and E to zero and other targets to +10. For A, that is \((0,10,10,0,0)\), containing no downstream-response information.

Figure 1. Author-created synthetic example. Blue is negative, brown positive and gray zero. Color indicates sign only; cell values show magnitude. Only other panel targets are raised, with no prediction of D or E. Units are arbitrary.

With cosine distance after removing A, the prediction's inner product with the true response is zero, giving distance one. Its inner product with the B response includes \(10\times(-5)\), giving a distance above one; C behaves similarly. The result appears to identify the truth by pushing wrong candidates away.

I compared row masking, panel masking of all targets, and pairwise masking of both targets for each comparison. Ties receive 0.5; zero-vector cosine distance is defined as one; distance differences within \(10^{-12}\) count as ties. For row and panel masking, mean pairwise wins correspond to a tie-adjusted normalized rank. This does not fully reproduce an old implementation's preprocessing or tie handling.

Figure 2. Synthetic calculation, averaged across three perturbations. Oracles use the true answers as positive controls; they are not trained predictors. All bars share a 0–1 scale.

The example establishes that outputs without downstream prediction can achieve perfect scores under a particular evaluation rule. It establishes neither actual model rankings, real-data effect sizes nor superiority of pairwise masking. Code, result CSV and a reproduction ZIP with inputs, dependencies and figures are available.

Panel-wide and pairwise target removal

Panel masking has the benefit of fixed coordinates. But if B responds when A is suppressed, that response also disappears whenever B is another panel target. Potentially relevant information is lost.

Kernfeld's appendix proposes removing both targets for each correct-versus-competing pair i,j. I add explicit tie handling:

\[ S_i=\frac{1}{N-1}\sum_{j\ne i} \left[ \mathbf 1\{a_{ij}<b_{ij}\} +\frac12\mathbf 1\{a_{ij}=b_{ij}\} \right], \]

\[ a_{ij}=d_{-(i,j)}(\hat x_i,x_i),\qquad b_{ij}=d_{-(i,j)}(\hat x_i,x_j). \]

This presentation and interpretation build on the original proposal. Each pair uses the same genes for both distances, then averages wins. That differs from ranking a single list of distances calculated using different coordinates for each candidate.

An AUC-like win fraction is an intuition, not equivalence to ordinary ROC-AUC or ranking by one fixed distance. Pair-dependent coordinates need not produce a common distance ordering across candidates.

Third-party targets irrelevant to a pair can remain. That is not always beneficial: validation must consider retained direct effects, gene dependence, multitarget perturbations, weak responses and cosine's norm dependence. Blocking the same spike in one toy example does not establish an optimal solution.

What recent studies support

Ahlmann-Eltze and colleagues found no deep-learning advantage over simple linear baselines in the gene-perturbation settings examined. Systema showed the need to distinguish shared systematic variation from perturbation-specific responses. Both motivate retaining simple references (Ahlmann-Eltze et al.; Systema).

Miller and colleagues in October 2026 assessed 18 metrics across 14 datasets by whether they distinguish positive and negative controls. Multiple deep models exceeded baselines on sensitive metrics. This checks a metric's ability to detect useful prediction, rather than calibration of predicted probabilities. The paper explicitly does not address unseen cellular backgrounds (Miller et al.).

These views do not simply negate each other. All-gene mean error can dilute weak responses, while metrics restricted to strongly changing genes can emphasize direct target effects. Intended signal, positive-control sensitivity and susceptibility to shortcuts each require examination.

Recent models also solve different tasks.

Capability summaries use UCE, State, Stack and Miller et al.. The last column is my interpretation. Stack is treated as a preprint in this review.

PIE was released on October 5, 2026. Its public README and announcement describe using knowledge sources, target-context controls and training-data perturbation responses. Predicting an unknown condition does not mean predicting with no observations at all. I describe only the design verifiable from public materials, without claiming detailed performance auditing or independent validation (public implementation; author announcement).

On the data side, Tahoe-100M in September 2026 reported a large drug-perturbation atlas spanning 50 cancer cell lines and 1,100 drug–dose conditions. Drug inhibition of a protein and CRISPRi suppression of its mRNA are different interventions; the target masking here does not automatically transfer to drug perturbations (Tahoe-100M).

A 2026 preprint by Nicol and colleagues reports inflated correlations when the same estimated control is subtracted from predictions and observations, introducing shared estimation error (Nicol et al.). For intuition, suppose estimation errors \(\epsilon_p,\epsilon_t,\epsilon_c\) are independent. An algebraic illustration is

\[ \operatorname{Cov}(\epsilon_p-\epsilon_c,\epsilon_t-\epsilon_c) =\operatorname{Var}(\epsilon_c) \]

Subtracting a shared control introduces a common component even with independent prediction and observation errors. This does not assert independence for general cellular data. Target removal alone does not solve these other evaluation problems.

A metric name does not specify evaluation conditions

Target masks, gene sets, preprocessing and tie handling change a score's meaning even when its name is unchanged. In the toy example, target-only and shared predictions also receive different scores under different rules.

The scPertEval preprint separates evaluation into representation, metric, score transformation and reporting strategy. This makes clear why scores with different preprocessing and normalization cannot be compared by metric name alone (scPertEval).

My panel-versus-pairwise comparison is limited to equations and artificial examples. It neither validates real predictive performance or population distributions nor turns score differences into evidence of downstream mechanism understanding.

Direct target effects and downstream responses

Kernfeld asks whether correctly predicting target downregulation supports understanding of downstream response. Arc's specifications additionally show that asymmetric target handling creates a real retrieval shortcut.

The synthetic example reproduces that mechanism. It does not establish that scFMs are useless or that pairwise masking solves evaluation.

I would rather read what was known before prediction, what was held out, which responses were predicted and which controls were exceeded than a model's name or size. Those conditions provide a common basis for discussing Virtual Cell progress and limits.

Sources

Links in the text identify primary sources. No extended quotations beyond the original blog or figures from existing papers are reproduced.

1. Kernfeld E. Blog, 2025-12-05: Target-gene prediction and retrieval critique.
2. Theodoris CV et al. Nature, 2023. 10.1038/s41586-023-06139-9.
3. Hao M et al. Nature Methods, 2024. 10.1038/s41592-024-02305-7.
4. Csendes G et al. Benchmarking foundation cell models for post-perturbation RNA-seq prediction. BMC Genomics, 2025. 10.1186/s12864-025-11600-2.
5. Kernfeld E et al. A comparison of computational methods for expression forecasting. Genome Biology, 2025. 10.1186/s13059-025-03840-y.
6. Ahlmann-Eltze C et al. Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines. Nature Methods, 2025. 10.1038/s41592-025-02772-6.
7. Viñas Torné R et al. Systema. published online 2025-08-25. 10.1038/s41587-025-02777-8.
8. Wei Z et al. Nature Methods 23, 2026 (published online 2025-12-11). 10.1038/s41592-025-02980-0.
9. Miller HE et al. Deep learning perturbation models can outperform baselines on calibrated metrics. Nature Biotechnology, 2026. 10.1038/s41587-026-03307-w.
10. Rosen Y et al. UCE. Nature, 2026. 10.1038/s41586-026-10689-z.
11. Adduri AK et al. State. Cell, 2026. 10.1016/j.cell.2026.07.052.
12. Dong M et al. Stack. bioRxiv, 2026 preprint. 10.64898/2026.01.09.698608.
13. Nicol PB et al. bioRxiv, 2026 preprint. 10.64898/2026.05.07.723486.
14. Arc Institute. cell-eval2 metrics reference, competition brief dated 2026-08-19. Links are pinned to checked commit 0ce25cf. The competition brief refers to implementation 0.15.0 and rule version 3.
15. Verma R et al. PIE. bioRxiv, 2026 preprint. 10.64898/2026.10.02.756297. The author announcement and public README were checked for this article. Abstract was also consulted. The full paper was not examined in detail.
16. Zhang J et al. Tahoe-100M. Cell, 2026. 10.1016/j.cell.2026.08.035.
17. Wong DR, Hill AS, Moccia R. Bioinformatics, 2025. 10.1093/bioinformatics/btaf317. The abstract was checked on PubMed.
18. Schäfer PSL et al. scPertEval. bioRxiv, 2026 preprint. 10.64898/2026.07.23.740433.

Reproduction and limitations

Extract the reproduction ZIP and run the following from its directory with Python 3.11 or later. Tested NumPy and Pillow versions are pinned in its requirements.txt. Fonts can change figure appearance across environments without changing calculated values.

python -m pip install -r requirements.txt
python toy_metrics.py

This generates toy-results.csv, toy-data.json and Figures 1–2. Artificial mean effects for three perturbations and five genes are supplied directly. No real data were retrieved, no scFM trained, no single-cell distribution reproduced and no Arc competition pipeline rerun.

Checks cover two positive controls and baseline scores for 100 shared predictions. Pairwise masking is examined only through this example and mathematical properties; real-data comparative validation remains outstanding.