糖心视频

Researchers from more than 50 international institutions have launched , a collaborative open-source platform to benchmark, improve, and run competitions for computational methods in single-cell genomics.

Co-led by Helmholtz Munich and Yale University, the initiative aims to standardize evaluations, foster reproducibility, and accelerate progress toward open challenges in this fast-moving field. An article introducing the platform has been in Nature Biotechnology.

A common language for a complex field

Single-cell genomics allows scientists to analyze individual cells at unprecedented resolution, revealing how they function, interact, and contribute to health and disease. But as the field has grown, so has the number of computational tools鈥攏ow numbering in the thousands鈥攄esigned to process and interpret this complex data.

This rapid growth presents a major challenge: how can researchers identify the most suitable tool, or determine the best combination of processing steps to achieve a specific analytical goal? Many tools are specialized, and evaluating their performance is challenging due to the limited availability of datasets with known, accurate outcomes (so-called ground truth).

As a result, researchers often turn to large-scale benchmarking studies. However, these studies can be inconsistent, quickly become outdated, and often make comparisons difficult to interpret鈥攎aking it challenging to identify the best method for a given task.

"We need a common language to measure what works鈥攁nd what doesn't鈥攖hat can stand the test of time," says Prof. Fabian Theis, Director of the Computational Health Center at Helmholtz Munich and Professor at the Technical University of Munich. "With Open Problems, we're introducing a reproducible, living, and transparent framework to guide tool development and evaluation鈥攐ne that the community can actively shape and use."

Transparent, reproducible, and community-driven

Open Problems currently includes 81 public datasets and tests 171 methods across 12 core tasks in single-cell analysis. Each method is evaluated using a suite of metrics鈥攓uantitative measures that show how well a method performs on a specific task. These metrics include accuracy, scalability, and robustness, among others, and are chosen based on the goals of each task. In total, 37 different metrics are used across the platform, with each task using the most relevant ones.

All evaluations run automatically in the cloud and follow standardized procedures to ensure the results are fully reproducible. Researchers can see how each method performs, explore the underlying code, and suggest improvements. To remain relevant and impactful over the long term, the platform is designed to be open to contributions: scientists can propose new tasks, add their own methods, join regular community calls, and take part in collaborative hackathons to help shape the future of the project.

Real-world benefits

By comparing tools side by side, Open Problems helps researchers identify the most effective methods for their specific scientific questions and often challenges established assumptions in the process.

As Dr. Smita Krishnaswamy, Associate Professor of Computer Science and Genetics at Yale University, explains, "We found that looking at overall patterns of gene activity gives more accurate results than focusing on individual genes when studying how cells communicate. And for some tasks, like identifying cell types across different datasets, a simple statistical model can actually outperform complex AI methods, making the analysis both faster and more efficient for many researchers."

The platform also powers major machine learning competitions, including the NeurIPS multimodal integration challenges. These global contests bring together experts in biology and artificial intelligence to solve real-world problems using common datasets and evaluation standards.

"Open Problems lowers the barrier for AI researchers outside biology to contribute to genomics," says Dr. Malte L眉cken, who co-led the project. "It's a blueprint for interdisciplinary innovation."

All code and results are openly available under a CC-BY license on .