MRS Meetings and Events


DS06.07.07 2023 MRS Fall Meeting

JARVIS-Leaderboard—A Comprehensive Benchmark of Materials Design Methods

When and Where

Nov 29, 2023
10:00am - 10:30am

Sheraton, Second Floor, Back Bay A



Kamal Choudhary1

National Institute of Standards and Technology1


Kamal Choudhary1

National Institute of Standards and Technology1
Reproducibility and validation are major hurdles for scientific development across many fields. Materials science in particular encompasses a variety of experimental and theoretical approaches that require careful benchmarking. Leaderboard efforts have been developed previously to mitigate these issues, however, a comprehensive comparison and benchmarking on an integrated platform with multiple data-modalities with both perfect and defect materials data is still lacking. This talk introduces the JARVIS-Leaderboard, an open-source and community-driven platform that facilitates benchmarking and enhances reproducibility. The platform allows users to set up benchmarks with custom tasks and enables contributions in the form of dataset, code, and meta-data submissions. We cover the following materials design categories: Artificial Intelligence (AI), Electronic Structure (ES), Force-fields (FF), Quantum Computation (QC), and Experiments (EXP). For AI, we cover several types of input data, including atomic structures, atomistic images, spectra, and texts. For ES, we consider multiple ES approaches, software packages, pseudopotentials, materials, and properties, comparing results to experiment. For FF, we compare multiple approaches for material property predictions. For QC, we benchmark Hamiltonian simulation using various quantum algorithms and circuits. Finally, for experiments, we use the round-robin approach to establish benchmarks. Currently, there are 1008 contributions to 225 benchmarks using over 100 different methods, and the leaderboard is continuously expanding. The JARVIS-Leaderboard is available at the website: .



Symposium Organizers

Mathieu Bauchy, University of California, Los Angeles
Ekin Dogus Cubuk, Google
Grace Gu, University of California, Berkeley
N M Anoop Krishnan, Indian Institute of Technology Delhi

Symposium Support

Patterns and Matter | Cell Press

Session Chairs

Mathieu Bauchy
Binquan Luan

In this Session

Publishing Alliance

MRS publishes with Springer Nature