Skip to content

JARVIS Leaderboard (Introduction)

This project provides benchmark-performances of various methods for materials science applications using the datasets available in JARVIS-Tools databases. Some of the categories are: Artificial Intelligence (AI), Electronic Structure (ES), Force-field (FF), Quantum Computation (QC) and Experiments (EXP). There are a variety of properties included in the benchmark. In addition to prediction results, we attempt to capture the underlyig software, hardware and instrumental frameworks to enhance reproducibility. This project is a part of the NIST-JARVIS infrastructure.

Usually, codes are kept in GitHub/GitLab etc., data is kept in Zenodo/FigShare/NIST Materials data etc., we recommend keeping the benchmarks in the JARVIS-Leaderboard to enhance reproducibility and transparency.

  • Number of contributors: 20
  • Number of methods: 159
  • Number of benchmarks: 296
  • Number of contributions: 1476
  • Number of datapoints: 8731126

Summary table

Category/Sub-cat.SinglePropertyPredictionSinglePropertyClassMLFFTextClassTokenClassTextSummaryTextGenAtomGenImageClassSpectraEigenSolver
AI4412111918113321-
ES776--------10-
FF50----------
QC----------6
EXP7--------18-

Quick start using GoogleColab notebook examples

  1. Analyzing_data_in_the_JARVIS_Leaderboard.ipynb
  2. Upload_benchmark_to_jarvis_leaderboard.ipynb
  3. alignn_jarvis_leaderboard.ipynb
  4. kgcnn_jarvis_leaderboard.ipynb
  5. MatMiner_on_JARVIS_DFT.ipynb
  6. Train ALIGNN-FF for FCC Copper
  7. Train ALLEGRO for Silicon
  8. AtomVision_Image_Classification.ipynb
  9. Text-generation ChemNLP/HuggingFace.ipynb
  10. Inverse design of superconductors
  11. QuantumComputation for FCC Aluminum.ipynb
  12. Pretrained CHGNet Prediction.ipynb
  13. Pretrained OpenCatalystProject Model.ipynb
  14. GPAW colab.ipynb
  15. Quantum Espresso colab.ipynb
  16. ocp20_load_pretrained_models.ipynb

For additional notebooks, visit JARVIS-Tools-Notebooks collections with more than 60 colab notebooks.

Terminologies used in this project

  • Categories: are of following types Artificial Intelligence (AI), Electronic Structure (ES), Force-field (FF), Quantum Computation (QC) and Experiments (EXP). Each of these categories are divided into sub-categories. These sub-categories include single-property-prediction, single-property-classification, machine-learning force-fields, text-classification, text-token classification, text-generation, image classification, image-segmentation, image-generation, spectra-prediction, and eigensolver. These categories and sub-categories are highly flexible and new entries can be easily added.

  • Sub-categories: include 1) SinglePropertyPrediction (where the output of a model/experiment is one single number for an entry), 2) SinglePropertyClass (where the output is class-ids, e.g., 0,1,.. instead of floating values), 3) ImageClass (for multi-class image classification), 4) textClass (for multi-label text classification), 5) MLFF (machine learning force-field), 6) Spectra (for multi-value data) and 7) EigenSolver (for Hamiltonian simulation).

  • Benchmarks: are ground truth data used to calculate metrics for each specific task (e.g a json.zip file).

  • Methods: are a set of precise specifications for evaluation against a benchmark. For example, within ES category, DFT with VASP-GGA-PAW-PBE are specifications, hence a method. Similarly, within AI category, descriptor/feature based models with MatMiner-chemical features and LightGBM software are specifications, hence a method.

  • Contributions: are individual data in form of csv.zip files for each benchmark and specific method. Each contribution has six components: method (e.g. AI), category (e.g. SinglePropertyPrediction), property (e.g. formation energy), dataset (e.g. dft_3d), data-split (e.g. test), metric (e.g. mae).

List of benchmarks

Click on the entries in the Benchmark column. You'll be able to see methods available for each benchmark, CSV file submitted for the contribution, JSON file for ground trutch data, run.sh script for running the method and Info for metadata associated with the method.

CategorySub-categoryBenchmarkMethodMetricScoreTeamDatasetSize
AIAtomGendft_3d_Tc_superconcdvae_modelRMSE0.3557CDVAEdft_3d1056
AIAtomGencarbon24_energy_per_atomcdvae_modelRMSE0.3581CDVAEcarbon2410153
AIImageClassstem_2d_image_bravais_classdensenet_modelACC0.8304JARVISstem_2d_image9150
AIMLFFalignn_ff_db_energyalignnff_wt0.1MAE0.0342JARVISalignn_ff_db307111
AIMLFFmlearn_Cu_energyalignnff_mlearn_onlyMAE1.0104JARVISmlearn_Cu293
AIMLFFmlearn_Ge_energyalignnff_fmult_mlearn_onlyMAE0.9439JARVISmlearn_Ge253
AIMLFFmlearn_Li_energyalignnff_fmult_mlearn_onlyMAE0.2735JARVISmlearn_Li270
AIMLFFmlearn_Mo_energyalignnff_fmult_mlearn_onlyMAE1.1066JARVISmlearn_Mo217
AIMLFFmlearn_Ni_energyalignnff_mlearn_onlyMAE1.4316JARVISmlearn_Ni294
AIMLFFmlearn_Si_energyalignnff_mlearn_onlyMAE1.0688JARVISmlearn_Si239
AIMLFFalignn_ff_db_forcesm3gnet_pretrainedMULTIMAE0.0887M3GNETalignn_ff_db307111
AIMLFFmlearn_Cu_forcesalignnff_fmult_mlearn_onlyMULTIMAE0.0357JARVISmlearn_Cu293
AIMLFFmlearn_Ge_forcesalignnff_fmult_mlearn_onlyMULTIMAE0.0726JARVISmlearn_Ge253
AIMLFFmlearn_Li_forcesalignnff_fmult_mlearn_onlyMULTIMAE0.0339JARVISmlearn_Li270
AIMLFFmlearn_Mo_forcesalignnff_fmult_mlearn_onlyMULTIMAE0.1229JARVISmlearn_Mo217
AIMLFFmlearn_Ni_forcesalignnff_fmult_mlearn_onlyMULTIMAE0.0448JARVISmlearn_Ni294
AIMLFFmlearn_Si_forcesalignnff_fmult_mlearn_onlyMULTIMAE0.0885JARVISmlearn_Si239
AIMLFFmlearn_Cu_stressesalignnff_fdMULTIMAE20.0515ALIGNNmlearn_Cu293
AIMLFFmlearn_Ge_stressesalignnff_fdMULTIMAE9.0436ALIGNNmlearn_Ge253
AIMLFFmlearn_Li_stressesalignnff_fdMULTIMAE2.6491ALIGNNmlearn_Li270
AIMLFFmlearn_Mo_stressesalignnff_mlearn_onlyMULTIMAE19.4086JARVISmlearn_Mo217
AIMLFFmlearn_Ni_stressesalignnff_fdMULTIMAE31.1771ALIGNNmlearn_Ni294
AIMLFFmlearn_Si_stressesalignnff_fmult_mlearn_onlyMULTIMAE11.5542JARVISmlearn_Si239
AISinglePropertyClassdft_3d_magmom_oszicarmatminer_xgboostACC0.9489UofTdft_3d52210
AISinglePropertyClassdft_3d_mbj_bandgapmatminer_xgboostACC0.9399UofTdft_3d18167
AISinglePropertyClassdft_3d_n_powerfactmatminer_xgboostACC0.8186UofTdft_3d23210
AISinglePropertyClassdft_3d_optb88vdw_bandgapmatminer_xgboostACC0.9361UofTdft_3d55713
AISinglePropertyClassdft_3d_p_Seebeckmatminer_xgboostACC0.9332UofTdft_3d23210
AISinglePropertyClassdft_3d_slmematminer_rfACC0.8422UofTdft_3d9062
AISinglePropertyClassdft_3d_spillagematminer_xgboostACC0.8364UofTdft_3d11375
AISinglePropertyPredictionsnumat_Band_gap_HSEalignn_modelMAE0.3765ALIGNNsnumat10386
AISinglePropertyPredictionqm9_std_jctc_Cvalignn_modelMAE0.0239ALIGNNqm9_std_jctc130829
AISinglePropertyPredictionqm9_std_jctc_Galignn_modelMAE0.0133ALIGNNqm9_std_jctc130829
AISinglePropertyPredictionqm9_std_jctc_Halignn_modelMAE0.014ALIGNNqm9_std_jctc130829
AISinglePropertyPredictionqm9_std_jctc_HOMOalignn_modelMAE0.0187ALIGNNqm9_std_jctc130829
AISinglePropertyPredictionhalide_peroskites_HSE_decomp_energyalignn_modelMAE0.0512ALIGNNhalide_peroskites229
AISinglePropertyPredictionhalide_peroskites_HSE_gapalignn_modelMAE0.1615ALIGNNhalide_peroskites229
AISinglePropertyPredictionqm9_std_jctc_LUMOalignn_modelMAE0.0175ALIGNNqm9_std_jctc130829
AISinglePropertyPredictionhalide_peroskites_PBE_decomp_energyalignn_modelMAE0.0477ALIGNNhalide_peroskites229
AISinglePropertyPredictionhalide_peroskites_PBE_gapalignn_modelMAE0.1223ALIGNNhalide_peroskites229
AISinglePropertyPredictionqm9_std_jctc_R2alignn_modelMAE0.6523ALIGNNqm9_std_jctc130829
AISinglePropertyPredictionhalide_peroskites_Ref_indalignn_modelMAE0.0233ALIGNNhalide_peroskites229
AISinglePropertyPredictionsupercon_chem_Tcmatminer_rfMAE4.8511UofTsupercon_chem16414
AISinglePropertyPredictionqm9_std_jctc_Ualignn_modelMAE0.0138ALIGNNqm9_std_jctc130829
AISinglePropertyPredictionqm9_std_jctc_U0alignn_modelMAE0.0146ALIGNNqm9_std_jctc130829
AISinglePropertyPredictionqm9_std_jctc_ZPVEalignn_modelMAE0.0019ALIGNNqm9_std_jctc130829
AISinglePropertyPredictionqm9_std_jctc_alphaalignn_modelMAE0.0557ALIGNNqm9_std_jctc130829
AISinglePropertyPredictiondft_3d_avg_elec_massalignn_modelMAE0.0853ALIGNNdft_3d17642
AISinglePropertyPredictiondft_3d_avg_hole_massalignn_modelMAE0.1239ALIGNNdft_3d17642
AISinglePropertyPredictionqmof_bandgapalignn_modelMAE0.2019ALIGNNqmof20424
AISinglePropertyPredictiondft_3d_bulk_modulus_kvkgcnn_coNGNMAE8.7022kgcnndft_3d19680
AISinglePropertyPredictiondft_3d_dfpt_piezo_max_dielectrickgcnn_coNGNMAE25.5553kgcnndft_3d4704
AISinglePropertyPredictiondft_3d_dfpt_piezo_max_dijkgcnn_coNGNMAE13.8868kgcnndft_3d3345
AISinglePropertyPredictionmegnet_e_formalignn_modelMAE0.0221ALIGNNmegnet69239
AISinglePropertyPredictionAGRA_CHO_eadalignn_modelMAE0.1062ALIGNNAGRA_CHO214
AISinglePropertyPredictionAGRA_CO_eadalignn_modelMAE0.1349ALIGNNAGRA_CO193
AISinglePropertyPredictionAGRA_COOH_eadalignn_modelMAE0.0485ALIGNNAGRA_COOH280
AISinglePropertyPredictionAGRA_O_eadalignn_modelMAE0.1134ALIGNNAGRA_O1000
AISinglePropertyPredictionAGRA_OH_eadalignn_modelMAE0.0762ALIGNNAGRA_OH875
AISinglePropertyPredictiontinnet_N_eadalignn_modelMAE0.0871ALIGNNtinnet_N327
AISinglePropertyPredictiontinnet_O_eadalignn_modelMAE0.3694ALIGNNtinnet_O745
AISinglePropertyPredictiontinnet_OH_eadalignn_modelMAE0.0882ALIGNNtinnet_OH746
AISinglePropertyPredictionvacancydb_efalignnff_wt0.1MAE0.8739JARVISvacancydb530
AISinglePropertyPredictionvacancydb_2D_efalignnff_wt0.1MAE1.0018JARVISvacancydb_2D72
AISinglePropertyPredictionvacancydb_elements_efalignnff_wt0.1MAE1.1992JARVISvacancydb_elements74
AISinglePropertyPredictionvacancydb_oxides_efalignnff_wt0.1MAE0.6091JARVISvacancydb_oxides65
AISinglePropertyPredictionvacancydb_oxides_train_test_efcrystal_feature_modelMAE0.7Crystal_Featuresvacancydb_oxides_train_test30
AISinglePropertyPredictiondft_3d_ehullkgcnn_coGNMAE0.0466kgcnndft_3d55364
AISinglePropertyPredictiondft_3d_encutkgcnn_coNGNMAE129.8266kgcnndft_3d55386
AISinglePropertyPredictionqe_tb_energy_per_atomkgcnn_coGNMAE0.0636kgcnnqe_tb829574
AISinglePropertyPredictiondft_3d_epsxkgcnn_coNGNMAE18.5738kgcnndft_3d44490
AISinglePropertyPredictiondft_3d_epsykgcnn_coNGNMAE18.5923kgcnndft_3d44490
AISinglePropertyPredictiondft_3d_epszkgcnn_coNGNMAE17.8104kgcnndft_3d44490
AISinglePropertyPredictiondft_3d_exfoliation_energymatminer_xgboostMAE40.887UofTdft_3d812
AISinglePropertyPredictionqe_tb_f_enpkgcnn_coGNMAE0.0956kgcnnqe_tb829574
AISinglePropertyPredictionqe_tb_final_energykgcnn_coGNMAE1.3185kgcnnqe_tb829574
AISinglePropertyPredictionmxene275_formation_energymxene_alignnMAE0.0561MXene_ALIGNNmxene275274
AISinglePropertyPredictiondft_3d_formation_energy_peratomkgcnn_coGNMAE0.0271kgcnndft_3d55713
AISinglePropertyPredictionssub_formula_energyElemNet2_TLMAE0.0924NorthWestern_Universityssub1726
AISinglePropertyPredictionqm9_std_jctc_gapalignn_modelMAE0.0345ALIGNNqm9_std_jctc130829
AISinglePropertyPredictionmegnet_gappbealignn_modelMAE0.218ALIGNNmegnet69239
AISinglePropertyPredictionqe_tb_indir_gapmatminer_rfMAE0.0278UofTqe_tb829574
AISinglePropertyPredictiondft_3d_kpoint_length_unitmatminer_xgboostMAE9.047UofTdft_3d55392
AISinglePropertyPredictionhmof_lcdmatminer_xgboostMAE0.4585UofThmof137651
AISinglePropertyPredictiondft_3d_magmom_oszicarkgcnn_coNGNMAE0.2437kgcnndft_3d52210
AISinglePropertyPredictionmag2d_chem_magnetic_momentElemNet2_TLMAE0.2997NorthWestern_Universitymag2d_chem226
AISinglePropertyPredictionhmof_max_co2_adspmatminer_xgboostMAE0.4622UofThmof137651
AISinglePropertyPredictiondft_3d_max_efgalignn_modelMAE19.1211ALIGNNdft_3d11865
AISinglePropertyPredictiondft_3d_mbj_bandgapkgcnn_coGNMAE0.264kgcnndft_3d18167
AISinglePropertyPredictiondft_3d_mepsxkgcnn_coNGNMAE23.3801kgcnndft_3d16809
AISinglePropertyPredictiondft_3d_mepsykgcnn_coNGNMAE23.3299kgcnndft_3d16809
AISinglePropertyPredictiondft_3d_mepszkgcnn_coNGNMAE22.842kgcnndft_3d16809
AISinglePropertyPredictionhmof_min_co2_adspalignn_modelMAE0.0383ALIGNNhmof137651
AISinglePropertyPredictionqm9_std_jctc_mualignn_modelMAE0.0248ALIGNNqm9_std_jctc130829
AISinglePropertyPredictiondft_3d_n_Seebeckkgcnn_coGNMAE39.2692kgcnndft_3d23210
AISinglePropertyPredictiondft_3d_n_powerfactalignn_modelMAE442.2993ALIGNNdft_3d23210
AISinglePropertyPredictiondft_3d_optb88vdw_bandgapkgcnn_coGNMAE0.1219kgcnndft_3d55713
AISinglePropertyPredictiondft_3d_optb88vdw_total_energykgcnn_coGNMAE0.0262kgcnndft_3d55713
AISinglePropertyPredictiondft_3d_ph_heat_capacitymatminer_rfMAE5.2757UofTdft_3d12054
AISinglePropertyPredictionhmof_pldmatminer_xgboostMAE0.5728UofThmof137651
AISinglePropertyPredictionocp100k_relaxed_energyalignn_modelMAE0.6289ALIGNNocp100k149886
AISinglePropertyPredictionocp10k_relaxed_energyalignn_modelMAE0.7623ALIGNNocp10k59886
AISinglePropertyPredictionocp_all_relaxed_energyalignn_modelMAE0.5989ALIGNNocp_all510214
AISinglePropertyPredictiondft_3d_shear_modulus_gvkgcnn_coNGNMAE8.4881kgcnndft_3d19680
AISinglePropertyPredictiondft_3d_slmekgcnn_coNGNMAE4.4428kgcnndft_3d9062
AISinglePropertyPredictiondft_3d_spillagekgcnn_coNGNMAE0.3463kgcnndft_3d11375
AISinglePropertyPredictionhmof_surface_area_m2cm3matminer_xgboostMAE83.3575UofThmof137651
AISinglePropertyPredictionhmof_surface_area_m2galignn_modelMAE91.1502ALIGNNhmof137651
AISinglePropertyPredictionhmof_void_fractionalignn_modelMAE0.0174ALIGNNhmof137651
AISpectraedos_pdos_ph_dosalignn_modelMULTIMAE0.0577ALIGNNedos_pdos14243
AITextClassarXiv_categoriessvc_model_text_abstractACC0.9082ChemNLParXiv100994
AITextClasspubchem_categoriesrandom_forest_text_title_abstractACC0.9674ChemNLPpubchem44500
AITextGenarxiv_gen_textChatGPT_May24ROUGE0.3006ChatGPTarxiv_gen490
AITextSummaryarxiv_summary_texttransformers_t5_baseROUGE0.2602ChemNLParxiv_summary87148
AITokenClassmat_scholar_ner_labelstransformers_xlnetACC0.7481ChemNLPmat_scholar_ner123267
ESSinglePropertyPredictiondft_3d_Tc_superconqe_pbesol_gbrvMAE3.3785JARVISdft_3d14
ESSinglePropertyPredictiondft_3d_Tc_supercon_JVASP_1014_Taqe_pbesol_gbrvMAE3.1395JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_Tc_supercon_JVASP_1151_MgB2qe_pbesol_gbrvMAE6.3148JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_Tc_supercon_JVASP_11981_Nb3Alqe_pbesol_gbrvMAE7.8436JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_Tc_supercon_JVASP_14492_NbOqe_pbesol_gbrvMAE2.2134JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_Tc_supercon_JVASP_14837_Vqe_pbesol_gbrvMAE12.957JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_Tc_supercon_JVASP_14960_V3Siqe_pbesol_gbrvMAE0.6329JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_Tc_supercon_JVASP_15938_Nb3Siqe_pbesol_gbrvMAE1.4986JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_Tc_supercon_JVASP_19679_ZrNqe_pbesol_gbrvMAE0.2645JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_Tc_supercon_JVASP_19889_NbCqe_pbesol_gbrv_tetrahedronMAE1.0JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_Tc_supercon_JVASP_20620_YB6qe_pbesol_gbrvMAE2.0599JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_Tc_supercon_JVASP_36335_NbNqe_pbesol_gbrvMAE1.5652JARVISdft_3d1
ESSinglePropertyPredictiondft_2d_Tc_supercon_JVASP_646_NbS2qe_pbesol_gbrvMAE2.2JARVISdft_2d1
ESSinglePropertyPredictiondft_2d_Tc_supercon_JVASP_655_NbSe2qe_pbesol_gbrvMAE1.9JARVISdft_2d1
ESSinglePropertyPredictiondft_3d_Tc_supercon_JVASP_816_Alqe_pbesol_gbrvMAE0.3995JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_Tc_supercon_JVASP_934_Nbqe_pbesol_gbrvMAE1.4233JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_Tc_supercon_JVASP_961_Pbqe_pbesol_gbrv_tetrahedronMAE1.8JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bandgapvasp_tbmbjMAE0.4981JARVISdft_3d54
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_1002_Sigpaw_gllbscMAE0.0048GPAWdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_104_TiO2tb3_modelsMAE0.3108TB3dft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_1130_LiFgpaw_gllbscMAE0.7701GPAWdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_113_ZrO2gpaw_gllbscMAE1.2866GPAWdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_1145_KClvasp_tbmbjMAE0.091JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_116_MgOgpaw_gllbscMAE0.6653GPAWdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_1174_GaAsvasp_tbmbjMAE0.199JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_1180_InNvasp_tbmbjMAE0.038JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_1183_InPvasp_tbmbjMAE0.033JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_1189_InSbgpaw_ldaMAE0.0238GPAWdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_1192_CdSevasp_tbmbjMAE0.01JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_1198_ZnTevasp_tbmbjMAE0.157JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_1201_CuClgpaw_gllbscMAE0.708GPAWdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_1216_Cu2Ogpaw_gllbscMAE0.9561GPAWdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_1267_BaTegpaw_gllbscMAE0.2532GPAWdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_1294_BaSegpaw_gllbscMAE0.5666GPAWdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_1300_MgSvasp_tbmbjMAE0.504JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_1312_BPvasp_tbmbjMAE0.186JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_1315_BaSvasp_tbmbjMAE0.601JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_1327_AlPvasp_tbmbjMAE0.063JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_1393_GaPvasp_tbmbjMAE0.023JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_1405_CaOgpaw_gllbscMAE0.4381GPAWdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_1408_AlSbvasp_tbmbjMAE0.087JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_1453_AlCuO2gpaw_gllbscMAE0.2589GPAWdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_1702_ZnSgpaw_gllbscMAE0.0562GPAWdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_17_BNvasp_tbmbjMAE0.085JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_1954_AgClvasp_tbmbjMAE0.372JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_23_CdTevasp_tbmbjMAE0.03JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_299_SnSevasp_optb88vdwMAE0.192JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_30_GaNgpaw_gllbscMAE0.3384GPAWdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_32_Al2O3gpaw_gllbscMAE0.488GPAWdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_39_AlNgpaw_gllbscMAE0.0119GPAWdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_54_MoS2vasp_tbmbjMAE0.047JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_57_MoSe2gpaw_pbeMAE0.034GPAWdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_5_TiO2tb3_modelsMAE0.2013TB3dft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_72_WS2tb3_modelsMAE0.0674TB3dft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_75_WSe2gpaw_pbeMAE0.0103GPAWdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_7630_BAsvasp_optb88vdwMAE0.045JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_7678_MgSevasp_optb88vdwMAE0.353JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_7762_MgTevasp_tbmbjMAE0.106JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_7844_AlNvasp_tbmbjMAE0.099JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_7860_SnTetb3_modelsMAE0.039TB3dft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_8003_CdSvasp_tbmbjMAE0.02JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_8082_SrTiO3tb3_modelsMAE0.6271TB3dft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_8158_SiCvasp_tbmbjMAE0.112JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_8169_GaNgpaw_gllbscMAE0.1977GPAWdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_8566_AgIvasp_tbmbjMAE0.821JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_8583_AgBrvasp_tbmbjMAE0.194JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_890_Geqe_pbe_ccECPMAE0.0495JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_9147_HfO2vasp_tbmbjMAE0.036JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_91_Cgpaw_gllbscMAE0.2281GPAWdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_95_CdSvasp_tbmbjMAE0.101JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_96_ZnSevasp_tbmbjMAE0.19JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bandgap_JVASP_97_InAsvasp_tbmbjMAE0.022JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulusvasp_opt86bMAE4.6619JARVISdft_3d21
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_1002_Sivasp_scanMAE0.669JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_1130_LiFvasp_pbe_mpMAE0.2MPdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_113_ZrO2qmcpack_team__dmc_003MAE42.0QMCPACKdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_116_MgOvasp_optb88vdwMAE4.33JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_1174_GaAsqmcpack_dmc_pbeMAE0.9669JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_1306_BaOqmcpack_team__dmc_002MAE2.35QMCPACKdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_1312_BPqmcpack_team__dmc_001MAE4.85QMCPACKdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_1390_SrOqmcpack_team__dmc_002MAE1.69QMCPACKdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_1405_CaOqmcpack_team__dmc_002MAE0.38QMCPACKdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_14590_Hfqmcpack_team__dmc_003MAE0.4QMCPACKdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_14604_Bavasp_optpbeMAE0.1JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_14606_Agvasp_opt86bMAE1.6JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_14612_Zrqmcpack_team__dmc_003MAE4.0QMCPACKdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_14813_Rbvasp_pbe_mpMAE0.1MPdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_182_SiCvasp_optcx13MAE7.1JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_20290_Sc2O3qmcpack_team__dmc_002MAE29.65QMCPACKdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_20326_NaFvasp_opt86bMAE0.5JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_20793_Y2O3qmcpack_team__dmc_002MAE23.7QMCPACKdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_21208_Srvasp_opt86bMAE0.0JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_23862_NaClvasp_opt86bMAE0.3JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_23864_LiClvasp_optb88vdwMAE0.13JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_25065_Livasp_optcx13MAE0.0JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_25114_Kvasp_optpbeMAE0.1JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_25180_Cavasp_opt86bMAE0.5JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_25213_Krqmcpack_team__dmc_001MAE0.24QMCPACKdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_25248_Xeqmcpack_team__dmc_001MAE0.19QMCPACKdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_34249_HfO2qmcpack_team__dmc_003MAE55.0QMCPACKdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_350_ZrO2qmcpack_team__dmc_003MAE93.0QMCPACKdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_7809_ZrO2qmcpack_team__dmc_003MAE30.0QMCPACKdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_7870_MnOqmcpack_team__dmc_002MAE10.0QMCPACKdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_7871_NiOqmcpack_team__dmc_002MAE5.0QMCPACKdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_79204_BNqmcpack_team__dmc_001MAE10.86QMCPACKdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_816_Alvasp_optcx13MAE0.2JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_819_Arqmcpack_team__dmc_001MAE0.42QMCPACKdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_834_Beqmcpack_team__dmc_001MAE2.37QMCPACKdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_867_Cuvasp_optb88vdwMAE0.6JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_87128_CoOqmcpack_team__dmc_002MAE1.0QMCPACKdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_890_Geqmcpack_dmc_pbeMAE2.1838JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_9147_HfO2qmcpack_team__dmc_003MAE8.0QMCPACKdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_91_Cvasp_optcx13MAE4.0JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_92796_La2O3qmcpack_team__dmc_002MAE20.34QMCPACKdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_963_Pdvasp_optcx13MAE5.1JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_984_Rhvasp_optb88vdwMAE8.3JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_cubic_lattice_param_avasp_optb88vdwMAE0.0491JARVISdft_3d64
ESSinglePropertyPredictionvacancydb_efvasp_optb88vdwMAE0.3026JARVISvacancydb39
ESSinglePropertyPredictiondft_3d_epsxvasp_optb88vdw_linoptMAE1.4638JARVISdft_3d16
ESSinglePropertyPredictiondft_3d_epsx_JVASP_1312_BPvasp_optb88vdw_linoptMAE1.8931JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_epsx_JVASP_1327_AlPvasp_optb88vdw_linoptMAE1.0729JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_epsx_JVASP_1393_GaPvasp_optb88vdw_linoptMAE0.4888JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_epsx_JVASP_1408_AlSbvasp_optb88vdw_linoptMAE0.3367JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_epsx_JVASP_1702_ZnSvasp_optb88vdw_dfptMAE1.3908JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_epsx_JVASP_17_BNvasp_optb88vdw_linoptMAE0.307JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_epsx_JVASP_182_SiCvasp_optb88vdw_linoptMAE0.3837JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_epsx_JVASP_2355_ZnGeP2vasp_optb88vdw_dfptMAE0.2855JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_epsx_JVASP_2376_ZnSiP2vasp_optb88vdw_linoptMAE0.1707JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_epsx_JVASP_23_CdTevasp_optb88vdw_linoptMAE2.9228JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_epsx_JVASP_54_MoS2vasp_optb88vdw_linoptMAE0.8649JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_epsx_JVASP_57_MoSe2vasp_optb88vdw_linoptMAE0.5049JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_epsx_JVASP_60_Te2Movasp_optb88vdw_linoptMAE0.7269JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_epsx_JVASP_72_WS2vasp_optb88vdw_dfptMAE2.4098JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_epsx_JVASP_75_WSe2vasp_optb88vdw_dfptMAE3.7241JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_epsx_JVASP_8041_HgTevasp_optb88vdw_linoptMAE3.2251JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_max_piezoelecvasp_optb88vdw_dfptMAE1.0853JARVISdft_3d16
ESSinglePropertyPredictiondft_3d_max_piezoelec_JVASP_110_BaTiO3vasp_optb88vdw_dfptMAE5.109JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_max_piezoelec_JVASP_1180_InNvasp_optb88vdw_dfptMAE0.0086JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_max_piezoelec_JVASP_1195_ZnOvasp_optb88vdw_dfptMAE0.7625JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_max_piezoelec_JVASP_1240_LiNbO3vasp_optb88vdw_dfptMAE0.7074JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_max_piezoelec_JVASP_1327_AlPvasp_optb88vdw_dfptMAE0.7648JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_max_piezoelec_JVASP_1372_AlAsvasp_optb88vdw_dfptMAE0.8201JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_max_piezoelec_JVASP_1408_AlSbvasp_optb88vdw_dfptMAE0.6711JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_max_piezoelec_JVASP_20778_BeOvasp_optb88vdw_dfptMAE0.4868JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_max_piezoelec_JVASP_30_GaNvasp_optb88vdw_dfptMAE0.313JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_max_piezoelec_JVASP_3450_TiPbO3vasp_optb88vdw_dfptMAE4.3759JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_max_piezoelec_JVASP_35711_GaSbvasp_optb88vdw_dfptMAE0.84JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_max_piezoelec_JVASP_39_AlNvasp_optb88vdw_dfptMAE1.1309JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_max_piezoelec_JVASP_41_SiO2vasp_optb88vdw_dfptMAE0.0115JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_max_piezoelec_JVASP_57695_BNvasp_optb88vdw_dfptMAE0.4967JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_max_piezoelec_JVASP_7648_ZnSvasp_optb88vdw_dfptMAE0.3974JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_max_piezoelec_JVASP_8047_ZnSevasp_optb88vdw_dfptMAE0.4686JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_slmevasp_tbmbjMAE5.0925JARVISdft_3d5
ESSinglePropertyPredictiondft_3d_slme_JVASP_1174_GaAsvasp_tbmbjMAE3.94JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_slme_JVASP_266_InPvasp_tbmbjMAE9.14JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_slme_JVASP_7112_H6PbCI3Nvasp_tbmbjMAE0.52JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_slme_JVASP_7757_CdTevasp_tbmbjMAE6.77JARVISdft_3d1
ESSinglePropertyPredictiondft_3d_slme_JVASP_8554_InCuSe2vasp_tbmbjMAE8.35JARVISdft_3d1
ESSpectradft_3d_dielectric_functionvasp_tbmbjMULTIMAE2.88JARVISdft_3d4
ESSpectradft_3d_dielectric_function_JVASP_1002_Sivasp_tbmbjMULTIMAE3.3922JARVISdft_3d1
ESSpectradft_3d_dielectric_function_JVASP_1174_GaAsvasp_tbmbjMULTIMAE2.672JARVISdft_3d1
ESSpectradft_3d_dielectric_function_JVASP_266_InPvasp_tbmbjMULTIMAE2.5309JARVISdft_3d1
ESSpectradft_3d_dielectric_function_JVASP_890_Gevasp_tbmbjMULTIMAE2.9248JARVISdft_3d1
EXPSinglePropertyPredictiondft_3d_Tc_supercon_JVASP_1151_MgB2ppms_magnetizationMAE0.22PML-PPMSdft_3d1
EXPSinglePropertyPredictiondft_3d_Tc_supercon_JVASP_19679_ZrNppms_magnetizationMAE0.1PML-PPMSdft_3d1
EXPSinglePropertyPredictiondft_3d_Tc_supercon_JVASP_20166_Mo2Cppms_magnetizationMAE1.44PML-PPMSdft_3d1
EXPSinglePropertyPredictiondft_3d_Tc_supercon_JVASP_30369_NbS2ppms_magnetizationMAE1.8PML-PPMSdft_3d1
EXPSinglePropertyPredictiondft_3d_Tc_supercon_JVASP_31795_NbSe2ppms_magnetizationMAE1.1PML-PPMSdft_3d1
EXPSinglePropertyPredictiondft_3d_Tc_supercon_JVASP_36335_NbNppms_magnetizationMAE0.9PML-PPMSdft_3d1
EXPSinglePropertyPredictiondft_3d_Tc_supercon_JVASP_45_FeSeppms_magnetizationMAE0.5PML-PPMSdft_3d1
EXPSpectradft_3d_XRD_JVASP_19821_MgB2bruker_d8MULTIMAE0.02MML-BrukerD8dft_3d1
EXPSpectranist_isodb_co2_RM_885210.1007s10450-018-9958-x.Lab01MULTIMAE0.0213FACTlabnist_isodb1
EXPSpectramidas_stress_strain_vibroscopy_kevlar129favimat_5MULTIMAE0.014FAVIMATmidas1
FFSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_1002_SiSi.tersoffMAE1.4267JARVIS-FFdft_3d1
FFSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_816_AlNiAlH_jea.eam.alloyMAE0.0267JARVIS-FFdft_3d1
FFSinglePropertyPredictiondft_3d_bulk_modulus_JVASP_867_CuMendelev_Cu2_2012.eam.fsMAE0.8367JARVIS-FFdft_3d1
FFSinglePropertyPredictionbiobench_deltaFtiwarylabMAE0.9477tiwarylabbiobench3
FFSinglePropertyPredictionbiobench_left_handed_populationtiwarylabMAE0.1087tiwarylabbiobench3
FFSinglePropertyPredictionbiobench_right_handed_populationtiwarylabMAE0.0307tiwarylabbiobench3
FFSinglePropertyPredictionlj_2d_liquid_viscositylammps_2d_liquid_einsteinMAE0.0244LAMMPSlj_2d_liquid1
QCEigenSolverdft_3d_electron_bands_JVASP_816_Al_WTBHqiskit_vqd_SU2_c6MULTIMAE0.003JARVISdft_3d1

A tree diagram for directory and file structure

Tree

How to contribute

1. How to add a contribution (csv.zip):

For a short demo, checkout this google colab-notebook

Prerequisites (for those not familiar with GitHub):

A GitHub username and a basic familiarity with GitHub is needed to work with the JARVIS-Leaderboard. See below:

If you do not have a GitHub account, sign up for one at: https://github.com/signup Related article: https://www.toolsqa.com/git/how-to-create-github-account/

GitHub

Suppose you choose your username as “knc6”, you’ll have a webpage at https://github.com/knc6 . Ofcourse, choose some other username of your choice, but for the following demo, we will use the “knc6” as an example username.

Once you have the GitHub account, go to the website https://github.com/usnistgov/jarvis_leaderboard. Here Fork (make your own copy) of the JARVIS-Leaderbord repo by clicking on the “Fork” button. Also, feel free to click on the “Star” button if you like the JARVIS-Leaderboard project.

GitHub

Now, go to your forked version of the JARVIS-Leaderboard. For a username “knc6”, you’ll have a repo at https://github.com/knc6/jarvis_leaderboard. Notice, we have “knc6” in the URL instead of usnistgov. Ofcourse, use your own username instead of knc6.

GitHub

Now, we will make a pull request (request to contribute your edits in the main repo). If you are not familiar with pull requests checkout this link

GitHub

If successful with basic GitHub setup, follow the guidelines:

  1. Go to your forked jarvis_leaderboard repo.
  2. git clone https://github.com/USERNAME/jarvis_leaderboard. NOTE: use your own GitHub USERNAME, e.g. knc6, instead of usnistgov

    Note if you do not use forked version, you won't be able to make a pull request

  3. cd jarvis_leaderboard

  4. conda create --name leaderboard python=3.8
  5. source activate leaderboard
  6. Install the package: python setup.py develop
  7. Let's add a contribution for Silicon bandgap using DFT PBE (an Electronic structure approach)

cd jarvis_leaderboard/contributions/

mkdir vasp_pbe_teamX , you can give any reaosnable name to the benchmark folder in place of vasp_pbe_teamX

cd vasp_pbe_teamX

cp ../vasp_optb88vdw/ES-SinglePropertyPrediction-bandgap_JVASP_1002_Si-dft_3d-test-mae.csv.zip .

vi ES-SinglePropertyPrediction-bandgap_JVASP_1002_Si-dft_3d-test-mae.csv.zip

Note: do not change filenames, e.g., replace dft with qmc etc., which will cause errors

After pressing eneter twice, you'll see the file content as id,prediction

Just modify the predicting value to your model/measurement value

Save the file (":wq!" and ":q!")

Add metadata.json and run.sh files to capture metadata and enhance reproducibility. The metadata file must have at least your project_url and model_name info. The project_url couls be a publication/GitHub page etc.

Note: An admin will run your run.sh to check if he/she can reproduce your benchmark

Now, cd ../../../

python jarvis_leaderboard/rebuild.py

which will compile all data, compare with reference dataset and calculate metrices

Hoping there's no error, try: mkdocs serve

Ensure vasp_pbe_teamX row exists at:

http://127.0.0.1:8000/usnistgov/jarvis_leaderboard/ES/SinglePropertyPrediction/bandgap_JVASP_1002_Si/

Now add changes, git add jarvis_leaderboard/contributions/vasp_pbe_teamX

Commit your changes, git commit -m 'Adding my PBE Si result.'

git push

Now go to your forked github repo and make a pull reuqest (PR) to usnistgov/jarvis_leaderboard in develop branch

If you are not familiar with pull requests checkout this link

Note: only admins are allowed to make pull requests to main branch

Once an admin approve the PR, you'll see your results on the official leaderboard

  1. Another example for AI mode as follows:

Populate the dataset for a benchmark, e.g.:

python jarvis_leaderboard/populate_data.py --benchmark_file AI-SinglePropertyPrediction-exfoliation_energy-dft_3d-test-mae --output_path=Out

Currently, this script works for atomistic tasks only, addition of other tasks will be available soon.

Train you model(s), e.g.:

pip install alignn

wget https://raw.githubusercontent.com/usnistgov/alignn/main/alignn/examples/sample_data/config_example.json

train_folder.py --root_dir "Out" --config "config_example.json" --output_dir="temp"

Create a folder in the jarvis_leaderboard/contributions folder under respective submodule, e.g.:

mkdir contributions/my_awesome_model

Add comma-separated zip file (.csv.zip) file(s) corresponding to benchmark(s), e.g.:

cp temp/prediction_results_test_set.csv .

mv prediction_results_test_set.csv AI-SinglePropertyPrediction-exfoliation_energy-dft_3d-test-mae.csv

zip AI-SinglePropertyPrediction-exfoliation_energy-dft_3d-test-mae.csv AI-SinglePropertyPrediction-exfoliation_energy-dft_3d-test-mae.csv.zip

mv AI-SinglePropertyPrediction-exfoliation_energy-dft_3d-test-mae.csv.zip jarvis_leaderboard/contributions/my_awesome_model

Add metadata info in the metadata.json file, e.g.:

cp jarvis_leaderboard/contributions/alignn_models/metadata.json jarvis_leaderboard/contributions/my_awesome_model

Also, add a run.py, run.sh and Dockerfile scripts to reproduce the model predictions.

Run python jarvis_leaderboard/rebuild.py to check there are no errors

Run mkdocs serve to check if the new benchmark exists, e.g. at page http://127.0.0.1:8000/usnistgov/jarvis_leaderboard/AI/SinglePropertyPrediction/exfoliation_energy/

Add. commit and push your changes, e.g.:

git add jarvis_leaderboard/contributions/my_awesome_model

git commit -m 'Adding my awesome_model to jarvis_leaderboard

git push origin main

Make a pull request from your fork to the source repo at usnistgov/jarvis_leaderboard develop branch

Notes:

  1. The word: SinglePropertyPrediction: task type, test, property: exfoliation_energy, dataset: dft_3d, method: AI, metric: mae have been joined with '-' sign. This format should be used for consistency in webpage generation.

  2. The test data splits are pre-determined, if the exact test IDs are not used, then the code might result in errors.

2. How to add a new benchmark (json.zip):

  1. Create a json.zip file in the jarvis_leaderboard/benchmarks folder under respective sub-category, e.g.:

    e.g. jarvis_leaderboard/benchmarks/AI/SinglePropertyPrediction/dft_3d_exfoliation_energy.json.zip.

    An example of a new json.zip (mydb_myprop.json) file can be found here as well.

  2. In the .json file should have train, val, test keys with array of ids and their values.

    Note train and val can be empty dictionaries if the benchmarks are other than AI method

  3. Add a .md file, e.g.: jarvis_leaderboard/docs/AI/SinglePropertyPrediction/exfoliation_energy.md. This is where contributions performers will be kept and website info will be generated.

  4. An example for creating such a file is provided in: jarvis_leaderboard/benchmarks/AI/SinglePropertyPrediction/transform_from_figshare.py

  5. Then follow the instructions for "Adding model benchmarks to existing dataset"

Notes: A new benchmark must be linked with a peer-reviewed article and must have a DOI to ensure a minimum quality assurance for the data. We recommend adding your large dataset in Figshare or similar repository and then integrate it in JARVIS-Tools We also recommend to use JARVIS-Tools for generating dataset/models/benchmarks which can help us maintain the benchmark for long term.
Methods used for generating the data and referece are given below:

Method used for results Methods for comparison
EXP EXP/ES/analytical results
ES ES/EXP
FF ES/EXP
QC Classical/analytical results
AI Test set data

Citation

Large Scale Benchmark of Materials Design Methods

Acronyms

  1. MAE: Mean Absolute Error
  2. ACC: Classification accuracy
  3. MULTIMAE: MAE sum of multple entries, Euclidean distance
  4. For names of datasets and associated propertiesm refer to datasets in JARVIS-Tools

Help

Ask a question/raise an issue on GitHub. You can also email Kamal Choudhary if needed (kamal.choudhary@nist.gov). However, we recommend using the GitHub issues for any questions/concerns.

License

This template is served under the NIST license.
Read the LICENSE file for more info.