Table of Contents

• Introduction
• Summary table
• Quick start using colab
• Terminologies
• List of benchmarks
• Tree diagram
• How to contribute
• License
• Citation

JARVIS Leaderboard (Introduction)

This project provides benchmark-performances of various methods for materials science applications using the datasets available in JARVIS-Tools databases. Some of the categories are: Artificial Intelligence (AI), Electronic Structure (ES), Force-field (FF), Quantum Computation (QC) and Experiments (EXP). There are a variety of properties included in the benchmark. In addition to prediction results, we attempt to capture the underlyig software, hardware and instrumental frameworks to enhance reproducibility. This project is a part of the NIST-JARVIS infrastructure.

Usually, codes are kept in GitHub/GitLab etc., data is kept in Zenodo/FigShare/NIST Materials data etc., we recommend keeping the benchmarks in the JARVIS-Leaderboard to enhance reproducibility and transparency.

• Number of contributors: 20

• Number of methods: 159

• Number of benchmarks: 296

• Number of contributions: 1476

• Number of datapoints: 8731126

Summary table

Category/Sub-cat.	SinglePropertyPrediction	SinglePropertyClass	MLFF	TextClass	TokenClass	TextSummary	TextGen	AtomGen	ImageClass	Spectra	EigenSolver
AI	441	21	119	18	1	1	3	3	2	1	-
ES	776	-	-	-	-	-	-	-	-	10	-
FF	50	-	-	-	-	-	-	-	-	-	-
QC	-	-	-	-	-	-	-	-	-	-	6
EXP	7	-	-	-	-	-	-	-	-	18	-

Quick start using GoogleColab notebook examples

1. Analyzing_data_in_the_JARVIS_Leaderboard.ipynb
2. Upload_benchmark_to_jarvis_leaderboard.ipynb
3. alignn_jarvis_leaderboard.ipynb
4. kgcnn_jarvis_leaderboard.ipynb
5. MatMiner_on_JARVIS_DFT.ipynb
6. Train ALIGNN-FF for FCC Copper
7. Train ALLEGRO for Silicon
8. AtomVision_Image_Classification.ipynb
9. Text-generation ChemNLP/HuggingFace.ipynb
10. Inverse design of superconductors
11. QuantumComputation for FCC Aluminum.ipynb
12. Pretrained CHGNet Prediction.ipynb
13. Pretrained OpenCatalystProject Model.ipynb
14. GPAW colab.ipynb
15. Quantum Espresso colab.ipynb
16. ocp20_load_pretrained_models.ipynb

For additional notebooks, visit JARVIS-Tools-Notebooks collections with more than 60 colab notebooks.

Terminologies used in this project

• Categories: are of following types Artificial Intelligence (AI), Electronic Structure (ES), Force-field (FF), Quantum Computation (QC) and Experiments (EXP). Each of these categories are divided into sub-categories. These sub-categories include single-property-prediction, single-property-classification, machine-learning force-fields, text-classification, text-token classification, text-generation, image classification, image-segmentation, image-generation, spectra-prediction, and eigensolver. These categories and sub-categories are highly flexible and new entries can be easily added.

• Sub-categories: include 1) SinglePropertyPrediction (where the output of a model/experiment is one single number for an entry), 2) SinglePropertyClass (where the output is class-ids, e.g., 0,1,.. instead of floating values), 3) ImageClass (for multi-class image classification), 4) textClass (for multi-label text classification), 5) MLFF (machine learning force-field), 6) Spectra (for multi-value data) and 7) EigenSolver (for Hamiltonian simulation).

• Benchmarks: are ground truth data used to calculate metrics for each specific task (e.g a json.zip file).

• Methods: are a set of precise specifications for evaluation against a benchmark. For example, within ES category, DFT with VASP-GGA-PAW-PBE are specifications, hence a method. Similarly, within AI category, descriptor/feature based models with MatMiner-chemical features and LightGBM software are specifications, hence a method.

• Contributions: are individual data in form of csv.zip files for each benchmark and specific method. Each contribution has six components: method (e.g. AI), category (e.g. SinglePropertyPrediction), property (e.g. formation energy), dataset (e.g. dft_3d), data-split (e.g. test), metric (e.g. mae).

List of benchmarks

Click on the entries in the Benchmark column. You'll be able to see methods available for each benchmark, CSV file submitted for the contribution, JSON file for ground trutch data, run.sh script for running the method and Info for metadata associated with the method.

Category	Sub-category	Benchmark	Method	Metric	Score	Team	Dataset	Size
AI	AtomGen	dft_3d_Tc_supercon	cdvae_model	RMSE	0.3557	CDVAE	dft_3d	1056
AI	AtomGen	carbon24_energy_per_atom	cdvae_model	RMSE	0.3581	CDVAE	carbon24	10153
AI	ImageClass	stem_2d_image_bravais_class	densenet_model	ACC	0.8304	JARVIS	stem_2d_image	9150
AI	MLFF	alignn_ff_db_energy	alignnff_wt0.1	MAE	0.0342	JARVIS	alignn_ff_db	307111
AI	MLFF	mlearn_Cu_energy	alignnff_mlearn_only	MAE	1.0104	JARVIS	mlearn_Cu	293
AI	MLFF	mlearn_Ge_energy	alignnff_fmult_mlearn_only	MAE	0.9439	JARVIS	mlearn_Ge	253
AI	MLFF	mlearn_Li_energy	alignnff_fmult_mlearn_only	MAE	0.2735	JARVIS	mlearn_Li	270
AI	MLFF	mlearn_Mo_energy	alignnff_fmult_mlearn_only	MAE	1.1066	JARVIS	mlearn_Mo	217
AI	MLFF	mlearn_Ni_energy	alignnff_mlearn_only	MAE	1.4316	JARVIS	mlearn_Ni	294
AI	MLFF	mlearn_Si_energy	alignnff_mlearn_only	MAE	1.0688	JARVIS	mlearn_Si	239
AI	MLFF	alignn_ff_db_forces	m3gnet_pretrained	MULTIMAE	0.0887	M3GNET	alignn_ff_db	307111
AI	MLFF	mlearn_Cu_forces	alignnff_fmult_mlearn_only	MULTIMAE	0.0357	JARVIS	mlearn_Cu	293
AI	MLFF	mlearn_Ge_forces	alignnff_fmult_mlearn_only	MULTIMAE	0.0726	JARVIS	mlearn_Ge	253
AI	MLFF	mlearn_Li_forces	alignnff_fmult_mlearn_only	MULTIMAE	0.0339	JARVIS	mlearn_Li	270
AI	MLFF	mlearn_Mo_forces	alignnff_fmult_mlearn_only	MULTIMAE	0.1229	JARVIS	mlearn_Mo	217
AI	MLFF	mlearn_Ni_forces	alignnff_fmult_mlearn_only	MULTIMAE	0.0448	JARVIS	mlearn_Ni	294
AI	MLFF	mlearn_Si_forces	alignnff_fmult_mlearn_only	MULTIMAE	0.0885	JARVIS	mlearn_Si	239
AI	MLFF	mlearn_Cu_stresses	alignnff_fd	MULTIMAE	20.0515	ALIGNN	mlearn_Cu	293
AI	MLFF	mlearn_Ge_stresses	alignnff_fd	MULTIMAE	9.0436	ALIGNN	mlearn_Ge	253
AI	MLFF	mlearn_Li_stresses	alignnff_fd	MULTIMAE	2.6491	ALIGNN	mlearn_Li	270
AI	MLFF	mlearn_Mo_stresses	alignnff_mlearn_only	MULTIMAE	19.4086	JARVIS	mlearn_Mo	217
AI	MLFF	mlearn_Ni_stresses	alignnff_fd	MULTIMAE	31.1771	ALIGNN	mlearn_Ni	294
AI	MLFF	mlearn_Si_stresses	alignnff_fmult_mlearn_only	MULTIMAE	11.5542	JARVIS	mlearn_Si	239
AI	SinglePropertyClass	dft_3d_magmom_oszicar	matminer_xgboost	ACC	0.9489	UofT	dft_3d	52210
AI	SinglePropertyClass	dft_3d_mbj_bandgap	matminer_xgboost	ACC	0.9399	UofT	dft_3d	18167
AI	SinglePropertyClass	dft_3d_n_powerfact	matminer_xgboost	ACC	0.8186	UofT	dft_3d	23210
AI	SinglePropertyClass	dft_3d_optb88vdw_bandgap	matminer_xgboost	ACC	0.9361	UofT	dft_3d	55713
AI	SinglePropertyClass	dft_3d_p_Seebeck	matminer_xgboost	ACC	0.9332	UofT	dft_3d	23210
AI	SinglePropertyClass	dft_3d_slme	matminer_rf	ACC	0.8422	UofT	dft_3d	9062
AI	SinglePropertyClass	dft_3d_spillage	matminer_xgboost	ACC	0.8364	UofT	dft_3d	11375
AI	SinglePropertyPrediction	snumat_Band_gap_HSE	alignn_model	MAE	0.3765	ALIGNN	snumat	10386
AI	SinglePropertyPrediction	qm9_std_jctc_Cv	alignn_model	MAE	0.0239	ALIGNN	qm9_std_jctc	130829
AI	SinglePropertyPrediction	qm9_std_jctc_G	alignn_model	MAE	0.0133	ALIGNN	qm9_std_jctc	130829
AI	SinglePropertyPrediction	qm9_std_jctc_H	alignn_model	MAE	0.014	ALIGNN	qm9_std_jctc	130829
AI	SinglePropertyPrediction	qm9_std_jctc_HOMO	alignn_model	MAE	0.0187	ALIGNN	qm9_std_jctc	130829
AI	SinglePropertyPrediction	halide_peroskites_HSE_decomp_energy	alignn_model	MAE	0.0512	ALIGNN	halide_peroskites	229
AI	SinglePropertyPrediction	halide_peroskites_HSE_gap	alignn_model	MAE	0.1615	ALIGNN	halide_peroskites	229
AI	SinglePropertyPrediction	qm9_std_jctc_LUMO	alignn_model	MAE	0.0175	ALIGNN	qm9_std_jctc	130829
AI	SinglePropertyPrediction	halide_peroskites_PBE_decomp_energy	alignn_model	MAE	0.0477	ALIGNN	halide_peroskites	229
AI	SinglePropertyPrediction	halide_peroskites_PBE_gap	alignn_model	MAE	0.1223	ALIGNN	halide_peroskites	229
AI	SinglePropertyPrediction	qm9_std_jctc_R2	alignn_model	MAE	0.6523	ALIGNN	qm9_std_jctc	130829
AI	SinglePropertyPrediction	halide_peroskites_Ref_ind	alignn_model	MAE	0.0233	ALIGNN	halide_peroskites	229
AI	SinglePropertyPrediction	supercon_chem_Tc	matminer_rf	MAE	4.8511	UofT	supercon_chem	16414
AI	SinglePropertyPrediction	qm9_std_jctc_U	alignn_model	MAE	0.0138	ALIGNN	qm9_std_jctc	130829
AI	SinglePropertyPrediction	qm9_std_jctc_U0	alignn_model	MAE	0.0146	ALIGNN	qm9_std_jctc	130829
AI	SinglePropertyPrediction	qm9_std_jctc_ZPVE	alignn_model	MAE	0.0019	ALIGNN	qm9_std_jctc	130829
AI	SinglePropertyPrediction	qm9_std_jctc_alpha	alignn_model	MAE	0.0557	ALIGNN	qm9_std_jctc	130829
AI	SinglePropertyPrediction	dft_3d_avg_elec_mass	alignn_model	MAE	0.0853	ALIGNN	dft_3d	17642
AI	SinglePropertyPrediction	dft_3d_avg_hole_mass	alignn_model	MAE	0.1239	ALIGNN	dft_3d	17642
AI	SinglePropertyPrediction	qmof_bandgap	alignn_model	MAE	0.2019	ALIGNN	qmof	20424
AI	SinglePropertyPrediction	dft_3d_bulk_modulus_kv	kgcnn_coNGN	MAE	8.7022	kgcnn	dft_3d	19680
AI	SinglePropertyPrediction	dft_3d_dfpt_piezo_max_dielectric	kgcnn_coNGN	MAE	25.5553	kgcnn	dft_3d	4704
AI	SinglePropertyPrediction	dft_3d_dfpt_piezo_max_dij	kgcnn_coNGN	MAE	13.8868	kgcnn	dft_3d	3345
AI	SinglePropertyPrediction	megnet_e_form	alignn_model	MAE	0.0221	ALIGNN	megnet	69239
AI	SinglePropertyPrediction	AGRA_CHO_ead	alignn_model	MAE	0.1062	ALIGNN	AGRA_CHO	214
AI	SinglePropertyPrediction	AGRA_CO_ead	alignn_model	MAE	0.1349	ALIGNN	AGRA_CO	193
AI	SinglePropertyPrediction	AGRA_COOH_ead	alignn_model	MAE	0.0485	ALIGNN	AGRA_COOH	280
AI	SinglePropertyPrediction	AGRA_O_ead	alignn_model	MAE	0.1134	ALIGNN	AGRA_O	1000
AI	SinglePropertyPrediction	AGRA_OH_ead	alignn_model	MAE	0.0762	ALIGNN	AGRA_OH	875
AI	SinglePropertyPrediction	tinnet_N_ead	alignn_model	MAE	0.0871	ALIGNN	tinnet_N	327
AI	SinglePropertyPrediction	tinnet_O_ead	alignn_model	MAE	0.3694	ALIGNN	tinnet_O	745
AI	SinglePropertyPrediction	tinnet_OH_ead	alignn_model	MAE	0.0882	ALIGNN	tinnet_OH	746
AI	SinglePropertyPrediction	vacancydb_ef	alignnff_wt0.1	MAE	0.8739	JARVIS	vacancydb	530
AI	SinglePropertyPrediction	vacancydb_2D_ef	alignnff_wt0.1	MAE	1.0018	JARVIS	vacancydb_2D	72
AI	SinglePropertyPrediction	vacancydb_elements_ef	alignnff_wt0.1	MAE	1.1992	JARVIS	vacancydb_elements	74
AI	SinglePropertyPrediction	vacancydb_oxides_ef	alignnff_wt0.1	MAE	0.6091	JARVIS	vacancydb_oxides	65
AI	SinglePropertyPrediction	vacancydb_oxides_train_test_ef	crystal_feature_model	MAE	0.7	Crystal_Features	vacancydb_oxides_train_test	30
AI	SinglePropertyPrediction	dft_3d_ehull	kgcnn_coGN	MAE	0.0466	kgcnn	dft_3d	55364
AI	SinglePropertyPrediction	dft_3d_encut	kgcnn_coNGN	MAE	129.8266	kgcnn	dft_3d	55386
AI	SinglePropertyPrediction	qe_tb_energy_per_atom	kgcnn_coGN	MAE	0.0636	kgcnn	qe_tb	829574
AI	SinglePropertyPrediction	dft_3d_epsx	kgcnn_coNGN	MAE	18.5738	kgcnn	dft_3d	44490
AI	SinglePropertyPrediction	dft_3d_epsy	kgcnn_coNGN	MAE	18.5923	kgcnn	dft_3d	44490
AI	SinglePropertyPrediction	dft_3d_epsz	kgcnn_coNGN	MAE	17.8104	kgcnn	dft_3d	44490
AI	SinglePropertyPrediction	dft_3d_exfoliation_energy	matminer_xgboost	MAE	40.887	UofT	dft_3d	812
AI	SinglePropertyPrediction	qe_tb_f_enp	kgcnn_coGN	MAE	0.0956	kgcnn	qe_tb	829574
AI	SinglePropertyPrediction	qe_tb_final_energy	kgcnn_coGN	MAE	1.3185	kgcnn	qe_tb	829574
AI	SinglePropertyPrediction	mxene275_formation_energy	mxene_alignn	MAE	0.0561	MXene_ALIGNN	mxene275	274
AI	SinglePropertyPrediction	dft_3d_formation_energy_peratom	kgcnn_coGN	MAE	0.0271	kgcnn	dft_3d	55713
AI	SinglePropertyPrediction	ssub_formula_energy	ElemNet2_TL	MAE	0.0924	NorthWestern_University	ssub	1726
AI	SinglePropertyPrediction	qm9_std_jctc_gap	alignn_model	MAE	0.0345	ALIGNN	qm9_std_jctc	130829
AI	SinglePropertyPrediction	megnet_gappbe	alignn_model	MAE	0.218	ALIGNN	megnet	69239
AI	SinglePropertyPrediction	qe_tb_indir_gap	matminer_rf	MAE	0.0278	UofT	qe_tb	829574
AI	SinglePropertyPrediction	dft_3d_kpoint_length_unit	matminer_xgboost	MAE	9.047	UofT	dft_3d	55392
AI	SinglePropertyPrediction	hmof_lcd	matminer_xgboost	MAE	0.4585	UofT	hmof	137651
AI	SinglePropertyPrediction	dft_3d_magmom_oszicar	kgcnn_coNGN	MAE	0.2437	kgcnn	dft_3d	52210
AI	SinglePropertyPrediction	mag2d_chem_magnetic_moment	ElemNet2_TL	MAE	0.2997	NorthWestern_University	mag2d_chem	226
AI	SinglePropertyPrediction	hmof_max_co2_adsp	matminer_xgboost	MAE	0.4622	UofT	hmof	137651
AI	SinglePropertyPrediction	dft_3d_max_efg	alignn_model	MAE	19.1211	ALIGNN	dft_3d	11865
AI	SinglePropertyPrediction	dft_3d_mbj_bandgap	kgcnn_coGN	MAE	0.264	kgcnn	dft_3d	18167
AI	SinglePropertyPrediction	dft_3d_mepsx	kgcnn_coNGN	MAE	23.3801	kgcnn	dft_3d	16809
AI	SinglePropertyPrediction	dft_3d_mepsy	kgcnn_coNGN	MAE	23.3299	kgcnn	dft_3d	16809
AI	SinglePropertyPrediction	dft_3d_mepsz	kgcnn_coNGN	MAE	22.842	kgcnn	dft_3d	16809
AI	SinglePropertyPrediction	hmof_min_co2_adsp	alignn_model	MAE	0.0383	ALIGNN	hmof	137651
AI	SinglePropertyPrediction	qm9_std_jctc_mu	alignn_model	MAE	0.0248	ALIGNN	qm9_std_jctc	130829
AI	SinglePropertyPrediction	dft_3d_n_Seebeck	kgcnn_coGN	MAE	39.2692	kgcnn	dft_3d	23210
AI	SinglePropertyPrediction	dft_3d_n_powerfact	alignn_model	MAE	442.2993	ALIGNN	dft_3d	23210
AI	SinglePropertyPrediction	dft_3d_optb88vdw_bandgap	kgcnn_coGN	MAE	0.1219	kgcnn	dft_3d	55713
AI	SinglePropertyPrediction	dft_3d_optb88vdw_total_energy	kgcnn_coGN	MAE	0.0262	kgcnn	dft_3d	55713
AI	SinglePropertyPrediction	dft_3d_ph_heat_capacity	matminer_rf	MAE	5.2757	UofT	dft_3d	12054
AI	SinglePropertyPrediction	hmof_pld	matminer_xgboost	MAE	0.5728	UofT	hmof	137651
AI	SinglePropertyPrediction	ocp100k_relaxed_energy	alignn_model	MAE	0.6289	ALIGNN	ocp100k	149886
AI	SinglePropertyPrediction	ocp10k_relaxed_energy	alignn_model	MAE	0.7623	ALIGNN	ocp10k	59886
AI	SinglePropertyPrediction	ocp_all_relaxed_energy	alignn_model	MAE	0.5989	ALIGNN	ocp_all	510214
AI	SinglePropertyPrediction	dft_3d_shear_modulus_gv	kgcnn_coNGN	MAE	8.4881	kgcnn	dft_3d	19680
AI	SinglePropertyPrediction	dft_3d_slme	kgcnn_coNGN	MAE	4.4428	kgcnn	dft_3d	9062
AI	SinglePropertyPrediction	dft_3d_spillage	kgcnn_coNGN	MAE	0.3463	kgcnn	dft_3d	11375
AI	SinglePropertyPrediction	hmof_surface_area_m2cm3	matminer_xgboost	MAE	83.3575	UofT	hmof	137651
AI	SinglePropertyPrediction	hmof_surface_area_m2g	alignn_model	MAE	91.1502	ALIGNN	hmof	137651
AI	SinglePropertyPrediction	hmof_void_fraction	alignn_model	MAE	0.0174	ALIGNN	hmof	137651
AI	Spectra	edos_pdos_ph_dos	alignn_model	MULTIMAE	0.0577	ALIGNN	edos_pdos	14243
AI	TextClass	arXiv_categories	svc_model_text_abstract	ACC	0.9082	ChemNLP	arXiv	100994
AI	TextClass	pubchem_categories	random_forest_text_title_abstract	ACC	0.9674	ChemNLP	pubchem	44500
AI	TextGen	arxiv_gen_text	ChatGPT_May24	ROUGE	0.3006	ChatGPT	arxiv_gen	490
AI	TextSummary	arxiv_summary_text	transformers_t5_base	ROUGE	0.2602	ChemNLP	arxiv_summary	87148
AI	TokenClass	mat_scholar_ner_labels	transformers_xlnet	ACC	0.7481	ChemNLP	mat_scholar_ner	123267
ES	SinglePropertyPrediction	dft_3d_Tc_supercon	qe_pbesol_gbrv	MAE	3.3785	JARVIS	dft_3d	14
ES	SinglePropertyPrediction	dft_3d_Tc_supercon_JVASP_1014_Ta	qe_pbesol_gbrv	MAE	3.1395	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_Tc_supercon_JVASP_1151_MgB2	qe_pbesol_gbrv	MAE	6.3148	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_Tc_supercon_JVASP_11981_Nb3Al	qe_pbesol_gbrv	MAE	7.8436	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_Tc_supercon_JVASP_14492_NbO	qe_pbesol_gbrv	MAE	2.2134	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_Tc_supercon_JVASP_14837_V	qe_pbesol_gbrv	MAE	12.957	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_Tc_supercon_JVASP_14960_V3Si	qe_pbesol_gbrv	MAE	0.6329	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_Tc_supercon_JVASP_15938_Nb3Si	qe_pbesol_gbrv	MAE	1.4986	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_Tc_supercon_JVASP_19679_ZrN	qe_pbesol_gbrv	MAE	0.2645	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_Tc_supercon_JVASP_19889_NbC	qe_pbesol_gbrv_tetrahedron	MAE	1.0	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_Tc_supercon_JVASP_20620_YB6	qe_pbesol_gbrv	MAE	2.0599	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_Tc_supercon_JVASP_36335_NbN	qe_pbesol_gbrv	MAE	1.5652	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_2d_Tc_supercon_JVASP_646_NbS2	qe_pbesol_gbrv	MAE	2.2	JARVIS	dft_2d	1
ES	SinglePropertyPrediction	dft_2d_Tc_supercon_JVASP_655_NbSe2	qe_pbesol_gbrv	MAE	1.9	JARVIS	dft_2d	1
ES	SinglePropertyPrediction	dft_3d_Tc_supercon_JVASP_816_Al	qe_pbesol_gbrv	MAE	0.3995	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_Tc_supercon_JVASP_934_Nb	qe_pbesol_gbrv	MAE	1.4233	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_Tc_supercon_JVASP_961_Pb	qe_pbesol_gbrv_tetrahedron	MAE	1.8	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap	vasp_tbmbj	MAE	0.4981	JARVIS	dft_3d	54
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_1002_Si	gpaw_gllbsc	MAE	0.0048	GPAW	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_104_TiO2	tb3_models	MAE	0.3108	TB3	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_1130_LiF	gpaw_gllbsc	MAE	0.7701	GPAW	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_113_ZrO2	gpaw_gllbsc	MAE	1.2866	GPAW	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_1145_KCl	vasp_tbmbj	MAE	0.091	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_116_MgO	gpaw_gllbsc	MAE	0.6653	GPAW	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_1174_GaAs	vasp_tbmbj	MAE	0.199	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_1180_InN	vasp_tbmbj	MAE	0.038	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_1183_InP	vasp_tbmbj	MAE	0.033	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_1189_InSb	gpaw_lda	MAE	0.0238	GPAW	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_1192_CdSe	vasp_tbmbj	MAE	0.01	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_1198_ZnTe	vasp_tbmbj	MAE	0.157	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_1201_CuCl	gpaw_gllbsc	MAE	0.708	GPAW	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_1216_Cu2O	gpaw_gllbsc	MAE	0.9561	GPAW	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_1267_BaTe	gpaw_gllbsc	MAE	0.2532	GPAW	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_1294_BaSe	gpaw_gllbsc	MAE	0.5666	GPAW	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_1300_MgS	vasp_tbmbj	MAE	0.504	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_1312_BP	vasp_tbmbj	MAE	0.186	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_1315_BaS	vasp_tbmbj	MAE	0.601	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_1327_AlP	vasp_tbmbj	MAE	0.063	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_1393_GaP	vasp_tbmbj	MAE	0.023	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_1405_CaO	gpaw_gllbsc	MAE	0.4381	GPAW	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_1408_AlSb	vasp_tbmbj	MAE	0.087	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_1453_AlCuO2	gpaw_gllbsc	MAE	0.2589	GPAW	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_1702_ZnS	gpaw_gllbsc	MAE	0.0562	GPAW	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_17_BN	vasp_tbmbj	MAE	0.085	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_1954_AgCl	vasp_tbmbj	MAE	0.372	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_23_CdTe	vasp_tbmbj	MAE	0.03	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_299_SnSe	vasp_optb88vdw	MAE	0.192	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_30_GaN	gpaw_gllbsc	MAE	0.3384	GPAW	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_32_Al2O3	gpaw_gllbsc	MAE	0.488	GPAW	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_39_AlN	gpaw_gllbsc	MAE	0.0119	GPAW	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_54_MoS2	vasp_tbmbj	MAE	0.047	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_57_MoSe2	gpaw_pbe	MAE	0.034	GPAW	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_5_TiO2	tb3_models	MAE	0.2013	TB3	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_72_WS2	tb3_models	MAE	0.0674	TB3	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_75_WSe2	gpaw_pbe	MAE	0.0103	GPAW	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_7630_BAs	vasp_optb88vdw	MAE	0.045	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_7678_MgSe	vasp_optb88vdw	MAE	0.353	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_7762_MgTe	vasp_tbmbj	MAE	0.106	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_7844_AlN	vasp_tbmbj	MAE	0.099	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_7860_SnTe	tb3_models	MAE	0.039	TB3	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_8003_CdS	vasp_tbmbj	MAE	0.02	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_8082_SrTiO3	tb3_models	MAE	0.6271	TB3	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_8158_SiC	vasp_tbmbj	MAE	0.112	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_8169_GaN	gpaw_gllbsc	MAE	0.1977	GPAW	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_8566_AgI	vasp_tbmbj	MAE	0.821	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_8583_AgBr	vasp_tbmbj	MAE	0.194	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_890_Ge	qe_pbe_ccECP	MAE	0.0495	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_9147_HfO2	vasp_tbmbj	MAE	0.036	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_91_C	gpaw_gllbsc	MAE	0.2281	GPAW	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_95_CdS	vasp_tbmbj	MAE	0.101	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_96_ZnSe	vasp_tbmbj	MAE	0.19	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bandgap_JVASP_97_InAs	vasp_tbmbj	MAE	0.022	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus	vasp_opt86b	MAE	4.6619	JARVIS	dft_3d	21
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_1002_Si	vasp_scan	MAE	0.669	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_1130_LiF	vasp_pbe_mp	MAE	0.2	MP	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_113_ZrO2	qmcpack_team__dmc_003	MAE	42.0	QMCPACK	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_116_MgO	vasp_optb88vdw	MAE	4.33	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_1174_GaAs	qmcpack_dmc_pbe	MAE	0.9669	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_1306_BaO	qmcpack_team__dmc_002	MAE	2.35	QMCPACK	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_1312_BP	qmcpack_team__dmc_001	MAE	4.85	QMCPACK	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_1390_SrO	qmcpack_team__dmc_002	MAE	1.69	QMCPACK	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_1405_CaO	qmcpack_team__dmc_002	MAE	0.38	QMCPACK	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_14590_Hf	qmcpack_team__dmc_003	MAE	0.4	QMCPACK	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_14604_Ba	vasp_optpbe	MAE	0.1	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_14606_Ag	vasp_opt86b	MAE	1.6	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_14612_Zr	qmcpack_team__dmc_003	MAE	4.0	QMCPACK	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_14813_Rb	vasp_pbe_mp	MAE	0.1	MP	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_182_SiC	vasp_optcx13	MAE	7.1	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_20290_Sc2O3	qmcpack_team__dmc_002	MAE	29.65	QMCPACK	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_20326_NaF	vasp_opt86b	MAE	0.5	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_20793_Y2O3	qmcpack_team__dmc_002	MAE	23.7	QMCPACK	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_21208_Sr	vasp_opt86b	MAE	0.0	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_23862_NaCl	vasp_opt86b	MAE	0.3	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_23864_LiCl	vasp_optb88vdw	MAE	0.13	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_25065_Li	vasp_optcx13	MAE	0.0	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_25114_K	vasp_optpbe	MAE	0.1	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_25180_Ca	vasp_opt86b	MAE	0.5	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_25213_Kr	qmcpack_team__dmc_001	MAE	0.24	QMCPACK	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_25248_Xe	qmcpack_team__dmc_001	MAE	0.19	QMCPACK	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_34249_HfO2	qmcpack_team__dmc_003	MAE	55.0	QMCPACK	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_350_ZrO2	qmcpack_team__dmc_003	MAE	93.0	QMCPACK	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_7809_ZrO2	qmcpack_team__dmc_003	MAE	30.0	QMCPACK	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_7870_MnO	qmcpack_team__dmc_002	MAE	10.0	QMCPACK	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_7871_NiO	qmcpack_team__dmc_002	MAE	5.0	QMCPACK	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_79204_BN	qmcpack_team__dmc_001	MAE	10.86	QMCPACK	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_816_Al	vasp_optcx13	MAE	0.2	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_819_Ar	qmcpack_team__dmc_001	MAE	0.42	QMCPACK	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_834_Be	qmcpack_team__dmc_001	MAE	2.37	QMCPACK	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_867_Cu	vasp_optb88vdw	MAE	0.6	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_87128_CoO	qmcpack_team__dmc_002	MAE	1.0	QMCPACK	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_890_Ge	qmcpack_dmc_pbe	MAE	2.1838	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_9147_HfO2	qmcpack_team__dmc_003	MAE	8.0	QMCPACK	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_91_C	vasp_optcx13	MAE	4.0	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_92796_La2O3	qmcpack_team__dmc_002	MAE	20.34	QMCPACK	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_963_Pd	vasp_optcx13	MAE	5.1	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_984_Rh	vasp_optb88vdw	MAE	8.3	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_cubic_lattice_param_a	vasp_optb88vdw	MAE	0.0491	JARVIS	dft_3d	64
ES	SinglePropertyPrediction	vacancydb_ef	vasp_optb88vdw	MAE	0.3026	JARVIS	vacancydb	39
ES	SinglePropertyPrediction	dft_3d_epsx	vasp_optb88vdw_linopt	MAE	1.4638	JARVIS	dft_3d	16
ES	SinglePropertyPrediction	dft_3d_epsx_JVASP_1312_BP	vasp_optb88vdw_linopt	MAE	1.8931	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_epsx_JVASP_1327_AlP	vasp_optb88vdw_linopt	MAE	1.0729	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_epsx_JVASP_1393_GaP	vasp_optb88vdw_linopt	MAE	0.4888	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_epsx_JVASP_1408_AlSb	vasp_optb88vdw_linopt	MAE	0.3367	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_epsx_JVASP_1702_ZnS	vasp_optb88vdw_dfpt	MAE	1.3908	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_epsx_JVASP_17_BN	vasp_optb88vdw_linopt	MAE	0.307	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_epsx_JVASP_182_SiC	vasp_optb88vdw_linopt	MAE	0.3837	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_epsx_JVASP_2355_ZnGeP2	vasp_optb88vdw_dfpt	MAE	0.2855	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_epsx_JVASP_2376_ZnSiP2	vasp_optb88vdw_linopt	MAE	0.1707	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_epsx_JVASP_23_CdTe	vasp_optb88vdw_linopt	MAE	2.9228	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_epsx_JVASP_54_MoS2	vasp_optb88vdw_linopt	MAE	0.8649	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_epsx_JVASP_57_MoSe2	vasp_optb88vdw_linopt	MAE	0.5049	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_epsx_JVASP_60_Te2Mo	vasp_optb88vdw_linopt	MAE	0.7269	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_epsx_JVASP_72_WS2	vasp_optb88vdw_dfpt	MAE	2.4098	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_epsx_JVASP_75_WSe2	vasp_optb88vdw_dfpt	MAE	3.7241	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_epsx_JVASP_8041_HgTe	vasp_optb88vdw_linopt	MAE	3.2251	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_max_piezoelec	vasp_optb88vdw_dfpt	MAE	1.0853	JARVIS	dft_3d	16
ES	SinglePropertyPrediction	dft_3d_max_piezoelec_JVASP_110_BaTiO3	vasp_optb88vdw_dfpt	MAE	5.109	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_max_piezoelec_JVASP_1180_InN	vasp_optb88vdw_dfpt	MAE	0.0086	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_max_piezoelec_JVASP_1195_ZnO	vasp_optb88vdw_dfpt	MAE	0.7625	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_max_piezoelec_JVASP_1240_LiNbO3	vasp_optb88vdw_dfpt	MAE	0.7074	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_max_piezoelec_JVASP_1327_AlP	vasp_optb88vdw_dfpt	MAE	0.7648	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_max_piezoelec_JVASP_1372_AlAs	vasp_optb88vdw_dfpt	MAE	0.8201	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_max_piezoelec_JVASP_1408_AlSb	vasp_optb88vdw_dfpt	MAE	0.6711	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_max_piezoelec_JVASP_20778_BeO	vasp_optb88vdw_dfpt	MAE	0.4868	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_max_piezoelec_JVASP_30_GaN	vasp_optb88vdw_dfpt	MAE	0.313	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_max_piezoelec_JVASP_3450_TiPbO3	vasp_optb88vdw_dfpt	MAE	4.3759	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_max_piezoelec_JVASP_35711_GaSb	vasp_optb88vdw_dfpt	MAE	0.84	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_max_piezoelec_JVASP_39_AlN	vasp_optb88vdw_dfpt	MAE	1.1309	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_max_piezoelec_JVASP_41_SiO2	vasp_optb88vdw_dfpt	MAE	0.0115	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_max_piezoelec_JVASP_57695_BN	vasp_optb88vdw_dfpt	MAE	0.4967	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_max_piezoelec_JVASP_7648_ZnS	vasp_optb88vdw_dfpt	MAE	0.3974	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_max_piezoelec_JVASP_8047_ZnSe	vasp_optb88vdw_dfpt	MAE	0.4686	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_slme	vasp_tbmbj	MAE	5.0925	JARVIS	dft_3d	5
ES	SinglePropertyPrediction	dft_3d_slme_JVASP_1174_GaAs	vasp_tbmbj	MAE	3.94	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_slme_JVASP_266_InP	vasp_tbmbj	MAE	9.14	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_slme_JVASP_7112_H6PbCI3N	vasp_tbmbj	MAE	0.52	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_slme_JVASP_7757_CdTe	vasp_tbmbj	MAE	6.77	JARVIS	dft_3d	1
ES	SinglePropertyPrediction	dft_3d_slme_JVASP_8554_InCuSe2	vasp_tbmbj	MAE	8.35	JARVIS	dft_3d	1
ES	Spectra	dft_3d_dielectric_function	vasp_tbmbj	MULTIMAE	2.88	JARVIS	dft_3d	4
ES	Spectra	dft_3d_dielectric_function_JVASP_1002_Si	vasp_tbmbj	MULTIMAE	3.3922	JARVIS	dft_3d	1
ES	Spectra	dft_3d_dielectric_function_JVASP_1174_GaAs	vasp_tbmbj	MULTIMAE	2.672	JARVIS	dft_3d	1
ES	Spectra	dft_3d_dielectric_function_JVASP_266_InP	vasp_tbmbj	MULTIMAE	2.5309	JARVIS	dft_3d	1
ES	Spectra	dft_3d_dielectric_function_JVASP_890_Ge	vasp_tbmbj	MULTIMAE	2.9248	JARVIS	dft_3d	1
EXP	SinglePropertyPrediction	dft_3d_Tc_supercon_JVASP_1151_MgB2	ppms_magnetization	MAE	0.22	PML-PPMS	dft_3d	1
EXP	SinglePropertyPrediction	dft_3d_Tc_supercon_JVASP_19679_ZrN	ppms_magnetization	MAE	0.1	PML-PPMS	dft_3d	1
EXP	SinglePropertyPrediction	dft_3d_Tc_supercon_JVASP_20166_Mo2C	ppms_magnetization	MAE	1.44	PML-PPMS	dft_3d	1
EXP	SinglePropertyPrediction	dft_3d_Tc_supercon_JVASP_30369_NbS2	ppms_magnetization	MAE	1.8	PML-PPMS	dft_3d	1
EXP	SinglePropertyPrediction	dft_3d_Tc_supercon_JVASP_31795_NbSe2	ppms_magnetization	MAE	1.1	PML-PPMS	dft_3d	1
EXP	SinglePropertyPrediction	dft_3d_Tc_supercon_JVASP_36335_NbN	ppms_magnetization	MAE	0.9	PML-PPMS	dft_3d	1
EXP	SinglePropertyPrediction	dft_3d_Tc_supercon_JVASP_45_FeSe	ppms_magnetization	MAE	0.5	PML-PPMS	dft_3d	1
EXP	Spectra	dft_3d_XRD_JVASP_19821_MgB2	bruker_d8	MULTIMAE	0.02	MML-BrukerD8	dft_3d	1
EXP	Spectra	nist_isodb_co2_RM_8852	10.1007s10450-018-9958-x.Lab01	MULTIMAE	0.0213	FACTlab	nist_isodb	1
EXP	Spectra	midas_stress_strain_vibroscopy_kevlar129	favimat_5	MULTIMAE	0.014	FAVIMAT	midas	1
FF	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_1002_Si	Si.tersoff	MAE	1.4267	JARVIS-FF	dft_3d	1
FF	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_816_Al	NiAlH_jea.eam.alloy	MAE	0.0267	JARVIS-FF	dft_3d	1
FF	SinglePropertyPrediction	dft_3d_bulk_modulus_JVASP_867_Cu	Mendelev_Cu2_2012.eam.fs	MAE	0.8367	JARVIS-FF	dft_3d	1
FF	SinglePropertyPrediction	biobench_deltaF	tiwarylab	MAE	0.9477	tiwarylab	biobench	3
FF	SinglePropertyPrediction	biobench_left_handed_population	tiwarylab	MAE	0.1087	tiwarylab	biobench	3
FF	SinglePropertyPrediction	biobench_right_handed_population	tiwarylab	MAE	0.0307	tiwarylab	biobench	3
FF	SinglePropertyPrediction	lj_2d_liquid_viscosity	lammps_2d_liquid_einstein	MAE	0.0244	LAMMPS	lj_2d_liquid	1
QC	EigenSolver	dft_3d_electron_bands_JVASP_816_Al_WTBH	qiskit_vqd_SU2_c6	MULTIMAE	0.003	JARVIS	dft_3d	1

A tree diagram for directory and file structure

How to contribute

1. How to add a contribution (csv.zip):

For a short demo, checkout this google colab-notebook

Prerequisites (for those not familiar with GitHub):

A GitHub username and a basic familiarity with GitHub is needed to work with the JARVIS-Leaderboard. See below:

If you do not have a GitHub account, sign up for one at: https://github.com/signup Related article: https://www.toolsqa.com/git/how-to-create-github-account/

Suppose you choose your username as “knc6”, you’ll have a webpage at https://github.com/knc6 . Ofcourse, choose some other username of your choice, but for the following demo, we will use the “knc6” as an example username.

Once you have the GitHub account, go to the website https://github.com/usnistgov/jarvis_leaderboard. Here Fork (make your own copy) of the JARVIS-Leaderbord repo by clicking on the “Fork” button. Also, feel free to click on the “Star” button if you like the JARVIS-Leaderboard project.

Now, go to your forked version of the JARVIS-Leaderboard. For a username “knc6”, you’ll have a repo at https://github.com/knc6/jarvis_leaderboard. Notice, we have “knc6” in the URL instead of usnistgov. Ofcourse, use your own username instead of knc6.

Now, we will make a pull request (request to contribute your edits in the main repo). If you are not familiar with pull requests checkout this link

If successful with basic GitHub setup, follow the guidelines:

1. Go to your forked jarvis_leaderboard repo.

2. git clone https://github.com/USERNAME/jarvis_leaderboard. NOTE: use your own GitHub USERNAME, e.g. knc6, instead of usnistgov

   Note if you do not use forked version, you won't be able to make a pull request

3. cd jarvis_leaderboard

4. conda create --name leaderboard python=3.8
5. source activate leaderboard
6. Install the package: python setup.py develop
7. Let's add a contribution for Silicon bandgap using DFT PBE (an Electronic structure approach)

cd jarvis_leaderboard/contributions/

mkdir vasp_pbe_teamX , you can give any reaosnable name to the benchmark folder in place of vasp_pbe_teamX

cd vasp_pbe_teamX

cp ../vasp_optb88vdw/ES-SinglePropertyPrediction-bandgap_JVASP_1002_Si-dft_3d-test-mae.csv.zip .

vi ES-SinglePropertyPrediction-bandgap_JVASP_1002_Si-dft_3d-test-mae.csv.zip

Note: do not change filenames, e.g., replace dft with qmc etc., which will cause errors

After pressing eneter twice, you'll see the file content as id,prediction

Just modify the predicting value to your model/measurement value

Save the file (":wq!" and ":q!")

Add metadata.json and run.sh files to capture metadata and enhance reproducibility. The metadata file must have at least your project_url and model_name info. The project_url couls be a publication/GitHub page etc.

Note: An admin will run your run.sh to check if he/she can reproduce your benchmark

Now, cd ../../../

python jarvis_leaderboard/rebuild.py

which will compile all data, compare with reference dataset and calculate metrices

Hoping there's no error, try: mkdocs serve

Ensure vasp_pbe_teamX row exists at:

http://127.0.0.1:8000/usnistgov/jarvis_leaderboard/ES/SinglePropertyPrediction/bandgap_JVASP_1002_Si/

Now add changes, git add jarvis_leaderboard/contributions/vasp_pbe_teamX

Commit your changes, git commit -m 'Adding my PBE Si result.'

git push

Now go to your forked github repo and make a pull reuqest (PR) to usnistgov/jarvis_leaderboard in develop branch

If you are not familiar with pull requests checkout this link

Note: only admins are allowed to make pull requests to main branch

Once an admin approve the PR, you'll see your results on the official leaderboard

1. Another example for AI mode as follows:

Populate the dataset for a benchmark, e.g.:

python jarvis_leaderboard/populate_data.py --benchmark_file AI-SinglePropertyPrediction-exfoliation_energy-dft_3d-test-mae --output_path=Out

Currently, this script works for atomistic tasks only, addition of other tasks will be available soon.

Train you model(s), e.g.:

pip install alignn

wget https://raw.githubusercontent.com/usnistgov/alignn/main/alignn/examples/sample_data/config_example.json

train_folder.py --root_dir "Out" --config "config_example.json" --output_dir="temp"

Create a folder in the jarvis_leaderboard/contributions folder under respective submodule, e.g.:

mkdir contributions/my_awesome_model

Add comma-separated zip file (.csv.zip) file(s) corresponding to benchmark(s), e.g.:

cp temp/prediction_results_test_set.csv .

mv prediction_results_test_set.csv AI-SinglePropertyPrediction-exfoliation_energy-dft_3d-test-mae.csv

zip AI-SinglePropertyPrediction-exfoliation_energy-dft_3d-test-mae.csv AI-SinglePropertyPrediction-exfoliation_energy-dft_3d-test-mae.csv.zip

mv AI-SinglePropertyPrediction-exfoliation_energy-dft_3d-test-mae.csv.zip jarvis_leaderboard/contributions/my_awesome_model

Add metadata info in the metadata.json file, e.g.:

cp jarvis_leaderboard/contributions/alignn_models/metadata.json jarvis_leaderboard/contributions/my_awesome_model

Also, add a run.py, run.sh and Dockerfile scripts to reproduce the model predictions.

Run python jarvis_leaderboard/rebuild.py to check there are no errors

Run mkdocs serve to check if the new benchmark exists, e.g. at page http://127.0.0.1:8000/usnistgov/jarvis_leaderboard/AI/SinglePropertyPrediction/exfoliation_energy/

Add. commit and push your changes, e.g.:

git add jarvis_leaderboard/contributions/my_awesome_model

git commit -m 'Adding my awesome_model to jarvis_leaderboard

git push origin main

Make a pull request from your fork to the source repo at usnistgov/jarvis_leaderboard develop branch

Notes:

1. The word: SinglePropertyPrediction: task type, test, property: exfoliation_energy, dataset: dft_3d, method: AI, metric: mae have been joined with '-' sign. This format should be used for consistency in webpage generation.

2. The test data splits are pre-determined, if the exact test IDs are not used, then the code might result in errors.

2. How to add a new benchmark (json.zip):

1. Create a json.zip file in the jarvis_leaderboard/benchmarks folder under respective sub-category, e.g.:

   e.g. jarvis_leaderboard/benchmarks/AI/SinglePropertyPrediction/dft_3d_exfoliation_energy.json.zip.

   An example of a new json.zip (mydb_myprop.json) file can be found here as well.

2. In the .json file should have train, val, test keys with array of ids and their values.

   Note train and val can be empty dictionaries if the benchmarks are other than AI method

3. Add a .md file, e.g.: jarvis_leaderboard/docs/AI/SinglePropertyPrediction/exfoliation_energy.md. This is where contributions performers will be kept and website info will be generated.

4. An example for creating such a file is provided in: jarvis_leaderboard/benchmarks/AI/SinglePropertyPrediction/transform_from_figshare.py

5. Then follow the instructions for "Adding model benchmarks to existing dataset"

Notes: A new benchmark must be linked with a peer-reviewed article and must have a DOI to ensure a minimum quality assurance for the data. We recommend adding your large dataset in Figshare or similar repository and then integrate it in JARVIS-Tools We also recommend to use JARVIS-Tools for generating dataset/models/benchmarks which can help us maintain the benchmark for long term.
Methods used for generating the data and referece are given below:

Method used for results	Methods for comparison
EXP	EXP/ES/analytical results
ES	ES/EXP
FF	ES/EXP
QC	Classical/analytical results
AI	Test set data

Citation

Large Scale Benchmark of Materials Design Methods

Acronyms

1. MAE: Mean Absolute Error
2. ACC: Classification accuracy
3. MULTIMAE: MAE sum of multple entries, Euclidean distance
4. For names of datasets and associated propertiesm refer to datasets in JARVIS-Tools

Help

Ask a question/raise an issue on GitHub. You can also email Kamal Choudhary if needed (kamal.choudhary@nist.gov). However, we recommend using the GitHub issues for any questions/concerns.

License

This template is served under the NIST license.
Read the LICENSE file for more info.