Skip to content

Proceedings - Plain-Language Adaptation of Biomedical Abstracts 2024

Enhancing Accessibility of Medical Texts through Large Language Model-Driven Plain Language Adaptation

Ting-Wei Chang (Department of Computer Science, Information Engineering, National Taiwan University, Taiwan) Hen-Hsen Huang (Institute of Information Science, Academia Sinica, Taiwan) Hsin-Hsi Chen (Department of Computer Science, Information Engineering, National Taiwan University, Taiwan, AI Research Center (AINTU), National Taiwan University, Taiwan)

Abstract

This paper addresses the challenge of making complex healthcare information more accessible through automated Plain Language Adaptation (PLA). PLA aims to simplify technical medical language, bridging a critical gap between the complexity of healthcare texts and patients’ reading comprehension. Recent advances in Large Language Models (LLMs), such as GPT and BART, have opened new possibilities for PLA, especially in zero-shot and few-shot learning contexts where task-specific data is limited. In this work, we leverage the capabilities of LLMs such as GPT-4o-mini, Gemini-1.5-pro, and LLaMA for text simplification. Additionally, we incorporate Mixture-of-Agents (MoA) techniques to enhance adaptability and robustness in PLA tasks. Key contributions include a comparative analysis of prompting strategies, finetuning with QLoRA on different LLMs, and the integration of MoA technique. Our findings demonstrate the effectiveness of LLM-driven PLA, showcasing its potential in making healthcare information more comprehensible while preserving essential content.

Bibtex 
@inproceedings{ntu_nlp-trec2024-papers-proc-1,
    author = {Ting-Wei Chang (Department of Computer Science and Information Engineering, National Taiwan University, Taiwan) 
Hen-Hsen Huang (Institute of Information Science, Academia Sinica, Taiwan) 
Hsin-Hsi Chen (Department of Computer Science and Information Engineering, National Taiwan University, Taiwan, AI Research Center (AINTU), National Taiwan University, Taiwan)},
    title = {Enhancing Accessibility of Medical Texts through Large Language Model-Driven Plain Language Adaptation},
    booktitle = {The Thirty-Third Text REtrieval Conference Proceedings (TREC 2024), Gaithersburg, MD, USA, November 15-18, 2024},
    series = {NIST Special Publication},
    volume = {xxx-xxx},
    publisher = {National Institute of Standards and Technology (NIST)},
    year = {2024},
    trec_org = {ntu_nlp},
    trec_runs = {task2_moa_tier3_post, task2_moa_tier1_post, task2_moa_tier2_post, gemini-1.5-pro_demon5_replace-demon5, gemini-1.5-flash_demon5_replace-demon5, gpt-4o-mini _demon5_replace-demon5},
    trec_tracks = {plaba}
   url = {https://trec.nist.gov/pubs/trec33/papers/ntu_nlp.plaba.pdf}
}

MaLei at the PLABA Track of TAC-2024: RoBERTa for Task 1 – LLaMA3.1 and GPT-4o for Task 2

Zhidong Ling, Zhihao Li, Pablo Romero, Lifeng Han, Goran Nenadic

Abstract

This report is the system description of the \textsc{MaLei} team (\textbf{Manchester} and \textbf{Leiden}) for the shared task Plain Language Adaptation of Biomedical Abstracts (PLABA) 2024 (we had an earlier name BeeManc following the last year).

This report contains two sections corresponding to the two sub-tasks in PLABA-2024. In task one, we applied fine-tuned ReBERTa-Base models to identify and classify the difficult terms, jargon, and acronyms in the biomedical abstracts and reported the F1 score. Due to time constraints, we didn't finish the replacement task. In task two, we leveraged Llamma3.1-70B-Instruct and GPT-4o with the one-shot prompts to complete the abstract adaptation and reported the scores in BLEU, SARI, BERTScore, LENS, and SALSA. From the official Evaluation from PLABA-2024 on Task 1A and 1B, our \textbf{much smaller fine-tuned RoBERTa-Base} model ranked 3rd and 2nd respectively on the two sub-tasks, and the \textbf{1st on averaged F1 scores across the two tasks} from 9 evaluated systems. Our LLaMA-3.1-70B-instructed model achieved the \textbf{highest Completeness} score for Task-2. We share our source codes, fine-tuned models, and related resources at \url{https://github.com/HECTA-UoM/PLABA-2024}

Bibtex 
@inproceedings{UM-trec2024-papers-proc-1,
    author = {Zhidong Ling, Zhihao Li, Pablo Romero, Lifeng Han, Goran Nenadic},
    title = {{MaLei} at the PLABA Track of TAC-2024: RoBERTa for Task 1 -- LLaMA3.1 and GPT-4o for Task 2},
    booktitle = {The Thirty-Third Text REtrieval Conference Proceedings (TREC 2024), Gaithersburg, MD, USA, November 15-18, 2024},
    series = {NIST Special Publication},
    volume = {xxx-xxx},
    publisher = {National Institute of Standards and Technology (NIST)},
    year = {2024},
    trec_org = {UM},
    trec_runs = {GPT, LLaMa 3.1 70B instruction (2nd run), Roberta-base},
    trec_tracks = {plaba}
   url = {https://trec.nist.gov/pubs/trec33/papers/UM.plaba.pdf}
}

MaLei at the PLABA Track of TAC-2024: RoBERTa for Task 1 – LLaMA3.1 and GPT-4o for Task 2

Zhidong Ling, Zihao Li, Pablo Romero, Lifeng Han, Goran Nenadic

Abstract

This report is the system description of the \textsc{MaLei} team (\textbf{Manchester} and \textbf{Leiden}) for shared task Plain Language Adaptation of Biomedical Abstracts (PLABA) 2024 (we had an earlier name BeeManc following last year).

This report contains two sections corresponding to the two sub-tasks in PLABA-2024. In task one, we applied fine-tuned ReBERTa-Base models to identify and classify the difficult terms, jargon and acronyms in the biomedical abstracts and reported the F1 score. Due to time constraints, we didn't finish the replacement task. In task two, we leveraged Llamma3.1-70B-Instruct and GPT-4o with the one-shot prompts to complete the abstract adaptation and reported the scores in BLEU, SARI, BERTScore, LENS, and SALSA. From the official Evaluation from PLABA-2024 on Task 1A and 1B, our \textbf{much smaller fine-tuned RoBERTa-Base} model ranked 3rd and 2nd respectively on the two sub-tasks, and the \textbf{1st on averaged F1 scores across the two tasks} from 9 evaluated systems. Our LLaMA-3.1-70B-instructed model achieved the \textbf{highest Completeness} score for Task-2. We share our source codes, fine-tuned models, and related resources at \url{https://github.com/HECTA-UoM/PLABA2024}

Bibtex 
@inproceedings{UM-trec2024-papers-proc-2,
    author = {Zhidong Ling, Zihao Li, Pablo Romero, Lifeng Han, Goran Nenadic},
    title = {{MaLei} at the PLABA Track of TAC-2024: RoBERTa for Task 1 -- LLaMA3.1 and GPT-4o for Task 2},
    booktitle = {The Thirty-Third Text REtrieval Conference Proceedings (TREC 2024), Gaithersburg, MD, USA, November 15-18, 2024},
    series = {NIST Special Publication},
    volume = {xxx-xxx},
    publisher = {National Institute of Standards and Technology (NIST)},
    year = {2024},
    trec_org = {UM},
    trec_runs = {GPT, LLaMa 3.1 70B instruction (2nd run), Roberta-base},
    trec_tracks = {plaba}
   url = {https://trec.nist.gov/pubs/trec33/papers/UM.plaba.pdf}
}

UM_FHS at TREC 2024 PLABA: Exploration of Fine-tuning and AI agent approach for plain language adaptations of biomedical text

Primoz Kocbek (University of Maribor)Leon Kopitar (University of Maribor)Zhihong Zhang (Columbia University)Emirhan Aydın (Manisa Celal Bayar University)Maxim Topaz (Columbia University)Gregor Stiglic (University of Maribor)

Abstract

This paper describes our submissions to the TREC 2024 PLABA track with the aim to simplify biomedical abstracts for a K8-level audience (13–14 years old students). We tested three approaches using OpenAI’s gpt-4o and gpt-4o-mini models: baseline prompt engineering, a two-AI agent approach, and fine-tuning. Adaptations were evaluated using qualitative metrics (5-point Likert scales for simplicity, accuracy, completeness, and brevity) and quantitative readability scores (Flesch-Kincaid grade level, SMOG Index). Results indicated that the two-agent approach and baseline prompt engineering with gpt-4o-mini models show superior qualitative performance, while fine-tuned models excelled in accuracy and completeness but were less simple. The evaluation results demonstrated that prompt engineering with gpt-4o-mini outperforms iterative improvement strategies via two-agent approach as well as fine-tuning with gpt-4o. We intend to expand our investigation of the results and explore advanced evaluations.

Bibtex 
@inproceedings{um_fhs-trec2024-papers-proc-1,
    author = {Primoz Kocbek (University of Maribor)
Leon Kopitar (University of Maribor)
Zhihong Zhang (Columbia University)
Emirhan Aydın (Manisa Celal Bayar University)
Maxim Topaz (Columbia University)
Gregor Stiglic (University of Maribor)},
    title = {UM_FHS at TREC 2024 PLABA: Exploration of Fine-tuning and AI agent approach for plain language adaptations of biomedical text},
    booktitle = {The Thirty-Third Text REtrieval Conference Proceedings (TREC 2024), Gaithersburg, MD, USA, November 15-18, 2024},
    series = {NIST Special Publication},
    volume = {xxx-xxx},
    publisher = {National Institute of Standards and Technology (NIST)},
    year = {2024},
    trec_org = {um_fhs},
    trec_runs = {plaba_um_fhs_sub1, plaba_um_fhs_sub2, plaba_um_fhs_sub3},
    trec_tracks = {plaba}
   url = {https://trec.nist.gov/pubs/trec33/papers/um_fhs.plaba.pdf}
}

MaLei at the PLABA Track of TREC 2024: RoBERTa for Term Replacement – LLaMA3.1 and GPT-4o for Complete Abstract Adaptation

Zhidong Ling, Zihao Li, Pablo Romero, Lifeng Han, Goran Nenadic

Abstract

This report is the system description of the \textsc{MaLei} team (\textbf{Manchester} and \textbf{Leiden}) for shared task Plain Language Adaptation of Biomedical Abstracts (PLABA) 2024 (we had an earlier name BeeManc following last year).

This report contains two sections corresponding to the two sub-tasks in PLABA-2024. In task one, we applied fine-tuned ReBERTa-Base models to identify and classify the difficult terms, jargon and acronyms in the biomedical abstracts and reported the F1 score. Due to time constraints, we didn't finish the replacement task. In task two, we leveraged Llamma3.1-70B-Instruct and GPT-4o with the one-shot prompts to complete the abstract adaptation and reported the scores in BLEU, SARI, BERTScore, LENS, and SALSA. From the official Evaluation from PLABA-2024 on Task 1A and 1B, our \textbf{much smaller fine-tuned RoBERTa-Base} model ranked 3rd and 2nd respectively on the two sub-tasks, and the \textbf{1st on averaged F1 scores across the two tasks} from 9 evaluated systems. Our LLaMA-3.1-70B-instructed model achieved the \textbf{highest Completeness} score for Task-2. We share our source codes, fine-tuned models, and related resources at \url{https://github.com/HECTA-UoM/PLABA2024}

Bibtex 
@inproceedings{UM-trec2024-papers-proc-3,
    author = {Zhidong Ling, Zihao Li, Pablo Romero, Lifeng Han, Goran Nenadic},
    title = {{MaLei} at the PLABA Track of TREC 2024: RoBERTa for Term Replacement -- LLaMA3.1 and GPT-4o for Complete Abstract Adaptation},
    booktitle = {The Thirty-Third Text REtrieval Conference Proceedings (TREC 2024), Gaithersburg, MD, USA, November 15-18, 2024},
    series = {NIST Special Publication},
    volume = {xxx-xxx},
    publisher = {National Institute of Standards and Technology (NIST)},
    year = {2024},
    trec_org = {UM},
    trec_runs = {GPT, LLaMa 3.1 70B instruction (2nd run), Roberta-base},
    trec_tracks = {plaba}
   url = {https://trec.nist.gov/pubs/trec33/papers/UM.plaba.pdf}
}

Biomedical Text Simplification Models Trained on Aligned Abstracts and Lay Summaries

Jan Bakker (University of Amsterdam)Taiki Papandreou-Lazos (University of Amsterdam)Jaap Kamps (University of Amsterdam)

Abstract

This paper documents the University of Amsterdam’s participation in the TREC 2024 Plain Language Adaptation of Biomedical Abstracts (PLABA) Track. We investigated the effectiveness of text simplification models trained on aligned pairs of sentences in biomedical abstracts and plain language summaries. We participated in Task 2 on Complete Abstract Adaptation and conducted post-submission experiments in Task 1 on Term Replacement. Our main findings are the following. First, we used text simplification models trained on aligned real-world scientific abstracts and plain language summaries. We observed better performance for the context-aware model relative to the sentence-level model. Second, our experiments show the value of training on external corpora and demonstrate very reasonable out-of-domain performance on the PLABA data. Third, more generally, our models are conservative and cautious in gratuitous edits or information insertions. This approach ensures the fidelity of the generated output and limits the risk of overgeneration or hallucination.

Bibtex 
@inproceedings{UAmsterdam-trec2024-papers-proc-2,
    author = {Jan Bakker (University of Amsterdam)
Taiki Papandreou-Lazos (University of Amsterdam)
Jaap Kamps (University of Amsterdam)},
    title = {Biomedical Text Simplification Models Trained on Aligned Abstracts and Lay Summaries},
    booktitle = {The Thirty-Third Text REtrieval Conference Proceedings (TREC 2024), Gaithersburg, MD, USA, November 15-18, 2024},
    series = {NIST Special Publication},
    volume = {xxx-xxx},
    publisher = {National Institute of Standards and Technology (NIST)},
    year = {2024},
    trec_org = {UAmsterdam},
    trec_runs = {UAms-ConBART-Cochrane, UAms-BART-Cochrane},
    trec_tracks = {plaba}
   url = {https://trec.nist.gov/pubs/trec33/papers/UAmsterdam.plaba.pdf}
}

Biomedical Text Simplification Models Trained on Aligned Abstracts and Lay Summaries

Jan Bakker (University of Amsterdam)Taiki Papandreou-Lazos (University of Amsterdam)Jaap Kamps (University of Amsterdam)

Abstract

This paper documents the University of Amsterdam’s participation in the TREC 2024 Plain Language Adaptation of Biomedical Abstracts (PLABA) Track. We investigated the effectiveness of text simplification models trained on aligned pairs of sentences in biomedical abstracts and plain language summaries. We participated in Task 2 on Complete Abstract Adaptation and conducted post-submission experiments in Task 1 on Term Replacement. Our main findings are the following. First, we used text simplification models trained on aligned real-world scientific abstracts and plain language summaries. We observed better performance for the context-aware model relative to the sentence-level model. Second, our experiments show the value of training on external corpora and demonstrate very reasonable out-of-domain performance on the PLABA data. Third, more generally, our models are conservative and cautious in gratuitous edits or information insertions. This approach ensures the fidelity of the generated output and limits the risk of overgeneration or hallucination.

Bibtex 
@inproceedings{UAmsterdam-trec2024-papers-proc-3,
    author = {Jan Bakker (University of Amsterdam)
Taiki Papandreou-Lazos (University of Amsterdam)
Jaap Kamps (University of Amsterdam)},
    title = {Biomedical Text Simplification Models Trained on Aligned Abstracts and Lay Summaries},
    booktitle = {The Thirty-Third Text REtrieval Conference Proceedings (TREC 2024), Gaithersburg, MD, USA, November 15-18, 2024},
    series = {NIST Special Publication},
    volume = {xxx-xxx},
    publisher = {National Institute of Standards and Technology (NIST)},
    year = {2024},
    trec_org = {UAmsterdam},
    trec_runs = {UAms-ConBART-Cochrane, UAms-BART-Cochrane},
    trec_tracks = {plaba}
   url = {https://trec.nist.gov/pubs/trec33/papers/UAmsterdam.plaba.pdf}
}