Facilitating Language Technologies for Crisis Preparedness and Response

2024

1. NLP4PI

   

   From Text to Maps: LLM-Driven Extraction and Geotagging of Epidemiological Data

   Karlyn K. Harrod^*, Prabin Bhandari^*, and Antonios Anastasopoulos

   In Proceedings of the Third Workshop on NLP for Positive Impact code here , Nov 2024

   🏅 Abs DOI Bib PDF

   Best Paper Award

   Epidemiological datasets are essential for public health analysis and decision-making, yet they remain scarce and often difficult to compile due to inconsistent data formats, language barriers, and evolving political boundaries. Traditional methods of creating such datasets involve extensive manual effort and are prone to errors in accurate location extraction. To address these challenges, we propose utilizing large language models (LLMs) to automate the extraction and geotagging of epidemiological data from textual documents. Our approach significantly reduces the manual effort required, limiting human intervention to validating a subset of records against text snippets and verifying the geotagging reasoning, as opposed to reviewing multiple entire documents manually to extract, clean, and geotag. Additionally, the LLMs identify information often overlooked by human annotators, further enhancing the dataset‘s completeness. Our findings demonstrate that LLMs can be effectively used to semi-automate the extraction and geotagging of epidemiological data, offering several key advantages: (1) comprehensive information extraction with minimal risk of missing critical details; (2) minimal human intervention; (3) higher-resolution data with more precise geotagging; and (4) significantly reduced resource demands compared to traditional methods.

   @inproceedings{harrod-etal-2024-text,
     title = {From Text to Maps: {LLM}-Driven Extraction and Geotagging of Epidemiological Data},
     author = {Harrod, Karlyn K. and Bhandari, Prabin and Anastasopoulos, Antonios},
     editor = {Dementieva, Daryna and Ignat, Oana and Jin, Zhijing and Mihalcea, Rada and Piatti, Giorgio and Tetreault, Joel and Wilson, Steven and Zhao, Jieyu},
     booktitle = {Proceedings of the Third Workshop on NLP for Positive Impact},
     month = nov,
     year = {2024},
     address = {Miami, Florida, USA},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2024.nlp4pi-1.24/},
     doi = {10.18653/v1/2024.nlp4pi-1.24},
     pages = {258--270},
   }

2. ClimateNLP

   

   Unlearning Climate Misinformation in Large Language Models

   Michael Fore, Simranjit Singh, Chaehong Lee, Amritanshu Pandey, Antonios Anastasopoulos, and Dimitrios Stamoulis

   In Proceedings of the 1st Workshop on Natural Language Processing Meets Climate Change (ClimateNLP 2024), Aug 2024

   Abs DOI Bib PDF

   Misinformation regarding climate change is a key roadblock in addressing one of the most serious threats to humanity. This paper investigates factual accuracy in large language models (LLMs) regarding climate information. Using true/false labeled Q&A data for fine-tuning and evaluating LLMs on climate-related claims, we compare open-source models, assessing their ability to generate truthful responses to climate change questions. We investigate the detectability of models intentionally poisoned with false climate information, finding that such poisoning may not affect the accuracy of a model‘s responses in other domains. Furthermore, we compare the effectiveness of unlearning algorithms, fine-tuning, and Retrieval-Augmented Generation (RAG) for factually grounding LLMs on climate change topics. Our evaluation reveals that unlearning algorithms can be effective for nuanced conceptual claims, despite previous findings suggesting their inefficacy in privacy contexts. These insights aim to guide the development of more factually reliable LLMs and highlight the need for additional work to secure LLMs against misinformation attacks.

   @inproceedings{fore-etal-2024-unlearning,
     title = {Unlearning Climate Misinformation in Large Language Models},
     author = {Fore, Michael and Singh, Simranjit and Lee, Chaehong and Pandey, Amritanshu and Anastasopoulos, Antonios and Stamoulis, Dimitrios},
     editor = {Stammbach, Dominik and Ni, Jingwei and Schimanski, Tobias and Dutia, Kalyan and Singh, Alok and Bingler, Julia and Christiaen, Christophe and Kushwaha, Neetu and Muccione, Veruska and A. Vaghefi, Saeid and Leippold, Markus},
     booktitle = {Proceedings of the 1st Workshop on Natural Language Processing Meets Climate Change (ClimateNLP 2024)},
     month = aug,
     year = {2024},
     address = {Bangkok, Thailand},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2024.climatenlp-1.14/},
     doi = {10.18653/v1/2024.climatenlp-1.14},
     pages = {178--192},
   }

3. JAMIA

   Clinical risk prediction using language models: benefits and considerations

   Angeela Acharya, Sulabh Shrestha, Anyi Chen, Joseph Conte, Sanja Avramovic, Siddhartha Sikdar, Antonios Anastasopoulos, and Sanmay Das

   Journal of the American Medical Informatics Association, Feb 2024

   Abs DOI Bib PDF

   The use of electronic health records (EHRs) for clinical risk prediction is on the rise. However, in many practical settings, the limited availability of task-specific EHR data can restrict the application of standard machine learning pipelines. In this study, we investigate the potential of leveraging language models (LMs) as a means to incorporate supplementary domain knowledge for improving the performance of various EHR-based risk prediction tasks.We propose two novel LM-based methods, namely “LLaMA2-EHR” and “Sent-e-Med.” Our focus is on utilizing the textual descriptions within structured EHRs to make risk predictions about future diagnoses. We conduct a comprehensive comparison with previous approaches across various data types and sizes.Experiments across 6 different methods and 3 separate risk prediction tasks reveal that employing LMs to represent structured EHRs, such as diagnostic histories, results in significant performance improvements when evaluated using standard metrics such as area under the receiver operating characteristic (ROC) curve and precision-recall (PR) curve. Additionally, they offer benefits such as few-shot learning, the ability to handle previously unseen medical concepts, and adaptability to various medical vocabularies. However, it is noteworthy that outcomes may exhibit sensitivity to a specific prompt.LMs encompass extensive embedded knowledge, making them valuable for the analysis of EHRs in the context of risk prediction. Nevertheless, it is important to exercise caution in their application, as ongoing safety concerns related to LMs persist and require continuous consideration.

   @article{10.1093/jamia/ocae030,
     author = {Acharya, Angeela and Shrestha, Sulabh and Chen, Anyi and Conte, Joseph and Avramovic, Sanja and Sikdar, Siddhartha and Anastasopoulos, Antonios and Das, Sanmay},
     title = {Clinical risk prediction using language models: benefits and considerations},
     journal = {Journal of the American Medical Informatics Association},
     volume = {31},
     number = {9},
     pages = {1856-1864},
     year = {2024},
     month = feb,
     issn = {1527-974X},
     doi = {10.1093/jamia/ocae030},
     url = {https://doi.org/10.1093/jamia/ocae030},
     eprint = {https://academic.oup.com/jamia/article-pdf/31/9/1856/58868302/ocae030.pdf},
   }