From Spatial Language to Spatial Data - a simulation-based approach
funded by the NSF (2021-)
Project Description
User experience related to spatial information is currently directly linked to the representation of the data; that is, geographic co-ordinates pinpoint locations on maps and routing algorithms determine the best route based on distance and time. In contrast, human interaction with the world is based on experience, learning and reasoning on qualitative factors, including spatial concepts, such as near/far, behind, next to, inside. Considering how spatial information is conveyed in natural language, there is no unique mapping between the spatial expressiveness and quantifiable spatial concepts. For the most part this is attributed to the highly contextualized nature of human language; that is, what human language is interpreted in part by who and where it was said and what other words surrounded the comment. The challenge in this project is on devising means to better understand people’s perception of space by deciphering such spatial language terms. This will lead to novel text and audio-based interfaces for the consumption of geospatial data such as when asking for or giving directions in a way that is intuitive to people or for systems that more effectively assist the visually impaired.
The technical aims of the project are divided into two thrusts. The first thrust develops a simulation to crowdsource geospatial language expression data by having users interact in a virtual environment. The spatial language expressions and interactions are captured using quantitative models. The second thrust then uses these user-generated descriptions to evaluate several modeling approaches that include (i) studying specific urban settings to identify contextual factors and (ii) exploring neural approaches to modeling the problem of grounding language to this spatial context. The resulting models can then be used to automatically translate language to geospatial information and, in the reverse direction, to train dialogue agents that can generate enriched, contextualized route and scene descriptions with natural, useful geospatial language expressions.
In urban science, understanding mobility patterns and analyzing how people move around cities helps improve the overall quality of life and supports the development of more livable, efficient, and sustainable urban areas. A challenging aspect of this work is the collection of mobility data through user tracking or travel surveys, given the associated privacy concerns, noncompliance, and high cost. This work proposes an innovative AI-based approach for synthesizing travel surveys by prompting large language models (LLMs), aiming to leverage their vast amount of relevant background knowledge and text generation capabilities. Our study evaluates the effectiveness of this approach across various U.S. metropolitan areas by comparing the results against existing survey data at different granularity levels. These levels include (i) pattern level, which compares aggregated metrics such as the average number of locations traveled and travel time, (ii) trip level, which focuses on comparing trips as whole units using transition probabilities, and (iii) activity chain level, which examines the sequence of locations visited by individuals. Our work covers several proprietary and open-source LLMs, revealing that open-source base models like Llama-2, when fine-tuned on even a limited amount of actual data, can generate synthetic data that closely mimics the actual travel survey data and, as such, provides an argument for using such data in mobility studies.
@inproceedings{bhandari-etal-24-urban,author={Bhandari, Prabin and Anastasopoulos, Antonios and Pfoser, Dieter},title={Urban Mobility Assessment Using LLMs},year={2024},month=nov,isbn={9798400711077},publisher={Association for Computing Machinery},address={New York, NY, USA},url={https://doi.org/10.1145/3678717.3691221},doi={10.1145/3678717.3691221},booktitle={Proceedings of the 32nd ACM International Conference on Advances in Geographic Information Systems},pages={67–79},numpages={13},keywords={Large Language Models, Travel Data, Travel Survey, Travel Survey Data Simulation},location={Atlanta, GA, USA},series={SIGSPATIAL '24},}
SIGSPATIAL
Trajectory Anomaly Detection with Language Models
Jonathan Kabala Mbuya, Dieter Pfoser, and Antonios Anastasopoulos
In Proceedings of the 32nd ACM International Conference on Advances in Geographic Information Systems, Atlanta, GA, USA, Nov 2024
This paper presents a novel approach for trajectory anomaly detection using an autoregressive causal-attention model, termed LM-TAD. This method leverages the similarities between language statements and trajectories, both of which consist of ordered elements requiring coherence through external rules and contextual variations. By treating trajectories as sequences of tokens, our model learns the probability distributions over trajectories, enabling the identification of anomalous locations with high precision. We incorporate user-specific tokens to account for individual behavior patterns, enhancing anomaly detection tailored to user context. Our experiments demonstrate the effectiveness of LM-TAD on both synthetic and real-world datasets. In particular, the model outperforms existing methods on the Pattern of Life (PoL) dataset by detecting user-contextual anomalies and achieves competitive results on the Porto taxi dataset, highlighting its adaptability and robustness. Additionally, we introduce the use of perplexity and surprisal rate metrics for detecting outliers and pinpointing specific anomalous locations within trajectories. The LM-TAD framework supports various trajectory representations, including GPS coordinates, staypoints, and activity types, proving its versatility in handling diverse trajectory data. Moreover, our approach is well-suited for online trajectory anomaly detection, significantly reducing computational latency by caching key-value states of the attention mechanism, thereby avoiding repeated computations. The code to reproduce experiments in this paper can be found at the following link: https://github.com/jonathankabala/LMTAD.
@inproceedings{10.1145/3678717.3691257,author={Mbuya, Jonathan Kabala and Pfoser, Dieter and Anastasopoulos, Antonios},title={Trajectory Anomaly Detection with Language Models},year={2024},month=nov,isbn={9798400711077},publisher={Association for Computing Machinery},address={New York, NY, USA},url={https://doi.org/10.1145/3678717.3691257},doi={10.1145/3678717.3691257},booktitle={Proceedings of the 32nd ACM International Conference on Advances in Geographic Information Systems},pages={208–219},numpages={12},keywords={Anomalous Trajectories, Anomaly Detection, Language Modeling, Self-Supervised Learning, Trajectory Data},location={Atlanta, GA, USA},series={SIGSPATIAL '24},}
2023
SIGSPATIAL
Are Large Language Models Geospatially Knowledgeable?
Prabin Bhandari, Antonios Anastasopoulos, and Dieter Pfoser
In Proceedings of the 31st ACM International Conference on Advances in Geographic Information Systems, Hamburg, Germany, Oct 2023
Despite the impressive performance of Large Language Models (LLM) for various natural language processing tasks, little is known about their comprehension of geographic data and related ability to facilitate informed geospatial decision-making. This paper investigates the extent of geospatial knowledge, awareness, and reasoning abilities encoded within such pretrained LLMs. With a focus on autoregressive language models, we devise experimental approaches related to (i) probing LLMs for geo-coordinates to assess geospatial knowledge, (ii) using geospatial and non-geospatial prepositions to gauge their geospatial awareness, and (iii) utilizing a multidimensional scaling (MDS) experiment to assess the models’ geospatial reasoning capabilities and to determine locations of cities based on prompting. Our results confirm that it does not only take larger but also more sophisticated LLMs to synthesize geospatial knowledge from textual information. As such, this research contributes to understanding the potential and limitations of LLMs in dealing with geospatial information.
@inproceedings{bhandari-etal-23-geospatially,author={Bhandari, Prabin and Anastasopoulos, Antonios and Pfoser, Dieter},title={Are Large Language Models Geospatially Knowledgeable?},year={2023},month=oct,isbn={9798400701689},publisher={Association for Computing Machinery},address={New York, NY, USA},url={https://doi.org/10.1145/3589132.3625625},doi={10.1145/3589132.3625625},booktitle={Proceedings of the 31st ACM International Conference on Advances in Geographic Information Systems},articleno={75},numpages={4},keywords={geospatial reasoning, geospatial awareness, geospatial knowledge, large language models},location={Hamburg, Germany},series={SIGSPATIAL '23},}
MRL
Geographic and Geopolitical Biases of Language Models
Fahim Faisal, and Antonios Anastasopoulos
In Proceedings of the 3rd Workshop on Multi-lingual Representation Learning (MRL) Code here , Dec 2023
@inproceedings{faisal-anastasopoulos-2023-geographic,title={Geographic and Geopolitical Biases of Language Models},author={Faisal, Fahim and Anastasopoulos, Antonios},editor={Ataman, Duygu},booktitle={Proceedings of the 3rd Workshop on Multi-lingual Representation Learning (MRL)},month=dec,year={2023},address={Singapore},publisher={Association for Computational Linguistics},url={https://aclanthology.org/2023.mrl-1.12/},doi={10.18653/v1/2023.mrl-1.12},pages={139--163},}
INLG
Trustworthiness of Children Stories Generated by Large Language Models
Prabin Bhandari, and Hannah Brennan
In Proceedings of the 16th International Natural Language Generation Conference Code here , Sep 2023
Large Language Models (LLMs) have shown a tremendous capacity for generating literary text. However, their effectiveness in generating children‘s stories has yet to be thoroughly examined. In this study, we evaluate the trustworthiness of children‘s stories generated by LLMs using various measures, and we compare and contrast our results with both old and new children‘s stories to better assess their significance. Our findings suggest that LLMs still struggle to generate children‘s stories at the level of quality and nuance found in actual stories.
@inproceedings{bhandari-brennan-2023-trustworthiness,title={Trustworthiness of Children Stories Generated by Large Language Models},author={Bhandari, Prabin and Brennan, Hannah},editor={Keet, C. Maria and Lee, Hung-Yi and Zarrie{\ss}, Sina},booktitle={Proceedings of the 16th International Natural Language Generation Conference},month=sep,year={2023},address={Prague, Czechia},publisher={Association for Computational Linguistics},url={https://aclanthology.org/2023.inlg-main.24/},doi={10.18653/v1/2023.inlg-main.24},pages={352--361}}
