Reading Notes

Personal reading notes. Original paper: arXiv:2606.07172v1.

arxiv_id: 2606.07172v1

Paper Reading Note: Textual Supervision Enhances Geospatial Representations in Vision-Language Models

Paper Overview

Research Motivation and Background

The paper points out that geospatial understanding is a crucial yet underexplored dimension in the development of machine learning systems, particularly in tasks such as image geolocation and spatial reasoning. Although visual models have made tremendous progress over the past decade, such as Convolutional Neural Networks (CNNs) and Vision Transformers (ViT), there is currently a lack of systematic research on whether these models implicitly encode location information during training.

The authors propose that with the development of Vision-Language Models (VLMs) and multimodal foundation models, the complexity of internal representations has increased, making it more difficult to interpret how knowledge is encoded. This opacity can lead to unexpected results, such as geographic disparities reflecting uneven generalization between regions. To improve fairness and transparency, the paper aims to investigate the extent to which pure vision models and vision-language models internalize global location knowledge during training and fine-tuning.

Methodology: Linear Probing Analysis

To assess whether potential features related to geospatial tasks exist within the internal layers of the models, the authors adopted a Linear Probing method. This method aims to analyze whether the model’s internal layers have learned geospatial structures without additional geospatial supervision.

The paper designed an analysis framework as shown in Figure 1. The authors extract the residuals of the [CLS] token or the last token from each layer of the model, denoted as $A(l)$. Subsequently, they fit a Ridge Regression model $W$ to attempt to extract geospatial information (i.e., latitude and longitude) from these hidden dimensions. By calculating the $R^2$ value, the authors evaluate the feasibility of extracting geographic information from specific layers.

Figure 1: Schematic of the Linear Probing Analysis Setup. The authors fit a Ridge Regression model $W$ to the residuals $A(l)$ of each layer $l$ to investigate whether geospatial information (i.e., latitude and longitude) can be extracted from these hidden dimensions based on the $R^2$ value.

Core Findings

Through the analysis of three model families—pure vision architectures (e.g., ViT), vision-language models (e.g., CLIP), and large-scale multimodal foundation models (e.g., LLaVA, Qwen, Gemma)—the paper reveals the following key findings:

1. Textual Supervision Enhances Geospatial Representations: The paper argues that textual supervision can enhance the learning of geospatial representations. Language serves as an effective complementary modality that aids in encoding spatial context.
2. Multimodal Models Perform Better: The study finds that vision-language models and large-scale multimodal foundation models exhibit stronger geospatial structures compared to pure vision architectures.
3. Inter-layer Representation Differences:
   • Pure Vision Models: Tend to exhibit stronger geospatial representations in the final layers.
   • Vision-Language Models (VLMs): Possess better geospatial representations in the early layers of their language model blocks.
4. Role of Prompting: The authors show that prompting VLMs allows geospatial information to propagate to subsequent layers, leading to improved representation quality in some cases.

Experimental Validation and Representation Editing

To comprehensively evaluate the performance of different model representations, the authors conducted evaluations on various image clusters, including people, landmarks, and everyday objects, grouped by degree of localizability. This setup helps reveal systematic gaps in spatial accuracy.

Additionally, the paper explores the editability of geospatial representations. The authors edit geospatial representations through dimension swapping to observe how this operation affects the model’s perception or output. Figure 6 illustrates a schematic of this process, indicating that by modifying specific geospatial dimensions, the model’s perception or prediction of an image’s geographic location can be altered.

Figure 6: Schematic of Geospatial Representation Editing. This figure demonstrates how editing geospatial representations via dimension swapping changes perception or prediction results.

Summary and Outlook

The main contribution of this paper lies in the systematic investigation of geospatial representations learned by pure vision architectures, vision-language models, and large-scale multimodal foundation models. The research results indicate that multimodal learning is a key direction for advancing geospatial AI.

The paper suggests that language plays an effective complementary modality role in encoding spatial context. Future research should focus further on how to reduce geographic bias through multimodal learning, improve model generalization capabilities across different regions, and enhance the transparency of internal geographic knowledge encoding in models. This work provides important empirical evidence for understanding how foundation models implicitly learn world knowledge.