ORIGINAL RESEARCH article

Front. Earth Sci., 03 September 2026

Sec. Geoinformatics

Volume 14 - 2026 | https://doi.org/10.3389/feart.2026.1884556

MM-DIPR: perception-aware multimodal learning for tourism destination image understanding and recommendation

  • 1. Faculty of Humanities and Social Sciences, Yanbian University, Yanbian, Jilin, China

  • 2. Finance Department, Hubei Institute of Technology, Huangshi, Hubei, China

Abstract

Tourism destination image encompasses the multifaceted perceptions of visitors that influence their choice of destination. However, current computational methodologies often analyze visual and textual signals separately and seldom link image perception to the quality of recommendations. This study introduces multimodal destination image perception and recommendation (MM-DIPR), a framework utilizing public data that amalgamates visual encoding, natural language processing, geospatial context, and graph-based ranking into a cohesive perception-aware pipeline. MM-DIPR discerns seven interpretable perception attributes, namely, landmarkness, naturalness, cultural/historical salience, esthetic appeal, crowding proxy, sentiment affect, and activity affordance, through a dedicated perception bottleneck that facilitates cross-modal alignment and graph ranking. We establish a reproducible benchmark using solely public and openly licensed datasets (Google Landmarks Dataset v2, YFCC100M, Places365, Yelp Open Dataset, TREC Contextual Suggestion, Wikimedia/Wikidata/Wikivoyage, Overture Maps Places, and SNAP Gowalla/Brightkite) and assess MM-DIPR against 11 baselines (including spatiotemporal and knowledge-graph POI models) across six complementary tasks, namely, perception classification, image–text retrieval, personalized point-of-interest (POI) recommendation, cold-start recommendation, cross-city transfer, and explanation faithfulness. A formal spatial autocorrelation analysis confirms that all seven perception attributes exhibit statistically significant positive spatial autocorrelation (Moran’s ; ), grounding the framework’s geospatial design choices. Experimental findings indicate that the perception bottleneck significantly enhances ranking accuracy (Recall@10 of 0.213 and NDCG@10 of 0.152 on the Yelp benchmark; ) and cold-start robustness compared to strong multimodal baselines. Ablation studies validate the independent contribution of each module, while diversity analysis reveals that calibrated re-ranking enhances long-tail exposure with minimal accuracy loss. A controlled annotation study with 2,000 images and three annotators confirms substantial inter-annotator agreement for all attributes. Qualitative explanation panels ground recommendations in public visual and textual evidence, thus providing practical utility for destination management practitioners. All experimental code and derived benchmark artifacts will be made available to support reproducibility.

1 Introduction

The selection of a tourism destination represents one of the most consequential decisions in consumer behavior (; ; ). Visitors allocate significant financial and temporal resources based on expectations increasingly influenced by digital artifacts, including user-generated photographs, online reviews, travel guides, and, more recently, algorithmic recommendation feeds (; Zheng et al., 2015). The concept of tourism destination image (TDI) encompasses the cognitive and affective impressions a potential visitor holds about a destination, and a substantial body of tourism research has demonstrated its predictive power over both the intention to visit and visitor satisfaction (; ).

Despite this extensive theoretical foundation, computational approaches to modeling destination image have evolved largely independently of recommendation systems, with limited interaction. Image analysis in tourism has predominantly focused on classifying or clustering the visual content of destination photographs (Stepchenkova and Zhan, 2013; ), while recommendation systems optimize ranking objectives without incorporating interpretable perceptual constructs (Zhang and Chen, 2020). This disciplinary divide leads to two practical challenges. First, recommender systems lack perceptually grounded representations that could elucidate why a destination ranks highly for a given user, a limitation with direct implications for trust and destination management. Second, perception research does not readily generalize to ranking, cold-start, or cross-destination transfer, complicating the utilization of insights in operational platforms.

An additional, equally significant issue is data reproducibility. Much of the existing literature on tourism recommendation relies on proprietary or scraped datasets from commercial platforms, whose licensing terms, population biases, and long-term availability are uncertain (Zangerle and Bauer, 2022). This constrains replication, external audit, and fair comparison across studies.

This study addresses both the challenges by proposing MM-DIPR, a multimodal destination image perception and recommendation framework that:

  • Introduces an explicit perception bottleneck that facilitates the learning of seven interpretable TDI attributes, serving as an intermediate representation shared between cross-modal alignment and graph-based personalized ranking; the bottleneck is validated by a controlled annotation study showing substantial inter-annotator agreement ( for all attributes).

  • Develops a spatially grounded, transparent, and reproducible benchmark utilizing solely publicly available and openly licensed datasets, including open geospatial data (Overture Maps Places, Wikidata, and Wikivoyage), with a formal spatial autocorrelation analysis (Moran’s ) confirming that perception attributes exhibit statistically significant spatial structure.

  • Assesses the quality of perception, retrieval, recommendation, cold-start robustness, cross-city generalization, diversity, and explanation faithfulness within a unified experimental framework, with an expanded baseline set that includes spatiotemporal and knowledge-graph POI recommendation models.

  • Offers licensing-aware data pipeline code and derived benchmark artifacts to facilitate future research endeavors.

The remainder of this article is organized as follows: Section 2 reviews related work. Section 3 describes the public dataset construction. Section 4 formalizes the tasks. Section 5 presents the MM-DIPR architecture. Section 6 details the experimental setup. Section 7 reports main, cold-start, cross-city, and ablation results. Section 8 analyses diversity, explanation, and error. Section 9 discusses implications, and Section 10 concludes.

2 Related work

2.1 Tourism destination image and user-generated content

Theoretical frameworks for TDI typically differentiate between cognitive (attribute-based), affective (feeling-based), and conative (behavioral intention) components (). The proliferation of digital photography and social media has positioned user-generated photographs and reviews as pivotal in image formation, both for individual tourists and destination management organizations (; Stepchenkova and Zhan, 2013). Computational studies have utilized visual analysis to classify the esthetics of tourism photographs (Zhang et al., 2022), deduce perceived destination attributes from Flickr images (Yang et al., 2014), and analyze textual sentiment in review corpora (). These studies illustrate that perception is quantifiable; yet, they infrequently link perception to subsequent recommendation tasks.

2.2 Multimodal representation learning

The domain of vision–language representation learning has progressed rapidly. CLIP () demonstrated that contrastive pre-training on image–text pairs at a web scale produces representations that are transferable to a variety of downstream tasks. BLIP-2 () introduced a lightweight querying transformer to connect frozen image encoders with large language models. DINOv2 () revealed that self-supervised pre-training with curated data generates universal visual features that outperform supervised alternatives across numerous tasks. SigLIP (Zhai et al., 2023) enhanced contrastive learning with sigmoid losses (). These advancements provide a robust foundation for encoding tourism visual content; however, their direct application to TDI necessitates domain-specific adaptation concerning perception attributes absent in generic pre-training distributions.

2.3 Point-of-interest and tourism recommendation

Point-of-interest (POI) recommendation has been extensively examined through collaborative filtering (Rendle et al., 2009), visually-aware ranking (), and graph neural network methodologies (; Wang et al., 2019). Context-aware and spatiotemporal extensions integrate time, mobility, and sequential behavior (; ). Tourism-specific research has analyzed hybrid content-collaborative methods (; ) and addressed cold-start challenges stemming from sparse coverage of less-visited attractions (Zhang and Chen, 2020; ). Explainability in recommender systems has emerged as a distinct research trajectory (Zhang and Chen, 2020; Yaqoob et al., 2023), although the evaluation of faithfulness assessing whether the cited evidence genuinely influences recommendations remains insufficiently explored.

2.4 Responsible data practices in recommendation research

Concerns regarding reproducibility have led to calls for standardized, openly licensed benchmark datasets within the recommender systems community (Zangerle and Bauer, 2022; ). Tourism datasets, in particular, are often derived from proprietary platform APIs or scraped under ambiguous terms, thereby limiting auditability and replication (Zhang et al., 2023). Our research directly addresses this issue by constructing a benchmark exclusively from public sources with documented licensing, preprocessing, and version control.

3 Public dataset construction

3.1 Dataset selection principles

All datasets utilized in this study must adhere to four criteria: (1) the dataset must be publicly accessible without the need for proprietary access agreements; (2) the license or terms of service must allow for academic research use; (3) the specific extraction version and access date must be documented; and (4) the dataset must not contain any records that necessitate consent revocation under applicable privacy regulations. Data scraped from commercial tourism platforms (e.g., TripAdvisor and Booking.com) and Kaggle mirrors of uncertain provenance are explicitly excluded.

3.2 Dataset overview

Table 1 provides a summary of the public sources utilized. Collectively, these sources offer the visual, textual, geospatial, interaction, and entity-linking modalities required for the six tasks formalized in Section 4.

TABLE 1

Dataset/SourceModalitiesRole in studyLicense/Caveat
Google Landmarks Dataset v2 (GLDv2) (Weyand et al., 2020)Landmark images, recognition/retrieval ground truthVisual encoder tuning; retrieval evaluationApache 2.0 image list; validate URL availability
YFCC100M (Thomee et al., 2016)Creative Commons Flickr metadata, titles, tags, and geoGeotagged tourism UGC; weak image–text alignmentCC and Yahoo terms; document actual retrieval yield
Places365 (Zhou et al., 2018)1.8 M scene images, 365 categoriesScene perception pre-training/evaluationMIT license; labels are scene categories, not TDI-specific
Yelp Open Dataset (Yelp Inc, 2023)Business, reviews, photos, check-ins, and attributesPOI recommendation and text–image alignmentAcademic use; filter to tourism categories
TREC Contextual Suggestion ()User profiles, POI suggestions, and relevance judgmentsPersonalized POI evaluation with reusable test collectionsNIST public; cite exact track year
Wikimedia/Wikidata/Wikivoyage (Vrandečić and Krötzsch, 2014; Wikivoyage contributors, 2023)Freely licensed media, structured POI metadata, and travel textPOI knowledge graph; image provenanceCC-BY, CC0, and CC-BY-SA; license differs per content type
Overture Maps Places ()Open POI entities, categories, and geometryEntity normalization; tourism-category filteringCDLA Permissive 2.0; version every extraction
SNAP Gowalla/Brightkite ()Check-in records, location, and timeImplicit feedback baseline; cold-start stress testsPublic for research; use aggregated/anonymous analysis

Dataset and source matrix for MM-DIPR are publicly accessible and verified for academic research licensing. It is imperative that extraction versions and any associated caveats are updated at the time of data download.

3.3 Entity construction and linking

The point-of-interest (POI) entity table is developed by incorporating the Overture Maps Places release (version 2024-04-16.0, extracted 2024-05-01) and standardizing records with Wikidata entity identifiers where applicable. Duplicate entries across sources are resolved through coordinate-radius matching with a 50-m threshold, followed by string similarity filtering using the Jaro–Winkler method with a threshold of 0.92 (yielding a de-duplication yield of 8.3% of raw records across all cities). Wikivoyage and Wikipedia guide texts are linked to POI entities via Wikidata QIDs, offering multilingual textual descriptions. Yelp business records are aligned with entity table entries through coordinate matching and fuzzy category alignment. SNAP check-in records are spatially joined to the entity table within a 100-m radius to derive implicit user–POI interaction matrices. Geographic context features attached to each entity include: (i) decimal-degree WGS84 coordinates; (ii) a city-level region identifier; (iii) a three-level Overture Maps category hierarchy (e.g., eat_and_drink > restaurant > italian_restaurant); (iv) a Wikivoyage region tag used as a soft spatial prior during graph construction.

3.4 Image–text linking and privacy filtering

Images associated with each POI entity are sourced from GLDv2 landmark identifiers, YFCC100M geotag-radius lookups, and Wikimedia Commons category pages. All images undergo a three-stage filtering process: (1) license verification against an approved list of Creative Commons and equivalent terms; (2) face detection and blurring using a publicly available OpenCV DNN detector to mitigate inadvertent personal data exposure; and (3) near-duplicate removal via perceptual hashing with a pHash Hamming distance of eight or less. Text associated with each entity combines Wikivoyage sentences, Wikipedia lead paragraphs, and Yelp review aggregates (tip summaries, not raw reviews, to limit re-identification risk). The dataset construction pipeline, including all filter thresholds and random seeds, is released with the article.

3.5 Benchmark splits

Three complementary split strategies are constructed. In-city split: a random 70/10/20 train/validation/test division within each city, preserving user–POI interaction distribution. Cold-start split: POIs with fewer than five training interactions are withheld from training and evaluated separately, stratified by the interaction count . Cross-city split: models are trained on all POIs and user interactions from Phoenix and Pittsburgh (training cities) and evaluated zero-shot on six unseen cities: Las Vegas, Chicago, Seattle, Boston, Austin, and Denver (test cities).

Table 2 reports the exact record counts per city and split after all filtering. Across all eight cities, the benchmark contains 40,542 unique POI entities, 283,946 images (after license and privacy filtering), 80,306 unique users, and 1,418,622 check-in interactions. Image source distribution across the benchmark is reported in Figure 1.

TABLE 2

CityPOIsImagesUsersCheck-insRole
Phoenix4,81238,49011,240187,431Train
Pittsburgh3,26726,1387,820130,280Train
Las Vegas5,14041,12012,308204,150Test
Chicago6,02348,17814,421239,623Test
Seattle4,39835,18210,521175,073Test
Boston5,21141,68812,480207,625Test
Austin3,78930,3129,056150,900Test
Denver3,10224,8167,420123,540Test
Total40,542283,94680,3061,418,622

Detailed per-city dataset statistics after all filtering steps. The raining cities are Phoenix and Pittsburgh; the remaining cities are used exclusively for cross-city evaluation.

FIGURE 1

4 Problem formulation

Let be the set of users, be the set of destination/POI items, be the set of public images associated with item , be the set of textual evidence (reviews, descriptions, and tags), and be the geospatial and category attributes of item . Let represent the contextual history for user .

Task 1: Destination image perception: Learning a perception function that maps multimodal item evidence to a -dimensional vector of interpretable perception attributes (e.g., landmarkness and naturalness) and a dense latent perception embedding.

Task 2: Image–text retrieval: Given a query image (or text ), retrieving the top- most relevant text descriptions (or images) from the POI corpus, evaluated by Recall@, MAP@, and MRR.

Task 3: Personalized POI recommendation: Learning a scoring function and return a ranked list of items for each user, evaluated by Recall@, NDCG@, and MAP@.

Task 4: Cold-start generalization: Evaluating on new POIs with no training interactions, measuring the performance by interaction stratum and relative improvement over interaction-only baselines.

Task 5: Cross-city transfer: Evaluating on POIs and users from cities entirely unseen during training (see Section 7.4 for the full protocol, clarification of the zero-shot scope, user overlap, and inference procedure).

Task 6: Explanation faithfulness: For each recommended item, generating an evidence tuple of representative images, text snippets, and perception attributes. Evaluating faithfulness by a deletion protocol: removing cited evidence, re-scoring the item, and measuring the score drop relative to random-evidence deletion.

The joint training objective is:where (i) is a binary cross-entropy loss over binarized (above/below median) perception attribute labels; (ii) is a contrastive InfoNCE image–text alignment loss (Equation 2); (iii) is a Bayesian personalized ranking (BPR) loss (Rendle et al., 2009), optimizing the relative ordering of observed over unobserved interactions; (iv) is the expected calibration error (ECE) penalizing overconfident attribute predictions; and (v) is the negative mean pairwise cosine distance in perception embedding space, applied only at the re-ranking stage to promote diverse top- selections. Hyper-parameters are tuned on the validation split; their sensitivity is analyzed in Section 6.6 and Figure 2.

FIGURE 2

5 The MM-DIPR framework

Figure 3 provides a high-level overview of MM-DIPR, and Figure 4 provides details of the module interactions and data flow.

FIGURE 3

FIGURE 4

5.1 Module M1: licensing-aware entity construction

A comprehensive POI entity table is constructed utilizing data from Overture Maps Places and Wikidata (refer to Section 3). Each entity record encompasses coordinates, a normalized name, a Wikidata QID (when available), a tourism category derived from Overture and Wikivoyage hierarchies, and a media license manifest. This manifest is instrumental in guiding the downstream image filter to ensure that no unlicensed images are incorporated into the training or evaluation processes.

5.2 Module M2: visual encoding

Images associated with each POI are encoded using two complementary backbones: a CLIP ViT-B/32 encoder () for semantic image–language grounding and a DINOv2 ViT-S/14 encoder () for capturing robust low-level and mid-level visual features. A Places365-style scene head (Zhou et al., 2018) appends scene-category logits, while a GLDv2-trained landmark head appends landmark probability scores. The final image embedding for POI is achieved through the L2-normalized concatenation of these four representations, followed by a learned projection to a dimension of .

5.3 Module M3: textual encoding

The text associated with each POI, including Wikivoyage sentences, Wikipedia lead paragraphs, Yelp tip aggregates, and tag strings from YFCC100M, is encoded using a multilingual pre-trained transformer [mDeBERTa-v3-base ()]. Sentence-level embeddings are mean-pooled within each text type, and type-specific embeddings are integrated via a learned gating network into a unified -dimensional text embedding.

5.4 Module M4: cross-modal alignment

Image and text embeddings are aligned with a contrastive InfoNCE loss (van den Oord et al., 2018):where is a learned temperature. Hard negatives are sampled from geographically nearby or visually similar POIs to discourage trivial separation and encourage within-domain discrimination (Robinson et al., 2021).

5.5 Module M5: perception bottleneck

The perception bottleneck is designed to learn a set of interpretable perception attributes, corresponding to theoretically grounded TDI dimensions (): (1) landmarkness, (2) naturalness, (3) cultural/historical salience, (4) esthetic appeal, (5) crowding proxy, (6) sentiment affect, and (7) activity affordance. Weak supervision is obtained from several sources: (i) Places365 scene logits for naturalness and activity affordance, (ii) GLDv2 landmark scores for landmarkness, (iii) Wikivoyage category tags for cultural/historical salience, and (iv) Yelp sentiment aggregates for affect. The bottleneck head comprises a two-layer MLP that generates sigmoid-activated scores from the fused multimodal embedding. These scores serve as both interpretable outputs and regularizers, ensuring that the latent POI embedding retains perception-relevant structure.

Relationship to POI evaluation categories. The seven perception attributes are distinct from the six POI evaluation categories (landmark, natural scenery, cultural/historical, beach/resort, museum/gallery, and urban attractions) used in the classification experiment. Perception attributes are continuous scores that any POI can exhibit to varying degrees, whereas POI categories are discrete labels used to organize the evaluation benchmark. Table 3 maps each attribute to its primary POI category association and weak supervision signal.

TABLE 3

AttributeTDI dimensionWeak supervisionPrimary POI category
LandmarknessCognitive (attribute)GLDv2 landmark scoresLandmark
NaturalnessCognitive (attribute)Places365 natural scenesNatural scenery
Cultural/historical salienceCognitive (attribute)Wikivoyage category tagsCultural/historical
Asthetic appealAffectivePlaces365 esthetic proxyBeach/resort
Crowding proxyCognitive (attribute)YFCC100M density estimatesUrban attractions
Sentiment affectAffectiveYelp sentiment aggregatesMuseum/gallery
Activity affordanceConative (behavior)Places365 activity scenesAll categories

Perception attribute–category correspondence, TDI theory dimension, weak supervision source, and expected category association.

5.6 Module M6: graph recommendation

A heterogeneous graph is constructed with users and POIs as nodes, incorporating four types of edges: user–POI interactions, POI–POI category similarity, image–POI association, and geospatial proximity (Euclidean distance km, weighted by inverse distance). A LightGCN-style () propagation scheme is utilized to aggregate neighborhood information over two layers (justified by the sensitivity analysis in Figure 2) without non-linear activation, leading to propagated user and item embeddings. The ranking score is computed as in Equation 3:where denotes the aggregated embedding after graph propagation, is the initial user embedding (retained for training stability), is the perception embedding from Module M5 projected to dimension , and are jointly learned scalars. The additive injection of into the item embedding is deliberate: unlike concatenation, which would opaquely entangle perception with graph-propagated features, additive injection preserves the interpretability of the individual perception attribute scores while allowing the ranker to modulate their influence via . The ablation study (Section 7; Figure 5) shows that setting (w/o Perc) causes a 13.6% decrease in point in absolute Recall@10.

FIGURE 5

5.7 Module M7: diversity re-ranking

In accordance with the maximal marginal relevance (MMR) framework (), a re-ranking procedure is utilized to select items from the top-50 candidates, aiming to optimize a convex combination of relevance and intra-list diversity, as regulated by . Diversity is quantified as the mean pairwise cosine distance within the perception embedding space, thereby encouraging the re-ranker to favor destinations with complementary perceptual profiles.

5.8 Module M8: explanation generation

For each recommended POI, the explanation module provides: (i) the top-three perception attributes ranked by magnitude, (ii) the public image from the POI’s image set that exhibits the highest alignment similarity (subject to licensing constraints), and (iii) the review sentence from the Yelp tip corpus that is most semantically similar. The faithfulness of the explanation is assessed through a deletion protocol: the top-cited image is removed from the visual input, and the resultant decrease in score is measured and compared to the mean decrease in score from 100 random-image deletions. An explanation is deemed faithful if the deletion of top-cited evidence leads to a significantly larger decrease in score than random deletions (paired -test; ).

5.9 Algorithm summary

This section presents the algorithmic summary of the MM-DIPR training procedure in Algorithm 1.

Algorithm 1

  • Require: Public POI table, public images, public texts, public check-in data, license manifest

  • Ensure: Trained parameters ; benchmark artefacts

  • 1: Data preparation: filter by license, version sources, apply privacy filter

  • 2: Entity linking: normalize POI entities; link images, text, and interactions

  • 3: Embedding: extract for all POIs (Modules M2–M3)

  • 4: Graph construction: build heterogeneous user–POI–image graph

  • 5: for each training epoch do

  • 6:  Sample mini-batch of (user, positive POI, negative POI) triples

  • 7:  Compute cross-modal alignment loss with hard negatives (M4)

  • 8:  Compute perception attribute scores; compute (M5)

  • 9:  Perform two-layer LightGCN propagation; compute ranking scores (M6)

  • 10:  Compute BPR loss

  • 11:  Update via AdamW with (Equation 1)

  • 12: end for

  • 13: Apply MMR diversity re-ranking at inference time (M7)

  • 14: Generate and evaluate evidence-grounded explanations (M8)

MM-DIPR training procedure.

5.10 Annotation protocol and construct validation

To validate the perception bottleneck’s seven attributes beyond weak supervision, we conducted a controlled annotation study on a stratified random sample of 2,000 images (approximately 333 images per POI category). Three trained annotators independently scored each image on all seven attributes using a five-point Likert scale (1 = not at all; 5 = strongly). Scores were binarized at the per-attribute median score (computed on the pilot set of 100 images) to yield binary present/absent labels before computing agreement statistics.

Annotation guidelines were developed iteratively through pilot rounds on 100 images that were not included in the final sample, achieving consensus on operationalized definitions for ambiguous attributes (e.g., crowding proxy operationalized as the presence of identifiable people or queue structures visible in the image; activity affordance operationalized as the range of observable physical activities that the scene supports). Disagreements were adjudicated by majority vote, and the majority-vote label was used as the consensus (gold) label for all downstream comparisons.

Inter-annotator agreement calculation: With three annotators, Cohen’s was computed as the mean of the three pairwise values (annotators A vs. B, A vs. C, and B vs. C) for each attribute, following the mean-pairwise approach of . We report the mean pairwise rather than Fleiss’s to maintain comparability with the two-annotator literature; reported values are averages over the three pairs.

Results (Figure 6) show that all seven attributes achieve substantial or near-perfect inter-annotator agreement , exceeding the widely adopted threshold of 0.61 for substantial agreement (). Crowding proxy and sentiment affect show the lowest agreement, consistent with their reliance on subjective interpretation; these are flagged as lower-confidence attributes.

FIGURE 6

F1 measures were defined. Two F1 scores are compared. Weak-supervision F1 is the macro-averaged F1 of the weak-label predictions (derived from Places365, GLDv2, Wikivoyage, and Yelp proxies, as described in Section 5) against the consensus annotation labels. Human-annotation F1 is the macro-averaged F1 of each individual annotator’s labels against the majority-vote consensus, averaged across the three annotators and then across attributes. Comparison of the weak-supervision F1 to human-annotation F1 reveals a maximum gap of 4.1% points (crowding proxy: 0.812 vs. 0.853), confirming that the public-data proxies provide reliable operational approximations of the intended constructs.

6 Experimental setup

6.1 Datasets and splits

The experiments utilize the Yelp Open Dataset (Yelp Inc., 2023) for personalized POI recommendation (Tasks 3–5) and the GLDv2+Wikimedia split for image–text retrieval (Task 2). Perception classification (Task 1) utilizes Places365 scene labels and a manually annotated subset of 2,000 images, incorporating the seven perception attributes detailed in Section 5. The TREC Contextual Suggestion 2016 () offers additional evaluation for recommendations through graded relevance judgments. The SNAP Gowalla dataset () provides supplementary check-in baselines.

Split sizes: For the in-city split, the two training cities (Phoenix and Pittsburgh) together yield 8,079 POIs, 317,711 check-in interactions, 19,060 unique users, and 64,628 image–text pairs, with a 70/10/20 train/validation/test partition (train: 5,655 POIs and 222,398 interactions; validation: 808 POIs and 31,771 interactions; test: 1,616 POIs, 63,542 interactions). For the cold-start split, 1,613 POIs (20% of training-city POIs, those with interactions) are withheld from training and used exclusively for cold-start evaluation. For the cross-city split, all six test cities contribute 28,663 POIs, 1,100,911 interactions, and 61,246 users; no interactions or user histories from these cities appear during training.

User deduplication: Users are identified by their hashed Yelp pseudonym. A user whose hashed identifier appears in more than one city is counted only once in the global user pool (80,306 total unique users), but their city-specific interaction records are retained separately. No user from a test city is present in the training split.

6.2 Baselines

The following seven baselines are used for the core comparison (Table 4):

TABLE 4

MethodR@5R@10R@20N@5N@10N@20
Popularity0.0420.0680.1010.0310.0380.049
BPR0.0730.1120.1630.0580.0710.089
VBPR0.0910.1380.1970.0740.0890.111
Text-only0.0880.1340.1910.0700.0860.107
Image-only0.0850.1300.1860.0680.0830.104
LightGCN0.1090.1610.2260.0870.1060.131
Late fusion0.1180.1740.2410.0960.1170.143
MM-DIPR0.1480.2130.2910.1270.1520.183
vs. late fusion+25.4%+22.4%+20.7%+32.3%+29.9%+28.0%

Complete results of the point-of-interest (POI) recommendation on the Yelp Open Dataset benchmark.

Bold text indicates the best result for each metric, while underlined text denotes the best baseline. All differences between MM-DIPR and the bold-underlined baselines are statistically significant (, paired bootstrap).

Popularity/category heuristics: The rank by global check-in count and tourism-category prior, a necessary tourism-specific baseline.

BPR (Rendle et al., 2009): Matrix factorization with Bayesian personalized ranking loss; interaction signals only.

VBPR (): Extends BPR with visual item factors from a pre-trained CNN; strong visual recommendation baseline.

Text-only: BPR-style ranker using mDeBERTa text embeddings as static item factors; no image or graph component.

Image-only. BPR-style ranker with CLIP ViT-B/32 image embeddings only.

LightGCN (): Graph-based collaborative filtering; no multimodal content.

Late fusion: Combines text-only and image-only ranking scores via learned linear interpolation after independent training.

MM-DIPR (ours): The full proposed framework.

Section 7.6 and Figure 7 additionally compare MM-DIPR against four stronger baselines: GeoMF () (geography-aware matrix factorization), STGCN-POI () (spatiotemporal graph convolution network for POI recommendation), SASRec () (self-attentive sequential recommender), and KGAT (Wang et al., 2019) (knowledge-graph attention network), yielding 11 total baselines (seven core + four extended). BERT4Rec (Sun et al., 2019) appears in Appendix A (Table A1) for completeness (it uses the same sequential self-attention paradigm as SASRec, but with a masked-language-model training objective), but it is not reported as a separate baseline in the main results to avoid redundancy with SASRec; its performance is Recall@10, between SASRec and KGAT.

FIGURE 7

TABLE A1

MethodRecall@10
(mean std)
NDCG@10
(mean std)
Popularity0.068 0.0000.038 0.000
BPR0.112 0.0020.071 0.002
VBPR0.138 0.0030.089 0.002
Text-only0.134 0.0030.086 0.002
Image-only0.130 0.0030.083 0.002
LightGCN0.161 0.0020.106 0.002
Late fusion0.174 0.0030.117 0.002
GeoMF0.158 0.0030.103 0.002
STGCN-POI0.166 0.0030.109 0.002
SASRec0.171 0.0030.114 0.002
BERT4Rec0.168 0.0030.111 0.002
KGAT0.163 0.0020.108 0.002
MM-DIPR0.2130.0020.1520.002

Mean standard deviation of Recall@10 and NDCG@10 over five random seeds (0–4) on the Yelp Open Dataset benchmark (in-city split). All differences between MM-DIPR and the best baseline are statistically significant (, paired bootstrap, ).

6.3 Ablations

Eight ablated variants isolate the contribution of each module: (i) without the perception bottleneck (w/o Perc), (ii) without the graph ranker (w/o Graph), (iii) without the geo/context branch (w/o Geo), (iv) without the text encoder (w/o Text), (v) without the visual encoder (w/o Visual), (vi) with shuffled image–text pairs as a sanity check (Shuffled), (vii) without diversity re-ranking (w/o Div), and (viii) without geographic edges in the heterogeneous graph (w/o GeoEdges), which removes geospatial proximity edges while retaining all other edge types (user–POI interactions, POI–POI category similarity, and image–POI association), directly testing the model-level contribution of explicit spatial relations in the graph structure (results are reported in Section 7.7).

6.4 Metrics

Perception metrics include macro-F1, AUROC, and calibration error (ECE). For retrieval, the metrics are Recall@5, MAP@10, and MRR. Recommendation metrics encompass Recall@, Precision@, NDCG@, and MAP@ for . Diversity is assessed through intra-list diversity (ILD: the mean pairwise cosine distance among top- recommendation embeddings), catalog coverage: the fraction of all POIs appearing in at least one recommendation list, and long-tail exposure: the fraction of recommendations drawn from POIs in the bottom 20% by interaction count. Explanation is evaluated using the faithfulness deletion score.

Statistical significance is determined using paired bootstrap tests with 10,000 resamples and a significance level of over users/queries. The paired bootstrap resamples users (with replacement), recomputes the per-method metric on each resample, and counts the proportion of resamples in which MM-DIPR does not exceed the baseline; this proportion is the one-sided -value. All tests are conducted at the user level, not the interaction level. All reported confidence intervals are 95% bootstrap confidence intervals computed over 10,000 resamples. The effect sizes are reported as Cohen’s for continuous comparisons and as percentage change relative to the best baseline for recommendation metrics.

Each model is run with five independent random seeds (seed ). The results in Table 4 report the mean; standard deviations across runs are reported in Appendix A (Table A1). Standard deviations for MM-DIPR are across all metrics, confirming stability.

6.5 Implementation details

The MM-DIPR model is implemented using PyTorch 2.2. Visual encoders are initialized from public HuggingFace checkpoints and fine-tuned with a learning rate that is lower than that of the projection layers. The graph ranker is trained using the AdamW optimizer with a learning rate of , a weight decay of , and a batch size of 2,048. The temperature parameter is initialized to 0.07 and is learned jointly. All experiments are conducted on a single NVIDIA A100 40 GB GPU. Hyper-parameters are selected through grid search based on the validation NDCG@10; the full search space is reported in Appendix A (Table A2), and the joint sensitivity of and is visualized in Figure 2. All baselines are tuned under the same validation protocol (grid search on validation NDCG@10; no test-set access during hyperparameter selection).

TABLE A2

ModelHyperparameterSearch spaceSelected
BPR/VBPREmbedding dim{64, 128, and 256}256
Learning rate{1e-3, 5e-4, and 1e-4}1e-3
L2 regularization{1e-4, 1e-5, and 0}1e-5
LightGCN/KGATEmbedding dim{64, 128, and 256}256
Graph layers {1, 2, and 3}2
Learning rate{1e-3, 5e-4, and 1e-4}1e-3
SASRec/BERT4RecHidden size{64, 128, and 256}128
Attention heads{1, 2, and 4}2
Dropout{0.1, 0.2, and 0.5}0.2
GeoMF/STGCN-POIEmbedding dim{64, 128, and 256}128
Spatial bandwidth{0.5, 1, and 2} km1 km
Learning rate{1e-3, 5e-4, and 1e-4}5e-4
MM-DIPR (ours)Projection dim {128, 256, 512, and 1,024}512
Graph layers {1, 2, 3, and 4}2
Temperature learned (init 0.07)learned
(align){0.01, 0.05, 0.10, 0.20, and 0.50}0.10
(rank){0.01, 0.05, 0.10, 0.20, and 0.50}0.10
(cal){0.01, 0.05, and 0.10}0.05
(div){0.1, 0.2, 0.3, 0.4, and 0.5}0.3
Learning rate{1e-3, 5e-4, and 1e-4}5e-4

Hyperparameter search spaces for all models.

Bold values mark the selected configuration for MM-DIPR. All baselines were tuned under the same protocol (grid search on validation NDCG@10, five seeds).

6.6 Hyperparameter sensitivity

Figure 2 reports a joint sensitivity grid over and (validation NDCG@10) and individual sensitivity curves for graph propagation depth and projection dimension . The selected configuration (, , , and ) is marked by a star. Performance is robust within a broad region around the optimum, supporting the generalizability of the results.

7 Results

7.1 Personalized POI recommendation

Figure 8 presents the Recall@10 and NDCG@10 metrics for all baseline models and MM-DIPR on the Yelp benchmark. MM-DIPR achieves a Recall@10 of 0.213 and an NDCG@10 of 0.152, indicating statistically significant improvements over the strongest baseline (late fusion, 0.174/0.117) by 22.4% and 29.9%, respectively (, Cohen’s , and , respectively). These enhancements are consistent across and on the TREC contextual suggestion split, demonstrating that the improvements are not artifacts of a specific evaluation depth.

FIGURE 8

Table 4 provides the full numerical results at .

7.2 Ablation study

Figure 5 illustrates the results of the ablation study. The removal of the perception bottleneck leads to the most significant decline for a single module (Recall@10: , ), indicating that the perception layer plays a crucial role beyond the mere fusion of raw multimodal embeddings. The elimination of the graph ranker leads to a similarly substantial decrease , whereas the removal of the geo/context branch, text encoder, or visual encoder leads to moderate yet notable degradations. The shuffled image–text variant, serving as a sanity check, deteriorates to a performance level close to the popularity baseline, thereby affirming the complexity of the alignment objective.

7.3 Cold-start performance

Figure 9 presents the Recall@10 metric, categorized by the number of interactions per POI. For POIs with no prior training interactions, known as strict cold-start scenarios, interaction-only methods such as BPR and LightGCN inherently lead to zero recall. In contrast, content-based methods maintain some level of signal. The MM-DIPR method achieves a Recall@10 of 0.061 for cold-start POIs, establishing a meaningful baseline facilitated by visual–perceptual representations. Furthermore, MM-DIPR surpasses late fusion by 9.7% points within the 1–5 interaction category. The relative benefit of MM-DIPR decreases as the number of interactions increases, indicating that the perception layer is particularly beneficial when collaborative signals are limited.

FIGURE 9

7.4 Cross-city transfer protocol

The term cross-city transfer in this article refers specifically to POI-level zero-shot transfer: the model is trained on all POIs and user interactions from Phoenix and Pittsburgh and is then evaluated on POIs from six unseen cities without any fine-tuning on those cities’ data. We clarify the following aspects explicitly in response to the reviewer’s concerns.

What is zero-shot? The zero-shot condition applies to cities and their POIs: no POI entity, image, text description, or graph structure from the six test cities (Las Vegas, Chicago, Seattle, Boston, Austin, and Denver) is used during training or model selection. The model relies entirely on transferable multimodal and spatial representations learned from the two training cities.

Test-user histories and user overlap: All test-city users are unseen during training (confirmed by global user deduplication, Section 6). Their interaction histories within test cities are available at inference time for ranking: the evaluation follows a leave-last-out protocol within each test user’s history (final interaction = positive test item; prior interactions = observable history). Users with fewer than three interactions in a test city are excluded from that city’s evaluation set. Users do not overlap across training and test cities.

What is not zero-shot? Test-city images and textual descriptions are encoded by the trained model at inference time (modality encoders generalize across cities by design). No graph propagation involves test-city nodes during training; the item embedding is computed from content features only, which explains why content-rich cities (Las Vegas and Boston) benefit most from the transfer.

Labeling: This evaluation is accurately described as cross-city transfer (trained on city A + B and evaluated on cities C–H) rather than user cold-start or full zero-shot learning. The six-city results should be treated as an exploratory analysis of geographic generalization; no causal claims about spatial factors are made.

7.5 Cross-city transfer results

For the experimental protocol of this evaluation, see Section 7.4 above. Figure 10 presents the NDCG@10 results for six previously unexamined test cities. The MM-DIPR model consistently surpasses all baseline models across these cities, with the most significant improvements observed in Las Vegas and Boston, where it achieves increases of 63% and 61% over BPR, respectively. The comparatively modest enhancements in Austin and Denver align with the lower density of POIs and the availability of images in these locations, indicating that perception-aware representations are most advantageous in image-rich destination contexts; a quantified analysis of the POI-density–performance relationship is provided in the spatial analysis (Section 7.7).

FIGURE 10

7.6 Extended baseline comparison (spatiotemporal and knowledge-graph models)

Figure 7 reports Recall@10 and NDCG@10 for all 11 baselines (seven core plus four extended). MM-DIPR (Recall@10 = 0.213) outperforms the best spatiotemporal model GeoMF (0.158) by 34.8%, the best sequential model SASRec (0.171) by 24.6%, and the knowledge-graph model KGAT (0.163) by 30.7% ( in all cases). The consistent advantage over geographically-aware (GeoMF) and graph-structured (KGAT and LightGCN) baselines confirms that the gains are attributable to the perception bottleneck and cross-modal alignment, not to any single architectural component.

7.7 Spatial autocorrelation analysis

To establish the geoinformatics foundation of the framework, we compute Moran’s () for each of the seven perception attributes across all 40,542 benchmark POIs. The spatial weights matrix is row-standardized and uses an inverse-distance kernel: if , else 0, where is the Euclidean distance between the POIs and in decimal degrees projected to meters. Significance is assessed by a permutation test with 999 random permutations of the attribute scores; the reported -values are the proportion of permuted Moran’s values exceeding the observed statistic (one-sided and upper tail).

Figure 11 presents the Moran’s scatter plot for landmarkness (panel a), per-attribute Moran’s values (panel b), a city model NDCG@10 heatmap (panel c), and a POI-density vs. NDCG@10 scatter (panel d). All seven attributes exhibit statistically significant positive autocorrelation (, all ), confirming that perception is not spatially random. Landmarkness shows the strongest autocorrelation , consistent with the geographic clustering of tourist attractions; crowding proxy shows the weakest autocorrelation , reflecting diffuse distribution of high-foot-traffic environments.

FIGURE 11

The POI-density–performance scatter (panel d) reveals a positive correlation between city-level POI count and MM-DIPR’s NDCG@10, confirming that geospatial data quality is a significant performance determinant. This correlation is based on six data points and should be interpreted as exploratory; it indicates a plausible relationship between data coverage and model performance but does not establish a causal mechanism. This finding has direct implications for usage in data-sparse regions.

Geographic-edges ablation: To directly assess the model contribution of the geospatial proximity edges in the heterogeneous graph (rather than merely post hoc spatial analysis), we compare MM-DIPR against the w/o GeoEdges ablation variant introduced in Section 6. Removing the geographic edges from the graph reduces Recall@10 from 0.213 to 0.198 (, ) and NDCG@10 from 0.152 to 0.141 (, ). This confirms that the spatial proximity structure encoded in the graph provides meaningful complementary signal beyond POI-category and image-based edges, grounding the Moran’s analysis in a model-level causal test.

7.8 Perception classification and retrieval

The perception bottleneck achieves a macro-F1 score of 0.894 on the six-category perception classification task (Figure 12). The highest per-category F1 score is observed for natural scenery (0.934), while the lowest is for historic sites (0.868), where visual ambiguity with Urban Landmarks constitutes the primary source of confusion. In the context of image–text retrieval (Figure 13), the MM-DIPR encoder surpasses CLIP ViT-B/32 by 9.3% points on Recall@5 and the CLIP + DINOv2 ensemble by 3.7% points on MAP@10. This demonstrates that domain-adapted contrastive training with perception-aware hard negatives enhances retrieval performance beyond that of off-the-shelf foundation models.

FIGURE 12

FIGURE 13

8 Analysis

8.1 Perception attribute profiles by POI category

In a comparative analysis of image–text retrieval across various encoder variants, the MM-DIPR encoder demonstrates superior performance, as evidenced by its highest scores in Recall@5, MAP@10, and MRR. The error bars indicate 95% bootstrap confidence intervals (GLDv2 + Wikimedia test split). Figure 14 presents radar plots of the mean perception attribute scores categorized by POIs. The profiles exhibit theoretically consistent patterns: nature and parks POIs achieve the highest scores in naturalness and esthetic appeal, historic sites excel in cultural/historical salience and landmarkness, and beaches/resorts attain the highest scores in sentiment affect. These cluster-level patterns substantiate the perception bottleneck’s capacity to discern TDI-relevant structures through weak supervision, even in the absence of explicit perception-label training data.

FIGURE 14

8.2 Accuracy–diversity trade-off

Figure 15 demonstrates the impact of varying the MMR re-ranking weight on NDCG@10, intra-list diversity (ILD), catalog coverage, and long-tail exposure. At , which was determined through validation, MM-DIPR achieves an ILD of 0.520 and a catalog coverage of 0.392, with only a 4.6% decrease in NDCG@10 compared to the configuration without re-ranking. Long-tail exposure increases monotonically with , aligning with the re-ranker’s propensity to promote less-frequented destinations with complementary perceptual profiles, an outcome directly pertinent to sustainable tourism policy.

FIGURE 15

8.3 Explanation faithfulness

The evaluation of deletion faithfulness reveals that the removal of the top-cited image leads to an average score reduction of 0.048 (SD = 0.019), in contrast to 0.011 (SD = 0.014) for random-image deletion (, , Cohen’s ). Regarding textual evidence, the removal of the top-cited review sentence leads to a score reduction of 0.031 compared to 0.009 for random deletion . These findings confirm that the evidence cited by Module M8 is causally relevant to the recommendation score, offering–stronger faithfulness assurances than attention–weight visualizations.

8.4 Error and bias analysis

Four systematic failure patterns have been identified. Over-touristed landmark bias: High-popularity landmarks (e.g., Eiffel Tower analogs in the data) are over-represented in Wikimedia images, inflating their landmarkness scores and biasing recommendations towards them. Although partially mitigated by the MMR re-ranker, this remains a residual concern. Nighttime and seasonal variability: The perception bottleneck’s scene head, pre-trained on daytime scenes, assigns inconsistent naturalness and esthetic appeal scores to night or winter images of the same POI. Low-resource cities: Cross-city transfer performance degrades more significantly for cities with sparse overture POI coverage, indicating that entity quality is a critical determinant of performance. Multilingual review noise: Non-English Yelp reviews outside the top languages are handled noisily by the mDeBERTa tokenizer, potentially diluting textual representations for non-Anglophone destinations.

Geographic bias analysis: The underperformance in Austin (NDCG@10 = 0.121) and Denver (NDCG@10 = 0.132) relative to the benchmark average (0.133) is quantifiably linked to lower POI entity density (3,789 and 3,102 POIs, respectively, vs. the benchmark average of 5,072 POIs for test cities). The POI-density–NDCG@10 correlation (, Figure 11, panel d) confirms this relationship. We propose that future geoinformatics-oriented extensions of this framework could apply spatial Kriging or Gaussian process regression to impute perception scores in data-sparse regions, leveraging the demonstrated spatial autocorrelation structure (Moran’s for all attributes) to propagate perception estimates from neighboring data-rich POIs.

9 Discussion

9.1 Theoretical contributions

The MM-DIPR framework operationalizes the cognitive–affective TDI model () as a computable bottleneck that links perception theory to ranking models. The perception attribute profiles (Figure 14) provide empirical validation for the multidimensionality postulate, demonstrating that different destination categories exhibit distinct perceptual signatures consistent with theoretical expectations, which can be extracted from public UGC without the need for survey data. This serves as a bridge between the qualitative tradition of TDI research and the quantitative tradition of recommender systems.

9.2 Practical contributions

For destination management organizations, the perception scores and explanation panels serve as a monitoring tool: variations in aggregate scores of landmarkness, naturalness, or esthetic appeal over time may indicate shifts in UGC-mediated destination image, facilitating proactive marketing responses. The diversity re-ranker’s capability to highlight long-tail destinations with high perceptual complementarity provides a technically sound mechanism for promoting lesser-known attractions within sustainable tourism strategies ().

9.3 Limitations

Three limitations require explicit acknowledgment. First, the benchmark is primarily derived from North American Yelp sub-graphs, which limits its generalizability to destinations in the Global South, where Overture and Wikidata coverage is less comprehensive. Second, while the controlled annotation study (Section 5.10) validates the perception attributes on a 2,000-image subset and achieves substantial inter-annotator agreement, the remaining 98% of benchmark images rely on weak supervision proxies; future work should extend human annotation to a larger, more geographically diverse image pool. Third, recommendation evaluation relies on implicit feedback (check-ins), which conflates visitation with satisfaction and may favor popular but average destinations over niche but highly satisfying ones.

9.4 Future work

Four directions are prioritized. Controlled user studies with explicit preference elicitation would complement the implicit evaluation. Multilingual perception modeling using cross-lingual text encoders could enhance coverage for non-Anglophone destinations. Real-time integration of trip constraints (transport, opening hours, and seasonal closures) would increase practical utility. Finally, causal analysis of the relationship between algorithmically promoted perception attributes and actual visitor flows using natural experiments around platform algorithm changes would advance the tourism policy implications.

10 Conclusion

This study introduces MM-DIPR, a perception-aware multimodal framework designed for the understanding and recommendation of tourism destination images. By incorporating a perception bottleneck that learns interpretable TDI attributes as an intermediate representation shared between cross-modal alignment and graph ranking, MM-DIPR demonstrates statistically significant improvements over 11 baseline models across six evaluation tasks, including cold-start and cross-city scenarios. Furthermore, it retains the capability to generate evidence-based explanations validated through a faithfulness deletion protocol. The framework is constructed entirely from publicly available, openly licensed datasets with documented provenance, and all experimental code and derived benchmark artifacts will be released to facilitate reproducibility. We anticipate that MM-DIPR will serve as both a methodological template for multimodal tourism research and a practical tool for destination management practitioners seeking perceptually grounded, transparent recommendation systems.

Statements

Data availability statement

The original contributions presented in the study are included in the article/supplementary material; further inquiries can be directed to the corresponding author.

Author contributions

LD: Conceptualization, Data curation, Formal Analysis, Funding acquisition, Investigation, Methodology, Project administration, Software, Supervision, Validation, Visualization, Writing – original draft, Writing – review and editing. QL: Conceptualization, Formal Analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Writing – original draft, Writing – review and editing.

Funding

The author(s) declared that financial support was not received for this work and/or its publication.

Conflict of interest

The author(s) declared that this work was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Generative AI statement

The author(s) declared that generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

  • 1

    ArtsteinR.PoesioM. (2008). Survey article: inter-coder agreement for computational linguistics. Comput. Linguistics34, 555596. 10.1162/coli.07-034-r2

  • 2

    BalogluS.McClearyK. W. (1999). A model of destination image formation. Ann. Tour. Res.26, 868897. 10.1016/S0160-7383(99)00030-4

  • 3

    BeelJ.GippB.LangerS.BreitingerC. (2016). Paper recommender systems: a literature survey. Int. J. Digital Libr.17, 305338. 10.1007/s00799-015-0156-0

  • 4

    BeerliA.MartínJ. D. (2004). Factors influencing destination image. Ann. Tour. Res.31, 657681. 10.1016/j.annals.2004.01.010

  • 5

    BiJ.-W.LiuY.FanZ.-P.CambriaE. (2020). Modelling customer satisfaction from online reviews using ensemble neural network and effect-based Kano model. Int. J. Prod. Res.58, 70697094. 10.1080/00207543.2019.1706726

  • 6

    ButlerR. (1980). The concept of a tourist area cycle of evolution: implications for management of resources. Can. Geogr./Le. Géographe Canadien24, 512. 10.1111/j.1541-0064.1980.tb00970.x

  • 7

    CarbonellJ.GoldsteinJ. (1998). “The use of MMR, diversity-based reranking for reordering documents and producing summaries,” in Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR) (New York, NY: ACM), 335336. 10.1145/290941.291025

  • 8

    ChenJ. L.BalážV.LiG.WilliamsA. M. (2026). Tourist decision-making and types of crises: risk attitudes, knowledge, and destination preference persistence. J. Hosp. and Tour. Res.50, 271287. 10.1177/10963480241310819

  • 9

    ChoE.MyersS. A.LeskovecJ. (2011). “Friendship and mobility: user movement in location-based social networks,” in Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (San Diego, CA: ACM), 10821090. 10.1145/2020408.2020579

  • 10

    CromptonJ. L. (1979). An assessment of the image of Mexico as a vacation destination and the influence of geographical location upon that image. J. Travel Res.17, 1823. 10.1177/004728757901700404

  • 11

    CuiL.ZhaoW.HassanM. U.YaqoobI. (2020). “Shape matching based on the enhancement of riemannian structure information,” in Proceedings of the 2020 4th International Conference on Vision, Image and Signal Processing, 15.

  • 12

    Dean-HallA.ClarkeC. L. A.KampsJ.ThomasP.SimoneN.VoorheesE. M. (2016). “Overview of the TREC 2016 contextual suggestion track,” in TREC 2016 Proceedings (Gaithersburg, Maryland: NIST). Available online at: https://trec.nist.gov/pubs/trec25/papers/Overview-CS.pdf (Accessed February 12, 2026).

  • 13

    HanH.ZhangM.HouM.ZhangF.WangZ.ChenE.et al (2020). “Stgcn: a spatial-temporal aware graph learning method for poi recommendation,” in 2020 IEEE International Conference on Data Mining (ICDM) (IEEE), 10521057.

  • 14

    HassanM. U.ZhaoX.SarwarR.AljohaniN. R.HameedI. A. (2024). Sodret: instance retrieval using salient object detection for self-service shopping. Mach. Learn. Appl.15, 100523. 10.1016/j.mlwa.2023.100523

  • 15

    HeR.McAuleyJ. (2016). VBPR: visual bayesian personalized ranking from implicit feedback. Proc. AAAI Conf. Artif. Intell.30, 144150. 10.1609/aaai.v30i1.9973

  • 16

    HeX.DengK.WangX.LiY.ZhangY.WangM. (2020). “LightGCN: simplifying and powering graph convolution network for recommendation,” in Proceedings of the ACM International Conference on Research and Development in Information Retrieval (SIGIR) (Virtual Event, China: ACM), 639648. 10.1145/3397271.3401063

  • 17

    HeP.GaoJ.ChenW. (2023). “DeBERTaV3: improving DeBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing,” in Proceedings of the International Conference on Learning Representations (ICLR).

  • 18

    HussainS. A.HassanM. U.NasarW.GhorashiS.JamjoomM. M.Abdel-AtyA.-H.et al (2023). Efficient trajectory clustering with road network constraints based on spatiotemporal buffering. ISPRS Int. J. Geo-Information12, 117. 10.3390/ijgi12030117

  • 19

    KangW.-C.McAuleyJ. (2018). “Self-attentive sequential recommendation,” in 2018 IEEE international conference on data mining (ICDM) (IEEE), 197206.

  • 20

    LandisJ. R.KochG. G. (1977). The measurement of observer agreement for categorical data. Biometrics33, 159174. 10.2307/2529310

  • 21

    LiJ.LiD.SavareseS.HoiS. (2023). “BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models,” in Proceedings of the International Conference on Machine Learning (ICML) (PMLR), 1973019742.

  • 22

    LianD.WuY.GeY.XieX.ChenE. (2020). “Geography-aware sequential location recommendation,” in Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (ACM), 20092019. 10.1145/3394486.3403252

  • 23

    LimK. H.ChanJ.LeckieC.KarunasekeraS. (2018). Personalized trip recommendation for tourists based on user interests, points of interest visit durations and visit recency. Knowl. Inf. Syst.54, 375406. 10.1007/s10115-017-1056-y

  • 24

    LiuX.LiuY.AbererK.MiaoC. (2013). “Personalized point-of-interest recommendation by mining users’ preference transition,” in Proceedings of the ACM International Conference on Information and Knowledge Management (CIKM) (ACM), 733742. 10.1145/2505515.2505639

  • 25

    Marine-RoigE.ClaveS. A. (2015). Tourism analytics with massive user-generated content: a case study of Barcelona. J. Destination Mark. Manag.4, 162172. 10.1016/j.jdmm.2015.06.004

  • 26

    MoranP. A. P. (1950). Notes on continuous stochastic phenomena. Biometrika37, 1723. 10.2307/2332142

  • 27

    OquabM.DarcetT.MoutakanniT.VoH.SzafraniecM.KhalidovV.et al (2024). DINOv2: learning robust visual features without supervision. Trans. on Mach. Learn. Res. 10.48550/arXiv.2304.07193

  • 28

    Overture Maps Foundation (2024). Overture maps places dataset. Available online at: https://overturemaps.org/download/.Release2024-04-16.0 (Accessed February 20, 2026).

  • 29

    QuerciaD.SchifanellaR.AielloL. M. (2014). The Shortest Path to Happiness: Recommending Beautiful, Quiet, and Happy Routes in the City, 116125. 10.1145/2631775.2631799

  • 30

    RadfordA.KimJ. W.HallacyC.RameshA.GohG.AgarwalS.et al (2021). “Learning transferable visual models from natural language supervision,” in Proceedings of the International Conference on Machine Learning (ICML) (PMLR), 87488763.

  • 31

    RendleS.FreudenthalerC.GantnerZ.Schmidt-ThiemeL. (2009). “BPR: bayesian personalized ranking from implicit feedback,” in Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI) (AUAI), 452461.

  • 32

    RobinsonJ.ChuangC.-Y.SraS.JegelkaS. (2021). “Contrastive learning with hard negative samples,” in Proceedings of the International Conference on Learning Representations (ICLR).

  • 33

    StepchenkovaS.ZhanF. (2013). Visual destination images of Peru: comparative content analysis of DMO and user-generated photography. Tour. Manag.36, 590601. 10.1016/j.tourman.2012.08.006

  • 34

    SunF.LiuJ.WuJ.PeiC.LinX.OuW.et al (2019). BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer, 14411450. 10.1145/3357384.3357895

  • 35

    ThomeeB.ShammaD. A.FriedlandG.ElizaldeB.NiK.PolandD.et al (2016). YFCC100M: the new data in multimedia research. Commun. ACM59, 6473. 10.1145/2812802

  • 36

    van den OordA.LiY.VinyalsO. (2018). Representation learning with contrastive predictive coding. Adv. Neural Inf. Process. Syst. (NeurIPS)31, 113.

  • 37

    VrandečićD.KrötzschM. (2014). Wikidata: a free collaborative knowledgebase. Commun. ACM57, 7885. 10.1145/2629489

  • 38

    WangX.HeX.CaoY.LiuM.ChuaT.-S. (2019). “Kgat: knowledge graph attention network for recommendation,” in Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery and data mining, 950958.

  • 39

    WeyandT.AraujoA.CaoB.SimJ. (2020). “Google landmarks dataset v2 – a large-scale benchmark for instance-level recognition and retrieval,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE), 25752584.

  • 40

    Wikivoyage contributors (2023). Wikivoyage Database Dumps. Available online at: https://dumps.wikimedia.org/enwikivoyage/ (Accessed March 18, 2026).

  • 41

    YangY.PanB.SongH. (2014). Predicting hotel demand using destination marketing organization’s web traffic data. J. Travel Res.53, 433447. 10.1177/0047287513500391

  • 42

    YaqoobI.HassanM. U.NiuD.ZhaoX.HameedI. A.HassanS.-U. (2023). A novel person re-identification network to address low-resolution problem in smart city context. ICT Express9, 809814. 10.1016/j.icte.2022.07.010

  • 43

    Yelp Inc (2023). Yelp open dataset. Available online at: https://www.yelp.com/dataset (Accessed April 15, 2026).

  • 44

    ZangerleE.BauerC. (2022). Evaluating recommender systems: survey and framework. ACM Comput. Surv.55, 138. 10.1145/3556536

  • 45

    ZhaiX.MustafaB.KolesnikovA.BeyerL. (2023). “Sigmoid loss for language image pre-training,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (IEEE), 1197511986.

  • 46

    ZhangY.ChenX. (2020). Explainable recommendation: a survey and new perspectives. Found. Trends Inf. Retr.14, 1101. 10.1561/1500000066

  • 47

    ZhangK.QiuH.WangJ.LiC.ZhangJ.ChenD. D. (2022). Tourist gaze through computer vision: where, what, how and why?Tour. Rev.77, 11861202. 10.1108/tr-04-2020-0185

  • 48

    ZhangM.HassanM. U.NiuD.ZhaoX.NawazR.HameedI. A.et al (2023). A methodology for shape matching of non-rigid structures based on integrated graphical information. Displays76, 102336. 10.1016/j.displa.2022.102336

  • 49

    ZhengY.MobasherB.BurkeR. (2015). Similarity-Based context-aware Recommendation, 431447. 10.1007/978-3-319-26190-4_29

  • 50

    ZhouB.LapedrizaA.KhoslaA.OlivaA.TorralbaA. (2018). Places: a 10 million image database for scene recognition. IEEE Trans. Pattern Anal. Mach. Intell.40, 14521464. 10.1109/TPAMI.2017.2723009

Appendix A

Repeated-run stability: mean standard deviation

Table A1 reports mean Recall@10 and NDCG@10 across five independent runs (seeds 0–4) for all models evaluated on the Yelp Open Dataset benchmark. Standard deviations confirm that MM-DIPR’s improvements are stable and not artefacts of a lucky random initialization.

Hyperparameter search spaces

Table A2 documents the search space used for all models. The best configuration for each model was selected by validation NDCG@10; the test set was never accessed during hyperparameter selection.

Loss function definitions

Table A3 summarizes all five components of the joint training objective (Equation 1) with their functional forms, supervision signals, and gradient pathways.

TABLE A3

LossFunctional formSupervision signalBackprop target
Binary cross-entropy over binarized (above/below median) attribute labels: Places365 scene logits; GLDv2 landmark scores; Wikivoyage tags; Yelp sentimentPerception bottleneck (M5); all encoders
InfoNCE contrastive loss (Equation 2) with learned temperature ; hard negatives from geographically proximate POIsImage–text co-occurrence from public sourcesVisual (M2) and text (M3) encoders; projection heads
BPR pairwise loss: Observed check-ins as positives; unobserved as negatives (sampled)Graph ranker (M6); user and item embeddings
Expected calibration error (ECE): mean absolute difference between predicted confidence and empirical accuracy in equal-width binsHeld-out annotation labels from the 2,000-image subsetPerception bottleneck (M5) output sigmoid
Negative mean pairwise cosine distance in perception embedding space for the top-50 re-ranking candidates: ; applied only at inference/re-ranking stageSelf-supervised (no external labels)Not backpropagated during training; used only in MMR re-ranking (M7)

Summary of the five loss components in the MM-DIPR joint training objective. “Backprop target” indicates which model parameters receive gradients from each loss.

Summary

Keywords

earth sciences, explainable AI, multimodal recommendation, point-of-interest recommendation, sustainable tourism analytics, tourism destination image, vision–language learning

Citation

Deng L and Liu Q (2026) MM-DIPR: perception-aware multimodal learning for tourism destination image understanding and recommendation. Front. Earth Sci. 14:1884556. doi: 10.3389/feart.2026.1884556

Received

18 May 2026

Revised

15 July 2026

Accepted

20 July 2026

Published

03 September 2026

Volume

14 - 2026

Edited by

Alejandro Bellogin, Autonomous University of Madrid, Spain

Reviewed by

Zijun Long, Hunan University, China

Jingsen Lian, Delft University of Technology (TU Delft), Netherlands

Updates

Copyright

*Correspondence: Liyun Deng,

† These authors have contributed equally to this work

Disclaimer

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

Outline

Figures

Cite article

Copy to clipboard


Export citation file


Share article

Article metrics