Vision–language guided semantic-geometric transformer for memory-efficient 3D scene understanding

Abstract

Recent 3D Transformers have become a dominant framework for point-cloud segmentation by modeling spatial context in sparse 3D scenes. However, geometry and color alone provide limited high-level semantic cues, especially for cluttered boundary regions, visually similar objects, and long-tail categories. To address this issue, we propose a segmentation framework guided by Contrastive Language–Image Pre-training (CLIP) that enriches sparse 3D tokens with vision–language semantic priors. Specifically, dense CLIP features extracted from multi-view RGB images are projected onto 3D points through visibility-aware alignment and view pooling, and are fused with relative geometric offsets and color cues to form semantically aware sparse voxel tokens. To better exploit the aligned CLIP semantics during local token interactions, we build on contextual relative signal encoding (cRSE) and introduce a decoupled CLIP-induced semantic residual that forms semantic-geometric attention biases for local window attention. We further adapt block-wise online softmax computation to generate and consume these biases on the fly. Experiments on ScanNet, ScanNet200, and S3DIS demonstrate competitive segmentation performance, improved instance-level discrimination, and a 25.7% reduction in peak online 3D-stage training memory compared with the materialized attention implementation when cached CLIP features are used.

1 Introduction

3D scene understanding is fundamental to embodied intelligence, and point-cloud segmentation provides essential spatial-semantic representations for tasks such as scene mapping, robot navigation, and object interaction. To model complex spatial context in sparse 3D data, recent research has moved from point-based local aggregation methods (Zhao et al., 2021; Zhang et al., 2022) toward Transformer-based architectures (Lu et al., 2025; Wu et al., 2024; Lai et al., 2022; He et al., 2024; Han et al., 2024). In particular, window-based 3D Transformers (Yang et al., 2025a,b; Liu et al., 2021) have emerged as a practical design choice because localized attention achieves a favorable balance between efficiency and receptive field while preserving geometric inductive bias (Liu et al., 2023; Guo et al., 2022). However, their geometry-centric design still leaves high-level semantic understanding insufficient, especially in cluttered scenes (Akwensi et al., 2024; Kim et al., 2026; Li et al., 2024; Khan et al., 2024).

This semantic limitation arises from two factors. First, semantic cues in point clouds are sparse and ambiguous: geometry and optional color alone provide limited high-level semantic cues, making it difficult to maintain consistent predictions for visually similar structures, cluttered boundary regions, and long-tail categories across distant regions (Peng et al., 2023; Takmaz et al., 2023; Jatavallabhula et al., 2023; Tai et al., 2026). Second, directly incorporating external semantic priors into Transformer-based 3D architectures can be memory-intensive. Existing attempts to enhance 3D representations with multi-view image encoders or vision–language models (VLMs), including Contrastive Language–Image Pre-training (CLIP), typically rely on high-dimensional tokens and dense pairwise interactions, which rapidly exhaust GPU memory and restrict training to smaller windows or batch sizes (Dao et al., 2022; Chen et al., 2025; Yamazaki et al., 2024). These limitations call for a practical solution that incorporates VLM-derived semantic priors into sparse 3D representations while avoiding excessive memory overhead in semantic-geometric attention.

To address these challenges, we propose a CLIP-guided 3D segmentation framework that aligns vision–language semantic features with 3D point clouds and integrates them into sparse voxel tokens for memory-efficient window-based Transformer segmentation. Specifically, dense CLIP features extracted from multi-view RGB images are projected onto 3D points through visibility-aware alignment and view pooling, and are fused with relative geometric offsets and color cues to construct semantically aware sparse voxel tokens. Building on contextual relative signal encoding (cRSE) (Wu et al., 2021), we introduce a semantic-geometric extension that combines relative geometry, appearance variation, and a decoupled CLIP-induced discrepancy residual as a dynamic pairwise attention bias. To make this extension practical, block-wise online softmax computation is adapted to generate and consume bias tiles without materializing the full pairwise tensor.

Our contributions are threefold:

• We propose a CLIP-guided feature alignment framework that projects dense CLIP features from multi-view RGB images onto 3D points through visibility-aware alignment and view pooling. The aligned semantic features are fused with relative geometric offsets and color cues to construct semantically aware sparse voxel tokens for point-cloud segmentation.

• We extend contextual relative encoding with a decoupled CLIP-induced semantic residual and quantify its incremental contribution to the structural semantic–geometric attention bias within local windows.

• We adapt block-wise online softmax computation to generate and fuse cRSE bias tiles on the fly. This implementation avoids full pairwise bias materialization and reduces peak online training memory for the 3D stage by 25.7% at a window capacity of N_w = 32 on a single 24GB GPU.

2 Related work

2.1 3D scene segmentation

Early 3D scene segmentation methods primarily relied on point-based local aggregation [e.g., PointNet++ (Qi et al., 2017)], sparse convolutional operators, such as KPConv (Thomas et al., 2019) and SparseUNet (Choy et al., 2019; Gezawa et al., 2023), or efficient graph-convolutional architectures, such as MLGCN (Khodadad et al., 2024), to extract multi-scale geometric features from irregular 3D data. More recently, 3D Transformers have emerged as a powerful alternative for scalable context modeling (Wu et al., 2024; He et al., 2024; Lu and Zhang, 2026). To mitigate the quadratic complexity of global self-attention, partition-based or window-based Transformers (Lai et al., 2022; Wu et al., 2024; Yang et al., 2025b; Li et al., 2025) restrict attention to local or shifted spatial windows, offering a favorable trade-off between receptive field and computational efficiency. Despite their strong scalability, these architectures still rely predominantly on geometry-centric inductive biases (Zhao et al., 2021; Wu et al., 2022; Park et al., 2022), such as relative position encodings derived from spatial offsets. Although such inductive biases are effective for local geometry modeling, they provide limited guidance when spatially adjacent tokens correspond to different semantic categories, as frequently observed in cluttered scenes and long-tail settings. Existing window-based 3D Transformers mainly rely on geometric relative encoding to modulate local attention, while high-level semantic discrepancy is rarely converted explicitly into pairwise attention biases. To address this limitation, our semantic-geometric cRSE extension introduces aligned CLIP-induced discrepancy into local attention. It measures the discrepancy between neighboring voxel tokens in CLIP space and adds a decoupled residual to the structural attention bias.

2.2 3D vision–language learning

The strong open-vocabulary priors of vision–language models (VLMs) such as CLIP (Radford et al., 2021) have inspired a growing body of research on grounding 2D semantics in 3D scenes (Peng et al., 2023; Takmaz et al., 2023; Jatavallabhula et al., 2023; Chen et al., 2023; Yamazaki et al., 2024; Tai et al., 2026). Representative methods commonly transfer image-level, region-level, or dense 2D semantic features into 3D representations through multi-view correspondences. For example, OpenScene (Peng et al., 2023) projects 2D open-vocabulary features onto 3D points to enable open-vocabulary 3D scene understanding, while ConceptFusion (Jatavallabhula et al., 2023) fuses multi-view visual concepts into 3D maps for open-set scene representation. OpenMask3D (Takmaz et al., 2023) further exploits 2D foundation-model features for open-vocabulary 3D instance segmentation, and CLIP2Scene (Chen et al., 2023) investigates CLIP-guided 3D scene understanding by transferring 2D vision–language knowledge to point cloud representations. These methods demonstrate that projecting or aggregating 2D VLM features can effectively enrich point clouds with high-level semantic cues. Recent post-2024 studies have further explored vision–language priors for 3D scene understanding. MPEC (Wang Y. et al., 2025) introduces masked point–entity contrastive learning with 3D entity–language alignment and uses a frozen CLIP text encoder for 3D representation learning. PGOV3D (Zhang et al., 2025) develops a partial-to-global curriculum based on multi-view RGB-D observations, CLIP-related language supervision, a multimodal large language model, and a 2D segmentation foundation model. Open-Vocabulary Octree-Graph (Wang Z. et al., 2025) uses CLIP and OVSeg features to construct zero-shot open-vocabulary semantic and instance-level 3D scene representations. These recent studies confirm the continuing importance of vision–language features in 3D scene understanding, while covering different supervision and evaluation settings. While effective, most existing methods mainly use the projected CLIP features as additional point-level, voxel-level, or region-level descriptors, for example through feature concatenation, aggregation, or auxiliary semantic guidance (Wang Z. et al., 2025; Tai et al., 2026). MPEC further studies entity–language representation learning, whereas PGOV3D and Open-Vocabulary Octree-Graph primarily target open-vocabulary or zero-shot scene understanding. These settings differ from the closed-set supervised segmentation protocol considered in this work. Such designs enhance the semantic representation of 3D tokens, but they rarely use the semantic discrepancy between local tokens in the CLIP feature space to directly modulate attention computation. In contrast, our framework not only aligns multi-view CLIP features with 3D points through visibility-aware projection and view pooling, but also converts the aligned CLIP semantics into a dynamic pairwise bias for local attention. Specifically, we measure the semantic discrepancy between neighboring voxel tokens in the CLIP feature space and incorporate this discrepancy into cRSE as an additive bias to the attention logits. This design allows CLIP semantics to guide token-to-token attention weights within local windows, rather than serving only as static input features.

2.3 Memory-efficient attention

The memory bottleneck of self-attention has motivated extensive research on efficient attention mechanisms. In natural language processing (NLP) and 2D vision, hardware-aware methods such as FlashAttention (Dao et al., 2022; Dao, 2024) use I/O-aware block-wise tiling and streaming softmax computation to avoid materializing large intermediate attention matrices. Recent sparse or structured attention methods further improve scalability by restricting attention to selected tokens or local regions (Yuan et al., 2025). In 3D point cloud understanding, efficient Transformer designs commonly reduce attention cost through local windows, partition-based attention, voxelization, token reduction, or hardware-aware sparse computation (Wu et al., 2024; Chen et al., 2025; He et al., 2024). However, applying fused attention principles to sparse 3D architectures remains challenging. Point cloud windows usually contain irregular numbers of tokens, and sparse indexing introduces non-negligible gather/scatter overhead in window partitioning and token aggregation (Wu et al., 2024; Chen et al., 2025; He et al., 2024). Moreover, additional pairwise bias terms, such as relative positional encoding or semantic-geometric attention bias, are not directly supported by standard fused attention implementations (Dao et al., 2022; Dao, 2024; Yuan et al., 2025). In our cRSE attention, the bias for each token pair depends on relative geometry, appearance variation, and CLIP-induced semantic discrepancy. A naive implementation would pre-compute and store the full bias tensor of size H×N_w×N_w, resulting in auxiliary memory. To reduce this overhead, we use a block-wise streaming integration strategy. Instead of materializing the full cRSE bias tensor, we compute only the bias tile required by the current key/value block, add it directly to the attention logits, and discard it after updating the running softmax accumulation. This reduces cRSE bias storage from full pairwise tensor materialization to block-level intermediate buffers, enabling memory-efficient semantic-geometric attention for sparse 3D Transformers.

Table 1 summarizes the mechanism-level differences between our method and closely related approaches. Unlike methods that use CLIP features only as input-level descriptors or rely solely on structural relative encoding, our method introduces a decoupled pairwise semantic residual and integrates the resulting cRSE bias without materializing the full bias tensor.

3 Methodology

Figure 1 summarizes the pipeline: frozen multi-view CLIP features are aligned with 3D points, embedded into sparse voxel tokens, and processed by a Swin3D-style encoder equipped with semantic-geometric cRSE. A block-wise implementation generates cRSE bias tiles only when they are consumed by attention.

Figure 1

3.1 Input representation and multimodal alignment

We align VLM-derived priors from multi-view images with 3D points before sparse voxel processing (Peng et al., 2023; Chen et al., 2023; Jatavallabhula et al., 2023; Wang Z. et al., 2025). Unlike point-level feature transfer alone, the aligned features are later reused to construct pairwise semantic attention biases.

3.1.1 Voxelization and 2D semantic extraction

Given an input 3D point cloud with coordinates and colors, we denote the i-th point as (x_i, c_i), where and . The point cloud is first discretized into a sparse voxel grid for subsequent sparse 3D processing (Choy et al., 2019; Graham et al., 2018). Meanwhile, the associated multi-view RGB images are fed into a frozen pre-trained CLIP image encoder Φ_CLIP to extract dense 2D semantic feature maps:

Here, V is the number of selected views, v is the camera-view index, H′ and W′ denote the height and width of the CLIP patch-feature map, respectively, namely the numbers of patch positions along its two spatial axes, and C_clip is the feature-channel dimension. In our implementation, H′ = W′ = 24 and C_clip = 1024. In practice, F^(v) is formed from the last-layer CLIP patch-token features excluding the global classification ([CLS]) token, reshaped into a 2D patch grid and normalized before feature sampling. These feature maps provide the 2D semantic cues that are subsequently projected and aligned with 3D points.

3.1.2 Visibility-aware projection

To align 2D CLIP semantics with 3D points (Hu et al., 2021; Dai and Nießner, 2018), each 3D point is projected onto the image plane of a camera view v. Given the intrinsic matrix K^(v) and extrinsic matrix [R^(v)|t^(v)], the projected pixel coordinate of point x_i is computed as

In Equation 1, K^(v)∈ℝ^{3 × 3} is the camera intrinsic matrix, while R^(v)∈ℝ^{3 × 3} and t^(v)∈ℝ³ define the world-to-camera transformation. The vector is the homogeneous 3D coordinate, and is the homogeneous image coordinate. The symbol ~ denotes equality up to a nonzero scale, and π(·) performs perspective division. To suppress erroneous semantic alignment caused by occlusion or geometric inconsistency (Peng et al., 2023), we further apply a visibility-aware depth test. Let denote the projected depth of point x_i in view v, and let be the corresponding rendered or sensor depth. The visibility indicator is defined as

Here, 1(·) is the indicator function. The threshold δ = 0.05m is fixed for visibility filtering in all experiments and is not learned during training. Only visible projections with are used for CLIP feature sampling and subsequent view pooling.

3.1.3 View pooling and aligned point representation

Since a 3D point may be visible in multiple views, we aggregate the corresponding CLIP features (Chen et al., 2023) through view pooling. Let Ω_img denote the image domain and define the valid-view set of point x_i as

Let be the CLIP feature sampled at for . We compute the aligned CLIP semantic feature as

where is the view-specific pooling weight. We define the point-level CLIP-validity flag as

In Equation 4, indicates that point i receives at least one valid CLIP observation from the selected views. If , we set s_i = 0 and record a separate invalidity flag, thereby avoiding an undefined zero denominator. In all experiments, we use uniform weights, namely for every valid projection, so that Equation 3 reduces to average pooling over the valid views of each point. The aligned point representation is subsequently combined with the intra-voxel coordinate offset, RGB color, and aligned CLIP feature CLIP_{f, i}≡s_i.

3.2 Semantically aware initial embedding

The aligned point features are converted into semantically aware sparse voxel tokens for the Swin3D-style backbone (Yang et al., 2025b).

3.2.1 Fine-scale sparse voxelization and relative embedding

Let v_s be the voxel size, ν_i = ⌊x_i/v_s⌋ be the integer voxel coordinate obtained by element-wise flooring, ν(i) be the voxel index assigned to point i, and c_ν(i) be its center. The brackets in Equation 5 denote channel-wise concatenation. The input embedding retains intra-voxel geometry and concatenates it with color and CLIP semantics (Lai et al., 2022; Choy et al., 2019; Peng et al., 2023):

Here c_i is the RGB vector. This embedding exposes geometry, appearance, and semantics from the first layer.

3.2.2 Sparse convolutional encoding

Point embeddings in each non-empty voxel are average-pooled and passed through a 3 × 3 × 3 sparse convolution, batch normalization (BN), and a rectified linear unit (ReLU) (Graham et al., 2018; Choy et al., 2019). The resulting tensor provides the initial sparse tokens. Here, N₁ is the number of non-empty voxels, or equivalently the number of active sparse tokens, at the initial stage, and C₁ is the corresponding feature-channel dimension. In our implementation, C₁ = 48.

3.3 Semantic-geometric window attention via cRSE

Here, cRSE refers to contextual relative signal encoding, following the contextual relative-position formulation in (Wu et al. 2021). In each window, p_i is the center of voxel token i and c_i is its mean RGB color. The structural branch therefore uses token-level differences in geometry and appearance rather than raw point coordinates. Given the semantically aware initial sparse tensor from the semantically aware initial embedding stage, we feed it into a hierarchical Swin3D-style backbone (Yang et al., 2025b) to learn multi-scale semantic-geometric representations. The backbone comprises four stages, where successive blocks alternate between window-based self-attention (W-MSA3D) and shifted-window self-attention (SW-MSA3D) (Liu et al., 2021) to enable efficient local context modeling and cross-window information exchange. However, standard 3D window attention mainly relies on geometric relative position encodings when modeling pairwise token relations, making it less effective in distinguishing physically adjacent regions with different semantics. We therefore extend this structural encoding (Wu et al., 2021) with a CLIP-induced semantic residual while retaining geometry and appearance cues.

3.3.1 CLIP-induced semantic discrepancy

For voxel index ν, let

denote the set of points assigned to voxel ν. We first count the points in this voxel that receive at least one valid CLIP observation:

The voxel-level CLIP feature is then obtained by averaging the valid point-level features:

Equations 6, 7 define the voxel-level CLIP-validity statistics and the pooled voxel-level CLIP feature, respectively. For a voxel token i, let ν_i denote the voxel represented by that token. The voxel-level validity flag is distinct from the point–view visibility flag and the point-level aggregated flag . Specifically, if at least one point assigned to voxel ν_i receives a valid CLIP observation from any selected view.

For tokens i and j in the same local window, the semantic discrepancy is defined as

In Equation 8, ϵ = 10⁻⁶ is the numerical-stability constant used in all experiments. A larger value of indicates a stronger CLIP-space mismatch. The semantic discrepancy is set to zero whenever either represented voxel lacks a valid CLIP observation.

3.3.2 Joint context and cRSE bias generation

To keep high-level semantics separate from low-level transformations, we form Δz_ij = [p_i−p_j||c_i−c_j] and inject semantic discrepancy through a residual branch. The head-specific bias is

In Equation 9, || denotes channel-wise concatenation, h∈{1, …, H} indexes one of the H heads in the current attention block. The transformations t_Q and t_K map Δz_ij to a shared latent dimension, w_h projects this representation to a scalar bias for head h, and λ_{s, h} is a non-negative head-specific semantic coefficient implemented with a softplus parameterization. The negative residual suppresses attention between semantically inconsistent neighbors without entangling CLIP features with structural transformations.

3.3.3 Integration into window attention

The bias is added directly to window attention:

In Equation 10, q_{i, h}, k_{j, h}, and v_{j, h} are the query, key, and value vectors for tokens i and j in the current local window. They are obtained as , , and , where , , and are learned head-specific projection matrices. The dimension d is the per-head feature dimension, the index j ranges over valid key/value tokens in the same window, and softmax_j normalizes the logits over those tokens.

Figure 2 illustrates the complete cRSE computation and streaming-attention process. The original voxel tokens generate the query, key, and value features, while relative geometry, color difference, and CLIP-space are the structural and semantic branches that process relative and semantic discrepancy to construct the attention bias. This bias is generated and consumed tile by tile during block-wise attention.

Figure 2

3.4 Memory-efficient streaming integration for cRSE

Naively storing incurs auxiliary memory. We instead partition keys and values into tiles. Let N_w be the window capacity and let the keys and values be partitioned into T tiles. For a fixed query, tile t produces , where K_t, V_t, and are the key, value, and cRSE-bias tiles. The index j runs over entries in tile t. The variables m_t, ℓ_t, and a_t denote the running maximum, softmax normalization factor, and value accumulator after tile t. We initialize m₀ = −∞, ℓ₀ = 0, and a₀ = 0; padding and shifted-window masks are applied to α_t before the update. Equation 11 updates the running maximum, normalization factor, and value accumulator tile by tile.

Here, α_{t, j} and v_{t, j} are the j-th logit and value vector in tile t, while o is the final attention output for the query. The required cRSE tile is generated together with each dot-product tile, consumed by this recurrence, and discarded. Thus, the forward computation retains block-level bias buffers instead of the full pairwise bias tensor. The reported memory measurements cover the online 3D segmentation stage with fixed, pre-computed CLIP features; they do not include offline CLIP extraction, virtual-view rendering, alignment, or cache storage.

3.5 Multi-scale decoding and segmentation heads

U-Net-style sparse decoder mirrors the four encoder stages, using transposed sparse convolutions and skip connections to restore voxel resolution (Ronneberger et al., 2015; Choy et al., 2019). The semantic head predicts per-voxel class logits, which are mapped to raw points by the deterministic point-to-voxel correspondence. A lightweight instance branch predicts center offsets, embeddings, and objectness; at inference, objectness-filtered points are clustered jointly by shifted coordinates and embeddings, followed by semantic voting and mask-level non-maximum suppression (NMS). These standard heads are retained only to evaluate whether the proposed representation transfers to both semantic and instance segmentation.

The semantic loss combines cross-entropy and Lovász–Softmax (Berman et al., 2018). The instance loss combines offset regression, objectness binary cross-entropy (BCE), and discriminative embedding losses. The complete objective is

In Equation 12, and are the semantic cross-entropy and Lovász–Softmax losses. The terms , , and denote the offset-regression, objectness, and discriminative-embedding losses, respectively. The coefficients are fixed to λ_ins = 1.0, λ_obj = 1.0, and λ_emb = 1.0 in all experiments. Definitions of the standard instance losses, point grouping procedure, and main inference thresholds are provided in the Supplementary material.

4 Experiments

4.1 Experimental setup

4.1.1 Datasets and protocols

We evaluate on ScanNet v2 (Dai et al., 2017), ScanNet200 (Rozenberszki et al., 2022), and the Stanford Large-Scale 3D Indoor Spaces (S3DIS) dataset (Armeni et al., 2016). ScanNet v2 contains 1,201 training scans and 312 validation scans, together with 100 held-out test scans used by the official benchmark. We train on the official training split and report results on the 312-scan validation split; the hidden test set is not used. For ScanNet, semantic and instance segmentation are evaluated over 20 and 18 categories, respectively. ScanNet200 uses the same 1,201/312/100 training–validation–test partition and extends the label space to 200 semantic and 200 instance categories, which are grouped into head, common, and tail subsets according to occurrence frequency. The 100-scan hidden test split is not used, and all ScanNet200 results are reported on the validation split. S3DIS contains six areas, 271 rooms, and 13 annotated categories. Under the Area-5 protocol, the 203 rooms from Areas 1, 2, 3, 4, and 6 are used for training, whereas the 68 rooms in Area 5 are used exclusively for evaluation; no separate S3DIS validation split is used.

4.1.2 Evaluation metrics

For semantic segmentation, the intersection-over-union for class c and its class average are

In Equation 13, TP_c, FP_c, and FN_c denote the numbers of true-positive, false-positive, and false-negative points for class c, respectively, and C is the number of evaluated semantic classes. For instance segmentation, mask IoU is the intersection-over-union between a predicted and a ground-truth point mask. A prediction is a true positive when it has the correct category, reaches mask IoU ≥0.5 with an unmatched ground-truth instance, and is selected by one-to-one score-ordered matching. Let r∈[0, 1] denote recall and let p_c(r) denote the interpolated precision of class c at recall r. We report

In Equation 14, C_ins is the number of valid instance categories. Head/common/tail AP₅₀ values are the same class-wise average restricted to each ScanNet200 frequency group.

4.1.3 Multimodal input protocol

ScanNet provides approximately 2.5 million posed RGB-D frames with calibrated camera intrinsics, camera poses, and depth maps, and ScanNet200 reuses the same RGB-D sequences. For each ScanNet or ScanNet200 scene, frames with invalid poses or missing depth maps are removed. The remaining frames are ordered by temporal index, and 32 approximately uniformly spaced frames are selected using i_k = round(k(M−1)/31), where k = 0, …, 31 and M is the number of valid frames. The resulting view list is fixed across all experiments. S3DIS provides colored point clouds but does not provide posed RGB-D video sequences in the same form as ScanNet. We therefore deterministically generate virtual RGB-D views from the normalized room bounding box using eight azimuth directions, four normalized camera heights, fixed camera intrinsics, and z-buffer rendering. Each camera is oriented toward the room center. Exactly 32 deterministic candidate views are generated for each S3DIS room using the fixed camera layout described in the Supplementary material. Candidate views with projected-point coverage below τ_cov = 0.05 are excluded from feature alignment. Consequently, each room uses a deterministic valid subset of up to 32 views; the exact number depends on the room-specific coverage test, and rejected views are not replaced. The remaining valid views are used for CLIP feature extraction and 2D–3D alignment. We freeze the CLIP ViT-L/14 encoder and resize each RGB image to 336 × 336. The normalized last-layer patch tokens form a 24 × 24 × 1, 024 feature map. Features are bilinearly sampled, retained when the depth discrepancy is below 0.05 m, and uniformly averaged across valid views. Points without valid observations receive a zero feature and a separate validity mask. Point-aligned CLIP features are cached in half precision and used during both 3D training and inference. Detailed camera parameters and the complete pipeline are provided in the Supplementary material.

4.1.4 Implementation details

The 3D network is implemented in PyTorch/OpenMMLab and trained on one 24GB RTX 4090. We use AdamW with learning rate 10⁻³, weight decay 5 × 10⁻², cosine decay, 1500 warmup iterations, batch size 4, voxel size v_s = 0.02 m, and base window size N_w = 32. We apply standard geometric and color augmentations. All controlled variants share the same training schedule, instance head, and post-processing settings. Unless otherwise stated, the main experiments use seed 42. The sequential component comparison in Table 2 is reported from a separately executed single-run diagnostic, whereas Table 3 reports an independent repeated-seed evaluation using seeds 41, 42, and 43. Results from these two experiment groups are therefore not treated as the same training runs. The Supplementary material provides the deterministic virtual-camera configuration and the complete ordered protocol for view generation, CLIP extraction, 2D–3D alignment, feature caching, 3D training, inference, and profiling.

4.2 Comparison with state-of-the-art methods

We report both published benchmark results and controlled retrained variants. The retrained variants use the same backbone, prediction head, data split, training schedule, and post-processing, and provide the component-wise comparison of the CLIP input, geometry–color cRSE, and semantic residual.

4.2.1 Protocol-matched controlled comparison

Table 2 presents a protocol-matched cross-task component comparison. Starting from the retrained Swin3D configuration, the CLIP input, geometry–color cRSE, and semantic residual are added successively while keeping the backbone, prediction head, data split, training schedule, and post-processing fixed. The table reports both ScanNet semantic mIoU and ScanNet200 instance AP₅₀, providing a common cumulative view of the component contributions across two segmentation tasks.

On ScanNet, mIoU increases from 73.30 for the retrained Swin3D configuration to 76.12 with CLIP input, 76.48 with geometry–color cRSE, and 76.82 for the full model. Thus, the main performance increase is associated with the aligned CLIP input, while geometry–color cRSE and the semantic residual provide smaller incremental gains of 0.36 and 0.34 mIoU points, respectively. On ScanNet200, the corresponding AP₅₀ values are 28.66, 35.80, 39.94, and 40.86. The semantic residual adds 0.92 AP₅₀ points after the CLIP input and geometry–color cRSE have already been enabled. A finer branch-level analysis on ScanNet is provided separately in Table 4.

4.2.2 Published benchmark comparisons

Tables 5, 6 compare our results with selected published benchmarks under their reported evaluation settings. To broaden the comparison with recent 3D methods using CLIP or related vision–language models, we include MPEC in the benchmark tables and summarize the reported results of PGOV3D and Open-Vocabulary Octree-Graph in Table 7.

The mean and standard deviation reported for ScanNet are obtained from the repeated-seed evaluation. The single-run results in Table 2 come from the separate controlled component experiment and are used to measure the incremental contribution of each component.

For instance segmentation, the full model achieves an AP₅₀ of 76.9% on ScanNet, 40.70 ± 0.26 on ScanNet200 across three seeds, and 69.6% on S3DIS Area 5. The corresponding comparisons with published methods are reported in Table 6.

Our full model achieves an AP₅₀ of 76.9 on ScanNet, 40.70 ± 0.26 on ScanNet200 over three seeds, and 69.6 on S3DIS Area 5. On ScanNet200 and S3DIS, these results exceed the published values listed in the table, whereas Mask3D obtains the highest ScanNet AP₅₀. The incremental contributions of the CLIP input, geometry–color cRSE, and semantic residual are examined separately in Table 2.

4.2.3 Recent CLIP/VLM-based contextual comparisons

Recent CLIP/VLM-based methods published after 2024, together with our full model, are summarized in Table 7. The reported results are listed with their corresponding tasks and evaluation settings.

4.2.4 Broader literature reference

Table 8 summarizes additional representative published point-cloud segmentation results on ScanNet, ScanNet200, and S3DIS. Recent CLIP/VLM-based results are reported separately in Table 7. Semantic-segmentation results in this table are reported using mIoU, whereas instance-segmentation results are reported using AP₅₀. For methods whose official implementations or complete experimental configurations were unavailable, we report the values from the corresponding publications rather than reproducing those methods under our experimental setup.

4.3 Ablation studies

We first evaluate the proposed components on the ScanNet validation split under the same training and evaluation settings. The largest improvement is obtained by adding the aligned CLIP input. Geometry–color cRSE provides a further gain, while the semantic residual improves mIoU by an additional 0.3–0.4 points in the reported experiments. These results indicate that the semantic residual acts as a complementary refinement of the semantic–geometric attention bias. For variants marked with †, we report the mean over seeds 41, 42, and 43, rounded to one decimal; the remaining diagnostic variants are evaluated with seed 42.

The cross-task cumulative comparison is given in Table 2. A finer branch-level analysis of the ScanNet semantic task is provided in Table 4, while alternative semantic-residual formulations are examined in Table 9.

4.3.1 Statistical stability across random seeds

To assess the stability of the observed improvements, we independently retrain the key configurations with seeds 41, 42, and 43. Table 3 reports the results for each run together with the sample mean and standard deviation. On ScanNet, the mean mIoU increases from 73.40 for the 3D-only baseline to 75.97 with CLIP input, 76.33 with geometry–color cRSE, and 76.67 for the full model. The full model exceeds the geometry–color cRSE configuration by 0.3, 0.3, and 0.4 points for the three seeds, respectively. On ScanNet200, the mean instance AP₅₀ increases from 28.40 for the 3D-only configuration with our instance head to 35.60 with CLIP input and to 40.70 for the full model, showing consistent gains across random seeds.

4.3.2 Semantic-bias formulation

Table 9 compares pairwise CLIP-discrepancy formulations while keeping the CLIP input, structural cRSE, backbone, prediction head, and training settings fixed. The CLIP-input-plus-structural-cRSE configuration serves as the reference and obtains 76.45 mIoU in this formulation study. Directly coupling the cosine discrepancy with the structural branch provides only a marginal gain, whereas decoupled ℓ₂ and two-layer MLP residuals reach 76.61 and 76.75 mIoU, respectively. The proposed decoupled cosine residual reaches 76.82 mIoU, corresponding to a 0.37-point improvement over the structural reference. Compared with the two-layer MLP residual, it reduces peak memory by 0.3 GB and step time by 13 ms while improving mIoU by 0.07 points. These single-run results motivate the selected formulation, while the small differences among the strongest variants should be interpreted cautiously.

4.3.3 Hyperparameter sensitivity

The semantic residual also depends on how its coefficient λ_{s, h} is constrained and shared. Table 10a compares fixed, unconstrained, and softplus-parameterized variants while keeping the semantic-bias formulation unchanged. Fixed coefficients of 0.25 and 0.50 obtain 76.57 and 76.67 mIoU, respectively. Without a non-negativity constraint, the learned scalar becomes −0.08, reversing the intended suppression of semantically inconsistent neighbors and reducing mIoU to 76.51. A non-negative learned scalar improves the result to 76.76. The proposed per-head softplus parameterization reaches 76.82 mIoU, with learned coefficients ranging from 0.18 to 0.71 across heads. This indicates that different heads can assign different weights to semantic disagreement. The 0.06-point gain over the learned scalar is modest, while the per-head parameterization provides greater flexibility.

We further evaluate the sensitivity to the depth-consistency threshold δ in Equation 2, which determines whether a projected observation is retained during visibility-aware CLIP alignment. Table 10b reports the valid-point coverage and ScanNet validation mIoU for three threshold values, with uniform view pooling and all other experimental settings held fixed. Reducing δ from 0.05m to 0.02m decreases the percentage of points receiving at least one valid CLIP observation from 89.7% to 87.4% and reduces mIoU from 76.82 to 74.94. Increasing δ to 0.10m slightly increases the valid-point coverage to 89.9%, but reduces mIoU to 75.15. These results indicate that a stricter threshold removes useful observations, whereas a looser threshold can retain less reliable 2D–3D correspondences. We therefore use δ = 0.05m in all reported experiments as a balance between semantic-feature coverage and alignment reliability.

4.3.4 Instance-level and long-tail analysis on ScanNet200

The ScanNet and ScanNet200 ablations examine complementary aspects of the proposed framework rather than repeating the same experiment. Table 4 evaluates semantic segmentation on ScanNet using mIoU and separates the geometry, color, and semantic-residual branches. In contrast, the following ScanNet200 study evaluates instance segmentation using overall and group-wise Head/Common/Tail AP₅₀ under a larger and more imbalanced category space. We analyze instance segmentation across the Head, Common, and Tail groups of ScanNet200. Table 11 reports overall and group-wise AP₅₀ for the controlled configurations. Overall AP₅₀ increases from 28.4 for the 3D-only configuration to 35.6 with CLIP input, 39.8 with geometry–color cRSE, and 40.7 for the full model. Tail AP₅₀ increases from 28.1 to 34.3, 39.7, and 40.2, respectively. Figure 3 compares the published Swin3D result with the full model, showing a 20.7-point Tail AP₅₀ difference, from 19.5 to 40.2. This difference reflects the complete system rather than the semantic residual alone. Within the controlled chain, adding the semantic residual improves Tail AP₅₀ by 0.5 points, from 39.7 to 40.2, after CLIP input and geometry–color cRSE have been enabled.

Figure 3

4.4 Memory and efficiency analysis

To isolate the contribution of the memory-efficient streaming integration, we compare the proposed block-wise implementation with a materialized cRSE implementation that stores the full pairwise bias tensor. Both implementations are evaluated on a single 24 GB RTX 4090 GPU with batch size 4 and the same window capacity.

As shown in Table 12, the materialized implementation at N_w = 32 serves as the ablation setting without streaming integration. Replacing it with the streaming implementation reduces peak online 3D-stage training memory from 18.3 to 13.6 GB, reduces the training step time from 564 to 461 ms, increases throughput from 1.80 to 2.20 iterations/s, and reduces inference latency from 142 to 127 ms per scene. These correspond to a 25.7% reduction in peak memory, an 18.3% reduction in training step time, and a 10.6% reduction in inference latency. The same streaming implementation supports window capacities of N_w = 48 and N_w = 64, with peak memory usage of 15.2 and 18.0 GB, respectively. Offline CLIP extraction, virtual-view rendering, 2D–3D alignment, and feature-cache storage are excluded from these measurements.

The online profile excludes multimodal preprocessing, whose costs are reported separately in Table 13. ScanNet and ScanNet200 require 19.3 and 19.9 s per scene, respectively, for loading the selected RGB-D frames, extracting CLIP features, and performing 2D–3D alignment. S3DIS requires 28.4 s because its virtual views must also be rendered. Cached features occupy 122–144 MB per scene, and 89.5%–94.8% of points receive at least one valid projection. The window statistics show irregular occupancy: as the window capacity increases from 32 to 64, the median token count increases from 19 to 35, while the 95th percentile reaches the capacity in all settings. The median unused capacity is computed as 1 − median(N)/N_w and is reported as an occupancy statistic, not as padding allocated by the streaming kernel.

4.5 Qualitative results

Figure 4 illustrates representative ScanNet cases involving a small object, an ambiguous local boundary, and adjacent furniture with similar appearance. The middle column shows the Swin3D baseline; the red boxes identify the regions discussed in the text. Relative to that baseline, the full model better preserves the small region in these examples and reduces local label confusion.

Figure 4

To complement the ScanNet examples, Figure 5 presents qualitative semantic-segmentation results on S3DIS Area 5, including the input scenes, ground-truth annotations, and predictions from the full model.

Figure 5

To complement the semantic examples, Figure 6 shows representative ScanNet200 instance-segmentation results. The examples compare the input scene, ground-truth instance annotations, the baseline prediction, and the prediction of the full model. The highlighted regions illustrate cases involving small or visually ambiguous instances and show the differences between the two configurations.

Figure 6

5 Conclusion

In this paper, we presented a CLIP-guided point-cloud segmentation framework for bridging the semantic-geometric gap in sparse 3D scene understanding. Dense CLIP features are aligned with 3D points through visibility-aware projection and view pooling to enrich sparse voxel tokens. Building on contextual relative position encoding, the proposed semantic-geometric extension adds a decoupled CLIP-induced discrepancy residual to geometry and appearance biases in local attention. Its block-wise implementation generates and consumes bias tiles during online softmax, avoiding full pairwise bias materialization. Experiments on ScanNet, ScanNet200, and S3DIS show that the largest performance gains arise from aligned CLIP input and structural semantic–geometric modeling, while the decoupled semantic residual provides a smaller complementary refinement. The full system also improves long-tail recognition and reduces the memory required for cRSE integration under the reported setting.

5.1 Limitations and future work

Although aligned CLIP semantics provide effective high-level cues for 3D point-cloud segmentation, the current framework still depends on the availability and quality of multi-view RGB observations, camera calibration, and 2D–3D alignment. Its performance may therefore be affected by inaccurate poses, occlusions, incomplete view coverage, or noisy depth consistency during visibility-aware projection. For ScanNet and ScanNet200, the aligned CLIP features are obtained from native posed RGB-D sequences. For S3DIS, however, posed RGB-D video sequences are not available in the same form, and the aligned CLIP features are obtained from virtual RGB-D views rendered from colored point clouds. Therefore, the S3DIS results should be interpreted under a rendered-view alignment setting rather than a native RGB-D sequence setting.

Another limitation is the system-level cost of multimodal preprocessing. The proposed streaming integration reduces the peak GPU memory of cRSE attention inside the 3D backbone, especially compared with naive pairwise bias materialization. CLIP feature extraction, virtual view rendering, feature storage, and 2D–3D alignment remain offline preprocessing steps; their measured time and cache footprint are reported in Table 13. Thus, the memory reduction refers specifically to semantic-geometric attention during 3D-network training rather than the total peak resource cost of an online end-to-end multimodal system.

In addition, the current empirical evaluation focuses on static indoor benchmarks. Future work will explore more robust cross-modal alignment strategies, efficient online or cached CLIP feature preprocessing pipelines, and extensions of the proposed memory-efficient semantic-geometric attention paradigm to large-scale outdoor environments such as autonomous driving point clouds. We will also investigate 3D-native foundation models to reduce the dependence on dense 2D-to-3D feature projection.

References

• 1

  AkwensiP. H.WangR.GuoB. (2024). Preformer: a memory-efficient transformer for point cloud semantic segmentation. Int. J. Appl. Earth Observ. Geoinf. 128:103730. doi: 10.1016/j.jag.2024.103730

  • CrossRef
  • Google Scholar
  • View reference in article

• 2

  ArmeniI.SenerO.ZamirA. R.JiangH.BrilakisI.FischerM.et al. (2016). “3D semantic parsing of large-scale indoor spaces,” in Proceedings of the IEEE conference on computer vision and pattern recognition (Las Vegas, NV), 1534–1543. doi: 10.1109/CVPR.2016.170

  • CrossRef
  • Google Scholar
  • View reference in article

• 3

  BermanM.TrikiA. R.BlaschkoM. B. (2018). “The lovász-softmax loss: a tractable surrogate for the optimization of the intersection-over-union measure in neural networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition (Salt Lake City, UT), 4413–4421. doi: 10.1109/CVPR.2018.00464

  • CrossRef
  • Google Scholar
  • View reference in article

• 4

  ChenL.MeyerG. P.ZhangZ.WolffE. M.VernazaP. (2025). “Flash3D: super-scaling point transformers through joint hardware-geometry locality,” in Proceedings of the computer vision and pattern recognition conference (Nashville, TN), 6595–6604. doi: 10.1109/CVPR52734.2025.00618

  • CrossRef
  • Google Scholar
  • View reference in article

• 5

  ChenR.LiuY.KongL.ZhuX.MaY.LiY.et al. (2023). “CLIP2Scene: towards label-efficient 3D scene understanding by CLIP,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (Vancouver, BC), 7020–7030. doi: 10.1109/CVPR52729.2023.00678

  • CrossRef
  • Google Scholar
  • View reference in article

• 6

  ChenS.FangJ.ZhangQ.LiuW.WangX. (2021). “Hierarchical aggregation for 3d instance segmentation,” in Proceedings of the IEEE/CVF international conference on computer vision (), 15467–15476. doi: 10.1109/ICCV48922.2021.01518

  • CrossRef
  • Google Scholar
  • View reference in article

• 7

  ChoyC.GwakJ.SavareseS. (2019). “4D spatio-temporal convnets: minkowski convolutional neural networks,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (Montreal, QC), 3075–3084. doi: 10.1109/CVPR.2019.00319

  • CrossRef
  • Google Scholar
  • View reference in article

• 8

  DaiA.ChangA. X.SavvaM.HalberM.FunkhouserT.NießnerM. (2017). “ScanNet: richly-annotated 3D reconstructions of indoor scenes,” in Proceedings of the IEEE conference on computer vision and pattern recognition (Honolulu, HI), 5828–5839. doi: 10.1109/CVPR.2017.261

  • CrossRef
  • Google Scholar

• 9

  DaiA.NießnerM. (2018). “3DMV: joint 3D-multi-view prediction for 3D semantic scene segmentation,” in Proceedings of the European conference on computer vision (ECCV) (Munich), 452–468. doi: 10.1007/978-3-030-01249-6_28

  • CrossRef
  • Google Scholar
  • View reference in article

• 10

  DaoT. (2024). “FlashAttention-2: faster attention with better parallelism and work partitioning,” in International conference on learning representations (Vienna).

  • Google Scholar
  • View reference in article

• 11

  DaoT.FuD.ErmonS.RudraA.RéC. (2022). FlashAttention: fast and memory-efficient exact attention with IO-awareness. Adv. Neural Inf. Process. Syst. 35, 16344–16359. doi: 10.52202/068431-1189

  • CrossRef
  • Google Scholar
  • View reference in article

• 12

  GezawaA. S.LiuC.JiaH.NanehkaranY. A.AlmutairiM. S.ChiromaH. (2023). An improved fused feature residual network for 3D point cloud data. Front. Comput. Neurosci. 17:1204445. doi: 10.3389/fncom.2023.1204445

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 13

  GrahamB.EngelckeM.Van Der MaatenL. (2018). “3D semantic segmentation with submanifold sparse convolutional networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition (Salt Lake City, UT), 9224–9232. doi: 10.1109/CVPR.2018.00961

  • CrossRef
  • Google Scholar
  • View reference in article

• 14

  GuoX.SunY.ZhaoR.KuangL.HanX. (2022). “SWPT: spherical window-based point cloud transformer,” in Proceedings of the Asian conference on computer vision (Macao), 3034–3050. doi: 10.1007/978-3-031-26319-4_24

  • CrossRef
  • Google Scholar
  • View reference in article

• 15

  HanT.ChenY.MaJ.LiuX.ZhangW.ZhangX.et al. (2024). Point cloud semantic segmentation with adaptive spatial structure graph transformer. Int. J. Appl. Earth Observ. Geoinf. 133:104105. doi: 10.1016/j.jag.2024.104105

  • CrossRef
  • Google Scholar
  • View reference in article

• 16

  HeC.LiR.ZhangG.ZhangL. (2024). “ScatterFormer: efficient voxel transformer with scattered linear attention,” in European Conference on computer vision (Milan: Springer), 74–92. doi: 10.1007/978-3-031-73397-0_5

  • CrossRef
  • Google Scholar
  • View reference in article

• 17

  HuW.ZhaoH.JiangL.JiaJ.WongT.-T. (2021). “Bidirectional projection network for cross dimension scene understanding,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, (Nashville, TN), 14373–14382. doi: 10.1109/CVPR46437.2021.01414

  • CrossRef
  • Google Scholar
  • View reference in article

• 18

  JatavallabhulaK. M.KuwajerwalaA.GuQ.OmamaM.ChenT.MaaloufA.et al. (2023). “ConceptFusion: open-set multimodal 3D mapping,” in Robotics: science and systems (Daegu). doi: 10.15607/RSS.2023.XIX.066

  • CrossRef
  • Google Scholar
  • View reference in article

• 19

  JiangL.ZhaoH.ShiS.LiuS.FuC.-W.JiaJ. (2020). “Pointgroup: dual-set point grouping for 3D instance segmentation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (Seattle, WA), 4867–4876. doi: 10.1109/CVPR42600.2020.00492

  • CrossRef
  • Google Scholar
  • View reference in article

• 20

  KhanM.ShahzadM.KhanS.FrazM.ZhuX. (2024). Beyond local patches: preserving global-local interactions by enhancing self-attention via 3D point cloud tokenization. Pattern Recognit. 155:110712. doi: 10.1016/j.patcog.2024.110712

  • CrossRef
  • Google Scholar
  • View reference in article

• 21

  KhodadadM.Shiraee KasmaeeA.MahyarH.RezanejadM. (2024). MLGCN: an ultra-efficient graph convolutional neural model for 3D point cloud analysis. Front. Artif. Intell. 7:1439340. doi: 10.3389/frai.2024.1439340

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 22

  KimH.YoonJ.YoonS. M. (2026). Dynamic window transformer for three-dimensional indoor scene segmentation. Neurocomputing671:132746. doi: 10.1016/j.neucom.2026.132746

  • CrossRef
  • Google Scholar
  • View reference in article

• 23

  LaiX.LiuJ.JiangL.WangL.ZhaoH.LiuS.et al. (2022). “Stratified transformer for 3D point cloud segmentation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (New Orleans, LA), 8500–8509. doi: 10.1109/CVPR52688.2022.00831

  • CrossRef
  • Google Scholar
  • View reference in article

• 24

  LiD.ZhengS.ChenZ.LiX.WangL.DuJ. (2024). Hierarchical local global transformer for point clouds analysis. Int. J. Appl. Earth Observ. Geoinf. 129:103813. doi: 10.1016/j.jag.2024.103813

  • CrossRef
  • Google Scholar
  • View reference in article

• 25

  LiZ.ChenL.LiuY.ZhaoS.GuanQ. (2025). Rwaformer: a lightweight road lidar point cloud segmentation network based on transformer. Front. Comput. Sci. 7:1542813. doi: 10.3389/fcomp.2025.1542813

  • CrossRef
  • Google Scholar
  • View reference in article

• 26

  LiangD.ZhouX.XuW.ZhuX.ZouZ.YeX.et al. (2024). Pointmamba: a simple state space model for point cloud analysis. Adv. Neural Inf. Process. Syst. 37, 32653–32677. doi: 10.52202/079017-1026

  • CrossRef
  • Google Scholar

• 27

  LiuZ.LinY.CaoY.HuH.WeiY.ZhangZ.et al. (2021). “Swin transformer: hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision (Montreal, QC), 10012–10022. doi: 10.1109/ICCV48922.2021.00986

  • CrossRef
  • Google Scholar
  • View reference in article

• 28

  LiuZ.YangX.TangH.YangS.HanS. (2023). “FlatFormer: flattened window attention for efficient point cloud transformer,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (Vancouver, BC), 1200–1211. doi: 10.1109/CVPR52729.2023.00122

  • CrossRef
  • Google Scholar
  • View reference in article

• 29

  LuD.XuL.ZhouJ.GaoK. Y.LiJ. (2025). 3DLST: 3D learnable supertoken transformer for lidar point cloud scene segmentation. Int. J. Appl. Earth Observ. Geoinf. 140:104572. doi: 10.1016/j.jag.2025.104572

  • CrossRef
  • Google Scholar
  • View reference in article

• 30

  LuJ.ZhangY. (2026). Geo-refined point transformer: coordinate-aware excitation and positional upsampling for 3D scene segmentation. J. Comput. Commun. 14, 46–65. doi: 10.4236/jcc.2026.141004

  • CrossRef
  • Google Scholar
  • View reference in article

• 31

  ParkC.JeongY.ChoM.ParkJ. (2022). “Fast point transformer,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) (New Orleans, LA), 16949–16958. doi: 10.1109/CVPR52688.2022.01644

  • CrossRef
  • Google Scholar
  • View reference in article

• 32

  PengB.WuX.JiangL.ChenY.ZhaoH.TianZ.et al. (2024). “OA-CNNS: omni-adaptive sparse cnns for 3D semantic segmentation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (Seattle, WA), 21305–21315. doi: 10.1109/CVPR52733.2024.02013

  • CrossRef
  • Google Scholar
  • View reference in article

• 33

  PengS.GenovaK.JiangC.TagliasacchiA.PollefeysM.FunkhouserT.et al. (2023). “OpenScene: 3D scene understanding with open vocabularies,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (Vancouver, BC), 815–824. doi: 10.1109/CVPR52729.2023.00085

  • CrossRef
  • Google Scholar
  • View reference in article

• 34

  QiC. R.YiL.SuH.GuibasL. J. (2017). “Pointnet++: deep hierarchical feature learning on point sets in a metric space,” in Advances in neural information processing systems, Vol. 30 (Long Beach, CA).

  • Google Scholar
  • View reference in article

• 35

  QianG.LiY.PengH.MaiJ.HammoudH.ElhoseinyM.et al. (2022). Pointnext: revisiting pointnet++ with improved training and scaling strategies. Adv. Neural Inf. Process. Syst. 35, 23192–23204. doi: 10.52202/068431-1685

  • CrossRef
  • Google Scholar
  • View reference in article

• 36

  RadfordA.KimJ. W.HallacyC.RameshA.GohG.AgarwalS.et al. (2021). “Learning transferable visual models from natural language supervision,” in International conference on machine learning (Virtual: PmLR), 8748–8763.

  • Google Scholar
  • View reference in article

• 37

  RonnebergerO.FischerP.BroxT. (2015). “U-Net: convolutional networks for biomedical image segmentation,” in International Conference on Medical image computing and computer-assisted intervention (Munich: Springer), 234–241. doi: 10.1007/978-3-319-24574-4_28

  • CrossRef
  • Google Scholar
  • View reference in article

• 38

  RozenberszkiD.LitanyO.DaiA. (2022). “Language-grounded indoor 3D semantic segmentation in the wild,” in European conference on computer vision (Tel Aviv: Springer), 125–141. doi: 10.1007/978-3-031-19827-4_8

  • CrossRef
  • Google Scholar

• 39

  SchultJ.EngelmannF.HermansA.LitanyO.TangS.LeibeB. (2023). “Mask3D: mask transformer for 3D semantic instance segmentation,” in 2023 IEEE international conference on robotics and automation (ICRA) (London: IEEE), 8216–8223. doi: 10.1109/ICRA48891.2023.10160590

  • CrossRef
  • Google Scholar
  • View reference in article

• 40

  TaiH.HeQ.QianY.HuX.LiX.LiuY.et al. (2026). Open-vocabulary SAM3D: towards training-free open-vocabulary 3D scene understanding. IEEE Trans. Circuits Syst. Video Technol. 36, 10943–10956. doi: 10.1109/TCSVT.2026.3680794

  • CrossRef
  • Google Scholar

• 41

  TakmazA.FedeleE.SumnerR. W.PollefeysM.TombariF.EngelmannF. (2023). “Openmask3D: open-vocabulary 3D instance segmentation,” in Advances in neural information processing systems, Vol. 36 (New Orleans, LA). doi: 10.52202/075280-2989

  • CrossRef
  • Google Scholar
  • View reference in article

• 42

  ThomasH.QiC. R.DeschaudJ.-E.MarcoteguiB.GouletteF.GuibasL. J. (2019). “KPConv: flexible and deformable convolution for point clouds,” in Proceedings of the IEEE/CVF international conference on computer vision (Seoul), 6411–6420. doi: 10.1109/ICCV.2019.00651

  • CrossRef
  • Google Scholar
  • View reference in article

• 43

  WangP.-S. (2023). OctFormer: octree-based transformers for 3D point clouds. ACM Trans. Graphics42, 1–11. doi: 10.1145/3592131

  • CrossRef
  • Google Scholar
  • View reference in article

• 44

  WangY.JiaB.ZhuZ.HuangS. (2025). “Masked point-entity contrast for open-vocabulary 3D scene understanding,” in 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (Nashville, TN: IEEE), 14125–14136. doi: 10.1109/CVPR52734.2025.01318

  • CrossRef
  • Google Scholar
  • View reference in article

• 45

  WangZ.SuY.LiC.WangD.HuangY.LiX.et al. (2025). “Open-vocabulary octree-graph for 3D scene understanding,” in Proceedings of the IEEE/CVF international conference on computer vision (Honolulu, HI), 7037–7047. doi: 10.1109/ICCV51701.2025.00661

  • CrossRef
  • Google Scholar
  • View reference in article

• 46

  WuK.PengH.ChenM.FuJ.ChaoH. (2021). “Rethinking and improving relative position encoding for vision transformer,” in Proceedings of the IEEE/CVF international conference on computer vision (), 10033–10041. doi: 10.1109/ICCV48922.2021.00988

  • CrossRef
  • Google Scholar

• 47

  WuX.JiangL.WangP.-S.LiuZ.LiuX.QiaoY.et al. (2024). “Point transformer V3: simpler faster stronger,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (Montreal, QC), 4840–4851. doi: 10.1109/CVPR52733.2024.00463

  • CrossRef
  • Google Scholar

• 48

  WuX.LaoY.JiangL.LiuX.ZhaoH. (2022). Point transformer V2: grouped vector attention and partition-based pooling. Adv. Neural Inf. Process. Syst. 35, 33330–33342. doi: 10.52202/068431-2415

  • CrossRef
  • Google Scholar

• 49

  YamazakiK.HanyuT.VoK.PhamT.TranM.DorettoG.et al. (2024). “Open-fusion: real-time open-vocabulary 3D mapping and queryable scene representation,” in 2024 IEEE international conference on robotics and automation (ICRA) (Yokohama: IEEE), 9411–9417. doi: 10.1109/ICRA57147.2024.10610193

  • CrossRef
  • Google Scholar

• 50

  YangY.-Q.GuoY.-X.LiuY. (2025a). Swin3D++: effective multi-source pretraining for 3D indoor scene understanding. Comput. Vis. Media11, 465–481. doi: 10.26599/CVM.2025.9450437

  • CrossRef
  • Google Scholar

• 51

  YangY.-Q.GuoY.-X.XiongJ.-Y.LiuY.PanH.WangP.-S.et al. (2025b). Swin3D: a pretrained transformer backbone for 3D indoor scene understanding. Comput. Vis. Media11, 83–101. doi: 10.26599/CVM.2025.9450383

  • CrossRef
  • Google Scholar

• 52

  YuanJ.GaoH.DaiD.LuoJ.ZhaoL.ZhangZ.et al. (2025). “Native sparse attention: hardware-aligned and natively trainable sparse attention,” in Proceedings of the 63rd annual meeting of the association for computational linguistics, Vol. 1 (long papers) (Vienna), 23078–23097. doi: 10.18653/v1/2025.acl-long.1126

  • CrossRef
  • Google Scholar

• 53

  ZhangC.WanH.ShenX.WuZ. (2022). PVT: point-voxel transformer for point cloud learning. Int. J. Intell. Syst. 37, 11985–12008. doi: 10.1002/int.23073

  • CrossRef
  • Google Scholar

• 54

  ZhangS.ZhangS.DengJ.ShenY.MaM.ZhangY. (2025). “PGOV3D: open-vocabulary 3D semantic segmentation with partial-to-global curriculum,” in Proceedings of the 33rd ACM international conference on multimedia (Dublin), 5070–5079. doi: 10.1145/3746027.3755735

  • CrossRef
  • Google Scholar

• 55

  ZhaoH.JiangL.JiaJ.TorrP. H.KoltunV. (2021). “Point transformer,” in Proceedings of the IEEE/CVF international conference on computer vision (Montreal, QC), 16259–16268. doi: 10.1109/ICCV48922.2021.01595

  • CrossRef
  • Google Scholar