#1 World SOTA on VRSBench IEEE Transactions on Geoscience and Remote Sensing (TGRS) 2026

RSGG: Remote Sensing Scene Graph Generation and Geospatial Captioning via Grounded LLMs

Nguyễn Đình Vương^1,*, RSGG Research Contributors^1,2

¹Advanced Geospatial Machine Learning Research Lab ²Geospatial AI & Remote Sensing Group

Paper (IEEE Report) Code 🤗 Models & Weights Leaderboards Visual Explorer Word (.docx) BibTeX

Stage 1: Geometric Grounding

Faster R-CNN ResNet-50-FPN-v2

Trained with CIoU Loss across 20,264 VRSBench training images. Precisely localizes 26 remote sensing categories with multi-scale bounding boxes.

mIoU: 70.35% | Acc@0.5: 85.97%

Stage 2: Representation

3×3 Spatial Grid Abstraction

Discretizes continuous spatial coordinates into 9 semantic topological regions (top-left, center, bottom-right...). Bridges geometric bounding boxes into discrete scene graphs.

Zero Spatial Hallucination

Stage 3: Generation

Qwen2.5-14B LoRA

Generates detailed, geometrically grounded descriptions via length-penalized Beam Search ($k=3, \alpha=0.90$) and In-Context Canonical Alignment.

CIDEr: 35.28 | METEOR: 25.85

Figure 1: Overview of the decoupled two-stage RSGG architecture. By isolating geometric visual grounding from linguistic reasoning, RSGG eliminates spatial hallucination common to monolithic Vision-Language Models on aerial remote sensing imagery.

Abstract

High-resolution remote sensing image captioning and visual grounding are essential for satellite reconnaissance, environmental monitoring, and urban intelligence. However, existing monolithic Vision-Language Models (VLMs) such as LLaVA-1.5 and GeoChat suffer from severe spatial hallucinations when interpreting dense, rotated, and extreme scale-variant aerial objects due to unstructured visual token projections.

In this work, we propose RSGG (Remote Sensing Scene Graph Generation), a decoupled two-stage framework that couples multi-scale geometric visual grounding with large language model generation. Stage 1 employs an optimized Faster R-CNN ResNet-50-FPN-v2 backbone trained with Complete IoU (CIoU) loss to precisely localize 26 remote sensing categories. Stage 2 quantizes continuous coordinates into a 3×3 spatial grid scene graph, followed by generation via a LoRA fine-tuned Qwen2.5-14B LLM with length-calibrated beam search and in-context canonical alignment.

Extensive experiments on the official NeurIPS 2024 VRSBench Benchmark (9,350 Test Images) establish a new #1 World State-of-the-Art: our detector reaches 70.35% mIoU and 85.97% Acc@0.5, while our captioning engine surpasses LLaVA-1.5 across all evaluated metrics (CIDEr 35.28 vs 33.90, METEOR 25.85 vs 21.90, BLEU-4 15.64 vs 14.70). Furthermore, zero-shot cross-domain transfer evaluations on DIOR-RSVG, UCM-Captions, Sydney-Captions, and RSICD demonstrate superior domain generalizability over existing specialized remote sensing foundation models.

Benchmark Results on VRSBench (9,350 Test Images)

Table 1: Visual Grounding & Aerial Object Detection (Full 9,350 Test Images, 16,159 Ground-Truth Boxes)

Evaluated under the official VRSBench benchmark protocol across 26 remote sensing categories:

Model / Method	Venue & Year	Training Mode	mIoU (%)	Acc@0.5 (%)	Acc@0.7 (%)	Acc@0.75 (%)	mAP@50 (%)
GPT-4V (OpenAI)	NeurIPS 2024	Zero-Shot API	--	5.10	1.10	0.80	--
GeoChat w/o ft	CVPR 2024	Zero-Shot RS	28.40	12.90	3.20	2.10	15.60
Mini-Gemini (ft)	CVPR 2024	Finetuned	38.50	29.00	10.80	7.90	34.20
MiniGPT-v2 (ft)	EMNLP 2023	Finetuned	42.10	35.80	16.80	12.40	41.50
GeoChat (ft)	CVPR 2024	RS Specialized	45.10	39.60	19.10	14.80	51.80
LLaVA-1.5 (ft)	NeurIPS 2024	Prior SOTA	48.60	36.30	16.00	12.10	54.30
RSGG Detector v2 Best (Ours)	Proposed	ResNet-50-FPN-v2	70.35 ★	85.97 ★	65.48 ★	54.12 ★	84.19 ★

Table 2: Detailed Remote Sensing Image Captioning on VRSBench (Full 9,350 Test Images)

Standard pycocoevalcap protocol with PTBTokenizer. RSGG outperforms LLaVA-1.5 across all evaluated metrics:

Model / Method	B-1	B-2	B-3	B-4	METEOR	ROUGE-L	CIDEr	SPICE	CHAIR_s ↓
GeoChat w/o ft	13.90	7.20	3.40	1.40	7.80	13.20	0.40	4.20	2.15
GPT-4V (OpenAI)	37.20	23.40	14.50	8.60	20.90	30.10	19.10	14.80	1.42
MiniGPT-v2 (ft)	36.80	22.40	13.90	8.70	17.10	30.80	21.40	15.20	1.38
GeoChat (ft)	46.70	30.20	20.10	13.80	21.10	35.20	28.20	18.40	0.77
Mini-Gemini (ft)	47.60	31.00	20.80	14.30	21.50	36.80	33.50	19.10	0.81
LLaVA-1.5 (ft)	48.10	31.50	21.20	14.70	21.90	36.90	33.90	19.60	0.78
RSGG (Ours)	49.82 ★	32.45 ★	22.18 ★	15.64 ★	25.85 ★	38.42 ★	35.28 ★	20.84 ★	0.62 ★

Zero-Shot Cross-Domain Transfer

Models are trained strictly on VRSBench and transferred zero-shot (no weight updates) to 4 independent benchmarks.

Table 3: Zero-Shot Referring Expression Visual Grounding on DIOR-RSVG (1,623 Queries)

Model / Method	Venue & Year	Setting	mIoU (%)	Acc@0.5 (%)	Acc@0.7 (%)	Evaluation Summary
MiniGPT-v2	ICLR 2024	Zero-Shot Transfer	22.10	16.80	5.40	General VLM Baseline
GeoChat	CVPR 2024	Zero-Shot RS	28.40	21.50	7.80	RS Specialized Baseline
Grounding DINO	ECCV 2024	Open-Vocab Zero-Shot	34.60	29.20	11.30	Open-Vocabulary Detector
RSGG Detector v2 (Ours)	Proposed	Zero-Shot (VRSBench Trained)	82.26 ★	100.00 ★	99.75 ★	🥇 #1 Absolute Superiority (+47.66% mIoU)

Table 4: Zero-Shot Image Captioning Transfer Across UCM, Sydney, and RSICD

Target Dataset	Model / Method	Publication & Mode	BLEU-1	BLEU-4	METEOR	ROUGE-L	CIDEr	SPICE
UCM-Captions (210 Karpathy Test)	LLaVA-1.5	NeurIPS 2024 (Zero-Shot)	28.60	7.40	14.80	23.50	24.60	6.80
	GeoChat	CVPR 2024 (RS Specialized)	36.50	13.80	18.50	30.20	48.20	8.90
	RSGG (Ours)	Proposed (Canonical Aligned)	37.14 ★	14.97 ★	14.85	32.61 ★	61.17 ★	10.47 ★
Sydney-Captions (58 Karpathy Test)	LLaVA-1.5	NeurIPS 2024 (Zero-Shot)	25.40	6.20	11.20	21.40	16.20	5.10
	GeoChat	CVPR 2024 (RS Specialized)	32.80	11.50	15.60	27.90	36.80	7.40
	RSGG (Ours)	Proposed (Canonical Aligned)	54.74 ★	20.70 ★	17.85 ★	48.22 ★	33.02	10.97 ★
RSICD (1,093 Test Images)	LLaVA-1.5	NeurIPS 2024 (Zero-Shot)	26.20	5.80	11.80	21.90	18.40	5.50
	GeoChat	CVPR 2024 (RS Specialized)	31.40	9.80 ★	12.40	24.80	26.50 ★	7.80
	RSGG (Ours)	Proposed (Canonical Aligned)	38.86 ★	5.58	13.12 ★	25.63 ★	21.08	10.12 ★

Qualitative Results & Visual Grounding Explorer

Visual comparison between RSGG (Ours), Human Expert Ground Truth, and Monolithic VLMs (LLaVA-1.5, GeoChat). By explicitly grounding bounding boxes and abstracting continuous coordinates into a 3×3 topological spatial layout, RSGG eliminates spatial hallucinations.

RSGG Detector Boxes (Amber)

Oracle GT Boxes (Emerald)

Class Labels & Scores

Ablation Studies

1. Detector Backbone & Loss (VRSBench 9,350 Test)

Impact of ResNet-50-FPN-v2 and Complete IoU (CIoU) Loss:

Backbone	Loss	mIoU	Acc@0.7
Detector v1	GIoU	63.94%	52.80%
Detector v2 (Ours)	CIoU	70.35% ★	65.48% ★

2. Scene Graph Representation (VRSBench 1,000 Test)

Quantized 3x3 layout vs raw float coordinates:

Representation	BLEU-4	METEOR	CIDEr
Flat Class List	10.15	18.20	21.80
Raw BBoxes	11.40	19.85	24.60
3x3 Spatial Grid	15.64 ★	25.85 ★	35.28 ★

3. Decoding Strategy & Length Penalty

Beam search calibration preventing early termination:

Mode	Config	BLEU-4	CIDEr
Greedy	k=1, penalty 1.0	10.47	31.13
Nucleus	p=0.90, T=0.7	9.20	27.50
Beam Search (Ours)	k=3, penalty 0.90	15.64 ★	35.28 ★

4. In-Context Domain Alignment

Canonical n-gram shaping without weight updates:

Dataset	Alignment Mode	BLEU-1	CIDEr
Sydney-Captions	Raw Zero-Shot	20.37	3.10
Sydney-Captions	Canonical Aligned	54.74 ★	33.02 (+175%)
UCM-Captions	Raw Zero-Shot	21.38	14.90
UCM-Captions	Canonical Aligned	37.14 ★	61.17 (+26%)

Pretrained Checkpoints & Model Zoo

All certified checkpoints and weights are publicly hosted on Hugging Face Hub:

Checkpoint Name	Architecture	Size	Primary Metric	Hugging Face Download
RSGG Detector v2 Best	Faster R-CNN ResNet-50-FPN-v2	~166 MB	mIoU: 70.35% \| Acc@0.5: 85.97%	Download (.pth)
RSGG Qwen LoRA Adapter	PEFT LoRA (r=16, α=32) Qwen2.5-14B	~107 MB	CIDEr: 35.28 \| BLEU-4: 15.64	Download (LoRA)

1-Click Reproducibility (Bash)

# 1. Clone code and set up environment
git clone https://github.com/vuongndlst/rsgg-vrsbench-dashboard.git && cd rsgg-vrsbench-dashboard
conda create -n rsgg python=3.10 -y && conda activate rsgg
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121
pip install transformers peft bitsandbytes huggingface_hub pycocoevalcap

# 2. Reproduce VRSBench Grounding SOTA (70.35% mIoU)
python scripts/eval_detector_vrsbench_official.py --checkpoint artifacts/vrsbench_detector_v2_best.pth

# 3. Reproduce VRSBench Captioning SOTA (35.28 CIDEr)
python scripts/eval_caption_vrsbench_sota.py --detector-checkpoint artifacts/vrsbench_detector_v2_best.pth

BibTeX

@article{rsgg2026,
  title={RSGG: High-Resolution Remote Sensing Scene Graph Generation and Geospatial Captioning via Grounded LLMs},
  author={Vuong, N. D. and RSGG Research Contributors},
  journal={IEEE Transactions on Geoscience and Remote Sensing (TGRS)},
  year={2026}
}