--- license: apache-2.0 language: - en tags: - OneScience - Earth Science - Remote Sensing Image-Text Retrieval - Vision-Language Model frameworks: PyTorch datasets: - RSITMD - RSICD - UCM-Captions ---

RemoteCLIP

# Model Introduction RemoteCLIP is a vision-language foundation model for remote sensing imagery and text. It aligns remote sensing visual semantics with natural-language descriptions through CLIP dual encoders and bidirectional contrastive learning, supporting cross-modal retrieval and transfer to downstream remote sensing tasks. Paper: RemoteCLIP: A Vision Language Foundation Model for Remote Sensing https://arxiv.org/abs/2306.11029 # Model Description RemoteCLIP was proposed by a research team from the National University of Defense Technology and other institutions. The model performs continual pretraining with RSITMD, RSICD, UCM-Captions, and remote sensing image-text data constructed by converting task datasets. It is suitable for remote sensing image-text retrieval, zero-shot classification, and vision-language representation learning. # Use Cases | Scenario | Description | | :---: | :--- | | Remote sensing image-text retrieval | Compute cross-modal similarities between image and text features. | | Multi-positive contrastive learning | Use `pair_ids` to express positive relationships such as one image paired with multiple texts. | | Zero-shot scene classification | Match natural-language class prompts with image features to recognize remote sensing scenes without training an additional classification head. | | Few-shot visual recognition | Transfer vision-language representations and perform few-shot fine-tuning or linear probing for remote sensing classification with limited labels. | | Local engineering validation | Use a small amount of synthetic data to check the training, inference, and evaluation workflows. | | Multi-GPU training | Launch distributed training with `torchrun`. | # Usage Guide ## 1. OneCode Experience intelligent one-click AI4S programming through the OneCode online environment: [Click to Experience Intelligent One-Click AI4S Programming](https://web-2069360198568017922-iaaj.ksai.scnet.cn:58043/home) ## 2. Download and Installation ```bash hf download OneScience-Group/RemoteCLIP --local-dir ./RemoteCLIP cd RemoteCLIP ``` ### Environment Dependencies **Hardware Requirements** - A GPU or DCU is recommended. - CPU can be used for small-configuration connectivity validation; full training and inference will be slow. - DCU users must install DTK in advance. DTK 25.04.2 or above, or the OneScience recommended version matching the current cluster, is recommended. **DCU Environment** ```bash # Please activate DTK and CONDA first conda create -n onescience311 python=3.11 -y conda activate onescience311 # uv installation is supported pip install onescience[earth-dcu] -i http://mirrors.onescience.ai:3141/pypi/simple/ --trusted-host mirrors.onescience.ai ``` **GPU Environment** ```bash # Please activate CONDA first conda create -n onescience311 python=3.11 -y libstdcxx-ng=12 libgcc-ng=12 gcc_linux-64=12 gxx_linux-64=12 conda activate onescience311 # uv installation is supported pip install onescience[earth-gpu] -i http://mirrors.onescience.ai:3141/pypi/simple/ --trusted-host mirrors.onescience.ai ``` ### Training Data Introduction By default, 8 training and 4 test synthetic image-text samples are used to validate the engineering workflow. They are saved as `data/train.npz` and `data/test.npz`, respectively. Tokens follow the OpenAI CLIP BPE vocabulary of 49,408 entries and the SOT, EOT, and padding sequence constraints. The synthetic data preserves the official model input specifications of 3-channel `224x224` images and CLIP text sequences of length 77. Real data must be preprocessed and converted to the following NPZ training protocol. This protocol is consistent with the model input specification but is not the download format of the original datasets. ```text images: float32 [N,3,224,224] tokens: int64 [N,77] pair_ids: int64 [N] ``` `fake_data.py` automatically writes the `protocol` and `data_source` protocol metadata. These fields must be retained when using real data. ```bash python scripts/fake_data.py ``` ### Training ```bash python scripts/train.py ``` For multi-GPU training, use: ```bash torchrun --nproc_per_node=8 scripts/train.py ``` Training optimizes the image and text dual encoders with a multi-positive bidirectional contrastive objective and saves a checkpoint and aggregate training metrics. The default configuration is intended for quick workflow validation. Formal experiments should use the image-text data scale, model configuration, and training duration corresponding to the paper. ```text result/checkpoints/remoteclip.pt result/training/metrics.json ``` ### Training Weights This repository will provide RemoteCLIP training weights in the `weight/` folder. The weight files will be uploaded soon and are expected to be available in the near future. ### Inference ```bash python scripts/inference.py ``` Inference loads the training checkpoint, computes test-set image and text features and their similarities, and saves the results to: ```text result/output/retrieval.npz ``` ### Evaluation and Visualization ```bash python scripts/result.py ``` Using `pair_ids`, evaluation reports bidirectional retrieval R@1, R@5, R@10, and mean recall, and generates a similarity heatmap. Results on synthetic data are only for engineering workflow validation and do not represent full-paper performance. ```text result/evaluation/metrics.json result/evaluation/similarity_matrix.png ``` # Official OneScience Resources | Platform | OneScience Main Repository | Skills Repository | | --- | --- | --- | | Gitee | https://gitee.com/onescience-ai/onescience | https://gitee.com/onescience-ai/oneskills | | GitHub | https://github.com/onescience-ai/OneScience | https://github.com/onescience-ai/oneskills | # Citation and License This repository is a reproduction of the original RemoteCLIP paper. Use of the code and data in this repository remains subject to the licenses and terms of use of their respective projects.