Andrii Bidochko

• Updated: July 15, 2026
• 7 min read

TMI: Text-to-Image Meets Image-to-Image for Complementary Data Synthesis to Boost Long-Tailed Instance Segmentation

Direct Answer

The paper TMI: Text-to-Image Meets Image-to-Image for Complementary Data Synthesis to Boost Long‑Tailed Instance Segmentation introduces a hybrid data‑generation pipeline that couples large‑scale text‑to‑image (T2I) synthesis with a context‑aware image‑to‑image (I2I) editor called VRAIN. By blending the breadth of T2I with the precision of I2I, the authors dramatically improve segmentation performance on rare categories in the LVIS benchmark.

Background: Why This Problem Is Hard

Instance segmentation models are only as good as the data they see during training. In real‑world image collections, the distribution of object categories follows a long‑tailed pattern: a handful of classes dominate the pixel count while hundreds of fine‑grained categories appear only a few times. This imbalance creates two intertwined challenges:

• Insufficient examples for rare classes – deep networks struggle to learn discriminative features when the signal is scarce.
• Fine‑grained inter‑class ambiguity – many tail classes differ by subtle visual cues (e.g., “red‑capped mushroom” vs. “brown‑capped mushroom”), demanding high‑quality, context‑rich training samples.

Traditional remedies—re‑sampling, loss re‑weighting, or synthetic copy‑paste—address the quantity problem but introduce new flaws. Copy‑paste methods often place objects on unrealistic backgrounds, breaking scene coherence and widening the domain gap between synthetic and real images. Conversely, modern text‑to‑image generators (e.g., diffusion models) can produce diverse scenes, yet they inherit noisy pseudo‑labels from the prompting process and tend to under‑represent the tail of the vocabulary.

What the Researchers Propose

The authors present a two‑stage framework named TMI (Text‑to‑Image Meets Image‑to‑Image). The core idea is to let each synthesis paradigm compensate for the other’s weakness:

1. T2I Generation Branch – A large diffusion model receives carefully crafted textual prompts that enumerate target categories. This branch supplies a wide variety of scenes, lighting conditions, and object poses, ensuring that even rare classes appear in many contexts.
2. Teacher‑Student Label Filter – A high‑capacity “teacher” segmentation model evaluates the T2I outputs and retains only those pixels that match the prompt‑specified categories with high confidence. The filtered masks become the “student” labels used for downstream training, dramatically reducing label noise.
3. VRAIN (Verified Rare‑class Augmentation via INstructed editing) – An I2I editor that inserts verified instances into real‑world background images. VRAIN leverages a separate instruction‑guided diffusion network to synthesize objects that blend seamlessly with the surrounding context, respecting occlusion, lighting, and perspective.

By chaining these components, TMI delivers both breadth (many scenes) and depth (high‑fidelity, context‑aware edits) without sacrificing label reliability.

How It Works in Practice

The operational workflow can be broken down into four logical stages, each of which can be run independently or as part of an automated pipeline:

1. Prompt Engineering & T2I Sampling

For every target category, a natural‑language prompt is generated (e.g., “a photo of a rare blue‑spotted salamander perched on a mossy log”). A diffusion model such as Stable Diffusion produces a batch of images per prompt, guaranteeing diverse backgrounds and viewpoints.

2. Teacher‑Student Mask Verification

A pre‑trained, high‑capacity instance segmentation model (the teacher) runs inference on each synthetic image. Pixels that the teacher assigns to the prompted class with a confidence above a calibrated threshold are kept; everything else is discarded. The resulting masks become the student labels, which are far less noisy than raw diffusion outputs.

3. VRAIN‑Driven I2I Editing

VRAIN receives two inputs: a real‑world “canvas” image and a verified object mask from the previous step. An instruction‑conditioned diffusion network then renders the object directly onto the canvas, adjusting color balance, shadows, and occlusion cues to match the scene. The process is guided by a semantic map that tells VRAIN where the object should sit (e.g., “on the left side of a wooden table”).

4. Dataset Assembly & Model Training

The final synthetic dataset is a union of:

• Pure T2I images with teacher‑filtered masks.
• VRAIN‑edited composites that embed rare objects into natural scenes.

These images are mixed with the original LVIS training set and fed to a standard Mask R‑CNN or a transformer‑based segmentation backbone. Because the synthetic portion is heavily enriched for tail classes, the model receives a balanced learning signal.

Evaluation & Results

The authors benchmarked TMI on the LVIS v1.0 dataset, a widely used long‑tailed instance segmentation benchmark containing over 1,200 categories. Evaluation focused on three standard metrics:

• AP (Average Precision) – overall detection quality.
• AP_r (Rare‑class AP) – performance on categories with fewer than 10 training instances.
• AP_c (Common‑class AP) – performance on well‑represented categories.

Key findings include:

• When added to a ResNet‑101 backbone, TMI raised overall AP by **+4.0 points** compared to the strongest prior data‑synthesis baseline.
• Rare‑class AP saw a **+9.5 point** jump, confirming that VRAIN’s targeted edits effectively close the long‑tail gap.
• Scaling the backbone to Swin‑Transformer‑L further amplified gains, indicating that the synthetic data remains beneficial even for high‑capacity models.
• Ablation studies showed that removing either the teacher‑student filter or the VRAIN module reduced performance by more than 3 AP points, underscoring the necessity of both components.

These results demonstrate that a hybrid T2I + I2I pipeline can produce synthetic data that is not only abundant but also semantically and visually aligned with real‑world distributions.

Why This Matters for AI Systems and Agents

Long‑tailed instance segmentation is a bottleneck for any AI system that must understand complex visual environments—autonomous robots, retail inventory bots, or augmented‑reality assistants. By delivering high‑quality, tail‑rich training data without manual annotation, TMI enables several practical advances:

• Reduced annotation cost – Companies can replace expensive human labeling pipelines with automated synthesis, freeing budget for model iteration.
• Improved robustness in edge cases – Agents that rely on segmentation (e.g., a warehouse robot distinguishing rare tool types) become more reliable because the model has seen those objects in realistic contexts.
• Faster domain adaptation – VRAIN’s ability to embed objects into existing scenes means synthetic data can be tailored to a specific deployment environment (factory floor, retail shelf, etc.) with minimal effort.
• Seamless integration with AI orchestration platforms – The modular nature of TMI fits naturally into workflow automation tools. For example, the Workflow automation studio can schedule prompt generation, teacher‑student filtering, and VRAIN editing as discrete tasks, allowing data engineers to scale synthesis on demand.

In practice, a developer could spin up a pipeline that continuously harvests new product images from a catalog, runs them through VRAIN to inject rare accessories, and feeds the output directly into a segmentation model that powers an AI marketing agent. The result is a self‑reinforcing loop where the agent’s perception improves as the synthetic data pool expands.

What Comes Next

While TMI marks a significant step forward, several open challenges remain:

• Prompt diversity automation – Current prompt engineering relies on heuristics. Future work could explore LLM‑driven prompt generation that automatically discovers under‑represented attribute combinations.
• Cross‑modal consistency – Ensuring that the textual description, the generated mask, and the final I2I edit remain perfectly aligned is non‑trivial, especially for highly deformable objects.
• Real‑time synthesis for on‑device agents – Reducing the computational footprint of diffusion models would allow edge devices to generate tail‑class examples on the fly.
• Evaluation beyond LVIS – Applying TMI to other long‑tailed benchmarks (e.g., OpenImages, COCO‑Stuff) will test its generality.

Addressing these points could unlock new use cases such as:

• Dynamic augmentation for OpenAI ChatGPT integration where visual context is generated on demand.
• Real‑time visual assistants that adapt to user‑specific objects via Telegram integration on UBOS, delivering personalized image synthesis directly to a chat interface.
• Enterprise‑grade pipelines built on the Enterprise AI platform by UBOS, where synthetic data is versioned, audited, and governed alongside production models.

In summary, TMI’s hybrid synthesis strategy opens a practical pathway to bridge the long‑tail gap in instance segmentation, and its modular design invites immediate adoption within existing AI infrastructure.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.