Andrii Bidochko

• Updated: July 23, 2026
• 7 min read

Multi-Conditioned Diffusion Synthesis of Sand Boils for Low-Resource Earthen-Levee Inspection

Direct Answer

The paper introduces a diffusion‑based synthesis pipeline that creates realistic sand‑boil images for earthen‑levee inspection using a fine‑tuned Stable Diffusion XL model, a multi‑branch ControlNet stack, and a soft‑mask inpainting protocol. This matters because it supplies high‑quality, label‑accurate synthetic data where real annotations are scarce, enabling safer, data‑driven monitoring of critical flood‑control infrastructure.

Background: Why This Problem Is Hard

Earthen levees protect millions of people from catastrophic flooding, yet they develop “sand boils” – localized eruptions of sand and water that signal internal erosion. Detecting these defects at the pixel level requires large, expertly annotated image collections. In practice, field crews capture only a handful of usable photos per inspection season, and manual labeling is both time‑consuming and error‑prone.

Existing computer‑vision pipelines for defect detection rely on two fragile assumptions:

• Abundant labeled data: Standard supervised models need thousands of diverse examples, a luxury unavailable for rare civil‑engineering anomalies.
• Simple compositing tricks: Prior attempts to augment data by pasting sand‑boil cutouts onto background scenes suffer from visible seams, color mismatches, and inconsistent lighting, which degrade model generalization.

Because sand‑boils are safety‑critical, any false negative can lead to undetected failure, while false positives waste costly remediation resources. The scarcity of reliable training data therefore creates a bottleneck that hampers the deployment of AI‑assisted inspection tools.

What the Researchers Propose

The authors present a three‑layered framework that synthesizes inspection‑grade imagery while preserving the authenticity of the defect region:

1. Stable Diffusion XL fine‑tuned with DreamBooth: A small curated set of real sand‑boil photos teaches the diffusion model the visual signature of the defect without overfitting.
2. Multi‑branch ControlNet stack: Separate ControlNet branches ingest edge maps, depth cues, and soft‑mask guidance, allowing the model to respect the geometry of the original scene while generating new surrounding context.
3. Soft‑mask inpainting protocol: Instead of naïve cut‑and‑paste, the pipeline masks only the defect pixels and re‑renders the surrounding area, eliminating seams and preserving color consistency.

Additionally, a taxonomy‑driven “Prompt Atlas” expands a single domain description into a stratified, CLIP‑validated prompt bank, enabling text conditioning for new defect classes without code changes.

How It Works in Practice

Conceptual Workflow

The end‑to‑end pipeline follows a linear yet modular sequence:

1. Reference collection: Engineers gather a minimal set (≈30) of high‑resolution sand‑boil photographs, each paired with a manually drawn soft mask that isolates the defect.
2. DreamBooth fine‑tuning: The Stable Diffusion XL backbone is adapted to the sand‑boil visual domain using the reference set, learning a compact embedding that captures texture, granularity, and moisture sheen.
3. Prompt Atlas generation: A domain taxonomy (e.g., “wet sand eruption”, “dry sand plume”) is fed to a CLIP‑based validator, producing a diversified prompt list that guides the diffusion process.
4. ControlNet conditioning: For each synthesis request, the system extracts edge, depth, and mask maps from the reference image. These maps travel through dedicated ControlNet branches, ensuring structural fidelity.
5. Soft‑mask inpainting: The diffusion model fills in the background around the masked defect, preserving the original defect pixels. The result is a seamless composite where the sand‑boil appears naturally embedded.
6. Post‑generation filtering: A CLIP admissibility filter discards out‑of‑distribution outputs, leaving only high‑confidence synthetic samples.

Component Interactions

Each component plays a distinct role while sharing a common data contract:

• DreamBooth embedding supplies the semantic “essence” of sand‑boils to the diffusion core.
• ControlNet branches enforce geometric constraints, preventing the model from hallucinating unrealistic terrain around the defect.
• Soft‑mask inpainting acts as a safeguard, guaranteeing that the defect pixels remain untouched, which is crucial for downstream segmentation label reliability.
• Prompt Atlas injects textual diversity, allowing the same visual defect to appear under varied lighting, weather, and seasonal contexts.

This separation of concerns makes the pipeline extensible: swapping a ControlNet branch for a new modality (e.g., thermal maps) or expanding the Prompt Atlas to cover other infrastructure defects can be done without retraining the entire system.

Evaluation & Results

Test Scenarios

The researchers evaluated the pipeline on three fronts:

• Quantity of synthetic output: From the 30 real references, the system generated 1,020 candidate images.
• Quality filtering: A CLIP‑based admissibility filter retained 815 images, representing a 79.9% pass rate.
• Distributional fidelity: They compared the synthetic set against the real reference distribution using Fréchet Inception Distance (FID) and Kernel Inception Distance (KID).

Key Findings

Across the metrics, the diffusion‑synthesized images consistently outperformed a Poisson‑noise baseline:

Beyond raw numbers, qualitative audits revealed no memorization of the original images—synthetic samples displayed novel terrain configurations while preserving the defect’s visual signature. The soft‑mask preset emerged as the most reliable for downstream labeling, whereas other presets offered higher diversity at the cost of occasional label drift.

Why This Matters for AI Systems and Agents

For AI practitioners building inspection agents, the pipeline delivers two immediate advantages:

• Data‑centric scalability: Synthetic images can be generated on demand, allowing continuous model retraining as new field conditions emerge, without the need for costly annotation campaigns.
• Label provenance: Because the defect pixels are never altered, each synthetic image comes with a built‑in segmentation mask, eliminating the “label‑generation” step that typically introduces noise.

These benefits translate into more robust defect‑detection agents that can be deployed on edge devices, integrated into autonomous drone surveys, or embedded within larger UBOS platform overview for real‑time monitoring. Moreover, the modular ControlNet architecture aligns with the emerging practice of “agent‑orchestrated AI pipelines,” where a supervisory agent selects the appropriate conditioning branch (e.g., depth vs. edge) based on sensor availability.

From a business perspective, organizations can reduce inspection labor costs, accelerate response times, and improve regulatory compliance—all while leveraging a reproducible, open‑source workflow that fits within existing Workflow automation studio environments.

What Comes Next

While the study establishes a solid foundation, several open challenges remain:

• Downstream segmentation performance: The authors deliberately left the impact on actual segmentation models for future work. Empirical studies are needed to quantify gains in detection accuracy when training on the synthetic set.
• Generalization to other defect types: Extending the Prompt Atlas to cracks, vegetation overgrowth, or seepage zones will test the taxonomy‑driven approach’s adaptability.
• Real‑time synthesis: Integrating the pipeline with streaming sensor feeds (e.g., UAV video) could enable on‑the‑fly data augmentation for continual learning agents.

Potential next steps for developers include:

1. Embedding the soft‑mask inpainting module into a OpenAI ChatGPT integration that automatically generates synthetic samples based on natural‑language inspection requests.
2. Connecting the pipeline to a Telegram integration on UBOS so field crews can submit raw photos and receive instantly augmented datasets for local model fine‑tuning.
3. Leveraging Enterprise AI platform by UBOS to orchestrate large‑scale batch generation, quality filtering, and versioned dataset storage.

By addressing these avenues, the community can move from proof‑of‑concept to production‑grade, AI‑enhanced levee monitoring systems that operate reliably under diverse environmental conditions.

References

• Thapa, P. J., Naeem, A. B., Dey, A., Katwal, A., & Hoque, M. T. (2026). Multi‑Conditioned Diffusion Synthesis of Sand Boils for Low‑Resource Earthen‑Levee Inspection. arXiv preprint arXiv:2607.08794.
• Rombach, R., Blattmann, A., et al. (2022). “High‑Resolution Image Synthesis with Latent Diffusion Models.”
• Huang, G., et al. (2023). “ControlNet: Adding Conditional Control to Diffusion Models.”
• Radford, A., et al. (2021). “Learning Transferable Visual Models From Natural Language Supervision.”

Ready to explore how synthetic data can accelerate your AI projects? Visit the UBOS homepage for more tools, templates, and partner opportunities.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.