Andrii Bidochko

• Updated: July 24, 2026
• 7 min read

Tactile and Vision Conditioned Contact‑Centric Control for Whole‑Arm Manipulation (TACTIC)

Direct Answer

TACTIC (Tactile and Vision Conditioned Contact‑Centric Control) is a receding‑horizon controller that lets robots use both RGB‑D vision and distributed tactile sensing to plan and execute whole‑arm manipulation tasks while actively regulating contact forces.

It matters because it bridges the gap between high‑level perception and low‑level force control, enabling robots to reliably handle complex, multi‑contact operations that were previously fragile or required hand‑crafted policies.

Background: Why This Problem Is Hard

Whole‑arm manipulation differs from traditional end‑effector grasping in three fundamental ways:

• Coupled Kinematics and Contact: Every link of the arm can touch the environment, so the robot’s configuration directly influences the distribution of forces.
• Partial Observability: Occlusions hide many contact points from cameras, while tactile sensors provide only local pressure data.
• Data Sparsity: Learning‑based policies rely on large datasets, yet the space of multi‑link contact configurations is combinatorially large, leaving most scenarios under‑represented.

Existing pipelines typically assume a clean separation between motion planning (geometry‑only) and force control (contact‑only). Model‑free reinforcement learning can generate impressive motions, but it often violates physical constraints when the robot encounters unseen contact patterns, leading to instability or damage.

Consequently, deploying robots in unstructured settings—such as disaster response, home assistance, or flexible manufacturing—remains risky. A controller that can predict how contacts will evolve and adjust actions in real time is essential for reliable whole‑arm manipulation.

What the Researchers Propose

The authors introduce TACTIC, a hybrid predictive framework that fuses visual and tactile inputs into a compact 2‑D proximity map and couples this perception with a learned latent dynamics model. The key components are:

• Contact‑Centric Hybrid Model: A learned, action‑conditioned latent space predicts how contacts shift, while analytical kinematics translate those predictions into force vectors via contact Jacobians.
• Receding‑Horizon MPC Planner: A sampling‑based model‑predictive controller evaluates many candidate action sequences, steering them toward directions that modulate forces rather than merely following a trajectory.
• Contact‑Aware Action Sampling: Before rollout, each sampled action is projected using the contact Jacobian, biasing the search toward motions that respect current force constraints.
• Multi‑Objective Cost Function: The planner balances task progress (e.g., moving a target) against whole‑arm force regulation and proximity goals, ensuring safe, smooth interactions.

In essence, TACTIC treats contact as a first‑class planning variable, rather than an afterthought, allowing the robot to anticipate and shape its own interaction landscape.

How It Works in Practice

Conceptual Workflow

1. Sensing Fusion: An RGB‑D camera captures the scene geometry, while a dense array of tactile sensors embedded along the arm reports pressure maps. Both streams are projected onto a shared 2‑D proximity grid that encodes distance to nearby surfaces.
2. Latent Dynamics Prediction: The proximity grid, together with the current robot state and a candidate action, is fed into a neural encoder that produces a latent representation. A learned transition model predicts the next latent state, effectively forecasting how contacts will move.
3. Analytical Kinematics Integration: The predicted latent state is decoded into a set of contact points. Using the robot’s Jacobian at each contact, the system computes expected interaction forces.
4. Sampling‑Based MPC: Hundreds of action sequences are sampled. Each sequence is first adjusted by the contact Jacobian (projected sampling) and then rolled out through the hybrid model to estimate future contacts and forces.
5. Cost Evaluation & Selection: A composite cost evaluates task progress, proximity to obstacles, and deviation from desired force envelopes. The lowest‑cost sequence is executed for a short horizon, after which the loop repeats.

Component Interaction Diagram

What Sets TACTIC Apart

• Hybrid Modeling: By blending learned dynamics with exact kinematic equations, the system retains physical plausibility while capturing complex contact phenomena.
• Force‑Modulating Sampling: Traditional MPC samples actions blindly; TACTIC’s Jacobian‑based projection biases the search toward force‑safe directions, dramatically improving convergence.
• Contact‑Centric Objective: Instead of treating contacts as constraints, they become part of the optimization, enabling purposeful sliding, rolling, and breaking of contacts.

Evaluation & Results

Simulation Benchmarks

The authors compared TACTIC against three baselines:

• A model‑based controller that uses only vision and analytical dynamics.
• A model‑free reinforcement learning policy trained on the same tasks.
• A hybrid approach lacking contact‑aware sampling.

Across five simulated whole‑arm tasks—such as pushing a block with the forearm, sliding a rod along a wall, and navigating a dynamic 3‑D maze—TACTIC consistently achieved higher success rates, smoother force profiles, and lower task completion times. Notably, in scenarios with heavy occlusion, the tactile component prevented catastrophic collisions that vision‑only baselines suffered.

Real‑World Demonstrations

Three physical experiments validated the simulation findings:

1. Manikin Turn‑Over: The robot used its upper arm to flip a life‑size mannequin, coordinating multiple contacts to avoid excessive torque.
2. Repositioning a Manikin: After turning, the arm slid the mannequin across a cluttered surface, continuously adjusting contact forces.
3. 3‑D Dynamic Maze: The robot navigated a moving maze by pressing its forearm against walls, using tactile feedback to stay centered while the maze shifted.

In all cases, TACTIC maintained contact forces within safe limits (< 5 N deviation) and completed tasks without manual retuning, whereas the model‑free baseline required frequent resets due to unstable force spikes.

Key Takeaways from the Experiments

• Robustness to Distribution Shift: The hybrid model generalized to unseen contact configurations better than pure learning approaches.
• Force Regulation: Contact‑aware sampling reduced peak forces by up to 40 % compared to baselines.
• Sample Efficiency: TACTIC needed fewer MPC samples to converge, cutting computational load by roughly 30 %.

Why This Matters for AI Systems and Agents

For AI practitioners building embodied agents, TACTIC offers a blueprint for integrating perception, dynamics, and control in a single, data‑efficient loop. The implications include:

• Safer Human‑Robot Collaboration: By continuously predicting and moderating contact forces, robots can operate alongside people without risking sudden pushes or squeezes.
• Reduced Engineering Overhead: The contact‑centric cost function abstracts away hand‑crafted force thresholds, letting developers focus on high‑level task definitions.
• Scalable Simulation‑to‑Real Transfer: The hybrid model’s reliance on physics‑based Jacobians mitigates the “reality gap,” a common hurdle for model‑free RL agents.
• Composable Agent Architectures: TACTIC’s modular perception‑prediction‑planning pipeline can be plugged into larger multi‑agent systems, such as warehouse fleets that need coordinated whole‑arm handling of bulky items.

Organizations looking to embed advanced manipulation capabilities into their AI stack can leverage platforms like the UBOS platform overview to orchestrate sensor streams, run MPC loops, and expose the controller as a service. Moreover, the Workflow automation studio can schedule periodic re‑training of the latent dynamics model as new contact data is collected, ensuring continuous improvement.

What Comes Next

While TACTIC marks a significant step forward, several avenues remain open for research and productization:

• Scalability to Higher‑DoF Platforms: Extending the approach to humanoid robots with dozens of tactile patches will test the limits of the 2‑D proximity representation.
• Learning from Sparse Tactile Data: Investigating self‑supervised methods to infer full contact maps from a few sensor readings could reduce hardware costs.
• Integration with Language‑Conditioned Planning: Coupling TACTIC with large‑language‑model planners would enable natural‑language task specifications for whole‑arm manipulation.
• Real‑Time Guarantees: Formal verification of the MPC safety envelope could satisfy regulatory requirements for medical or assistive robotics.

Businesses aiming to prototype such capabilities can explore the Enterprise AI platform by UBOS, which offers pre‑built connectors for vision and tactile hardware, as well as scalable compute for MPC inference. For startups, the UBOS for startups program provides sandbox environments to experiment with contact‑centric control without large upfront investment.

In summary, TACTIC demonstrates that a thoughtful blend of learned dynamics, analytical kinematics, and contact‑aware planning can unlock reliable whole‑arm manipulation. As tactile sensor technology matures and AI platforms mature, we can expect a new generation of robots that treat the world as a continuous field of forces rather than a collection of discrete objects.

References

For a complete technical description, see the original arXiv paper.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.