GelSlim 5.0: Unifying Vision and Touch
with a Single Fingertip Camera

Paper Code soon Data soon Video
GelSlim 5.0 switches between a proximity mode with UV off and a tactile mode with UV on. VisTacFusion combines the two observations to predict depth, surface normals, and planar object pose.
UV off: see through. UV on: feel contact. VisTacFusion fuses both.

Abstract

Vision and touch provide complementary cues for object pose and local contact geometry, but external cameras are often occluded at the contact site. We present GelSlim 5.0, a fingertip sensor that acquires visual and tactile observations with one fixed camera and a continuous UV-responsive contact layer. By switching illumination, the camera sequentially acquires through-membrane visual images and contact-sensitive tactile images from a shared viewpoint, avoiding inter-camera extrinsic calibration.

To combine these observations, we introduce VisTacFusion, a multimodal transformer with separate prediction paths for planar object pose and contact geometry. Asymmetric bottleneck fusion incorporates visual context while preserving tactile spatial features for depth and surface-normal estimation. Modality dropout enables inference with paired inputs or either modality alone.

On real validation data from 12 known objects, paired inputs reduce depth MSE, angular error, and translation L₁ by 61.2%, 72.3%, and 61.3%, respectively, relative to tactile-only inference with the same model. Relative to visual-only inference, the corresponding reductions are 93.7%, 6.1%, and 8.6%.

Vision and Touch Resolve Different Ambiguities

Vision
Touch

GelSlim 5.0 Sensor

Exploded view of GelSlim 5.0 with UV-responsive layer, transparent elastomer, lens, UV LEDs, sensor body, camera, and LED driver PCB; illumination states showing the layer opaque with UV on and transparent with UV off; prototype photographs in both modes with a 30 mm sensing area. Enlarge figure ↗
Hardware. (a) Assembly. (b) UV on: opaque; UV off: transparent. (c) Prototype in both modes.
Hex nut
Screw

UV illumination switches the observation modality; the camera and lens stay fixed.

Data

Real

Side view
Sensor view

Simulation

Top: rendered tactile contact. Bottom: rendered visual view.

VisTacFusion

VisTacFusion fuses visual and tactile features through 16 bottleneck tokens, then uses separate prediction paths:

  1. DPT decoder for contact depth and surface normals.
  2. MLP for 2D pose.

Modality dropout allows the same model to operate with paired or single-modality inputs.

VisTacFusion architecture: frozen SITR and MAE encoders, asymmetric bottleneck fusion, gated visual conditioning, a DPT geometry decoder, and a pose MLP. Enlarge figure ↗

Results

Paired-input improvement over:

Depth MSE

61.2% reduction

0.0170 → 0.0066 mm²

Angular error

72.3% reduction

2.350° → 0.652°

Translation L₁

61.3% reduction

0.0137 → 0.0053

Same model, different inputs. The pair is best on every metric.

Comparison with baselines

Input:
V-shape, corner, Y-junction, and rod contacts: simulated and real inputs, reference depth and normals, and visual-only, tactile-only, and paired predictions. Enlarge figure ↗
Real-sample predictions. Vision misses contact detail; touch recovers it; the pair is cleanest.

Live Real Predictions

Contact geometry from touch. Rows: predicted depth, predicted normals, tactile input.
Planar pose from vision.

Video