Time-Consistent Surface Mapping

Interactive demos of the paper

Time Consistent Surface Mapping for Deformable Object Shape Control

Shape control deforms an object into a desired shape with robots. Its reference is hard to define for 3D objects without visual texture or distinctive features. The paper defines it with a surface map between the perceived mesh of the object and the target mesh, computed with functional maps: every point of the object is compared with its image on the target, using all of the object's geometry.

A map computed anew at every frame is not a usable reference: each frame brings a new mesh, with another number of points in another order, and the map can jump between solutions of similar cost. The method keeps the eigenbasis, the tracked points, and the functional map consistent from one frame to the next, copes with targets that are not isometric to the current shape, and runs above 5 Hz. A shape controller then learns the response of the object to each gripper from the tracked points.

These pages compute the method in the browser: on meshes perceived by a simulated depth camera, on objects deformed with ARAP as in the simulations of the paper.

\C(I)=(\PhiM_I)^\T\A\,\Pi\,\PhiN_J \Csgn(k), \X(k) \rho=\sqrt{A(\Mc)/A(\Nc)} \uv_g=-\alpha\Lh_g^+\e
Functional maps of increasing size at k = 1, from C(I_initial) to C(I_max); at the next frame the refinement starts from the top-left I_update block of the previous C(I_max). On the right, the reference shape N and the current shape M(k) with the surface map as colors.
Fig. 4 of the paper. At the first frame, the functional map grows from \C(\Ii) to \C(\Imax); at every next frame, the refinement starts from the top-left \Iu\times\Iu block of the previous map, so that the map follows the deformation.

Interactive pages

in the order of the paper

Video

the method, the simulations, and the experiments of the paper

The video presents the shape control framework (Fig. 1): a 3D sensor provides the mesh of the object at every iteration, the time-consistent surface map compares it with the target mesh, and the control law defines a 6 DoF action for each robot.

It shows the simulations with ARAP of Fig. 8 and the five experiments of Sec. 4.3 with two ABB IRB120 robots and an Intel RealSense D415 camera: a Mexican hat, a T-shaped pool noodle, a pillow, a rectangular foam sheet, and a foam cut-out, with the surface maps of the method next to those of ZoomOut computed at every frame. Open on YouTube

Framework

Fig. 1 of the paper

Overview: an object deformed by two robots towards a target shape; the RGB-D camera provides the mesh at every iteration, which the time-consistent surface mapping compares with the target mesh to define the reference of the shape controller.
A 3D sensor provides the mesh of the object at every iteration; the time-consistent surface map compares it with the target mesh and defines the reference of the shape controller, which computes a 6 DoF action for each robot.

Results in the paper

click a figure to enlarge it

Code

the MATLAB code of the paper, a Python implementation, and the source of these demos

The repository of the paper contains the MATLAB code that reproduces the surface maps of the five experiments (timeconsistentsurfacemapping.m) from the recorded meshes, and compares them with ZoomOut in shape error, time, and mesh size. A Python implementation of the method, with a 3D viewer of the experiments, is in python/.

% MATLAB R2022b (Statistics, Computer Vision, Lidar, Optimization toolboxes)
timeconsistentsurfacemapping      % the five experiments of Sec. 4.3

python python/demo.py --experiment 5   # Python (see python/README.md)

The demos are static files (HTML, CSS, and JavaScript, with KaTeX and three.js included) in the folder docs of the same repository, next to the web version of the paper (docs/paper). The method, the simulator, and the camera (docs/js/sim) also run in Node.js, where tests/test_sim.mjs checks them.

python run.py                  # http://127.0.0.1:8765/
python tools/build_web.py      # after editing the sources in docs (needs Node.js)
node tests/test_sim.mjs        # checks of the simulation modules