NeurIPS 2026

PEACH

Point Cloud Encoding for Accurate Context Handling

Point Cloud Sequence Encoding for Material-conditioned Graph Network Simulators

Philipp Dahlinger1 Balázs Gyenes1 Niklas Freymuth1 Luca Geminiani2 Tobias Würth3 Johannes Mitsch3 Nadja Klein2 Luise Kärger3 Gerhard Neumann1

1Autonomous Learning Robots 2Methods for Big Data 3Institute of Vehicle System Technology
Karlsruhe Institute of Technology (KIT), Karlsruhe

Deforming Block
Sheet Deformation
Trampoline

PEACH predictions (blue) with the ground truth mesh overlaid as a red wireframe. Materials and initial conditions are not seen during training.

TL;DR

A learned physics simulator usually needs to know the material it simulates. PEACH infers it instead from a few point cloud recordings of the process, as they come out of a depth camera. It needs no meshes, no parameter labels at test time and no optimization loop, and it transfers zero-shot from simulation to a real robot scene.

1–8
context point cloud sequences
plus one initial mesh at inference
0
real point clouds seen during training
zero-shot sim-to-real transfer
45%
lower real-world error than the Oracle
at maximal sheet stretching (t = 0.2 s)
~140×
faster than test-time optimization
0.58 s vs. 82 s to adapt and simulate
Overview of the PEACH framework: context encoding of point cloud sequences, mesh simulation, and auxiliary losses.
Overview of PEACH. Point cloud sequences from several simulated or real trajectories are encoded into a single latent material description by our spatio-temporal encoder. The aggregated latent vector conditions a mesh-based simulator, which predicts complete trajectories from an initial mesh. During training, auxiliary losses on the ground truth physical properties and a Signed Distance Field (SDF) improve accuracy. At inference, only the context point cloud sequences and an initial geometry are needed.

Abstract

Graph Network Simulators (GNSs) have emerged as powerful surrogates for complex physics-based simulation, offering inherent differentiability and orders-of-magnitude speedups over traditional solvers. However, GNSs typically assume access to the underlying material parameters, such as stiffness or viscosity, severely limiting their utility in realistic experimental settings. While recent meta-learning approaches address the parameter dependency by inferring properties from mesh trajectories, reconstructing a mesh from an observed scene is challenging. In this work, we introduce Point Cloud Encoding for Accurate Context Handling (PEACH), a novel framework that applies in-context learning on point clouds to adapt a learned simulator to unseen physical properties during inference. Our approach relies on a novel spatio-temporal point cloud sequence encoder, as well as two forms of auxiliary supervision to help improve simulation fidelity. We demonstrate that PEACH is capable of accurate zero-shot sim-to-real transfer on a challenging, dynamic scene. Experiments on simulation scenes show that PEACH even outperforms mesh-based baselines on prediction accuracy, while being much more practical for real-world deployment.

How PEACH works

Every task is one fixed set of physical properties, such as stiffness, damping or thickness. PEACH turns a handful of point cloud recordings of that task into a latent material description, which then conditions a mesh simulator.

  1. Point cloud sequences over time
    1

    Observe

    Record 1–8 point cloud sequences of the process, e.g. with a depth camera. Points carry no correspondences across time, so the motion of individual points is never observed directly.

  2. Encoder turning point cloud sequences into sequence embeddings
    2

    Encode in 4D space-time

    Each sequence becomes one point cloud in 4D space-time. Farthest point sampling and kNN form space-time patches, which a PointMAE tokenizer on Fourier features embeds. A single transformer layer and attention pooling produce one latent code per sequence.

  3. Aggregation of sequence embeddings into the latent material description
    3

    Aggregate

    A learned element-wise softmax aggregation merges any number of sequence codes into a single latent material description r. More context gives a better estimate.

  4. Graph network simulator predicting the trajectory from the canonical mesh and latent material description
    4

    Simulate

    The trajectory-level MaNGO simulator takes r and an initial mesh (e.g. from CAD) and predicts the entire trajectory in one forward pass. It interleaves message passing in space with 1D convolutions over time.

Why 4D space-time patches?

PEACH treats each point cloud sequence as one point cloud in space-time, with time scaled by τ. Drag the slider to see how the patches change.

patch center (farthest point sampling)
Loading…
Drag to rotate, scroll to zoom, right-drag to pan.
MLP predicting material properties from the latent material description, trained with an MSE loss

Aux. loss 1 Material regression

A small MLP predicts the ground truth physical parameters ρ from the latent description r. This grounds the latent space in physically meaningful quantities and doubles as an explicit system identification.

Model predicting the signed distance field from sequence embeddings and query points, trained with an MSE loss

Aux. loss 2 Signed distance field

Query points in space-time read the nearest patch tokens and predict a tanh-squashed signed distance to the mesh surface. This keeps fine geometric detail in the tokenizer. SDF labels come from geometry alone, so this loss also works when physical properties are unknown.

For sim-to-real transfer, the simulated training point clouds are augmented with multi-scale noise, structured region dropout and sporadic outlier points. No real-world data is used for training.

Simulation results

We evaluate on four simulation scenes, covering 2D and 3D deformables and elastic to viscoelastic materials. Test tasks have physical properties and initial conditions that are not seen during training.

The four simulation scenes: Deforming Block, Sheet Deformation, Bending Beam and Trampoline, with PEACH, PSTNet Encoder and No Context predictions on Trampoline.
The four simulation scenes. Qualitative predictions of PEACH and selected baselines are shown in blue, pink and gray, respectively, with ground truth meshes overlaid in red.

Compare methods side by side

All videos run in sync. The red wireframe is the ground truth mesh.

frame 0

Simulation accuracy

Full rollout MSE with 8 context trajectories (log scale, lower is better), mean over 4 seeds.

Best point cloud encoder, by a wide margin

PEACH beats the PSTNet and GNN point cloud encoders on every scene. These baselines differ from PEACH only in the encoder.

Point clouds instead of privileged meshes

PEACH is more accurate than mesh-context MaNGO on three of four scenes, although MaNGO sees ground truth mesh trajectories. On Bending Beam, PEACH even surpasses the Oracle, because it can learn an effective stiffness that compensates for simulator approximation errors.

No optimization loop

Test-time optimization via gradient descent through the simulator is strong on clean simulated data. It is ~140× slower, though (82 s vs. 0.58 s), and it degrades on noisy, occluded real-world observations.

0.11101001000 sPEACH (ours)0.58 sTest-Time Opt.82 sFEM (forward only)240 s

Generalizes out of distribution

On Deforming Block tasks with Poisson's ratios outside the training range, PEACH stays close to Oracle performance. No Context is over an order of magnitude worse.

Zero-shot sim-to-real

A robot drops balls of different size and weight onto latex sheets of different thickness, while a stereo camera records the scene. There are no meshes and no ground truth parameters, only point clouds. PEACH, trained purely in simulation, adapts from this context.

A Franka Panda releases a ball above an elastic membrane stretched across a 3D-printed frame, slowed down 5× (every camera frame is shown). ArUco markers on the corners calibrate the camera.
Qualitative real-world comparison at maximal sheet stretching between No Context and PEACH, with the observed point cloud in red.
At maximal stretching (t = 0.2 s), the context-free simulator regresses to the mean and barely deforms. PEACH follows the observed point cloud (red). The bottom of the sheet is occluded by the ball.
Balls
ØMassMaterial
40 mm269 gSteel, solid
40 mm69 gSteel, hollow
50 mm110 gSteel, hollow
60 mm162 gSteel, hollow
40 mm21 gPLA, 50% infill
Latex sheets
ColorThickness
Black0.381 mm
Green0.254 mm
Yellow0.152 mm

Error over time on real recordings

Mean point-to-mesh distance between the predicted mesh and the observed point cloud, over all real-world trials, with the band from the paper. Click a legend entry to hide or show it.

At maximal stretching (t = 0.2 s), PEACH's error is 45% lower than the Oracle's (measured and literature material parameters) and 59% lower than No Context's. All methods predict the free fall well, but the baselines deteriorate once the ball hits the sheet.

Explore in 3D: predicted mesh vs. observed point cloud

Drag to orbit, scroll to zoom and right-drag to pan. Both views share one camera and one timeline.

Preview with placeholder geometry. The real recordings and PEACH predictions will replace this soon.
PEACH (ours)
Loading…
No Context
Loading…
t = 0.000 s

What does the latent space learn?

PCA projection of PEACH latent codes

Trampoline: color is the loading ratio (mass over Young's modulus × thickness), which governs how far the sheet deforms. Marker size is the relaxation time.

We project the latent material codes of all test tasks onto their first two principal components. The latent space has no imposed structure, yet physical properties emerge in it. On Trampoline, the first component separates tasks by relaxation time and the second by loading ratio.

Two statistical checks back this up:

  • Distances mean something. Pairwise task distances in the latent space follow the distances in physical parameters (plots below; all Mantel tests give p < 10−3).
  • More context, less uncertainty. The variance of the latent code shrinks with context size at nearly the rate of averaging independent observations (βr ≈ 1).

Latent distances follow physical distances

Every pair of test tasks is one sample: distance between their physical parameters ρ vs. distance between their latent descriptions r. Darker hexagons hold more task pairs.

BibTeX

@inproceedings{dahlinger2026peach,
  title     = {Point Cloud Sequence Encoding for Material-conditioned Graph Network Simulators},
  author    = {Dahlinger, Philipp and Gyenes, Bal{\'a}zs and Freymuth, Niklas and Geminiani, Luca and W{\"u}rth, Tobias and Mitsch, Johannes and Klein, Nadja and K{\"a}rger, Luise and Neumann, Gerhard},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}