StreamME: Simplify 3D Gaussian Avatar within Live Stream

We propose StreamME, a method focuses on ultra-fast 3D avatar reconstruction. The StreamME synchronously records and reconstructs a head avatar from live video streams without any pre-cached data, enabling seamless integration of the reconstructed appearance into downstream applications. This exceptionally fast training strategy, which we refer to as on-the-fly training, is central to our approach. Our method is built upon 3D Gaussian Splatting (3DGS), eliminating the reliance on MLPs in deformable 3DGS and relying solely on geometry, which significantly improves the adaptation speed to facial expression motion. To further ensure high efficiency in on-the-fly training, we introduced a simplification strategy based on primary points, which distributes the point clouds more sparsely across the facial surface, optimizing points number while maintaining rendering quality. Leveraging the on-the-fly training capabilities, our method protects the facial privacy and reduces communication bandwidth in VR system or online conference. Additionally, it can be directly applied to downstream appliaction such as animation, toonify, and relighting.

Figure 2. The overview the pipeline of StreamME. We list three components at here. i) The 3DGS Properties Warming-Up (Optional): we introduce two auxiliary learnable 3D Gaussian attribute texture and illumination, refining the UV vertex positions to improve facial geometry detail (e.g. here, we show the coarse displacement for the vertices around the hair). This step is optional, and the users may also opt to use the tracked head without displacement. ii) Anchor Duplication and Simplification: we freeze the Tex and SH attributes and introduce a binary learnable mask, initialized with all values set to 1, from the UV vertices sampled on the mesh. Natural face©Xuan Gao et al. (CC BY).

Figure 3. We employ the proposed Idx for anchor binding, elimination the motion irrelevant points and simplification the over-small size/opacity points to reduce computational overhead. (a) The anchor points are sampled from the mesh and multiple duplicated for detail representation. (b) The motion-aware Idx tends to remove points, as there are none motion gradients around the forehead in canonical space. (c) The learnable masks from Idx for deleting the small size and opacity points within training. Please zoom-in for details. Natural face©Xuan Gao et al. (CC BY).