Repository navigation
Technical Details
How version 0.1.0 runs Idol inside Forge Neo. For everyday use, see Getting Started.
Forge Neo itself loads the checkpoint and the other files with its own loader, samples with its own samplers and schedules, and manages memory as it does for any model. The extension teaches it what Idol is through runtime hooks, which hand anything that is not Idol to the original Forge function untouched. No file of Forge changes. The structure follows the MiniMax H3 extension.
The model code is adapted from the Idol SDK, without its TensorDict and plyfile dependencies.
Idol is a rectified-flow transformer over 8,192 latent tokens of 16 channels and one camera token of 5, conditioned on the picture. The denoiser has 24 joint blocks and refiners of 2 blocks, with 1,024 channels.
| Step | What runs |
|---|---|
| Prepare the picture | BiRefNet removes the background unless the picture has real alpha; the alpha is eroded by 1 pixel; a square crop around the object, resized to 1024 × 1024, on black |
| Encode it | DINOv3 ViT-H+/16 gives 4,101 tokens of 1,280 channels; the FLUX.2 VAE gives 4,096 tokens of 128 channels, behind 5 empty ones |
| Sample | Forge's sampler on the packed latent |
| Decode | An octree decoder places one anchor per 32 Gaussians; a Gaussian decoder turns each anchor into 32 Gaussians |
Forge's sampler works on one tensor, so the object latent and the camera travel packed as [B, 1, 1, N], and the two image encoders' tokens as one context of 1,408 channels. The picture stands in for the prompt, and zeros for the negative prompt. Forge's prompt cache is cleared for every generation, as it knows nothing of the picture.
The script takes the original picture in before_process, before Forge flattens its alpha. Forge's own copy only feeds a placeholder latent. The seed for the VAE's sampling and the octree's random placement comes from process_batch, the one the sampler uses and the infotext shows: other extensions can still replace p.all_seeds in process.
Precision. As in the SDK, the transformer blocks run in fp16 and the embeddings and norms in fp32, whatever the file stores: every Forge layer casts its weights to the type of its input. The latent and the context stay in fp32 on the way in.
With identical weights on a reduced model on the CPU, the denoiser, the decoders and the PLY match the SDK exactly (difference 0.0). With the fp16 weights of the repack, the denoiser differs by about 0.0003. Forge's Simple schedule with shift 3 matches the SDK's time grid to within 1e-7 at 5, 8 and 50 steps. These compare the code, on small random models; the results with the real weights have been judged by eye.
Idol Turbo distills the denoiser into two phase LoRAs of rank 32 on 112 Linear layers, with the guidance folded in: slow for t >= 0.5, fast below. Forge's LoRA system applies one set of weights for a whole generation, so the extension adds them through forward hooks and picks the phase from the timestep at each step. The hooks come off when Turbo is unticked.
Version 3 of Niantic's format, the last with gzip: a 16-byte header, then positions as 24-bit fixed point with 12 fractional bits, alphas, colours, scales, and rotations as "smallest three" quaternions. Y and Z flip from the PLY's frame. Read back with PlayCanvas's splat-transform, the real axolotl moved by at most 0.00012 in position, 0.4% in visible colour and 0.2% in opacity. Colours outside the format's range saturate, 0.27% of the values in that object.
The extraction starts from ComfyUI's SplatToMesh: density and colour grids, Surface Nets, floater removal, optional Taubin smoothing. With the real objects, that alone gave about four times the faces, an inner wall joined to the outer one, a lumpy back, and flat faces that vanished: most Gaussians are flat and far thinner than a voxel, and fall between the grid's sample points. So the extension adds:
- A minimum thickness of 0.6 voxel for each Gaussian
- A solid field: a blur of the density, a low threshold of 0.07 of Otsu's level, a closing of 2 voxels, the holes filled, and a second blur, with Surface Nets at 0.5
- A margin around the grid, so that a solid reaching its edge still closes
- A blur of the colour grids, so a vertex settled between Gaussians still reaches them
The threshold was chosen by searching across four test objects. The result is one closed surface, wound outwards, in linear vertex colours with an unlit material; the Khronos glTF validator reports no errors or warnings. A few edges remain non-manifold, from Surface Nets' ambiguous cells.
CPU tests run without Forge, with stand-ins for the pieces of it the extension touches:
python -m pytest tests
They include the tensor names and shapes of the real release files, from their headers, against the model's layout. tools/turntable.py makes the GIFs of the Examples.
Idol for Forge Neo
Using it
Reference