实验定位 V3 架构交付 / 待训练验证
A80 是三版路线中的第 3 版，也是当前最可能带来较大提升的模型架构版本：Use VGGT multi-view observations to predict per-region/per-step guidance budgets inside TRELLIS.2 sparse and SLat denoising, instead of selecting or repairing meshes after generation.
它的关键变化是从 mesh 后处理、candidate selector、GPT 正交视图转换，转向 TRELLIS.2 sparse/SLat denoising 内部的区域预算控制。当前 claim level 是 architecture_proposal_grounded_by_replay_and_proxy_evidence，尚未声称 full TRELLIS.2 generation quality improvement。
A80 is an implementation-facing architecture version, not a completed training run. It must be validated by A80.2/A80.3 before being claimed as a method result.
实验设计（Image-2 风格）
模块设计（Image-2 风格）
架构名：VGGT-guided Regional Budget Sampler / 版本：A80-v0。输入是不定数量非正交视角；VGGT 提供几何 observation；Region Budget Head 输出 per-region λ、trust radius、lock mask、coverage priority 和 face-budget prior；TRELLIS.2 sparse/SLat sampler 在每个关键 denoising step 内执行 controlled update。
| module | input | output | purpose |
|---|---|---|---|
| VGGT Observation Encoder | N non-orthogonal RGB views, masks, estimated cameras, VGGT depth/world points/confidence | view-aligned observation tokens plus region summaries: conflict, visible recovery, unobserved closure room, confidence and depth-sign residual | Convert unordered non-orthogonal images into geometry-aware evidence without forcing GPT orthographic redraws. |
| Surface-region State Tokenizer | current sparse/SLat preview support, carrier/proxy surface samples, VGGT observation summaries | K region tokens with state logits over late-safe, visible-recovery, saturated-damage, defer, topology-risk | Expose the A79 state taxonomy as learnable tokens rather than hand-coded if/else rules. |
| Region Budget Head | region tokens, denoising timestep embedding, current support geometry statistics | per-region lambda, trust radius, lock mask probability, recall/coverage priority and face-budget prior | Replace A66 global damping and A67 fixed lambdas with continuous state-conditioned budgets. |
| TRELLIS.2 Sparse/SLat Sampler Controller | base denoising prediction, VGGT residual direction, predicted budgets | controlled pred_x0/noise update for sparse support and SLat features | Apply guidance before decoder topology is committed, where A10/A11 showed token-only control is insufficient but sampler-time intervention is feasible. |
| Topology and Face-budget Guard | preview decoded support stats, predicted face budget, topology-risk logits | rollback/dampen/continue decision and local budget clipping | Address TRELLIS.2's non-watertight/high-face pathologies before mesh extraction. |
实验结果（表格）
evidence basis
| claim | support | strength | limitation |
|---|---|---|---|
| Post-hoc candidate ranking is not enough. | A76 candidate-specific observation residual ranked 77 candidates but did not beat A75/A66. | strong within current artifacts | small number of object identities and synthetic replay candidates. |
| Sampler-time intervention is the right insertion point. | A10/A62 showed sampler trajectory can be controlled; A77 showed online replay states explain oracle choices better than post-hoc ranking. | moderate | A77 is replay/diagnostic, not a trained sampler. |
| State-balanced observation modeling is necessary. | A79 covers 5/5 required proxy states, but only 3 have full trajectory coverage. | moderate | defer/topology-failure states are proxy-only. |
| Global damping improves Chamfer but can damage recall. | A66 best no-GT policy has mean ΔChamfer=-0.0004090911015402626 and mean ΔF@5=-0.0006439499805592311. | strong for active 3-case replay | not yet trained across large variable-view sets. |
| A78 gives a positive but insufficient generalization signal. | A78 LOO joint=-0.0010409518764472024 vs A66 joint=-0.0009298794336918137, oracle match=2/3. | moderate | 3 held-out folds are too few for publication-level evidence. |
training targets
| target | source | loss | why |
|---|---|---|---|
| state classification | A79 proxy labels plus future full trajectories | state-balanced cross entropy | A78 failed when a state type was absent from the train split. |
| regional budget regression | A65/A66/A67/A73/A78 oracle and no-GT replay rows | Huber loss on lambda/trust radius with per-state reweighting | Continuous budgets have headroom over hard route, but scalar/global lambda is too weak. |
| multi-view observation consistency | rendered candidate views vs VGGT depth/silhouette/normal | robust depth residual + silhouette IoU loss + normal consistency | A58 showed MV consistency is useful as a recall signal but brittle as a hard post-hoc gate. |
| topology/face budget | A43/A71 topology metrics and face count audits | binary topology-risk loss + log-face-budget regularizer | TRELLIS.2 raw meshes can be non-watertight or extremely dense; later repair fixes topology but can hurt geometry. |
| GT geometry quality when available | ProObjaverse/Benchmark100 objects with held-out rendered views and GT mesh | Chamfer + F@2/F@5 surrogate + normal consistency | PSNR-only view metrics cannot capture water-tightness, face explosion or hidden geometry. |
minimal code hooks
| hook | change |
|---|---|
| sample_sparse_structure | after each denoising prediction, call budget_head(step, sparse_state, vggt_obs) and mix base/guided pred_x0 per region. |
| sample_slat | apply stronger visible-region lock and topology/face-budget clipping in late timesteps. |
| preview decoder | decode lightweight support/occupancy previews every M steps for topology-risk and support drift checks. |
| dataset loader | emit variable number of non-orthogonal views, cameras, masks, VGGT cache and state-balanced labels. |
expected improvements
| metric | expected direction | mechanism |
|---|---|---|
| watertightness | increase raw mesh watertight rate before repair | topology-risk logits and late-step SLat clipping reduce sheet/fragment creation. |
| face count | lower median face count without post-hoc decimation | face-budget prior penalizes over-fragmented support states during generation. |
| Chamfer | match or exceed A53/A66 on active cases | continuous region budget keeps beneficial depth-sign outward corrections while damping saturated damage. |
| F@5 / recall | avoid A66's recall loss | coverage-priority output explicitly protects visible-region recovery and unobserved closure states. |
| 20-view render consistency | reduce worst-view depth/silhouette error | per-view VGGT residual enters the sampler before final topology is fixed. |
可视化结果
实验结论
- A80 是目前最值得投入实现的架构：它直接针对 TRELLIS.2 raw mesh 的非水密、高面片数，以及 A66/A70 暴露的 Chamfer/F@5 tradeoff。
- 模型创新点不是“VGGT + TRELLIS.2”简单拼接，而是将 VGGT observation residual 变成 denoising-time per-region budget。
- 这条路线比 GPT 正交图预处理更有研究价值，因为它不强迫输入重绘，也不把多视角一致性问题外包给图像生成模型。
- 还不能说已经达到 A 类会议标准；必须完成 A80.2/A80.3 的 frozen-controller 和 sampler integration 实测。
下一步想法
inference flow
- Run VGGT once on all available non-orthogonal views to estimate camera, depth, world points and confidence.
- Initialize TRELLIS.2 multi-view generation with existing conditioning.
- At each sparse denoising step, project the current support preview into VGGT observation coordinates and form region tokens.
- Predict region budgets and apply controlled guidance to sparse pred_x0 with trust-region clipping.
- Repeat the same budget prediction for SLat denoising, with stronger visible locks and face-budget penalties near late timesteps.
- Decode mesh; run only light topology validation/simplification, not geometry-changing repair as the main method.
risks
- VGGT depth/camera errors can inject wrong signed residuals; the trust-region gate must be conservative under low confidence.
- Proxy labels from A79 are not sufficient for final training; at least several dozen full sampler trajectories are needed.
- Face-budget regularization can oversmooth thin structures if it is applied globally instead of region-conditionally.
- A80's current evidence is architecture-level and replay-backed; it is not yet a measured full-generation improvement.
implementation phases
- A80.1 offline dataset：state-balanced trajectory cache with VGGT observation tensors, region labels, per-step sparse/SLat summaries and GT/no-GT metrics.；退出标准：At least 20 cases per major state family or an explicit missing-state report.
- A80.2 frozen TRELLIS.2 controller：train only observation encoder projection and budget head; TRELLIS.2 weights frozen.；退出标准：improves A78/A66 replay joint and does not reduce watertightness on validation.
- A80.3 sampler integration：actual sparse/SLat denoising hooks with rollback and budget clipping.；退出标准：5-case and 100-case eval show raw watertightness/face count/Chamfer/F@5 gains.
- A80.4 paper ablation：remove VGGT encoder, remove state balance, remove topology guard, remove coverage loss.；退出标准：each ablation explains a failure mode seen in A1-A79.