arXiv:2611.18727v1 Announce Type: cross
Abstract: Instruction-based image editing models must condition on the source image, either by sequence concatenation, which appends source tokens to the target sequence (FLUX.1 Kontext), or by channel-wise concatenation, which stacks the source latent onto the noisy latent and widens the input projection (InstructPix2Pix, FLUX.1 Fill). The FLUX.1 Kontext report found channel-wise concatenation worse in initial experiments at 1K resolution with a 12B model and a large corpus. We explore channel-wise concatenation as the lower-cost option for a common production regime: structure-preserving edits that change appearance (sky, exposure, foliage, cleanup) but not layout, about 1,400 training pairs, a 4B non-distilled backbone, and 2K frames. It keeps the sequence at its original length, where sequence concatenation doubles it and roughly quadruples attention cost, and needs only single-scale classifier-free guidance (two forward passes per step instead of three). Starting from FLUX.2 Klein 4B base, we widen only the input projection and train both schemes with the same trainer, data, and hyperparameters. Channel-wise concatenation keeps the output pixel-aligned with the source; in our runs sequence concatenation, which must learn the correspondence between 10,880 target and 10,880 source tokens from fewer than 1,500 pairs, displaced structure, an observation we report without claiming a general ranking. On 29 held-out images the editor reaches 20.3 dB PSNR and 0.198 LPIPS against the human edit (unedited source: 15.0 dB), and counterfactual conditioning tests show it largely learned a global grade of the source rather than a selective edit, which we report as the open problem of this recipe. We also compare against an SD3.5-Medium editor trained with the same surgery and report the full training recipe, including failure modes. Code: https://github.com/complete-dope/LatentFusion
Abstract: Instruction-based image editing models must condition on the source image, either by sequence concatenation, which appends source tokens to the target sequence (FLUX.1 Kontext), or by channel-wise concatenation, which stacks the source latent onto the noisy latent and widens the input projection (InstructPix2Pix, FLUX.1 Fill). The FLUX.1 Kontext report found channel-wise concatenation worse in initial experiments at 1K resolution with a 12B model and a large corpus. We explore channel-wise concatenation as the lower-cost option for a common production regime: structure-preserving edits that change appearance (sky, exposure, foliage, cleanup) but not layout, about 1,400 training pairs, a 4B non-distilled backbone, and 2K frames. It keeps the sequence at its original length, where sequence concatenation doubles it and roughly quadruples attention cost, and needs only single-scale classifier-free guidance (two forward passes per step instead of three). Starting from FLUX.2 Klein 4B base, we widen only the input projection and train both schemes with the same trainer, data, and hyperparameters. Channel-wise concatenation keeps the output pixel-aligned with the source; in our runs sequence concatenation, which must learn the correspondence between 10,880 target and 10,880 source tokens from fewer than 1,500 pairs, displaced structure, an observation we report without claiming a general ranking. On 29 held-out images the editor reaches 20.3 dB PSNR and 0.198 LPIPS against the human edit (unedited source: 15.0 dB), and counterfactual conditioning tests show it largely learned a global grade of the source rather than a selective edit, which we report as the open problem of this recipe. We also compare against an SD3.5-Medium editor trained with the same surgery and report the full training recipe, including failure modes. Code: https://github.com/complete-dope/LatentFusion
This article has been indexed from cs.AI updates on arXiv.org
