One-dimensional Adapter to Rule Them All: Concepts, Diffusion Models and Erasing Applications

Mengyao Lyu1,2*, Yuhong Yang1,2*, Haiwen Hong3⚑, Hui Chen1,2†, Xuan Jin3, Yuan He3, Hui Xue3, Jungong Han1,2, Guiguang Ding1,2†
1Tsinghua University, 2BNRist, 3Alibaba Group
*Equal contribution; Project lead; Corresponding authors.
SPM achieves concept elimination in a non-invasive, precise, customizable, and transferable manner.


The prevalent use of commercial and open-source diffusion models (DMs) for text-to-image generation prompts risk mitigation to prevent undesired behaviors. Existing concept erasing methods in academia are all based on full parameter or specification-based fine-tuning, from which we observe the following issues: 1) Generation alternation towards erosion: Parameter drift during target elimination causes alternations and potential deformations across all generations, even eroding other concepts at varying degrees, which is more evident with multi-concept erased; 2) Transfer inability & deployment inefficiency: Previous model-specific erasure impedes the flexible combination of concepts and the training-free transfer towards other models, resulting in linear cost growth as the deployment scenarios increase.

To achieve non-invasive, precise, customizable, and transferable elimination, we ground our erasing framework on one-dimensional adapters to erase multiple concepts from most DMs at once across versatile erasing applications. The concept-Semi-Permeable structure is injected as a Membrane (SPM) into any DM to learn targeted erasing, and meantime the alteration and erosion phenomenon is effectively mitigated via a novel Latent Anchoring fine-tuning strategy. Once obtained, SPMs can be flexibly combined and plug-and-play for other DMs without specific re-tuning, enabling timely and efficient adaptation to diverse scenarios. During generation, our Facilitated Transport mechanism dynamically regulates the permeability of each SPM to respond to different input prompts, further minimizing the impact on other concepts. Quantitative and qualitative results across ~40 concepts, 7 DMs and 4 erasing applications have demonstrated the superior erasing of SPM.

Limitations of Previous Methods

Teaser image.

Generation alternation is one of the issues that previous concept erasing methods are now facing. The generations of non-erasing-target concepts could differ from the original model, and this may even escalate into concept erosion when more and more concepts are editted from the model.

Inefficiency, on the other side, also limits the usage of previous erasing methods in industry and communities. Their full-finetuning or specification-based erasing strategies are non-transferable across models, and non-customizable for complex and evolving application scenarios.

In contrast, our SPM can precisely erase targeted concept from DMs while preserving non-targets, as a one-size-fits-all lightweight, customizable, and model-transferable solution.

Our Method

Framework image.

A one-dimentional adapter which we call Semi-Permeable Membrane (SPM) is employed in a general diffusion model. We design the Latent Anchoring during the finetuning process to efficiently alleviate the alternation and erosion phenomenon in a self-supervised manner. During the generation phase, we introduce Facilitated Transport mechanism to ensure that only threatening prompts amplify the permeability rate of corresponding SPMs, while the generation of safe prompts (e.g., Picasso style) remain unharmed.


General Concept Removal

Single concept removal image.

Samples of "graffiti of the {concept}" after erasing Snoopy. Our SPM exhibits sufficient elimination on the targeted concept Snoopy, while the impact on non-targets is negligible.

Single concept removal (transfer) image.

Training-free transfer results. Once obtained (e.g. from SD v1.4 in this case), SPMs can transfer to other specialized models without re-tuning, and express both its target concept erasing and non-target preservation capabilities well.

Artistic Style Removal

Artistic style removal image.

Samples from DMs with artistic styles removed. SPMs can erase targeted styles (upper Van Gogh and lower Picasso) while preserving others, unlike prior works that show an evident trade-off between erasing and preservation.

Explicit Content Removal

Explicit content removal image.

NudeNet evaluation results on the I2P benchmark. The numbers on the left count the exposed body parts generated by the original SD v1.4. The binplots show the decrement with different methods applied for nudity mitigation, encompassing data-filtering (SD v2.0) and different erasing strategies. Compared to the prior works, SPM effectively eliminates explicit contents across different nude categories.

Explicit content removal cases.

Samples conditioned on the I2P prompts with the concept nudity erased. In each row, from left to right, we present the prompt from the I2P dataset, the generation outputs of SD v1.4, SD v1.4 + SPM, ChillOutMix, and ChillOutMix + SPM.

More Results

Further results, such as numerical comparison, multi-concept erasure, memorized image erasure, applications on other DMs (e.g. SD v2.x, SDXL) as well as more qualitative samples, can be found in the main text and the supplementary material of our paper.


        title={One-Dimensional Adapter to Rule Them All: Concepts, Diffusion Models and Erasing Applications}, 
        author={Mengyao Lyu and Yuhong Yang and Haiwen Hong and Hui Chen and Xuan Jin and Yuan He and Hui Xue and Jungong Han and Guiguang Ding},
        booktitle={2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},