PixelMan: Consistent Object Editing with Diffusion Models via Pixel Manipulation and Generation [AAAI-25]

📄Paper | 🌐Website | 📰Poster | 🪧Slides | 📹Video

The official implementation for "PixelMan: Consistent Object Editing with Diffusion Models via Pixel Manipulation and Generation", accepted to AAAI-25.

PixelMan: Consistent Object Editing with Diffusion Models via Pixel Manipulation and Generation
Liyao Jiang, Negar Hassanpour, Mohammad Salameh, Mohammadreza Samadi, Jiao He, Fengyu Sun, Di Niu
AAAI-25

Description

The official implementation for "PixelMan: Consistent Object Editing with Diffusion Models via Pixel Manipulation and Generation", accepted to AAAI-25.

• In this work, we propose PixelMan, an inversion-free and training-free method for achieving consistent object editing via Pixel Manipulation and generation.

• PixelMan maintains image consistency by directly creating a duplicate copy of the source object at target location in the pixel space, and we introduce an efficient sampling approach to iteratively harmonize the manipulated object into the target location and inpaint its original location.

• The key to ensuring image consistency is anchoring the output image to be generated to the pixel-manipulated image as well as introducing various consistency-preserving optimization techniques during inference.

• Moreover, we propose a leak-proof SA manipulation technique to enable cohesive inpainting by addressing the attention leakage issue which is a root cause of failed inpainting.

• If you like our project, please give us a ⭐ on Github!

📰 News

• Feb 27, 2025, 12:30pm-2:30pm: Poster presentation at AAAI-25 in Philadelphia
• Feb 6, 2025: The code is now available on GitHub
• Jan 30, 2025: The paper is now available on arXiv
• Dec 09, 2024: PixelMan is accepted to AAAI-25.

Table of Contents

• PixelMan: Consistent Object Editing with Diffusion Models via Pixel Manipulation and Generation [AAAI-25]
  • Description
  • 📰 News
  • Table of Contents
  • Dependencies
  • Models
  • How to Run Demo
  • How to Run Experiments (Comparisons and Ablation)
  • Datasets
    • COCOEE - from "https://github.com/Fantasy-Studio/Paint-by-Example".
    • ReS - from "https://github.com/Yikai-Wang/ReS".
  • Acknowledgements
  • Bibtex

Dependencies

• Python >= 3.9
• PyTorch >= 2.0.1
• CLIP

pip install -r requirements.txt

Models

• Diffusion Model (SDv1.5) will be automatically downloaded through diffusers
• Download auxiliary models using "src/demo/download.py"

python src/demo/download.py

How to Run Demo

We provide a demo on gradio.

python demo_pixelman.py

Then, go to http://0.0.0.0:7860 for the demo on object repositioning

How to Run Experiments (Comparisons and Ablation)

# Edit Images with each method on two datasets
sh scripts/run_experiments.sh

# Evaluate the metrics on edited images
sh scripts/run_metrics.sh

# Note: Evaluation metrics output are saved under "outputs/metrics/<dataset>/sd1p5_<steps>/" folder, named as "aggregated_<method_name>.json" and "individual_<method_name>.json"

Datasets

COCOEE - from "https://github.com/Fantasy-Studio/Paint-by-Example".

• We include the images, labeled masks, and moving diff vectors under "datasets/COCOEE/"

ReS - from "https://github.com/Yikai-Wang/ReS".

• We include the images, masks, and moving diff vectors under "datasets/ReS/"

Acknowledgements

• This codebase is based on the implementation in "https://github.com/MC-E/DragonDiffusion" for Dragondiffusion and DiffEditor.
• The Self-Guidance baseline method implementation follows "https://colab.research.google.com/drive/1SEM1R9mI9cF-aFpqg3NqHP8gN8irHuJi?usp=sharing".

Bibtex

If you find our method and paper useful, we kindly ask that you cite our paper:

@inproceedings{jiang2025pixelman,
    title = {PixelMan: Consistent Object Editing with Diffusion Models via Pixel Manipulation and Generation},
    author = {Liyao Jiang and Negar Hassanpour and Mohammad Salameh and Mohammadreza Samadi and Jiao He and Fengyu Sun and Di Niu},
    booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
    year={2025}
}