Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

high resolution #11

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

high resolution #11

wants to merge 5 commits into from

Conversation

ClashLuke
Copy link

@ClashLuke ClashLuke commented Jul 9, 2024

Quite a few things are happening here. We'll probably have to clean up the code, but the concepts are simple.

  1. Instead of using .repeat_interleave() and .expand(), this PR uses only .expand() greatly reducing memory usage (this change alone allows 3072x3072 on one 3090)
  2. A simple "size" argument for the generator
  3. Warmup / Cooldown for guidance factor instead of a pure constant factor
  4. CFG only when using FABRIC (perfect when using it only on the first half of the steps, which reduces runtime cost from (2+K)F to (2+K)F/2 + F while improving image quality noticeably)
  5. fused forward pass of positive samples and batch, removing the entire "hidden state caching" and CPU offload business, shaving off another 5% - 10% of runtime while halving memory usage in my one test

Finally, this PR removes negative images. As they are an intended feature, adding them back into the code might be required. (Not difficult, just tedious.)

@ClashLuke
Copy link
Author

ClashLuke commented Jul 9, 2024

(3) and (4) together allow the creation of 1024x1024 images as below

image

compared to the unguided baseline

image

and default fabric

image

@dvruette
Copy link
Contributor

dvruette commented Jul 9, 2024

Very cool, thank you! Already looking great, love the optimizations and high-res images.

As far as what's required to merge: Yes, indeed, negative feedback is kind of an integral feature so we should try and add back support for that. Also, if possible, we should avoid requiring the xformers dependency, especially when nowadays PyTorch 2.0 is pretty much the norm and most people have access to F.scaled_dot_product_attention.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants