Code

Brandon/flux model loading#6739

Code

Merged

brandonrising merged 113 commits intomaininvoke-ai/InvokeAI:mainfrom brandon/flux-model-loadinginvoke-ai/InvokeAI:brandon/flux-model-loadingCopy head branch name to clipboardAug 27, 2024

+3,717-72

Commits

Commits on Aug 26, 2024

• Bump diffusers version to include FLUX support.

  

  RyanJDick

  authored and

  brandonrising

  committed
  5256f26

• Update imports for compatibility with bumped diffusers version.

  

  RyanJDick

  authored and

  brandonrising

  committed
  562c2cc

• Update HF download logic to work for black-forest-labs/FLUX.1-schnell.

  

  RyanJDick

  authored and

  brandonrising

  committed
  b617631

• First draft of FluxTextToImageInvocation.

  

  RyanJDick

  authored and

  brandonrising

  committed
  6a068cc

• Add sentencepiece dependency for the T5 tokenizer.

  

  RyanJDick

  authored and

  brandonrising

  committed
  5149a3e

• Use the FluxPipeline.encode_prompt() api rather than trying to run the two text encoders separately.

  

  RyanJDick

  authored and

  brandonrising

  committed
  761e49f

• Got FLUX schnell working with 8-bit quantization. Still lots of rough edges to clean up.

  

  RyanJDick

  authored and

  brandonrising

  committed
  5e1b3e9

• Minor improvements to FLUX workflow.

  

  RyanJDick

  authored and

  brandonrising

  committed
  e71c7d0

• Make 8-bit quantization save/reload work for the FLUX transformer. Reload is still very slow with the current optimum.quanto implementation.

  

  RyanJDick

  authored and

  brandonrising

  committed
  f7753be

• Add support for 8-bit quantizatino of the FLUX T5XXL text encoder.

  

  RyanJDick

  authored and

  brandonrising

  committed
  28654ec

• Make float16 inference work with FLUX on 24GB GPU.

  

  RyanJDick

  authored and

  brandonrising

  committed
  40e9a4e

• WIP - experimentation

  

  RyanJDick

  authored and

  brandonrising

  committed
  df1ac07

• Make quantized loading fast.

  

  RyanJDick

  authored and

  brandonrising

  committed
  11279ab

• Make quantized loading fast for both T5XXL and FLUX transformer.

  

  RyanJDick

  authored and

  brandonrising

  committed
  bcb7f8e

• Split a FluxTextEncoderInvocation out from the FluxTextToImageInvocation. This has the advantage that we benfit from automatic caching when the prompt isn't changed.

  

  RyanJDick

  authored and

  brandonrising

  committed
  3f1fbc6

• wip

  

  RyanJDick

  authored and

  brandonrising

  committed
  06d35c3

• NF4 loading working... I think.

  

  RyanJDick

  authored and

  brandonrising

  committed
  b1cf2f5

• NF4 inference working

  

  RyanJDick

  authored and

  brandonrising

  committed
  31c8d76

• Clean up NF4 implementation.

  

  RyanJDick

  authored and

  brandonrising

  committed
  17f5952

• LLM.int8() quantization is working, but still some rough edges to solve.

  

  RyanJDick

  authored and

  brandonrising

  committed
  52ff3c7

• More improvements for LLM.int8() - not fully tested.

  

  RyanJDick

  authored and

  brandonrising

  committed
  307a130

• WIP on moving from diffusers to FLUX

  

  RyanJDick

  authored and

  brandonrising

  committed
  27f33bb

• Setup flux model loading in the UI

  brandonrising

  committed
  e157ff3

• Remove changes to v1 workflow

  brandonrising

  committed
  6779e03

• Manage quantization of models within the loader

  brandonrising

  committed
  4bb3438

• Run Ruff

  brandonrising

  committed
  6b0f5f4

• Run Ruff

  brandonrising

  committed
  3814cd7

• Some UI cleanup, regenerate schema

  brandonrising

  committed
  a6ad70e

• Add backend functions and classes for Flux implementation, Update the way flux encoders/tokenizers are loaded for prompt encoding, Update way flux vae is loaded

  brandonrising

  committed
  f3096a8

• Run ruff, setup initial text to image node

  brandonrising

  committed
  5fc6c28

• Add nf4 bnb quantized format

  brandonrising

  committed
  68d28db

• Remove unused param on _run_vae_decoding in flux text to image

  brandonrising

  committed
  f3ebbe1

• Working inference node with quantized bnb nf4 checkpoint

  brandonrising

  committed
  efab4a3

• Install sub directories with folders correctly, ensure consistent dtype of tensors in flux pipeline and vae

  brandonrising

  committed
  95a2d97

• Select dev/schnell based on state dict, use correct max seq len based on dev/schnell, and shift in inference, separate vae flux params into separate config

  brandonrising

  committed
  98151ce

• Fix FLUX output image clamping. And a few other minor fixes to make inference work with the full bfloat16 FLUX transformer model.

  

  RyanJDick

  authored and

  brandonrising

  committed
  5d7e154

• Add tqdm progress bar to FLUX denoising.

  

  RyanJDick

  authored and

  brandonrising

  committed
  870ecd3

• Fix support for 8b quantized t5 encoders, update exception messages in flux loaders

  brandonrising

  committed
  24829b9

• Fix styling/lint

  brandonrising

  committed
  f36c6d0

• Add t5 encoders and clip embeds to the model manager

  brandonrising

  committed
  bebc6d3

• Some cleanup of the tags and description of flux nodes

  brandonrising

  committed
  3f845d9

• exclude flux models from main model dropdown

  

  Mary Hipp

  authored and

  brandonrising

  committed
  8b3e386

• add default workflow for flux t2i

  

  Mary Hipp

  authored and

  brandonrising

  committed
  35c263a

• Rename t5Encoder -> t5_encoder.

  

  RyanJDick

  authored and

  brandonrising

  committed
  ec360ee

• Address minor review comments.

  

  RyanJDick

  authored and

  brandonrising

  committed
  c822c3d

• Update doc string for import_local_model and remove access_token since it's only usable for local file paths

  brandonrising

  committed
  19238ed

• Switch inheritance class of flux model loaders

  brandonrising

  committed
  ede26a7

• Various styling and exception type updates

  brandonrising

  committed
  f0408bb

• More flux loader cleanup

  brandonrising

  committed
  9e888b1

• Remove duplicate log_time(...) function.

  

  RyanJDick

  authored and

  brandonrising

  committed
  6afb113

• Add docs to the requantize(...) function explaining why it was copied from optimum-quanto.

  

  RyanJDick

  authored and

  brandonrising

  committed
  519bf71

• Move requantize.py to the quatnization/ dir.

  

  RyanJDick

  authored and

  brandonrising

  committed
  c549a49

• update flux_model_loader node to take a T5 encoder from node field instead of hardcoded list, assume all models have been downloaded

  

  maryhipp

  authored and

  brandonrising

  committed
  41fb09b

• add case for clip embed models in probe

  

  maryhipp

  authored and

  brandonrising

  committed
  c66ccad

• add FLUX schnell starter models and submodels as dependenices or adhoc download options

  

  maryhipp

  authored and

  brandonrising

  committed
  9020a8a

• fix(ui): only exclude flux main models from linear UI dropdown, not model manager list

  

  Mary Hipp

  authored and

  brandonrising

  committed
  7264920

• fix(ui): pass base/type when installing models, add flux formats to MM badges

  

  Mary Hipp

  authored and

  brandonrising

  committed
  3fe9582

• feat(ui): create new field for t5 encoder models in nodes

  

  Mary Hipp

  authored and

  brandonrising

  committed
  24831d4

• tsc and lint fix

  

  Mary Hipp

  authored and

  brandonrising

  committed
  6899762

• fix schema

  

  Mary Hipp

  authored and

  brandonrising

  committed
  f67c4da

• update default workflow

  

  maryhipp

  authored and

  brandonrising

  committed
  a04d479

• fix(worker) fix T5 type

  

  maryhipp

  authored and

  brandonrising

  committed
  192eda7

• add better workflow description

  

  maryhipp

  authored and

  brandonrising

  committed
  3c861fd

• add better workflow name

  

  maryhipp

  authored and

  brandonrising

  committed
  dcfdc00

• Fix bug in InvokeInt8Params that was causing it to use double the necessary VRAM.

  

  RyanJDick

  authored and

  brandonrising

  committed
  f51dd36

• Update load_flux_model_bnb_llm_int8.py to work with a single-file FLUX transformer checkpoint.

  

  RyanJDick

  authored and

  brandonrising

  committed
  3c9811f

• Add docs to the quantization scripts.

  

  RyanJDick

  authored and

  brandonrising

  committed
  9982bc2

• Fix max_seq_len field description.

  

  RyanJDick

  authored and

  brandonrising

  committed
  c5c60f5

• Remove automatic install of models during flux model loader, remove no longer used import function on context

  brandonrising

  committed
  5f3e325

• Run ruff

  brandonrising

  committed
  bc6e1ba

• Undo changes to the v2 dir of frontend types

  brandonrising

  committed
  407796c

• added FLUX dev to starter models

  

  maryhipp

  authored and

  brandonrising

  committed
  d4ec434

• Don't install bitsandbytes on macOS

  brandonrising

  committed
  374dc82

• Attribute black-forest-labs/flux for much of the flux code

  brandonrising

  committed
  80a46d2

• Mark FLUX nodes as prototypes.

  

  RyanJDick

  authored and

  brandonrising

  committed
  5406a2f

• Make FLUX get_noise(...) consistent across devices/dtypes.

  

  RyanJDick

  authored and

  brandonrising

  committed
  c9c4e47

• Tidy is_schnell detection logic.

  

  RyanJDick

  authored and

  brandonrising

  committed
  22a3b3d

• Add comment about incorrect T5 Tokenizer size calculation.

  

  RyanJDick

  authored and

  brandonrising

  committed
  5307a6f

• Rename field positive_prompt -> prompt.

  

  RyanJDick

  authored and

  brandonrising

  committed
  5e9ef4b

• Move prepare_latent_image_patches(...) to sampling.py with all of the related FLUX inference code.

  

  RyanJDick

  authored and

  brandonrising

  committed
  0e9f6f7

• Run FLUX VAE decoding in the user's preferred dtype rather than float32. Tested, and seems to work well at float16.

  

  RyanJDick

  authored and

  brandonrising

  committed
  b5c937e

• Update macos test vm to macOS-14

  brandonrising

  committed
  f34a923

• Load and unload clip/t5 encoders and run inference separately in text encoding

  brandonrising

  committed
  b8d4630

• Only import bnb quantize file if bitsandbytes is installed

  brandonrising

  committed
  a31c02b

• Switch flux to using its own conditioning field

  brandonrising

  committed
  9899e42

• Add script for quantizing a T5 model.

  

  RyanJDick

  authored and

  brandonrising

  committed
  cfcd860

• Fixes to the T5XXL quantization script.

  

  RyanJDick

  authored and

  brandonrising

  committed
  4089ff2

• Update the T5 8-bit quantized starter model to use the BnB LLM.int8() variant.

  

  RyanJDick

  authored and

  brandonrising

  committed
  54c48c3

• Remove all references to optimum-quanto and downgrade diffusers.

  

  RyanJDick

  authored and

  brandonrising

  committed
  5af214b

• Update docs for T5 quantization script.

  

  RyanJDick

  authored and

  brandonrising

  committed
  f4612a9

• Move quantization scripts to a scripts/ subdir.

  

  RyanJDick

  authored and

  brandonrising

  committed
  18c0ec3

• Downgrade revert torch version after removing optimum-qanto, and other minor version-related fixes.

  

  RyanJDick

  authored and

  brandonrising

  committed
  098db5c

• Update t5 encoder formats to accurately reflect the quantization strategy and data type

  brandonrising

  committed
  dbdd851

• Switch the CLIP-L start model to use our hosted version - which is much smaller.

  

  RyanJDick

  authored and

  brandonrising

  committed
  1d6c83b

• Replace swish() with torch.nn.functional.silu(h). They are functionally equivalent, but in my test VAE deconding was ~8% faster after the change.

  

  RyanJDick

  authored and

  brandonrising

  committed
  1413ff9

• Setup scaffolding for in progress images and add ability to cancel the flux node

  brandonrising

  committed
  bd1b37d

• Remove dependency on flux config files

  brandonrising

  committed
  d159fe6

• ruff

  

  RyanJDick

  authored and

  brandonrising

  committed
  877b88e

• Remove flux repo dependency

  

  RyanJDick

  authored and

  brandonrising

  committed
  ae94e48

• Downgrade accelerate and huggingface-hub deps to original versions.

  

  RyanJDick

  authored and

  brandonrising

  committed
  f046a38

• ruff format

  

  RyanJDick

  authored and

  brandonrising

  committed
  9f6f404

• Remove outdated TODO.

  

  RyanJDick

  authored and

  brandonrising

  committed
  9a530c7

• Only install starter models if not already installed

  brandonrising

  committed
  bb80697

• Remove in progress images until we're able to make the valuable

  brandonrising

  committed
  642a953

• Remove no longer used code in the flux denoise function

  brandonrising

  committed
  a90d098

• Fix type error in tsc

  brandonrising

  committed
  40a3fa5

• Run ruff

  brandonrising

  committed
  b9238b6

• Rename params for flux and flux vae, add comments explaining use of the config_path in model config

  brandonrising

  committed
  5a5ca10

• update default workflow for flux

  

  Mary Hipp

  authored and

  brandonrising

  committed
  bf59ab3

• remove prompt

  

  Mary Hipp

  authored and

  brandonrising

  committed
  bd2692b

• Run ruff

  brandonrising

  committed
  5d42e67

• default workflow: add steps to exposed fields, add more notes

  

  Mary Hipp

  authored and

  brandonrising

  committed
  c510234

• Update starter model size estimates.

  RyanJDick

  committed
  3b29bad