Fix Flux multiple Lora loading bug

[maxs-kan]

What does this PR do?

The current approach of checking for a key with a base_layer suffix may lead to a bug when multiple Lora models are loaded. If the first loaded Lora model does not have weights for layer n, and the second one does, loading the second model will lead to an error since the transformer state dict currently does not have key n.base_layer.weight. So I explicitly check for the presence of a key with the base_layer suffix.

@yiyixuxu

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[hlky]

Hi @maxs-kan, thanks for your contribution, can you share some example lora checkpoints that may lead to a bug?

[maxs-kan]

Sure, try in the same order:
pipe.load_lora_weights(hf_hub_download("TTPlanet/Migration_Lora_flux","Migration_Lora_cloth.safetensors"), adapter_name="cloth")
pipe.load_lora_weights("alimama-creative/FLUX.1-Turbo-Alpha", adapter_name="turbo")