Lora + Controlnet

[fefespn]

can we train a controlnet with lora weights on on flux ? maybe combining the lora weights with the model then doing the controlnet ?

[dancasas]

I believe you can do that with the current repository as is. For example, imagine you have trained a LoRA that it is stored locally under loras/lora.safetensors, you can then combine it with the canny ControNet like this:

main.py \
 --prompt "cyberpank dining room, full hd, cinematic" \
 --image image_datasets/input_canny1.png --control_type canny \
 --repo_id XLabs-AI/flux-controlnet-canny-v3 \
 --name flux-canny-controlnet-v3.safetensors \
 --lora_local_path loras/lora.safetensors \
 --use_lora \
 --lora_weight 0.7 \
 --use_controlnet --model_type flux-dev \
 --width 1024 --height 1024  --timestep_to_start_cfg 1 \
 --num_steps 25 --true_gs 4 --guidance 4