How to solve the problem of experiments stalling?

[YUjh0729]

Hello,

When I train the model, the experiment stops at a certain epoch and doesn't continue training. The GPU usage is at 1% and the memory usage is 12GB, indicating that the experiment is still running. However, it stays stuck at the current epoch for an entire night, preventing the experiment from progressing. What could be the problem? Can you help explain this?

Thank you.

[zcyrique]

Hi @YUjh0729 ,
I'm having the same issue as you! Were you able to solve it? Any help would be greatly appreciated.
@JunMa11, any help on this one?
Thank you!

[YUjh0729]

Hi @zcyrique ,
I've tried all the solutions from the issues, but none of them resolved the issue.

[zcyrique]

Thank you @YUjh0729 for reaching out, Let wait for anyone who may have solved this issue for help.