Resume Training

[KyleBoyer]

Based on the code, it seems the main method in the captcha_train.py deletes recursively the training data. For neural networks that take longer to train, it is useful to be able to fine tune the network along the way. Unless I am interpreting the code wrong?

[jor3l]

When running the captcha_train it ends or you just running for as long as you want? I'm new to this and wondering if it actually stops after a while. Thanks

Edit: looking at the code it seems to stop when should_stop says or I'm wrong?

[KyleBoyer]

You run it for as long as you want. I encourage you to run the training in the background using nohup and disown. That way you can evaluate the network to see how accurate it is, before you cancel the training. This bug was about canceling training, and then restarting where left off. I manually fixed it for my use case.

…

Sent from my iPhone X

On Feb 8, 2018, at 10:18 AM, Jose Vera ***@***.***> wrote: When running the captcha_train it ends or you just running for as long as you want? I'm new to this and wondering if it actually stops after a while. Thanks — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[jor3l]

I see now, I'll try with 2 million captchas and see if the accuracy is good enough. With 50k it wast about 10%. Thanks for the info

[KyleBoyer]

How long did you train for? I trained with 50k for about 16hrs and got high 90% accuracy. I also am training on CPU only with an i7-6700K. However my captchas were only 4 numbers.

…

Sent from my iPhone X

On Feb 8, 2018, at 12:44 PM, Jose Vera ***@***.***> wrote: I see now, I'll try with 2 million captchas and see if the accuracy is good enough. With 50k it wast about 10%. Thanks for the info — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[jor3l]

For about 2 hours and I just started generating the 2 million captchas to train on DigitalOcean. Will leave it working for a week or so, hopefully I get a good accuracy.

[Ashesh3]

@KyleBoyer hey! i am facing the same issue, after restarting captcha_train.py, it restarts from step 0 deleting all previous data, I'd be great if you could tell how you fixed it for your case!

Thanks!

[LxKxC]

• My captche program runs on raspberry PI 3B+, It has been running for three days.....
  
  ....

[LxKxC]

Where to control training step?

[mohammadsaleh]

Hi,
I have this issue too.
After restarting captcha_train.py, it restarts from step 0 and deleting all previous data, How fix this problem?!