Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
download: fix logging level of workers on Windows
A gotcha from my Unix-only design. multiprocessing is simply awful on Windows -- expensive process creation, no signals (only crappy signal emulation; I miss SIGTSTP in particular), and TIL, no fork. So logger level set inside a function simply isn't inherited by worker processes (because it's not a side effect of import). Were I to redo the whole thing with Windows in mind, I might choose threads over processes -- but then we face the problem of no reliable termination mechanism, and a download thread could hang indefinitely* (unless we dig low enough to be able to inject synchronization mechanisms into network code, or find a library that's flexible enough to do so -- certainly not requests or urllib3) unless we force a dirty exit with unjoined threads. Trade-offs, trade-offs. * Make no mistake: iter_content could hang, too. Actually, maybe my iter_content_with_timeout https://gist.github.com/zmwangx/0dcbdbe6f67f3540dd73e777ef1b2a89 could solve the problem within requests, but god knows whether eventlet works on Windows, or whether it works but has other gotchas.
- Loading branch information