Add max_threads_per_process and mp_context to pca by channel computation and PCA metrics

[alejoe91]

No description provided.

[zm711]

After tests pass I can test this in lab. PCA is so brutal on Windows! Hopefully setting this also gets us a speed boost too (although could just be a Windows problem).

[alejoe91]

Since I'm at it, I'll propagate the same trick to the PCA metrics.

For PCA computation, I'm getting 5-6x speed up on linux!

[alejoe91]

@zm711 let me know if this speeds things up on Windows! @jonahpearl can you also give it a try to see if it fixes the PCA hanging?

[zm711]

In a seminar this morning, I'll test this on an old dataset this afternoon :)

[zm711]

So testing now it seems like we don't get a speed up at all on Windows. What version of sklearn do you have? Basically no matter what I set for n_jobs I still use all processors at 100% activity. I tried varying the max_threads_per_process between 2-4 and also saw no difference in calculation time. So I think this is either an OS issue (Windows is just slow for this) or maybe a env issue.

ADDITION: since I never had the hanging issue I don't know if this fixes that though!

[alejoe91]

Mmm interesting! Have you tried max_threads to 1?

[zm711]

Mmm interesting! Have you tried max_threads to 1?

Nope, but I can try first thing tomorrow morning when I'm back at the Windows station.

[samuelgarcia]

OK for me.

[zm711]

Same speed for 1 thread. And still uses all cores to the max. I think this must be a Windows scheduler thing. It doesn't make sense to bounce processes around on different processors, but it must be happening.

[jonahpearl]

No luck for me but I do have an update which i'll put in the issue