Testable packages on crates.io

[jsgf]

A big chunk of my workflow is importing crates from crates.io for consumption by a non-Cargo build system (Buck). As part of this flow, I'd love to be able to validate the imported crates by running their own test suite.

Unfortunately this isn't formally supported. As far as I can tell there are two things that can fail:

1. the tests may have test fixtures and other files which are not packaged and uploaded to crates.io - for good reason as they might be large
2. the tests may have circular dev dependencies with respect to other crates

I wonder if both these can be solved by splitting off the testable parts of the crate into a separate artifact which can be optionally downloaded for people wanting to run tests on them. For example, off the top of my head, cargo vendor --testable ....

Those issues aside, I suspect a very large number of crates on crates.io are actually completely testable as-is. It would be nice to have some metadata to indicate that they are stand-alone testable.

I've talked about this before (with @alexcrichton I think), but @nellshamrell suggested I file an actual issue. I'm not sure if there should be a corresponding Cargo issue as well, since there's clearly some overlap.

[est31]

This would also be very helpful for anyone who doesn't want to run tests, because it'd reduce the size of the .crate file and thus speed up the download phase of any cargo build. Firefox for example has to manually blacklist test files in their vendored rust crates.