Use sharded BAM output by default on Spark, and write utilities and design APIs to make it as easy to work with as single BAMs

[droazen]

No description provided.

[tomwhite]

I've sketched what this might look like at https://github.com/broadinstitute/gatk/compare/tw_sharded_bam_reader. I'd be happy for someone else to take this over.

A few notes about the status:

• Currently the Hadoop Path API is used. This should be changed to use the nio Path API (Add support for java.nio.file.Path to SamReaderFactory in htsjdk #1426)
• SamReader.PrimitiveSamReaderToSamReaderAdapter is not public so htsjdk will need a change to make this easier to integrate. Another possibility would be to build this into htsjdk itself.
• Many of the methods on SamReader assume that an index is present. To support these, there needs to be more work on how indexes are created and merged for sharded BAMs.
• Comprehensive tests are needed.

[droazen]

This one will not be done for alpha-1 -- moving milestones

[tomwhite]

I think the core of this work belongs in Hadoop-BAM, along with the code to merge sharded BAM files. I'll leave this open for the GATK integration.

Opened HadoopGenomics/Hadoop-BAM#110.