Optimize memory usage of CXG conversion

[Bento007]

Motivation

Currently resources allocated to the CXG conversion step are not well utilized. Many VCPUs are allocated to the container to meet AWS memory/VCPU requirement, but very little of the actual process uses multiple CPUs. Reducing the memory needed will allow us to reduce the number of VCPUs, and support processing more cxg in parallel. This will save on compute costs by reducing the compute size. In the future we can rewrite the code to make better use of multiple CPUS to speed up CXG conversion.

Definition of Done

• The size of the dataset does not have a significant(20% diff) impact on the maximum memory used by cxg_conversion.

Tasks

• investigate lines of code with the comment # big memory usage.
• Make changes to optimize memory usage of those lines.
• Rerun the memory profiler and verify memory usage has decreased.
• modifying the tile_db_ctx parameters could help.

  single-cell-data-portal/backend/layers/processing/h5ad_data_file.py

  Line 33 in bd179bf

  tile_db_ctx_config = {

related issue

• chore: memory profile processing pipeline #7300
• Consolidate sparsity checks in CLI and CXG conversion steps. #7308

[nayib-jose-gloria]

Estimate (incl testing): 1-1.5 weeks

@Bento007 for comment if you agree