just a note.

[PGTBoos]

not a bad idea at all,
Dont you think the tensorflow / pytorch community would like to improve it with your idea?
These days if you can compress video 3% more then it has a huge impact on data storage centers.
If you can train an AI faster, 10%,... boy that's huge... think of the impact on data centers, training time.
You should draw some attention from nvidea / intel.