Add "Simple pread/pwrite" disk IO type

[HanabishiRecca]

I think everyone is aware of infamous LT20 memory-mapped files problems at this point. In case someone not up to the topic: arvidn/libtorrent#6667.
Despite it being on its way to a pread-disk-io implementation (arvidn/libtorrent#7013), I am sick of waiting and fighting with mmap problems.

Here is a simple way to fix it completely by cutting the problem roots:
LT20 currently falls back to pread io for small files (<640K). But that's actually a tunable mmap_file_size_cutoff, never used by qBittorrent. So the fix is very straightforward, we simply increase that value.

This patch implements a new disk IO type, which forces all files to use the pread/pwrite fallback.
Well, technically files <32 TiB in size, as this is the max int value we can put there. But that should be enough for everyone. ©

It mitigates all mmap related problems, such as:

1. Memory hog and OOM crashes. Such as
   v4.4.X.X x64 - Insane memory usage(Linux) #17650
   Memory leak with memory mapped file IO #19914
   Memory Leak, crash #20893
   Huge memory consumption #20925
   and the rest countless duplicates.

2. Generally subpar performance in comparison with LT12. Especially on network drives. Such as
   File checking speed for 4.4.0 is much slower than 4.3.9 #16043
   Network slows down after installing qBittorrent 4.6.0 (lt20 qt6) #19902
   Abnormally slow IO performance on CIFS network share #19407
   and many more.

I run with this change for a month and have not seen any negative effects. Only pure relief.
It is still a bit subpar to LT12, as it doesn't have the client cache. But OS cache still works and I don't see any decremental in that regard in comparison with mmap.

[glassez]

One of the questions now: do we want to make it a configurable option instead? Maybe someone does want mmap files for some reason. And it's generally good to have more options after all.

I believe that there are a huge number of users (including me) who are not affected by mmap-related problems at all because of their use cases. At the same time, I am not sure that disabling mmaping will not bring other problems in some use cases, because mmap IO subsystem was not originally designed to work in this mode. Therefore, it would be better to provide mmap_file_size_cutoff as an option.

[glassez]

Therefore, it would be better to provide mmap_file_size_cutoff as an option.

Or maybe provide a way of disabling memory mapped files via another "Disk IO type" which behind the scenes would be just mmap disk IO with disabled memory mapping.

[HanabishiRecca]

Or maybe provide a way of disabling memory mapped files via another "Disk IO type" which behind the scenes would be just mmap disk IO with disabled memory mapping.

I like that idea more. Switching Disk IO type also requires client restart, could work as additional safety measure.

Exposing the raw int field is not very useful imo. Because some arbitrary values don't really mean anything. And advising people with mmap problems to set it to something like 2147483647 would be weird.
So on/off behavior looks better.

[HanabishiRecca]

There is also another related option: disk_write_mode. Maybe worth setting to always_pwrite along the way.

[HanabishiRecca]

Done.
This option also could be repurposed in the future, when the actual pread-disk-io would be in production.

[glassez]

There is also another related option: disk_write_mode. Maybe worth setting to always_pwrite along the way.

Will it provide any additional benefit? Isn't setting mmap_file_size_cutoff to maximum value enough?

[HanabishiRecca]

Will it provide any additional benefit?

No, it is redundant and doesn't do anything in this case. It uses pwrite if either of conditions met.

Isn't setting mmap_file_size_cutoff to maximum value enough?

I see it as an additional safety net. But if you think it is excessive, I'll remove it.

[glassez]

I see it as an additional safety net.

Well, so be it.

[nagyimre1980]

Will 5.0.0 include it?

[glassez]

@HanabishiRecca
Thank you!

[USBhost]

Wasn't the fallback single-threaded?

[HanabishiRecca]

No, it uses as many threads as you set via Asynchronous I/O threads option.

[KyleSanderson]

@HanabishiRecca this doesn't seem to be in the 5.0.x branch - do you know if this will make it for 5.0.1?

[USBhost]

Isn't it the new default? Basically called default?

[HanabishiRecca]

this doesn't seem to be in the 5.0.x branch - do you know if this will make it for 5.0.1?

There is a milestone on the right. I'm not in the developers team, just a regular user, so it does not depend on me.

Isn't it the new default? Basically called default?

No. You need to enable it explicitly (Disk IO type => Simple pread/pwrite).

[USBhost]

scratch that I was wrong. It's not on 5.0.0

[HanabishiRecca]

It's not on 5.0.0

Of course not. This PR was only merged yesterday, after the 5.0.0 release.

I clearly described testing steps in that message arvidn/libtorrent#6667 (comment).

[Pentaphon]

Has anybody done any testing to confirm this works as intended?

[USBhost]

try this arvidn/libtorrent#6667 (comment)
@Pentaphon

[glassez]

@HanabishiRecca
Could you create PR to my v5.0 branch in order for it to be backported?

[HanabishiRecca]

Unfortunately GitHub does not allow forking the same repo twice.
Is there a problem with cherry-picking?

[glassez]

Unfortunately GitHub does not allow forking the same repo twice.

Why do you need it?

Is there a problem with cherry-picking?

There are conflicts regarding the WebUI, which I prefer not to get involved with solving.

[HanabishiRecca]

Ok, I did it manually.

[buroa]

This PR made my year. Thank you @HanabishiRecca

[mreilaender]

Thank you very much @HanabishiRecca! For me with memory mapped IO under Linux, QBitTorrent didn't release the memory and simply kept growing eventually forcing other processes to use SWAP.

Using Pread fixed the issue without any obvious side effects :)