Abnormally slow IO performance on CIFS network share

[constantins2001]

Versions

libtorrent version (or branch): 2.0.9-r2
platform/architecture: ARM64

What is the problem?

IO on mounted CIFS (SMB) network share are abnormally low.

• Downloading to the network share directly results in ~5.4 MB/s speeds.
• Downloading to local SSD results in speeds in excess of 26 MB/s
• Automatically moving to network share via qBittorrent results in about 5 MB/s speeds.
• Moving the file manually from local SSD to network share via the mv command results in speeds of ~35.7 MB/s.

This indicates qBittorrent has drastically slower speeds when moving downloads/downloading to network shares compared to a manual move operation.

Steps to reproduce

1. Mount CIFS share
   mount.cifs -o user=<XXX>,pass=<XXX>,cache=loose,actimeo=120,fsc //<URL>/<share> <mountpoint>
2. Set qBittorent download/move location to <mountpoint>
3. Download torrent
4. Observe performance impact

Additional context

I'll be happy to provide a full testing environment, just contact me directly.

Related to qbittorrent/qBittorrent#19407

[nagyimre1980]

2.0px branch unusable, bad. don't bother with it! it will never be good

[master255]

@nagyimre1980 Why do you think that?

[master255]

@constantins2001 Do you have a comparison with version 1.2 of the library?

[stalkerok]

How about #6561 qbittorrent/qBittorrent#18983 qbittorrent/qBittorrent#16043 and many more performance issues that I didn't comment on, so it's hard for me to track them down. Is that not enough to consider this branch bad?

[master255]

@stalkerok Are these problems compared to the old version 1.2 of the library? Or are they general problems?

[stalkerok]

These are problems only 2.0.x

[master255]

@arvidn Too many changes at one time (1.2->2). Is it possible to roll back the change of disk write type? (memory mapped files)
This can increase disk write speed and improve the popularity of the library 2.0 version.

[master255]

@stalkerok You need to test 1.2 torrents and 2.0 torrents on Url-seeded torrent files. This will rule out problems with other torrent mechanisms. This is pure write speed. Use this ready-made torrent file.
511.65-notebook-win10-win11-64bit-international-dch-whql.zip

[constantins2001]

@constantins2001 Do you have a comparison with version 1.2 of the library?

qbittorrent/qBittorrent#19407 (comment)

[constantins2001]

Is it possible to roll back the change of disk write type? (memory mapped files)

I reverted back to version 2 due to the following issue with 1.2.19:
qbittorrent/qBittorrent#19407 (comment)

Better slowish write speeds than an unstable experience, but still I'd love to use the full potential of my platform...

[stalkerok]

https://user-images.githubusercontent.com/63958081/259389632-14476ece-989c-4394-b232-38e9da440111.png
RC_2_0 is NOT ABLE to download/upload 100MB/s from HDD.
Disk I/O type (POSIX-compliant) saves the situation a little, but these are far from the same results.
Also unable to match RC_1_2 is this new disk I/O #7013
So RC_1_2 remains the best.

[master255]

@stalkerok Problem with the upload or download speed?

[constantins2001]

• Automatically moving to network share via qBittorrent results in about 5 MB/s speeds.

same issue on 1.2.19, only downloads directly to the share are fast

[master255]

Looks like Arvin has abandoned library development. I don't know why. Maybe he has a creative crisis because of the wrong transition to version 2.+.
Or maybe he's just tired of torrents.

It must be hard to develop a program and not see the results of its work.

[paxter]

Using 2.x with NFS share. Upload/download rates above 100 MBit/s are no problem. I think it's working fine with the right OS and/or kernel version.

[constantins2001]

Using 2.x with NFS share

NFS and SMB seem to be very different in many regards

[constantins2001]

Update:
I moved to a non network share environment with 2x 8 TB HDDs (HGST HUH721008AL) in RAID 0 and now easily reach the limit of my network connection (1 Gbit/s).
The issues many others seem to experience with HDDs don't show for me.

[KyleSanderson]

Update: I moved to a non network share environment with 2x 8 TB HDDs (HGST HUH721008AL) in RAID 0 and now easily reach the limit of my network connection (1 Gbit/s). The issues many others seem to experience with HDDs don't show for me.

Yeah, it's a bit silly but that's expected. Remember each time you're doing this you're incurring a network roundtrip which can be multiple ms at the very least. Writing sequentially is also a very different from writing "random" blocks, especially with that latency involved. I saw you turned your dirty to nothing which was obviously a mistake on your part.

But yeah, mmap is never going to be as performant as the classic C linux api which 1.2 used. The future remains io_uring here for linux. The backends are interchangeable, so 2.0 can recover from this (and be baller if the networking is moved as well). Just takes time.

[constantins2001]

you're incurring a network roundtrip which can be multiple ms

Shouldn't matter, in combination with the write cache the 1 Gbit/s connection should easily be saturated (see OP).

Writing sequentially is also a very different from writing "random" blocks, especially with that latency involved.

Due to the same reasons as above I also don't think this is the issue. The move operation is just as slow as the download using random blocks (see OP).

I saw you turned your dirty to nothing which was obviously a mistake on your part.

Only on the old libtorrent, it crashed my system with the default settings.
I think once it filled the write cache the system switched to sync operations and in combination with the network share that somehow locked up the whole system. The config I posted above was the only way for me to prevent the system from locking up. I also observed that behavior with Transmission (which prompted my switch to qBittorrent in the first place).

[evulhotdog]

Using 2.x with NFS share. Upload/download rates above 100 MBit/s are no problem. I think it's working fine with the right OS and/or kernel version.

@paxter Can you share your configuration with us?

[arvidn]

libtorrent 2.0 defaults to using memory mapped files for disk I/O. This turned out to be sub-optimal in a lot of common cases (perhaps most cases).

My top priority is to implement a new disk I/O subsystem in master of libtorrent, which is multi-threaded, cached and uses pread/pwrite to be thread-safe (and windows equivalents). Once this is ready I will cut libtorrent 2.1 which includes this.

In the meantime, you may want to try to posix disk I/O subsystem available in libtorrent 2.0. It's not multi-threaded nor cached, so it will likely not perform optimally, but for a network share, it might be better than mmap.

[master255]

@arvidn It would be better to do DHT development (switch to tcp and broadcast traffic through other nodes (as in libp2p)). But if you'd rather work with the disk subsystem, then ok.

[arvidn]

It would be better to do DHT development

patches are welcome

[master255]

It would be better to do DHT development

patches are welcome

I'm developing more important programs. Higher level. And it would be easier for me to do it if you improved the DHT.

[master255]

It would be better to do DHT development

patches are welcome

Okay. I realize it's very complicated. So just help me on my tickets. I think that should be enough.

[paxter]

Using 2.x with NFS share. Upload/download rates above 100 MBit/s are no problem. I think it's working fine with the right OS and/or kernel version.

@paxter Can you share your configuration with us?

Which configuration? It's a default nfs share on bsd. Nothing special.

[stalkerok]

@arvidn, why can't the old disk I/O subsystem from 1.2 be used in 2.1, which works fine, if not by default, then at least as configurable parameters? You can't guarantee that the new disk subsystem will be better and there will be no problems with it.

[KyleSanderson]

@arvidn, why can't the old disk I/O subsystem from 1.2 be used in 2.1, which works fine, if not by default, then at least as configurable parameters? You can't guarantee that the new disk subsystem will be better and there will be no problems with it.

the p stands for position, which allows you to avoid the file pointer position and read and write to certain segments (as long as append isn't used). This is a general POSIX backend that was standardised in 2001 (not nearly as fast or efficient as io_uring or IoRing) - that should still be better if fully pipelined than the older one despite being 22 year-old technology. The risk here is the local cache, that's going to dump into the page cache, then finally spooled to disk.

People are having problems with the classic backend with multiple files not being committed to the page cache when cross-seeding (same data, different torrents). I'm unclear/concerned the local cache is going to still hamper these efforts, and the locking isn't going to be as tight as one would expect.

the real struggle for NVMe boxes right now (on 1.2) is the fact the network is single threaded. There's been custom TLS crypto and similar shoved in there that just kills the entire session if it's toggled on. io_uring "fixes" this as well by allowing you to do all that crypto for the next set of work between submissions (which could be threaded or not).

Anyway, it sounds like 2.1 is the next 1.2, and 2.0 will hopefully be forgotten in short order (it may even take calling it 3.0, but if there's a large problem when that's done it won't be possible to recover from this).

[Smexhy]

Versions

libtorrent version (or branch): 2.0.9-r2 platform/architecture: ARM64

What is the problem?

IO on mounted CIFS (SMB) network share are abnormally low.

• Downloading to the network share directly results in ~5.4 MB/s speeds.
• Downloading to local SSD results in speeds in excess of 26 MB/s
• Automatically moving to network share via qBittorrent results in about 5 MB/s speeds.
• Moving the file manually from local SSD to network share via the mv command results in speeds of ~35.7 MB/s.

This indicates qBittorrent has drastically slower speeds when moving downloads/downloading to network shares compared to a manual move operation.

Steps to reproduce

1. Mount CIFS share mount.cifs -o user=<XXX>,pass=<XXX>,cache=loose,actimeo=120,fsc //<URL>/<share> <mountpoint>
2. Set qBittorent download/move location to <mountpoint>
3. Download torrent
4. Observe performance impact

Additional context

I'll be happy to provide a full testing environment, just contact me directly.

Related to qbittorrent/qBittorrent#19407

This is driving me crazy as well. Literally tried everything, but the speeds are terrible. Wish I could just go around the NAS, but I can’t.

[the-hotmann]

I encounter the very same issue.

When I use qBittorrent via docker-compose and want it to access a SMB/CIFS share when everything is super super slow:

  torrent:
    image: lscr.io/linuxserver/qbittorrent:4.6.6
    container_name: torrent
    environment:
      - PUID=1000
      - PGID=33
    volumes:
      - "./torrent/config/:/config/:rw"
      - "./torrent/custom-services.d/:/custom-services.d/:ro"
      - "SMB_SHARE:/downloads/:rw"
    deploy:
      resources:
        limits:
          memory: 1G
    restart: unless-stopped

volumes:
  SMB_SHARE:
    name: "SMB_SHARE"
    driver: "local"
    driver_opts:
      type: "cifs"
      o: "seal,addr=###addr###,password=###pass###,rw,uid=1000,gid=33,file_mode=0644,dir_mode=0755"
      device: "//###device###/"

Did I do something wrong, or is it just qBittorrent not handeling SMB/CIFS Shares properly?
Any recommendations are welcome!

qbittorrent/qBittorrent#19407 (comment)

[...] this seems to be a quite niche use case.

I dont think it is a niche issue. Storage these days can be very cheap, or expensive. If you just buy/rent storage, it is cheap. If you buy a server with more ssd-raid storage it will be expensive, but you actually will never use the ssd-raid storage speed. So I think it would make a lot of sense to improve the handling/speed of SMB shares.

[the-hotmann]

• Downloading to the network share directly results in ~5.4 MB/s speeds.
  [...]
• Automatically moving to network share via qBittorrent results in about 5 MB/s speeds.

Yes, I see these 5MB/s limits aswell. When I move stuff on my own server I see about 500MB/s, but when to the SMB share it drops to 5MB/s.

Anyway, I can access the SMB share from my server with about 200-250MB/s. So the share is not the bottleneck!

[HanabishiRecca]

I am trying to solve this problem at qBittorrent side right now. qbittorrent/qBittorrent#21300
If someone wishes to test the patch and feed back with results, it would be great.

[the-hotmann]

How can one test this?
I use a dockerised version of this image: lscr.io/linuxserver/qbittorrent Will there be a test/bets release with this patch?

[onedr0p]

@HanabishiRecca I am using Simple pread/pwrite in qBittorrent 5.0.1. I can say memory usage is a lot better than before, like really better, so thanks a lot.

On topic with this version of QB and Simple pread/pwrite, it took 40 seconds to move a 10GB torrent file from local SSD (incomplete dir) to an NFS share (complete dir) (~285MB/s) which is roughly the speed of my local SSD.

[HanabishiRecca]

Cool. There are more not yet released optimization patches on libtorrent side.
If you are a Linux user, you probably want to give this build a try: #6667 (comment)