Compactor OOM when run nexmark q19,q20 with minio storage

[huangjw806]

test setting:

KUBEBENCH_CONFIG="{'benchmark':{'risingwave':{'storage':{'type':'minio'}},'minio':{'enabled':true,'node_selector':'risingwave.cloud/nodegroup-name:ondemand-ng-test-8c16g-ssd-0','resources':{'cpu':{'limit':'8','request':'7'},'mem':{'limit':'13Gi','request':'13Gi'}},'persistence':{'enabled':false}}}}"
RW_META_STORE="postgresql"
RW_VERSION="nightly-20240507"

test pipeline:

https://buildkite.com/risingwave-test/nexmark-benchmark/builds/3648
https://buildkite.com/risingwave-test/nexmark-benchmark/builds/3639

compactor pod state:

State:          Running
  | Started:      Thu, 09 May 2024 08:54:08 +0000
  | Last State:     Terminated
  | Reason:       OOMKilled
  | Exit Code:    137
  | Started:      Thu, 09 May 2024 08:49:00 +0000
  | Finished:     Thu, 09 May 2024 08:49:04 +0000
  | Ready:          True
  | Restart Count:  7
  | Limits:
  | memory:  4Gi
  | Requests:
  | memory:   2Gi

• enhancement(compactor): record potential oom risks in compactor components #15946

[hzxa21]

Compactor memory usage is 2x larger in MinIO than in S3. Findings:

1. Object store throughput is higher in MinIO than in S3.
2. Object store rates are similar
3. Object store latency is also higher in MinIO than in S3.

1 and 2 make sense but 3 is surprising. We suspect that 3 is the reason for the OOM because when the slower the upload request, the more buffer will be accumulated in compactor because the uploads are concurrent.

MinIO run:

S3 run:

[hzxa21]

See #15946 (comment)