Quota Update
Harbor provides the function of quota management, you can configure different quotas for each project to limit the occupation of space for each project, but in the scenario of high concurrent push to the same project, because the update is the same quota_usage record in the db, so it will cause a lot of optimistic locking, which will cause the db connection number spike and a lot of occupation.
In the v2.9, harbor introduces the quota update by redis to resolve the issue. The basic idea is to shift the implementation of optimistic locking mechanism from the database (Postgres) to utilizing the Watch mechanism in Redis. This change makes the updating of quota usage asynchronous, resulting in improved throughput and reduced database connections. However, the downside is that the displayed project quota usage in the UI may experience some delay and won't be updated in real-time. Nevertheless, the delay will be kept within a few minutes.
USAGE
Apply the environment variable QUOTA_UPDATE_PROVIDER to the following options, or not set for default behavior(update quota by DB).
- QUOTA_UPDATE_PROVIDER="redis"
- QUOTA_UPDATE_PROVIDER="db" (by default)
NOTICE
If you have multiple core instances, please make sure the configuration is consistent, both use DB to update quota or both use Redis to update, otherwise, it may result in inconsistent data, leading to incorrect quota usage.
Benchmark scenario is pushing a total of 5000 artifacts to the same project at 500 concurrence. (Harbor deployed by docker-compose in 16 CPU cores and 16G MEM VM)
script reference: https://github.com/goharbor/perf/blob/main/scripts/test/push-artifacts-to-same-project.js
update quota by DB
update quota by Redis
The average db connections was less than 25 after updating quota via Redis. This is a very important enhancement, as long periods of high db connection usage may cause other requests from harbor to become unavailable.
update quota by DB
update quota by Redis
Switching to Redis and seeing a several-fold increase in CPU usage for core is reasonable. This is because the throughput has increased, resulting in handling more requests per second, which in turn consumes more resources for token parsing. It does not incur any additional costs.
update quota by DB
update quota by Redis
The resources occupied by DB do not show significant differences, but the overall time consumption is reduced after using Redis.
update quota by DB
update quota by Redis
Obviously, after updating the quota with Redis, it will require more resources, which is also expected.
After using Redis, the response time of requests from different dimensions has shown significant improvement, ranging from several times to tens of times faster.
The throughput that can be processed per second has increased by more than 4 times after switching to Redis.