to #757, use non-blocking cursor to avoid too many query to src mongodb #843
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
当oplog增量很少几乎没有时,mongo driver的cursor.Next()方法会不间断的查询源库,直至有新的oplog返回。在不间断的查询当中会消耗不少collector和源库的性能。
我测试的环境是两台aliyun的4核vps,平时空载状态下,源和目标的cpu占用就一直在30~40%。现象基本与问题#757的描述相同。
因此我将fetcher()方法内的cursor.Next()修改为cursor.TryNext(),并当无新的oplog返回时停顿1秒,借此解决oplog空闲时cpu过高的问题。
已经在我的环境测试通过,制造了10g左右的oplog,同步性能没影响(似乎cpu利用率变高了,原先9分钟同步完的,修改后6分钟左右就能同步完),而空闲时的cpu也降低到了1%以内。
但我毕竟不太了解fetcher中用cursor.Next()不间断查询是不是有特定需求,例如对数据及时性的要求等等。因此冒昧提出PR,请多多指教。