Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

to #757, use non-blocking cursor to avoid too many query to src mongodb #843

Open
wants to merge 1 commit into
base: develop
Choose a base branch
from

Conversation

smiley-yoyo
Copy link

当oplog增量很少几乎没有时,mongo driver的cursor.Next()方法会不间断的查询源库,直至有新的oplog返回。在不间断的查询当中会消耗不少collector和源库的性能。

我测试的环境是两台aliyun的4核vps,平时空载状态下,源和目标的cpu占用就一直在30~40%。现象基本与问题#757的描述相同。

因此我将fetcher()方法内的cursor.Next()修改为cursor.TryNext(),并当无新的oplog返回时停顿1秒,借此解决oplog空闲时cpu过高的问题。

已经在我的环境测试通过,制造了10g左右的oplog,同步性能没影响(似乎cpu利用率变高了,原先9分钟同步完的,修改后6分钟左右就能同步完),而空闲时的cpu也降低到了1%以内。

但我毕竟不太了解fetcher中用cursor.Next()不间断查询是不是有特定需求,例如对数据及时性的要求等等。因此冒昧提出PR,请多多指教。

@CLAassistant
Copy link

CLAassistant commented Aug 17, 2023

CLA assistant check
All committers have signed the CLA.

@marknewmail
Copy link

the another way is block for a certain amount of time for new data before returning no data like this,
func (or *OplogReader) EnsureNetwork() (err error) {
...
findOptions := options.Find().SetBatchSize(int32(BatchSize)).
SetNoCursorTimeout(true).
SetCursorType(options.TailableAwait).
SetMaxAwaitTime(time.Millisecond * 1000)
...
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants