Need fullsync despite 'fullsync_on_connect' set to true?

[mfredholm]

Hello,
I have installed 3 servers with Ubuntu 20.04 and the package https://files.tiot.jp/riak/kv/3.0/3.0.3/ubuntu/focal64/riak_3.0.3-OTP22.3_amd64.deb

It seems to work fine, the servers all have default configuration except I hade set nodename and listeners plus in advanced.config I did set 'cluster_mgr' IP and port.

The first to nodes (10.251.0.33 and 10.251.0.34) formed 'Staging-A' and I tested to add a key/value with:

$ curl -XPUT -H "Content-Type: text/plain" -d "world" http://10.251.0.33:8098/buckets/tbucket/keys/hello

I could then read it from both nodes fine;
(curl http://10.251.0.33:8098/buckets/tbucket/keys/hello)

I then set clusternames and then formed replication from Staging-A (10.251.0.33) to Staging-B (10.251.0.35) with

$ riak repl clustername Staging-A
$ riak repl connect 10.251.0.35:908
$ riak repl realtime enable Staging-B
$ riak repl realtime start Staging-B

However I could not find the key/value previously added:

$ curl http://10.251.0.35:8098/buckets/tbucket/keys/hello
not found

I tested adding an 'hello2' key which I could read fine though on all nodes so it replicated well.

Then I tried fullsync with:

$ riak repl fullsync enable Staging-B
$ riak repl fullsync start Staging-B

After that I could read the first key on the 'Staging-B' node 10.251.0.35.

Did I miss something or is this some bug?

regards
Mats

[martinsumner]

Real-time replication is triggered by a hook in the PUT process - if real-time replication is enabled then the hook will be enabled, and any PUT made will be queued for replication. Before realtime there is no hook enabled, and so any changes made prior to real-time replication will not be queued for replication. So you should not expect to find the key "hello" in Staging-B, if real-time replication was enabled after the key was PUT in Staging-A.

Triggering a full-sync will prompt a full comparison between Staging-A and Staging-B, and this will detect that the key "hello" is different, and resolve that difference by replicating from Staging-A to Staging-B.

Real-time enable - means replicate future PUTs; full sync enable - means compare the history of PUTs and repair any deltas. So this looks like a normal and correct sequence of events to me.

[mfredholm]

Ok, I understand, thanks!

[martinsumner]

Sorry, I missed the text in the title of the post about 'fullsync_on_connect' being set to true. Were you expecting the "hello" key to be replicated because the initial riak-repl connect 10.251.0.35:9080 should have triggered a fullsync?