Run offline state sync in the background #3367

laevandus · 2024-08-09T11:38:48Z

🔗 Issue Links

Resolves stream-chat-swift/issues/2021

🎯 Goal

Allow regular API calls to be made while offline sync is running which gives quicker UI updates when UI update requires API call to finish.

📝 Summary

Add a sync v2 implementation controllable with a StreamRuntimeCheck._isSyncV2Enabled

🛠 Implementation

Adds a new version of the offline state sync which only pauses regular API requests when handling automatic retries. The sync is more lightweight and only syncs currently active channels (active = server state has been requested in this session aka calling synchronize). If sync is not possible (too many events), then data is fully refreshed for active channel lists (same as synchronize but not resetting to the first page).

Improvements

Does not block other API requests if sync takes a bit of time (in debug and up-to-date data it runs <0.5 seconds)
Less data exchanges as it only syncs active channels (for other channels we use controller synchronize or get methods anyway)

Notes

These changes do not touch the code (direct call to syncExistingChannelsEvents) which is triggered by the ChatRemoteNotificationHandler. This code path triggers the request.fetchLimit = 100 case which the regular sync actually does not. We should look this in a separate task.
I kept refreshing channel lists in V2 because UI-tests were relying on these (and probably some other hidden cases as well). Could be changed in next iterations.

Summary of the V2 mode

Recovery mode (pauses regular API requests while it is running)

Enter recovery
Runs offline API requests
Exit recovery

Background mode (other regular API requests are allowed to run at the same time)

Collect all the active channel ids (from instances of Chat, ChannelList, ChatChannelController, ChatChannelListController)
Ask updates from the /sync endpoint for these channels
2.a Apply changes
2.b When there are too many events, just refresh channel lists (channels for current pages in ChannelList, ChatChannelListController)

Comparison

Description	V1	V2
Channel list sync	All in the local storage	Only active channel lists
Resets channel list to the first page	Yes, always	No, refreshes currently loaded channels, UI does not just when data changes
Always starts watching channels	Yes	No
Deletes channels	Yes, when reseting channel list and channel not referenced by active instances	No
Blocks regular API requests	Yes	Only when retrying offline requests, otherwise no

🧪 Manual Testing Notes

Case 1: Channel list updates

Log in, go offline
Other user triggers channel list updates (sends new messages etc)
Come back online, channel list updates

Case 2: Sending messages while offline

Log in, go offline
Send some messages, edit some, add reactions etc
Come back online, messages are sent and edits are visible for other participants

Case 3: Not resetting channel list (#2021)

Log in
Scroll down in the channel list to load multiple pages
Go offline
Trigger updates using a different account (for verifying that channel list updates when coming back online)
Come back online. Channel list does not reset to the first page.

☑️ Contributor Checklist

I have signed the Stream CLA (required)
This change should be manually QAed
Changelog is updated with client-facing changes
Changelog is updated with new localization keys
New code is covered by unit tests
Comparison screenshots added for visual changes
Affected documentation updated (docusaurus, tutorial, CMS)

github-actions · 2024-08-09T11:42:24Z

	1 Warning
⚠️	Big PR

Generated by 🚫 Danger

Stream-SDK-Bot · 2024-08-09T11:47:03Z

SDK Size

`title`	`develop`	`branch`	`diff`	`status`
StreamChat	6.73MB	6.76MB	0.03MB	🟢
StreamChatUI	4.41MB	4.41MB	0.0MB	🟢

Stream-SDK-Bot · 2024-08-09T11:49:57Z

SDK Performance

`target`	`metric`	`benchmark`	`branch`	`performance`	`status`
MessageList	Hitches total duration	10 ms	5.01 ms	49.9% 🔼	🟢
	Duration	2.6 s	2.54 s	2.31% 🔼	🟢
	Hitch time ratio	4 ms per s	1.97 ms per s	50.75% 🔼	🟢
	Frame rate	75 fps	78.17 fps	4.23% 🔼	🟢
	Number of hitches	1	0.6	40.0% 🔼	🟢
ChannelList	Hitches total duration	12.5 ms	21.69 ms	-73.52% 🔽	🔴
	Duration	2.6 s	2.55 s	1.92% 🔼	🟢
	Hitch time ratio	5 ms per s	8.52 ms per s	-70.4% 🔽	🔴
	Frame rate	72 fps	74.35 fps	3.26% 🔼	🟢
	Number of hitches	1.2	1.8	-50.0% 🔽	🔴

Stream-SDK-Bot · 2024-08-09T12:36:59Z

SDK Size

`title`	`develop`	`branch`	`diff`	`status`
StreamChat	6.73MB	6.76MB	+35KB	🟢
StreamChatUI	4.41MB	4.41MB	0KB	🟢

nuno-vieira

LGTM! ✅ Nice that this is behind a flag so that we can minimize the risk on this one 👍

martinmitrevski

Looks good to me ✅ We should do solid testing here @testableapple

martinmitrevski · 2024-08-12T14:01:29Z

Sources/StreamChat/Config/StreamRuntimeCheck.swift

+    /// For *internal use* only
+    ///
+    /// Uses version 2 for offline state sync.
+    public static var _isSyncV2Enabled = true


We should also have a ticket to remove this in 2-3 releases and only support V2.

Added a ticket

martinmitrevski · 2024-08-12T14:30:45Z

Sources/StreamChat/Repositories/SyncRepository.swift

+        //
+        if config.isLocalStorageEnabled {
+            apiClient.enterRecoveryMode()
+            operations.append(ExecutePendingOfflineActions(offlineRequestsRepository: offlineRequestsRepository))


I assume there are no changes in this logic, right? We have some logic for sending new messages - after some interval they shouldn't be sent when you come online. Just making sure we still have that.

martinmitrevski · 2024-08-12T14:33:04Z

Sources/StreamChat/Repositories/SyncRepository.swift

+    /// 1. Collect all the **active** channel ids (from instances of `Chat`, `ChannelList`, `ChatChannelController`, `ChatChannelListController`)
+    /// 2. Apply updates from the /sync endpoint for these channels
+    /// 3. Refresh channel lists (channels for current pages in `ChannelList`, `ChatChannelListController`)
+    private func syncLocalStateV2(lastSyncAt: Date, completion: @escaping () -> Void) {


We also have some internal docs about the sync mechanism, that probably require revisiting.

It is still there, that logic is in OfflineRequestsRepository what did not change.

… requests while it runs

sonarqubecloud · 2024-08-14T06:35:09Z

Quality Gate passed

Issues
150 New issues
0 Accepted issues

Measures
0 Security Hotspots
84.4% Coverage on New Code
0.1% Duplication on New Code

See analysis details on SonarCloud

* main: (154 commits) Unblock release on workflow dispatch (GetStream#3389) [CI] Run release action on workflow dispatch (GetStream#3386) [CI] Do not unshallow branches on CI [CI] Do not unshallow branches on CI [CI] Update git config on merging develop and release branches [CI] Update git config on merging develop and release branches [CI] Pull origin release branch on release [CI] Pull origin release branch on release [CI] Fetch all branches on release (GetStream#3385) [CI] Fetch all branches on release (GetStream#3385) Bump 4.62.0 Store pending message along with regular messages (GetStream#3378) [CI] Set up a default base branch when automatically creating PRs (GetStream#3383) Fix reading the local database state just after the initial write (GetStream#3373) [CI] Add space to SDK size badge (GetStream#3379) Update channel type documentation to reflect allowed characters (GetStream#3377) Reuse CurrentChatUserController instead of recreating it when sending typing events (GetStream#3372) Run offline state sync in the background (GetStream#3367) [CI] Bump fastlane plugin version (GetStream#3376) [CI] Bump fastlane actions plugin (GetStream#3375) ...

laevandus added 🔧 WIP A PR that is Work in Progress 🌐 SDK: StreamChat (LLC) Tasks related to the StreamChat LLC SDK ✅ Feature An issue or PR related to a feature labels Aug 9, 2024

laevandus requested a review from a team as a code owner August 9, 2024 11:38

laevandus force-pushed the feature/background-offline-sync branch from 4e4b18b to 3ad3ed3 Compare August 9, 2024 11:39

laevandus added 🤞 Ready For QA A PR that is Ready for QA and removed 🔧 WIP A PR that is Work in Progress labels Aug 9, 2024

laevandus changed the title ~~[WIP] Run offline state sync in the background~~ Run offline state sync in the background Aug 9, 2024

laevandus force-pushed the feature/background-offline-sync branch from 8831244 to 4884f7a Compare August 12, 2024 06:16

nuno-vieira approved these changes Aug 12, 2024

View reviewed changes

laevandus force-pushed the feature/background-offline-sync branch from 4884f7a to f184d6e Compare August 12, 2024 10:47

martinmitrevski approved these changes Aug 12, 2024

View reviewed changes

testableapple added 🟢 QAed A PR that was QAed and removed 🤞 Ready For QA A PR that is Ready for QA labels Aug 13, 2024

testableapple approved these changes Aug 13, 2024

View reviewed changes

laevandus added 4 commits August 14, 2024 09:06

Run offline state sync in the background instead of pausing other API…

b0c9aa3

… requests while it runs

Add test for refreshing loaded channels in channel list

3add9c5

Fix typo

5f8ceb4

Always refresh channel lists

a201597

laevandus force-pushed the feature/background-offline-sync branch from c1d523f to a201597 Compare August 14, 2024 06:06

laevandus enabled auto-merge (squash) August 14, 2024 06:06

laevandus merged commit 6c19a98 into develop Aug 14, 2024
15 checks passed

laevandus deleted the feature/background-offline-sync branch August 14, 2024 06:56

Stream-SDK-Bot mentioned this pull request Aug 15, 2024

4.62.0 Release #3384

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run offline state sync in the background #3367

Run offline state sync in the background #3367

laevandus commented Aug 9, 2024 •

edited

Loading

github-actions bot commented Aug 9, 2024 •

edited

Loading

Stream-SDK-Bot commented Aug 9, 2024 •

edited

Loading

Stream-SDK-Bot commented Aug 9, 2024

Stream-SDK-Bot commented Aug 9, 2024 •

edited

Loading

nuno-vieira left a comment

martinmitrevski left a comment

martinmitrevski Aug 12, 2024

laevandus Aug 14, 2024

martinmitrevski Aug 12, 2024

martinmitrevski Aug 12, 2024

laevandus Aug 14, 2024

sonarqubecloud bot commented Aug 14, 2024

Run offline state sync in the background #3367

Run offline state sync in the background #3367

Conversation

laevandus commented Aug 9, 2024 • edited Loading

🔗 Issue Links

🎯 Goal

📝 Summary

🛠 Implementation

Improvements

Notes

Summary of the V2 mode

Comparison

🧪 Manual Testing Notes

Case 1: Channel list updates

Case 2: Sending messages while offline

Case 3: Not resetting channel list (#2021)

☑️ Contributor Checklist

github-actions bot commented Aug 9, 2024 • edited Loading

Stream-SDK-Bot commented Aug 9, 2024 • edited Loading

SDK Size

Stream-SDK-Bot commented Aug 9, 2024

SDK Performance

Stream-SDK-Bot commented Aug 9, 2024 • edited Loading

SDK Size

nuno-vieira left a comment

Choose a reason for hiding this comment

martinmitrevski left a comment

Choose a reason for hiding this comment

martinmitrevski Aug 12, 2024

Choose a reason for hiding this comment

laevandus Aug 14, 2024

Choose a reason for hiding this comment

martinmitrevski Aug 12, 2024

Choose a reason for hiding this comment

martinmitrevski Aug 12, 2024

Choose a reason for hiding this comment

laevandus Aug 14, 2024

Choose a reason for hiding this comment

sonarqubecloud bot commented Aug 14, 2024

Quality Gate passed

laevandus commented Aug 9, 2024 •

edited

Loading

github-actions bot commented Aug 9, 2024 •

edited

Loading

Stream-SDK-Bot commented Aug 9, 2024 •

edited

Loading

Stream-SDK-Bot commented Aug 9, 2024 •

edited

Loading