v1.26.0

Dragonfly v1.26.0 - Chrismukkah edition 🔯 🎄🔯 🎄🔯

This marks our final release for the year, featuring numerous stability fixes and performance enhancements.

Important features and changes:

1. Numerous Search and Json fixes #4070 #4012 #4205 #4228 #4231 #4311 #4343
2. Account for list type memory, fixes #3800
3. Native support of GCS storage! Just point your --dir to gs::\mybucket\df_snapshots\
4. Fix metrics compatibility with prometheus 3.0 scraping #4277
5. Series of STREAM improvements bringing Dragonfly to be compatible with redis/valkey 7.2 streams: #4192, #4237, #4239 #4240 #4281 #4285

What's Changed

• chore: remove old io by @dranikpg in #3953
• chore: RdbSerializer::SaveListObject supports QList by @romange in #4101
• chore: Implement list Pop/Erase functionality with QList by @romange in #4099
• chore: change Namespaces to be a global pointer by @romange in #4032
• fix: update instruction how to build from the source for ubuntu by @BorysTheDev in #4103
• fix: test_noreply_pipeline flakiness by @romange in #4102
• chore: Implement LMOVE over QList by @romange in #4104
• fix(search_family): Process wrong field types in indexes for the FT.SEARCH and FT.AGGREGATE commands by @BagritsevichStepan in #4070
• chore: implement Erase for a range by @romange in #4106
• chore: decouple reply_builder from ConnectionContext by @romange in #4069
• chore: support rdb loading and container utils with QList. by @romange in #4109
• feat(server) : snapshot traverse physical buckets by @adiholden in #4084
• fix: skip Send() in SinkReplyBuilder::Flush() when vec is empty by @kostasrim in #4114
• fix: regression in squashing code when determining eval commands by @romange in #4116
• fix: slow regression tests tests by @kostasrim in #4117
• test: add test for snapshoting during migration by @BorysTheDev in #4108
• feat server: experimental_new_io flag add as deprecated by @adiholden in #4127
• chore: simplify BumpUps deduplication by @romange in #4098
• fix: Do not use cc_ in connection if it's null by @chakaz in #4131
• fix: Use MOVED error type for moved replies by @chakaz in #4125
• chore(deps): bump github/codeql-action from 3.27.0 to 3.27.1 in the actions group by @dependabot in #4115
• fix server: fix stats of pipeline squashed commands by @adiholden in #4132
• chore: fix plain node insertion by @romange in #4134
• chore: improve the state machine of RedisParser by @romange in #4085
• chore: run tests with list_experimental_v2 enabled by @romange in #4112
• chore: optimize info command by @romange in #4137
• fix server: fix write to slowlog on squashing flow by @adiholden in #4138
• chore:update fakeredis, remove irrelevant tests by @cunla in #4014
• chore: fix a potential crash during client list by @romange in #4141
• fix: migration ACK response processing by @BorysTheDev in #4140
• feat: add INFO memory section for squashing replies memory consuming by @BorysTheDev in #4147
• fix: allow SELECT in multi/exec if it's a noop by @romange in #4146
• chore(deps): bump github/codeql-action from 3.27.1 to 3.27.4 in the actions group by @dependabot in #4148
• chore: tune logs and improve restrict denied error by @romange in #4145
• fix: UB during cmd squashing reply size calculation by @BorysTheDev in #4149
• fix: enforce load limits when loading snapshot by @romange in #4136
• fix: test_replication_all failure by @romange in #4155
• fix: Huge entries fail to load outside RDB / replication by @chakaz in #4154
• chore: allow slow and precise memory measurement of an object by @romange in #4160
• chore: fix a regression build break by @romange in #4162
• chore: log exit code of failing dragonfly in tests by @romange in #4166
• fix: remove DenseSet::IteratorBase::TraverseApply by @kostasrim in #4170
• adjusted grafana dashboard to be more user friendly by @applike-ss in #4165
• chore: fix shutdown sequence in Dragonfly server by @romange in #4168
• feat(server): add eval_ro and evalsha_ro by @andydunstall in #4091
• chore: Hide replicas from CLUSTER subcmds in managed mode by @chakaz in #4174
• fix(tools): Prevent dragonfly.logrotate from stopping logrotate service by @s0xDk in #4176
• chore: more parser improvements by @romange in #4177
• fix(release/helm): allow empty commits for rerun by @Pothulapati in #4163
• fix(search_family): Fix LOAD fields parsing in the FT.AGGREGATE and FT.SEARCH commands by @BagritsevichStepan in #4012
• feat: Huge values breakdown in cluster migration by @chakaz in #4144
• feat: cluster migrations restarts if timeout happens by @BorysTheDev in #4081
• fix: deduplicate mget response by @romange in #4175
• fix: stop using openssl for container healthchecks by @romange in #4181
• chore: produce core files in regtests by @romange in #4185
• chore: stop periodic task earlier during the shutdown process by @romange in #4187
• chore(deps): bump github/codeql-action from 3.27.4 to 3.27.5 in the actions group by @dependabot in #4186
• test: skip test_cluster_mgr because of unclosed instance by @BorysTheDev in #4191
• chore: fix false positives sanitizers by @kostasrim in #4190
• fix: debug object encoding names by @romange in #4188
• feat: add ability reading stream_listpacks_2/3 rdb types by @BorysTheDev in #4192
• chore: get back on the decision to put a hard limit on command interface by @romange in #4203
• chore: qlist improvements by @romange in #4194
• fix: stream memory tracking by @kostasrim in #4067
• chore: fully support qlist in all list_family commands by @romange in #4201
• feat(contrib/helm): evaluate the provided passwordSecretName value as a template by @oOraph in #4046
• fix(search_family): Fix crash when no SEPARATOR is specified in the FT.CREATE command by @BagritsevichStepan in #4205
• chore: update command interface for hset/set families by @romange in #4209
• fix: cluster_mgr script by @BorysTheDev in #4210
• bug(replication): snapshot cleanup fix in transition to stable sync by @adiholden in #4211
• test: fix and unskip test_migration_timeout_on_sync by @BorysTheDev in #4216
• chore: Add more qlist tests by @romange in #4217
• chore: change the interface of stream and server commands by @romange in #4219
• chore: add noop docs subcmd by @kostasrim in #4214
• chore: remove tail field from qlist by @romange in #4220
• chore: qlist code clean ups by @romange in #4223
• chore: Track QList memory by @romange in #4226
• chore: change the interface of hll, generic and list commands...

v1.25.6

This is a patch release.

What's Changed

• fix: crash during getting info about replication (#4328)
• fix: potential crash - do not check-fail in OpRestore (#4332)

Full Changelog: v1.25.5...v1.25.6

v1.25.5

This is a patch release.

What's Changed

1. Fix crash when no SEPARATOR is specified in the FT.CREATE command (#4205)
2. Fix crash when defragmenting StringSet (#4283)
3. Fix crash caused by destructor in RaxTreeMap (#4228 )

Full Changelog: v1.25.4...v1.25.5

v1.25.4

This is a patch release.

What's Changed

1. fix: rdb load fix loading plain lists 4959bef
2. fix conflict on cherry-pick rdb load fix 976586c

full change log: v1.25.3...v1.25.4

v1.25.3

This is a patch release.

What's Changed

1. fix: stop using openssl for container healthchecks (#4181)
2. fix: deduplicate mget response (#4184)
3. fix(tools): Prevent dragonfly.logrotate to stop logrotate service (#4176)
4. fix: remove DenseSet::IteratorBase::TraverseApply (#4170)
5. chore: Hide replicas from CLUSTER subcmds in managed mode (#4174)

Full Changelog: v1.25.2...v1.25.3

v1.25.2

This is a patch release.

1. fix: Huge entries fail to load outside RDB / replication (#4154)
   fixes #4143
2. chore: optimize info command (#4137)
   decrease latency for "info server" and "info replication" commands
3. fix: Use MOVED error type for moved replies (#4125)
   Use type MOVED for moved replies, making them all the same under # Errorstats
   Fixes #4118
4. fix: a potential crash during client list (#4141)

v1.25.1

This is a patch release.

fix: regression in squashing code when determining eval commands (#4116)

The regression was caused by #3947 and it causes crashes in bullmq.
It has not been found till now because python client sends commands in uppercase.
Fixes #4113

Signed-off-by: Roman Gershman [email protected]

v1.25.0

Dragonfly v1.25.0

Some prominent changes include:

• FT.SEARCH fixes: #3956 #3955 #4043
• Prefix search - supporting queries like "@color:{gre* | orange | yellow*}: #3972
• GCS save/load support #4006
• Few Improvements around replication/snapshotting - it should be more efficient around large entries #4041
• zrevrank/zrank support withscore option #4001

What's Changed

• chore: ignore applying the same cluster config twice by @BorysTheDev in #3932
• test: add test to reproduce a lot of memory consumtion during migration by @BorysTheDev in #3939
• chore(refactoring): header clean ups by @romange in #3943
• chore: get rid of ToUpper call and use AsciiStrToUpper by @romange in #3944
• chore: remove ToUpper calls in main_service by @romange in #3947
• chore: get rid of ToUpper/ToLower mutations on arguments by @romange in #3950
• chore(facade): Fix bad new IO glue by @dranikpg in #3940
• feat: track request sizes histograms by @romange in #3951
• chore: pass SinkReplyBuilder and Transaction explicitly. Part1 by @romange in #3946
• chore: disable test_cluster_memory_consumption_migration by @kostasrim in #3948
• fix(search_family): Temporary remove the error when a field name does not have the '@' sign at the beginning in the FT.AGGREGATE command by @BagritsevichStepan in #3956
• chore: tune test_rss_used_mem_gap by @kostasrim in #3958
• test: add test to test big collections or collections with big values by @BorysTheDev in #3959
• chore: increase load in test_noreply_pipeline by @kostasrim in #3960
• chore(fakeredis): Enable JSON tests in the Fakeredis tests by @BagritsevichStepan in #3773
• chore: add defrag logic for zsets by @kostasrim in #3836
• chore: pass SinkReplyBuilder and Transaction explicitly. Part2 by @romange in #3954
• chore: pass SinkReplyBuilder and Transaction explicitly. Part3 by @romange in #3966
• fix: crash if we OOM during migration process by @BorysTheDev in #3968
• fix(cluster): fix backoff on unknown migration by @andydunstall in #3965
• chore: pass SinkReplyBuilder and Transaction explicitly. Part4 by @romange in #3967
• chore: get rid of MutableSlice by @romange in #3952
• chore: reduce dependency of debug/memory commands on ConnectionContext by @romange in #3977
• fix: properly set object with DenseLinkKey by @romange in #3980
• fix(memory-leak): add missing destructor to RaxTreeMap by @kostasrim in #3975
• fix: dangling string in DebugCmd::Reload by @kostasrim in #3981
• chore: pass SinkReplyBuilder and Transaction explicitly. Part5 by @romange in #3976
• chore: pass SinkReplyBuilder and Transaction explicitly. Part6 by @romange in #3987
• fix(build): Reduced checks permission in build to read #3821 by @irufus in #3916
• fix: properly set ttl bit during object replacement by @romange in #3991
• fix: replace string_view with string in RaxTreeMap::FindIterator by @kostasrim in #3982
• fix: prevent starting migrations from replica by @BorysTheDev in #3985
• feat(search): Prefix search for tags by @dranikpg in #3972
• chore: pass SinkReplyBuilder and Transaction explicitly. Part8 by @romange in #3992
• chore: pass SinkReplyBuilder and Transaction explicitly. Part7 by @romange in #3988
• chore: pass SinkReplyBuilder and Transaction explicitly. Part9 by @romange in #3996
• fix: keep the ttl bit when dense_set grows with expiring items by @romange in #3995
• refactor(workflow): changes default permissions to read for few workflows increasing security by @BLANKatGITHUB in #3905
• fix(search_family): Fix indexes loading in the FT.SEARCH and Ft.AGGREGATE commands by @BagritsevichStepan in #3955
• feat: introduce metrics/logs of when pipelining is being throttled by @romange in #4000
• chore: clean up conn_use_incoming_cpu heuristic by @romange in #3978
• chore: refactor snapshot expanding logic by @romange in #4003
• chore: introduce GcsSnapshotStorage by @romange in #4004
• fix: flaky test_failover test by @romange in #4007
• chore: pass SinkReplyBuilder and Transaction explicitly. Part10 by @romange in #3998
• fix: build break in search_family by @romange in #4008
• feat(zset_family): support WITHSCORE in zrevrank/zrank commands (#3921) by @Diskein in #4001
• fix: separate Heartbeat and ShardHandler to fibers by @kostasrim in #3936
• chore: improve verbosity logging during shutdown by @romange in #4010
• chore: skip AggregateWithLoadOptionHard on sanitizers (false positive) by @kostasrim in #4013
• chore: pass RedisReplyBuilder explicitly from dragonfly connection by @romange in #4009
• chore: print info stats if test_noreply_pipeline fails by @romange in #4016
• chore: flush journal to channel for non auto journal commands by @kostasrim in #3945
• chore: support load/save from GCS by @romange in #4006
• chore: pull helio by @romange in #4036
• chore: decrease logs severity for OOM from error to warning by @BorysTheDev in #4031
• feat(pre-commit): adds senstive data check like api keys by @BLANKatGITHUB in #4030
• test: update test_big_containers by @BorysTheDev in #4025
• fix: reduce memory consumption during migration by @BorysTheDev in #4017
• chore: add tx queue head debug info in AnalyzeTxQueue by @kostasrim in #4026
• chore: refactor part of s3 logic for loading a snapshot. by @romange in #4044
• chore: add retry to test_noreply_pipeline test by @romange in #4045
• chore(DenseSet): defrag all links in a chain by @kostasrim in #4019
• chore: add ForceUpdate to BPTree by @kostasrim in #3993
• fix(search_family): Fix crash in FT.PROFILE command for invalid queries by @BagritsevichStepan in #4043
• feat(dependabot): Add dependabot to check for vulnerabilies and updat… by @BLANKatGITHUB in #4035
• chore: pass reply_builder explicitly to pubsub module by @romange in #4021
• chore(deps): bump numpy from 1.24.1 to 2.1.3 in /tools by @dependabot in #4063
• chore(deps): bump github.com/redis/go-redis/v9 from 9.5.1 to 9.7.0 in /tools/replay by @dependabot in #4062
• chore(deps): bump tomli from 2.0.1 to 2.0.2 in /tests/dragonfly by @dependabot in #4059
• chore(deps): bump github.com/gruntwork-io/terratest from 0.41.11 to 0.47.2 in /contrib/charts/dragonfly by @dependabot in #4058
• chore(deps): bump redis-om from 0.2.2 to 0.3.3 in /tests/dragonfly by @dependabot in #4060
• chore(deps): bump mikepenz/action-junit-report from 4 to 5 by @dependabot in #4052
• chore(deps): bump mozilla-actions/sccache-action from 0.0.4 to 0.0.6 by @dependabot in #4051
• chore(deps): bump ossf/scorecard-action from 2.3.1 to 2.4.0 by @dependabot in #4053
• chore(deps): bump github/codeql-action from 3.24.9 t...

v1.24.0

Dragonfly v1.24.0

It's a maintenance release with mostly performance and bug fixes for edge cases.

Some prominent changes include:

1. feat: add HEXPIRE and FIELDEXPIRE by @NegatioN #3780
2. fix BITPOS command responses by @Diskein #3910
3. add allcommands/nocommands to ACL rules by @Lakshyadevelops #3783

What's Changed

• fix: Fix flaky test test_acl_revoke_pub_sub_while_subscribed by @chakaz in #3768
• chore: Forbid replicating a replica by @chakaz in #3779
• fix: GETRANGE params validation by @BorysTheDev in #3781
• fix: bitop do not add dst key if result is empty by @kostasrim in #3751
• chore(acl_family): add allcomands and nocommands by @Lakshyadevelops in #3783
• chore: remove goto statements by @kostasrim in #3791
• chore: ScheduleInternal refactoring by @romange in #3794
• test: update test_noreply_pipeline to prevent false fail by @BorysTheDev in #3801
• fix: bitcount invalid range by @BorysTheDev in #3792
• fix(generic_family): Update indexes in the RESTORE and RENAME commands by @BagritsevichStepan in #3803
• chore: cleanup not used opcodes in replication by @adiholden in #3804
• chore: tune TieredStorageTest.MemoryPressure by @romange in #3805
• fix: make sure dfly_bench reliably connects by @romange in #3802
• chore: reformat listpack according to valkey 8 by @romange in #3810
• fix: empty strings and Nan in the sort command by @BorysTheDev in #3809
• chore: refactor XRead by @dranikpg in #3775
• chore: singlehop XREAD by @dranikpg in #3817
• chore: introduce an optimized integer compare algorithm for lists by @romange in #3813
• chore: add a benchmark comparing lpStringToInt64 to SimpleAtoi by @romange in #3815
• feat(rdb_load): add support for loading huge sets by @andydunstall in #3807
• chore: now it's not needed to allocate quicklistIter on heap by @romange in #3814
• fix(rdb_load): fix loading huge hmaps with ttl by @andydunstall in #3825
• chore: allow config set notify_keyspace_events by @kostasrim in #3790
• chore: do not close connections at the end of pytest by @kostasrim in #3811
• fix(rdb_load): fix appending to an expired key by @andydunstall in #3829
• chore: add logs to test_acl_cat_commands_multi_exec_squash by @kostasrim in #3826
• chore: optimize element removal for integer lists by @romange in #3827
• refactor: remove CmdArgParser::ToUpper() by @BorysTheDev in #3831
• chore: add pending_ops to disk stats by @romange in #3837
• feat(rdb_load): add support for loading huge hmaps and zsets by @andydunstall in #3823
• add dense_set.SetExpiryTime in preparation for fieldexpire by @NegatioN in #3780
• test: more stabe test_bgsave_and_save by @adiholden in #3843
• chore: WrapSds from family_utils.h by @dranikpg in #3818
• chore: add defrag logic for sets by @kostasrim in #3833
• chore: extend RobjWrapper::sz_ to 2^56 by @romange in #3849
• feat(rdb_load): add support for loading huge lists by @andydunstall in #3850
• fix(rdb_load): fix partial reads dropping elements by @andydunstall in #3853
• refactor: refactor fakeredis tests by @cunla in #3852
• chore: simplify CloneBatch code by @romange in #3862
• feat: add HEXPIRE and FIELDEXPIRE by @NegatioN in #3842
• chore: improve performance of ClearInternal by @romange in #3863
• chore: Remove DenseSet::AddOrFindDense and AddSds by @romange in #3864
• chore: Implement AddMany method by @romange in #3866
• chore: ClearInternal now can clear partially by @romange in #3867
• feat(rdb_load): add support for loading huge streams by @andydunstall in #3855
• chore: Refactor AddMany by @romange in #3869
• chore: run memory decommit after snapshot load/save by @kostasrim in #3828
• chore: reduce usage of ToUpper by @romange in #3874
• opt: Optimize AllocationTracker to be efficient when enabled by @chakaz in #3875
• chore: allow rdb version 12 by @kostasrim in #3860
• chore: lock keys for optimistic transactions by @kostasrim in #3865
• fix: Do not publish to connections without context by @chakaz in #3873
• fix: Do not kill Dragonfly on failed DFLY LOAD by @chakaz in #3892
• chore: collection size by @dranikpg in #3844
• chore: skip flaky test_noreply_pipeline by @kostasrim in #3903
• fix(cluster): fix unknown migration error by @andydunstall in #3899
• chrore: parse cgroup v2 by @kostasrim in #3857
• feat(search): STOPWORDS by @dranikpg in #3851
• chore: schedule chains by @romange in #3819
• fix!: fix BITPOS command responses (#3893) by @Diskein in #3910
• fix: prevents Dragonfly from blocking in epoll during snapshotting by @romange in #3911
• fix: macos build by @romange in #3912
• chore(search): Rax TreeMap by @dranikpg in #3909
• chore: Add --allocator_tracker for default tracking by @chakaz in #3901
• chore: relax assertion in test_noreply_pipeline by @kostasrim in #3908
• feat(search): Prefix search* by @dranikpg in #3913
• fix: pre-commit ci workflow by @kostasrim in #3917
• fix: Support replicating Valkey and Redis 7.2 by @chakaz in #3927
• feat(server): use listpack node encoding for list by @adiholden in #3914
• fix: Fix test_flushall_in_full_sync by @chakaz in #3929
• fix: dragonfly_connection should only access the original reply_builder by @romange in #3924
• chore(search): Fix deprecated functions by @dranikpg in #3933

New Contributors

• @Lakshyadevelops made their first contribution in #3783
• @NegatioN made their first contribution in #3780
• @Diskein made their first contribution in #3910

Huge thanks to all the contributors! ❤️

Full Changelog: v1.23.0...v1.24.0

v1.23.2

This is a patch release.
fix: rdb load fix loading plain lists 4959bef

full change log: v1.23.1...v1.23.2