v1.23.1

Dragonfly v1.23.1

This is a patch release.
Update urgency - low.

fix: use less memory when loading huge values: #3829, #3825, #3807
fix: crash in stream code + performance improvements: #3775
fix: crash when using RENAME with search indexes: #3803

Full Changelog: v1.23.0...v1.23.1

v1.23.0

Dragonfly v1.23.0

Some prominent changes include:

• Fix container healthchecks for both privileged and non-privileged environments #3644
• Fix FT.AGGREGATE command (#3657 #3660 )
• Fix regression with the S3 access #3717
• Added ZRANGESTORE command #3757
• introducerss_oom_deny_ratio flag (#3702) - now Dragonfly will reject write commands if its RSS memory usage
  crosses maxmemory*rss_oom_deny_ratio. The default value of this flag is set to 1.25.
• Support for "CLIENT ID" and "CLIENT SETINFO" commands (#3673 and #3672)
• Better memory tracking for JSON objects (#3641)

What's Changed

• fix(docker/healthcheck): run netstat port retreival command as dfly by @Pothulapati in #3647
• test: remove redundant tests by @cunla in #3642
• chore: allow disabling io_uring registered buffers by @romange in #3650
• CmdArgParser improvement by @BorysTheDev in #3633
• fix: deadlock in the cluster migration process by @BorysTheDev in #3653
• fix: fix multi mget exec error message by @BorysTheDev in #3662
• test: fix MC test_expiration by @BorysTheDev in #3663
• chore: add some community suggestions to README by @Niennienzz in #3664
• fix: recursive calls in the allocation tracker by @romange in #3665
• chore: cosmetic changes around Snapshot functions by @romange in #3652
• fix(search_family): Fix FT.AGGREGATE GROUPBY option by @BagritsevichStepan in #3657
• chore: Add CLIENT ID command by @chakaz in #3672
• deprecate RecordsPopper and serialize channel records during push by @romange in #3667
• feat(server): Support CLIENT SETINFO by @chakaz in #3673
• server logs: change script error to warning by @adiholden in #3670
• fix(search_family): Fix LOAD option behavior in the FT.AGGREGATE command by @BagritsevichStepan in #3660
• test: unskip cluster tests and add debug info by @BorysTheDev in #3681
• chore: adjust RdbChannel sizes by @romange in #3676
• chore: add a script that parses allocator tracking logs by @romange in #3687
• chore: limit number of descriptors in the exec map by @romange in #3688
• fix(server): fix replication master deadlock on cancelation flow by @adiholden in #3686
• fix: join for cancel incoming migration by @BorysTheDev in #3692
• fix(json_family): Fix the JSON.SET bug if the path is in legacy mode and is not the root by @BagritsevichStepan in #3693
• fix: test_acl_revoke_pub_sub_while_subscribed by @kostasrim in #3680
• test: fix test_cluster_replication_migration by @BorysTheDev in #3699
• fix: test_replicaof_reject_on_load assert failure by @kostasrim in #3697
• fix(json_family) Add NOESCAPE option to the JSON.GET command by @BagritsevichStepan in #3685
• chore: pull latest helio by @kostasrim in #3700
• feat(core): Support RISCV RVV by @howjmay in #3655
• fix: allow parsing extra spaces on acl files by @kostasrim in #3703
• Update dflycluster slot-migration-status reply by @BorysTheDev in #3707
• chore: add clone benchmark by @romange in #3709
• fix: add default value has_mc_flag field by @BorysTheDev in #3710
• chore: symbolize stack traces in tests upon crash by @romange in #3714
• fix: fix s3 load snapshot by @andydunstall in #3717
• fix(cluster): Do not Pause() replication / migrations by @chakaz in #3716
• chore: add log info on failed commands by @kostasrim in #3694
• chore: enable ReplyGuard in ReplyBuilder2 by @kostasrim in #3705
• tests: fix and enable s3 snapshot test by @andydunstall in #3720
• fix(json_family): Fix JSON.ARRPOP command in legacy mode should not return WRONGTYPE error by @BagritsevichStepan in #3683
• fix(json_family): Separate double and int values during the comparison of the JSON objects by @BagritsevichStepan in #3711
• fix: Fix test test_client_pause_with_replica by @chakaz in #3729
• fix: Fix cluster_mgr.py by @chakaz in #3730
• fix: Fix test_take_over_seeder by @chakaz in #3733
• chore: Remove ReqSerializer by @dranikpg in #3721
• fix(json_family): Fix out of bound ranges for the JSON.ARR* commands by @BagritsevichStepan in #3712
• fix: properly track json memory usage by @kostasrim in #3641
• test: add tests for replication by @BorysTheDev in #3734
• test: unskip test for debugging purpose by @BorysTheDev in #3738
• refactor: use CmdArgParser for XGROUP command by @BorysTheDev in #3739
• test: add test for replication deadlock on replication timeout by @adiholden in #3691
• chore: Switch ports for cluster_mgr_test.py by @chakaz in #3741
• chore: change log level to warning for empty keys by @kostasrim in #3722
• chore: add SetReplies by @dranikpg in #3727
• chore: introduce a Clone function for the dense set by @romange in #3740
• chore: add missing await in test_take_over_seeder by @kostasrim in #3744
• fix: allow the healthcheck run in non-privileged containers as well by @romange in #3731
• fix: add value range check for SETBIT command by @BorysTheDev in #3750
• fix: zset store conclude transaction on error by @andydunstall in #3755
• chore: refactor a lambda function into a named one by @romange in #3753
• chore: fix return on bad status by @dranikpg in #3758
• feat(helm): add hostNetwork, topologySpreadConstraint and clusterIP su… by @katapultcloud in #3389
• feat(server): introduce oom_deny_commands flag by @adiholden in #3718
• feat(server): introduce rss oom limit by @adiholden in #3702
• fix: test deadlock with processing the stdout of sed by @romange in #3735
• chore: add integrity checks to consumer->pel by @romange in #3754
• fix: make snapshotting process more responsive by @romange in #3759
• fix: tune test_replicaof_reject_on_load parameters by @romange in #3762
• chore: logs on assert fail for test_acl_cat_commands_multi_exec_squash by @kostasrim in #3749
• chore: add more error logs around ziplist parsing checks by @romange in #3764
• fix: substr/getrange result for invalid range by @BorysTheDev in #3766
• feat(zset_family): add ZRANGESTORE by @andydunstall in #3757
• fix server: fix last error reply by @adiholden in #3728
• chore: give up on InlinedVector due to spurious warnings with optional by @romange in #3765
• fix: mark pubusb commands as unsupported for cluster by @BorysTheDev in #3767
• chore: some renames + fix a typo in RETURN_ON_BAD_STATUS by @romange in #3763
• fix: setrange with empty value doesn't modify the DB by @BorysTheDev in #3771

New Contributors

• @howjmay made their first contribution in #3655
• @katapultcloud made their first contribution in https://githu...

v1.22.2

Dragonfly v1.22.2

This is a patch release.

fix(docker/healthcheck): run netstat port retreival command as dfly (#3647)
fix: data inconsistency on replica during replication (#3710)

v1.22.1

Dragonfly v1.22.1

This is a patch release. Urgency to update: low

fix: infinite recursive calls in the allocation tracker #3665

Full Changelog: v1.22.0...v1.22.1

v1.22.0

Dragonfly v1.22.0

Some prominent changes include:

• Lots of stability fixes around JSON and Search APIS:
  #3459, #3284, #3537, #3544,#3580, #3582, #3578, #3591, #3597, #3539
• Support HELLO AUTH handshake: #3486
• Implement FT.TAGVALS: #3493
• Introduce Dragonfly specific lua pragmas: #3517
• Memcache now replicates flags correctly: #3538
• Memcache API now supports absolute timestamps as well: #3607
• PEXPIRETIME and EXPIRETIME are implemented by @Eunoia1729 - thank you!
• Last but not least, a significant contribution by @cunla: a massive test suite, likely the largest in Dragonfly's history, with 12K lines of code. This contribution has sparked further efforts to enhance Dragonfly's compatibility and stability. The initial results are included in this release.

What's Changed

• fix: missing logs on ci timeout by @kostasrim in #3452
• fix: JSON.MSET command by @BorysTheDev in #3459
• chore: integrate quicklist changes from valkey by @romange in #3440
• feat(json_family): Add support of the JSON legacy mode by @BagritsevichStepan in #3284
• fix: pytest teardown exception in os.remove(LAST_LOGS) by @kostasrim in #3463
• chore: change how we track memory_budget during evictions by @romange in #3457
• chore: add timeout to replication sockets by @romange in #3434
• chore(server): Remove old blocking debug by @dranikpg in #3460
• chore: SinkReplyBuilder2 with vec batching by @dranikpg in #3454
• feat: Allow pre-declaring Lua SHAs to run with undeclared keys by @chakaz in #3465
• chore(server): Use parser more in list_family by @dranikpg in #3461
• fix: Make replica's REPLCONF IP-ADDRESS optional by @chakaz in #3473
• chore: make tiered_test more reliable by @romange in #3477
• chore(facade): Update SinkReplyBuilder2 by @dranikpg in #3474
• refactor: set info_replication_valkey_compatible=true by @BorysTheDev in #3467
• chore: relax the version check in test_replicate_old_master by @romange in #3478
• chore: Download Dragonfly versions only when needed by @chakaz in #3479
• fix: division by zero bug by @romange in #3482
• test: fix test_scripts by @BorysTheDev in #3485
• chore(facade): RedisReplyBuilder2 base by @dranikpg in #3475
• chore(facade): MCReplyBuilder2 by @dranikpg in #3480
• chore(facade): RedisReplyBuilder2 (extensions) by @dranikpg in #3481
• fix: big value serialization corner cases by @kostasrim in #3430
• fix: HELLO AUTH with non default user by @kostasrim in #3486
• fix: flake in teardown of test_unknown_dfly_env by @kostasrim in #3487
• fix: test_replicate_old_master missing serialization_max_chunk_size by @kostasrim in #3498
• chore: add logs to debug tiered memory failures by @romange in #3499
• feat: implement FT.TAGVALS by @romange in #3493
• test: fix test_acl_deluser by @BorysTheDev in #3488
• fix: sanitizers post failure on google chat by @kostasrim in #3496
• chore(generic_family): Fix bad data format error in the RESTORE command by @BagritsevichStepan in #3501
• fix: sanitizers false positives by @kostasrim in #3495
• feat: Expose replica_reconnect_count for Prometheus metrics by @BagritsevichStepan in #3370
• fix: pytest warnings by @kostasrim in #3497
• fix: missing virtual destructors in ReplyBuilder2 by @kostasrim in #3506
• chore: split engine_shard file from engine_shard_set by @romange in #3507
• chore: introduce a secondary TaskQueue for shards by @romange in #3508
• fix: clang warnings by @romange in #3509
• chore: reduce pipelining latency by reusing existing shard fibers by @romange in #3494
• fix: bugs around the growth of a tiered file by @romange in #3516
• chore: Change Lua embedded flags syntax by @chakaz in #3517
• feat(cluster): Allow appending RDB to existing store by @chakaz in #3505
• fix: disable ThreadLocalMutex when big value ser is off by @kostasrim in #3521
• chore(io): Introduce (carefully) new io with use_new_io flag by @dranikpg in #3513
• chore(io): Optimize repeated ReservePiece calls by @dranikpg in #3525
• fix: disable sanitizers false positive by @kostasrim in #3522
• chore(config): make pipeline_squash configurable by @dranikpg in #3529
• chore(traffic loger): use pipelining and print/analyze commands by @dranikpg in #3527
• fix: skip empty objects on load and replication by @kostasrim in #3514
• fix: deadlock in Heartbeat() by @kostasrim in #3530
• fix(search_family): Add error on creating indexes from non-zero databases by @BagritsevichStepan in #3537
• fix: macos build by @kostasrim in #3536
• fix: MC flags size and serialization #3134 by @BorysTheDev in #3538
• fix(transaction): Don't set continuation for blocking by @dranikpg in #3540
• fix: named volume permissions in docker by @romange in #3518
• fix(jsonpath): Add JsonPath grammar for the child identifier in brackets by @BagritsevichStepan in #3533
• fix: build errors in sanitizers daily workflow by @kostasrim in #3542
• test: tests using fakeredis to check compatibility by @cunla in #3534
• chore: fix search replication by @dranikpg in #3547
• chore: add deallocation logs in the allocation tracker by @romange in #3549
• fix: fakeredis tests support python 3.10 by @cunla in #3550
• feat(server_family): Add backup/restore Prometheus metrics by @BagritsevichStepan in #3520
• docs: Fakeredis tests README by @cunla in #3552
• fix: zinterstore correctly finds weights by @romange in #3554
• fix(search_family): Fix query parsing for the integer tags in OR expression by @BagritsevichStepan in #3544
• chore(server): Fix watch by @dranikpg in #3557
• chore(search): fix numeric index query in rev order by @dranikpg in #3555
• chore: allow limiting pipelining queue by length by @romange in #3551
• fix: add unsupported_server_types to fakeredis tests by @cunla in #3560
• fix: string compatibility issues by @romange in #3564
• fix(server): Fix ZRANGEBYLEX limit params by @dranikpg in #3562
• fix: compatibility around list and string commands by @romange in #3565
• chore(server): Unify zset arg parsing by @dranikpg in #3563
• fix: compatibility around list,string and sort commands by @romange in #3568
• chore(server): Sort correctly in ZINTER by @dranikpg in #3566
• chore(server): Unify zset boolean operations into single function by @dranikpg in #3567
• chore(transaction): More blocking tests by @dranikpg in #3546
• fix: delete empty dense sets in HGetGeneric by @kostasrim in #3543
• chore(search): Silence query parser error by @dranikpg in https:...

v1.21.4

Dragonfly v1.21.4

This is a patch release.

chore: fix search replication (#3547)

v1.21.3

Dragonfly v1.21.3

This is a patch release.

fix: display real replica remote address in INFO logs
fix: crash with concurrent multi-shard BLMOVE operations #3540

This follows up on Dragonfly v1.21.2 release.

v1.21.2

Dragonfly v1.21.2

This is a patch release.

fix: disable code that can cause potential deadlocks during the replication (#3521)

This follows up on Dragonfly v1.21.0 release, that includes the following prominent features:

• Alpha release of SSD Data tiering - enabled with flag --tiered_prefix some/path/basename
• Very basic support of multi-tenancy #3260
• HSETEX now supports NX option, see our docs.
• Added support for JSON.MERGE and for JSON.MSET.
• valkey's replica-announce-ip and --cluster--announc--ip are consolidated via --announce-ip flag.

Full Changelog: v1.21.1...v1.21.2

v1.21.1

Dragonfly v1.21.1

This is a patch release.

fix: the replication from older masters to newer versions (#3473)

This follows up on Dragonfly v1.21.0 release, that includes the following prominent features:

• Alpha release of SSD Data tiering - enabled with flag --tiered_prefix some/path/basename
• Very basic support of multi-tenancy @3260
• HSETEX now supports NX option, see our docs.
• Added support for JSON.MERGE and for JSON.MSET.
• valkey's replica-announce-ip and --cluster--announc--ip are consolidated via --announce-ip flag.

v1.21.0

Dragonfly v1.21.0

Some prominent changes include:

• Alpha release of SSD Data tiering - enabled with flag --prefix some/path/basename
• Very basic support of multi-tenancy @3260
• HSETEX now supports NX option, see our docs.
• Added support for JSON.MERGE and for JSON.MSET.
• valkey's replica-announce-ip and --cluster--announc--ip are consolidated via --announce-ip flag.

What's Changed

• feat(server): master stop sending exec opcode to replica by @adiholden in #3289
• chore: optimize zpopminmax operations by @romange in #3291
• fix(json_family): Fix error in JsonFamilyTest.MGet by @BagritsevichStepan in #3285
• fix: define macro WITH_AWS in cmake when flag is ON by @kostasrim in #3296
• chore: Add 'memory arena show' command by @romange in #3298
• fix: missing logs on pytest failures #3255 by @BorysTheDev in #3272
• chore: refactor compact_object and introduce materialize method by @romange in #3300
• fix(acl): loading interleaved plain and hashed passwords by @kostasrim in #3297
• chore: Add CompactObj Raw methods by @romange in #3303
• chore: On invalid TTL, print the TTL first by @chakaz in #3306
• chore: moving functions + renaming in tiered storage by @romange in #3304
• fix(tests): Add missing awaits by @chakaz in #3273
• chore: refactoring around tiered storage by @romange in #3299
• feat: yield when serialization is in progress by @kostasrim in #3220
• refactor: Use DbContext, OpArgs and Transaction to access DbSlice by @chakaz in #3311
• chore: Separate tiered serialization format from object values by @romange in #3310
• chore: add more community links to README by @Niennienzz in #3308
• fix(cluster): Join on specified attempt id by @chakaz in #3305
• chore(acl): add test with requirepass and aclfile by @kostasrim in #3312
• chore: skip test_cluster_flushall_during_migration by @kostasrim in #3316
• fix (pytest): generate unique random dbfilename for tests by @adiholden in #3317
• chore: bypass decoding/encoding of the data when performing offloading by @romange in #3315
• refactor: acl helpers and global tables by @kostasrim in #3309
• chore(tiering): Fixes by @dranikpg in #3225
• fix(migration): Use transactions! by @dranikpg in #3266
• fix: forbid DFLYCLUSTER commads set for emulated cluster mode by @BorysTheDev in #3307
• feat(namespaces): Initial support for multi-tenant by @chakaz in #3260
• chore: improve dfly_bench stats by @romange in #3320
• chore: remove replace_deleted flag from hnswlib by @dranikpg in #3323
• feat(test): Improve benchmark workflow by @adiholden in #3330
• fix: Proper shutdown sequence with Namespaces by @chakaz in #3333
• fix(test): copy logs for failed test during TEARDOWN phase by @BorysTheDev in #3331
• fix(json_family): fix JSON.STRAPPEND command for JSON legacy mode by @BagritsevichStepan in #3264
• chore(tiering): add protection against overruning memory budget by @romange in #3327
• chore: Add coordinated omission mode by @romange in #3332
• chore: implement sequential pass without the overlapping traffic by @romange in #3335
• feat: add an option to flush serialized entries on threshold limit by @kostasrim in #3241
• feat(hset_family): Add NX option to HSETEX by @BagritsevichStepan in #3295
• Fix blocking commands moved error by @BorysTheDev in #3334
• fix: ub in RegisterOnChange and regression tests for big values by @kostasrim in #3336
• fix: Cancel outgoing migration when retrying / closing by @chakaz in #3339
• chore: Make KeyIndex iterable by @dranikpg in #3326
• fix: AllocationTracker::Remove return value was reversed by @romange in #3341
• chore: remove redundant metrics from memory stats by @romange in #3345
• fix: corruption in replication stream by @romange in #3344
• chore: clean up TaskQueue since we do not need multiple fibers for it by @romange in #3348
• chore: pull helio by @romange in #3350
• chore: Log connection context when issuing dangerous cmds by @chakaz in #3352
• chore: small rename and add dcheck on LocalBlockingCounter by @kostasrim in #3356
• refactor: reduce number of logs for cluster by @BorysTheDev in #3357
• chore: fixes to dfly_bench by @romange in #3353
• chore: fix test_parser_memory_stats flakiness by @romange in #3354
• chore(server): Introduce StringSetWrapper by @dranikpg in #3347
• fix: do not upload offload values on a first hit by @romange in #3360
• chore(tiering): Range functions + small refactoring by @dranikpg in #3207
• fix: failure in test_cluster_fuzzymigration by @romange in #3363
• fix(server): Require >=1 args to GETEX by @chakaz in #3366
• fix(transaction): Fix namespace access by @dranikpg in #3364
• chore: disable compression on big values by @kostasrim in #3358
• fix: protect OnJournalEntry with ConditionGuard by @kostasrim in #3367
• chore: Introduce CoolQueue by @romange in #3365
• chore: small fixes around tiering by @romange in #3368
• chore: add a test for HeapSize() function by @romange in #3349
• chore: simplify computation of used_mem_current by @romange in #3372
• chore: disable cluster_fuzzymigration by @kostasrim in #3373
• chore(replica): remove unused methods in the Replica class by @BagritsevichStepan in #3374
• fix(transaction): Properly store block cancel status by @dranikpg in #3371
• chore: Track db_slice table memory instantly by @romange in #3375
• chore: add mem test for big values and default the flag by @kostasrim in #3369
• update: replication_acks_interval flag to 1000 by @kostasrim in #3378
• chore: dfly_bench - print ongoing error counts by @romange in #3382
• chore: introduce a cool queue that gradually retires cool items by @romange in #3377
• chore: update cached stats inside PollExecution by @romange in #3376
• fix: Fix test_take_over_seeder by @chakaz in #3385
• chore: reenable evictions upon insertion to avoid OOM rejections by @romange in #3387
• fix: remove fiber guard from non atomic section by @kostasrim in #3381
• chore: do not preempt on db_slice::RegisterOnChange by @kostasrim in #3388
• fix: disable inline transactions when db_slice has registered callbacks by @kostasrim in #3391
• fix: test_big_value_serialization_memory_limit shutdown timeout by @kostasrim in #3390
• chore: set serialization_max_chunk_size to 1 byte by @kostasrim in #3379
• chore: tiere...