10 Jul 18:17

5adb348

v1.2.12 (Jul 03, 2019)

Overview of Changes

[IMPORTANT] This version includes a backward incompatible file format change for YSQL. This means that if you have an existing cluster with
YSQL schema, then you will not be able to connect to the YSQL database after upgrading to v1.2.12. There are two ways to work around this problem:

If you don't care about preserving your existing data, then you can delete your older cluster, upgrade to v1.2.12 and create a new cluster.
If you want to continue using your existing data, then please save your older data (in the form of SQL statements), delete the older cluster, upgrade to v1.2.12, create a new cluster and then load the previously saved data.

YSQL Changes

[YSQL] Fix scan operator when range key is not fully-specified.#1576
[YSQL] Fix ORDER BY range column of a secondary index.#1591
[YSQL] Add YSQL RPCs to tserver utilz page.#1446

System Improvements

Resolve ABA problem in LockFreeStackResolve ABA problem in LockFreeStack.#1561
Fix recursive locking in Batcher.#1584
Use boost::atomic for 16-bytes structures.#1189
Fix integer overflow during Raft log read path.#1606
Fix concurrent access to primary table info in RaftGroupMetadata.#1631
A new priority-aware thread pool implementation for compactions and flushes.#1314
Create mem trackers that reflects TC Malloc stats.#1633
Make sure we don't read an invalid replicate index from the footer.#1625

Performance Benchmarks

This section gives an overview of the performance of this release. The benchmarks were done on the following setup:

3 nodes with 16-vCPUs
Replication factor = 3
Each key-value is ~64 bytes (combined)
See setup details for more info

Summary

API	Workload	Type	Throughput	Latency	Avg CPU on DB node
YCQL	KeyValue	100% Reads	218K	1.17ms	48%
YCQL	KeyValue	100% Writes	73K	3.5ms	44%
YCQL	Batched Writes	100% Writes	247K	26ms	52%
YCQL	Secondary Index	100% Reads	187K	1.4ms	44%
YCQL	Secondary Index	100% Writes	5K	12.9ms	51%

Details

YCQL KeyValue

	Workload CassandraKeyValue with 256 writers and 0 readers: 
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
| Metric                         | mean      | min       | max       | std.dev   | median    | 1%        | 5%        | 10%       | 90%       | 95%       | 99%       |
| Read throughput, ops/sec       | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         |
| Read latency, ms/op            | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         | 0         |
| Write throughput, ops/sec      | 73037.05  | 68884.95  | 74102.43  | 1211.67   | 73480.55  | 68884.95  | 68884.95  | 71993.35  | 74051.57  | 74102.43  | 74102.43  |
| Write latency, ms/op           | 3.50      | 3.45      | 3.72      | 0.06      | 3.48      | 3.45      | 3.45      | 3.45      | 3.55      | 3.72      | 3.72      |
| Load tester CPU, user, %       | 16.91     | 14.90     | 18.10     | 0.88      | 17.30     | 14.90     | 14.91     | 15.18     | 17.70     | 18.08     | 18.10     |
| Load tester CPU, system, %     | 9.20      | 7.60      | 10.10     | 0.83      | 9.65      | 7.60      | 7.60      | 7.72      | 9.90      | 10.09     | 10.10     |
| Cluster node CPU, user, %      | 32.93     | 23.80     | 38        | 3.45      | 32.45     | 23.80     | 24.31     | 30.13     | 37.20     | 37.49     | 38        |
| Cluster node CPU, system, %    | 15.43     | 12.70     | 16.60     | 0.82      | 15.40     | 12.70     | 13.22     | 14.43     | 16.30     | 16.50     | 16.60     |
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+

	Workload CassandraKeyValue with 192 writers and 16 readers: 
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
| Metric                         | mean      | min       | max       | std.dev   | median    | 1%        | 5%        | 10%       | 90%       | 95%       | 99%       |
| Read throughput, ops/sec       | 18493.93  | 17156.62  | 19190.41  | 587.15    | 18606.10  | 17156.62  | 17156.62  | 17440.30  | 19151.33  | 19190.41  | 19190.41  |
| Read latency, ms/op            | 0.87      | 0.83      | 0.93      | 0.03      | 0.86      | 0.83      | 0.83      | 0.83      | 0.92      | 0.93      | 0.93      |
| Write throughput, ops/sec      | 60571.82  | 58544.09  | 62509.31  | 983.23    | 60550.72  | 58544.09  | 58544.09  | 59035.93  | 62380.73  | 62509.31  | 62509.31  |
| Write latency, ms/op           | 3.17      | 3.07      | 3.28      | 0.05      | 3.17      | 3.07      | 3.07      | 3.08      | 3.25      | 3.28      | 3.28      |
| Load tester CPU, user, %       | 17.68     | 16        | 20.70     | 0.87      | 17.55     | 16        | 16.06     | 17.20     | 18.39     | 20.58     | 20.70     |
| Load tester CPU, system, %     | 8.73      | 8.10      | 10.40     | 0.54      | 8.50      | 8.10      | 8.11      | 8.22      | 9.58      | 10.36     | 10.40     |
| Cluster node CPU, user, %      | 35.62     | 32.90     | 38.80     | 1.71      | 34.95     | 32.90     | 33.41     | 33.72     | 38.10     | 38.39     | 38.80     |
| Cluster node CPU, system, %    | 15.38     | 14.90     | 15.80     | 0.19      | 15.40     | 14.90     | 15.01     | 15.10     | 15.60     | 15.70     | 15.80     |
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+

	Workload CassandraKeyValue with 128 writers and 64 readers: 
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
| Metric                         | mean      | min       | max       | std.dev   | median    | 1%        | 5%        | 10%       | 90%       | 95%       | 99%       |
| Read throughput, ops/sec       | 71301.76  | 64468.86  | 78149.80  | 6074.55   | 69364.13  | 64468.86  | 64468.86  | 64563.72  | 78080.94  | 78149.80  | 78149.80  |
| Read latency, ms/op            | 0.90      | 0.82      | 0.99      | 0.08      | 0.92      | 0.82      | 0.82      | 0.82      | 0.99      | 0.99      | 0.99      |
| Write throughput, ops/sec      | 42974.61  | 40464.12  | 45473.25  | 1883.46   | 42860     | 40464.12  | 40464.12  | 40927.76  | 45053.88  | 45473.25  | 45473.25  |
| Write latency, ms/op           | 2.98      | 2.82      | 3.16      | 0.13      | 2.98      | 2.82      | 2.82      | 2.84      | 3.13      | 3.16      | 3.16      |
| Load tester CPU, user, %       | 21.46     | 19.10     | 23.30     | 1.28      | 21.60     | 19.10     | 19.15     | 20        | 23.08     | 23.29     | 23.30     |
| Load tester CPU, system, %     | 10.17     | 9.20      | 10.90     | 0.56      | 10.30     | 9.20      | 9.21      | 9.31      | 10.80     | 10.89     | 10.90     |
| Cluster node CPU, user, %      | 38.31     | 28.30     | 41.60     | 3.01      | 39.80     | 28.30     | 34.10     | 34.34     | 41.09     | 41.20     | 41.60     |
| Cluster node CPU, system, %    | 13.11     | 11.10     | 14        | 0.51      | 13.15     | 11.10     | 11.66     | 12.80     | 13.50     | 13.60     | 14        |
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+

	Workload CassandraKeyValue with 64 writers and 128 readers: 
+--------------------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
| Metric                         | mean      | min       | max       | std.dev   | median    | 1%        | 5%        | 10%       | 90%       | 95%       | 99%       |
| Read throughput, ops/sec       | 113115.50 | 111032.59 | 114690.37 | 1211.91   | 113593.04 | 111032.59 | 111032.59 | 111280.88 | 114639.78 | 114690.37 | 114690.37 |
| Read latency, ms/op            | 1.13      | 1.11      | 1.15      | 0.01      | 1.13      | 1.11      | 1.11      | 1.12      | 1.15      | 1.15      | 1.15      |
| Write throughput, ops/sec      | 21349.23  | 20717.49  | 21753.92  | 264.27    | 21335.10  | 20717.49  | 20717.49  | 20915.45  | 21699.52  | 21753.92  | 21753.92  |
| Write latency, ms/op           | 3.00      | 2.94      | 3.09      | 0.04      | 3         | 2.94      | 2.94      | 2.95      | 3.06      | 3.09      | 3.09      |
| Load tester CPU, user, %       | 23.54     | 21.80     | 27.90     | 1.24      | 23.50     | 21.80     | 21.81     | 22.03     | 24.36     | 27.7...

Assets 2

17 Apr 21:48

rkarthik007

v1.2.4

f2ecaa1

v1.2.4 (Apr 02, 2019)

Overview of Changes

If you don't care about preserving your existing data, then you can delete your older cluster, upgrade to v1.2.4 and create a new cluster.
If you want to continue using your existing data, then please save your older data (in the form of SQL statements), delete the older cluster, upgrade to v1.2.4, create a new cluster and then load the previously saved data.

YSQL Changes

Added support for primary keys. It is now possible to create a table with a primary key. #955
Added support for temporary tables. #805
Added support for CYCLE option in CREATE SEQUENCE statement. #993
Enabled SET TRANSACTION READ WRITE statement. #1081
Support RelationType in TableInfo to distinguish between system tables, user tables and user indexes. #906

Bug Fixes

Fixed a crash in tserver that is observed while inserting null value into an indexed column. #945
Fixed a bug where indexes on system catalog tables were not being updated during DDL. #1083
Fixed detection of single row transactions on tables that have primary keys.
Fixed MiniMaster startup race condition in test infrastructure. #1040

System Improvements

Reduce per-connection overhead for cache preloading. #1090 #1091
Faster release of sending buffers to reduce memory consumption when leader sends large updates to followers who are lagging behind. #1079
Parallelize DNS resolution while filling system.peers and system.local tables #1000
Script for post-processing test results and generating JSON-based result files.
Improve stability of QLTransactionTest.IntentsCleanupAfterRestart. #1102

Key Commits

[f2ecaa1] [YSQL] Fix single-row-txn detection after primary key change
[90680ca] [#1102]: Improve stability of QLTransactionTest.IntentsCleanupAfterRestart
[a4c614f] JSON-based test reports
[33835b0] [#1079]: Release sending buffers as soon as possible
[1bc77da] ENG-4682 (#906): Support RelationType (user table, index, system table) in TableInfo
[1dedcd1] ENG-4943 #945 [YSQL] Correct handling for NULL value in indexed column
[465b6d6] [YSQL] #1090 #1091 Reduce per-connection overhead for cache preloading
[382b89c] [YSQL] ENG-4601 (#805): Add support for temporary tables
[5165f01] #1081 [YSQL] Enable READ WRITE transaction
[9f31c41] ENG-4927: #1040: Fix MiniMaster startup race condition
[be0908b] [YSQL] #955: Handle primary key as an index
[4b48e25] ENG-4872: #1000: Parallel DNS resolution while filling system.peers and system.local tables
[4096f59] ENG-4900: Add external mini cluster test with TLS support
[34ee0fc] [YSQL] #1083: Handle sys catalog index updates
[ac52116] #993: [YSQL] Add support for CYCLE option in CREATE SEQUENCE
[3149f36] ENG-4363, #668: Support JSON operator on INSERT (Part 1 - Refactoring)
[a3ea483]
[b3a60ee] [yb-docker-ctl] Save initdb log (#1087)
[b131fbc] ENG-4888: #1031: Make YCQL index name optional
[9d333f7] #1064: Fix data race with YQL permissions cache
[4a56fe1] #1038: Change FLAGS_remote_boostrap_rate_limit_bytes_per_sec to FLAGS_remote_bootstrap_rate_limit_bytes_per_sec
[56a6029] #752: Add TLS encryption support to yb-admin.
[d9b6385] [YSQL] #1066: Fix memory leak in ResourceOwner
[fb4c1a2] Added yb-docker-ctl to CE (#1077)
[046b4b6] ENG-4855: #977: Handle restart read in YSQL
[30725bf] ENG-4343: #655: Pick transaction read time at tablet server when possible
[0341b29] [YSQL] #1067: Buffer create index write operations only for initdb
[78ca75a] [#1050]: Add Tablet Server capabilities
[9d724d5] ENG-4790: Removing most of yb-loadtester from repo due to migration to standalone yb-sample-apps repo
[bbcc67b] ENG-4891: Removing the SQL RPCs button.
[14279cc] [YCQL] #1042: Fix ASAN issue in YCQL executor restart
[cc3584f] ENG-4800: #952: Use call buffer for receive from socket
[8715be3] [YSQL] #1051: Bulk insert in initdb
[987abc6] ENG-4877 (#1009): yb-ctl enhancements
[0f261a9] [YSQL] 1049 ASAN reported memory leaks in Postgres
[9eaf15b] Add a new pattern for NFS directories: /Volumes/net
[3a61698] [#1041]: Skip dumping of finished call
[a322d95] [YSQL] #711 Add support for all datatypes
[7e2c89c] [#1030]: Add transactional retry writes tests 2) Second
[request]licated. Fixed by returning AlreadyPresent status and handling it in
[callbac] not handled correctly. For instance when request cannot be processed because of service shutdown, we could stop retrying to apply the operation. Fixed by correct handling of such errors.
[509c5a2] #921: Fixed: [YCQL] improve error messages related to index DDL
[e48b7da] ENG-4468: Always perform only one server rpc call for readiness check.
[0f8fdd9] [YSQL] ENG-4831 (#969): Enable drop cascade
[85836b5] Allow sp...

Assets 2

13 Mar 16:33

rkarthik007

v1.2.0

9093416

v1.2.0 (Mar 12, 2019)

Overview of Changes

Theme of this release: PostgreSQL, PostgreSQL, PostgreSQL! YSQL is a distributed PostgreSQL that can scale-out, tolerate failures and has no single point of failure.

Support has been added for a ton of Postgres functionality. The following is a list of some of the functionality:

DDL statements

CREATE, DROP and TRUNCATE tables

Data types

All primitive types including numeric types (integers and floats), text data types, byte arrays, date-time types
Support for UUID and SERIAL
Support for JSONB

DML statements

INSERT, UPDATE, SELECT and DELETE statements
JOIN, WHERE clauses, GROUP BY, ORDER BY, LIMIT, OFFSET and SEQUENCES

Transactions

ABORT, ROLLBACK, BEGIN, END, and COMMIT

Miscellaneous

Expressions: Built-in functions and operators
VIEWs
EXPLAIN PLAN
PREPARE-BIND-EXECUTE
JDBC Support

Key Commits

[aef1a1f] ENG-4683 (#874): Fix user table UI to exclude transactions and sequences tables
[c60921f] ENG-4681 [YSQL] Support INTERVAL for date/time datatypes
[4927d38] [YSQL] ENG-4807 (#957) Fix logs wrongly reporting that YugaByte transactions are not enabled
[4e69172] ENG-4780 (#930): Fix postgres server crash while creating index on existing table with data
[f004732] #972: ENG-4808: Fixed TabletInvoker::SelectTabletServer
[652bb5e] [#959]: Groups threads by stack trace in /threadz
[d33eaa9] #665: [YSQL] Enable support for sequences
[53a1331] ENG-4801 (#944): yb-ctl improvements for easier onboarding
[dce09da] ENG-4585: #796: Created test that covers case when thread pool is not yet ready for intents cleanup
[bb40833] ENG-4468: (#871) Add support for checking if tserver is ready for IO requests.
[e570c1c] [ENG-3928] [#967] Wait for NFS-based paths to exist in run_tests_on_spark.py
[3399b88] [YSQL] #255: Separate YCQL, YSQL and YEDIS namespaces
[064d366] ENG-4749: #887: Handle failure of delete at start of iteration in PgLibPqTest.SerializableColoring test
[88bb343] #961 Fix instantiation of round-robin policy
[1037c47] ENG-4577: Log Replay Count Bugfix and Test #782
[57ca114] ENG-4725: #932: Write RPC should take into account retryable_rpc_single_call_timeout_ms
[e9424ab] [YSQL] #943 Optimize single-row write transactions to avoid distributed txn path.
[c545fb5] ENG-4783: Misc fixes in shared lock manager
[45bc122] ENG-4783: Fixed LockBatch move ctor/assignment
[3db5d55] Cleanup of stray processes, NFS issues workaround, DXR integration, adjusting RPC timeout and using random PG ports in YSQL
[36431db] Update PostgreSQL code as of the following commit:
[3952394] [YSQL] ENG-4757: #902: Restart transaction when it was first executed operation and read restart is required
[d563e50] ENG-4577: Log Replay Count Bugfix and Test #782
[1ed9e95] ENG-3558: #920: Replace our own implementation of VarInt with BIGNUM from OpenSSL
[2f6dedc] [#836][ENG-4638] Fix KEYS to not Surface Deleted Keys
[7cf6ccd] [#918]: Add a pool of pre-created transaction records to avoid a Raft round-trip for transaction creation on the write path
[7b59cc8] #897: Fixed yb-tserver failure on SELECT COUNT(column) where column contains null value
[b0314eb] [YCQL] #916 Making STATIC an unreserved-keyword. Removing REFERENCES from keyword list.
[c886b86] [YSQL] #917: Support DROP INDEX
[81a80ab] [YCQL] (#878) : Add uuid() builtin function support.
[99cc736] ENG-4672: Provide more information (version,
[a101e74] ENG-4656 (#852): Enabled pg_depend
[c839f8c] Avoid using NFS-shared local Maven repository directory for running Java tests on macOS
[396ff88] [#914]: Split doc_operation into multiple files
[78776a5] [YCQL] #835 Allow OFFSET to be used as column names and identifiers
[362502d] ENG-4636 (#834): Make transaction failure messages more user friendly
[a746743] #904: ENG-4735: Fixed race between deleting obsolete files and creating new output SST file
[845cdb4] ENG-4717: #891: Not submitted write operation should block tablet peer shutdown
[1125d1e] #771 [YEDIS] Close connection on pub/sub limit
[ae1e30c] #768 Ping in subscribe mode should return a multi-bulk string
[dd97847] [YSQL] ENG-4734 Fix data race with ysql_catalog_version #903
[08c9bbf] ENG-4410 (#696): Retry resolving master addresses instead of erroring out immediately
[3ce7386] ENG-4680 (#873): Save postgres logs in tserver/logs directory
[1dd8e4b] ENG-4692 (#868): Enable access share mode for table lock
[63a0d61] ENG-4692 (#868): Enable support for read only transactions
[3779a86] #862: ENG-4728: Use libbacktrace only in debug builds and in single-threaded mode to avoid high memory consumption
[827ff02] [YSQL] Basic framework for cache consistency with multi-head DDLs #679
[299c718] Add pre- for leader election logging
[bfd2019] #895: Rate-limit the warning of unsupported YCQL consistency level.
[ff38980] Break the long test TestPgRegressPgTypes into two separate tests
[492c67c] Bump up master memory limit threshold for ASAN and TSAN
[[abc6a2e](https://github.com/YugaByte/yugabyte-db/commit/abc6a2ead184c764a7732cfb2d...

Assets 4

19 Sep 23:35

ravimurthy

v1.1.0

0eb062a

v1.1.0 (Sep 17, 2018)