ydb/ROADMAP.md

YDB Roadmap

Legend

We use the following symbols as abbreviations:

1. ㉓ - feature appeared in the Roadmap for 2023;
2. ㉔ - feature appeared in the Roadmap for 2024;
3. ✅ - feature has been released;
4. 🚧 - feature is partially available and is under development;
5. ❌ - feature has been refused;
6. 🔥 - not yet released, but we are in rush.

Query Processor

1. ㉔ Unique secondary indexes
2. ㉔ Apply indexes automatically to optimize data fetching
3. ㉔ Default values for table columns
4. ㉔ Asynchronous LLVM JIT query compilation
5. ㉔ Parameters in DECLARE clause are becoming optional, better SQL compatibility
6. 🚧㉔ Cost-based optimizer for join order and join algorithm selection
7. ㉔ INSERT INTO table FROM SELECT for large datasets
8. ㉔ Support for transactional writes into both row and column tables
9. ㉔ Support for computed columns in a table
10. ㉔ Support for temporary tables
11. ㉔ Support for VIEW SQL clause
12. ㉔ Data Spilling in case there is issufient amount of RAM
13. ㉔ TPC-H, TPC-DS for 100TB dataset
14. ✅ ㉓ Support for Snapshot Readonly transactions mode
15. 🚧 ㉓ Better resource management for KQP Resource Manager (share information about nodes resources, avoid OOMs)
16. ✅ ㉓ Switch to New Engine for OLTP queries
17. ✅ ㉓ Support not null for PK (primary key) table columns
18. ✅ ㉓ Aggregates and predicates push down to column-oriented tables
19. ✅ ㉓ Optimize data formats for data transition between query phases
20. ✅ ㉓ Index Rename/Rebuild
21. ✅ ㉓ KQP Session Actor as a replacement for KQP Worker Actor (optimize to reduce CPU usage)
22. PostgreSQL compatibility
    • ✅ ㉓ Support PostgreSQL datatypes serialization/deserialization in YDB Public API
    • 🚧 ㉓ PostgreSQL compatible query execution (TPC-C, TPC-H queries should work)
    • ✅ ㉓ Support for PostgreSQL wire protocol
23. ㉓ Support a single Database connection string instead of multiple parameters
24. ㉓ Support constraints in query optimizer
25. Query Processor 3.0 (a set of tasks to be more like traditional database in case of query execution functionality)
    • ㉓ Support for Streaming Lookup Join via MVCC snapshots (avoid distributed transactions, scalability is better)
    • ㉓ Universal API call for DML, DDL with unlimited results size for OLTP/OLAP workload (aka ExecuteQuery)
    • ✅ ㉓ Support for secondary indexes in ScanQuery
    • ✅ ㉓ Transaction can see its own updates (updates made during transaction execution are not buffered in RAM anymore, but rather are written to disk and available to read by this transaction)
26. ✅ ㉓ Computation graphs caching (compute/datashard programs) (optimize CPU usage)
27. 🚧 ㉓ RPC Deadline & Cancellation propagation (smooth timeout management)
28. ✅ ㉓ DDL for column-oriented tables

Database Core (Tablets, etc)

1. ㉔ Volatile transactions. YDB Distributed transactions 2.0, minimize network round trips in happy path
2. ㉔ Table statistics for cost-based optimizer
3. ㉔ Memory optimization for row tables (avoid full SST index loading, dynamic cache adjusting)
4. ㉔ Reduce minimum requirements for the number of cores to 2 for YDB node
5. ㉔ Incremental backup and Point-in-time recovery
6. ㉔ ALTER CHANGEFEED
7. ㉔ Async Replication between YDB databases (column tables, topics)
8. ㉔ Async Replication between YDB databases (schema changes)
9. ㉔ Support for Debezium format
10. ㉔ Topics autoscaling (increase/decrease number of partitions in the topic automatically)
11. ㉔ Extended Kafka API protocol to YDB Topics support (balance reads, support for v19)
12. ㉔ Schema for YDB Topics
13. ㉔ Message-level parallelism in YDB Topics
14. ✅ ㉓ Get YDB topics (aka pers queue, streams) ready for production
15. ✅ ㉓ Turn on MVCC support by default
16. ✅ ㉓ Enable Snapshot read mode by default (take and use MVCC snapshot for reads instead of running distributed transaction for reads)
17. ✅ ㉓ Change Data Capture (be able to get change feed of table updates)
18. 🔥 ㉓ Async Replication between YDB databases (first version, row tables, w/o schema changes)
19. ✅ ㉓ Background compaction for DataShards
20. ✅ ㉓ Compressed Backups. Add functionality to compress backup data
21. ㉓ Process of Extending State Storage without cluster downtime. If a cluster grows from, say, 9 nodes to 900 State Storage configuration stays the same (9 nodes), it leads to a performance bottleneck.
22. Split/Merge DataShards BY LOAD by default. Most users require this feature turned on by default
23. ✅ ㉓ Support PostgreSQL datatypes in tablet local database
24. Basic histogram for DataShards (first step towards cost based optimizations)
25. ✅ ㉓ Transaction can see its own updates (updates made during transaction execution are not buffered in RAM anymore, but rather are written to disk and available to read by this transaction)
26. ㉓ Data Ingestion from topic to table (implement built-in compatibility to ingest data to YDB tables from topics)
27. ㉓ Support snapshot read over read replicas (consistent reads against read replicas)
28. ㉓ 🚧 Transactions between topics and tables
29. ✅ ㉓ Support for Kafka API compatible protocol to YDB Topics

Hardcore or system wide

1. ㉔ Tracing capabilities
2. ㉔ Automatically balance tablet channels via BlobStorage groups
3. ✅ ㉓ Datashard iterator reads via MVCC
4. ❌ (refused) ㉓ Switch to TRope (or don't use TString/std::string directly, provide zero-copy data passing between components)
5. ㉓ Avoid Node Broker as SPF (NBS must work without Node Broker under emergency conditions)
6. ㉓ Subscriptions in SchemeBoard (optimize interaction with SchemeBoard via subsription to updates)

Security

1. ✅ ㉓ Basic LDAP Support
2. ㉔ Support for OpenID Connect
3. ㉔ Authentication via KeyCloack
4. ㉔ Support for SASL framework

BlobStorage

1. ㉔ BlobStorage latency optimization (p999), less CPU consumption
2. ㉔ ActorSystem performance optimizations
3. ㉔ Optimize ActorSystem for ARM processors
4. ㉔ Effortless initial cluster deployment (provide only nodes and disks description)
5. ㉔ Reduce number of BlobStorage groups for a database (add ability to remove unneeded groups)
6. ㉓ "One leg" storage migration without downtime (migrate 1/3 of the cluster from one AZ to another for mirror3-dc erasure encoding)
7. ✅ ㉓ ActorSystem 1.5 (dynamically reassign threads in different thread pools)
8. ✅ ㉓ Publish an utility for BlobStorage management (it's called ds_tool for now, improve it and open)
9. ㉓ Self-heal for degrated BlobStorage groups (automatic self-heal for groups with two broken disks, get VDisk Donors production ready)
10. ㉓ BlobDepot (a component for smooth blobs management between groups)
11. ㉓ Avoid BSC (BlobStorage Controller) as SPF (be able to run the cluster without BSC in emergency cases)
12. ㉓ BSC manages static group (reconfiguration of the static BlobStorage group must be done BlobStorage Controller as for any other group)
13. ㉓ (Semi-)Hard disk space separation (Better guarantees for disk space usage by VDisks on a single PDisk)
14. ㉓ Reduce space amplification (Optimize storage layer)
15. ✅ ㉓ Storage nodes decommission (Add ability to remove storage nodes)

Analytical Capabilities

1. ㉔ Backup for column tables
2. ㉔ Column tables autosharding
3. ㉓ 🚧 Log Store (log friendly column-oriented storage which allows to create 1+ million tables for logs storing)
4. ㉓ 🚧 Column-oriented Tables (introduce a Column-oriented tables in additon to Row-orinted tables)
5. ㉓ Tiered Storage for Column-oriented Tables (with the ability to store the data in S3)

Federated Query

1. ✅ ㉓ Run the first version

Embedded UI

Detailed roadmap could be found at YDB Embedded UI repo.

Command Line Utility

1. 🚧 ㉓ Use a single ydb yql instead of ydb table query or ydb scripting
2. ✅ ㉓ Interactive CLI

Tests and Benchmarks

1. ㉓ Built-in load test for DataShards in YCSB manner
2. ✅ ㉓ ydb workload for topics
3. Jepsen tests support

Experiments

1. ❌ (refused) Try RTMR-tablet for key-value workload