By Peter Simpson, OneTick Product Owner, KX
OneTick 1.27 is our annual major release. It rolls up six minor releases shipped every two months over the past year, all of which have already run in production for more than 100 customers on OneTick Cloud. Now those improvements are ready for your own deployed environments. In this first post of a three-part series, we look at the headline: speed and storage.
OneTick 1.27 runs the same benchmark suite in about 31% of the time 1.26 needed when swapping from OneTick's proprietary storage format to Parquet, and stores the same data in roughly 20% less space. The gains come from one strategic bet: partitioned Parquet is now OneTick’s primary storage format.
We test every major release against roughly 160 benchmark queries. They cover the analytics our customers run every day: aggregations, joins, filters and merge queries across tick data (trades and quotes), one-minute bars and daily data. Each query runs on one, two and four cores, both cold (caches cleared, everything restarted) and warm.
|
Comparison (1 core) |
Total query time |
|
OneTick 1.26, OneTick archives |
9,100+ seconds |
|
OneTick 1.27, partitioned Parquet |
2,860 seconds |
That is roughly a threefold speedup on a single core. The gain narrows a little as cores are added (as some of the test queries are single core specific), but stays large. Queries that touch the fewest columns benefit most: a row count query now runs about 100 times faster than before.
For teams staying on traditional OneTick archives, performance between 1.26 and 1.27 is essentially unchanged. Both formats remain fully supported, but our development effort, and our own cloud, is moving to Parquet.
OneTick archives have traditionally been stored as row stores. Partitioned Parquet is a column store, so query cost tracks the number of columns you retrieve. Row counts that read no fields are fastest. Point-in-time queries that pull every column from a very wide schema can be slower than on archives, up to 13 times on two cores in our tests. The practical advice is simple: select only the fields you need, and Parquet rewards you.
We also cut how much data leaves the disk in the first place. 1.27 adds predicate pushdown and delta encoding for time fields, both aimed at fast queries even on slower network disks and object stores. We also leverage parquet page indexes, and add our own index across partitioned parquet output.
On our US equities consolidated tape, "US Comp," Parquet with Z Standard compression takes about 80% of the space of OneTick’s proprietary format with native compression plus Z Standard. Convert your archives and you can expect a footprint roughly 20% smaller, on top of the speed gains.
Instead of processing one row at a time, OneTick increasingly processes blocks of rows. Vector processing began in 1.24 and 1.25 and reached aggregations in 1.26. In 1.27 it extends to joins, presorting, merging across flows and symbols, numeric functions and CASE statements, with aggregations and sorting optimized a third time. The effect shows most clearly on Parquet.
Querying every symbol in a database used to take two queries: one to build a symbol list, another to fetch the data. 1.27 adds two special symbols that remove that step.
_ALL_DATA_ is built for tables with many symbols and few rows per symbol: static and reference data, day records, stat records and market phases. For high-volume data such as every quote, the standard methods remain the right choice.
At petabyte scale, storage tiering decides your bill. OneTick Cloud keeps more than 2.5 PB on S3: about 98% sits in the object store, with the remainder on network disk and locally attached SSDs. Until now, tiers were set per database.
With Parquet, 1.27 sets alternative locations per table and per catalog. That matters because tables differ hugely in size. A day of the US Composite holds about 46 GB of quotes, 10 GB of MBBO and 3.5 GB of trades, but only about 170 MB of catalog data. Check your entitlement logs for the tables users hit most, and put just those on fast storage.
You do not have to convert everything at once. Parquet and OneTick archive locations can sit side by side in a single database, for example Parquet for recent years and archives for older history. The DB Reloader tool, part of the standard product, converts archives to Parquet and lets you reconcile the two datasets and compare their sizes. Our recommendation: convert one database, reconcile, run your own queries, and measure.
Next in the series: how OneTick 1.27 makes your data easier to reach, through an MCP server for AI agents, richer SQL, and faster APIs.