Positioning Theseus exclusively for 10TB+ workloads restricted our scope and shrank our addressable market. This was intentional. Theseus was built to make use of the latest and greatest software + hardware coming out of NVIDIA, and that just doesn’t exist on the cloud (GPUDirect Storage, NVLink, Infiniband, etc.).

Initially, our go-to-market strategy targeted the most demanding use cases and those that justified the cost of enterprise-class GPUs, leveraging state-of-the-art DGX/HGX architectures. But as GPUs proliferated across public clouds and prices dropped, customers kept asking for a cloud-native Theseus.

Until recently, we simply couldn’t hit the cost/performance thresholds needed to justify cloud deployments.

In this latest release, we’ve unlocked dramatic cloud performance gains, so we figured we should show our latest cloud performance from as low as 1GB to 30TB. The feedback from our benchmarking blog, which compares DuckDB to Theseus, was clear: “Theseus doesn’t perform well at small scale.” In the last few sprints, and by harnessing affordable NVIDIA T4 and L4 GPU instances on AWS, Theseus performs incredibly well at both small and large scales. Because Theseus is a distributed runtime, you can simply add more GPU resources to dial in the optimal balance of cost and performance.

Full view of the benchmark results

Benchmarking Setup and Methodology

Below are TPC-H benchmark results comparing DuckDB’s and Theseus' total runtime and cost to complete these jobs at various scale factors from AWS S3.

Data

• All queries for both engines are run off Parquet files stored in AWS S3. That data is never pulled or pre-loaded onto local drives.
• That data is not pre-sorted or prepped in any way. The order it comes out of the TPC-H dbgen is the order it is converted into Parquet files.

Compute

• DuckDB - We are using different sizes of r5d EC2 instances from .xlarge-.24xlarge.
• Theseus - We are using different node counts of g4dn and g6 EC2 instances, ranging from 1 node up to 32 nodes.

Queries

• We’re just running the TPC-H standard queries with no rewrites exactly the same on both engines.

If you read our benchmarking report, you’ll know we’re not fans of using a simple column chart to compare two systems (see our methodologies). It typically supports cherry-picking of results that benefit the publishing vendor. Unfortunately, it’s not always clean and easy to compare data systems fairly using the SPACE chart. For simplicity and brevity, we will use bar charts to compare Theseus with DuckDB and provide ALL OF THE DATA (even what we didn’t plot) used to build the charts.

Results: TPC-H SF1 (1GB)

We’re starting at a small scale factor because this is where we have rarely shown data. You can see below that DuckDB and Theseus effectively cost the same. We would recommend using DuckDB due to the simplicity of spinning it up and getting going.

AWS EC2 Instance

• DuckDB - 1 x r5d.xlarge ($0.29 per hour)
• Theseus - 1 x g4dn.xlarge ($0.53 per hour)

Results: TPC-H SF10 (10GB)

At 10 gigabytes, the divergence starts to get a little larger, where the performance might make sense, and the TCO is marginally better for Theseus, but very minimally.

AWS EC2 Instance

• DuckDB - 1 x r5d.xlarge ($0.29 per hour)
• Theseus - 1 x g4dn.xlarge ($0.53 per hour)

Results: TPC-H SF100 (100GB)

As we continue to scale, it makes sense to scale up the DuckDB and Theseus instances, which, as the raw data demonstrates, are not the cheapest runs, but the performance delta just makes them the most “reasonable” choice.

AWS EC2 Instance

• DuckDB - 1 x r5d.8xlarge ($2.02 per hour)
• Theseus - 4 x g4dn.xlarge ($2.10 per hour)

Results: TPC-H SF1000 (1TB)

As we often say, the more we scale, the more Theseus makes sense, where the difference in performance and costs begin to greatly exceed what is possible today with CPU-based engines.

AWS EC2 Instance

• DuckDB - 1 x r5d.8xlarge ($2.02 per hour)
• Theseus - 4 x g6.4xlarge ($5.29 per hour)

Results: TPC-H SF10,000 (10TB)

As expected, the performance differential on very large data sets becomes very high, where the TCO and the performance require a rethink of how these systems should be done. In this case, we could only scale up the DuckDB instance vertically with a larger server, but that has limits, while with Theseus, we can scale our distributed runtime to 16 x L4 GPUs and optimize performance and cost.

AWS EC2 Instance

• DuckDB - 1 x r5d.24xlarge ($6.91 per hour)
• Theseus - 16 x g6.4xlarge ($21.17 per hour)

Wrap-up

Contrary to our messaging for the past couple of years, Theseus performance scales seamlessly to fit your workload from 1 GB to 30 TB on cost-effective T4 and L4 AWS EC2 instances. We intend to post a lot more numbers soon, but figured we would share this as soon as we got the numbers ourselves.