The benchmark became a debugging project
I wanted to see how NVIDIA Sirius, a GPU-native SQL engine, compared with DuckDB on the Join Order Benchmark: 113 queries over more than 74 million rows of IMDB data.
Before I could measure speed, the workload exposed two correctness problems. Some queries crashed when a join returned no rows. Others computed string aggregates inside cuDF but could not return the result through Sirius. The useful question changed from “How fast is it?” to “What is stopping the real workload from finishing?”
I followed each failure across the GPU boundary
The first bug came from launching a CUDA conversion kernel for an empty result. I added an early return for zero-row inputs so an empty join behaved like a normal database result instead of an invalid GPU launch.
The second bug appeared when Sirius tried to turn a cuDF string scalar into its own column format. I added the missing materialization path for the string bytes, offsets, and metadata. That allowed string MIN and MAX aggregates to move cleanly through the engine.
Both fixes were merged upstream into Sirius.
Then I could measure the result honestly
Once the full workload ran correctly, I compared Sirius with DuckDB and checked that their answers matched. With the data already resident on the GPU, Sirius averaged roughly a 6× speedup across the benchmark.
The project changed how I think about performance work. A benchmark is most useful when it tests the system end to end. In this case, the failures were not a distraction from the benchmark: they showed me exactly where the engine needed to become more complete.