The Composable Codex

The Wall & The Machine

The engine is the heart of any data system. But the heart is fickle. Optimizing system performance can feel like a game of whack-a-mole. The second you push forward on one performance area, another one becomes the bottleneck. Even if you make all the right moves, you will hit a performance wall with CPU-based systems.

Here is a hard truth: future data systems will have to jump the wall, sooner or later. They will do it by moving to machines that operate on accelerated hardware. To hit the next wave of hardware innovations, composable engines that can "run anywhere" will rely on software and hardware ecosystems that are built to evolve.

Sign-up to receive an email when future chapters drop.

Get Notified
In the pipeline

The Wall & The Machine

What to expect in this chapter :
  • Execution engine
  • Execution runtime

  • Hardware
  • Accelerated hardware
  • CPUs
  • GPUs
  • Optimization
  • Distributed execution
  • Serialization

  • Arrow memory format
  • Calcite
  • Velox