Spotlight

Multilanguage Tooling is Table Stakes

We support and maintain open standards built to improve language interoperability. Engineers can scale from local to distributed – all while using their language and tools of choice.

Developers, engineers, analysts, and data scientists can now code in SQL, Python, R, Java, Go, and more – seamlessly operating on the same data inside whatever engine they choose.

Use Arrow and ADBC to start at the finish line and leave data conversions behind

Blog | ADBC for Database Developers


Arrow Database Connectivity (ADBC) gives users a single way to get Arrow data in and out of databases. It bypasses the need for time-intensive, row-to-columnar based data conversions or vendor-specific SDKs. Think of it like JDBC and ODBC, but for Arrow Data, with a strong focus on offering cross-language APIs to maximize productivity. Learn more about this standard and get access to resources to get started in this featured post.

ADBC GitHub Repo
ADBC Documentation
ADBC Primer Blog
Engineering
Library
Technical Level 2/5
  
    >>> from adbc_driver_sqlite.dbapi import connect   
  >>> conn = connect()   
  >>> cur = conn.cursor()   
  >>> cur.execute("SELECT 1, 2.0, 'hi'")   
  >>> cur.fetch_arrow_table()   
  pyarrow.Table   
  1: int64   
  2.0: double   
  'hi': string   
  ----   
  1: [[1]]   
  2.0: [[2]]   
  'hi': [["hi"]]


↳ Example using Python and SQLite. The Python API is based on Python’s DBAPI standard to make it easy to use.

Bridge Ibis and data engines with Substrait

Blog | Supercharge Code Portability


Substrait is a cross-language, interoperable specification for data compute operations. It connects data analysis tools, like Ibis and dpylr, with compute engines and hardware. It is a flexible way for APIs and compute engines to share the intermediate representations (IR) for analytical computations. In this featured post, we demonstrate how Substrait enables users to write Ibis expressions without targeting a specific backend – effectively increasing productivity through code portability.

  
  from ibis_substrait.compiler.core 
import SubstraitCompiler   
  compiler = SubstraitCompiler()   
  plan = compiler.compile(topfilms)   
  # type (plan) --> ‹class 'substrait.ibis.plan_pb2.Plan'>



↳ Import Ibis and compile into a Substrait plan.

Engineering
Code
Technical Level 3/5

Simplify database connectivity for both clients and vendors

Blog | Extend Flight SQL with Flight


Avoid converting between columnar and row-based data. Instead of building the networking layer from scratch, Flight SQL extends Flight — a fast, low-level protocol designed for transporting Arrow data — to avoid unnecessary data copies by hooking into frameworks like gRPC. Get a lay of the land and understand how these two standards work together to simplify database connectivity for both clients and vendors.


↳ Current State: Every client must support every vendor




↳ Vision: How Flight SQL connects the client API to a database

Dig into additional technical resources discussing how to optimize costs and increase flexibility for data system design. 

Discover more on our blog and follow us on social.

Resources