Technical Resources: Language Interoperability in Data System Design · Voltron Data
See how standards enable interoperability between languages
Use Arrow and ADBC to start at the finish line and leave data conversions behind
Blog | ADBC for Database Developers
Arrow Database Connectivity (ADBC) gives users a single way to get Arrow data in and out of databases. It bypasses the need for time-intensive, row-to-columnar based data conversions or vendor-specific SDKs. Think of it like JDBC and ODBC, but for Arrow Data, with a strong focus on offering cross-language APIs to maximize productivity. Learn more about this standard and get access to resources to get started in this featured post.
>>> from adbc_driver_sqlite.dbapi import connect
>>> conn = connect()
>>> cur = conn.cursor()
>>> cur.execute("SELECT 1, 2.0, 'hi'")
>>> cur.fetch_arrow_table()
pyarrow.Table
1: int64
2.0: double
'hi': string
----
1: [[1]]
2.0: [[2]]
'hi': [["hi"]]
↳ Example using Python and SQLite. The Python API is based on Python’s DBAPI standard to make it easy to use.
Bridge Ibis and data engines with Substrait
Blog | Supercharge Code Portability
Substrait is a cross-language, interoperable specification for data compute operations. It connects data analysis tools, like Ibis and dpylr, with compute engines and hardware. It is a flexible way for APIs and compute engines to share the intermediate representations (IR) for analytical computations. In this featured post, we demonstrate how Substrait enables users to write Ibis expressions without targeting a specific backend – effectively increasing productivity through code portability.
from ibis_substrait.compiler.core
import SubstraitCompiler
compiler = SubstraitCompiler()
plan = compiler.compile(topfilms)
# type (plan) --> ‹class 'substrait.ibis.plan_pb2.Plan'>
↳ Import Ibis and compile into a Substrait plan.
Simplify database connectivity for both clients and vendors
Blog | Extend Flight SQL with Flight
Avoid converting between columnar and row-based data. Instead of building the networking layer from scratch, Flight SQL extends Flight — a fast, low-level protocol designed for transporting Arrow data — to avoid unnecessary data copies by hooking into frameworks like gRPC. Get a lay of the land and understand how these two standards work together to simplify database connectivity for both clients and vendors.
Flight SQL Documentation
Flight SQL Intro Blog
↳ Current State: Every client must support every vendor
↳ Vision: How Flight SQL connects the client API to a database
Dig into additional technical resources discussing how to optimize costs and increase flexibility for data system design.
Discover more on our blog and follow us on social.
Resources