🧠 Second Brain

Ibis (Python)

Last updated Feb 16, 2025

Ibis is a Python library that provides a lightweight, universal interface for data wrangling. It helps Python users explore and transform data of any size, stored anywhere.

Ibis has three primary components:

A dataframe API for Python. Python users can write Ibis code to manipulate tabular data.
Interfaces to 15+ query engines. Wherever data is stored, people can use Ibis as their API of choice to communicate with any of those query engines.
Deferred execution. Ibis uses deferred execution, so execution of code is pushed to the query engine. Users can execute at the speed of their backend, not their local computer.

Why Use Ibis?

Ibis aims to be a future-proof solution to interacting with data using Python and can accomplish this goal through its main features:

Familiar API: Ibis’s API design borrows from popular APIs like pandas and dplyr that most users already know and like to use.
Consistent syntax: Ibis aims to be a universal Python API for tabular data of any size, big or small.
Deferred execution: Ibis pushes code execution to the query engine and only moves required data into memory when necessary. Analytics workflows are faster and more efficient
Interactive mode: Ibis provides an interactive mode in which users can quickly diagnose problems, explore data, and mock up workflows and pipelines locally.
10+ supported backends: Ibis supports multiple query engines and DataFrame APIs. Use one interface to transform with your data wherever it lives: from DataFrames in pandas to Parquet files through DuckDB to tables in BigQuery.
Minimize rewrites: Teams can often keep their Ibis code the same regardless of backend changes, like increasing or decreasing computing power, changing the number or size of their databases, or switching backends entirely.
Flexibility when you need it: When Ibis doesn’t support something, it provides a way to jump directly into SQL.

Upgraded to DuckDB
“default DuckDB backend, and DuckDB is much more performant”: Farewell pandas, and thanks for all the fish. – Ibis

Common Use Cases

Speed up prototype to production. Scale code written and tested locally to a distributed system or cloud SQL engine with minimal rewrites.
Boost performance of existing Python or pandas code. For example a general rule of thumb for pandas is “Have 5 to 10 times as much RAM as the size of your dataset”. When a dataset exceeds this rule using in-memory frameworks like pandas can be slow. Instead, using Ibis will significantly speed up your workflows because of its deferred execution. Ibis also empowers you to switch to a faster database engine, without changing much of your code.
Get rid of long, error-prone, f-strings. Ibis provides one syntax for multiple query engines and dataframe APIs that lets you avoid learning new flavors of SQL or other framework-specific code. Learn the syntax once and use that syntax anywhere.

Backends

Ibis acts as a universal frontend to the following systems:

The list of supported backends is continuously growing. Anyone can get involved in adding new ones! Learn more about contributing to ibis in our contributing documentation at https://github.com/ibis-project/ibis/blob/master/docs/CONTRIBUTING.md

# Installation

Install Ibis from PyPI with:

1

pip install 'ibis-framework[duckdb]'

Getting Started with Ibis

We provide a number of tutorial and example notebooks in the ibis-examples. The easiest way to try these out is through the online interactive notebook environment provided here:

You can also get started analyzing any dataset, anywhere with just a few lines of Ibis code. Here’s an example of how to use Ibis with a SQLite database.

Download the SQLite database from the ibis-tutorial-data GCS (Google Cloud Storage) bucket, then connect to it using ibis.

1

curl -LsS -o geography.db 'https://storage.googleapis.com/ibis-tutorial-data/geography.db'

Connect to the database and show the available tables

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


>>> import ibis
>>> from ibis import _
>>> ibis.options.interactive = True
>>> con = ibis.sqlite.connect("geography.db")
>>> con.tables
Tables
------
- countries
- gdp
- independence

Choose the countries table and preview its first few rows

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


>>> countries = con.tables.countries
>>> countries.head()
┏━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ iso_alpha2 ┃ iso_alpha3 ┃ iso_numeric ┃ fips   ┃ name                 ┃ capital          ┃ area_km2 ┃ population ┃ continent ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ string     │ string     │ int32       │ string │ string               │ string           │ float64  │ int32      │ string    │
├────────────┼────────────┼─────────────┼────────┼──────────────────────┼──────────────────┼──────────┼────────────┼───────────┤
│ AD         │ AND        │          20 │ AN     │ Andorra              │ Andorra la Vella │    468.0 │      84000 │ EU        │
│ AE         │ ARE        │         784 │ AE     │ United Arab Emirates │ Abu Dhabi        │  82880.0 │    4975593 │ AS        │
│ AF         │ AFG        │           4 │ AF     │ Afghanistan          │ Kabul            │ 647500.0 │   29121286 │ AS        │
│ AG         │ ATG        │          28 │ AC     │ Antigua and Barbuda  │ St. Johns        │    443.0 │      86754 │ NA        │
│ AI         │ AIA        │         660 │ AV     │ Anguilla             │ The Valley       │    102.0 │      13254 │ NA        │
└────────────┴────────────┴─────────────┴────────┴──────────────────────┴──────────────────┴──────────┴────────────┴───────────┘

Show the 5 least populous countries in Asia

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


>>> (
...     countries.filter(_.continent == "AS")
...     .select("name", "population")
...     .order_by(_.population)
...     .limit(5)
... )
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ name                           ┃ population ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ string                         │ int32      │
├────────────────────────────────┼────────────┤
│ Cocos [Keeling] Islands        │        628 │
│ British Indian Ocean Territory │       4000 │
│ Brunei                         │     395027 │
│ Maldives                       │     395650 │
│ Macao                          │     449198 │
└────────────────────────────────┴────────────┘

Origin: Upcoming Data Engineering Tools : r/dataengineering
References:
Created 2023-09-25