Python Microsoft Fabric - Lakehouses
Fabric lakehouses - creating them, loading data into OneLake files and listing tables.
A lakehouse is Fabric's central data store - files and Delta tables in one item, backed by OneLake. Lakehouses are items inside a workspace, so the item API is how you create and manage them, while the OneLake data plane is how you move data in and out. You create a Fabric connection in the Dashboard and both are available to your services.
Listing lakehouses
conn.list_items with the Lakehouse type filter returns the lakehouses of a workspace.
# -*- coding: utf-8 -*-
# Zato
from zato.server.service import Service
class ListLakehouses(Service):
input = 'workspace_id'
def handle(self):
# Get the connection by its Dashboard name
conn = self.microsoft.fabric['My Fabric']
# List only the lakehouse items
response = conn.list_items(self.request.input.workspace_id, 'Lakehouse')
lakehouses = []
for item in response['value']:
lakehouses.append({
'id': item['id'],
'name': item['displayName'],
})
self.response.payload = {'lakehouses': lakehouses}
Creating a lakehouse
A lakehouse is created like any other item - with conn.create_item and the Lakehouse type.
# -*- coding: utf-8 -*-
# Zato
from zato.server.service import Service
class CreateSalesLakehouse(Service):
input = 'workspace_id'
def handle(self):
conn = self.microsoft.fabric['My Fabric']
lakehouse = conn.create_item(
self.request.input.workspace_id,
'Sales data',
'Lakehouse',
description='Sales data for analytics and reporting',
)
self.response.payload = {'lakehouse_id': lakehouse['id']}
Loading data into a lakehouse
Files land in the lakehouse's Files section through the OneLake data plane - conn.onelake_write takes the workspace, the path and the bytes to write. Once files are in place, a notebook or pipeline turns them into Delta tables.
# -*- coding: utf-8 -*-
# stdlib
import csv
import io
# Zato
from zato.server.service import Service
class LoadDailySales(Service):
input = 'workspace_id', 'lakehouse_name'
def handle(self):
conn = self.microsoft.fabric['My Fabric']
# Build a CSV file out of today's sales
buffer = io.StringIO()
writer = csv.writer(buffer)
writer.writerow(['order_id', 'amount'])
writer.writerow(['ORD-001', '250.00'])
writer.writerow(['ORD-002', '99.90'])
data = buffer.getvalue().encode('utf-8')
# Write it to the lakehouse's Files section
path = '{}.Lakehouse/Files/sales/daily.csv'.format(self.request.input.lakehouse_name)
conn.onelake_write(self.request.input.workspace_id, path, data)
self.response.payload = {'status': 'loaded', 'bytes_written': len(data)}
Listing tables
The lakehouse table listing endpoint is reached through the generic conn.get method, giving you each table's name and format.
# -*- coding: utf-8 -*-
# Zato
from zato.server.service import Service
class ListLakehouseTables(Service):
input = 'workspace_id', 'lakehouse_id'
def handle(self):
conn = self.microsoft.fabric['My Fabric']
# List the tables of a lakehouse
path = '/workspaces/{}/lakehouses/{}/tables'.format(
self.request.input.workspace_id, self.request.input.lakehouse_id)
response = conn.get(path)
tables = [table['name'] for table in response['data']]
self.response.payload = {'tables': tables}
Querying lakehouse data with SQL
Each lakehouse has a SQL analytics endpoint that speaks the same protocol as SQL Server, so querying tables with SQL goes through Zato's regular SQL connections rather than through this connection - create an outgoing SQL connection pointing at the lakehouse's SQL endpoint and use self.out.sql as with any other database.