Skip to content

DataFrame and Arrow Data Bridge

Edition: Commercial

wolfxl.data is part of the Commercial edition (authenticated package index). Community (MIT, public PyPI, 2.0.9) does not include it. The release notes list this bridge under Unreleased. See Install for the difference between the editions.

wolfxl.data moves typed tabular data between Excel files and Python data systems through one native Arrow path. Read, scan, and write share a single option contract and a single set of type rules, so a value that survives a write is the value a later read returns.

Every optional dependency stays optional: importing wolfxl or wolfxl.data imports neither pandas, Polars, PyArrow, nor DuckDB, and no adapter imports one until it is called.

Entry points

Function Direction Result
scan_arrow(source, sheet) file or workbook content to Arrow one-shot Arrow C stream, batch at a time
read_arrow(source, sheet) file or workbook content to Arrow one pyarrow.RecordBatch
scan_polars(source, sheet) file or workbook content to Polars polars.LazyFrame, still lazy
read_polars(source, sheet) file or workbook content to Polars polars.DataFrame
write_arrow(data, path) Arrow to file one worksheet
write_arrow_sheets(sheets, path) Arrow to file ordered multi-sheet workbook
register_pandas_engine() pandas I/O enables read_excel and to_excel with engine="wolfxl"
import_pandas / import_polars / import_arrow / import_records / import_sql_cursor Python object to worksheet bounded DataWriteReport
export_pandas(worksheet) worksheet to pandas pandas.DataFrame
import_json(worksheet, document) JSON document to worksheet bounded DataWriteReport
range_to_json(worksheet) worksheet to canonical JSON text str
import_html(worksheet, document) selected HTML table to worksheet bounded DataWriteReport
load_workbook(source, encoding=..., delimiter=..., schema=...) CSV/TSV to workbook one materialized worksheet
load_workbook(source) SpreadsheetML 2003 XML to workbook materialized worksheets with scalar cells, common styles, merges, and translated formulas

scan_arrow, read_arrow, scan_polars, and read_polars accept the same read sources (below) and the same validated options: header, batch_rows, columns, start_row, max_rows, temporal, and strict. Invalid options are rejected before any workbook source is opened.

SpreadsheetML 2003 XML

Use load_workbook for a SpreadsheetML XML path, bytes, or binary stream. For output, use convert with ConversionOptions("spreadsheetml") or save to an .xml destination. Worksheet names, sparse cells, scalar values, common styles, and merged cells form the supported tier. Formula references are translated between SpreadsheetML R1C1 and workbook A1 notation; literal formula-looking text remains text.

These imports materialize the workbook. Streaming, in-place modification, encryption, and binary VBA retention are unsupported. Inspect conversion preservation records for dropped drawing, chart, and other unsupported artifacts; this format path does not imply rendering equivalence.

HTML and delimited text

Use import_html to select a table by zero-based index or HTML id. Set span_policy, styles, links, and images explicitly when the default merge, bounded-style, safe-link, and image-alt-text behavior is unsuitable. Use start_row and start_column for the destination origin. External resources are never fetched, nested tables default to refusal, and formula-like text remains literal unless explicitly requested otherwise.

Use load_workbook for CSV/TSV paths, bytes, or binary streams. Text is literal by default; schema opts individual columns into the shared scalar conversion rules. Specify encoding, a one-character delimiter, and ragged_rows="pad" or "reject" as needed. max_rows and max_cols can tighten Excel's grid limits. These imports materialize one worksheet; streaming and binary-package options are refused rather than ignored.

For HTML output, use ConversionOptions("html", sheet=..., cell_range=...) with the conversion API. Export covers a bounded worksheet rectangle, intersecting merged cells, supported CSS styles, and safe hyperlinks. Consult preservation records for dropped images, unsupported styles, and pagination boundaries; general browser or Excel render equivalence is not asserted.

Read sources

The first argument of every read entry point is either a filesystem path or workbook content:

Source Meaning
str or os.PathLike[str] path to an .xlsx file
bytes workbook content
bytearray workbook content, frozen first
contiguous memoryview workbook content, frozen first
seekable binary stream such as io.BytesIO workbook content, read from the current cursor
  • Bytes are content. A bytes value is never treated as a filesystem path, and a PathLike whose __fspath__ returns bytes is rejected with a message directing you to pass the workbook contents as bytes.
  • Streams. A stream's remaining content is read once, from its current cursor, when the read function is called; scanning then works from the frozen snapshot rather than the live stream. The cursor is restored on success and on failure, and a caller-owned stream is never closed. Text streams, nonseekable streams, and non-contiguous memoryviews are rejected with specific errors.
  • Freezing. bytearray, memoryview, and stream input become an immutable snapshot before the scanner or frame is returned, so later mutation or closure of the caller's object cannot corrupt a lazy scan or a repeated Polars collection. Immutable bytes are the workbook source itself.
  • Schema access. A scanner exports its frozen schema through __arrow_c_schema__ without consuming a data batch, so a consumer can inspect the schema and still take the stream afterwards.
  • Distributions. The read-only wolfxl-data package exposes scan_arrow, read_arrow, and scan_polars under this same source and option contract. Its optional extras are wolfxl-data[pyarrow] and wolfxl-data[polars], and it contains no write, calculation, render, or pivot capability. The full distribution adds read_polars, the Arrow writers, and the pandas engine.

Reading: type and null policy

Column types are frozen from a bounded inference window of the first 1,024 data rows in the requested range, then applied to every later row.

Worksheet content Arrow type
logical TRUE/FALSE Boolean
numbers, and serials under temporal="serial" Float64
text, and any column mixing kinds inside the window Utf8
date-formatted serials under temporal="native" Date32
time-formatted serials under temporal="native" Time64[us]
date-and-time serials under temporal="native" Timestamp[us], no time zone
  • Nulls. Every field is nullable. Blank and missing cells become null. A row that is entirely blank inside the requested range is still emitted, as nulls.
  • Formulas. Only the cached value of a formula cell contributes. Schema metadata records wolfxl.formula_mode = cached. WolfXL never recalculates during a scan.
  • Mixed columns. A column whose kinds disagree inside the inference window widens to Utf8, and its values are rendered as text. A value that first disagrees after the window is a late incompatible value: with strict=True (the default) it raises a data error naming the sheet, cell, and field; with strict=False it becomes null and the schema records wolfxl.loss_mode = null_on_incompatible.
  • Excel error cells (#REF!, #VALUE!, and siblings) have no Arrow scalar representation. They raise under strict=True and become null under strict=False.
  • Temporal metadata. Native temporal fields carry wolfxl.logical_type, wolfxl.number_format, and wolfxl.date_system (1900 or 1904). Sheet-level metadata records the sheet name, first row and column, header mode, temporal mode, and date system.
  • Header and projection. header=True consumes the first read row as field names; header=False generates positional names. columns selects fields by name or 1-based Excel column index and suppresses decoding of unselected cells. start_row is applied before header removal, and max_rows bounds emitted data rows.
  • Lifetime. A scanner is one-shot. The first export or iteration takes ownership; a second export raises OSError.

Writing: type and null policy

Arrow type Worksheet result
Null blank cell
Boolean logical TRUE/FALSE
Int8-Int64, UInt8-UInt64 number
Float32, Float64 number
Utf8, LargeUtf8 text
Date32 serial with a date style
Time64[us] serial with a time style
Timestamp[us], no time zone serial with a date-and-time style
Decimal128, Decimal256 exact text at the declared scale
Dictionary with integer keys and Utf8/LargeUtf8 values decoded text
  • Nulls. A null in any column, including a null dictionary key, becomes a blank cell rather than an empty string or a zero.
  • Decimal. Every Arrow-valid Decimal128/Decimal256 scale is accepted and written as exact text, preserving every digit and the declared scale. WolfXL does not convert a decimal to a double, because a double cannot carry the exact value; a caller who wants arithmetic in Excel must cast the column to a floating-point type first and accept that rounding.
  • Dictionary. Only dictionaries with Arrow integer keys (i8-i64, u8-u64) and Utf8/LargeUtf8 values are decoded to their text values. Every other key or value combination is rejected as an unsupported type.
  • Integers. An integer whose magnitude exceeds 2^53 cannot be represented exactly by Excel's number model and is rejected, naming the column and row. Cast the column to Utf8 to preserve every digit.
  • Non-finite numbers. NaN and infinities are rejected, naming the column and row. Excel has no cell value for either.
  • Header. header=True writes field names into the first worksheet row and that row counts against the row limit. header=False writes data only.
  • Index. Arrow has no index concept, so the Arrow write path never invents one. import_pandas(..., index=True) is the only path that materializes a pandas index, and it writes index levels as leading columns whose unnamed levels use an empty label.
  • Batching. Batches are consumed one at a time and never collected first, so an unbounded reader is streamed. Every batch of one sheet must share one schema; a mismatch names the offending batch index. Each producer must yield at least one batch.

Limits

  • 1,048,576 rows and 16,384 columns per worksheet, enforced per sheet, with the header row counted when header=True.
  • The limit is checked before rows are appended, so an oversized stream fails without producing a partial file.
  • import_polars and import_arrow additionally accept a caller max_rows bound and reject a larger source instead of truncating it.

Unsupported write types

Binary, large binary, fixed-size binary, list, large list, struct, map, union, interval, duration, Date64, Time32, and any extension type are rejected with column "<name>" has unsupported Arrow type <type>; cast it to a supported scalar type before writing.

Temporal types outside the supported set are rejected specifically: only Time64[us] and Timestamp[us] without a time zone are accepted. A timezone-aware timestamp is rejected rather than silently converted, because Excel has no time-zone-aware cell type and any conversion would silently pick a zone on the caller's behalf.

Rejection always happens before the destination path is replaced. A failed write leaves both the source data and any existing target file unchanged.

Ordered multi-sheet writes

from wolfxl.data import write_arrow_sheets

write_arrow_sheets(
    [("Ledger", ledger_table), ("Labels", labels_table)],
    "pack.xlsx",
)

sheets is an ordered sequence of (sheet_name, data) pairs or an ordered mapping; worksheet order in the published workbook is the given order. Each data is any object exposing __arrow_c_stream__ or __arrow_c_array__.

The wrapper validates every sheet name (type, length, Excel's forbidden characters) and every producer's Arrow protocol, and rejects a name that repeats an earlier one under Excel's case-insensitive worksheet-name comparison, before the native writer is called at all. The native writer then preflights every sheet before it creates output and publishes the workbook atomically. Either the whole workbook appears, or nothing does.

JSON import and export

import_json writes one JSON document into an already-owned worksheet and range_to_json reads one rectangle back as canonical JSON text. Both are standard-library only: no third-party JSON dependency is imported, and both are plain functions on wolfxl.data with no Worksheet method.

import json

from wolfxl import Workbook
from wolfxl.data import import_json, range_to_json

document = [
    {"id": 7, "when": "2026-08-17", "meta": {"tags": ["a", "b"]}, "note": "=1+1"},
    {"id": 8, "when": "2026-08-18", "meta": {"tags": []}, "note": "#REF!"},
]
schema = {"when": "date"}

workbook = Workbook()
import_json(workbook.active, document, schema=schema, strict=True)
workbook.save("json-round-trip.xlsx")

assert json.loads(range_to_json(workbook.active, nested=True)) == [
    {"id": 7, "when": "2026-08-17", "meta": {"tags": ["a", "b"]}, "note": "=1+1"},
    {"id": 8, "when": "2026-08-18", "meta": {"tags": []}, "note": "#REF!"},
]

Input and shapes

document is JSON text, UTF-8 bytes, bytearray, contiguous memoryview, or an already-decoded JSON value. Decoding refuses duplicate object keys and the non-standard constants NaN, Infinity, and -Infinity, because a first-wins key collapse and a non-finite number both have no honest cell representation.

Document Shape Result
array of objects, or one object records one row per object, keys as headers
array of arrays arrays header=True consumes the first row as the header
object whose values are all arrays columns one row per index across the columns
any other top-level array refused pass shape= to state the intent

shape="records", "arrays", or "columns" pins the reading; a top-level object that only looks columnar can then still be read as one record with array fields. header=False writes data only and labels unlabelled columns column_<n> by 1-based worksheet column. Record keys, header labels, and columns= labels must all be unique non-blank strings, matching range_to_records. With strict=True, every record must carry exactly the resolved key set; without it, unknown keys are ignored and absent keys become blank cells.

Schema and type policy

schema maps column names to declared types from the same native scalar vocabulary the template data layer uses, with the same optional: prefix: any, null, bool, string (str), int, number (float), decimal, date, time, and datetime. Without a declaration a value keeps its JSON kind, so nothing is inferred from text and an identifier such as "0012" stays the string "0012". Declared types are the only conversions performed:

Declared Accepted JSON Worksheet result
any any JSON value its own kind
null null blank cell
bool true/false, or those exact strings logical value
string string text
int integer, or a canonical integer string number
number, float integer, finite number, or a canonical number string number
decimal integer, or a canonical decimal string exact text at the declared scale
date, time, datetime ISO-8601 string Excel temporal value
optional:<type> any of the above, or null blank cell for null
  • Exactness. An integer whose magnitude exceeds 2^53 is refused by name rather than rounded; declare the column decimal or string to keep every digit. A decimal column is written as exact text and never converted through a double, matching the Arrow write policy, and a JSON number is refused for a decimal column because the digits are already gone by then.
  • Canonical literals. String conversions accept exact literals only: 123, -7, 1.50, 1e3, true, false. "007", "1.5" for int, and "yes" for bool are refused with the column location instead of being guessed.
  • Temporals. ISO-8601 text is required. Excel serial numbers are refused because the workbook's 1900 or 1904 date system would silently decide the value, and a timezone-aware datetime or time is refused because Excel has no time-zone-aware cell type. A time is stored as its canonical ISO-8601 text, because the scalar tabular writer has no time-only cell; the native Arrow writer's Time64[us] support is a separate path.
  • Unstorable types. binary and image are refused by name: Excel has no such cell.
  • Absent versus null. A record key that is absent writes a blank cell and never consults the schema, while an explicit null is written only when the declared type is any, null, or optional:. strict=True refuses absent keys outright, so the two cases stay distinguishable even though a cell can only store the blank.

Nested data

nested="json" (the default) stores a nested object or array as canonical compact JSON text: keys sorted, no spaces, ASCII-escaped, for example {"a":null,"z":1}. Every valid string is preserved, including a string that is byte-identical to that encoding.

range_to_json(nested=False) is the default and returns every cell as text, so nothing is ever guessed from cell content. range_to_json(nested=True) is an explicit opt-in that decodes a cell back into an object or array when, and only when, its text is byte-identical to the canonical encoding. That opt-in cannot distinguish a stored object or array from a caller string that is identical to the canonical encoding, which is exactly why it is opt-in: export a document that carries such text with the default nested=False. nested="error" refuses every nested value instead, so no text is reinterpreted on either side.

Text and formulas

formula_text="literal" (the default) stores text as text. A value beginning with = becomes neither a formula nor a cached formula, and text equal to an Excel error literal such as #REF! becomes neither an error cell: both are written with an explicit string type, on the type-preserving cell path. formula_text="formula" restores the platform typing, so "=SUM(A1:A2)" is written as a formula and can be recovered from a sheet with range_to_json.

Export

range_to_json emits keys in worksheet column order and always emits every key, so a blank cell is null and a key is never omitted. Values use their JSON-native kind, dates, times, and datetimes become ISO-8601 strings, an int becomes a JSON number, and a Decimal becomes its exact decimal text.

range_to_json(..., schema=...) accepts the same declared-type vocabulary and optional: prefix as import_json, and requires header=True. It is how a document is exported in the exact shape that reimports unchanged: a date cell reopens from a saved workbook as a datetime at midnight, and schema={"when": "date"} emits "2026-08-17" for it, while a cell that carries a time of day is refused rather than truncated, because dropping that time would not be reversible. A date is never widened to midnight on the way in. decimal emits exact decimal text, int, number, bool, and string validate the cell kind, and optional: admits a blank as null. A non-finite float, a duration, rich text, an image, or a binary value is refused with the offending A1 coordinate instead of being converted lossily. Exporting with header=False returns the rectangle as an array of value arrays, which is the exact inverse of an arrays import with the same header.

Refusal boundaries

Every shape, schema, type, nesting, text, and grid-limit rule is validated before the first cell is written, so a refused document leaves the worksheet, its dirty set, and its write queue untouched. Cell text longer than Excel's 32,767-character limit, an OOXML-illegal control character, and a lone surrogate are refused by location rather than truncated. Rows whose width disagrees with the resolved header, column arrays of differing length, unknown schema columns, and int values outside Excel's exact range are all refused by name.

SQL cursor streaming and import control

import_sql_cursor streams tabular data directly from a caller-owned DB-API 2.0 cursor into an existing worksheet in bounded batches:

import sqlite3

from wolfxl import Workbook
from wolfxl.data import SqlImportCancelled, import_sql_cursor

connection = sqlite3.connect("reporting.db")
cursor = connection.cursor()
cursor.execute("SELECT id, name, revenue FROM quarterly_sales")

wb = Workbook()
ws = wb.active

report = import_sql_cursor(
    ws,
    cursor,
    chunk_size=5000,
    on_progress=lambda r: print(f"Committed {r.rows_written} rows..."),
)
print(f"Imported {report.rows_written} rows across {len(report.columns)} columns.")

Streaming and caller ownership

  • Chunked consumption. Rows are consumed and advanced strictly via cursor.fetchmany(chunk_size). WolfXL never calls fetchall().
  • Caller ownership. The cursor and its connection remain owned by the caller. WolfXL never calls execute, commit, rollback, or close on the cursor or connection, even on error or cancellation.
  • Immediate batch commitment. Each batch returned by fetchmany is validated and committed directly to the worksheet using bounded native row buffers. Memory consumption remains strictly bounded by chunk_size.
  • Column metadata. By default, column names are extracted from cursor.description. Alternatively, pass columns=("col_a", "col_b", ...) to supply explicit column names without reading cursor.description.

Progress and cancellation

  • on_progress(report): Invoked after each successful batch of rows is committed to the worksheet with an immutable cumulative DataWriteReport. The report reflects actually committed rows (report.rows_written), not buffered or pre-fetched rows.
  • should_cancel(): Optional callable checked before initial mutation or cursor fetch, and before each subsequent fetchmany call. When it returns a truthy value, streaming halts immediately without consuming another cursor batch, and SqlImportCancelled is raised.
  • SqlImportCancelled: Exception raised on cancellation. Its .report attribute carries the exact, immutable DataWriteReport reflecting the partial state committed to the worksheet. If cancelled before initial mutation or fetch, report.rows_written == 0 and report.header_written is False.
  • Pre-flight validation. All arguments (worksheet, cursor, start_row, start_col, chunk_size, header, columns, on_progress, should_cancel) and Excel grid boundaries are validated before any worksheet mutation or cursor fetch.
  • Failure policy. Exceptions raised by on_progress or should_cancel propagate immediately without suppression or rollback. Partial writes remain committed to the worksheet as documented.

pandas

import io

import pandas as pd
import wolfxl.data

wolfxl.data.register_pandas_engine()

frame = pd.read_excel("book.xlsx", engine="wolfxl")                 # from a path
frame = pd.read_excel(io.BytesIO(workbook_bytes), engine="wolfxl")  # from memory

Registration is explicit and registers the paired reader and writer together. Importing WolfXL never changes how pandas.read_excel or DataFrame.to_excel behaves, repeated registration is a no-op, and a foreign reader or writer already registered under the name wolfxl is reported rather than replaced; if either side collides, neither registry is changed. The paired protocol is supported for pandas >=2.2,<3.1 and was pinned against pandas 3.0.5; a release outside that range fails before either registry is touched. pandas is imported only by the registration call; a missing pandas raises ImportError naming the install command.

Engine boundaries:

  • The reader accepts local .xlsx paths and the same in-memory sources as scan_arrow: workbook bytes, bytearray, contiguous memoryview, and seekable binary streams. A stream is read from its current cursor, restored afterwards, and never closed, and the source is frozen, so the caller may close or mutate it right after the call. pd.ExcelFile sheet enumeration works from memory the same way. storage_options is rejected: sources are local or in-memory only.
  • PyArrow is required at read time, and is imported only then. Worksheet values reach pandas one decoded Arrow column at a time; no cell is read through a Python worksheet object.
  • The first worksheet row is always consumed as Arrow field names, so pandas header=0 (the default) behaves as expected, and header=None sees that row as text.
  • nrows is forwarded as a native row bound, so a bounded read does not decode the whole sheet.
  • engine_kwargs accepts only temporal, strict, and batch_rows, forwarded to the scan. Any other key is rejected by name.
  • Read-side null, temporal, mixed-column, and error-cell policy is the policy documented above: pandas receives None for nulls and the same strict-mode failures.

The same registration adds a bounded writer for new .xlsx workbooks:

import pandas as pd
import wolfxl.data

wolfxl.data.register_pandas_engine()

frame.to_excel("ledger.xlsx", engine="wolfxl", freeze_panes=(1, 1))
with pd.ExcelWriter("pack.xlsx", engine="wolfxl") as writer:
    frame.to_excel(writer, sheet_name="Ledger")
    labels.to_excel(writer, sheet_name="Labels")

The writer covers pandas' ordinary DataFrame.to_excel / ExcelWriter cells: single or multiple worksheets, headers and index levels (including pandas' emitted header merges), startrow / startcol offsets, pandas' default missing-value output, scalar dates and datetimes with pandas' number formats, the standard pandas cell styles (font, fill, border, alignment, number format, protection), and freeze_panes.

Writer boundaries:

  • Append and update are unsupported: mode="a" and if_sheet_exists fail visibly before the destination is touched; the writer produces new workbooks (mode="w" / "wb") only.
  • Arbitrary Styler styles, and style dictionaries outside the standard pandas cell-style set, fail visibly; unsupported behavior is never delegated to another writer.
  • storage_options, writer engine_kwargs, and pandas autofilters are rejected.
  • A destination is a .xlsx filesystem path (str or os.PathLike[str]) or a writable binary stream. Bytes-like values are never interpreted as paths and are rejected as destinations, as are text streams.
  • A path destination is staged beside the target and published atomically only after the workbook saves; a failed or aborted write leaves any existing destination file unchanged and removes the staging file. A stream destination is written only after staging succeeds and is never closed by the engine.

export_pandas(worksheet) remains the in-memory counterpart for a worksheet that is already open in a Workbook; the engine covers whole workbooks, from a path or from memory.

Polars

from wolfxl.data import read_polars, scan_polars

lazy = scan_polars("book.xlsx", "Data")       # stays lazy until .collect()
frame = read_polars("book.xlsx", "Data")      # materialized on request

Both functions accept the same read sources as scan_arrow (a filesystem path or workbook content) under the same freezing rules, so a LazyFrame built from a stream or mutable buffer stays valid after the caller closes or overwrites it.

scan_polars registers the scan through the public polars.io.plugins.register_io_source hook; it does not register an invented pl.read_excel(engine="wolfxl"). Projection is forwarded to scan_arrow(columns=...), a pushed-down predicate is applied to each batch, and the LazyFrame decodes nothing until the consumer collects. read_polars is the eager sibling: Polars ingests the native stream through the Arrow PyCapsule interface in one call.

The declared schema is the scanner's frozen worksheet schema, taken through __arrow_c_schema__ without emitting a data batch, so empty and header-only sheets keep their schema. The frozen source also means one LazyFrame can be collected repeatedly. Rows removed by a pushed-down predicate do not consume a head or max_rows budget, a predicate may reference a column the projection drops, and filter, then select and limit, match the eager result. Conversion errors keep the sheet, field, and cell context of the read policy above.

Neither function imports pandas or PyArrow, and neither converts a value through a Python scalar. Polars is imported only when one of them is called; a missing Polars raises ImportError naming the install command, and a Polars release without register_io_source is reported instead of being patched around. The read-only wolfxl-data distribution exposes the same scan_polars behavior.

A Polars frame is also a valid write producer: it exposes __arrow_c_stream__, so write_arrow(frame, path) and write_arrow_sheets([("Sheet1", frame)], path) accept it directly under the write policy above.

DuckDB

import duckdb

from wolfxl.data import scan_arrow

connection = duckdb.connect()
rows = connection.from_arrow(scan_arrow("book.xlsx", "Data")).fetchall()
connection.close()

The scanner implements the Arrow PyCapsule stream protocol, so DuckDB's from_arrow consumes a worksheet directly under the same one-shot stream contract as any other consumer. WolfXL never imports DuckDB and adds no DuckDB extension or wrapper; DuckDB is simply another consumer of the stream.