Repository navigation
String columns are copied through a fixed-width numpy <U{maxlen} array #79
Description
Under the "julia" format, string columns go through a fixed-width numpy unicode array before reaching Julia. datasets' numpy extractor calls np.asarray(list_of_str), which allocates a <U{maxlen} (UTF-32) array, and py2jl's non-numeric branch then converts it back with x.tolist() (src/transforms.jl:~70).
using HuggingFaceDatasets, HuggingFaceDatasets.PythonCall
np = pyimport("numpy")
feats = datasets.Features(pydict(Dict(
"u16" => datasets.List(datasets.Value("uint16")),
"f64" => datasets.Value("float64"),
"s" => datasets.Value("string"))))
pyds = datasets.Dataset.from_dict(pydict(Dict(
"u16" => pylist([np.array([1, 2], dtype=np.uint16), np.array([3, 4], dtype=np.uint16)]),
"f64" => pylist([1.0000000001, 2.0]),
"s" => pylist(["a", "b"^50]))); features=feats)
ds = with_format(Dataset(pyds), "julia")julia> ds.py.with_format("numpy")[pyslice(0, 2)]["s"].dtype
<U50The temporary takes batch_size × maxlen × 4 bytes. For long-document corpora such as HuggingFaceFW/fineweb-edu (documents up to ~100k characters), a batch of 1000 documents allocates hundreds of MB just to produce a Vector{String}, and every string is converted twice. Extracting string columns directly from Arrow, or with the Python formatter (to_pylist), would avoid this.
Environment: HuggingFaceDatasets v0.4.2 (the relevant code is unchanged on master), Python datasets 4.8.5, Julia 1.13.2.
Activity
CarloLucibello commentedon Oct 19, 2026
This also applies to streaming datasets, where the impact is much larger.
load_dataset(...; streaming=true)sets the same numpy-backed format on theIterableDataset, so.iter(batch_size=...)yields each text column as one fixed-width<U{maxlen}array:One 485k-character document sets the width for the whole batch. Tokenizing FineWeb-Edu this way grew the process to more than 60 GB of RSS after 1B tokens, and ran about 6× slower than with plain lists (
ds.py.with_format(nothing)), which give a 10k-document batch in ~1.8 s.Read string columns without a fixed-width numpy array
Read string columns without a fixed-width numpy array (#87)