iter(batch_size=…) yields PythonCall wrappers instead of "julia"-format batches #105

In the default "julia" format, rows (for row in itds) and slices (ds[1:2]) are converted with py2jl. The batched iterator ds.iter(; batch_size), the only way to stream an IterableDataset in batches, is forwarded to Python instead. Its batches come back in PythonCall's default conversion: a PyIterable of PyDicts holding PyArrays, with string columns as PyArray{Py} of Python str.

using HuggingFaceDatasets   # v0.4.3
ds = Dataset((; text=["a", "bb", "ccc"], n=Int32[1, 2, 3]))
b = first(ds.iter(; batch_size=2))
typeof(b)          # PythonCall.PyDict{Any, Any}
typeof(b["text"])  # PythonCall.PyArray{PythonCall.Py, 1, true, true, PythonCall.Wrap.UnsafePyObject}
typeof(ds[1:2])    # Dict{String, Vector}
first(ds.to_iterable_dataset().iter(; batch_size=2))  # the same PyDict

For streaming, batches are the fast path. On a local 400k-document TinyStories parquet, rows iterate at 26k rows/s in the "julia" format, while 10k-row batches converted with py2jl reach ~105k rows/s. Converting text batches is cheap: py2jl adds ~0.9 s per 100k FineWeb-Edu documents (482 MB of text). So with converted batches, a tokenization pipeline can work on Vector{String} batches and stay within 1–5% of keeping the text in Python (measured with the tokenizers GPT-2 tokenizer).

A possible fix: route iter through _method_override for Dataset and IterableDataset, so that it returns a Julia iterator applying the dataset's jltransform to each batch, e.g. Iterators.map(jltransform, py.iter(; kws...)). The current workaround is to call py2jl(b) on each batch.