[alden-sift]

Summary

Flat dataset Parquet config detection fails for wide files (many columns and/or long column names) because the client sends the Parquet footer to the server's DetectConfig endpoint, and footers over the server's 20 MiB gRPC receive limit are rejected with RESOURCE_EXHAUSTED.

This PR detects PARQUET_FLATDATASET configs on the client from the footer schema with pyarrow, the same way HDF5, TDMS, ULog, and MCAP configs are detected. Single channel per row Parquet still uses server detection.

detect_config and import_from_path now also accept relative_start_time so files with relative time can be detected and imported without editing the config.

Also refactored by moving helper functions either into _internal module or moved them into ImportConfig as methods.

Dependency change: flat dataset Parquet detection now requires pyarrow. Install it with pip install sift-stack-py[parquet].

Validation

Recreation

detect_config on a Parquet file with 20,000 columns and long column names:

import os

import dotenv
import dotenv
from sift_client import SiftClient
from sift_client._internal.util.file import extract_parquet_footer
from sift_client.sift_types.data_import import DataTypeKey

dotenv.load_dotenv()
pq_file = "data/super-wide-and-big-name.parquet"
client = SiftClient(
    api_key=os.getenv("SIFT_API_KEY"),
    grpc_url=os.getenv("SIFT_GRPC_URL"),
    rest_url=os.getenv("SIFT_REST_URL"),
)
footer_bytes, _ = extract_parquet_footer(pq_file)
print(f"footer size: {len(footer_bytes)} bytes")
client.data_import.import_from_path(
    pq_file, asset="test-long-parquet-1", data_type=DataTypeKey.PARQUET_FLATDATASET
)

Before this change:

grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
        status = StatusCode.RESOURCE_EXHAUSTED
        details = "grpc: received message larger than max (29943806 vs. 20971520)"

After this change:

alden@MacBook-Pro eng-12274-test % /Users/alden/projects/eng-12274-test/.venv/bin/python /Users/alden/projects/eng-12274-test/main.py
size of footer bytes: 29943832
Upload [super-wide-and-big-name.parquet] |████████████████████████████████████████| 32.3MB/32.3MB [100%] in 0.7s (46.70MB/s) 

Behavior validation

• Automated test: ran the old (server side) and the new (client side) config detection on Parquet files covering every time type, scalar and complex types, structs, durations, and embedded configs to ensure identical behavior.
• Manual test: Parquet files imported through the client and through the webapp produced identical runs.
• Unit tests: _tests/_internal/test_parquet.py.
• Integration tests

[github-actions]

Python docs preview: https://sift-stack.github.io/sift/python/pr-819/

Deployed from dfdc9fb. The link may take up to a minute to become live as GitHub Pages propagates.