Providing schema hint to ArrowReaderMetadata causes resulting schema to omit virtual fields

[limenilbuz]

Describe the bug

I recently had a use case where I had to supply both a schema hint and virtual columns (specifically row numbers) to ArrowReaderOptions, which was then used to construct an ArrowReaderMetadata.

The original code only asked for the virtual columns. When I changed the code to supply both virtual columns AND a schema hint, I observed some interesting behavior.

When only supplying virtual columns, the result of ArrowReaderMetadata::schema included the virtual column fields as supplied in ArrowReaderOptions::with_virtual_columns.

When supplying BOTH virtual columns and schema hint, the result of ArrowReaderMetadata::schema did NOT include the virtual column fields.

This caused issues for my downstream code that did some inspection on this schema to fail, since the column no longer exists.

Initially, I tried to work around this issue by including the virtual column IN the schema hint. This caused issues in ArrowReaderMetadata::with_supplied_schema since the number of fields in the schema did not match the number of fields in the Parquet file, with the error "Incompatible supplied Arrow schema: expected X columns received Y".

I eventually landed on a workaround specific to my use case that is not translatable to a generic solution, but this behavior still left a lot to be desired.

To Reproduce

let row_number_field = Field::new("row_number", DataType::Int64, false).with_extension_type(RowNumber)

let options = ArrowReaderOptions::new().with_virtual_fields(vec![row_number_field]);
// uncomment this line:
// let options = options.with_schema(some_schema)

let metadata = ArrowReaderMetadata::load(some_reader, options)?;
assert!(metadata.schema().fields.find("row_number").is_some())

Expected behavior

I'd expect one of two things to happen:

1. Include the virtual column in the resulting schema
2. Don't error when schema hint contains virtual columns

Additional context

I'd understand if this behavior is not considered a bug. After all, the library is simply executing the contract of with_schema, but it still led to some head scratching.

I appreciate the work done for this library, thanks all!