Resolve DST-ambiguous and nonexistent local times when casting timestamps to a named timezone

[adriangb]

Which issue does this PR close?

• Closes Casting Timestamp(_, None) to a named timezone fails on DST-ambiguous and nonexistent local times #11037.

Rationale for this change

What a user hits today

Cast a naive timestamp to a named timezone. Pick a wall clock reading that the zone skips or repeats. The cast fails.

use arrow_array::TimestampSecondArray;
use arrow_cast::{cast_with_options, CastOptions};
use arrow_schema::{DataType, TimeUnit};

// 1_710_037_800 is 2024-03-10T02:30:00, held as a naive wall clock reading.
// America/New_York skips that hour: the clocks go forward at 02:00.
let array = TimestampSecondArray::from(vec![1_710_037_800]);
let to = DataType::Timestamp(TimeUnit::Second, Some("America/New_York".into()));

let strict = CastOptions { safe: false, ..Default::default() };
let lenient = CastOptions { safe: true, ..Default::default() };

cast_with_options(&array, &to, &strict);
cast_with_options(&array, &to, &lenient);

Today:

strict  -> Err("Cast error: Cannot cast timezone to different timezone")
lenient -> Ok, null_count = 1

After this PR:

strict  -> Ok, value = 1710055800   (2024-03-10T07:30:00Z, i.e. 03:30 EDT)
lenient -> Ok, null_count = 0

The safe: true result is the dangerous one. A whole column of DST-boundary readings turns to NULL, and nothing reports it.

How this reaches a SQL user

A DataFusion user reaches the same kernel through ordinary SQL. This is how the bug was found:

SELECT '2024-03-10 02:30:00'::timestamp AT TIME ZONE 'America/New_York';
-- Cast error: Cannot cast timezone to different timezone

The engine plans that as Timestamp(_, None) to Timestamp(_, Some("America/New_York")), so the error text above comes straight from this kernel. See apache/datafusion#25084. An upcoming type-coercion change inserts the same cast automatically for timestamptz - timestamp, so the failure gets more common, not less.

Why the kernel fails

adjust_timestamp_to_timezone reads each value as a wall clock time in the target zone. In an IANA timezone that reading is not always one instant:

• A "fall back" transition repeats an hour, so the reading is ambiguous.
• A "spring forward" transition skips an hour, so the reading is nonexistent.

The kernel resolved the offset with offset_from_local_datetime(..).single(). That is None for both cases, so both cases fail.

What changes are included in this PR?

A new private helper, resolve_local_offset, decides the two cases instead of rejecting them. It follows PostgreSQL and DuckDB:

• An ambiguous reading takes the later instant. This is the offset in effect after the transition. 2024-11-03T01:30:00 in America/New_York becomes 01:30-05:00 (EST).
• A nonexistent reading moves forward by the length of the gap. 2024-03-10T02:30:00 in America/New_York becomes 03:30-04:00 (EDT).

adjust_timestamp_to_timezone calls the helper in place of .single(). That is the whole behavioural change.

The forward shift needs the offset in effect before the transition. The code recovers it with one probe: it asks for the offset 24 hours earlier and takes the earliest result. offset_from_utc_datetime is not a valid shortcut, because it returns the post-transition offset. That is wrong for a southern hemisphere gap such as Australia/Sydney 2024-10-06T02:30:00.

Two properties of the timezone database make the probe safe:

• No two transitions in any of the 597 zones are closer than 167 hours. So the probe lands on the correct side of the transition. The smallest interval is America/Boa_Vista in 2000.
• No local gap is longer than 24 hours. So the probe lands outside the gap at all. Seven zones sit exactly at 24 hours — the dateline changes, such as Pacific/Apia in 2011. The next largest gap is 10 hours.

If the probe still fails, the helper returns None and the caller keeps the old error or NULL. With current tzdata that branch is unreachable.

Nothing else changes. Unit conversion, safe handling and the error message stay as they were. A fixed-offset timezone has no transitions, so it is unaffected.

Are these changes tested?

Yes. Six new tests sit next to the existing test_cast_timestamp_with_timezone_* tests in arrow-cast/src/cast/mod.rs:

• test_cast_timestamp_to_named_timezone_dst — America/New_York, with an unambiguous, an ambiguous and a nonexistent reading plus a null, under safe: false.
• test_cast_timestamp_to_named_timezone_dst_safe — the same input under safe: true, which no longer produces nulls.
• test_cast_timestamp_to_named_timezone_dst_southern_hemisphere — Australia/Sydney, where the transitions run the other way round.
• test_cast_timestamp_to_named_timezone_dst_nanosecond and ..._dst_changing_unit — the resolution composes with the unit paths.
• test_cast_timestamp_to_fixed_offset_timezone_unaffected — +08:00 on the same readings is unchanged.

Expected values come from chrono, built from the wall clock reading and the expected offset. They are not hardcoded. Revert the one-line kernel change and the five DST tests fail, while the fixed-offset test still passes.

Validation

The change was checked against 97,162 cases. These cover every transition of every IANA zone in 1890–1995 and 2023–2025: 15,871 in a gap and 14,370 ambiguous. There were zero disagreements with an independent implementation on the same tzdata.

Offsets resolve per row. One array that straddles both 2024 America/New_York transitions yields three distinct offsets, and every element matches PostgreSQL 17.11.

One unrelated discrepancy came up. chrono-tz 0.10.4 bundles tzdata 2025b, which puts the Europe/Chisinau and Europe/Tiraspol transitions an hour away from PostgreSQL, ICU and CPython. That reproduces on main without this PR. It needs a chrono-tz bump, not a change here.

A note on the test manifest

arrow-cast had no way to name an IANA zone in its own tests. Tz parses IANA names only when arrow-array is built with chrono-tz, and arrow-cast never enabled it.

This PR enables it on the dev-dependency only: arrow-array = { workspace = true, features = ["chrono-tz"] } under [dev-dependencies]. The new tests then run in every arrow-cast CI job. No public feature appears, and nothing changes for downstream crates.

One test needed an update. test_cast_string_to_timestamp_invalid_tz asserted the exact error tail only offset based timezones supported without chrono-tz feature, which is now failed to parse timezone. It asserts the stable prefix Parser error: Invalid timezone \"ZZTOP\": instead.

Local runs that pass: cargo test -p arrow-cast (default and --all-features, debug and release), cargo test -p arrow --features chrono-tz,prettyprint --test array_cast --test timezone, clippy with -D warnings on the CI feature combinations, and cargo doc --all-features with -D warnings.

Field research: PostgreSQL 17 and DuckDB

The resolution policy is the one arbitrary decision in this change, so it needs more than one reference. Both engines were measured directly: PostgreSQL 17.11 in postgres:17, and the DuckDB 1.5.2 CLI with the ICU extension. All times below are UTC.

Case	PostgreSQL 17.11	DuckDB 1.5.2	This PR
America/New_York 2024-11-03 01:30 — ambiguous	2024-11-03 06:30:00	2024-11-03 06:30:00	2024-11-03 06:30:00
America/Havana 2024-11-03 00:00 — ambiguous local midnight	2024-11-03 05:00:00	2024-11-03 05:00:00	2024-11-03 05:00:00
America/New_York 2024-03-10 02:30 — gap	2024-03-10 07:30:00	2024-03-10 07:30:00	2024-03-10 07:30:00
America/Sao_Paulo 2018-11-04 00:00 — local midnight does not exist	2018-11-04 03:00:00	2018-11-04 03:00:00	2018-11-04 03:00:00
Australia/Sydney 2024-10-06 02:30 — southern hemisphere gap	2024-10-05 16:30:00	2024-10-05 16:30:00	2024-10-05 16:30:00
Australia/Lord_Howe 2024-10-06 02:15 — 30 minute DST step	2024-10-05 15:45:00	2024-10-05 15:45:00	2024-10-05 15:45:00
Pacific/Chatham 2024-09-29 03:00 — +12:45 / +13:45	2024-09-28 14:15:00	2024-09-28 14:15:00	2024-09-28 14:15:00

The two ambiguous rows carry the argument. A gap has only one sensible answer, so it discriminates nothing. An ambiguous reading has two real answers, and the choice between them is the policy.

2024-11-03 01:30 in New York is either 05:30Z (EDT) or 06:30Z (EST). Both engines give 06:30Z. Havana's ambiguous midnight is either 04:00Z (CDT) or 05:00Z (CST). Both engines give 05:00Z. In both cases the answer is the later instant, which is what this PR does.

The case neither engine can arbitrate

A fixed-offset string such as '+05:30' has no agreed reading, so it is recorded here rather than hidden. Measured:

Spelling	PostgreSQL 17.11	Convention
SET TimeZone='+05:30' then '2024-01-01 12:00:00'::timestamptz	17:30Z	POSIX, west-positive
'2024-01-01 12:00:00'::timestamp AT TIME ZONE '+05:30'	17:30Z	POSIX, west-positive
timestamptz '2024-01-01 12:00:00 +05:30'	06:30Z	ISO, east-positive

PostgreSQL agrees with arrow-rs when the offset sits inside the literal. It disagrees when the offset acts as a zone name. DuckDB 1.5.2 rejects the zone-name spelling outright: Not implemented Error: Unknown TimeZone '+05:30'.

So neither engine settles it. This PR does not change that behaviour. It is tracked downstream in apache/datafusion#25170.

Are there any user-facing changes?

Yes. A cast that previously raised an error, or returned NULL under safe: true, now returns the instants described above. There is no API change.

This PR does not close DataFusion #25084 on its own

There are two independent paths to a DST-boundary failure. This PR fixes one of them:

-- cast kernel, fixed here
SELECT '2024-03-10 02:30:00'::timestamp AT TIME ZONE 'America/New_York';

-- string parser, NOT fixed here (see #11039)
SELECT '2024-03-10 02:30:00' AT TIME ZONE 'America/New_York';
SET datafusion.execution.time_zone = 'America/New_York';
SELECT '2024-03-10 02:30:00'::timestamptz;
-- Parser error: Error parsing timestamp from '2024-03-10 02:30:00': error computing timezone offset

To a user those are the same query, and the last spelling is the most natural one. Merge this alone and an explicit ::timestamp in the middle of an expression makes a query start to work. That is harder to explain than the current uniform failure. #11039 must land with this PR or close behind it, and must use the same policy so the two paths cannot drift.

This PR is sufficient for apache/datafusion#10308, which is entirely on the cast path.

Follow-up: arrow-array resolves ambiguity the opposite way

After this PR, arrow resolves an ambiguous local time two different ways in two crates:

	Ambiguous	In a gap
arrow-cast::resolve_local_offset (this PR)	the later instant	offset probed 24 hours earlier
arrow-array::types::from_naive_datetime (types.rs:347)	the earlier instant (Ambiguous(dt1, _))	None

That function is deliberately untouched here.

This matters downstream. DataFusion calls the arrow-array form directly with a Some(tz) in datafusion/functions/src/datetime/date_part.rs (date_to_scalar, four call sites). It builds local-midnight bounds for the date_part(YEAR, col) = <year> rewrite. Where that local midnight is ambiguous, the earlier instant becomes an upper bound, so valid rows can drop out. The behaviour predates this PR, but this PR widens the gap between the two policies.

The two want reconciling, or the difference wants documenting on both functions. Changing from_naive_datetime affects every caller of arrow-array, so it deserves its own review rather than a ride along with a cast fix.

Follow-up: a loud error becomes a quiet non-identity

to_local_time(ts) AT TIME ZONE tz is a documented DataFusion idiom. Across an ambiguous hour it is no longer the identity, and it no longer says so. Before this PR the second leg raised Cannot cast timezone to different timezone:

orig                       local                 roundtrip                  identity
2021-10-31T02:00:00+02:00  2021-10-31T02:00:00   2021-10-31T02:00:00+01:00  false
2021-10-31T02:30:00+02:00  2021-10-31T02:30:00   2021-10-31T02:30:00+01:00  false
2021-10-31T02:00:00+01:00  2021-10-31T02:00:00   2021-10-31T02:00:00+01:00  true

PostgreSQL behaves the same way. t = ((t AT TIME ZONE 'Europe/Brussels') AT TIME ZONE 'Europe/Brussels') is f for those same two instants. So this is the intended trade, not an argument against the change. But it is a real behaviour change, and downstream docs want to say so.

🤖 Generated with Claude Code

[adriangb]

cc @Jefffrey @Omega359

[adriangb]

Self-review QA pass — this is our own PR, so treat this as our own adversarial check, not an independent review.

No blocking defect. I re-measured every claim. Four findings, in priority order.

1. The stated safety argument for the 24-hour probe is not the load-bearing one

The PR justifies the probe with a transition-spacing figure: no two transitions in 597 zones are closer than 167 hours. That figure is correct. I reproduce it exactly (America/Boa_Vista 2000-10-15, 167.00 h). But it is not the property that makes the probe resolvable.

Two separate invariants are at work here:

• Transition spacing above 24 hours. This makes the probed offset the correct pre-transition offset. Margin: 167 hours. The PR states this one.
• Maximum local gap at or below 24 hours. This makes the probe land outside the gap at all. The PR does not state this one, and it has no margin.

Seven zones have a local gap of exactly 24.00 hours. These are the dateline changes:

Zone	Transition (UTC)	Local gap
Pacific/Kwajalein	1993-08-21T12:00Z	24.00 h
Kwajalein	1993-08-21T12:00Z	24.00 h
Pacific/Kiritimati	1994-12-31T10:00Z	24.00 h
Pacific/Kanton	1994-12-31T11:00Z	24.00 h
Pacific/Enderbury	1994-12-31T11:00Z	24.00 h
Pacific/Apia	2011-12-30T10:00Z	24.00 h
Pacific/Fakaofo	2011-12-30T11:00Z	24.00 h

The next largest gap is 10.00 h (Antarctica/Macquarie). So the distribution is bimodal, and the code sits exactly on the upper mode.

For a reading at the last second of such a gap the probe lands one second before the gap starts. The real margin is one second, not 167 hours. The code is correct today. It stops being correct the day tzdb gains a gap above 24 hours.

I tested all four surviving dateline zones at the first instant of the gap, the middle and the last second. All 13 readings match PostgreSQL 17.11 exactly:

Pacific/Apia       2011-12-30 00:00:00 -> 1325239200   (PG 1325239200)
Pacific/Apia       2011-12-30 00:00:01 -> 1325239201   (PG 1325239201)
Pacific/Apia       2011-12-30 12:00:00 -> 1325282400   (PG 1325282400)
Pacific/Apia       2011-12-30 23:59:59 -> 1325325599   (PG 1325325599)
Pacific/Kiritimati 1994-12-31 00:00:00 -> 788868000    (PG 788868000)
Pacific/Kwajalein  1993-08-21 23:59:59 -> 746020799    (PG 746020799)
Pacific/Fakaofo    2011-12-30 23:59:59 -> 1325329199   (PG 1325329199)
...

Please state the gap-length invariant next to the spacing invariant, both in the PR body and in the doc comment on resolve_local_offset. A future reader who bumps chrono-tz needs to know which of the two numbers to re-check.

2. The give-up branch is unreachable, so its callers are untested

LocalResult::None from the probe reaches .earliest(), which returns None. I could not construct any input that reaches it. Both invariants above exclude it:

• No gap exceeds 24 hours, so the probe never lands in another gap.
• No fall-back transition sits within 24 hours before a gap, so the probe never lands in an ambiguous interval either.

So the branch is dead code with real tzdata. I read the callers instead of testing them, and both are correct:

• safe: false — adjust returns None, try_unary raises CastError("Cannot cast timezone to different timezone"). The whole array fails, as before.
• safe: true — unary_opt writes a null. As before.

The ? in adjust covers three separate None sources: as_datetime, resolve_local_offset and from_naive_datetime. All three collapse into the same message. That is pre-existing and fine, but it means the new None source has no distinct diagnostic.

I confirmed the two options stay in step on real DST input:

cast  safe=false [Some(1710055800), Some(1730615400)]
cast  safe=true  [Some(1710055800), Some(1730615400)]

3. The fixed-offset claim is over-broad

The PR says PostgreSQL reads a fixed-offset string such as '+05:30' with the opposite sign. That is true for one spelling only. I measured both on PostgreSQL 17.11:

Spelling	Result	Sign convention
SET TimeZone='+05:30' then '2024-01-01 12:00:00'::timestamptz	17:30Z	POSIX, west-positive
'2024-01-01 12:00:00'::timestamp AT TIME ZONE '+05:30'	17:30Z	POSIX, west-positive
timestamptz '2024-01-01 12:00:00 +05:30'	06:30Z	ISO, east-positive

So PostgreSQL agrees with arrow-rs when the offset sits inside the literal, and disagrees when the offset acts as a zone name. DuckDB 1.5.2 rejects the zone-name spelling outright (Not implemented Error: Unknown TimeZone '+05:30'). Please say "as a zone name" in the PR body. The current wording reads as though PostgreSQL always disagrees.

4. The dev-dependency removes the no-chrono-tz coverage of arrow-cast

arrow-array = { workspace = true, features = ["chrono-tz"] } under [dev-dependencies] unifies across the graph. So cargo test -p arrow-cast now always builds Tz with IANA support, and arrow-cast has no test that exercises the offset-only Tz. That is exactly why test_cast_string_to_timestamp_invalid_tz needed the relaxed assertion.

The relaxed assertion is the right call. I raise this only so the lost coverage is on the record.

---

What I checked and found correct

The per-row claim holds. This was my main concern, because a once-per-array offset gives silently wrong values rather than a loud error. I built one 15-row array that straddles both 2024 America/New_York transitions, and compared every element against PostgreSQL 17.11:

Wall clock reading	arrow	PostgreSQL 17.11
2024-01-15 12:00:00	1705338000	1705338000
2024-03-10 01:30:00	1710052200	1710052200
2024-03-10 02:30:00 (gap)	1710055800	1710055800
2024-03-10 03:30:00	1710055800	1710055800
2024-06-15 12:00:00	1718467200	1718467200
2024-11-03 00:30:00	1730608200	1730608200
2024-11-03 01:30:00 (ambiguous)	1730615400	1730615400
2024-11-03 02:30:00	1730619000	1730619000
2024-12-15 12:00:00	1734282000	1734282000

Three distinct offsets appear in one array. A single offset per array cannot produce this.

A broad independent sweep agrees. I generated every DST transition of every zone from 2020 to 2025, then sampled the start and the midpoint of each gap and each repeated interval. That gives 5,012 readings across 226 zones: 2,506 in a gap, 2,506 ambiguous. Results:

• 0 unresolved readings under safe: true.
• 3,908 of them use a zone PostgreSQL 17.11 accepts. 16 disagree.
• All 16 are Europe/Chisinau and its alias Europe/Tiraspol.

That is the chrono-tz tzdata skew the PR already discloses. I confirmed it predates this PR. On the base commit, with no change applied:

main            Europe/Chisinau 2024-03-31 03:30 -> 1711845000
PostgreSQL 17.11                                -> 1711848600
DuckDB 1.5.2                                    -> 1711848600

PostgreSQL and DuckDB agree with each other, so chrono-tz is the outlier. It puts the Moldova spring transition at 02:00 local; the other two put it at 03:00 local. Non-boundary readings agree everywhere. Please add Europe/Tiraspol to the disclosure — it carries the same skew and the PR names only Europe/Chisinau.

The known divergence stays untouched and disclosed. git diff against the base commit shows this branch changes nothing under arrow-array/. arrow-array/src/types.rs:349 still reads Ambiguous(dt1, _) and None => None. Measured side by side in America/New_York:

2024-11-03 01:30:00	Result
from_naive_datetime	1730611800 = 05:30Z, the earlier instant
this PR's cast	1730615400 = 06:30Z, the later instant

Opposite, as the PR states. The disclosure in the body is accurate and prominent.

Test coverage. The six new tests cover the southern-hemisphere gap (Australia/Sydney), both safe modes, the nanosecond path, a unit change and a fixed offset. Australia/Lord_Howe, Pacific/Chatham, America/Sao_Paulo and America/Havana are not in this PR's tests, but #11054 adds them. I verified all four against PostgreSQL by hand and they pass on this branch too.

Suite. cargo test -p arrow-cast passes on this branch.