Resolve DST-ambiguous and nonexistent local times in string_to_datetime

[adriangb]

Which issue does this PR close?

• Closes string_to_datetime (and Utf8 → Timestamp(tz) casts) fail on DST-ambiguous and nonexistent local times #11039.

Important

This PR stacks on #11038 and must merge after it. The branch sits on fix-dst-timezone-cast, so the diff here includes #11038's commit. Review the second commit only, or wait for #11038 to merge and this rebases down to one commit.

Together, #11038 and this PR close apache/datafusion#25084. Neither closes it alone. See "Are there any user-facing changes?" below.

Rationale for this change

What a user hits today

Parse a timestamp string into a named timezone. Pick a wall clock reading that the zone skips or repeats. The parse fails.

use arrow_array::timezone::Tz;
use arrow_cast::parse::string_to_datetime;

let tz: Tz = "America/New_York".parse().unwrap();

// 02:30 does not exist on 2024-03-10: the clocks go forward at 02:00.
string_to_datetime(&tz, "2024-03-10 02:30:00");
// 01:30 happens twice on 2024-11-03: the clocks go back at 02:00.
string_to_datetime(&tz, "2024-11-03 01:30:00");

Today:

gap       -> Err("Parser error: Error parsing timestamp from '2024-03-10 02:30:00': error computing timezone offset")
ambiguous -> Err("Parser error: Error parsing timestamp from '2024-11-03 01:30:00': error computing timezone offset")

After this PR:

gap       -> Ok("2024-03-10T07:30:00+00:00")
ambiguous -> Ok("2024-11-03T06:30:00+00:00")

Every Utf8 / LargeUtf8 / Utf8View cast to Timestamp(_, Some(tz)) goes through that function. Under CastOptions { safe: true } the error becomes a silent NULL for the whole row.

How this reaches a SQL user

A DataFusion user reaches the parser through ordinary SQL. #11038 fixed the cast kernel, so the first line below now works and the other two still fail:

SELECT '2024-03-10 02:30:00'::timestamp AT TIME ZONE 'America/New_York';  -- cast kernel: fixed by #11038
SELECT '2024-03-10 02:30:00'            AT TIME ZONE 'America/New_York';  -- parser: this PR
SET datafusion.execution.time_zone = 'America/New_York';
SELECT '2024-03-10 02:30:00'::timestamptz;                                -- parser: this PR

To a user those are the same query, and the last spelling is the most natural one. Merge #11038 alone and an explicit ::timestamp in the middle of an expression makes a query start to work. That is harder to explain than the current uniform failure. So this is a blocker for #11038, not a follow-up.

Why the parser fails

string_to_datetime in arrow-cast/src/parse.rs resolves a wall clock reading with LocalResult::single(). That is None in two distinct cases:

• the reading is ambiguous — the repeated hour when the clocks go back;
• the reading is nonexistent — the skipped hour when the clocks go forward.

There are three such call sites, one per shape of input:

#	Site	Example input
1	date only, reads local midnight	'2018-11-04'
2	naive datetime, no trailing offset or zone	'2024-03-10 02:30:00'
3	datetime with a trailing IANA zone name	'2024-03-10 02:30:00 America/New_York'

What changes are included in this PR?

Overview

One shared function now decides every wall clock reading in arrow-cast. The three parser sites and the cast kernel all call it. The policy itself does not change.

Detail

The resolution policy is deliberately shared with #11038, not restated. Two copies of this policy that drift apart is exactly the bug this PR avoids. One such divergence already exists in the tree, and this PR leaves it alone. See the follow-up section below.

So #11038's resolve_local_offset moves out of cast/mod.rs into a new private arrow-cast/src/local_time.rs module and becomes pub(crate). It generalises from &Tz to any chrono::TimeZone, because string_to_datetime is generic over its target timezone. A thin resolve_local_datetime wrapper returns a DateTime<T>, which is the shape the three parser sites want. All four call sites now go through the one function.

The policy is unchanged from #11038:

• An ambiguous reading takes the later instant. This is the offset in effect after the transition.
• A nonexistent reading moves forward by the length of the gap. The code recovers the pre-transition offset with one probe 24 hours earlier, and takes the earliest result.

Two properties of the timezone database make the probe safe:

• No two transitions in any of the 597 zones are closer than 167 hours. So the probe lands on the correct side of the transition.
• No local gap is longer than 24 hours. So the probe lands outside the gap at all.

Both were verified exhaustively for #11038. This PR does not re-derive them.

On site 3, the zone named in the string

Site 3 resolves against a zone the user spelled out, rather than one taken from the target type. So it is fair to ask whether it deserves stricter treatment. It gets the same policy, for two reasons:

1. A named zone makes the zone unambiguous. It does nothing to make the wall clock reading unambiguous. '2024-03-10 02:30:00 America/New_York' is exactly as unresolvable as '2024-03-10 02:30:00' read in America/New_York, and the user has no other spelling for what they meant.
2. PostgreSQL 17 makes no distinction between the two spellings. SET TimeZone='America/New_York'; SELECT '2024-03-10 02:30:00'::timestamptz; and SELECT timestamptz '2024-03-10 02:30:00 America/New_York'; both return 2024-03-10 07:30:00+00. This was measured, not assumed.

A different rule for site 3 would put the "same query, different spelling, different outcome" split back inside a single function. That split is what this PR exists to remove.

Are these changes tested?

Yes. Every expected instant in the new tests came from a real PostgreSQL 17.11 (postgres:17). None were derived by hand. All three call sites are covered for both an ambiguous and a nonexistent reading:

• America/New_York and America/Los_Angeles — a whole-hour DST step.
• Australia/Lord_Howe — a thirty minute step, so a gap shifts 02:15 to 02:45, not 03:15. This rules out any implementation that assumes a one-hour gap.
• Pacific/Chatham — +12:45 / +13:45, an offset that is not a whole number of hours on either side.
• Australia/Sydney — southern hemisphere, so the gap falls in October and the repeated hour in April. This catches implementations that reach for offset_from_utc_datetime as a shortcut, which returns the post-transition offset and is wrong here.
• The date-only site, through the two zones whose transitions land on midnight and are therefore unreachable from any datetime input:
  • America/Sao_Paulo 2018-11-04 — local midnight does not exist, because Brazil started DST at 00:00. Result: 2018-11-04T03:00:00Z.
  • America/Havana 2024-11-03 — local midnight happens twice, because Cuba ends DST at 00:00. Result: the later instant, 2024-11-03T05:00:00Z.
• A fixed offset (+05:30), which has no transitions and must be unaffected.
• CastOptions { safe: true } against safe: false at the array level. The former produced a silent NULL and the latter an error. Both now produce the resolved instant, and genuinely unparseable input still nulls or errors as before.
• An array that straddles both of a zone's 2024 transitions, across StringArray, LargeStringArray and StringViewArray. This confirms the offset resolves per row, not once for the array.
• Sub-second precision survives the shift applied to a nonexistent reading.

Counterfactually validated. Revert the three call sites and keep the tests, and 9 of the new tests fail. The safe: true case fails with left: 0, which is the silent-NULL mode.

cargo test -p arrow-cast            # and --all-features, --doc, --doc --no-default-features
cargo clippy -p arrow-cast --all-targets --all-features -- -D warnings
cargo test -p arrow --features chrono-tz
cargo test -p arrow-csv -p arrow-json -p arrow-array
cargo fmt --all

arrow-cast already gained arrow-array/chrono-tz as a dev-dependency in #11038, so no manifest change is needed here.

Field research: PostgreSQL 17 and DuckDB

The resolution policy is the one arbitrary decision in this stack, so it needs more than one reference. Both engines were measured directly: PostgreSQL 17.11 in postgres:17, and the DuckDB 1.5.2 CLI with the ICU extension. All times below are UTC.

Case	PostgreSQL 17.11	DuckDB 1.5.2	This PR
America/New_York 2024-11-03 01:30 — ambiguous	2024-11-03 06:30:00	2024-11-03 06:30:00	2024-11-03 06:30:00
America/Havana 2024-11-03 00:00 — ambiguous local midnight	2024-11-03 05:00:00	2024-11-03 05:00:00	2024-11-03 05:00:00
America/New_York 2024-03-10 02:30 — gap	2024-03-10 07:30:00	2024-03-10 07:30:00	2024-03-10 07:30:00
America/Sao_Paulo 2018-11-04 00:00 — local midnight does not exist	2018-11-04 03:00:00	2018-11-04 03:00:00	2018-11-04 03:00:00
Australia/Sydney 2024-10-06 02:30 — southern hemisphere gap	2024-10-05 16:30:00	2024-10-05 16:30:00	2024-10-05 16:30:00
Australia/Lord_Howe 2024-10-06 02:15 — 30 minute DST step	2024-10-05 15:45:00	2024-10-05 15:45:00	2024-10-05 15:45:00
Pacific/Chatham 2024-09-29 03:00 — +12:45 / +13:45	2024-09-28 14:15:00	2024-09-28 14:15:00	2024-09-28 14:15:00

The two ambiguous rows carry the argument. A gap has only one sensible answer, so it discriminates nothing. An ambiguous reading has two real answers, and the choice between them is the policy.

2024-11-03 01:30 in New York is either 05:30Z (EDT) or 06:30Z (EST). Both engines give 06:30Z. Havana's ambiguous midnight is either 04:00Z (CDT) or 05:00Z (CST). Both engines give 05:00Z. In both cases the answer is the later instant, which is what this PR does.

The case neither engine can arbitrate

A fixed-offset string such as '+05:30' has no agreed reading, so it is recorded here rather than hidden. Measured:

Spelling	PostgreSQL 17.11	Convention
SET TimeZone='+05:30' then '2024-01-01 12:00:00'::timestamptz	17:30Z	POSIX, west-positive
'2024-01-01 12:00:00'::timestamp AT TIME ZONE '+05:30'	17:30Z	POSIX, west-positive
timestamptz '2024-01-01 12:00:00 +05:30'	06:30Z	ISO, east-positive

PostgreSQL agrees with arrow-rs when the offset sits inside the literal. It disagrees when the offset acts as a zone name. DuckDB 1.5.2 rejects the zone-name spelling outright: Not implemented Error: Unknown TimeZone '+05:30'.

So neither engine settles it. This PR does not change that behaviour. It is tracked downstream in apache/datafusion#25170.

Are there any user-facing changes?

Yes, and the change is intentional. A string that previously failed to parse, or silently became NULL, now parses to the instant PostgreSQL and DuckDB produce. There is no API change. resolve_local_offset stays crate-private.

Together with #11038 this closes DataFusion #25084

Neither PR closes it alone. #11038 covers the cast kernel. This PR covers the parser. A user hits both through the same SQL, so both must land.

One existing test changes expectation

arrow/tests/timezone.rs asserted the old error for two America/Los_Angeles cases. They move from test_parse_timezone_invalid to test_parse_timezone:

Input	Before	After (= PostgreSQL 17)
2023-03-12 02:05:06 America/Los_Angeles	error computing timezone offset	2023-03-12T10:05:06+00:00
2023-11-05 01:30:06 America/Los_Angeles	error computing timezone offset	2023-11-05T09:30:06+00:00

Downstream

DataFusion has one matching expectation, at datafusion/sqllogictest/test_files/datetime/timestamps.slt:2360-2366. It asserts that SELECT TIMESTAMPTZ '2023-03-12 02:00:00 America/Los_Angeles' errors, and its own comment already notes # postgresql: accepts. On the next arrow upgrade it becomes a query P that returns 2023-03-12T10:00:00Z, which is what PostgreSQL returns. This PR includes no DataFusion change.

Follow-up: arrow-array resolves ambiguity the opposite way

This PR creates one shared helper for arrow-cast, but one divergence remains in the tree. It is deliberately untouched here:

	Ambiguous	In a gap
arrow-cast::local_time::resolve_local_offset	the later instant	offset probed 24 hours earlier
arrow-array::types::from_naive_datetime (types.rs:347)	the earlier instant (Ambiguous(dt1, _))	None

Measured in America/New_York for 2024-11-03 01:30:00: from_naive_datetime gives 05:30Z, and this stack gives 06:30Z.

This matters downstream. DataFusion calls the arrow-array form directly with a Some(tz) in datafusion/functions/src/datetime/date_part.rs (date_to_scalar, four call sites). Where the local midnight it builds is ambiguous, the earlier instant becomes an upper bound, so valid rows can drop out. The behaviour predates this stack.

Changing from_naive_datetime affects every caller of arrow-array, so it deserves its own review rather than a ride along with a parser fix.

🤖 Generated with Claude Code

[adriangb]

Self-review QA pass — this is our own PR, so treat this as our own adversarial check, not an independent review.

No blocking defect. The highest-risk part of this stack is the change of resolve_local_offset from &Tz to <T: TimeZone>, so I attacked that first. It is behaviour-preserving, and I verified that empirically rather than by reading. Four findings follow, in priority order.

1. The generalisation does not change the original cast call site — verified byte for byte

This was the item that could regress #11038 silently. I ran the identical battery of cast-path readings on both branches and compared the raw output:

• 15 readings in one array that straddles both 2024 America/New_York transitions.
• 15 readings across Australia/Sydney, Australia/Lord_Howe, Pacific/Chatham, America/Sao_Paulo, America/Havana, Europe/Lisbon, Europe/Dublin, Asia/Tehran, Pacific/Apia, America/Santiago and Asia/Kolkata.
• 13 readings inside the four 24-hour dateline gaps.

43 rows, and the two files are identical. The cast path on this branch is bit-for-bit the cast path on #11038.

The refactor moves .fix() from inside the function to the call site. Option::map(|o| o.fix()) and then ?, versus ? and then .fix(), are the same operation in the same order. The Single, Ambiguous and None arms are otherwise unchanged. So the empirical result matches the reading of the diff.

One note on the new wrapper. resolve_local_datetime builds its result with tz.from_utc_datetime(&(*local - offset.fix())), not with from_local_datetime. That is deliberate and necessary — from_local_datetime would reject the two cases this function exists to resolve — but it is not obvious. A one-line comment at that call helps the next reader.

2. The stated safety argument for the 24-hour probe is not the load-bearing one

This PR repeats #11038's justification: no two transitions are closer than 167 hours. That figure is correct, and I reproduce it. But it is not the property that makes the probe resolvable.

Two separate invariants are at work:

• Transition spacing above 24 hours. This makes the probed offset the correct pre-transition offset. Margin: 167 hours.
• Maximum local gap at or below 24 hours. This makes the probe land outside the gap at all. This PR does not state it, and it has no margin.

Seven zones have a local gap of exactly 24.00 hours — the dateline changes: Pacific/Apia, Pacific/Fakaofo, Pacific/Kiritimati, Pacific/Kanton, Pacific/Enderbury, Pacific/Kwajalein and Kwajalein. The next largest gap in any zone is 10.00 hours. For a reading at the last second of a 24-hour gap the probe lands one second before it. The real margin is one second.

The code is correct today. I tested the first instant, the midpoint and the last second of all four surviving dateline gaps through the cast path, and all 13 readings match PostgreSQL 17.11 exactly. But the doc comment in local_time.rs now carries this claim for the whole crate, so please state both invariants there.

3. The give-up branch is unreachable, and this PR widens who depends on that

resolve_local_offset returns None only when the probe itself lands in a gap. Both invariants above exclude that, so the branch is dead code with real tzdata. I could not construct an input that reaches it.

This PR raises the stakes, because the branch now has four callers rather than one. I read all four. All handle None correctly:

Caller	None behaviour
adjust_timestamp_to_timezone, safe: false	CastError("Cannot cast timezone to different timezone")
adjust_timestamp_to_timezone, safe: true	null
string_to_datetime date-only site	err("error computing timezone offset")
string_to_datetime naive-datetime site	err("error computing timezone offset")
string_to_datetime named-zone site	err("error computing timezone offset")

The three parser sites return a ParseError, which the string cast then turns into a null under safe: true and propagates under safe: false. I confirmed both modes still behave that way, and that genuinely bad input is unaffected:

str  safe=false ["2024-03-10 02:30:00", "2024-11-03 01:30:00"] -> [Some(1710055800), Some(1730615400)]
str  safe=true  ["2024-03-10 02:30:00", "2024-11-03 01:30:00"] -> [Some(1710055800), Some(1730615400)]
str  safe=false ["not a timestamp"]  -> Parser error: ... error parsing date
str  safe=true  ["not a timestamp"]  -> [None]

4. The known divergence deserves the same prominence it has in #11038

arrow-array/src/types.rs:349 is untouched. I confirmed that: git diff against the base commit shows this branch changes nothing under arrow-array/, and the function still reads Ambiguous(dt1, _) and None => None. Measured in America/New_York:

2024-11-03 01:30:00	Result
arrow_array::types::from_naive_datetime	1730611800 = 05:30Z, the earlier instant
arrow_cast::local_time::resolve_local_offset	1730615400 = 06:30Z, the later instant

2024-03-10 02:30:00 gives None from the first and 1710055800 from the second.

#11038 carries a full table and a paragraph on why the two must be reconciled. This PR mentions it in one parenthesis inside "What changes are included". I recommend the fuller disclosure here too. This is the PR that creates the single shared helper, so it is the PR that makes the surviving divergence look deliberate.

---

What I checked and found correct

Both paths now share one policy — measured, not assumed. I ran 5,012 boundary readings across 226 zones through the cast path and the string path in the same run. Every DST transition of every zone from 2020 to 2025, sampled at the start and the midpoint of each gap and each repeated interval: 2,506 in a gap, 2,506 ambiguous.

• The cast path and the string path agree on all 5,012 readings.
• 0 unresolved readings under safe: true.

That agreement is the central claim of this PR, and it holds across the whole tzdb, not just the tested zones.

Against PostgreSQL. 3,908 of those readings use a zone PostgreSQL 17.11 accepts. 16 disagree, on both paths equally. All 16 are Europe/Chisinau and its alias Europe/Tiraspol. That is the chrono-tz tzdata skew #11038 discloses, and it predates both PRs:

main             Europe/Chisinau 2024-03-31 03:30 -> 1711845000
PostgreSQL 17.11                                  -> 1711848600
DuckDB 1.5.2                                      -> 1711848600

PostgreSQL and DuckDB agree with each other, so chrono-tz is the outlier. Please name Europe/Tiraspol alongside Europe/Chisinau.

Per-row resolution on the string path. One StringArray that straddles both 2024 America/New_York transitions produces three distinct offsets, and every element matches PostgreSQL:

String	arrow	PostgreSQL 17.11
2024-01-15 12:00:00	1705338000	1705338000
2024-03-10 01:30:00	1710052200	1710052200
2024-03-10 02:30:00 (gap)	1710055800	1710055800
2024-03-10 03:30:00	1710055800	1710055800
2024-11-03 01:30:00 (ambiguous)	1730615400	1730615400
2024-11-03 02:30:00	1730619000	1730619000
2024-12-15 12:00:00	1734282000	1734282000

Utf8, LargeUtf8 and Utf8View produce identical results on that array.

The four hard zones the PR names. All verified against PostgreSQL independently:

Zone	Reading	arrow and PostgreSQL
Australia/Lord_Howe	2024-10-06 02:15 (30-minute step)	1728143100
Pacific/Chatham	2024-09-29 03:00 (+12:45 / +13:45)	1727532900
Australia/Sydney	2024-10-06 02:30 (southern gap)	1728145800
America/Sao_Paulo	2018-11-04 (midnight does not exist)	1541300400
America/Havana	2024-11-03 (midnight happens twice)	1730610000

The site-3 argument holds. I confirmed the PostgreSQL claim directly. SET TimeZone='America/New_York'; SELECT '2024-03-10 02:30:00'::timestamptz; and SELECT timestamptz '2024-03-10 02:30:00 America/New_York'; both return 2024-03-10 07:30:00+00. So PostgreSQL makes no distinction between the two spellings, and neither does this PR.

No call site is missed. After this PR the only remaining local-time resolution in the tree outside local_time.rs is arrow-array/src/types.rs:349, which is the known divergence. arrow-array/src/timezone.rs only forwards to chrono.

Suite. cargo test -p arrow-cast passes: 395 lib tests and 13 doc tests. The doc-test count rises from 12 to 13, which is the new string_to_datetime example.

The fixed-offset caveat is over-broad in the body. I measured it. PostgreSQL uses POSIX west-positive signs only when '+05:30' acts as a zone name (SET TimeZone, or AT TIME ZONE), giving 17:30Z. With the offset inside the literal, timestamptz '2024-01-01 12:00:00 +05:30' gives 06:30Z, which agrees with arrow-rs. DuckDB 1.5.2 rejects the zone-name spelling outright. Please say "as a zone name".