Support hashing objects larger than 4GB on Windows by dscho · Pull Request #2138 · gitgitgadget/git

dscho · 2026-06-04T07:30:19Z

Philip Oakley has contributed these patches ~4.5 years ago, and they have been carried in Git for Windows ever since.

Now that there are already other patch series flying around that try to address various aspects about >4GB objects (which aren't handled well by Git until it stops forcing unsigned long to do size_t's job), it seems a good time to upstream these patches, too, at long last.

cc: Philip Oakley philipoakley@iee.email
cc: Patrick Steinhardt ps@pks.im

On LLP64 systems, such as Windows, the size of `long`, `int`, etc. is only 32 bits (for backward compatibility). Git's use of `unsigned long` for file memory sizes in many places, rather than size_t, limits the handling of large files on LLP64 systems (commonly given as `>4GB`). Provide a minimum test for handling a >4GB file. The `hash-object` command, with the `--literally` and without `-w` option avoids writing the object, either loose or packed. This avoids the code paths hitting the `bigFileThreshold` config test code, the zlib code, and the pack code. Subsequent patches will walk the test's call chain, converting types to `size_t` (which is larger in LLP64 data models) where appropriate. Signed-off-by: Philip Oakley <philipoakley@iee.email> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Continue walking the code path for the >4GB `hash-object --literally` test. The `hash_object_file_literally()` function internally uses both `hash_object_file()` and `write_object_file_prepare()`. Both function signatures use `unsigned long` rather than `size_t` for the mem buffer sizes. Use `size_t` instead, for LLP64 compatibility. While at it, convert those function's object's header buffer length to `size_t` for consistency. The value is already upcast to `uintmax_t` for print format compatibility. Note: The hash-object test still does not pass. A subsequent commit continues to walk the call tree's lower level hash functions to identify further fixes. Signed-off-by: Philip Oakley <philipoakley@iee.email> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Continue walking the code path for the >4GB `hash-object --literally` test to the hash algorithm step for LLP64 systems. This patch lets the SHA1DC code use `size_t`, making it compatible with LLP64 data models (as used e.g. by Windows). The interested reader of this patch will note that we adjust the signature of the `git_SHA1DCUpdate()` function without updating _any_ call site. This certainly puzzled at least one reviewer already, so here is an explanation: This function is never called directly, but always via the macro `platform_SHA1_Update`, which is usually called via the macro `git_SHA1_Update`. However, we never call `git_SHA1_Update()` directly in `struct git_hash_algo`. Instead, we call `git_hash_sha1_update()`, which is defined thusly: static void git_hash_sha1_update(git_hash_ctx *ctx, const void *data, size_t len) { git_SHA1_Update(&ctx->sha1, data, len); } i.e. it contains an implicit downcast from `size_t` to `unsigned long` (before this here patch). With this patch, there is no downcast anymore. With this patch, finally, the t1007-hash-object.sh "files over 4GB hash literally" test case is fixed. Signed-off-by: Philip Oakley <philipoakley@iee.email> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Just like the `hash-object --literally` code path, the `--stdin` code path also needs to use `size_t` instead of `unsigned long` to represent memory sizes, otherwise it would cause problems on platforms using the LLP64 data model (such as Windows). To limit the scope of the test case, the object is explicitly not written to the object store, nor are any filters applied. The `big` file from the previous test case is reused to save setup time; To avoid relying on that side effect, it is generated if it does not exist (e.g. when running via `sh t1007-*.sh --long --run=1,41`). Signed-off-by: Philip Oakley <philipoakley@iee.email> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

To complement the `--stdin` and `--literally` test cases that verify that we can hash files larger than 4GB on 64-bit platforms using the LLP64 data model, here is a test case that exercises `hash-object` _without_ any options. Just as before, we use the `big` file from the previous test case if it exists to save on setup time, otherwise generate it. Signed-off-by: Philip Oakley <philipoakley@iee.email> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

To verify that the `clean` side of the `clean`/`smudge` filter code is correct with regards to LLP64 (read: to ensure that `size_t` is used instead of `unsigned long`), here is a test case using a trivial filter, specifically _not_ writing anything to the object store to limit the scope of the test case. As in previous commits, the `big` file from previous test cases is reused if available, to save setup time, otherwise re-generated. Signed-off-by: Philip Oakley <philipoakley@iee.email> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

dscho · 2026-06-04T17:14:02Z

/submit

gitgitgadget · 2026-06-04T17:15:49Z

Submitted as pull.2138.git.1780593313.gitgitgadget@gmail.com

To fetch this version into FETCH_HEAD:

git fetch https://github.com/gitgitgadget/git/ pr-2138/dscho/PhilipOakley/hashliteral_t-v1

To fetch this version to local tag pr-2138/dscho/PhilipOakley/hashliteral_t-v1:

git fetch --no-tags https://github.com/gitgitgadget/git/ tag pr-2138/dscho/PhilipOakley/hashliteral_t-v1

gitgitgadget · 2026-06-04T21:57:01Z

Philip Oakley wrote on the Git mailing list (how to reply to this email):

On 04/06/2026 18:15, Johannes Schindelin via GitGitGadget wrote:
> Philip Oakley has contributed these patches ~4.5 years ago, and they have
> been carried in Git for Windows ever since.
> 
> Now that there are already other patch series flying around that try to
> address various aspects about >4GB objects (which aren't handled well by Git
> until it stops forcing unsigned long to do size_t's job), it seems a good
> time to upstream these patches, too, at long last.

Yay. I approve this message ;-)

Philip

> 
> Philip Oakley (6):
>   hash-object: demonstrate a >4GB/LLP64 problem
>   object-file.c: use size_t for header lengths
>   hash algorithms: use size_t for section lengths
>   hash-object --stdin: verify that it works with >4GB/LLP64
>   hash-object: add another >4GB/LLP64 test case
>   hash-object: add a >4GB/LLP64 test case using filtered input
> 
>  object-file.c          | 18 +++++++++---------
>  object-file.h          |  4 ++--
>  sha1dc_git.c           |  3 +--
>  sha1dc_git.h           |  2 +-
>  t/t1007-hash-object.sh | 39 +++++++++++++++++++++++++++++++++++++++
>  5 files changed, 52 insertions(+), 14 deletions(-)
> 
> 
> base-commit: 9ac3f193c05c2237e2b14ebaa1149e9fc8a1abe0
> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2138%2Fdscho%2FPhilipOakley%2Fhashliteral_t-v1
> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2138/dscho/PhilipOakley/hashliteral_t-v1
> Pull-Request: https://github.com/gitgitgadget/git/pull/2138

gitgitgadget · 2026-06-04T21:57:03Z

User Philip Oakley <philipoakley@iee.email> has been added to the cc: list.

gitgitgadget · 2026-06-09T02:36:56Z

Junio C Hamano wrote on the Git mailing list (how to reply to this email):

Philip Oakley <philipoakley@iee.email> writes:

> On 04/06/2026 18:15, Johannes Schindelin via GitGitGadget wrote:
>> Philip Oakley has contributed these patches ~4.5 years ago, and they have
>> been carried in Git for Windows ever since.
>> 
>> Now that there are already other patch series flying around that try to
>> address various aspects about >4GB objects (which aren't handled well by Git
>> until it stops forcing unsigned long to do size_t's job), it seems a good
>> time to upstream these patches, too, at long last.
>
> Yay. I approve this message ;-)

While I very much appreciate the effort to switch to size_t where
appropriate (and the places we historically used ulong for size of
in-core memory region are the most appropriate places), such an old
series crashes with in-flight topics big time.  Can we get an update
on a more recent base?

No need to rush, as I'll be slowly processing the backlog to catch
up with the list traffic for a few days.

Thanks.

gitgitgadget · 2026-06-15T11:31:23Z

@@ -561,9 +561,9 @@ int odb_source_loose_read_object_info(struct odb_source *source,
 }


Patrick Steinhardt wrote on the Git mailing list (how to reply to this email):

On Thu, Jun 04, 2026 at 05:15:09PM +0000, Philip Oakley via GitGitGadget wrote: > diff --git a/object-file.c b/object-file.c > index 1f5f9daf24..c648cecd80 100644 > --- a/object-file.c > +++ b/object-file.c > @@ -581,7 +581,7 @@ static void write_object_file_prepare(const struct git_hash_algo *algo, > /* Generate the header */ > *hdrlen = format_object_header(hdr, *hdrlen, type, len); > > - /* Sha1.. */ > + /* Hash (function pointers) computation */ > hash_object_body(algo, &c, buf, len, oid, hdr, hdrlen); > } > Thanks for updating this comment while at it :) > diff --git a/t/t1007-hash-object.sh b/t/t1007-hash-object.sh > index 7867fd1dbf..10382a815e 100755 > --- a/t/t1007-hash-object.sh > +++ b/t/t1007-hash-object.sh > @@ -261,7 +261,7 @@ test_expect_success '--stdin outside of repository (uses default hash)' ' > test_cmp expect actual > ' > > -test_expect_failure EXPENSIVE,SIZE_T_IS_64BIT,!LONG_IS_64BIT \ > +test_expect_success EXPENSIVE,SIZE_T_IS_64BIT,!LONG_IS_64BIT \ > 'files over 4GB hash literally' ' > test-tool genzeros $((5*1024*1024*1024)) >big && > test_oid large5GB >expect && Previously we required `!LONG_IS_64BIT`, because the test would have succeeded on platforms where it is 64 bit wide. But now that this test works on all platforms I rather wonder whether we should completely drop that prerequisite here, as we expect it to pass regardless of whether or not long is 64 bit now. Patrick

gitgitgadget · 2026-06-15T11:31:25Z

User Patrick Steinhardt <ps@pks.im> has been added to the cc: list.

gitgitgadget · 2026-06-15T11:31:33Z

@@ -563,7 +563,7 @@ int odb_source_loose_read_object_info(struct odb_source *source,
 static void hash_object_body(const struct git_hash_algo *algo, struct git_hash_ctx *c,


Patrick Steinhardt wrote on the Git mailing list (how to reply to this email):

On Thu, Jun 04, 2026 at 05:15:08PM +0000, Philip Oakley via GitGitGadget wrote: > From: Philip Oakley <philipoakley@iee.email> > > Continue walking the code path for the >4GB `hash-object --literally` > test. The `hash_object_file_literally()` function internally uses both > `hash_object_file()` and `write_object_file_prepare()`. Both function > signatures use `unsigned long` rather than `size_t` for the mem buffer > sizes. Use `size_t` instead, for LLP64 compatibility. > > While at it, convert those function's object's header buffer length to > `size_t` for consistency. The value is already upcast to `uintmax_t` for > print format compatibility. One thing I was wondering is whether we should rather migrate to a size that is consistent across different platforms. We could e.g. `typedef uint64_t objsize_t` and then use that going forward. I guess the question though is whether that'd buy us anything. In other words, are there any platforms that we care about where `size_t` is only 32 bit wide? And would such platforms even be able to handle such large objects? Patrick

gitgitgadget · 2026-06-15T11:31:51Z

+	test-tool genzeros $((5*1024*1024*1024)) >big &&
+	test_oid large5GB >expect &&
+	git hash-object --stdin --literally <big >actual &&
+	test_cmp expect actual


Patrick Steinhardt wrote on the Git mailing list (how to reply to this email):

On Thu, Jun 04, 2026 at 05:15:10PM +0000, Philip Oakley via GitGitGadget wrote: > diff --git a/t/t1007-hash-object.sh b/t/t1007-hash-object.sh > index 10382a815e..59efee3aff 100755 > --- a/t/t1007-hash-object.sh > +++ b/t/t1007-hash-object.sh > @@ -269,4 +269,12 @@ test_expect_success EXPENSIVE,SIZE_T_IS_64BIT,!LONG_IS_64BIT \ > test_cmp expect actual > ' > > +test_expect_success EXPENSIVE,SIZE_T_IS_64BIT,!LONG_IS_64BIT \ > + 'files over 4GB hash correctly via --stdin' ' > + { test -f big || test-tool genzeros $((5*1024*1024*1024)) >big; } && > + test_oid large5GB >expect && > + git hash-object --stdin <big >actual && > + test_cmp expect actual > +' Same comment here: can we drop the `!LONG_IS_64BIT` prereq? Patrick

gitgitgadget · 2026-06-15T11:32:14Z

+	{ test -f big || test-tool genzeros $((5*1024*1024*1024)) >big; } &&
+	test_oid large5GB >expect &&
+	git hash-object --stdin <big >actual &&
+	test_cmp expect actual


Patrick Steinhardt wrote on the Git mailing list (how to reply to this email):

On Thu, Jun 04, 2026 at 05:15:11PM +0000, Philip Oakley via GitGitGadget wrote: > diff --git a/t/t1007-hash-object.sh b/t/t1007-hash-object.sh > index 59efee3aff..f2722380ee 100755 > --- a/t/t1007-hash-object.sh > +++ b/t/t1007-hash-object.sh > @@ -277,4 +277,12 @@ test_expect_success EXPENSIVE,SIZE_T_IS_64BIT,!LONG_IS_64BIT \ > test_cmp expect actual > ' > > +test_expect_success EXPENSIVE,SIZE_T_IS_64BIT,!LONG_IS_64BIT \ > + 'files over 4GB hash correctly' ' > + { test -f big || test-tool genzeros $((5*1024*1024*1024)) >big; } && > + test_oid large5GB >expect && > + git hash-object -- big >actual && > + test_cmp expect actual > +' Same comment here. Nit: I feel like we could've easily introduced all of these tests in the first commit. Patrick

PhilipOakley added 6 commits June 4, 2026 09:15

dscho self-assigned this Jun 4, 2026

dscho mentioned this pull request Jun 5, 2026

Missing Warning Leads to Silent Data Loss When Committing Large Files (>4GB) git-for-windows/git#6012

Open

1 task

gitgitgadget Bot reviewed Jun 15, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support hashing objects larger than 4GB on Windows#2138

Support hashing objects larger than 4GB on Windows#2138
dscho wants to merge 6 commits into
gitgitgadget:masterfrom
dscho:PhilipOakley/hashliteral_t

dscho commented Jun 4, 2026 •

edited by gitgitgadget Bot

Loading

Uh oh!

dscho commented Jun 4, 2026

Uh oh!

gitgitgadget Bot commented Jun 4, 2026

Uh oh!

gitgitgadget Bot commented Jun 4, 2026

Uh oh!

gitgitgadget Bot commented Jun 4, 2026

Uh oh!

gitgitgadget Bot commented Jun 9, 2026

Uh oh!

gitgitgadget Bot Jun 15, 2026

Uh oh!

gitgitgadget Bot commented Jun 15, 2026

Uh oh!

gitgitgadget Bot Jun 15, 2026

Uh oh!

gitgitgadget Bot Jun 15, 2026

Uh oh!

gitgitgadget Bot Jun 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -561,9 +561,9 @@ int odb_source_loose_read_object_info(struct odb_source *source,
		}

		@@ -563,7 +563,7 @@ int odb_source_loose_read_object_info(struct odb_source *source,
		static void hash_object_body(const struct git_hash_algo algo, struct git_hash_ctx c,

Conversation

dscho commented Jun 4, 2026 • edited by gitgitgadget Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dscho commented Jun 4, 2026

Uh oh!

gitgitgadget Bot commented Jun 4, 2026

Uh oh!

gitgitgadget Bot commented Jun 4, 2026

Uh oh!

gitgitgadget Bot commented Jun 4, 2026

Uh oh!

gitgitgadget Bot commented Jun 9, 2026

Uh oh!

gitgitgadget Bot Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

gitgitgadget Bot commented Jun 15, 2026

Uh oh!

gitgitgadget Bot Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

gitgitgadget Bot Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

gitgitgadget Bot Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dscho commented Jun 4, 2026 •

edited by gitgitgadget Bot

Loading