This repository was archived by the owner on Jun 11, 2026. It is now read-only.

This repository was archived by the owner on Jun 11, 2026. It is now read-only.

DDP Hangs with uneven batches #80

Open

Labels

Dan Schwartz (danrsc)

opened

on Jan 10, 2022

If you run training with an uneven number of batches on different GPUs, a hang will occur. Since PyTorch 1.8.1, the join context manager on DDP exists to help with this situation. I don't believe PyMarlin handles this right now

Metadata

Assignees

No one assigned

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests