retrieval_skipped_piece_missing is firing on mainnet against pieces that were added on chain seconds earlier and have never been scheduled for removal.
One deal, dealbot-backend v1.12.0, 2026-08-20 UTC:
14:21:29.454 pieces_added pod vgmcl
14:22:02.256 pieces_confirmed pieceIds:[857] pod vgmcl
14:22:03.998 deal_creation_completed pod vgmcl
14:22:09.497 retrieval_skipped_piece_missing pod sscln pieceId: 857
"PDP pieceLive=false; marked deal cleaned_up and skipped retrieval"
Piece 857 was added on chain in block 6297643, timestamp 14:21:30. 5.5 seconds after the deal is logged complete, a different pod in the same deployment reports it not live and writes the deal off. A second deal the same day is identical in shape: pieces_confirmed [861] 11:23:02.971, deal_creation_completed 11:23:06.006, skipped 11:23:11.437.
Scale
Every retrieval_skipped_piece_missing on mainnet from 2026-08-13 15:00 to 2026-08-20 15:00 UTC, cross-checked against indexed fwss_piece_added and pdp_pieces_removed records. 232 events across 232 distinct deals:
- 232 of 232 pieces are present on chain
- 0 appear in any removal event
- gap from the block timestamp of the add to the skip: 34s min, 70s max, 41s mean
- 0 outside that window, 0 skipped before their own add
| provider |
skips |
| 1 ezpdpz-main |
82 |
| 36 Cello |
81 |
| 2 beck-main |
47 |
| 30 foc-mainnet |
10 |
| 32 ezpdpz-main2 |
10 |
| 34 StorageUsgp2 |
2 |
Both example pieces return pieceLive = true now. For 774/857 an eth_call to PDPVerifier.pieceLive(774, 857) returns false at block 6297642 and true at block 6297643, the block the add landed in.
Effect
The deal is marked cleaned_up and the retrieval candidate query filters on cleaned_up = false, so it leaves the sampling pool. No code path sets cleanedUp back to false once set. That is 232 deals in the week above.
The provider dashboard's Retrieval success% column excludes only pending, so skipped.piece_missing counts as a non-success. At 2026-08-20 16:55 UTC ezpdpz-main reads 403 successes and 29 skips over 72h with no retrieval failures recorded in the window, which puts the column below the 97% approval threshold on a check that never failed.
The pre-check was added in #561.
Happy to pull more samples or run anything specific.
retrieval_skipped_piece_missingis firing on mainnet against pieces that were added on chain seconds earlier and have never been scheduled for removal.One deal, dealbot-backend v1.12.0, 2026-08-20 UTC:
Piece 857 was added on chain in block 6297643, timestamp 14:21:30. 5.5 seconds after the deal is logged complete, a different pod in the same deployment reports it not live and writes the deal off. A second deal the same day is identical in shape:
pieces_confirmed [861]11:23:02.971,deal_creation_completed11:23:06.006, skipped 11:23:11.437.Scale
Every
retrieval_skipped_piece_missingon mainnet from 2026-08-13 15:00 to 2026-08-20 15:00 UTC, cross-checked against indexedfwss_piece_addedandpdp_pieces_removedrecords. 232 events across 232 distinct deals:Both example pieces return
pieceLive = truenow. For 774/857 aneth_calltoPDPVerifier.pieceLive(774, 857)returns false at block 6297642 and true at block 6297643, the block the add landed in.Effect
The deal is marked
cleaned_upand the retrieval candidate query filters oncleaned_up = false, so it leaves the sampling pool. No code path setscleanedUpback to false once set. That is 232 deals in the week above.The provider dashboard's
Retrieval success%column excludes onlypending, soskipped.piece_missingcounts as a non-success. At 2026-08-20 16:55 UTC ezpdpz-main reads 403 successes and 29 skips over 72h with no retrieval failures recorded in the window, which puts the column below the 97% approval threshold on a check that never failed.The pre-check was added in #561.
Happy to pull more samples or run anything specific.