getActivePieceCount fails with out-of-gas for large datasets

[frrist]

Problem

getActivePieceCount(uint256 setId) has O(n) complexity where n = nextPieceId[setId]. For datasets with tens of thousands of pieces, this exceeds the gas limit for eth_call requests.

Reproduction

On mainnet verifier (0xBADd0B92C1c71d02E7d520f64c0876538fa2557F):

# Works - small dataset
cast call 0xBADd0B92C1c71d02E7d520f64c0876538fa2557F "getActivePieceCount(uint256)(uint256)" 1 --rpc-url https://api.node.glif.io/rpc/v1
# Returns: 2

# Fails - large dataset (54,568 pieces)
cast call 0xBADd0B92C1c71d02E7d520f64c0876538fa2557F "getActivePieceCount(uint256)(uint256)" 89 --rpc-url https://api.node.glif.io/rpc/v1
# Error: out of gas

Note: getActivePieces with pagination works fine for the same dataset.

[rvagg]

Instead of adding new storage with new invariants, my gentle nudge here would be that since this is for external observation that using a subgraph or similar indexer might be the best way to deal with this? We have paginated active pieces if you want a canonical answer but for efficient lookup just having an external index that knows about pieces would be much more efficient and not push more complexity on chain.

Goldsky is on mainnet and calibnet and we've been using it for a bunch of our services. Chain Love runs a The Graph service that's also available. I use run my own instance of Ponder, indexing all the events and other details for contracts I care about, a Filecoin supported fork I maintain is here: https://github.com/rvagg/filecoin-ponder (you can see the way I'm using it in indexer/ @ https://github.com/filozone/foc-observer). Hugo has written his own Filecoin-focused indexer solution, like a lightweight Ponder that's designed for Filecoin, here: https://github.com/hugomrdias/foxer (early, it works but feature set is being expanded). Further, if this is something that your individual nodes need, Piri could also montior data sets themselves, watch for events, do occsional checking against the full pieces list. I know it's annoying having to do local state vs chain state, but the costs of increasing chain state is pretty high.

[rvagg]

Looks like this is popping up in a few places as a usability hurdle, maybe we need to bite the bullet on this one and add a slot..

[rjan90]

Looks like this is popping up in a few places as a usability hurdle, maybe we need to bite the bullet on this one and add a slot..

Just getting a bit more clarity on scope for the changes for this. By adding a slot here, it would mean a contract change and upgrade right. So we are not going with the offchain/indexer route you previously commented about here: #265 (comment) ?

As very high level next steps for this issue, does this sound correct:

1. Confirming that the direction for this is: "Proceed with on-chain active piece count storage"
2. Design the storage/migration
3. Decide migration/backfill behavior for existing datasets
4. Implement tests
5. Contract Upgrade

[rvagg]

off-chain indexer is the easy route, but it's not easy for tools like filecoin-pin or bare synapse, they don't come with indexer access

this would be a network upgrade

But also, I don't have a strategy for what to do about existing data sets with no count, we're either going to have to backfill them (somehow! maybe an owner call, but even that's race-prone so maybe impractical), or have a split-brain - new & old data sets where old don't have an accurate count but new do have an accurate count.

So while I'm seeing the pain inflicted by not having this, I don't have a clear path to fixing it on-chain. So that's the main next step here, design.