Support scalable WCT fan-out with up to 320 outputs

[HaiwangYu]

Support scalable WCT fan-out with up to 320 outputs

Summary

Wire-Cell Toolkit currently limits TBB-based FanoutCat components to a maximum of 10 output ports.

The current implementation is in:

tbb/inc/WireCellTbb/FanoutCat.h

It uses compile-time tuple-based TBB nodes, conceptually:

using tuple_type = std::tuple<msg_t, msg_t, ...>;
tbb::flow::function_node<msg_t, tuple_type>
tbb::flow::split_node<tuple_type>

The fan-out function produces one tuple element for each output, and the TBB split_node exposes one output port for each tuple element.

The implementation currently:

• asserts that the requested output count is between 1 and 10;
• explicitly dispatches to build_fanouter<N>();
• instantiates tuple-based implementations only for N = 1...10.

The required use case is substantially larger:

WCT should support fan-out multiplicities of up to 320 outputs.

For this scale, simply extending the existing template dispatch from 10 to 320 is probably not an appropriate solution.

Background

Issue #236 proposed implementing multi-layer fan-in and fan-out structures in Jsonnet to work around historical TBB tuple and port-count limits.

Recent oneTBB versions appear to have removed or increased the earlier small fixed limit. However, removal of the old TBB limit does not necessarily make a single 320-element tuple-based node practical.

Potential concerns include:

• compilation time;
• compiler memory use;
• object-file and binary size;
• construction of very large tuple types;
• explicit dispatch for hundreds of multiplicities;
• template error complexity;
• maintainability;
• runtime graph-construction cost;
• whether split_node is conceptually appropriate when all outputs carry the same message type.

Therefore, this issue should evaluate a scalable fan-out architecture rather than only increasing the existing hard-coded limit.

Required semantics

Before selecting an implementation, the required fan-out behavior should be documented.

Questions include:

1. Does every output receive the same input payload?
2. Is a separate msg_t wrapper created for each output?
3. Does each output need a distinct WCT sequence number?
4. Must each output preserve input ordering independently?
5. Must downstream nodes address outputs by port index?
6. Can outputs be connected or disconnected independently?
7. How should EOS be propagated?
8. What happens if one downstream branch is slow or rejects a message?
9. Is branch-specific backpressure required?
10. Are all 320 branches expected to be active simultaneously?

These semantics determine whether a broadcast node, dynamic distributor, or hierarchical fan-out tree is most appropriate.

Current WCT implementation

The current FanoutCat design is approximately:

                    tuple element 0 -> sequencer -> output 0
                   /
input -> function -> tuple element 1 -> sequencer -> output 1
                   \
                    ...
                     tuple element N -> sequencer -> output N

The output-port count is embedded in the C++ type.

This design is manageable for small multiplicities but does not naturally scale to hundreds of outputs.

Candidate approaches

Option A: Extend the tuple-based implementation to 320

The most direct change would be to instantiate:

build_fanouter<1>();
build_fanouter<2>();
// ...
build_fanouter<320>();

or use compile-time dispatch to generate the same range.

Advantages

• Minimal conceptual change.
• Existing port-index behavior is retained.
• Existing sequencing behavior may be preserved.
• Reuses the current TBB split_node design.

Disadvantages

• Potentially very high compile-time cost.
• Potentially high compiler memory consumption.
• Large tuple and template types.
• Increased binary size.
• Hundreds of template instantiations.
• Difficult compiler diagnostics.
• A 320-element split_node may not be practical even if technically supported.
• Supporting every value from 1 through 320 may be unnecessarily expensive.

Assessment

This option should be tested as a baseline, but it should not be assumed to be the preferred production solution.

Option B: Use broadcast semantics

If every downstream branch receives effectively the same message, the fan-out may be modeled as one sender connected to many successors.

Conceptually:

                  -> successor 0
                 /
input -> sender --> successor 1
                 \
                  -> ...
                   -> successor 319

A tbb::flow::broadcast_node<msg_t> or equivalent WCT sender may support many successors without encoding every successor as an element in a std::tuple.

Advantages

• Output count becomes a runtime graph property.
• No 320-element tuple.
• No template instantiation for every multiplicity.
• Simpler underlying TBB graph.
• Naturally represents identical-message broadcast.

Disadvantages

• WCT may currently require separately addressable output ports.
• Per-output sequence numbers may not be generated automatically.
• Per-branch buffering or ordering may differ from the current FanoutCat.
• A single sender interface may not satisfy WCT graph-port conventions.
• Slow-successor and rejected-message behavior must be clearly defined.

Assessment

This should be the preferred candidate when all outputs have equivalent message semantics and do not require distinct typed ports.

Option C: Implement a dynamic WCT fan-out distributor

Introduce a custom fan-out implementation whose number of branches is determined at runtime.

Conceptually:

input
  |
dynamic distributor
  |
  +-- branch node 0 -> output 0
  +-- branch node 1 -> output 1
  +-- ...
  +-- branch node 319 -> output 319

The distributor could maintain a runtime collection of branch nodes or sender endpoints rather than representing them in one tuple type.

Each branch could include its own sequencing or buffering node if required.

Advantages

• Supports arbitrary or configurable multiplicity.
• Retains individually addressable WCT output branches.
• Avoids very large tuple types.
• Can preserve per-output sequence behavior.
• The maximum can be controlled by runtime configuration.

Disadvantages

• Requires a new WCT/TBB adapter design.
• Lifetime, ownership, edge registration, and reset behavior must be handled carefully.
• Rejection and backpressure behavior must be defined.
• More invasive than simply extending the current templates.

Assessment

This is likely the most general long-term solution if WCT truly requires 320 distinct logical output ports.

Option D: Generate a hierarchical fan-out tree

Construct several bounded-size fan-out nodes in multiple layers.

For example, a branching factor of 8 can provide 320 leaves using several levels:

                         input
                           |
                    first-level fan-out
                  /    /    |    \     \
                 ...  bounded branches ...
                         |
                 second-level fan-outs
                         |
                    up to 320 leaves

A branching factor of 10 can support up to 1,000 leaves with three levels.

Advantages

• Reuses the existing small FanoutCat implementation.
• Avoids a 320-element tuple.
• Requires little or no new low-level TBB node design.
• Can be generated automatically from Jsonnet or a C++ graph helper.
• Each leaf remains a distinct graph endpoint.
• Compatible with older TBB versions.

Disadvantages

• Adds internal graph nodes and edges.
• Adds scheduling and message-forwarding overhead.
• Graph visualization becomes more complicated.
• Sequence numbers and EOS behavior across layers must be verified.
• Uneven final levels require careful generation.
• A Jsonnet-only solution may duplicate behavior that belongs in C++.

Assessment

This is a practical fallback and may be the lowest-risk near-term solution.

It should be compared against a broadcast or dynamic-distributor implementation.

Recommended direction

For a required maximum of 320 outputs, do not initially extend the existing tuple implementation directly to 320.

Instead:

1. Define the exact fan-out semantics required by WCT.
2. Determine whether one sender with 320 successors is sufficient.
3. Prototype a broadcast-based implementation.
4. Prototype or generate a bounded-branching hierarchical fan-out tree.
5. Benchmark both approaches.
6. Use direct tuple-based fan-out only as a baseline comparison.
7. Select the implementation based on correctness, scalability, and maintainability.

The likely choices are:

• Broadcast-based implementation when the outputs are semantically identical.
• Dynamic distributor when 320 separately addressable output ports are required.
• Hierarchical fan-out tree as a compatible and lower-risk fallback.

Proposed tasks

1. Document the 320-output use case

Add a concrete description of the expected topology and message behavior.

Document:

• number of outputs;
• typical and maximum number of active outputs;
• message rate;
• message size;
• whether payloads are shared or copied;
• ordering requirements;
• EOS requirements;
• per-output sequence-number requirements;
• backpressure expectations;
• whether output ports must be individually addressable.

2. Verify current oneTBB capability

Build minimal standalone tests using WCT's minimum supported TBB version.

Test a split_node with tuple sizes such as:

• 10;
• 16;
• 32;
• 64;
• 128;

Record:

• whether compilation succeeds;
• compilation time;
• peak compiler memory;
• object-file size;
• executable size;
• graph-construction time;
• runtime throughput;
• runtime memory.

This test is intended to characterize the direct tuple approach, not necessarily to select it.

3. Test one sender with many successors

Build a TBB test using one sender or broadcast_node<msg_t> connected to:

• 10 successors;
• 32 successors;
• 64 successors;
• 128 successors;
• 320 successors.

Measure:

• message-delivery correctness;
• throughput;
• latency;
• memory use;
• behavior when one successor is slow;
• behavior when one successor rejects a message;
• EOS behavior.

4. Prototype a dynamic WCT fan-out

Investigate a FanoutCat replacement that creates branch nodes dynamically.

Requirements should include:

• runtime-configurable output count;
• individually addressable WCT outputs, if required;
• no tuple proportional to output count;
• correct ownership and destruction;
• graph reset compatibility;
• independent sequencing where required;
• correct EOS propagation;
• clear failure behavior.

5. Prototype a hierarchical fan-out generator

Create a helper that accepts:

number of leaves
maximum branching factor

For example:

nleaves = 320
branching = 8

The helper should:

• generate exactly 320 leaf outputs;
• minimize tree depth;
• avoid unnecessary nodes;
• map logical output indices deterministically to leaves;
• hide internal node names where possible;
• preserve message and EOS behavior;
• provide useful graph diagnostics.

Determine whether this helper belongs in:

• C++;
• pgraph.jsonnet;
• a reusable graph-construction library.

6. Compare architectures

Benchmark at least:

1. direct 320-element tuple and split_node, if compilation succeeds;
2. one broadcast sender with 320 successors;
3. dynamic distributor with 320 branches;
4. hierarchical fan-out tree.

Compare:

• build time;
• compiler memory;
• binary size;
• startup time;
• runtime throughput;
• message latency;
• runtime memory;
• scheduling overhead;
• code complexity;
• graph complexity;
• compatibility with supported TBB versions.

7. Review message-copy behavior

Determine whether each branch receives:

• a copy of msg_t;
• a shared payload through a smart pointer;
• a separately constructed message;
• a separately assigned sequence number.

For 320 outputs, unnecessary deep copies could dominate runtime and memory use.

The implementation should share immutable payload data where possible while preserving branch-specific metadata when required.

8. Test ordering and EOS

Add tests covering:

• one message broadcast to 320 outputs;
• multiple consecutive messages;
• independently delayed branches;
• branch-local ordering;
• sequence-number behavior;
• EOS delivery to every output;
• disconnected outputs;
• partially connected outputs;
• graph shutdown and reset;
• exceptions or rejected messages.

9. Reassess issue #236

Issue #236 proposed multi-layer fan-in and fan-out Jsonnet helpers.

For a 320-output requirement, the multi-layer concept remains relevant even if recent oneTBB removed the original 10-port limit.

However, the rationale changes:

• Previously: work around a hard TBB port-count limit.
• Now: avoid impractical large tuple types and provide scalable runtime graph construction.

The new implementation should either:

• supersede issue Support for multi-layered fanout/fanin #236 with a scalable C++ solution;
• revive the hierarchical approach with updated justification; or
• provide both a broadcast implementation and a hierarchical compatibility fallback.

Non-goals

This issue does not necessarily require equivalent 320-input fan-in support.

A 320-input synchronization node has different buffering, matching, latency, and memory implications and should be evaluated separately.

In particular, a queueing join_node with 320 inputs waits for one message from every input before emitting a tuple. That behavior may not match the application's actual aggregation requirements.

Suggested implementation phases

Phase 1: Requirements and benchmarks

• Document exact output semantics.
• Test oneTBB tuple limits.
• Benchmark broadcast to 320 successors.
• Prototype a generated fan-out tree.

Phase 2: Initial scalable implementation

Implement either:

• broadcast-based fan-out; or
• generated hierarchical fan-out.

Support up to 320 outputs and add correctness tests.

Phase 3: Interface and performance refinement

• Hide internal implementation nodes.
• Optimize message sharing and sequence handling.
• Improve diagnostics.
• Consider replacing the fixed maximum with a configurable safety limit.

Suggested acceptance criteria

• WCT supports a configured fan-out of 320 outputs.
• The implementation does not require manually listing template cases from 1 through 320.
• A 320-element std::tuple is not required in the selected production design unless benchmarks clearly justify it.
• Every connected output receives each expected message exactly once.
• Every connected output receives EOS correctly.
• Per-output ordering behavior is documented and tested.
• Sequence-number behavior is documented and tested.
• Behavior with slow or rejecting successors is documented.
• Build-time and runtime performance are measured.
• Supported TBB or oneTBB versions are documented.
• The relationship to issue Support for multi-layered fanout/fanin #236 is documented.
• The selected approach supports 320 outputs without unreasonable compilation time, compiler memory, binary size, or runtime overhead.

Initial recommendation

The initial implementation should prioritize:

1. a single broadcast-style sender connected to 320 successors, if WCT's port model permits it;
2. otherwise, a generated bounded-branching fan-out tree;
3. a custom dynamic distributor as the longer-term solution if distinct WCT ports and branch-local behavior are required.

Directly extending the current tuple-based implementation to 320 should be treated as an experiment and benchmark, not the default design.

Relevant files

tbb/inc/WireCellTbb/FanoutCat.h
tbb/inc/WireCellTbb/FaninCat.h
cfg/pgrapher/experiment/dune-vd/funcs.jsonnet
cfg/pgraph.jsonnet

Related issue

• Support for multi-layered fanout/fanin #236 - reusable multi-layer fan-in and fan-out construction

[brettviren]

@HaiwangYu this is too verbose for me to absorb. Can you summarize what you intend to do? Just based on the verbosity and lack of understanding transmitted I'd mark this "wontfix".

I feel at least one motivating problem to solve needs to be described. We may then think if there are other ways to solve that problem.

The only whole-FD fanout I can think of involves getting a deposet between drifter and per anode depo transforms. This does have other solutions that allow for single-anode graphs and these solutions are better suited to CPU grid computing as well as hypothetical future "wire cell sim as a service" patterns.

Do you have other problems this large fan would solve?

Phlex's fold/unfold implementation of fanin/fanout may be a way to have rather large logical fans.

[HaiwangYu]

Hi @brettviren, sorry that this verbosity is mainly for AI to digest.

To summarize, with the new TBB discussion in #236, how do we change this hard-coded fanout here? https://github.com/WireCell/wire-cell-toolkit/blob/master/tbb/inc/WireCellTbb/FanoutCat.h#L107

[brettviren]

The only answer I know is the obvious one: extend that branch chain (changing to a switch or introducing else's would be ever slightly more efficient).

The core problem is that C++ has no way to translate a number held in a run-time variable to a template parameter other than writing code that implements every possible value of N and then having a run-time switch.

A hierarchical fan-of-fans is another solution and one we already have.

But, I still think we should put aside solutions until the problem is better understood. WHY do you think we require such large fanouts and HOW might we avoid the problem with different solutions?

[HaiwangYu]

The history of this issue is that I asked AI to dig a bit on your latest comment in #236 and thinking we need to do something if we need fanout > 10. So I made this issue to keep a record of the output for my discussion with AI. I don't want to implement this now and I can close this issue for now to tidy things up.

Sorry I may miss-understood your comments in #236 that tbb has no limits now. I thought you are thinking about something like a single layer fanout

Some other information:

• 320 is the FD-VD CRU number
• If we do need a solution, AI suggested something like a "broadcast_node", which could be a direction to pursue