Topology: YAML topology files and gateway priorities

[thiell]

Implements the design discussed in #374 (see the design comment). Closes #374.

YAML topology files

• New topology.yaml file syntax: routes as a list of gateways/targets objects. Selected by file extension (.yaml/.yml), so --topology works with both syntaxes; INI topology.conf remains fully supported.
• Unlike INI, the same gateways may appear in several routes: one gateway pool can reach several target groups, with symmetric failback — the core need from the PR Tree: support optional route weights in topology.conf (#374) #678 review.
• topology.yaml is preferred over topology.conf when both exist at the same config location.
• TopologyParser becomes a thin facade over per-format backends; graph building and validation are unchanged and format-agnostic. The INI code path is untouched and still covered by the existing tests.
• PyYAML stays an optional lazy import, only when a YAML topology is loaded (same pattern as YAML group files). pip, Fedora/EPEL and Debian packages already require it; without it, only .yaml topologies fail.

Gateway priorities

• In YAML, the gateways of a route may be a list of entries with descriptive attributes, in the style of modern YAML tools such as lnetctl: nodes (node set, required), weight (load share among gateways of equal priority, default 1) and priority (failover rank, default 1). A lower number means a higher priority: gateways of a given priority are used only when all gateways with a lower priority number are unreachable:

routes:
  - gateways:
      - nodes: gw1
      - nodes: gw2
        priority: 2         # gw1 default, gw2 failover
    targets:  rio[100-199]
  - gateways:
      - nodes: gw2
      - nodes: gw1
        priority: 2         # gw2 default, gw1 failover
    targets:  rio[200-299]

• A plain list of node sets (gateways: [gw1, gw2]) is a load-shared pool, equivalent to gateways: gw[1-2] — failover is always explicit with priority, and the entry form keeps the schema extensible for future per-gateway attributes.
• Gateways of equal priority share the load with least-connections as today, weighted by weight (integers >= 1). Weight 0 is rejected: a standby gateway is a higher priority number, so there is only one mechanism.
• Priorities apply to all traffic routed through the pool, including targets deeper in the tree.
• Routes may not have overlapping targets, and a gateway may not appear twice in one gateways list with different weights or priorities: both are rejected at parse time.
• Resolved priorities ride the pickled TopologyTree; trees from older versions are still accepted, and the wire protocol is unchanged.

Error reporting

• YAML errors name the route and the accepted keys: topology.yaml: route #3: unsupported gateway entry key(s): prio (expected: nodes, priority, weight).
• A topology.conf holding YAML syntax now says so, instead of just Invalid configuration file. INI parsing itself is unchanged.

Docs and tests

• config.rst/clush.rst document the YAML syntax, quoting rules (@group values must be quoted), precedence and priorities — including explicitly that a lower priority number wins; conf/topology.yaml.example added.
• New tests/TreeRouterTest.py covers gateway selection: priority order, failover, mutual failback, weighted distribution, pool equivalence of plain lists, multi-level trees, pickled-tree round-trip, pre-1.11 tree compatibility. YAML parser tests added to TreeTopologyTest, including INI/YAML tree equivalence.

Supersedes PR #678 (now a draft): the INI [weights] section is not kept, and its weighted least-connections selection is reused as the load-sharing policy among gateways of equal priority.

[thiell]

Renamed tier to priority as agreed in #374; docs now state explicitly that a lower number means a higher priority.

[degremont]

Thanks for the patch.
Overall good, but some ambiguity if in the config file, gateways or target nodes overlap.
By example, the following config is allowed, but routing for node5 is unclear:

routes:
- gateways: admin
  targets: gw[1-2]

# gw1 preferred
- gateways:
  - nodes: gw1
  - nodes: gw2
    priority: 2
  targets: node[0-9]

# gw2 preferred
- gateways:
  - nodes: gw2
  - nodes: gw1
    priority: 2
  targets: node[5-14]

• When targets overlap, and gateways are common, we should not allow that. (fine if this is totally different gateways)

Similarly, we should avoid:

gateways:
- nodes: gw[1-2]
- nodes: gw2
  priority: 2

Minor: we should tell to AI to not add ticket ID or PR ID in code comment, unless we will eventually have this kind of ref every where in the code. git blame can always tell. Exception if this is a very tricky piece of code, and details/explanation are needed, and reference to a ticket can help. But when this is just a new feature, I do not care about the original ticket ID.

[thiell]

Thanks, both confirmed and fixed.

• Overlapping targets with common gateways — worse than your example: all of node[0-14] follow whichever route is listed first, so even node[10-14] lose the preference they declared. Also covers the same-targets/contradictory-priorities variant.

TopologyError: Overlapping targets detected (node[5-9])! Cannot add route gw[1-2] -> node[5-14]

• Duplicate gateway in a gateways: list — reads as "gw1 default, gw2 failover" but actually load-shares round-robin. Same check for a repeat at equal priority with a different weight.

TopologyError: Duplicate gateway (gw2) in the gateways of gw[1-2]

• Overlapping targets with totally different gateways was already rejected, since 2011 — left as is:

TopologyError: Convergent path detected! Cannot add route gw[3-4] -> node[5-14]

• The rule lives in the graph layer, so INI is tightened too when one file spells the same pool two ways (gw[1-2]: and gw1,gw2:) with overlapping targets.
• Ticket IDs dropped from the code comments; kept the two GH#527 (bug-workaround refs).

[degremont]

LGTM

[thiell]

• schema errors name the route + accepted keys: topology.yaml: route #3: unsupported gateway entry key(s): prio (expected: nodes, priority, weight)
• YAML in a .conf now points to the .yaml extension (expected common user error)
• new tests: duplicate keys, anchors/merge keys, no section header; also checked on py2.7 + PyYAML 3.10
• Also dropped the PyYAML question from the description — optional lazy import, packages all require it already.

[thiell]

I briefly had a strict SafeLoader rejecting duplicate YAML keys (PyYAML keeps the last value silently) but dropped it: cluster.yaml group files tolerate the same thing, so it would have been inconsistent.