Uplinkd should probably go away

[andrewjstone]

Uplinkd in its own words:

 // This is a very limited function daemon.  It watches the smf database, looking
 // for entries in the uplinks/* property group.  Each entry contains a tofino
 // link name and one or more addresses.  For each link that has a corresponding
 // tfport interface, the daemon ensures that each of those addresses exists on
 // the interface, and cleans up any addresses on the interface that aren't
 // listed in the property.
 //
 // Interfaces on the system that don't have corresponding smf properties are
 // left untouched.  Entries in the database that don't have corresponding
 // illumos interfaces are ignored - this daemon does not create or destroy
 // interfaces.
 //
 // TODO: there is a gap at startup time.  If the daemon crashes, when we restart
 // we won't know which addresses are managed and which aren't.  This is only a
 // problem if the configured population changes between crashing and restarting.
 //
 // TODO: currently setting an interface into PtP mode seems to screw it up
 // somehow.  Even after removing that address, you can't add a new address to
 // it.  Maybe there is a way to reset an interface to broadcast mode after
 // deleting its point-to-point link(s)?

This is sort of an odd daemon. It looks at SMF properties to set addresses for uplink interfaces rather than providing a dropshot server for configuration. This is redundant information from the PortConfig in RackNetworkConfig that already gets plumbed to dpd. Dpd could easily perform these operations directly without having to parse SMF properties to learn the existing configuration. This would also prevent the bugs from the doc comment above, as reconcilliation in dpd wouldn't rely on smf refresh.

There is also additional parsing and reconciliation in the uplinkd_reconciler in omicron that would go away in this case. port_settings_apply would drive everything and this would go away.

[rcgoodfellow]

I think if uplinkd is to merge into something, it would be tfportd. dpd's domain is managing the switch ASIC and not the os/zone networking stack. An important aspect of the switching stack is that the configuration of the os/zone network data plane and the switching ASIC data plane are not the same. For example, in the switch zone we install underlay routes that are learned from DDM onto the switch ASIC for transit but not into the kernel's routing tables. This means communications between entities in the switch zone and the rest of the underlay network such as nexus instances take place symmetrically on a route that points from the switch zone to the global zone and goes through the global zone's underlay paths for off-sled communications.

Consider the alternative where nexus on sled1 sends a request to dpd on sled14 that comes in through one of sled14's network interfaces, gets routed to the switch zone, and then the switch zone sees that it has a direct route to sled14 through a switch interface and just sends the response that way. That's a very weird asymmetric path, but more important than being weird it puts our relatively low bandwidth scrimlet/ASIC PCIe link on a potentially high bandwidth data path. I suppose you could argue that the switch zone should get its own underlay prefix and that comms between switch zone entities and other endpoints on the underlay should be through the switch interfaces directly. However, that creates an odd situation where the switch zone has a route to the scrimlet it's living on both out a switch port and through the global zone. What we have now makes it so that host software communicating on the underlay always goes through host interfaces, which I think is the easiest design to reason about and get right.

That being said there is some overlap. Some of the network state we get such as link addresses do get added to both the switch ASIC data plane and the OS/zone data plane. And that is why you see seemingly redundant state across dpd and uplinkd. tfportd is in charge of the os/zone ports in the switch zone, so I think that is a more natural place for uplinkd logic to go, if it is to go away, but @bnaecker likely has thoughts there.

Providing an SFM versus Dropshot interface is a separate question. Something we got with SMF properties is persistence if the daemon gets knocked over (but clearly no persistence for cold boot). Now that we have reconcilers driving most things from nexus in the steady state, that is not as much of a concern. Uplinks are critically important to early networking before we hit the steady state, but I think with some of the recent work @jgallagher has done to have pre-nexus networking have more of a reconciler model, the concerns there may go away.