DSS: Heavy NRID subscription and eviction operations cause SCD latency spikes

[wing-utm-sharing-airspace]

What are the risks associated with hosting NRID and SCD within the same DSS? Industry has observed issues with high NRID subscription creation and NRID evict tool impacting latency for SCD. Any hypothesis here are welcome.

[BenjaminPelletier]

Using the same CockroachDB-DAR-store DSS pool to provide both NetRID and SCD means the following resources are being shared (and perhaps more I haven't identified explicitly):

• CockroachDB cluster
• core-service instances
• connections between CRDB nodes

Any time a resource is shared, there is some potential for one service to impact the other. That could look like:

• CockroachDB cluster -> high resource demands on one or more nodes from both services could exhaust resources (e.g., RAM, disk space, CPU capacity, sockets, etc)
• core-service instances -> high request demands from both services could exhaust serving resources (e.g., sockets, workers, database connections, etc)
• connections between CRDB nodes -> large-volume communication could exhaust available bandwidth

That said, I don't think I have ever seen a deployment where any of the limits above were hit, except possibly database connections with certain settings. I believe the binding constraint I have seen has always been contention between database transactions, and that type of constraint should not apply to NetRID and SCD interacting since both use entirely separate database tables.

So, I would be surprised if NetRID interactions impacted SCD latency (though of course I wouldn't rule it out!). But, it seems like a diagnostic test should not be too difficult: apply a steady-state SCD load (perhaps with Locust via loadtest, or benchmarker) and then perform a NetRID eviction half way through observing the load. If there is a change in SCD performance matching the timing of the NetRID eviction, then that is a strong candidate -- repeat once or twice, and then file an Issue with reproduction instructions.

Apart from NetRID eviction, are there any other demands on the system that might trigger on a schedule similar to NetRID eviction that would perhaps cause someone to confuse them with NetRID eviction?

[utmverse]

Apart from NetRID eviction, are there any other demands on the system that might trigger on a schedule similar to NetRID eviction that would perhaps cause someone to confuse them with NetRID eviction?

Would it be reasonable to investigate this from the backup angle as well? It looks like hourly backups are running, so it may be worth checking whether the memory increases correlate with the backup windows.