Currently each instance of TaskProxy server performs periodic tasks discovery using ListOperations method of HttpProxy. Discovered tasks are dumped to //sys/task_proxies/services table.
This approach will result in increased load for cluster if TaskProxy's instance count is big.
We can switch to leader-follower scheme, then only one instance will perform discovery and dump info to table, other instances will read that table.
Some details to achieve it:
- Switch from Deployment to StatefulSet, so we can define leader by ordinal=0
- We will have to expand
//sys/task_proxies/services table format, or create new utility table, so followers will read full info about jobs (i.e., currently end host/port is missing from table)
Currently each instance of TaskProxy server performs periodic tasks discovery using ListOperations method of HttpProxy. Discovered tasks are dumped to
//sys/task_proxies/servicestable.This approach will result in increased load for cluster if TaskProxy's instance count is big.
We can switch to leader-follower scheme, then only one instance will perform discovery and dump info to table, other instances will read that table.
Some details to achieve it:
//sys/task_proxies/servicestable format, or create new utility table, so followers will read full info about jobs (i.e., currently end host/port is missing from table)