Misleading run error messages on no capacity or no matching fleets

[r4victor]

The run error messages shown in the end of dstack apply (based on job_model.termination_reason_message) are often misleading and sometimes just incorrect. There are confusing for humans, but for agents it's even worse since they go down the rabbit hole fixing the wrong thing when getting an incorrect error. Some examples:

"No matching fleet found"/"no fleets" when the fleets have no capacity

1. Create one and only cloud fleet with nodes: 1. or nodes: 0..1.
2. Submit the first run – it provisions and occupies the only fleet instance.
3. Submit the second run – it fails as expected but the error message is:

terrible-quail-1 provisioning completed (failed)
No fleets
No matching fleet found. Possible reasons: https://dstack.ai/docs/guides/troubleshooting/#no-fleets

and dstack ps displays no fleets in status – which is weird since the fleets do match but all instances are busy and there is no more capacity.

"Fleet is at capacity" when the fleet has idle instances that do not match

1. Create one and only cloud fleet with nodes: 1 that is CPU-only.
2. Submit one run that requires GPU – it fails as expected but the error message is

No offers
Fleet is at capacity

which is confusing since there is a fleet with capacity but the instance do not match the requirements.

Plain "No offers" when there are offers but no cloud capacity

1. Try to submit a run that has offers but no capacity (e.g. h100 on AWS in one region).
2. The only message you get is

modern-rattlesnake-2 provisioning completed (failed)
No offers
Check dstack logs -d modern-rattlesnake-2 for more details.

and dstack ps shows no offers. In this case "No capacity" is the proper message / status.

Error messages are worse where there are many different fleets that can't be used for run provisioning for different reasons

For example, there can be a matching SSH fleet with all instances busy and a cloud fleet that does not match the run and another cloud fleet that has matching offers but no capacity. Any current message like No offers or "No matching fleet found. would be incorrect in such cases.

Another related problem is that job status message (displayed in dstack ps) is derived from the termination_reason_message:

dstack/src/dstack/_internal/server/services/jobs/__init__.py

Lines 599 to 603 in 6510087

	if (
	job_model.termination_reason_message
	and "No matching fleet found" in job_model.termination_reason_message
	):
	return "no fleets"

which is very fragile.

Solution

A simple solution that does not require big API changes is to enrich return type of find_optimal_fleet_with_offers(). It can return a FleetRejectionReason for every candidate fleet. Then, we could construct a richer termination_reason_message that shows which fleets were tried and why they were rejected. Since there can be many fleet candidates, we'll display only a subset (e.g. arbitrary or top 3 best suited such as matching but at full capacity):

No fleet can run this job:
  ssh-prod: instances don't match requested resources
  aws-main: at max fleet capacity (4/4 nodes)
  gcp-eu: no matching offers
... and 8 more fleet(s) rejected
See https://dstack.ai/docs/guides/troubleshooting/

We could also improve run plan generic "No matching instance offers available" warning with the same per-fleet info.

Note that we have two distinct cases: a) no fleet selected (various rejection reason as shown above) b) an optimal fleet is selected but all offers failed. In the case of b) we can also show a much more informative error message:

Failed to provision in fleet 'aws-main': tried 5 of 12 offers (limit reached), all failed.
Last errors: us-east-1 g5.xlarge: InsufficientInstanceCapacity; eu-west-1 g5.xlarge: RequestLimitExceeded

This again requires no API changes but only returning extra info in _FailedNewCapacityProvisioning.

The "no fleets" job status message can be derived by introducing a new JobTerminationReason.NO_FLEET_ASSIGNED enum value, which can be useful for other cases as well.

And we could also display a selected fleet for the run in dstack apply output.