[Bug]: Can over-allocate instances in fleet past nodes-max when using rest API

[ASmedberg-woolpert]

Steps to reproduce

1. Spin up dstack server (I've only tested using AWS as backend with a pretty simple layout)

2. Add a fleet with a node max:

fleet.yml

name: test-fleet
nodes: 0..3
resources:
  cpu: 1..

3. write a script to shove tasks in via the rest API:

import requests
import json

token = "*****"
header = {
            "Content-Type": "application/json",
            "Accept": "application/json",
            "Authorization": f"Bearer {token}"
        }
server_url = "http://127.0.0.1:3000"
project_name = "main"
sub_url = "runs/apply"

runs = 10
for i in range(runs):
    data = {
        "force": True,
        "plan": {
            "run_spec": {
                "configuration": {
                    "type": "task",
                    "image": "ubuntu:22.04",
                    "creation_policy": "reuse-or-create",
                    "commands": ["date", "sleep 6", "date"],
                    "fleets": ["test-fleet"]
                }
            }
        }
    }

    requests.post(f"{server_url}/api/project/{project_name}/{sub_url}",
                    headers=header, data=json.dumps(data))

4. Run it (in my case, on the server)

Actual behaviour

The runs appear as expected:

But extra instances appear, more than the node max:

And these are real instances, not just phantoms, as seen here in my AWS console:

Expected behaviour

Given that the max nodes on the fleet was set to 3, I would've expected only 3 instances to spin up.

dstack version

20.12

Server logs

<Full logs as file below>

Additional information

logs.txt

[ASmedberg-woolpert]

Forgot to put this in above:
Even working at a semi-human time scale doesn't appear to fix the problem. I put an arbitrary time.sleep(1) into my for loop and still got similar results, although I did get fewer (4) instances. I had to up that to time.sleep(3) to actually get the expected result (only 3 instances)

[r4victor]

@ASmedberg-woolpert, over provisioning is currently possible but there is a background logic that terminates all extra instances as soon as they are idle. So the fleet nodes limits are soft. This is a temporary limitation because hard limit would require a long fleet lock and make provisioning longer. We're planning to change this soon with #3665 and #3551 – already Work in Progress.