Skip to content

[Bug]: Error getting offers with empty imported fleet #3660

@jvstme

Description

@jvstme

Steps to reproduce

  1. Create an SSH fleet
  2. Add it to an export
  3. Delete instances in the fleet (dstack fleet delete <name> -i ...).
  4. Immediately after that, submit a run in the importer project.

Actual behaviour

$ dstack apply -b remote
Unexpected error: status code 500 when requesting 
http://localhost:3000/api/project/main/runs/get_plan. Check the server logs for backend issues, and
the CLI logs at (~/.dstack/logs/cli/latest.log) local CLI output

Expected behaviour

Run submitted successfully, possibly fails due to no capacity if there are no other fleets.

dstack version

master

Server logs

Traceback (most recent call last):
  File "/dstack/.venv/lib64/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/dstack/.venv/lib64/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
  File "/dstack/.venv/lib64/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/dstack/.venv/lib64/python3.10/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/dstack/.venv/lib64/python3.10/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/dstack/.venv/lib64/python3.10/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/dstack/.venv/lib64/python3.10/site-packages/starlette/middleware/base.py", line 176, in __call__
    with recv_stream, send_stream, collapse_excgroups():
  File "/usr/lib64/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/dstack/.venv/lib64/python3.10/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
    raise exc
  File "/dstack/.venv/lib64/python3.10/site-packages/starlette/middleware/base.py", line 178, in __call__
    response = await self.dispatch_func(request, call_next)
  File "/dstack/src/dstack/_internal/server/app.py", line 342, in log_http_metrics
    response: Response = await call_next(request)
  File "/dstack/.venv/lib64/python3.10/site-packages/starlette/middleware/base.py", line 156, in call_next
    raise app_exc
  File "/dstack/.venv/lib64/python3.10/site-packages/starlette/middleware/base.py", line 141, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/dstack/.venv/lib64/python3.10/site-packages/starlette/middleware/base.py", line 176, in __call__
    with recv_stream, send_stream, collapse_excgroups():
  File "/usr/lib64/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/dstack/.venv/lib64/python3.10/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
    raise exc
  File "/dstack/.venv/lib64/python3.10/site-packages/starlette/middleware/base.py", line 178, in __call__
    response = await self.dispatch_func(request, call_next)
  File "/dstack/src/dstack/_internal/server/app.py", line 288, in log_request
    response: Response = await call_next(request)
  File "/dstack/.venv/lib64/python3.10/site-packages/starlette/middleware/base.py", line 156, in call_next
    raise app_exc
  File "/dstack/.venv/lib64/python3.10/site-packages/starlette/middleware/base.py", line 141, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/dstack/.venv/lib64/python3.10/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/dstack/.venv/lib64/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/dstack/.venv/lib64/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/dstack/.venv/lib64/python3.10/site-packages/starlette/routing.py", line 714, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/dstack/.venv/lib64/python3.10/site-packages/starlette/routing.py", line 734, in app
    await route.handle(scope, receive, send)
  File "/dstack/.venv/lib64/python3.10/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/dstack/.venv/lib64/python3.10/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/dstack/.venv/lib64/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/dstack/.venv/lib64/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/dstack/.venv/lib64/python3.10/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
  File "/dstack/.venv/lib64/python3.10/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
  File "/dstack/.venv/lib64/python3.10/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
  File "/dstack/src/dstack/_internal/server/routers/runs.py", line 131, in get_plan
    run_plan = await runs.get_plan(
  File "/dstack/src/dstack/_internal/server/services/runs/__init__.py", line 356, in get_plan
    job_plans = await get_job_plans(
  File "/dstack/src/dstack/_internal/server/services/runs/plan.py", line 165, in get_job_plans
    fleet_model, instance_offers, backend_offers = await find_optimal_fleet_with_offers(
  File "/dstack/src/dstack/_internal/server/services/runs/plan.py", line 344, in find_optimal_fleet_with_offers
    candidate_fleet = fleet_model_to_fleet(candidate_fleet_model)
  File "/dstack/src/dstack/_internal/server/services/fleets.py", line 863, in fleet_model_to_fleet
    project_name=fleet_model.project.name,
  File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/orm/attributes.py", line 566, in __get__
    return self.impl.get(state, dict_)  # type: ignore[no-any-return]
  File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/orm/attributes.py", line 1086, in get
    value = self._fire_loader_callables(state, key, passive)
  File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/orm/attributes.py", line 1121, in _fire_loader_callables
    return self.callable_(state, passive)
  File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/orm/strategies.py", line 978, in _load_for_state
    return self._emit_lazyload(
  File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/orm/strategies.py", line 1079, in _emit_lazyload
    return loading.load_on_pk_identity(
  File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/orm/loading.py", line 694, in load_on_pk_identity
    session.execute(
  File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/orm/session.py", line 2365, in execute
    return self._execute_internal(
  File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/orm/session.py", line 2251, in _execute_internal
    result: Result[Any] = compile_state_cls.orm_execute_statement(
  File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement
    result = conn.execute(
  File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/engine/base.py", line 1416, in execute
    return meth(
  File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/sql/elements.py", line 523, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/engine/base.py", line 1638, in _execute_clauseelement
    ret = self._execute_context(
  File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/engine/base.py", line 1843, in _execute_context
    return self._exec_single_context(
  File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/engine/base.py", line 1983, in _exec_single_context
    self._handle_dbapi_exception(
  File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/engine/base.py", line 2355, in _handle_dbapi_exception
    raise exc_info[1].with_traceback(exc_info[2])
  File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/engine/base.py", line 1964, in _exec_single_context
    self.dialect.do_execute(
  File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/engine/default.py", line 945, in do_execute
    cursor.execute(statement, parameters)
  File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 172, in execute
    self._adapt_connection._handle_exception(error)
  File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 323, in _handle_exception
    raise error
  File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 149, in execute
    _cursor = self.await_(self._connection.cursor())
  File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 123, in await_only
    raise exc.MissingGreenlet(
sqlalchemy.exc.MissingGreenlet: greenlet_spawn has not been called; can't call await_only() here. Was IO attempted in an unexpected place? (Background on this error at: https://sqlalche.me/e/20/xd2s)

Additional information

No response

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions