-
Notifications
You must be signed in to change notification settings - Fork 217
Closed
Labels
bugSomething isn't workingSomething isn't working
Description
Steps to reproduce
- Create an SSH fleet
- Add it to an export
- Delete instances in the fleet (
dstack fleet delete <name> -i ...). - Immediately after that, submit a run in the importer project.
Actual behaviour
$ dstack apply -b remote
Unexpected error: status code 500 when requesting
http://localhost:3000/api/project/main/runs/get_plan. Check the server logs for backend issues, and
the CLI logs at (~/.dstack/logs/cli/latest.log) local CLI outputExpected behaviour
Run submitted successfully, possibly fails due to no capacity if there are no other fleets.
dstack version
master
Server logs
Traceback (most recent call last):
File "/dstack/.venv/lib64/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/dstack/.venv/lib64/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
return await self.app(scope, receive, send)
File "/dstack/.venv/lib64/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
await super().__call__(scope, receive, send)
File "/dstack/.venv/lib64/python3.10/site-packages/starlette/applications.py", line 112, in __call__
await self.middleware_stack(scope, receive, send)
File "/dstack/.venv/lib64/python3.10/site-packages/starlette/middleware/errors.py", line 187, in __call__
raise exc
File "/dstack/.venv/lib64/python3.10/site-packages/starlette/middleware/errors.py", line 165, in __call__
await self.app(scope, receive, _send)
File "/dstack/.venv/lib64/python3.10/site-packages/starlette/middleware/base.py", line 176, in __call__
with recv_stream, send_stream, collapse_excgroups():
File "/usr/lib64/python3.10/contextlib.py", line 153, in __exit__
self.gen.throw(typ, value, traceback)
File "/dstack/.venv/lib64/python3.10/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
raise exc
File "/dstack/.venv/lib64/python3.10/site-packages/starlette/middleware/base.py", line 178, in __call__
response = await self.dispatch_func(request, call_next)
File "/dstack/src/dstack/_internal/server/app.py", line 342, in log_http_metrics
response: Response = await call_next(request)
File "/dstack/.venv/lib64/python3.10/site-packages/starlette/middleware/base.py", line 156, in call_next
raise app_exc
File "/dstack/.venv/lib64/python3.10/site-packages/starlette/middleware/base.py", line 141, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
File "/dstack/.venv/lib64/python3.10/site-packages/starlette/middleware/base.py", line 176, in __call__
with recv_stream, send_stream, collapse_excgroups():
File "/usr/lib64/python3.10/contextlib.py", line 153, in __exit__
self.gen.throw(typ, value, traceback)
File "/dstack/.venv/lib64/python3.10/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
raise exc
File "/dstack/.venv/lib64/python3.10/site-packages/starlette/middleware/base.py", line 178, in __call__
response = await self.dispatch_func(request, call_next)
File "/dstack/src/dstack/_internal/server/app.py", line 288, in log_request
response: Response = await call_next(request)
File "/dstack/.venv/lib64/python3.10/site-packages/starlette/middleware/base.py", line 156, in call_next
raise app_exc
File "/dstack/.venv/lib64/python3.10/site-packages/starlette/middleware/base.py", line 141, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
File "/dstack/.venv/lib64/python3.10/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/dstack/.venv/lib64/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/dstack/.venv/lib64/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/dstack/.venv/lib64/python3.10/site-packages/starlette/routing.py", line 714, in __call__
await self.middleware_stack(scope, receive, send)
File "/dstack/.venv/lib64/python3.10/site-packages/starlette/routing.py", line 734, in app
await route.handle(scope, receive, send)
File "/dstack/.venv/lib64/python3.10/site-packages/starlette/routing.py", line 288, in handle
await self.app(scope, receive, send)
File "/dstack/.venv/lib64/python3.10/site-packages/starlette/routing.py", line 76, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/dstack/.venv/lib64/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/dstack/.venv/lib64/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/dstack/.venv/lib64/python3.10/site-packages/starlette/routing.py", line 73, in app
response = await f(request)
File "/dstack/.venv/lib64/python3.10/site-packages/fastapi/routing.py", line 301, in app
raw_response = await run_endpoint_function(
File "/dstack/.venv/lib64/python3.10/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
return await dependant.call(**values)
File "/dstack/src/dstack/_internal/server/routers/runs.py", line 131, in get_plan
run_plan = await runs.get_plan(
File "/dstack/src/dstack/_internal/server/services/runs/__init__.py", line 356, in get_plan
job_plans = await get_job_plans(
File "/dstack/src/dstack/_internal/server/services/runs/plan.py", line 165, in get_job_plans
fleet_model, instance_offers, backend_offers = await find_optimal_fleet_with_offers(
File "/dstack/src/dstack/_internal/server/services/runs/plan.py", line 344, in find_optimal_fleet_with_offers
candidate_fleet = fleet_model_to_fleet(candidate_fleet_model)
File "/dstack/src/dstack/_internal/server/services/fleets.py", line 863, in fleet_model_to_fleet
project_name=fleet_model.project.name,
File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/orm/attributes.py", line 566, in __get__
return self.impl.get(state, dict_) # type: ignore[no-any-return]
File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/orm/attributes.py", line 1086, in get
value = self._fire_loader_callables(state, key, passive)
File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/orm/attributes.py", line 1121, in _fire_loader_callables
return self.callable_(state, passive)
File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/orm/strategies.py", line 978, in _load_for_state
return self._emit_lazyload(
File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/orm/strategies.py", line 1079, in _emit_lazyload
return loading.load_on_pk_identity(
File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/orm/loading.py", line 694, in load_on_pk_identity
session.execute(
File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/orm/session.py", line 2365, in execute
return self._execute_internal(
File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/orm/session.py", line 2251, in _execute_internal
result: Result[Any] = compile_state_cls.orm_execute_statement(
File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement
result = conn.execute(
File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/engine/base.py", line 1416, in execute
return meth(
File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/sql/elements.py", line 523, in _execute_on_connection
return connection._execute_clauseelement(
File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/engine/base.py", line 1638, in _execute_clauseelement
ret = self._execute_context(
File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/engine/base.py", line 1843, in _execute_context
return self._exec_single_context(
File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/engine/base.py", line 1983, in _exec_single_context
self._handle_dbapi_exception(
File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/engine/base.py", line 2355, in _handle_dbapi_exception
raise exc_info[1].with_traceback(exc_info[2])
File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/engine/base.py", line 1964, in _exec_single_context
self.dialect.do_execute(
File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/engine/default.py", line 945, in do_execute
cursor.execute(statement, parameters)
File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 172, in execute
self._adapt_connection._handle_exception(error)
File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 323, in _handle_exception
raise error
File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 149, in execute
_cursor = self.await_(self._connection.cursor())
File "/dstack/.venv/lib64/python3.10/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 123, in await_only
raise exc.MissingGreenlet(
sqlalchemy.exc.MissingGreenlet: greenlet_spawn has not been called; can't call await_only() here. Was IO attempted in an unexpected place? (Background on this error at: https://sqlalche.me/e/20/xd2s)Additional information
No response
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working