Skip to content

Compute Worker - Fix submission files duplication#2285

Open
ihsaan-ullah wants to merge 10 commits intodevelopfrom
compute_worker_duplicate_submissions
Open

Compute Worker - Fix submission files duplication#2285
ihsaan-ullah wants to merge 10 commits intodevelopfrom
compute_worker_duplicate_submissions

Conversation

@ihsaan-ullah
Copy link
Collaborator

@ihsaan-ullah ihsaan-ullah commented Mar 21, 2026

Description

This PR has the following updates:

.env_sample

  • updated for minor changes to WORKER_BUNDLE_URL_REWRITE

src/apps/competitions/tasks.py

  • updated to send submission during both ingestion and scoring, and some other minor updates for clarity

computer_worker.py

  • separate class SubmissionStatus for submission statuses and related updates
  • separate class ProgramKind for program kind and related updates
  • introduced a new class Settings that gathers all env variables. This class is now used all over the code for accessing settings. NOTE: This can be further improved to convert strings to booleans so that we don't use == "true" or `== "false".
  • organized imports
  • renamed program to scoring_program
  • update code start function to clarify which functions will run during ingestion and during scoring, and separated submission and scoring program clearly
  • made submission available ragardless of ingestion or scoring in app/ingested_program
  • rewrote _run_program_directory function to simplify code and logs
  • moved some code from _run_program_directory to a new function _create_container for reusability and clarity
  • minor changes to watch_detailed_results function to avoid looping forever
  • replaced container_id (undeclared) by container.get("Id") in _run_container_engine_cmd`
  • [TO DISCUSS] commented some code (that was introduced in Worker status to FAILED instead of SCORING or FINISHED in case of failure #2030 ) at the end of the start function that was failing submission even if everything went well. Need to discuss this and update the code

Issues this PR resolves

A checklist for hand testing

  • Upload this modified mini autoML bundle to check in ingestion logs and scoring logs that submission files are accessible Modified_Bundle.zip
  • use this submission to test the bundle submittion.zip
  • Carefully check the code to make sure there are no bugs

Checklist

  • Code review by me
  • Hand tested by me
  • I'm proud of my work
  • Code review by reviewer
  • Hand tested by reviewer
  • CircleCi tests are passing
  • Ready to merge

…e submission during both ingestion and scoring
@ihsaan-ullah ihsaan-ullah marked this pull request as draft March 21, 2026 23:50
@ihsaan-ullah ihsaan-ullah marked this pull request as ready for review March 22, 2026 11:24
@ihsaan-ullah ihsaan-ullah changed the title Compute Worer - Fix submission files duplication Compute Worker - Fix submission files duplication Mar 22, 2026
@Didayolo Didayolo mentioned this pull request Mar 23, 2026
20 tasks
@Didayolo Didayolo marked this pull request as draft March 23, 2026 13:30
@Didayolo Didayolo requested review from Didayolo and ObadaS March 24, 2026 14:41
@Didayolo
Copy link
Member

  • Put the duplication on scoring (ingested program --> results) to avoid breaking current competition design
  • In the future we can optimize this more

@ihsaan-ullah ihsaan-ullah marked this pull request as ready for review March 24, 2026 18:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants