[15] Usability enhancements to CONTROL-CORE

[GaneshPatil7517]

Hello @pradeeban Sir, @Rahuljagwani Sir, and @mayureshkothare Sir,
While drafting my proposal for [15] Usability Enhancements to CONTROL-CORE, I came across a few questions and would really appreciate your guidance.

Q1.The current configuration is scattered across four separate files (concore.tools, concore.octave, concore.mcr, concore.sudo). Would you be open to consolidating these into a single TOML/YAML configuration file (e.g., concore.toml) while keeping backward compatibility with the old files? Or is the separation intentional for modularity?

Q2.I noticed that mkconcore.py generates separate shell scripts for Windows (start /B, taskkill), POSIX (xterm -e, kill), and Docker (docker run/stop). Currently, users must choose one target at generation time with -t. Would it be valuable to generate a unified cross-platform runner (e.g., using Python's subprocess module) so that a single generated study can run on any OS without regeneration?

Q3.The CLI's validate command already performs 12 checks including shell injection prevention and ZMQ port conflict detection. For the usability enhancement, should I focus on making validation part of a pre-flight check before concore run, or should validation and running remain decoupled?

Q4. I see that Phase 1 protocol conformance tests exist in tests/test_protocol_conformance.py covering Python's parse_params, initval, and write_zmq. Phase 2 is planned for C++, Java, and MATLAB cross-language validation. Is completing Phase 2 conformance testing in scope for this usability project, or should I focus purely on installation/deployment UX?

Q5.The wire format uses Python ast.literal_eval() syntax. The C++ parser in concore.hpp and Java parser in TestLiteralEval.java re-implement this independently. Have there been interoperability bugs between language implementations? I'm thinking about whether a formal wire format specification document would help prevent future drift.

[GaneshPatil7517]

What I Understand So Far
CONTROL-CORE is a decentralized, broker-less simulation framework for closed-loop neuromodulation control systems. Unlike traditional DAG-based workflow tools (CWL, WDL), concore uniquely supports dicycles (feedback loops), which are essential for modeling neural control systems.

The core protocol uses a file-based data-sharing mechanism with a wire format based on Python's ast.literal_eval() syntax, and it supports development in Python, C++, MATLAB/Octave, Verilog, and Java. An optional ZMQ transport layer enables distributed execution. The workflow generator (mkconcore.py) parses GraphML files from the concore Editor (DHGWorkflow) and produces platform-specific scripts for Windows, POSIX, and Docker environments.

The CLI (concore_cli) already offers 7 commands - init, run, validate, inspect, status, stop, and watch - with comprehensive validation (12 checks including shell injection detection and ZMQ port conflict analysis). The Docker workflow generates per-node containers with shared-volume communication.

However, the multi-platform and multi-language nature of the project introduces friction: configuration is scattered across four separate files (concore.tools, concore.octave, concore.mcr, concore.sudo), tool paths must be set manually, platform-specific scripts must be regenerated when switching between Windows/POSIX/Docker, and the Docker workflow lacks modern orchestration (e.g., Docker Compose).

Proposed Approach: "Zero-to-Study in Minutes"
Goal: A new user should go from pip install concore to running their first study in under 5 minutes, on any platform.

Phase 1 - Installation & Configuration Automation (Weeks 1–4)
1. Unified Configuration: Consolidate the four separate config files into a single concore.toml, with a backward-compatibility layer that still reads the old files.
2. Auto-detection Wizard: A new concore setup command that scans the system for installed compilers/runtimes (g++, python3, iverilog, octave, Docker) using shutil.which() and writes the configuration automatically.
3. Platform Installers: A Homebrew formula for macOS, an install script for Linux, and a PowerShell installer for Windows - each installs concore and triggers auto-detection.
4. Actionable Error Messages: When a tool is missing (e.g., iverilog not found), display platform-specific installation instructions instead of a generic error.

Phase 2 - Container-First Workflow (Weeks 5–8)
1. Docker Compose Generation: Extend concore run --target docker to produce a docker-compose.yml with proper service definitions, shared volumes, and dependency ordering - replacing the current separate shell scripts.
2. Minimal Per-Language Dockerfiles: Slim images for each language (Python ~150 MB, C++ ~200 MB) vs. the current larger images that bundle unnecessary packages.
3. Container Runtime Auto-detection: Detect Docker Desktop, Podman, or Rancher Desktop and configure DOCKEREXE automatically.
4. One-Command Studio: docker run -p 8080:8080 concore/studio - a pre-built image containing the full development environment.
5. CI/CD Templates: GitHub Actions workflow for running concore studies in CI.

Phase 3 - Cross-Platform Runner & UX Polish (Weeks 9–12)
1. Unified Python Runner: A Python-based runner (using subprocess) that replaces platform-specific shell scripts, so a single generated study runs on Windows, macOS, and Linux without regeneration.
2. Pre-flight Validation: Automatic concore validate before every concore run, with actionable hints.
3. Interactive Init Wizard: concore init asks the user for language, number of nodes, and generates a working starter study.
4. Watch Improvements: Configurable pagination, CSV export, and an optional web-based dashboard.
5. Cross-Language Smoke Tests: Phase 2 protocol conformance - C++ and MATLAB nodes communicating with Python nodes.
6. Documentation: "Your First Study" tutorials for each platform (Windows, macOS, Ubuntu, Docker), plus a troubleshooting FAQ.

Why This Approach
1. Root-cause focused: Systematically removes every friction point in the new-user journey.
2. Incrementally valuable: Each phase delivers standalone improvements. Even if Phase 3 isn't 100% complete, Phases 1–2 alone transform the onboarding experience.
3. Fully backward-compatible: The unified config reads old files, Docker Compose is an addition (not a replacement), and the Python runner is opt-in.
4. Measurable: Success = time from pip install concore to first running study, benchmarked on Windows/macOS/Ubuntu. Target: < 5 minutes.

Given my existing deep involvement in the project, I'm confident I can hit the ground running from Day 1 and deliver meaningful usability improvements across all target platforms.
I look forward to your feedback on this approach.

[pradeeban]

Sounds good, @GaneshPatil7517.

[pradeeban]

@GaneshPatil7517 While making the usability enhancements, we should refrain from excessive changes to the backend logic or "old" files as that is a mere distraction. Bug fixes are great. But refactoring maybe not. Just to be careful.

We can figure out the priorities eventually. You can list your approach in your proposal and we will comment if something is listed out of order. Perhaps some of these enhancements should be an "if time permits."

[GaneshPatil7517]

@GaneshPatil7517 While making the usability enhancements, we should refrain from excessive changes to the backend logic or "old" files as that is a mere distraction. Bug fixes are great. But refactoring maybe not. Just to be careful.

We can figure out the priorities eventually. You can list your approach in your proposal and we will comment if something is listed out of order. Perhaps some of these enhancements should be an "if time permits."

Thank you for the guidance, @pradeeban Sir
I've restructured my proposal based on your feedback. The key changes:

1. All core deliverables (D1–D7) are purely additive - new files and new CLI commands only. No modifications to mkconcore.py, concore.py, concore_base.py, or any existing backend logic.

2. Bug fixes only for old files - for example, I just fixed a Windows UnicodeEncodeError in the concore doctor output (commit 888f143), but the fix was in the new doctor.py file itself, not in any old code.

3. Anything touching backend logic is now a stretch goal (S1–S3), explicitly gated behind "if time permits" and mentor approval:

• S1: Unified concore.toml config (would need to modify mkconcore.py)
• S2: Cross-platform Python runner (would replace shell/batch generation)
• S3: Pre-flight validation

4. The concore setup wizard writes the existing config files (concore.tools, concore.octave, etc.) that mkconcore.py already reads - so zero backend change is needed.

I'll list the full priority order in the proposal so you can flag anything that's out of order. The timeline is: Weeks 1–4 doctor+setup, Weeks 5–8 Docker Compose+slim images, Weeks 9–10 init wizard+docs, Weeks 11–12 stretch goals only if core is merged.

The concore doctor command is already implemented and pushed as PR #495 - 22 tests passing, CI green. Happy to iterate on it based on your review.

[pradeeban]

Sounds good. Also, if a new feature is implemented, we could create new test cases. No need to change all the previous examples to demonstrate the new feature. That will be a waste of effort. At some point, we could always document - which of these samples are for the new changes. Something to consider for the main GSoC coding period. Not now.

[GaneshPatil7517]

hello @pradeeban Sir i have one question:
When concore compose generates docker-compose.yml, should the containers communicate through shared Docker volumes (mimicking the current file-based edge protocol), or should we use Docker networking with the ZMQ layer that already exists in 0mq/? Shared volumes are simpler and keep backward compatibility, but ZMQ over the network would make the Docker workflow truly distributed -which seems like where CONTROL-CORE is heading

[GaneshPatil7517]

That's a fair point My current thinking is to support both modes start with shared Docker volumes by default (since that maps directly to the existing file-based edge protocol and requires zero changes to how nodes communicate), and then offer a --network flag that uses the ZMQ layer from 0mq/ for truly distributed setups. That way concore compose works out of the box for existing studies, and users who need multi-host execution can opt into ZMQ networking. Would love to hear @pradeeban's & @Rahuljagwani take on this approach.

[pradeeban]

I don't think we should overthink the backward compatibility of the past use cases. We just need to create some improved/ported use cases and document the new ones vs old ones (during the GSoC coding period, that is. No break changes at this point). We do not have to limit ourselves too much for backward compatibility and we should not spend too much time porting old use cases/examples either.

[GaneshPatil7517]

Thank you for clarifying, @pradeeban That makes sense I was being overly conservative with the "don't touch anything" approach. I'll focus on building the best possible new workflows (shared volumes by default, ZMQ --network flag for distributed) and document how they compare to the existing Docker scripts. I won't break existing behavior, but I also won't constrain myself to only wrapping around the old logic. Will update my proposal to reflect this.

[pradeeban]

No, you were right about not touching old stuff, such as ratc, testsou, etc. Not because we do not want to break them (even if you break, we can always go back to previous tags and releases). But because we cannot spend too much time fixing research products that are already published.

[GaneshPatil7517]

That makes perfect sense, @pradeeban. The rationale is clear time is better spent building improved tooling than fixing published research outputs that already serve their purpose. My proposal focuses entirely on new CLI commands (setup, compose, init) and documentation, leaving ratc/, testsou/, and other published studies untouched. Thanks for the clarification.......

[GaneshPatil7517]

hey @pradeeban, @mayureshkothare and @Rahuljagwani i have two question

1. I've been thinking about Section 2 of your paper where you emphasize reproducible science. The setup wizard and Docker Compose generator make concore easier to use today, but I want to make sure we don't create a situation where a study from 2026 can't be reproduced in 2031 because my tools have evolved. Should the generated files (concore.toml, docker-compose.yml, runner.py) be designed as immutable artifacts that anyone can run standalone, even if the generator is gone? That way, backward compatibility is baked into the generated output, not just the generator itself.

2. Following up on reproducibility: when the concore setup wizard and concore init wizards generate their artifacts, should they also auto-generate companion documentation files? For example, a SETUP.md that records which tools were detected, their versions, and the concore version used; and a schema comment inside concore.toml explaining each field. That way, five years later, someone reading the generated study would understand exactly how and why it was configured the way it was almost like reproducibility metadata baked in.

[pradeeban]

1. We keep tagging different "releases." Even when the tools evolve, thanks to GitHub, we will have previous versions that will still work. We cannot stall the progress to keep a 2021 paper alive and reproducible from the main branch. People can go back to the 1.x and 2.x releases as we head to concore-3.x or later in 2031. So I think it is ok.
2. That is not a bad idea.

[GaneshPatil7517]

@pradeeban Thank you for clarifying that makes complete sense. So the approach is:

1. Reproducibility via version pinning & releases - Researchers who need exact reproducibility pin concore to a specific release (e.g.,concore==1.0.0 in their study's requirements.txt). GitHub tags handle the rest as the project evolves to 3.x. Smart.

2. Generated artifacts should be clean but not "immortal"- Focus on making concore.toml, docker-compose.yml, and runner.py well-structured and documented (with schema comments), but don't over-engineer internal metadata. The self-documenting approach is enough.

3. SETUP.md is a nice-to-have- I'll make it a stretch goal for Weeks
   9-10 if time permits, not a core deliverable.

This actually simplifies my design. Instead of worrying about "will this still work in 2031," I can focus on making the new workflows (setup, compose, init) rock-solid and well-tested during the 350-hour window.

Updated plan:

• Weeks 1-4: concore setup + concore init --interactive with clean schema
• Weeks 5-8: concore compose with shared volumes + slim Dockerfiles
• Weeks 9-10: Pre-flight validation, tutorials, final polish
• Stretch: SETUP.md metadata if time allows

Does this align with your vision?

[pradeeban]

Pls make sure to include these in the proposal. We will evaluate and get back to you with comments/questions. I see there are many potential paths.

[GaneshPatil7517]

Pls make sure to include these in the proposal. We will evaluate and get back to you with comments/questions. I see there are many potential paths.

Thank you, @pradeeban . I have incorporated these points in the proposal, including version-pinning-based reproducibility, additive/self-documenting artifacts, and SETUP.md as a stretch goal. I also kept the plan flexible across multiple paths so we can finalize priorities with mentor feedback during bonding.

[GaneshPatil7517]

hello @pradeeban and @Rahuljagwani
If I deliver concore setup, concore init --interactive, and concore compose, what evidence would you want before we declare this the recommended default workflow over the current script-based path: lower first-run failure rate, shorter time-to-first-successful-study, fewer mentor support questions, or something else? I want to design the GSoC work so it produces measurable proof, not just new commands.

[pradeeban]

@GaneshPatil7517 We will iterate over the parameters during the community bonding period. You do not have to finalize everything during the application period. Feel free to simply list out the alternatives in your proposal, highlighting the approach you would think works best. We will consult the user and developer community (once/if the project is accepted) before jumping into coding.

[GaneshPatil7517]

@GaneshPatil7517 We will iterate over the parameters during the community bonding period. You do not have to finalize everything during the application period. Feel free to simply list out the alternatives in your proposal, highlighting the approach you would think works best. We will consult the user and developer community (once/if the project is accepted) before jumping into coding.

Thank you @pradeeban. I've updated my proposal to list the alternatives for each deliverable with my recommended approach highlighted exactly as you suggested. The proposal is submitted and ready for your review.

Happy to iterate during community bonding once you've had a chance to evaluate.

[GaneshPatil7517]

Hi @pradeeban, @Rahuljagwani, and @mayureshkothare
I have successfully submitted my GSoC 2026 proposal on the portal for Usability Enhancements to CONTROL CORE.
Thank you for your guidance and feedback throughout the drafting process.
I will continue contributing actively and refining implementation details based on your suggestions.

[GaneshPatil7517]

Hi @pradeeban, @Rahuljagwani, and @mayureshkothare
Quick update: the STUDY.json metadata PR has been merged successfully, and the related RFC issue is now closed.
I also updated my proposal to reflect your guidance on reproducibility via releases/version pinning, keeping generated artifacts self-documenting, and treating SETUP.md as stretch scope.
Next, I will continue with focused usability work (setup/init/compose path) and share weekly progress updates.