0.7 alpha 2

[Chenglong-MS]

This pull request introduces comprehensive Docker support for Data Formulator, improves developer experience, and updates documentation and configuration for easier deployment and internationalization. The main changes include adding Docker and Docker Compose files, updating documentation to guide users on Docker usage, enhancing environment and ignore files for containerization, and adding i18n-related dependencies. There are also updates to default model configurations and improvements to the README for clarity and developer onboarding.

Dockerization & Deployment:

• Added a production-ready Dockerfile and docker-compose.yml for building and running Data Formulator with persistent workspace storage and health checks. [1] [2]
• Introduced a .dockerignore file to optimize Docker build context and exclude unnecessary files.

Documentation & Developer Experience:

• Expanded DEVELOPMENT.md and updated README.md with detailed Docker usage instructions, quickstart guides, and clarified installation options. Also improved developer onboarding messaging. [1] [2] [3] [4]
• Added .vscode/settings.json to streamline Python development and common terminal tasks.

Configuration & Environment:

• Enhanced .env.template with new options for logging, data directory, and UI languages. Updated default LLM model lists for OpenAI, Azure, and Ollama providers. [1] [2] [3]

Frontend Internationalization:

• Added i18next, i18next-browser-languagedetector, and react-i18next to dependencies in package.json to support future UI localization. [1] [2]
• Updated package.json scripts to include vitest for testing.This pull request introduces Docker support for the Data Formulator project, making it easier to run the application without local Python or Node.js setup. It also refactors CORS handling for better security and configuration, and includes several improvements and fixes to agent logic and logging. The changes span new Docker-related files, Python backend adjustments, and updates to agent code for more accurate metadata and logging.

Dockerization and Development Environment:

• Added a production-ready Dockerfile with a multi-stage build to bundle the frontend and backend, and a docker-compose.yml for easy orchestration and persistent workspace data. .dockerignore is included to optimize builds. [1] [2] [3]
• Updated DEVELOPMENT.md with Docker usage instructions and caveats about sandboxing in containerized environments.
• Added VS Code workspace settings for Python interpreter and terminal tool auto-approval.

Backend API and CORS Handling:

• Centralized and configurable CORS headers using an @after_request handler in agent_routes.py, removing duplicated and insecure Access-Control-Allow-Origin: * headers from individual endpoints. Now, CORS is controlled via the CORS_ORIGIN environment variable. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10]

Agent Logic and Metadata Improvements:

• Updated semantic type references and metadata in agent prompts and logic for more precise field categorization (e.g., using "Category" instead of "Type", refining monetary and discrete numeric types). [1] [2] [3]
• Improved guidance in agent prompts for scale/domain inference and clarified open-ended measure handling.
• Enhanced logging in agent_data_rec.py and agent_data_transform.py to include LLM token usage and clarify timing breakdowns. [1] [2]
• Adjusted image detail levels in chart/image payloads for agents to optimize LLM input and output. [1] [2] [3] [4]

Other Improvements:

• Added more detailed logging in agent routines for better observability during LLM calls. [1] [2]
• Minor corrections and consistency improvements in agent prompt templates and comments. [1] [2] [3]

These changes collectively improve deployment flexibility, security, and developer experience, while also enhancing the correctness and observability of agent operations.

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of lines (20,000). Try reducing the number of changed lines and requesting a review from Copilot again.

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of lines (20,000). Try reducing the number of changed lines and requesting a review from Copilot again.

In general, to fix this category of problem you should avoid returning exception messages or stack traces (even “sanitized” ones) directly to clients. Instead, log the full exception on the server for debugging, and send only a generic, pre-defined error message, optionally with a simple status or code that does not contain implementation details.

For this specific case, the problematic flow is in test_model:

except Exception as e:
    logger.warning(f"Error testing model {content['model'].get('id', '')}: {e}")
    is_global = content['model'].get('is_global', False)
    result = {
        "model": content['model'],
        "status": 'error',
        "message": "Connection failed, please check server configuration" if is_global
                   else sanitize_model_error(str(e)),
    }

The best fix with minimal functional change is:

• Keep logging the error server-side (possibly upgrade to logger.exception so the stack trace is recorded in logs).
• Stop passing str(e) through sanitize_model_error to the client.
• Replace the exception-derived user message with a generic message that does not depend on e. We can still differentiate global vs non-global if needed, but both should use safe, static text.

We do not need to change sanitize_error_message or the alias sanitize_model_error themselves; they may be used safely elsewhere. We only need to change the construction of result in the except block of test_model in py-src/data_formulator/agent_routes.py. No new imports or helpers are required.

---

In general, to fix this kind of issue you should avoid returning exception details to the client. Instead, log the full exception (including stack trace) on the server and send back a generic, non-sensitive error message such as “An internal error has occurred” or something similarly high-level. If you want to expose some context to the client, it should be a controlled, static message, not derived from Exception text.

For this specific code, the simplest, non-breaking change is to adjust the open_workspace route’s except block. We should keep logging the rich error detail on the server, but stop using sanitize_error_message(str(e)) in the HTTP response. Instead, return a fixed, generic error string. Because other endpoints already use a more structured sanitize_db_error_message pattern, this endpoint can simply respond with a generic message like "Failed to open workspace" or "An internal server error occurred while opening the workspace." without affecting upstream logic (clients are already checking status and possibly message as a human-readable string). We do not need to modify sanitize_error_message itself for this finding.

Concretely:

• In py-src/data_formulator/tables_routes.py, inside open_workspace, update the except block at lines 179–181:
  • Keep logger.error(f"Failed to open workspace: {e}") so developers see the details.
  • Change the return jsonify(status="error", message=sanitize_error_message(str(e))), 500 call to instead return a fixed string, e.g. message="Failed to open workspace".
• No additional imports or helper methods are required; we are only changing the error message content.

---

In general, to fix information exposure via exceptions, you should avoid returning raw (or semi-sanitized) exception messages to clients. Instead, log the detailed exception, including stack trace, on the server, and send back a generic, stable error message that does not depend on str(e) or any other implementation detail. If you wish to surface some context (e.g., “invalid JSON”), use a static message or one derived from validated input, not from the exception object.

For this specific case in create_table in py-src/data_formulator/tables_routes.py, the best minimal fix is:

• In the except Exception as e: block around json.loads(raw_data), stop embedding sanitize_error_message(str(e)) into the client-facing message.
• Instead:
  • Log the full exception and stack trace on the server using the module logger, e.g. logger.exception(...) or logger.error(..., exc_info=True).
  • Return a generic error string such as "Invalid JSON data, it must be a list of dictionaries", without including str(e) at all.
• This preserves existing functionality (the route still returns a 400 error indicating invalid JSON) while removing any dependence on attacker-influenced exception text.

No changes are needed to sanitize_error_message itself for this fix.

Concretely:

• Edit the except Exception as e: block around lines 464–467 in create_table to:
  • Add a logging call to record the exception.
  • Replace the existing jsonify({"status": "error", "message": f"...{sanitize_error_message(str(e))}..."}) with a variant that does not reference e or sanitize_error_message.

No additional imports or new helper methods are required; you can reuse the existing logger instance.

---

General approach: avoid sending any content derived from the exception object back to the client. Instead, log the exception (with stack trace) on the server and return a generic, user-friendly error message that does not depend on e. The sanitizer can still be used elsewhere if needed, but for this endpoint we should not expose the parsed exception text.

Concrete best fix here: in parse_file’s except block (lines 540–542 in py-src/data_formulator/tables_routes.py), keep the logging statement as is (it already logs with exc_info=True), but replace the JSON response so that "message" is a fixed, generic string such as "Failed to parse file", independent of e. This change preserves existing functionality (client still receives a 400 with an error status) while eliminating any residual risk of leaking stack-trace or internal details via the exception string.

Required changes:

• File: py-src/data_formulator/tables_routes.py
  • In the parse_file function’s except Exception as e: block, update the return statement on line 542 to use a static message instead of sanitize_error_message(str(e)).
• No changes are required in py-src/data_formulator/sanitize.py for this specific issue.
• No new imports or helper methods are needed.

---