Skip to content

Conversation

renardeinside
Copy link
Contributor

This PR introduces the application skeleton, based on React, Vite, TailwindCSS + ShadCN (frontend) and FastAPI in the backend.

Copy link

github-actions bot commented Aug 28, 2025

❌ 312/313 passed, 4 flaky, 1 failed, 1 skipped, 2h45m32s total

❌ test_e2e_workflow: pyspark.errors.exceptions.connect.SparkConnectGrpcException: () BAD_REQUEST: session_id is no longer usable. Generate a new session_id by detaching and reattaching the compute and then try again [sessionId=38b1a963-075a-4b6b-b459-5e37cd569e56, reason=INACTIVITY_TIMEOUT]. (requestId=91a94f33-912b-4c01-8101-26c3a84762c1) (12m4.725s)
pyspark.errors.exceptions.connect.SparkConnectGrpcException: () BAD_REQUEST: session_id is no longer usable. Generate a new session_id by detaching and reattaching the compute and then try again [sessionId=38b1a963-075a-4b6b-b459-5e37cd569e56, reason=INACTIVITY_TIMEOUT]. (requestId=91a94f33-912b-4c01-8101-26c3a84762c1)
[gw1] linux -- Python 3.12.11 /home/runner/work/dqx/dqx/.venv/bin/python
11:59 INFO [databricks.labs.dqx.installer.config_provider] Please answer a couple of questions to provide TEST_SCHEMA DQX run configuration. The configuration can also be updated manually after the installation.
11:59 INFO [databricks.labs.dqx.installer.install] Installing DQX v0.9.3+1520250918115943
11:59 INFO [databricks.labs.dqx.installer.dashboard_installer] Creating dashboards...
11:59 INFO [databricks.labs.dqx.installer.dashboard_installer] Reading dashboard assets from /home/runner/work/dqx/dqx/src/databricks/labs/dqx/queries/quality/dashboard...
11:59 INFO [databricks.labs.dqx.installer.dashboard_installer] Using 'main.dqx_test.output_table' output table as the source table for the dashboard...
11:59 WARNING [databricks.labs.lsql.dashboards] Parsing : No expression was parsed from ''
11:59 WARNING [databricks.labs.lsql.dashboards] Parsing unsupported field in dashboard.yml: tiles.00_2_dq_error_types.hidden
11:59 WARNING [databricks.labs.lsql.dashboards] Parsing : No expression was parsed from ''
11:59 INFO [databricks.labs.dqx.installer.dashboard_installer] Installing 'DQX_Quality_Dashboard' dashboard in '/Users/3fe685a1-96cc-4fec-8cdb-6944f5c9787e/.fylX/dashboards'
11:59 INFO [databricks.labs.dqx.installer.workflow_installer] Creating new job configuration for step=profiler
11:59 INFO [databricks.labs.dqx.installer.workflow_installer] Creating new job configuration for step=quality-checker
11:59 INFO [databricks.labs.dqx.installer.workflow_installer] Creating new job configuration for step=e2e
11:59 INFO [databricks.labs.dqx.installer.workflow_installer] Updating configuration for step=e2e job_id=463903331719231
11:59 INFO [databricks.labs.dqx.installer.workflow_installer] Updating configuration for step=e2e job_id=463903331719231
11:59 INFO [databricks.labs.dqx.installer.workflow_installer] Updating configuration for step=e2e job_id=463903331719231
11:59 INFO [databricks.labs.dqx.installer.install] Installation completed successfully!
11:59 INFO [databricks.labs.dqx.installer.workflow_installer] Started e2e workflow: https://DATABRICKS_HOST#job/463903331719231/runs/42656438113872
12:11 INFO [databricks.labs.dqx.installer.workflow_installer] Completed e2e workflow run 42656438113872 with state: RunResultState.SUCCESS
12:11 INFO [databricks.labs.dqx.installer.workflow_installer] Completed e2e workflow run 42656438113872 duration: 0:11:36.335000 (2025-09-18 11:59:58.413000+00:00 thru 2025-09-18 12:11:34.748000+00:00)
12:11 INFO [databricks.labs.dqx.installer.workflow_installer] ---------- REMOTE LOGS --------------
12:11 INFO [databricks.labs.dqx:prepare] DQX v0.9.3+1520250918115943 After workflow finishes, see debug logs at /Workspace/Users/3fe685a1-96cc-4fec-8cdb-6944f5c9787e/.fylX/logs/e2e/run-42656438113872-0/prepare.log
12:11 INFO [databricks.labs.dqx.quality_checker.e2e_workflow:prepare] End-to-end: prepare start for run config: TEST_SCHEMA
12:11 INFO [databricks.labs.dqx:finalize] DQX v0.9.3+1520250918115943 After workflow finishes, see debug logs at /Workspace/Users/3fe685a1-96cc-4fec-8cdb-6944f5c9787e/.fylX/logs/e2e/run-42656438113872-0/finalize.log
12:11 INFO [databricks.labs.dqx.quality_checker.e2e_workflow:finalize] End-to-end: finalize complete for run config: TEST_SCHEMA
12:11 INFO [databricks.labs.dqx.quality_checker.e2e_workflow:finalize] For more details please check the run logs of the profiler and quality checker jobs.
12:11 INFO [databricks.labs.dqx.installer.workflow_installer] ---------- END REMOTE LOGS ----------
12:11 INFO [databricks.labs.dqx.checks_storage] Loading quality rules (checks) from '/Users/3fe685a1-96cc-4fec-8cdb-6944f5c9787e/.fylX/checks.yml' in the workspace.
11:59 INFO [databricks.labs.dqx.installer.config_provider] Please answer a couple of questions to provide TEST_SCHEMA DQX run configuration. The configuration can also be updated manually after the installation.
11:59 INFO [databricks.labs.dqx.installer.install] Installing DQX v0.9.3+1520250918115943
11:59 INFO [databricks.labs.dqx.installer.dashboard_installer] Creating dashboards...
11:59 INFO [databricks.labs.dqx.installer.dashboard_installer] Reading dashboard assets from /home/runner/work/dqx/dqx/src/databricks/labs/dqx/queries/quality/dashboard...
11:59 INFO [databricks.labs.dqx.installer.dashboard_installer] Using 'main.dqx_test.output_table' output table as the source table for the dashboard...
11:59 WARNING [databricks.labs.lsql.dashboards] Parsing : No expression was parsed from ''
11:59 WARNING [databricks.labs.lsql.dashboards] Parsing unsupported field in dashboard.yml: tiles.00_2_dq_error_types.hidden
11:59 WARNING [databricks.labs.lsql.dashboards] Parsing : No expression was parsed from ''
11:59 INFO [databricks.labs.dqx.installer.dashboard_installer] Installing 'DQX_Quality_Dashboard' dashboard in '/Users/3fe685a1-96cc-4fec-8cdb-6944f5c9787e/.fylX/dashboards'
11:59 INFO [databricks.labs.dqx.installer.workflow_installer] Creating new job configuration for step=profiler
11:59 INFO [databricks.labs.dqx.installer.workflow_installer] Creating new job configuration for step=quality-checker
11:59 INFO [databricks.labs.dqx.installer.workflow_installer] Creating new job configuration for step=e2e
11:59 INFO [databricks.labs.dqx.installer.workflow_installer] Updating configuration for step=e2e job_id=463903331719231
11:59 INFO [databricks.labs.dqx.installer.workflow_installer] Updating configuration for step=e2e job_id=463903331719231
11:59 INFO [databricks.labs.dqx.installer.workflow_installer] Updating configuration for step=e2e job_id=463903331719231
11:59 INFO [databricks.labs.dqx.installer.install] Installation completed successfully!
11:59 INFO [databricks.labs.dqx.installer.workflow_installer] Started e2e workflow: https://DATABRICKS_HOST#job/463903331719231/runs/42656438113872
12:11 INFO [databricks.labs.dqx.installer.workflow_installer] Completed e2e workflow run 42656438113872 with state: RunResultState.SUCCESS
12:11 INFO [databricks.labs.dqx.installer.workflow_installer] Completed e2e workflow run 42656438113872 duration: 0:11:36.335000 (2025-09-18 11:59:58.413000+00:00 thru 2025-09-18 12:11:34.748000+00:00)
12:11 INFO [databricks.labs.dqx.installer.workflow_installer] ---------- REMOTE LOGS --------------
12:11 INFO [databricks.labs.dqx:prepare] DQX v0.9.3+1520250918115943 After workflow finishes, see debug logs at /Workspace/Users/3fe685a1-96cc-4fec-8cdb-6944f5c9787e/.fylX/logs/e2e/run-42656438113872-0/prepare.log
12:11 INFO [databricks.labs.dqx.quality_checker.e2e_workflow:prepare] End-to-end: prepare start for run config: TEST_SCHEMA
12:11 INFO [databricks.labs.dqx:finalize] DQX v0.9.3+1520250918115943 After workflow finishes, see debug logs at /Workspace/Users/3fe685a1-96cc-4fec-8cdb-6944f5c9787e/.fylX/logs/e2e/run-42656438113872-0/finalize.log
12:11 INFO [databricks.labs.dqx.quality_checker.e2e_workflow:finalize] End-to-end: finalize complete for run config: TEST_SCHEMA
12:11 INFO [databricks.labs.dqx.quality_checker.e2e_workflow:finalize] For more details please check the run logs of the profiler and quality checker jobs.
12:11 INFO [databricks.labs.dqx.installer.workflow_installer] ---------- END REMOTE LOGS ----------
12:11 INFO [databricks.labs.dqx.checks_storage] Loading quality rules (checks) from '/Users/3fe685a1-96cc-4fec-8cdb-6944f5c9787e/.fylX/checks.yml' in the workspace.
12:11 INFO [databricks.labs.dqx.installer.install] Deleting DQX v0.9.3+1520250918115943 from https://DATABRICKS_HOST
12:11 INFO [databricks.labs.dqx.installer.workflow_installer] Removing job_id=890803048336484, as it is no longer needed
12:11 INFO [databricks.labs.dqx.installer.workflow_installer] Removing job_id=463100849698510, as it is no longer needed
12:11 INFO [databricks.labs.dqx.installer.workflow_installer] Removing job_id=463903331719231, as it is no longer needed
12:11 INFO [databricks.labs.dqx.installer.install] Uninstalling DQX complete
[gw1] linux -- Python 3.12.11 /home/runner/work/dqx/dqx/.venv/bin/python

Flaky tests:

  • 🤪 test_quality_checker_workflow_with_custom_check_func_rel_path (3m0.599s)
  • 🤪 test_quality_checker_no_output_config_configured (3m12.921s)
  • 🤪 test_e2e_workflow_serverless (4m22.227s)
  • 🤪 test_e2e_workflow (7m28.185s)

Running from acceptance #2393

@mwojtyczka mwojtyczka requested a review from Copilot August 29, 2025 10:28
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces the application skeleton for a full-stack web application with a React frontend and FastAPI backend. The implementation includes a modern tech stack with TailwindCSS, ShadCN UI components, and comprehensive development tooling.

  • Adds React-based frontend with TanStack Router, React Query, and ShadCN UI components
  • Implements FastAPI backend with user authentication, database configuration, and API endpoints
  • Sets up development environment with TypeScript, ESLint, Prettier, and comprehensive build tooling

Reviewed Changes

Copilot reviewed 43 out of 48 changed files in this pull request and generated 2 comments.

File Description
ui/* Complete React frontend setup with routing, state management, and UI components
src/databricks/labs/dqx/app/* FastAPI backend with authentication, configuration, and API endpoints
pyproject.toml Added app dependencies for FastAPI, uvicorn, and database libraries
docs/dqx/docs/dev/contributing.mdx Added development setup documentation for the new app

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.


## Contributing to the DQX App

DQX App is a web application that provides a user interface for DQX. It is built with React and FastAPI.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we mention Lakehouse app as deployment mechanism to give a bit more context?

@@ -78,6 +86,7 @@ dependencies = [
"pylint~=3.3.1",
"pylint-per-file-ignores~=1.3",
"pylint-pytest==2.0.0a0",
"pylint_pydantic",
Copy link
Contributor

@mwojtyczka mwojtyczka Aug 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pin the version

return f"postgresql://{self.user}:{self.password}@{self.host}:{self.port}/{self.database}?sslmode=require"


class DatabaseManager(BaseModel):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should not belong to config

@@ -0,0 +1,5 @@
import logoUrl from "/logo.svg";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we use existing logo from docs, or does it have to live inside the app?


Add a new `.env` file in the root of the project with the following content:
```
DQX_DEV_TOKEN=<your-databricks-token>
Copy link
Contributor

@mwojtyczka mwojtyczka Sep 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should recommend using oauth. Earlier in the docs we have a note on this.
Is this DQX_DEV_TOKEN correct? it's not a standard evn var and it is not used anywhere in this PR

@@ -47,6 +47,14 @@ pii = [
# This may be required for the larger models due to Databricks connect memory limitations.
# The models cannot be delcared as dependency here buecase PyPI does not support URL-based dependencies which would prevent releases.
]
app = [
"fastapi",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should pin the versions


N.b. - The `DQX_DEV_TOKEN` is a Databricks token that you can generate in the Databricks UI or via the Databricks CLI:
```bash
databricks token create --lifetime-seconds 3600 --comment "DQX App Development Token"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
databricks token create --lifetime-seconds 3600 --comment "DQX App Development Token"
databricks tokens create --lifetime-seconds 3600 --comment "DQX App Development Token"


2. Run the backend in development mode:
```bash
uvicorn src.databricks.labs.dqx.app.app:app --reload
Copy link
Contributor

@mwojtyczka mwojtyczka Sep 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only way i could make it work on my machine was to enter the shell hatch shell or run it, or hatch run python -m uvicorn src.databricks.labs.dqx.app.app:app --reload, otherwise i was getting zsh: command not found: uvicorn

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants