How can I install drivers like ODBC when deploying my code to Dagster+ Serverless?

[garethbrickman]

When trying to deploy my code to Dagster+ Serverless I encounter this error during my CI/CD process: ImportError: libodbc.so.2: cannot open shared object file: No such file or directory

I think this is because I am using pyodbc(used for Microsoft SQL Server) as a dependency in my code and it requires special drivers be available in the environment it's trying to be installed in.

[garethbrickman]

You can install that required driver either as part of your CI/CD process by using a lifecycle hook pre-install script, or by creating and using your own custom base image.

To use a lifecycle hook pre-install script requires disabling PEX deploys. If using the GitHub Actions workflow, comment out this line to disable PEX-based deploys:

env:
  DAGSTER­_CLOUD­_URL: ...
  DAGSTER­_CLOUD­_API­_TOKEN: ...
  # ENABLE­_FAST­_DEPLOYS: 'true'

Then in the root of your repo you can add a shell script named dagster_cloud_pre_install.sh that contains the instructions to install the required driver. Example:

#!/bin/bash
set -e

echo "Starting dagster_cloud_pre_install.sh..."

# Update package lists
apt-get update

# Install the unixODBC development package required by pyodbc
apt-get install -y unixodbc-dev

echo "Pre-install script completed successfully."

Note that this script requires execute permission. To ensure that, run these commands before committing and pushing the file to your repo:

git add dagster_cloud_pre_install.sh
git update-index --chmod=+x dagster_cloud_pre_install.sh

[billy-moore-98]

Just jumping on here to note that if also using a dbt integration for a SQL Server db on serverless deployments, it might be useful to add the dagster_cloud_post_install.sh script to package your dbt code. This helped our use case.

 
#!/bin/bash
dagster-dbt project prepare-and-package --file ./dags/warehouse/project.py 

I note in the documentation for using dbt with Dagster+ it provides an update to the deployment yaml, but only for pex enabled deploys, which is disabled in this instance.