Develop Databricks App In Local Environment

Introduction

Working with Databricks apps locally allows developers to test, debug, and enhance their projects effectively. By leveraging tools like the Databricks CLI and VS Code, you can integrate your local development workflow seamlessly with the Databricks environment. In this guide, we’ll walk you through the step-by-step process to set up and work with Databricks apps in a local development environment.

Prerequisites

Before you begin, ensure the following are in place:

Install Databricks CLI
VS Code (Visual Studio Code)
Databricks access to create and manage apps
Databricks Token for authentication

Steps to Set Up Databricks Apps Locally

1. Create a Databricks App

Log in to your Databricks portal.
Navigate to Compute > Apps Tab > Create App.
Follow the on-screen instructions to create the app.

For a detailed walkthrough, refer to this guide.

2. Verify Databricks CLI Version

Open VS Code and launch the terminal. Run the following command to ensure the Databricks CLI is installed and working correctly:

  databricks -v

3. Export the Project to Local Environment

Use the Databricks CLI to export the project from the Databricks workspace to your local environment. Replace <workspace path> with your project’s workspace path:

  databricks workspace export-dir --workspace path for the project--

4. Set Up a Python Virtual Environment

Create a virtual environment for the project in VS Code. Use the following command:

  python3 -m venv .venv

5. Install Required Python Packages

Install the dependencies required for your project by running:

  pip install -r requirements.txt

6. Configure Databricks in VS Code

Install the Databricks extension in VS Code (if not already installed).
Once installed, select the Databricks extension and connect it to your project.
Ensure that a .databrickscfg configuration file exists in your project’s root directory to manage profiles. Below is an example configuration:

[ath-app-profile]

host = https://adb-4342637126508250.10.azuredatabricks.net

account_id = 492-6c53-4f8d-b9d1-b2421f7e57b0

client_id = 5d4ef9da-a48c1-b240-af863cecc7e7

client_secret = do187a69e7094d6b8afc76f661f025

[ath-app-token-profile]

host = https://adb-434263718250.10.azuredatabricks.net

token = dapife710791603a1019d17b0389e-3

7. Test the Application Locally

Run your application locally by executing the following command:

  python app.py

Check your application by accessing the generated URL to ensure it runs as expected.

8. Sync Code Back to Databricks

Once you’ve tested and finalized your changes, sync the code back to the Databricks environment using the following command:

  databricks sync --watch . -<-path of the databricks location->-

Conclusion

By following the above steps, you can effectively work on Databricks apps in your local environment. This approach simplifies testing and debugging while providing a seamless workflow for syncing back to the Databricks environment. For more details, check out the comprehensive Databricks Apps Guide.