Databricks with BioStudio

¶

Databricks ¶

Databricks is a unified analytics platform built on Apache Spark, designed for large-scale data processing. It integrates data engineering, data science, and business analytics into one environment. Delta Lake adds reliability with ACID transactions, while MLflow manages the machine learning lifecycle. The platform supports massive data volumes and complex workloads with scalability, offering collaborative notebooks and real-time co-editing to facilitate teamwork. Databricks integrates with AWS, Azure, Google Cloud, and various databases, providing enterprise-level security and compliance. It handles infrastructure management and is versatile for data warehousing, ETL, machine learning, and real-time analytics.
Databricks integrates seamlessly with Amazon S3, offering a robust environment for big data processing and analytics. It can read and write data directly to and from S3, allowing users to utilize S3 as a cost-effective and scalable storage solution. Databricks also allows you to mount S3 buckets to the Databricks file system (DBFS), making it easier to access and manage data stored in S3 as if it were part of the local file system.

BioStudio integration with Databricks ¶

BioStudio supports integration with Databricks, enabling users to leverage its capabilities for data exploration and job execution. BioStudio offers various tools to streamline these processes. Currently, BioStudio supports:

1. Mounting Cloud Object Storage: BioStudio can mount All - cloud object storage ( AWS, GCP, Azure ) on Databricks, syncing data from the user’s workspace.
2. SQL Warehouse Connection: BioStudio also supports connections to SQL warehouses for optimized data querying and analysis.
3. Cluster Connection: Users can connect to Databricks clusters for scalable computing.

Setup Databricks connection with BioStudio. ¶

                         +=============================+
                         |      User's Workspace       |
                         +=============================+
                                     |
                                     | Data Sync
                                     v
                         +=============================+
                         |   BioStudio Integration     |
                         |        with Databricks      |
                         +=============================+
                           |            |            |
                           |            |            |
     +---------------------+            |            +---------------------+
     |                                  |                                  |
     v                                  v                                  v
+-----------------+             +-----------------+             +-----------------+
| Cloud Object    |             | SQL Warehouse   |             | Databricks      |
| Storage (AWS,   |             | Connection      |             | Clusters        |
| GCP, Azure)     |             |                 |             |                 |
|                 |             |                 |             |                 |
+-----------------+             +-----------------+             +-----------------+
       |                               |                                |
       |                               |                                |
       v                               v                                v
+---------------------+       +-----------------+             +-----------------+
|  Amazon S3          |       | SQL Data        |             | EC2 Instances   |
|                     |       | Querying/       |             | (Configured     |
|                     |       | Analysis        |             | for HPC with    |
|                     |       |                 |             | FSx for Lustre) |
+---------------------+       +-----------------+             +-----------------+
       |
       | Connected via FSx for Lustre
       |
       v
+---------------------+
| High-Performance    |
| Parallel Workflows  |
| (SLURM/SGE Paradigm)|
+---------------------+

Mounting Cloud Object Storage - related to Databricks with BioStudio. ¶

🔆 This bucket is associated with Databricks.

🔆 This bucket is associated with Databricks.

🔆 Mount S3 bucket.

1. Login to BioStudio.
2. Click to Cloud Storage Icon.

3. Click to Create Connection.
4. It will connect to S3 bucket and mount.

5. Mount point can be view by clicking System Terminal.

6. Create one file.

7. It is available with Databricks.

Databricks Connection with BioStudio Visual Studio. ¶

🔆 Cluster Connection
Complete databricks including resouces can be connected to BioStudio using token.

1. Generate token to connect to BioStudio by clicking setting.

2. Select user -> Developer -> Access tokens -> Generate new token.

3. Open the VS Code application tool from BioStudio.

4. Configure databricks connection.

[Bioturing-databrick]
host = https://dbc-454984-e221564448e.cloud.databricks.com
token = dap2165411316rdtffdbvrdftgyrtfhgrd321654646

5. Provide host name.

6. Edit Databricks profile.

7. Provide all values.

8. Configuration has been saved.
9. Click to Databricks extention -> Configure Databricks to see the connectivity.

10. Select saved profile.

11. Connection succeed.

12. User connection is available with Databricks.
    
    
    
    

BioStudio connection with Databricks SQL warehouses. ¶

🔆 SQL Warehouses

• We need SQL Warehouse to did operartions like execute query.

1. Let’s start SQL warehouses server.

2. SQL warehouses server started.

3. We will use python to execute query with SQL Warehouses.

4. Generate token to execute query from BioStudio.

5. Create conda environment / kernel. Select Kernels -> fill all the values.

6. Check environment is ready to use.

7. Activate environment.

ub-lalit-e33d9fbdb4e96d0@colabdev-868c4b58bb-ctb8k:~$ conda env list
# conda environments:
#
databrick                /data/ub-lalit-e33d9fbdb4e96d0/.conda/envs/databrick
databrick-sqlwarehose     /data/ub-lalit-e33d9fbdb4e96d0/.conda/envs/databrick-sqlwarehose
base                     /miniconda/user

ub-lalit-e33d9fbdb4e96d0@colabdev-868c4b58bb-ctb8k:~$ conda activate databrick-sqlwarehose
(databrick-sqlwarehose) ub-lalit-e33d9fbdb4e96d0@colabdev-868c4b58bb-ctb8k:~$ 

8. Install the Databricks SQL Connector for Python library on your development machine by running.

pip install databricks-sql-connector

9. Create new notebook.

10. Select Kernel.

11. Run notebook.

BioStudio connection with Databricks Compute. ¶

🔆 Compute

1. Creater cluster.

• Cluster is ready now.

2. Attached cluster to BioStudio.

3. Cluster Connected.

4. We can select Kernel to run code.

5. Databrick power tool also available.