Databricks - Setup Guide

Databricks is a unified, open analytics platform for building, deploying, sharing, and maintaining enterprise-grade data, analytics, and AI solutions at scale. Fullstory customers with Anywhere: Warehouse can sync data to Databricks Unity Catalog with Ready to Analyze Views. This setup guide provides the following resources for getting started:

• Setting up Databricks (below)
• Setting up Databricks in Fullstory (below)
• Databricks developer documentation
• Ready to Analyze Views developer documentation
  • Data Model
  • Sample Queries
  • Sync Expectations
  • Data Dictionary

Setting up Databricks

Fullstory's Warehouse Data Sync for Databricks leverages Unity Catalog to manage data governance and Databricks SQL Warehouses to execute the loading processes.

The architecture utilizes Workload Identity Federation, allowing Fullstory to securely write data to a staging area and then load it into your Delta Lake managed tables without the need for long-lived static secrets.

Setting up Account level resources

1. Create a Service Principal.
   In your Databricks Account Console, create a new Service Principal. This identity will be used by Fullstory to authenticate and perform sync operations.

   

2. Create a Federation Policy for the Service Principal.
   To allow Fullstory to authenticate as your Service Principal, you must create a federation policy using the following configuration:

   • Issuer: https://accounts.google.com
   • Subject:
     • NA customers: 116984388253902328461
     • EU customers: 107589159240321051166
   • Audience: fullstory.com

   

3. Create a Workspace (optional).
   If you wish to use an existing workspace, skip this step. Otherwise, create a new workspace.
   Note down your workspace URL (e.g., https://xxx.x.gcp.databricks.com).
4. Assign Service Principal to the Workspace.
   Add the Service Principal created in Step 1 as a User of the workspace that you'd like to use.
   
   

Setting up Workspace level resources

Navigate to your workspace using the workspace URL and perform the following actions.

1. Identify or create your SQL Warehouse.
   Select an existing SQL warehouse or create a new one in your workspace. Note down the Warehouse ID.
   
   
2. Identify or create your Catalog.
   Select an existing Catalog or create a new one in your workspace. Note down the Catalog Name.
   
   

3. Configure Unity Catalog Permissions.
   For the Catalog you wish to use, assign the following permissions to your Service Principal:

   • USE CATALOG
   • USE SCHEMA
   • CREATE SCHEMA
   • CREATE TABLE

   

4. Set up Storage Credentials.
   Fullstory uses GCP as a staging layer. To ensure Databricks can read these files, a Storage Credential is used.

   1. In Unity Catalog, create a new Storage Credential.
   2. Ensure you select GCP Service Account as the credential type.
   3. Assign the READ FILES permission to your Service Principal.
   4. Note down the Credential Name and the GCP Service Account email associated with it.

   

   

   

Setting up Databricks in Fullstory

1. Log in to Fullstory and navigate to Settings > Anywhere > Warehouse.
2. Select Databricks from the list of available destinations.

3. Fill in the following fields using the information noted from the above steps:

   • Workspace URL
   • SQL Warehouse ID
   • Catalog Name
   • Storage Credential Name
   • GCP Service Account Email
   • Service Principal Application ID

   

4. Click Save and Test Connection.

Frequently Asked Questions

Does Fullstory use Databricks Delta Share?

No, Fullstory uses a traditional warehouse connection to sync data to Databricks, not Delta Share. With this approach:

• You own your data in your Databricks warehouse.
• You have full control over data retention and governance.
• You receive the same Ready to Analyze Views available for Snowflake, Redshift, and BigQuery.

What data does Fullstory sync to Databricks?

Fullstory syncs data using Ready to Analyze Views, which provides pre-built views optimized for analytics and business intelligence. This is the same data format used for Snowflake, Redshift, and BigQuery connections. For complete details about the data model, sample queries, and sync expectations, see the Ready to Analyze Views developer documentation.