Amazon Redshift

Amazon Redshift is a cloud-based data warehouse. With our Data Direct integration, you can sync Fullstory's structured, behavioral event data directly to serverless or provisioned Redshift destinations.

Note: This integration works with Redshift Serverless and Provisioned Redshift. Redshift Spectrum is not supported. 

Enabling the Redshift integration (AWS)

Terraform Infrastructure as Code (IaC)

Fullstory provides a terraform module to help simplify the setup of the necessary permissions. See the module docs for more:

https://registry.terraform.io/modules/fullstorydev/fullstory-redshift-setup/aws/latest

Manual Configuration

Notes:
- To experience all of the benefits of this integration, a paid version of Redshift is required.
- Fullstory should only be granted access to read/write the data that we will be managing as part of this sync.

The following guide assumes a provisioned or serverless Redshift cluster exists. If you do not have an existing cluster, please create one before continuing.

You should create a database in your Redshift cluster for Fullstory to insert data into. You can call this fullstory or whatever name suits your needs. Also, be sure you can create users and grant permissions on that database.

To ensure all sensitive data remains secure, Fullstory uses AWS web identity federation to access S3 and Redshift resources required for Data Direct. This role and the associated Redshift database user should not be given permission to access any other data.

Redshift Serverless Setup Instructions

If you use a provisioned Redshift cluster, please skip to Redshift Provisioned Setup Instructions

Redshift Serverless uses namespaces for managing database objects, users, and workgroups for controlling the compute resources. A database with its associated namespace and workgroup must be created for Data Direct.

Step 1: Obtain Redshift configuration information

You will need this information in the following steps, please copy the following identifiers into a scratch notepad for later from the Serverless dashboard

• Workgroup
• Workgroup ARN
• Endpoint

As an example:

Workgroup: fullstory-aws-redshift-wg
Workgroup ARN: arn:aws:redshift-serverless:us-west-1:123456789012:workgroup/e32a6cd1-d345-2e77-12ae-cd1234a765aa
Endpoint: fullstory-aws-redshift-wg.123456789012.us-west-1.redshift-serverless.amazonaws.com:5439/fullstory_aws_redshift_db

Redshift Provisioned Setup Instructions

If you use a serverless Redshift cluster, please see Redshift Serverless Setup Instructions.

An Amazon Redshift data warehouse is a collection of computing resources called nodes, which are organized into a group called a cluster. Each cluster runs an Amazon Redshift engine and contains one or more databases. A database with the cluster needs to be created for Data Direct.

Step 1: Obtain Redshift configuration information

You will need this information in the following steps. Please copy the following identifiers into a scratch notepad for later from the Provisioned clusters dashboard.

• Cluster identifier
• Database ARN
• Endpoint

The Database ARN can be formed from your region, account ID, cluster identifier and database name.

arn:aws:redshift:<region>:<account-id>:dbname:<cluster-identifier>:<database-name>

As an example:

Cluster identifier: redshift-cluster-test
Database ARN: arn:aws:redshift:us-west-1:123456789012:dbname:redshift-cluster-test/fullstory_aws_redshift_db
Endpoint: redshift-cluster-test.abcd.us-east-1.redshift.amazonaws.com:5439/fullstory

Configure access to your cluster

Step 1: Configure Public Access

For Fullstory to connect, your Redshift cluster must be publicly accessible. To enable this option, visit your Redshift cluster detail page, then:

1. Properties > Network and security settings > Edit > Publicly accessible.
2. Click Turn on Publicly accessible.

You must ensure an internet gateway and route table are properly configured for your VPC. Please see this AWS guide for more info on configuring an Internet gateway and route table.

Step 2: Configure security group to allow Fullstory's IP addresses

We will need to allow Fullstory’s IPs to connect to your Redshift cluster to sync data. You must also modify the security group attached to your Redshift cluster by allowing Redshift type traffic from a custom source.

1. Navigate to your VPC security group by clicking the link on your Redshift cluster’s detail page.
2. Properties > Network and security settings > VPC security group.
3. Click Edit inbound rules, then Add rule.
4. Select Redshift for the type, Custom for the source, and add the following IP ranges according to your data realm:

For North American customers:

8.35.195.0/29

For European customers:

34.89.210.80/29

5. Finally, click Save rules.

Setup IAM Roles and Policies

Fullstory uses AWS Web Identity Federation to access S3 and Redshift resources required for Data Direct to ensure all sensitive data remains secure. This role and the associated Redshift database user should not be permitted to access any other data.

Fullstory's services will assume the following role, and they will be able to upload data to S3 and load it into Redshift.

Step 1: Setup assumable role

1. Navigate to IAM roles.
2. Roles > Create Role.
3. Choose Web Identity.
4. For Identity Provider, select Google.
5. For Audience, please insert one of the following values according to your data realm:
   
   For North American customers:

116984388253902328461

     For European customers:

107589159240321051166

7. Click Next.
8. You do not need to add any permissions. Click Next.
9. Give the role a name.

Keep in mind that the name will be used in a SQL context, therefore the name must adhere to proper SQL formatting. Do not use "-" in the name. Learn more.

Example:

fullstory_redshift_role

10. Click Create role.
11. Navigate to the role you just created, then:
12. Select Trust relationships and replace the contents with the following to allow the Redshift service to assume this role so that it can load files from S3. Remember to replace the audience value with the one required for your data realm.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "accounts.google.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "accounts.google.com:aud": "<audience for your Fullstory region from above>"
        }
      }
    },
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "redshift.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

14. Click Update Policy.

Step 2: Setup policy to get cluster credentials

Create a policy that allows Fullstory services to authenticate with the Redshift cluster.

1. Navigate to IAM, then Policies > Create Policy.
2. With Policy Editor, select JSON and paste the apprpriate policy below (first policy is for serverless, second is for provisioned)

Be sure to insert your Redshift workgroup ARN (serverless) or Redshift database ARN (provisioned) that you recorded above.

{
  "Statement": [
    {
      "Action": "redshift-serverless:GetCredentials",
      "Effect": "Allow",
      "Resource": "arn:aws:redshift-serverless:<region>:<account_id>:workgroup/e32a6cd1-d345-2e77-12ae-cd1234a765aa"
    }
  ],
  "Version": "2012-10-17"
}

{
  "Statement": [
    {
      "Action": "redshift:GetClusterCredentialsWithIAM",
      "Effect": "Allow",
      "Resource": "arn:aws:redshift:<region>:<account-id>:dbname:<cluster-identifier>:<database-name>"
    }
  ],
  "Version": "2012-10-17"
}

4. Give the policy a name, then click Create policy.

Step 3: Attach the policy to the role

Now that we have created a role and a policy, we need to attach the policy to the role.

1. Navigate to IAM roles.
2. Search for the role created above.
3. Click your role in the list to navigate to the detail page.
4. Select Add permissions.
5. Select Attach policies.
6. Search for the policy created above.
7. Select your policy by clicking the checkbox next to the item in the list.
8. Click Add permissions.

S3 Setup Instructions

An AWS S3 bucket is required for the Redshift Integration. This provides a performant way to sync Fullstory's structured, behavioral event data directly to Redshift.

The S3 bucket must be created in the same AWS region as your Redshift cluster.

Step 1: Create an S3 bucket

You can create a new S3 bucket by navigating to the S3 dashboard:

1. Click Create bucket.
2. Give the bucket a name.
3. Select Block all public access.
4. Click Create bucket.

Step 2: Grant the IAM role access to the bucket

After creating the bucket, go to the new bucket's detail page and navigate to:

1. Permissions > Bucket policy.
2. Click Edit.
3. Paste the following policy replacing your role ARN and bucket name:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "<your role ARN>"
      },
     "Action": [
       "s3:PutObject",
       "s3:ListBucket",
       "s3:GetObjectVersion",
       "s3:GetObject",
       "s3:GetBucketLocation",
       "s3:DeleteObject"
     ],
     "Resource": [
       "arn:aws:s3:::<your bucket name>/*",
       "arn:aws:s3:::<your bucket name>"
     ]
    }
  ]
}

4. Click Save changes.

Redshift Database Setup

The Redshift Data Direct Integration requires a database account that has certain permissions with the database. Access the Query editor v2 in the Redshift console and apply the following commands.

Step 1: Create user and grant permissions

Run the following commands using SQL against your Redshift cluster:

CREATE USER IAMR:fullstory_redshift_role PASSWORD disable;
GRANT CREATE, TEMPORARY ON DATABASE <database> TO IAMR:fullstory_redshift_role;

This will create a user on the database that Fullstory can use to run queries and load your data. This user also needs to have the following permissions (which are typically available by default):

• On the created schema
  • CREATE, DROP tables, including TEMP tables
  • SELECT, INSERT, UPDATE and DELETE for the tables created by the user account
• On other schemas
  • SELECT on pg_namespace;
  • SELECT ON SYS monitoring views (SYS_*)
    • SYS_QUERY_DETAIL
    • SYS_QUERY_HISTORY
    • SYS_QUERY_TEXT
    • SYS_LOAD_DETAIL
    • SYS_LOAD_HISTORY
    • SYS_ERROR_DETAIL

Step 2: Allow Redshift to assume role to load files from S3

We need to allow Redshift to load files from S3. Navigate to Redshift then:

1. Properties > Cluster Permissions.
2. Click Manage IAM roles.
3. Select Associate IAM roles.
4. Search for the role you created earlier.
5. Select the role by clicking on the checkbox.
6. Click Associate IAM roles.

Enabling the Redshift Integration in Fullstory

Now that all of the AWS resources have been created, all that is left is to provide Fullstory with the correct identifiers to make a secure connection to load data.

1. In Fullstory, navigate to Settings > Integrations > Destinations.
2. Click Install next to the Redshift Option.
3. Select Serverless or Provisioned based on the type of Redshift cluster you are using.
4. Using the values recorded in the previous steps, fill in the form.
5. Click Save.

After saving, data will flow into your warehouse within an hour.

Some tips:

• The host should only include the Redshift host name. A proper host name should appear like mycluster-1.abc123.us-east-1.redshift.amazonaws.com without the port and database name.
• The IAM role ARN is only used for loading data into your warehouse. It is not capable of accessing or modifying users.
• The region should be a valid AWS region and should be the same region as your Redshift cluster and S3 bucket. Cross-region resources are not supported at this time.
• The role session name is optional; one will be assigned if not provided. Advice on when and how to use role session names can be found in AWS documentation.
• For Redshift Serverless, you will be required to enter the serverless workgroup name like my-warehouse-wg. Alternatively, for Redshift Provisioned clusters, you will need to enter your cluster ID like my-warehouse-cluster-id.

FAQ

Can you set up more than one data destination in your account?
Yes. Repeat setup steps for different destinations as needed.