Data Destinations

Available for the following Plan types:

FullStory Enterprise*

FullStory Business*

FullStory Free

*with the following add-on:

Data Destinations

Available to the following User roles:

Admin

Architect

Standard

 

Data Destinations is currently available as an early access release. If this is a feature you are interested in, please reach out to your account representative for more details.

When FullStory is deployed on your site or app, we ingest a massive amount of raw data that we then clean and structure into analytics ready event data. All of this data is used to power FullStory's search and digital experience analytics features when you want instant, out-of-the-box analysis. However, there are times when you need to dive deeper into the data to truly understand custom use cases. FullStory has already captured the most complete product experience dataset, and now we are making this data available to our customers via Data Destinations. 

What is Data Destinations?

Data Destinations enables FullStory customers to sync all of their digital experience data directly into their cloud data warehouse, such as Snowflake or Google BigQuery. FullStory will manage the structuring, delivery and deduplication of captured events without the need for customers to engage their engineering teams to build custom ETL pipelines.

With Data Destinations, you can merge FullStory’s digital experience with data from your other key business platforms to get a complete picture of your business's health. Now every analyst at your company can explore FullStory data directly through their preferred SQL interface or by using an existing Business Intelligence tool.

When should I use it?

For customers who want to integrate FullStory’s digital experience data into their own internal reporting pipelines, Data Destinations is the answer. With this technology, FullStory will sync all the events captured from your product into your data warehouse in an analytics-ready format, so that you can monitor the metrics that matter most to your business. Perform robust analyses of your product and model KPIs that are relevant to your business. 

How does it work?

We will manage a regular sync process that pushes all captured user events directly into your warehouse. We'll also handle all of the logic around retries and deduplication to ensure that you always have the most accurate data at your fingertips.

Integrations

There are 2 supported integrations that are powered by Data Destinations. Please review the additional documentation that supports installing these integrations:

Note: FullStory only supports one active destination per org

How to enable

To get started, navigate to Settings > Integrations > Manage > Data Exports and select either Snowflake or Big Query, depending on which warehouse you are using to store your data. Once you’ve saved your credentials, the integration will begin pushing data to your warehouse for as long as the integration remains active. Keep in mind, only one destination is supported at a time. If you’d like to change destinations, you must remove any existing Data Export integrations before doing so.

FullStory’s approach to data

Built to Support Planned Workflows

The Data Destinations feature is designed for customers who have dedicated analytics support, and are interested in automating their data ingestion and reporting process. This feature will only sync data captured during periods when the integration is active and will not sync data retroactively. This means, if you are looking to ingest all the data FullStory processes, then you should activate a destination when you first install the FullStory JS tag or SDK.

Future Proof Schemas

FullStory has designed our data model around an expectation for two main types of data: required fields and conditional fields. For fields that are required on every event we have extracted that field into a dedicated top level column. Our goal is to avoid sparse data in the raw table, so any field that is conditional to the event type is packed into the source_properties or event_properties columns as JSON objects. This data model allows FullStory to add new event_types and properties without having to run costly data migrations every time a new event is defined. FullStory will maintain a consistent and future-proof data model while enabling customers to handle denormalization and aggregation downstream.  

Sync Interval vs Update Window

Sync interval is a concept that tracks the cadence with which FullStory syncs new events to the warehouse. Captured events flow into FullStory constantly then get bundled up into a package each hour and pushed to the warehouse destination. All events that are ingested into FullStory’s servers within that hour are included in the package.

Due to the nature of web traffic, distributed systems and worldwide users, not all events will hit FullStory’s servers in chronological order. There are some scenarios where events come in out of order, like swan song events that appear hours after they occur. FullStory’s reporting pipeline treats pages and sessions as top level entities. All events that flow into the system will be associated with either a page, a session or both. This relationship is then used to derive metrics like “session length” and “active time on page”. When a late event is captured, FullStory will reprocess the page or session associated with the event to “put the events back in order”. When this reprocessing happens, FullStory will also push the updated events out to the warehouse, sometimes overwriting the data previously associated with that event. This is referred to as the Update Window, or the rolling lookback period where FullStory may update events in the warehouse. For a small set of edge cases, events may be reprocessed after the update window, causing small discrepancies between the FullStory UI and the data warehouse data. A very long update window reduces discrepancy but can make it harder to “close the books” from an accounting perspective.

Event Time vs Processed Time vs Updated time

Event time is an immutable field that records the timestamp of each event according to the user’s device.

Processed time is a timestamp field indicating the time that the event was captured and processed by FullStory’s servers. On average we see 95% of events captured and processed within 20 minutes of the original event time. A small number of events may reach FullStory’s servers much later than the original event time, including server side events. Depending on the contents of these late events, FullStory may need to reprocess them to report on the most accurate metrics, including session length, page active time, etc. In these scenarios, the Processed Time for all events in the session, or on the page, will be updated and the events will be re-synced to the warehouse.

Updated Time is the equivalent of running `current_timestamp` in your warehouse every time a record gets inserted or updated. This field tracks the last time a record in your warehouse was changed and can be used as a filter to determine which new records to pull into your query.

Moving Analysis to the View layer

Our approach to support a future-proof data model means that we are pushing all of the data FullStory captures into a pre-defined set of columns. This proves efficient from a data processing standpoint and sets clear expectations around the data contract. With that said, it has some implications when approaching queries. Specifically, FullStory’s underlying events table needs to be unpacked and denormalized into analytics-ready models in order to make querying easier, especially for BI tools. 

To help with this process, FullStory will deploy a set of standard views in your warehouse. These views will not solve for every use case possible, but they should cover the bulk of the metrics that are available through the FullStory UI and give analysts a starting point to work from. These views will adhere to all of the same data contracts as our underlying events table, meaning we will only make additive changes and will not deploy breaking changes (like removing a column) without providing advanced notice. We do recommend explicitly specifying columns in any production-grade SELECT statements in order to ensure you do not accidentally pull in new fields.

Source Info

Every event now contains contextual information about where that event came from and how the event was passed into the FullStory system. To keep your custom event payload from overloading with these extra parameters, we’ve implemented support for Source Properties. This will make it transparent where the event came from (web, app or server integration) and which FullStory API endpoints were invoked to pass the event. Now you can easily QA your instrumentation by performing data quality checks against your other systems.

Schemas and Event Definitions

Events Table Data model

Field name

Type

Description

event_id

STRING

unique key to reference each individual event

event_time

TIMESTAMP

timestamp associated with the event

processed_time

TIMESTAMP

timestamp the event was processed by FullStory's servers

updated_time

TIMESTAMP

timestamp the event was last inserted or updated in the  warehouse table

device_id

INTEGER

persistent storage identifier for the user's device

session_id

INTEGER

unique id associating multiple events to the same session

view_id

INTEGER

unique id associating multiple events to a single page, screen or tab

event_type

ENUM

classifier for the type of event

event_properties

JSON

json column containing all captured properties associated with each event

source_type

ENUM

classifier for the originating source of the event

source_properties

JSON

json column containing all the captured properties associated with the originating source


Event Types and their Properties

Event Type

Description

Event Properties

background

When a mobile app is running but it is not in focus on the screen.


Triggers when: Native Mobile lifecycle event when the app, or a activity within the app, moves from the foreground to the background

 

change

Finds activities where a user changed a specific element on your website, such as an input field.


Triggers when: Change to a text field

fs_suspicious_kind:string

target.masked:boolean

target.raw_selector:string

target.text:string

click

Finds activities where a user clicked on an element.


Triggers when: a pointing device button (such as a mouse's primary mouse button) is both pressed and released

fs_dead_count:number

fs_error_kind:string

fs_rage_count:number

is_click_long:boolean

is_click_unhandled:boolean

target.masked:boolean

target.raw_selector:string

target.text:string

consent

A call to FS.consent

consent_action_type:string

consent_scope:string

is_consent_given:boolean

console_message

Finds activities where a console error occurred. Surfaced by its console error message.


Triggers when: 

Web: https://developer.mozilla.org/en-US/docs/Web/API/console/error() and https://developer.mozilla.org/en-US/docs/Web/API/console/assert (false, ...)


Android: https://developer.fullstory.com/logging?lang=Java (’FS.LogLevel.ERROR’, ...)


iOS: https://developer.fullstory.com/logging?lang=Swift (’FSLOG_ERROR’, ...) or https://developer.fullstory.com/logging?lang=Swift (’FSLOG_ASSERT’, ...)

console_message_level:string

copy

Finds cases where a user highlights text on the screen and copies it.


Triggers when: User copies highlighted text via CTRL+C or right click

target.masked:boolean

target.raw_selector:string

target.text:string

crash

Finds activities where a crash occurred on a mobile application.


Triggers when: detect an Unhandled Exception in Java, which causes the app to force quit

 

custom

Finds activities where a FS.event API or Defined Event occurs.


Triggers when: A call to FS.event

event_name:string

event_properties:json

fs_api_errors.field:string

fs_api_errors.reason:string

fs_api_errors.value:string

element_seen

Finds activities where a user instrumented event is rendered on the page and if it was visible in the user’s viewport.


Triggers when: at least 25% of the watched element enters the user’s viewport.

element_render_duration_millis:number

element_start_time:number

element_start_type:string

element_type:string

element_visible_duration_millis:number

target.masked:boolean

target.raw_selector:string

target.text:string

exception

Finds activities where a code exception occurred. Surfaced by its exception message. This is only uncaught exceptions for now.


Triggers when: Javascript exception occurs in the browser. Code exceptions on Android / iOS.

exception_count:number

exception_source_file:string

is_exception_handled:boolean

form_abandon

Finds cases where a user started filling out a form but left before submitting their data.


Triggers when: a user navigates away from a page with a form filled out, but not submitted.

target.masked:boolean

target.raw_selector:string

target.text:string

highlight

Find activities where a user highlights content on a page.


Triggers when: content is highlighted on a page

target.masked:boolean

target.raw_selector:string

target.text:string

identify

A call to FS.identify or FS.setUserVars

fs_api_errors.field:string

fs_api_errors.reason:string

fs_api_errors.value:string

user_display_name:string

user_email:string

user_id:string

user_properties:json

keyboard_close

The user’s keyboard was closed


Triggers when: The keyboard is no longer visible after capturing keyboard visibility changed event (FSEventEventKeyboardVisibility)

 

keyboard_open

The user’s keyboard was opened.


Triggers when: The keyboard is visible after capturing keyboard visibility changed event (FSEventEventKeyboardVisibility)

 

load

Finds activities where a user refreshes the URL using their browser refresh button or keyboard shortcut.


Triggers when: A user initiated refresh of the current page

dom_content_time_millis:number

first_paint_time_millis:number

load_time_millis:number

low_memory

Finds activities where data capture stopped due to low device memory.


Triggers when: On Android, this event is triggered when there is 10% of RAM available. On iOS, we listen for this as the trigger for this event.

memory_available:number

memory_maximum:number

mouse_thrash

Find activities where a user rapidly moves their cursor back and forth in a short period of time.


Triggers when: Rapid mouse movement within a short timeframe

target.masked:boolean

target.raw_selector:string

target.text:string

navigate

Finds activities where a user navigates to a specific app screen.


Triggers when: User navigates to mobile app screen

fs_suspicious_kind:array

navigate_reason:string

page_properties

Define page names and additional properties related to the page (ie. Category, Cart Size).


Triggers when: A call to FS.setVars API

fs_api_errors.field:string fs_api_errors.reason:string fs_api_errors.value:string page_name:string page_properties:json

paste

Finds cases where users are pasting text into a field.


Triggers when: User pastes text into a field, either via CTRL+V or right click

target.masked:boolean

target.raw_selector:string

target.text:string

pinch_gesture

Finds activities where a user zoomed in or zoomed out using a touch device.


Triggers when: Multi-touch pinch event on a touch screen or multi-touch trackpad

pinch_direction:number

pinch_gesture_type:string

pinch_scale:number

pinch_visual_scale_change:number

target.masked:boolean

target.raw_selector:string

target.text:string

request

Surfaces network request errors when any captured network request has a status code ≥ 400.


Triggers when: A captured network request has a status code ≥ 400

request_duration_millis:number

request_method:string

request_status:number

request_url.full_url:string

request_url.hash_path:string

request_url.hash_query:string

request_url.host:string

request_url.path:string

request_url.query:string

 

These definitions are provided for informational purposes only.  Please remember that you can only use this information in accordance with FullStory's terms.

FAQ

  • How often will data be synced between FullStory and my Data Destination of choice?
    • Hourly.
  • Are there any plans to add other Data Destinations in the future?
    • Yes, we plan to add more destinations in the future based on customer demand. If there’s a destination you’d like added, make sure to let us know.
  • I see more event types in the event filter list in the UI than I see in the data. Is something missing from the data?
    • No, All captured data is included in the Data Destination Sync. Some of the “events” that are shown in the UI are derived from the same underlying event_type. For example, “Refreshed URL” in the UI is a navigate event_type with event_properties:navigate_reason = 'reload' in the warehouse.
  • I’ve created Events or Elements in the FullStory UI but I don’t see them in the warehouse.
    • There are a couple of searches that aren’t available yet which are all related to Pages, Defined events, and Named elements. Since the definitions for these higher order objects can be edited in the UI it poses some different challenges from a data perspective (they are not immutable events in the raw stream). Each of these concepts is a configuration that defines a reusable “search” in the UI and we render the results at query time so there is no underlying event to sync.
  • There are a number of events that have a NULL event_type. What are those records representing?
    • FullStory’s data capture pipeline is heavily distributed, so there is no guarantee that events arrive (through our ingestion pipeline) in chronological order. As our pipeline processes these events, it associates them with a view_id and orders them chronologically. If an event is synced to the warehouse and then FullStory receives a new event for the same view_id with an older timestamp, both events will be updated so they are in order. This will create two new events in the warehouse and null out the previously synced event.
  • Is it possible to configure event types that we do not want to sync? For example, if we decide that we do not want "change" events, is it possible to not have those sent?
    • Currently, FullStory only supports syncing all captured events to your warehouse. Events that you are not interested in can easily be filtered out in your SQL analysis. We are exploring the possibility of adding event filters prior to the sync. If this is something you would be interested in, please reach out for more information.
  • I see two tables, one called `events` and another called `raw_events` but raw events is empty. Is that expected?
    • Yes, the `raw_events` table is used in the Snowflake data load process as a temporary staging table. This table may include some data while the hourly load process is running but should be empty at all other times. Consumers of the data can ignore this table and should only use the `events` table for operational queries.
  • Does Data Destinations support a Bi-directional sync?
    • Not yet, but we would love to hear more about any use cases you’ve got in mind or any workflows that you’d like to enable.
  • I see a slight discrepancy between FullStory UI metrics and warehouse data.
    • All timestamps in the warehouse are in UTC whereas the FullStory UI localizes metrics based on your browser’s local timezone. To align these timezones, apply a timezone conversion from UTC to your local timezone in your SQL query.

 

Data Security Warning: If FullStory receives a data deletion request from a customer, and that customer's data has already been exported to their warehouse, FullStory will not be responsible for deleting the previously exported data, once it is out of FullStory’s possession. Once the data has been synced to a customer's system, it is the responsibility of the customer to ensure the data is secure and that they adhere to all privacy, security, and other applicable regulations.

Need to get in touch with us?

The FullStory Team awaits your every question.

Contact us