HTTP API - Using Segment Export

Fullstory’s Segment Export API provides an asynchronous workflow via a set of API endpoints that can be used together to download captured event data from Fullstory’s servers. Learn more about the Segment Export API in our API docs.

Fullstory offers an add-on called Data Direct to fully manage extracting and syncing event data captured by Fullstory into your BigQuery or Snowflake warehouses.

If you need to create your own Extract, Transform, & Load (ETL) pipeline for Segment Export data, you can follow the steps outlined below.

Workflow

The Segment Export API endpoints can be coordinated to perform the following steps:

1. Initiate an export job to aggregate data from Fullstory’s database.
2. Query for the status of this job (including % completion).
3. Once the job is done, get a URL to a data file in cloud storage.
4. Download the file.

Note: You will need a basic working knowledge of how to use your OS's shell + curl (or equivalent) to properly move through the below steps. Make sure your OS shell is up to date.

1. Initiate an export job

Using Create Segment Export, you can create a new segment export file that includes data from a designated segment. 

• The segmentId can be located in the URL of the Fullstory app when viewing that segment:
  
  
  • The segmentId in this example is: “htIXCSMv9Q9M" (You may also use List Segments to find the segmentId).
• Indicate if you want data about Individuals (TYPE_INDIVIDUAL) or Events (TYPE_EVENT). 
• You can also set the preferred format (FORMAT_CSV, FORMAT_JSON, or FORMAT_NDJSON) and time range of the data.
• Create an ‘Admin’ API key for Segment Export.

curl -X POST -H "Authorization: Basic <your FS API key>" "https://api.fullstory.com/segments/v1/exports" -d '{
  "segmentId": "everyone",
  "format": "FORMAT_CSV",
  "type": "TYPE_EVENT",
  "timeRange": {
    "start": "2020-10-13T00:00:00Z",
    "end": "2020-10-13T23:59:59Z"
  },
  "segmentTimeRange": {
    "start": "2020-10-13T00:00:00Z",
    "end": "2020-10-13T23:59:59Z"
  }
}'

Note: timeRange start and end values are inISO 8601 format. segmentTimeRange is required when overriding the default time period of a segment. The above example will export all individuals with activity on October 13, 2020. The start time is inclusive, and the end time is exclusive. So if an event occurs at the exact timestamp of your specified start time, it would be returned in your export data. But this would not be the case for events occurring at the exact timestamp of your end time - these would not be included in the export.

A successful response will appear as:

{"operationId": "abcdefg123"}

2. Query for the status of this job (including % completion)

Next, you will need to check the status of the operation created in the step above before continuing to Step 3.

Use the operationId from the prior response with the following command:

curl -H "Authorization: Basic <your FS API key>" 
"https://api.fullstory.com/operations/v1/{operationId}"

When an operation is still running, you will see a response such as this:

{
  "type": "SEARCH_EXPORT",
  "details": null,
  "results": null,
  "state": "PENDING",
  "errorDetails": "",
  "createdAt": "<timestamp>",
  "finishedAt": null,
  "estimatePctComplete": 10,
  "step": ""
}

Note: The raw responses will not contain new lines.

When the operation has successfully completed, you will see a response such as this:

{
  "type": "SEARCH_EXPORT",
  "details": null,
  "results": {
    "expires": "<timestamp>",
    "searchExportId": "efghijk456"
  },
  "state": "COMPLETED",
  "errorDetails": "",
  "createdAt": "<timestamp>",
  "finishedAt": "<timestamp>",
  "estimatePctComplete": 100,
  "step": ""
}

If you see a state of “ERROR”, check the errorDetails provided, resolve the conditions, and restart the export with Step 1 above.

3. Get a URL to a data file in cloud storage

Once the operation has successfully completed, you will then be able to retrieve a URL to the export file that is stored in cloud storage.

Use the searchExportId from the prior step in the following command:

curl -H "Authorization: Basic <your FS API key>" 
"https://api.fullstory.com/search/v1/exports/{searchExportId}/results"

A successful response will appear as follows:

{
"location": "<your location URL>",
"expires": "<timestamp>"
}

Note: The location URL will only be available until the time indicated in the expires value field.

The returned location URL will contain Unicode escape sequences. For example, “&” will be encoded as “\u0026”. Replace these escape sequences with their Unicode equivalents to create a well-formed URL. For example, the escape sequences in the following example URL

https://storage.googleapis.com/fullstoryapp-search-export/some\u0026long\u0026url

will need to be replaced so that the URL appears as

https://storage.googleapis.com/fullstoryapp-search-export/some&long&url

to create a well-formed URL.

Tip: In macOS terminal, you can use jq (a command-line JSON processor) to fix the location URL programmatically. For example, you can use:

(curl -H "Authorization: Basic <your FS API key>"
"https://api.fullstory.com/search/v1/exports/{searchExportId}/results" | jq

to get a well-formed URL programmatically.

Tip: In Windows PowerShell, you can use the ConvertFrom-Json cmdlet to:

(curl -H "Authorization: Basic <your FS API key>"
"https://api.fullstory.com/search/v1/exports/{searchExportId}/results" | ConvertFrom-Json).location

4. Download the file

You will then download the data export file from the URL provided in the previous step in a regular curl GET command (no auth header needed):

curl -JO <your location URL>

This command will save the export to the filename that's provided in the Content-Disposition header.

Again, the file must be downloaded before the location URL expires; otherwise, an expired token XML error will be returned.

  <Error>
  <Code>ExpiredToken</Code>
  <Message>The provided token has expired.</Message>
  <Details>Request signature expired at: 2022-10-26T15:07:49+00:00</Details>
  </Error>

If you receive an expired token XML error, restart from step 3 of this guide.

Note: The provided file will be gzipped and will need to be decompressed before further use.

Rate Limiting

Segment Export API is rate limited to 2 requests per minute, with a burst limit up to 50 for those running daily jobs with 30 minute bundles.

Additional Resources

A Segment Export CLI (which uses the API methods in this document) can be used to test out Segment Export capabilities: https://github.com/patrick-fs/segment-export-cli.