Skip to main content

Configuring telemetry destinations

Describes configuring telemetry destinations (AWS S3, Splunk HEC, HTTPS) for user events, conversation data, and audit events. Covers setup, authentication, and SIEM integration.

Updated over a week ago

SurePath AI can export User Activity and Audit Event logs to external destinations for integration with SIEM platforms, log management tools, and data warehouses. The platform supports multiple destination types including AWS S3 buckets, Splunk HTTP Event Collector (HEC) endpoints, and generic HTTPS endpoints. Each destination can be configured to receive specific telemetry streams including user events, conversation data, and audit events.

Organizations typically configure telemetry destinations during initial deployment to establish centralized monitoring and compliance reporting. The destinations operate independently, allowing admins to send different event types to different systems or configure multiple destinations for redundancy.

Understanding telemetry event types

SurePath AI generates three distinct types of telemetry data that can be sent to external destinations. Admins can enable or disable each event type independently for each destination, allowing precise control over what data flows to different systems.

User events

User events capture all user interactions with GenAI services that pass through SurePath AI. Each user event includes comprehensive details about the request, policy decisions, risk assessments, intent classifications, and message content. User events represent the most detailed telemetry stream and typically generate the highest volume of data.

User events include information such as the actor (user or application), destination service, HTTP request details, network information, policy violations, risk scores, detected sensitive data, intent classifications, and token usage metrics. This data enables security teams to monitor usage patterns, identify policy violations, track sensitive data exposure, and audit GenAI interactions across the organization.

Conversation data

Conversation data provides supplementary context about multi-turn conversations that occur between users and GenAI services. This telemetry stream captures conversation-level metadata and relationships between individual message exchanges. Conversation data helps organizations understand how users interact with GenAI services over extended sessions rather than viewing each message as an isolated event.

Audit events

Audit events record administrative actions and configuration changes within the SurePath AI platform. These events track activities such as policy modifications, user provisioning, connector configuration changes, and administrative access. Audit events are essential for compliance reporting and security investigations, as they provide a complete audit trail of who made what changes to the platform and when those changes occurred.

Organizations often send audit events to different destinations than user events because audit logs typically integrate with different downstream systems. For example, user events might flow to a security information and event management (SIEM) platform for threat detection, while audit events might go to a governance, risk, and compliance (GRC) system for compliance reporting.

Understanding the file structure and format

SurePath AI writes telemetry data in a structured format designed for efficient processing and long-term storage. Understanding the file structure, naming conventions, and content format helps organizations integrate with downstream analytics tools.

For comprehensive information about file structure, naming conventions, compression format, and the complete telemetry schema including all JSON fields, see Telemetry schema and format.

Accessing the telemetry configuration

Admins configure telemetry destinations through the SurePath AI admin interface. The telemetry configuration page displays all configured destinations and their current status, allowing admins to manage multiple destinations from a single location.

To access telemetry configuration:

  • Click Telemetry in the CONFIGURE section

The telemetry page displays a table with all configured destinations. Each destination entry shows the destination name, type (AWS S3 Bucket, Splunk HEC Index, or HTTPS Endpoint), which event types are enabled, activation status, and current status. Admins can add new destinations, edit existing configurations, or delete destinations from this page.

Configuring AWS S3 bucket destinations

AWS S3 bucket destinations use role assumption through an AWS connector to write telemetry files directly to an S3 bucket. This integration eliminates the need for long-term credentials and provides a secure, scalable method for storing telemetry data. Organizations typically use S3 destinations when they want to retain telemetry data for long-term analysis, compliance archiving, or integration with AWS-native analytics tools.

The S3 integration writes compressed NDJSON files to the bucket every 15 minutes. Files are organized in a hierarchical directory structure based on UTC timestamps, making it easy to locate and process events from specific time periods.

Prerequisites for AWS S3 destinations

Before configuring an S3 destination in SurePath AI, admins must complete the following prerequisites in AWS:

  • Create or identify an S3 bucket where telemetry files will be stored

  • Note the AWS region where the S3 bucket exists

  • Configure an AWS connector in SurePath AI using CloudFormation to establish role assumption permissions

The AWS connector uses a CloudFormation template that creates an IAM role with permissions to write to S3 buckets. SurePath AI assumes this role when writing telemetry files, which provides secure access without requiring long-term AWS credentials.

Setting up the AWS connector

Organizations need to configure an AWS connector before they can create S3 telemetry destinations. The connector establishes the trust relationship and permissions that allow SurePath AI to access AWS resources.

To set up an AWS connector:

  • Click Connectors in the CONFIGURE section

  • Click ADD CONNECTOR

  • Enter a descriptive name for the connector (e.g., "AWS Telemetry Connector")

  • Select AWS from the connector type dropdown

  • Click SAVE

  • Follow the CloudFormation deployment instructions that appear, which include a unique CloudFormation template URL for your organization

  • In the AWS console, create a new CloudFormation stack using the provided template URL

  • Complete the CloudFormation stack creation, noting the outputs including the IAM role ARN

  • Return to the SurePath AI connector configuration and enter the IAM role ARN

  • Click SAVE CHANGES

The CloudFormation template creates an IAM role with a trust policy that allows SurePath AI to assume the role. The role includes permissions to create and write to S3 buckets. Once the connector is configured and the role is verified, it can be used for multiple S3 telemetry destinations.

For detailed information about the CloudFormation template and role configuration, see the Updating Access Permissions for SurePath AI's CloudFormation Stack documentation.

Obtaining AWS S3 bucket information

Admins need to gather specific information from AWS before configuring the S3 destination in SurePath AI:

  • S3 Bucket Name: The exact name of the S3 bucket where telemetry files will be written. The bucket must already exist in AWS. SurePath AI does not create buckets automatically.

  • AWS Region: The AWS region code where the S3 bucket is located (e.g., us-east-1, us-west-2, eu-west-1). This must match the region where the bucket exists.

  • S3 Key Prefix (Optional): A directory path prefix that will be prepended to all telemetry file paths. For example, if the key prefix is set to surepath/production/, all files will be written under that path structure. This is useful for organizing telemetry files when multiple systems write to the same bucket or when implementing bucket lifecycle policies that apply to specific prefixes.

To find this information in the AWS console:

  • Open the AWS S3 console

  • Locate the destination bucket in the bucket list

  • Click on the bucket name to view its properties

  • Note the bucket name exactly as it appears

  • Note the AWS region shown in the bucket properties

Creating an S3 telemetry destination

Once the AWS connector is configured and the S3 bucket information is gathered, admins can create the telemetry destination in SurePath AI.

To create an S3 destination:

  • Click Telemetry in the CONFIGURE section

  • Click ADD DESTINATION

  • Enter a descriptive name for the destination in the Name field

  • Select AWS S3 Bucket from the Destination Type dropdown

  • Select the AWS connector from the Connector dropdown

  • Select the AWS region from the AWS Region dropdown

  • Enter the S3 bucket name in the AWS S3 Bucket Name field

  • (Optional) Enter a key prefix in the AWS S3 Key Prefix field if files should be written to a specific path within the bucket

  • Review the destination URLs that are displayed, which show the exact S3 paths where user events and audit events will be written

  • Enable the desired telemetry source activation options by toggling User Events, Conversation Data, and Audit Events as needed

  • Toggle Enabled to activate the destination

  • Click SAVE

After saving, SurePath AI validates the connector and bucket permissions. If the validation succeeds, the destination becomes active and begins writing telemetry files to the S3 bucket every 15 minutes. If validation fails, the destination status will show an error, and admins should verify that the connector is configured correctly and that the S3 bucket name and region are accurate.

The destination URLs displayed during configuration show the full S3 paths where files will be written. User events are written to a path that includes the version number and date hierarchy (e.g., s3://bucket-name/surepath-ai/user-events/v2/YYYY/MM/DD/HH/), while audit events use a similar structure under the audit-events path.

Configuring Splunk HEC Index destinations

Splunk HTTP Event Collector (HEC) destinations send telemetry events directly to Splunk indexes in real-time. This integration enables immediate visibility into GenAI usage and security events within Splunk dashboards, alerts, and analytics. Organizations use Splunk HEC destinations when they have existing Splunk deployments and want to correlate GenAI telemetry with other security and operational data.

The HEC integration sends events as soon as they are generated, providing near-real-time data delivery to Splunk. Events are formatted as JSON and include all relevant metadata, making them immediately searchable and compatible with Splunk's analytics capabilities.

Prerequisites for Splunk HEC destinations

Before configuring a Splunk HEC destination in SurePath AI, admins must complete the following prerequisites in Splunk:

  • Have admin access to Splunk Enterprise or Splunk Cloud Platform

  • Identify or create indexes where user events and audit events will be stored

  • Enable HTTP Event Collector on the Splunk deployment

  • Create an HEC token with permissions to write to the target indexes

  • Note the HEC URL endpoint for the Splunk deployment

Obtaining Splunk HEC configuration information

Admins need to gather specific information from Splunk before configuring the HEC destination in SurePath AI.

Enable HTTP Event Collector

HTTP Event Collector must be enabled on the Splunk deployment before tokens can receive events. On Splunk Cloud Platform, HEC is enabled by default. On Splunk Enterprise, admins must enable it manually.

To enable HEC on Splunk Enterprise:

  • Log in to Splunk Web with admin credentials

  • Click Settings > Data inputs

  • Click HTTP Event Collector

  • Click Global Settings

  • Click Enabled to enable HEC

  • (Optional) Configure the HTTP port if different from the default (8088)

  • (Optional) Enable SSL if the deployment should only accept HTTPS connections

  • Click Save

Create an HEC token in Splunk

HEC tokens provide authentication for external systems sending events to Splunk. Each token is associated with specific indexes and source type configurations.

To create an HEC token on Splunk Enterprise:

  • Click Settings > Add Data

  • Click Monitor

  • Click HTTP Event Collector

  • In the Name field, enter a descriptive name for the token (e.g., "SurePath AI Telemetry")

  • (Optional) In the Source name override field, enter a source name that will be assigned to all events (e.g., "surepath-ai")

  • (Optional) In the Description field, enter a description of the token's purpose

  • (Optional) Enable Indexer acknowledgment if required for guaranteed delivery

  • Click Next

  • In the Source type dropdown, select Automatic or choose a specific source type

  • In the Index section, select the indexes where events should be stored

  • Click Review

  • Review all settings and click Submit

  • Copy the token value that Splunk displays and save it securely for use in SurePath AI configuration

To create an HEC token on Splunk Cloud Platform:

  • Click Settings > Add Data

  • Click Monitor

  • Click HTTP Event Collector

  • In the Name field, enter a descriptive name for the token

  • (Optional) Enter a source name override and description

  • (Optional) Enable Indexer acknowledgment if supported and required

  • Click Next

  • Select or confirm the index where events should be stored (the index must already exist)

  • Click Review

  • Review all settings and click Submit

  • Copy the token value and save it securely

  • Click Track deployment progress to monitor token distribution across the Splunk Cloud deployment

  • Wait for the status to show "Done" before using the token

The token value is a 32-character globally unique identifier (GUID) that looks similar to B5A79AAD-D822-46CC-80D1-819F80D7BFB0. This value must be entered in the SurePath AI configuration exactly as displayed, though it is case-insensitive.

Determine the HEC URL

The HEC URL is the endpoint where SurePath AI will send events. The URL format depends on whether the Splunk deployment is on-premises (Splunk Enterprise) or cloud-based (Splunk Cloud Platform).

For Splunk Enterprise, the HEC URL typically follows this format:

https://<splunk-hostname>:8088/services/collector

Where <splunk-hostname> is the hostname or IP address of the Splunk server or load balancer, and 8088 is the default HEC port (which may differ if customized during HEC configuration).

For Splunk Cloud Platform, the HEC URL is provided by Splunk and typically follows this format:

https://http-inputs-<deployment-name>.splunkcloud.com:443/services/collector

Where <deployment-name> is the unique identifier for the Splunk Cloud deployment.

To find the exact HEC URL:

  • For Splunk Enterprise: Check the HEC global settings under Settings > Data inputs > HTTP Event Collector > Global Settings to confirm the port number, then construct the URL using the server hostname

  • For Splunk Cloud Platform: Consult Splunk Cloud documentation or support to obtain the correct HEC endpoint URL for the deployment

Identify index names

SurePath AI requires separate indexes for user events and audit events. Organizations can use existing indexes or create new ones specifically for SurePath AI telemetry.

To view existing indexes in Splunk:

  • Click Settings > Indexes

  • Review the list of available indexes

  • Note the exact names of the indexes where user events and audit events should be stored

To create a new index in Splunk Enterprise:

  • Click Settings > Indexes

  • Click New Index

  • Enter an Index Name (e.g., "surepath_user_events" or "surepath_audit_events")

  • Configure retention policies, size limits, and other settings as appropriate for organizational requirements

  • Click Save

For Splunk Cloud Platform, admins must submit a support request to create new indexes, as index creation is not available through the Splunk Web interface.

Organizations typically create separate indexes for user events and audit events to simplify data management, apply different retention policies, and control access permissions. User events typically generate much higher volumes than audit events, so separating them allows for independent capacity planning and retention management.

Create a source identifier

The source identifier is a text string that Splunk assigns to all events received through the HEC token. This identifier helps distinguish SurePath AI events from other data sources within Splunk indexes.

Organizations can choose any meaningful source identifier. Common patterns include:

  • surepath-ai

  • surepath:telemetry

  • genai:surepath

The source identifier is configured in the HEC token settings under the "Source name override" field when creating the token. If no source override is specified, Splunk uses a default source value based on the client connection.

Creating a Splunk HEC telemetry destination

Once all the required information is gathered from Splunk, admins can create the HEC telemetry destination in SurePath AI.

To create a Splunk HEC destination:

  • Click Telemetry in the CONFIGURE section

  • Click ADD DESTINATION

  • Enter a descriptive name for the destination in the Name field

  • Select Splunk HEC Index from the Destination Type dropdown

  • Enter the HEC URL in the HEC URL field

  • Enter the source identifier in the Source Identifier field

  • Enter the HEC token value in the HEC Token field

  • Enter the index name for user events in the User Event Index field

  • Enter the index name for audit events in the Audit Event Index field

  • (Optional) Toggle Allow self-signed certificates if the Splunk deployment uses self-signed SSL certificates

  • Enable the desired telemetry source activation options by toggling User Events, Conversation Data, and Audit Events as needed

  • Toggle Enabled to activate the destination

  • Click SAVE

After saving, SurePath AI validates the connection to the Splunk HEC endpoint by attempting to send a test event. If the validation succeeds, the destination becomes active and begins sending telemetry events to Splunk in real-time. If validation fails, the destination status will show an error, and admins should verify that the HEC URL, token, and index names are correct.

The "Allow self-signed certificates" option should only be enabled when connecting to Splunk deployments that use self-signed SSL certificates, such as development or test environments. Production Splunk deployments should use certificates signed by a trusted certificate authority, and this option should remain disabled.

Configuring HTTPS endpoint destinations

HTTPS endpoint destinations provide a generic integration method that sends telemetry events to any HTTPS endpoint. This destination type enables organizations to integrate SurePath AI telemetry with custom applications, third-party SIEM platforms, log aggregation services, or data warehouses that accept data over HTTP.

The HTTPS endpoint integration sends events in real-time as JSON payloads to the configured URL. Each event is sent as a separate HTTP POST request with authentication headers as specified in the destination configuration.

Understanding HTTPS endpoint requirements

The destination endpoint must meet the following requirements to receive SurePath AI telemetry:

  • Accept HTTP POST requests with JSON payloads

  • Support HTTPS (HTTP is not supported)

  • Provide a stable endpoint URL that does not change

  • Implement authentication using either Bearer tokens or custom header-based authentication

  • Return HTTP 200 status codes for successful event delivery

  • Handle individual events (SurePath AI sends one event per HTTP request)

Organizations implementing custom HTTPS endpoints should ensure that the endpoint can handle the expected volume of telemetry events. User events typically generate the highest volume, with event rates varying based on organizational GenAI usage patterns.

Obtaining endpoint configuration information

Before configuring an HTTPS endpoint destination, admins need to gather the following information from the destination system:

  • Endpoint URL: The complete HTTPS URL where telemetry events should be sent (e.g., https://telemetry.example.com/ingest/surepath)

  • Authentication Type: The authentication method required by the endpoint (Bearer token or custom header)

  • Authentication Header Name: For custom authentication, the exact name of the HTTP header that contains the authentication token (e.g., X-API-Key, Authorization, X-Auth-Token)

  • Token Value: The authentication token or API key required to authenticate requests to the endpoint

Creating an HTTPS endpoint telemetry destination

Once the endpoint information is gathered, admins can create the HTTPS endpoint destination in SurePath AI.

To create an HTTPS endpoint destination with Bearer authentication:

  • Click Telemetry in the CONFIGURE section

  • Click ADD DESTINATION

  • Enter a descriptive name for the destination in the Name field

  • Select HTTPS Endpoint from the Destination Type dropdown

  • Enter the complete endpoint URL in the Endpoint URL field

  • Select Bearer from the Authentication Type dropdown

  • Verify that the Authentication Header Name field shows "Authorization"

  • Enter the bearer token value in the Bearer Token field

  • (Optional) Toggle Allow self-signed certificates if the endpoint uses self-signed SSL certificates

  • Enable the desired telemetry source activation options by toggling User Events, Conversation Data, and Audit Events as needed

  • Toggle Enabled to activate the destination

  • Click SAVE

To create an HTTPS endpoint destination with custom authentication:

  • Click Telemetry in the CONFIGURE section

  • Click ADD DESTINATION

  • Enter a descriptive name for the destination in the Name field

  • Select HTTPS Endpoint from the Destination Type dropdown

  • Enter the complete endpoint URL in the Endpoint URL field

  • Select Custom from the Authentication Type dropdown

  • Enter the authentication header name in the Authentication Header Name field

  • Enter the token value in the Custom Token field

  • (Optional) Toggle Allow self-signed certificates if the endpoint uses self-signed SSL certificates

  • Enable the desired telemetry source activation options by toggling User Events, Conversation Data, and Audit Events as needed

  • Toggle Enabled to activate the destination

  • Click SAVE

After saving, SurePath AI validates the connection to the HTTPS endpoint by attempting to send a test event. If the validation succeeds, the destination becomes active and begins sending telemetry events in real-time. If validation fails, the destination status will show an error, and admins should verify that the endpoint URL, authentication type, and token are correct.

Bearer authentication is the most common authentication method for REST APIs and follows the standard HTTP Bearer authentication scheme. When Bearer authentication is selected, SurePath AI sends requests with an Authorization header containing the value Bearer <token>. Custom authentication allows organizations to specify any header name and token format required by their endpoint.

Managing telemetry destinations

After creating telemetry destinations, admins can modify configurations, enable or disable destinations, and monitor delivery status through the telemetry configuration interface.

Editing telemetry destinations

To modify an existing telemetry destination:

  • Click Telemetry in the CONFIGURE section

  • Locate the destination in the destination list

  • Click the destination name or click the menu icon (three dots) and select Edit

  • Modify the desired configuration settings

  • Click SAVE

Changes to telemetry destinations take effect immediately. If a destination is currently enabled and delivering events, modifying the configuration may cause a brief interruption in event delivery while the new settings are validated.

Enabling and disabling destinations

Destinations can be temporarily disabled without deleting the configuration. This is useful when troubleshooting destination issues, performing maintenance on the receiving system, or temporarily reducing telemetry volume.

To disable a destination:

  • Click Telemetry in the CONFIGURE section

  • Locate the destination in the destination list

  • Toggle the Enabled switch to off

To enable a destination:

  • Click Telemetry in the CONFIGURE section

  • Locate the destination in the destination list

  • Toggle the Enabled switch to on

When a destination is disabled, SurePath AI stops sending events to that destination immediately. Events generated while a destination is disabled are not queued or retried; they are simply not sent to the disabled destination. Other destinations continue to receive events normally.

Deleting telemetry destinations

Deleting a telemetry destination permanently removes the configuration and stops all event delivery to that destination. This action cannot be undone.

To delete a destination:

  • Click Telemetry in the CONFIGURE section

  • Locate the destination in the destination list

  • Click the menu icon (three dots) and select Delete

  • Confirm the deletion when prompted

Organizations should disable destinations first and verify that the destination is no longer needed before deleting it. Once deleted, the destination configuration must be recreated entirely if the destination needs to be restored.

Monitoring destination status

The telemetry destination list displays the current status of each configured destination. Status indicators help admins identify destinations that are operating normally versus those that have configuration issues or delivery failures.

Common status indicators include:

  • Active: The destination is enabled and successfully delivering events

  • Disabled: The destination is configured but currently disabled by an admin

  • Error: The destination encountered a configuration or delivery error

When a destination shows an error status, admins should click on the destination to view detailed error information. Common error conditions include invalid credentials, network connectivity issues, incorrect endpoint URLs, or insufficient permissions to write to the target system.

Telemetry data format and schema

All telemetry destinations receive events in a structured JSON format. The specific format and schema version depend on the destination type and when the destination was created.

SurePath AI uses schema version 2 for all new telemetry destinations. This version provides a comprehensive event structure with detailed policy decision information, risk assessments, intent classifications, and message content.

For detailed information about the telemetry schema, including all available JSON fields, data types, descriptions, and format examples, see Telemetry schema and format.

Best practices for telemetry configuration

Start with a single destination

Organizations implementing SurePath AI telemetry should start by configuring a single destination for user events. This allows admins to validate that events are flowing correctly, understand the data volume and format, and confirm that downstream systems can process the events before expanding to multiple destinations or adding audit events.

Use separate destinations for different event types

Consider configuring separate destinations for user events and audit events when the events need to flow to different systems or require different retention policies. For example, send user events to a SIEM platform for security monitoring while sending audit events to a compliance logging system.

Configure appropriate retention policies

For S3 destinations, implement S3 lifecycle policies to automatically archive or delete old telemetry files according to organizational data retention requirements. For Splunk destinations, configure index retention policies that align with compliance requirements and storage capacity.

Use descriptive naming

Name telemetry destinations descriptively to indicate the destination system and purpose (e.g., "Splunk Production SIEM", "AWS Compliance Archive", "QRadar Security Analytics"). This helps admins identify destinations quickly and understand their purpose when reviewing configurations.

Did this answer your question?