Connecting Splunk

Splunk Setup

Onboarding your Splunk observability system involves three stages: Connect, Enable, and Configure webhooks.

Prerequisite - Getting a Splunk API token

You will need an MCP URL and token to connect Splunk. Splunk Administrator needs to perform the following steps on their deployment

• Enable REST API access

• Enable token authentication on the deployment.

• Create a new role 'mcp_user', the new role does not need to have any capabilities.

• Assign the role 'mcp_user' to any users on the deployment who are authorized to use the MCP server.

• Create the token for the authorized users with audience as 'mcp' and set the appropriate expiration, if the user does not have the permission to create tokens themselves.

Splunk User needs to perform the following steps

• Get an appropriate token from the Splunk Administrator or create one themselves, if they have the permission. The audience for the token must be 'mcp'.

Step 1: Connect

Establish connection to your Splunk remote MCP endpoint with account access credentials

1. Open the hamburger menu and select Settings

2. Scroll to the Available - Telemetry section. Press Register next to Splunk

3. Enter your Splunk MCP server details:

   • Server Name - Unique identifier (e.g., my-splunk-server)

   • Endpoint URL - Your Splunk MCP server endpoint:

     https://YOUR_SPLUNK_DEPLOYMENT_NAME.api.scs.splunk.com/YOUR_SPLUNK_DEPLOYMENT_NAME/mcp/v1/

   • Description - Optional server description

   • Token Name - The name of the bearer token for authentication: my-splunk-token

   • Token Value - The bearer token value for authentication

4. Click Next

5. Review and Add

Step 2: Enable

Activate Splunk in a specific Agent space and configure appropriate scoping

1. From the agent spaces page, select an agent space and press view details (if you have not yet created an agent space see [REF])

2. Select the Capabilities tab

3. Scroll down to the Telemetry section

4. Press Add

5. Press radio button Available Sources

6. Select Radio button Splunk

7. Next

8. Review and press Save

9. Copy the Webhook URL and Webhook Secret

Step 3: Configure webhooks

Using the Webhook URL and Webhook Secret you can configure Splunk to send events to trigger an investigation, for example from an alarm. To ensure that events sent can be used by the DevOps Agent, make sure that the data transmitted to the web hook matches the data schema specified below. Events that do not match this schema may be ignored by DevOps Agent. Set the method and the headers

    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "Authorization": "Bearer <Token>",
    },
    

Send the body as a JSON string.

{
    eventType: 'incident';
    incidentId: string;
    action: 'created' | 'updated' | 'closed' | 'resolved';
    priority: "CRITICAL" | "HIGH" | "MEDIUM" | "LOW" | "MINIMAL";
    title: string;
    description?: string;
    timestamp?: string;
    service?: string;
    // The original event generated by service is attached here.
    data?: object;
}

Send webhooks with Splunk (note select no authorization and instead use the custom header option).

Learn more:

• Splunk's MCP Server Documentation

• Access requirements and limitations for the Splunk Cloud Platform REST API

• Manage authentication tokens in Splunk Cloud Platform

• Create and manage roles with Splunk Web