Monitor tag changes with serverless workflows and Amazon EventBridge - Tagging AWS Resources
Tag changes generate EventBridge eventsLambda and serverlessTutorial: Automatically stopping Amazon EC2 instances that are missing required tags

Monitor tag changes with serverless workflows and Amazon EventBridge

Amazon EventBridge supports tag changes on AWS resources. Using this EventBridge type, you can build EventBridge rules to match tag changes and route the events to one or more targets. For example, a target might be an AWS Lambda function to invoke automated workflows. This topic provides a tutorial for using Lambda to build a cost-effective serverless solution to securely process tag changes on your AWS resources.

Tag changes generate EventBridge events

EventBridge delivers a near real-time stream of system events that describe changes in AWS resources. Many AWS resources support tags, which are custom, user-defined attributes to easily organize and categorize AWS resources. Common use cases for tags are cost allocation categorization, access-control security, and automation.

With EventBridge, you can monitor for changes to tags and track the tag state on AWS resources. Previously, to achieve similar functionality, you might have continuously polled APIs and orchestrated multiple calls. Now, any change to a tag including individual service APIs, Tag Editor, and the Tagging API will initiate the tag change on resource event. The following example shows a typical EventBridge event prompted by a tag change. It shows the new, updated, or deleted tag keys, and their associated values.

{
  "version": "0",
  "id": "bddcf1d6-0251-35a1-aab0-adc1fb47c11c",
  "detail-type": "Tag Change on Resource",
  "source": "aws.tag",
  "account": "123456789012",
  "time": "2018-09-18T20:41:38Z",
  "region": "us-east-1",
  "resources": [
    "arn:aws:ec2:us-east-1:123456789012:instance/i-0000000aaaaaaaaaa"
  ],
  "detail": {
    "changed-tag-keys": [
      "a-new-key",
      "an-updated-key",
      "a-deleted-key"
    ],
    "tags": {
      "a-new-key": "tag-value-on-new-key-just-added",
      "an-updated-key": "tag-value-was-just-changed",
      "an-unchanged-key": "tag-value-still-the-same"
    },
    "service": "ec2",
    "resource-type": "instance",
    "version": 3,
  }
}

All EventBridge events have the same top-level fields:

  • version – By default, this value is set to 0 (zero) in all events.

  • id – A unique value is generated for every event. This can be helpful in tracing events as they move through rules to targets and are processed.

  • detail-type – Identifies, in combination with the source field, the fields and values that appear in the detail field.

  • source – Identifies the service that was the source of the event. The source for tag changes is aws.tag.

  • time – The timestamp of the event.

  • region – Identifies the AWS Region where the event originated.

  • resources – This JSON array contains Amazon Resource Names (ARNs) that identify resources that are involved in the event. This is the resource where tags have changed.

  • detail – A JSON object, whose content is different depending on event type. For tag change on resource, the following detailed fields are included:

    • changed-tag-keys – The tag keys that changed by this event.

    • service – The service that the resource belongs to. In this example, the service is ec2, which is Amazon EC2.

    • resource-type – The type of resource of the service. In this example, it is an Amazon EC2 instance.

    • version – The version of the tag set. The version starts at 1 and increments when tags are changed. You can use the version to verify the order of tag change events.

    • tags – The tags attached to the resource after the change.

For more information, see Amazon EventBridge event patterns in the Amazon EventBridge User Guide.

By using EventBridge, you can create rules that match specific event patterns based on the different fields. We demonstrate how to do this in the tutorial. Also, we show how an Amazon EC2 instance can be stopped automatically if a specified tag isn’t attached to the instance. We use the EventBridge fields to create a pattern to match the tag events for the instance that launches a Lambda function.

Lambda and serverless

AWS Lambda follows the serverless paradigm to run code in the cloud. You run code only when it’s needed, without thinking about servers. You pay only for the exact compute time you use. Even though it’s called serverless, it doesn’t mean that there are no servers. Serverless in this context means that you don’t have to provision, configure, or manage the servers that are used to run your code. AWS does all of that for you, so you can focus on your code. For more information about Lambda, see the AWS Lambda Product Overview.

Tutorial: Automatically stopping Amazon EC2 instances that are missing required tags

As your pool of AWS resources and AWS accounts that you manage grows, you can use tags to make it easier to categorize your resources. Tags are commonly used for critical use cases such as cost allocation and security. To effectively manage AWS resources, your resources need to be consistently tagged. Often, when a resource is provisioned, it gets all the appropriate tags. However, a later process can result in a tag change that results in drift from the corporate tag policy. By monitoring changes to your tags, you can spot tag drift and immediately respond. This gives you more confidence that the processes that depend on your resources being properly categorized will produce the desired results.

The following example demonstrates how to monitor for tag changes on Amazon EC2 instances to verify that a specified instance continues to have the required tags. If the instance's tags change and the instance no longer has the required tags, a Lambda function is invoked to shut down the instance automatically. Why would you want to do this? It ensures that all resources are tagged according to your corporate tag policy, for effective cost allocation, or to be able to trust security based on attribute-based access control (ABAC).

Important

We strongly recommend that you perform this tutorial in a non-production account where you can't inadvertently shut down important instances.

The example code in this tutorial intentionally limits the impact of this scenario to only the instances on a list of instance IDs. You must update the list with instance IDs that you are willing to shut down for the test. This helps ensure that you can't accidentally shut down every instance in a Region in your AWS account.

After testing, make sure that all of your instances are tagged according to your company's tagging strategy. Then, you can remove the code that limits the function to only the instance IDs on the list.

This example uses JavaScript and the 16.x version of Node.js. The example uses example AWS account ID 123456789012 and the AWS Region US East (N. Virginia) (us-east-1). Replace these with your own test account ID and Region.

Note

If your console uses a different Region for its default, ensure that you switch the Region you're using in this tutorial whenever you change consoles. A common cause of this tutorial failing is having the instance and function in two different Regions.

If you use a different Region than us-east-1, ensure that you change all references in the following code examples to your chosen Region.

Step 1. Create the Lambda function

To create the Lambda function

  1. Open the AWS Lambda management console.

  2. Choose Create function and then choose Author from scratch.

  3. For Function name, type AutoEC2Termination.

  4. For Runtime, choose Node.js 16.x.

  5. Keep all other fields at their default values, and choose Create function.

  6. On the Code tab of the AutoEC2Termination detail page, open the index.js file to view its code.

    • If a tab with index.js is open, you can choose the edit box in that tab to edit its code.

    • If a tab with index.js isn't open, right-click the index.js file under the AutoEC2Terminator folder in the navigation pane. Then choose Open.

  7. In the index.js tab, paste the following code in the editor box, replacing anything that is already present.

    Replace the value RegionToMonitor with the Region that you want to run this function in.

    // Set the following line to specify which Region's instances you want to monitor
// Only instances in this Region are succesfully stopped on a match

const RegionToMonitor = "us-east-1"

// Specify the instance ARNs to check.
// This limits the function for safety to avoid the tutorial shutting down all instances in account
// The first ARN is a "dummy" that matches the test event you create in Step 3.
// Replace the second ARN with one that matches a real instance that you want to monitor and that you can 
// safely stop

const InstanceList = [
    "i-0000000aaaaaaaaaa",
    "i-05db4466d02744f07"
];

// The tag key name and value that marks a "valid" instance. Instances in the previous list that
// do NOT have the following tag key and value are stopped by this function

const ValidKeyName = "valid-key";
const ValidKeyValue = "valid-value";

// Load and configure the AWS SDK
const AWS = require('aws-sdk');
// Set the AWS Region
AWS.config.update({region: RegionToMonitor});
// Create EC2 service object.
const ec2 = new AWS.EC2({apiVersion: '2016-11-15'});

exports.handler = (event, context, callback) => {

  // Retrieve the details of the reported event.
  var detail = event.detail;
  var tags = detail["tags"];
  var service = detail["service"];
  var resourceType = detail["resource-type"];
  var resource = event.resources[0];
  var resourceSplit = resource.split("/");
  var instanceId = resourceSplit[resourceSplit.length - 1];

  // If this event is not for an EC2 resource, then do nothing.
  if (!(service === "ec2")) {
    console.log("Event not for correct service -- no action (", service, ")" );
    return;
  }

  // If this event is not about an instance, then do nothing.
  if (!(resourceType === "instance")) {
    console.log("Event not for correct resource type -- no action (", resourceType, ")" );
    return;
  }

  // CAUTION - Removing the following 'if' statement causes the function to run against 
  //           every EC2 instance in the specified Region in the calling AWS account. 
  //           If you do this and an instance is not tagged with the approved tag key 
  //           and value, this function stops that instance.

  // If this event is not for the ARN of an instance in our include list, then do nothing.
  if (InstanceList.indexOf(instanceId)<0) {
    console.log("Event not for one of the monitored instances -- no action (", resource, ")");
    return;
  }

  console.log("Tags changed on monitored EC2 instance (",instanceId,")");

  // Check attached tags for expected tag key and value pair
  if ( tags.hasOwnProperty(ValidKeyName) && tags[ValidKeyName] == "valid-value"){
    // Required tags ARE present
    console.log("The instance has the required tag key and value -- no action");
    callback(null, "no action");
    return;
  }
  
  // Required tags NOT present
  console.log("This instance is missing the required tag key or value -- attempting to stop the instance");

  var params = {
    InstanceIds: [instanceId], 
    DryRun: true
  };

  // call EC2 to stop the selected instances
  ec2.stopInstances(params, function(err, data) {
    if (err && err.code === 'DryRunOperation') {
      // dryrun succeeded, so proceed with "real" stop operation
      params.DryRun = false;
      ec2.stopInstances(params, function(err, data) {
        if (err) {
          console.log("Failed to stop instance");
          callback(err, "fail");
        } else if (data) {
          console.log("Successfully stopped instance", data.StoppingInstances);
          callback(null, "Success");
        }
      });
    } else {
      console.log("Dryrun attempt failed");
      callback(err);
    }
  });
};

  8. Choose Deploy to save your changes and make the new version of the function active.

This Lambda function checks the tags of an Amazon EC2 instance, as reported by the tag change event in EventBridge. In this example, if the instance in the event is missing the required tag key valid-key or if that tag doesn't have the value valid-value, then the function tries to stop the instance. You can change this logical check or the tag requirements for your own specific use cases.

Keep the Lambda console window open in your browser.

Step 2. Set up the required IAM permissions

Before the function can successfully run, you must grant the function the permission to stop an EC2 instance. The AWS provided role lambda_basic_execution doesn't have that permission. In this tutorial, you modify the default IAM permission policy that is attached to the function's execution role named AutoEC2Termination-role-uniqueid. The minimum additional permission required for this tutorial is ec2:StopInstances.

For more information about creating Amazon EC2 specific IAM policies, see Amazon EC2: Allows starting or stopping an EC2 Instance and modifying a security group, programmatically and in the console in the IAM User Guide.

To create an IAM permission policy and attach it to the Lambda function's execution role

  1. In a different browser tab or window, open the Roles page of the IAM console.

  2. Start typing the role name AutoEC2Termination, and when it appears in the list, choose the role name.

  3. On the role's Summary page, choose the Permissions tab and choose the name of the one policy that is already attached.

  4. On the policy's Summary page, choose Edit policy.

  5. On the Visual Editor tab, choose Add additional permissions.

  6. For Service, choose EC2.

  7. For Actions, choose StopInstances. You can type Stop in the search bar, and then choose StopInstances when it appears.

  8. For Resources, choose All resources, choose Review policy, and then choose Save changes.

    This automatically creates a new version of the policy and sets that version as the default.

    Your final policy should look similar to the following example.

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "ec2:StopInstances",
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "logs:CreateLogGroup",
            "Resource": "arn:aws:logs:us-east-1:123456789012:*"
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:us-east-1:123456789012:log-group:/aws/lambda/AutoEC2Termination:*"
        }
    ]
}

Step 3. Do a preliminary test of your Lambda function

In this step, you submit a test event to your function. The Lambda test functionality works by submitting a manually provided test event. The function processes the test event just as if the event had come from EventBridge. You can define multiple test events with different values to exercise all of the different parts of your code. In this step, you submit a test event that indicates that an Amazon EC2 instance's tags changed, and the new tags don't include the required tag key and value.

To test your Lambda function

  1. Return to the window or tab with the Lambda console and open the Test tab for your AutoEC2Termination function.

  2. Choose Create new event.

  3. For Event name, enter SampleBadTagChangeEvent.

  4. In the Event JSON, replace the text with the sample event shown in the following example text. You don't need to modify the accounts, Region, or instance ID for this test event to work correctly.

    {
  "version": "0",
  "id": "bddcf1d6-0251-35a1-aab0-adc1fb47c11c",
  "detail-type": "Tag Change on Resource",
  "source": "aws.tag",
  "account": "123456789012",
  "time": "2018-09-18T20:41:38Z",
  "region": "us-east-1",
  "resources": [
    "arn:aws:ec2:us-east-1:123456789012:instance/i-0000000aaaaaaaaaa"
  ],
  "detail": {
    "changed-tag-keys": [
      "valid-key"
    ],
    "tags": {
      "valid-key": "NOT-valid-value"
    },
    "service": "ec2",
    "resource-type": "instance",
    "version": 3
  }
}

  5. Choose Save, and then choose Test.

    The test appears to fail, but that's OK.

    You should see the following error in the Execution results tab under Response.

    {
  "errorType": "InvalidInstanceID.NotFound",
  "errorMessage": "The instance ID 'i-0000000aaaaaaaaaa' does not exist",
  ...
}

    The error occurs because the instance specified in the test event doesn't exist.

    The information on the Execution results tab, in the Function Logs section , demonstrates that your Lambda function successfully attempted to stop an EC2 instance. However, it failed because the code initially attempts a DryRun operation to stop the instance, which indicated that the instance ID was not valid. 

    START RequestId: 390c1f8d-0d9b-4b44-b087-8de64479ab44 Version: $LATEST
2022-11-30T20:17:30.427Z    390c1f8d-0d9b-4b44-b087-8de64479ab44    INFO    Tags changed on monitored EC2 instance ( i-0000000aaaaaaaaaa )
2022-11-30T20:17:30.427Z    390c1f8d-0d9b-4b44-b087-8de64479ab44    INFO    This instance is missing the required tag key or value -- attempting to stop the instance
2022-11-30T20:17:31.206Z    390c1f8d-0d9b-4b44-b087-8de64479ab44    INFO    Dryrun attempt failed
2022-11-30T20:17:31.207Z    390c1f8d-0d9b-4b44-b087-8de64479ab44    ERROR   Invoke Error     {"errorType":"InvalidInstanceID.NotFound","errorMessage":"The instance ID 'i-0000000aaaaaaaaaa' does not exist","code":"InvalidInstanceID.NotFound","message":"The instance ID 'i-0000000aaaaaaaaaa' does not exist","time":"2022-11-30T20:17:31.205Z","requestId":"a5192c3b-142d-4cec-bdbc-685a9b7c7abf","statusCode":400,"retryable":false,"retryDelay":36.87870631147607,"stack":["InvalidInstanceID.NotFound: The instance ID 'i-0000000aaaaaaaaaa' does not exist","    at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/services/ec2.js:50:35)","    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)","    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)","    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:686:14)","    at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)","    at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)","    at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10","    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)","    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:688:12)","    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:116:18)"]}
END RequestId: 390c1f8d-0d9b-4b44-b087-8de64479ab44

  6. To prove that the code doesn't try to stop the instance when the correct tag is used, you can create and submit another test event.

    Choose the Test tab above Code source. The console displays your existing SampleBadTagChangeEvent test event.

  7. Choose Create new event.

  8. For Event name, type SampleGoodTagChangeEvent.

  9. In line 17, delete NOT- to change the value to valid-value.

  10. At the top of the Test event window, choose Save, and then choose Test.

    The output displays the following, which demonstrates that the function recognizes the valid tag and doesn't attempt to shut down the instance.

    START RequestId: 53631a49-2b54-42fe-bf61-85b9e91e86c4 Version: $LATEST
2022-12-01T23:24:12.244Z    53631a49-2b54-42fe-bf61-85b9e91e86c4    INFO    Tags changed on monitored EC2 instance ( i-0000000aaaaaaaaaa )
2022-12-01T23:24:12.244Z    53631a49-2b54-42fe-bf61-85b9e91e86c4    INFO    The instance has the required tag key and value -- no action
END RequestId: 53631a49-2b54-42fe-bf61-85b9e91e86c4

    Keep the Lambda console open in your browser.

Step 4. Create the EventBridge rule that launches the function

Now you can create an EventBridge rule that matches the event and points to your Lambda function.

To create the EventBridge rule

  1. In a different browser tab or window, open the EventBridge console to the Create Rule page.

  2. For Name, enter ec2-instance-rule, and then choose Next.

  3. Scroll down to Creation method and choose Custom pattern (JSON editor).

  4. In the editing box, paste the following pattern text, and then choose Next.

    {
  "source": [
    "aws.tag"
  ],
  "detail-type": [
    "Tag Change on Resource"
  ],
  "detail": {
    "service": [
      "ec2"
    ],
    "resource-type": [
      "instance"
    ]
  }
}

    This rule matches Tag Change on Resource events for Amazon EC2 instances and invokes whatever you specify as the Target in the next step.

  5. Next, add your Lambda function as the target. In the Target 1 box, under Select a target, choose Lambda function.

  6. Under Function, choose the AutoEC2Termination function that you created previously, and then choose Next.

  7. On the Configure tags page, choose Next. Then on the Review and create page, choose Create rule. This also automatically grants permission for EventBridge to invoke the specified Lambda function.

Step 5. Test the complete solution

You can test your final result by creating an EC2 instance and watching what happens when you change its tags.

To test the monitoring solution with a real instance

  1. Open the Amazon EC2 console to the Instances page.

  2. Create an Amazon EC2 instance. Before you launch it, attach a tag with the key valid-key and the value valid-value. For information about how to create and launch an instance,see Step 1: Launch an instance in the Amazon EC2 User Guide for Linux Instances. In the procedure To launch an instance, in step 3, where you enter the Name tag, also choose Add additional tags, choose Add tag, and then enter the Key of valid-key and Value of valid-value. You can Proceed without a key pair if this instance is solely for the purposes of this tutorial and you plan on deleting this instance after you complete it. Return to this tutorial when you reach the end of Step 1; you don't need to do Step 2: Connect to your instance.

  3. Copy the InstanceId from the console.

  4. Switch from the Amazon EC2 console to the Lambda console. Choose your AutoEC2Termination function, choose the Code tab, and then choose the index.js tab to edit your code.

  5. Change the second entry in the InstanceList by pasting the value you copied from the Amazon EC2 console. Ensure that the RegionToMonitor value matches the Region that contains the instance you pasted.

  6. Choose Deploy to make your changes active. The function is now ready to be activated by tag changes to that instance in the specified Region.

  7. Switch from the Lambda console to the Amazon EC2 console.

  8. Change the Tags attached to the instance by either deleting the valid-key tag or by changing that key's value.

    Note

    For information about how to change the tags on a running Amazon EC2 instance, see Add and delete tags on an individual resource in the Amazon EC2 User Guide for Linux Instances.

  9. Wait a few seconds, and then refresh the console. The instance should change its Instance state to Stopping and then to Stopped.

  10. Switch from the Amazon EC2 console to the Lambda console with your function, and choose the Monitor tab.

  11. Choose the Logs tab, and in the Recent invocations table, choose the most recent entry in the LogStream column.

    The Amazon CloudWatch console opens to the Log events page for the last invocation of your Lambda function. The last entry should look similar to the following example.

    2022-11-30T12:03:57.544-08:00    START RequestId: b5befd18-2c41-43c8-a320-3a4b2317cdac Version: $LATEST
2022-11-30T12:03:57.548-08:00    2022-11-30T20:03:57.548Z b5befd18-2c41-43c8-a320-3a4b2317cdac INFO Tags changed on monitored EC2 instance ( arn:aws:ec2:us-west-2:123456789012:instance/i-1234567890abcdef0 )
2022-11-30T12:03:57.548-08:00    2022-11-30T20:03:57.548Z b5befd18-2c41-43c8-a320-3a4b2317cdac INFO This instance is missing the required tag key or value -- attempting to stop the instance
2022-11-30T12:03:58.488-08:00    2022-11-30T20:03:58.488Z b5befd18-2c41-43c8-a320-3a4b2317cdac INFO Successfully stopped instance [ { CurrentState: { Code: 64, Name: 'stopping' }, InstanceId: 'i-1234567890abcdef0', PreviousState: { Code: 16, Name: 'running' } } ]
2022-11-30T12:03:58.546-08:00    END RequestId: b5befd18-2c41-43c8-a320-3a4b2317cdac

Summary

This tutorial demonstrated how to create an EventBridge rule to match against a tag change on a resource event for Amazon EC2 instances. The rule pointed to a Lambda function that automatically shuts down the instance if it doesn't have the required tag.

The Amazon EventBridge support for tag changes on AWS resources opens possibilities to build event-driven automation across many AWS services. Combining this capability with AWS Lambda provides you with tools to build serverless solutions that access AWS resources securely, scale on demand, and are cost effective.

Other possible use cases for the tag-change-on-resource EventBridge event include:

  • Launch a warning if someone accesses your resource from an unusual IP address – Use a tag to store the source IP address of each visitor that accesses your resource. Changes to the tag generates a CloudWatch event. You can use that event to compare the source IP address to a list of valid IP addresses and activate a warning email if the source IP address isn't valid.

  • Monitor if there are changes to your tag-based access control for a resource – If you have set up access to a resource using attribute (tag) based access control (ABAC), you can use EventBridge events generated by any changes to the tag to prompt an audit by your security team.