Monitor Oracle GoldenGate logs by using Amazon CloudWatch - AWS Prescriptive Guidance
SummaryPrerequisites and limitationsArchitectureToolsEpicsTroubleshootingRelated resources

Monitor Oracle GoldenGate logs by using Amazon CloudWatch

Created by Chithra Krishnamurthy (AWS)

Environment: Production

Technologies: Databases

Workload: Oracle

AWS services: Amazon CloudWatch; Amazon SNS

Summary

Oracle GoldenGate provides real-time replication between Amazon Relational Database Service (Amazon RDS) for Oracle databases, or between Oracle databases hosted on Amazon Elastic Compute Cloud (Amazon EC2). It supports both unidirectional and bidirectional replication.

When you use GoldenGate for replication, monitoring is critical to verify that the GoldenGate process is up and running, to make sure that the source and target databases are in sync.

This pattern explains the steps to implement Amazon CloudWatch monitoring for a GoldenGate error log, and how to set alarms to send notifications for specific events such as STOP or ABEND so you can take appropriate actions to resume replication quickly.

Prerequisites and limitations

Prerequisites

  • GoldenGate installed and configured on an EC2 instance, so you can set up CloudWatch monitoring on those EC2 instances. If you want to monitor GoldenGate across AWS Regions for bidirectional replication, you must install the CloudWatch agent in each EC2 instance where the GoldenGate process is running.

Limitations

  • This pattern explains how to monitor the GoldenGate process by using CloudWatch. CloudWatch doesn’t monitor replication lag or data synchronization issues during replication. You must run separate SQL queries to monitor replication lag or data-related errors, as explained in the GoldenGate documentation.

Product versions

  • This document is based on the implementation of Oracle GoldenGate 19.1.0.0.4 for Oracle on Linux x86-64. However, this solution is applicable to all major versions of GoldenGate.

Architecture

Target technology stack

  • GoldenGate binaries for Oracle installed on an EC2 instance

  • Amazon CloudWatch

  • Amazon Simple Notification Service (Amazon SNS)

Target architecture

Target architecture for monitoring GoldenGate logs on AWS

Tools

AWS services

  • Amazon CloudWatch is a monitoring service that is used in this pattern to monitor GoldenGate error logs.

  • Amazon SNS is a message notification service that is used in this pattern to send email notifications.

Other tools

  • Oracle GoldenGate  is a data replication tool that you can use for Amazon RDS for Oracle databases or Oracle databases that are hosted on Amazon EC2.

High-level implementation steps

  1. Create an AWS Identity and Access Management (IAM) role for the CloudWatch agent.

  2. Attach the IAM role to the EC2 instance where GoldenGate error logs are generated.

  3. Install the CloudWatch agent on the EC2 instance.

  4. Configure the CloudWatch agent configuration files: awscli.conf and awslogs.conf.

  5. Start the CloudWatch agent.

  6. Create metric filters in the log group.

  7. Set up Amazon SNS.

  8. Create an alarm for the metric filters. Amazon SNS sends email alerts when those filters capture events.

For detailed instructions, see the next section.

Epics

TaskDescriptionSkills required

Create the IAM role.

Access to AWS resources requires permissions, so you create IAM roles to include the permissions necessary for each server to run the CloudWatch agent.

To create the IAM role:

  1. Sign in to the AWS Management Console and open the IAM console at https://console.aws.amazon.com/iam/.

  2. In the navigation pane, choose Roles, and then choose Create role.

  3. For Trusted entity type, choose AWS service.

  4. For Common use cases, choose EC2, and then choose Next.

  5. In the list of policies, select the check box next to CloudWatchAgentServerPolicy. If necessary, use the search box to find the policy.

  6. Choose Next.

  7. For Role name, enter a name for your new role, such as goldengate-cw-monitoring-role or another name that you prefer.

  8. (Optional) For Role description, enter a description.

  9. Confirm that CloudWatchAgentServerPolicy appears under Policy name.

  10. (Optional) Add one or more tags (key-value pairs) to organize, track, or control access for this role, and then choose Create role.

AWS general
TaskDescriptionSkills required

Attach the IAM role to the EC2 instance where GoldenGate error logs are generated.

The error logs generated by GoldenGate have to be populated to CloudWatch and monitored, so you need to attach the IAM role you created in step 1 to the EC2 instance where GoldenGate is running.

To attach an IAM role to an instance:

  1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.

  2. In the navigation pane, choose Instances, and then find the instance where GoldenGate is running.

  3. Select the instance, and then choose Actions, Security, Modify IAM role.

  4. Select the IAM role created in the first step to attach to your instance, and then choose Save.

AWS general
TaskDescriptionSkills required

Install the CloudWatch agent on the GoldenGate EC2 instance.

To install the agent, run the command:

sudo yum install -y awslogs
AWS general

Edit the agent configuration files.

  1. Run the following command.

    sudo su -

  2. Edit this file to update the AWS Region as necessary.

    cat /etc/awslogs/conf
[plugins]
cwlogs = cwlogs
[default]
region = us-east-1

  3. Edit the /etc/awslogs/awslogs.conf file to update the file name, log group name, and the date/time format. You must specify the date/time to match the date format in ggserror.log; otherwise, the log stream won’t flow into CloudWatch. For example:

    datetime_format = %Y-%m-%dT%H:%M:%S%z
file = /u03/oracle/oragg/ggserr.log
log_group_name = goldengate_monitor
AWS general

Start the CloudWatch agent.

To start the agent, use the following command.

$ sudo service awslogsd start

After you start the agent, you can view the log group in the CloudWatch console. The log stream will have the contents of the file.

AWS general
TaskDescriptionSkills required

Create metric filters for the keywords ABEND and STOPPED.

When you create metric filters for the log group, whenever the filters are identified in the error log, it starts an alarm and sends an email notification based on the Amazon SNS configuration.

To create metric filters:

  1. Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.

  2. Choose the name of the log group.

  3. Choose Actions, and then choose Create metric filter.

  4. For the Filter pattern, specify a pattern such as ABEND.

  5. Choose Next, and then enter a name for your metric filter.

  6. Under Metric details, for Metric namespace, enter a name for the CloudWatch namespace where the metric will be published. If the namespace doesn't already exist, make sure that Create new is selected.

  7. For Metric value, enter 1, because your metric filter is counting occurrences of the keywords in the filter.

  8. Set Unit to None.

  9. Choose Create metric filter. You can find the metric filter that you created from the navigation pane.

  10. Create another metric filter for the STOPPED pattern. Within one log group, you can create multiple metric filters and set alarms individually.

CloudWatch
TaskDescriptionSkills required

Create an SNS topic.

In this step, you configure Amazon SNS to create alarms for the metric filters.

To create an SNS topic:

  1. Sign in to the Amazon SNS console at https://console.aws.amazon.com/sns/home.

  2. In the Create topic box, enter a topic name such as goldengate-alert, and then choose Next step.

  3. For Type, choose Standard.

  4. Scroll to the end of the form and choose Create topic. The console opens the new topic's Details page.

Amazon SNS

Create a subscription.

To create a subscription to the topic:

  1. In the left navigation pane, choose Subscriptions.

  2. On the Subscriptions page, choose Create subscription.

  3. On the Create subscription page, choose the Topic ARN field to see a list of the topics in your AWS account.

  4. Choose the topic that you created in the previous step.

  5. For Protocol, choose Email.

  6. For Endpoint, enter an email address that can receive notifications.

  7. Choose Create subscription.
The console opens the new subscription's Details page.

  8. Check your email inbox for a message from AWS Notifications, and then choose Confirm subscription in the email.

Amazon SNS opens your web browser and displays a subscription confirmation with your subscription ID.

Amazon SNS
TaskDescriptionSkills required

Create an alarm for the SNS topic.

To create an alarm based on a log group-metric filter:

  1. Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.

  2. From the navigation pane, choose Logs, and then choose Log groups.

  3. Choose the log group that includes your metric filter.

  4. Choose Metric filters.

  5. In the Metric filters tab, select the check box for the metric filter that you want to base your alarm on.

  6. Choose Create alarm.

  7. For Conditions, specify the following in each section:

    • For Threshold type, choose Static .

    • For Whenever <metric-name> is . . ., choose Greater.

    • For than . . ., specify 0.

  8. Choose Next.

  9. Under Notification:

    • For Alarm state trigger, choose In alarm.

    • For Send notification to following SNS topic, choose Select an existing topic.

    • In the email box, select the Amazon SNS topic that you created in the previous step.

  10. Choose Next.

  11. For Name and description, enter a name and description for your alarm.

    Note: For the description, you can specify the instance name so that the notification email is descriptive.

  12. For Preview and create, check that your configuration is correct, and then choose Create alarm.

After these steps, whenever these patterns are detected in the GoldenGate error log file (ggserr.log) that you are monitoring, you will get an email notification.

CloudWatch

Troubleshooting

IssueSolution

The log stream from the GoldenGate error log doesn’t flow into CloudWatch.

Check the /etc/awslogs/awslogs.conf file to verify the file name, log group name, and the date/time format. You must specify the date/time to match the date format in ggserror.log. Otherwise, the log stream won’t flow into CloudWatch.

Related resources