Additional resources - Centralized Logging with OpenSearch
GrafanaOpenSSL 1.1 InstallationCreate Instance Group for CentOS 7Upload SSL Certificate to IAMFix version `GLIBC_2.25' not found issue

Additional resources

Grafana

This section introduces how to set up a Grafana environment. If you want the solution to generate dashboards in Grafana automatically, you must perform the following deployment. If you only want to store the data in Amazon S3 without creating dashboards, you can skip this section.

Step 1: Install Grafana

Note

Skip this step if you already have a Grafana environment.

Prerequisite:

An EC2 instance has been launched, supporting both x86 and ARM architecture.

The following steps provide an example using m6g.medium instance type, ARM architecture, and Amazon 2023. For more details, refer to Install Grafana. 

# Edit/etc/yum.repos.d/grafana.repo file，input below content
[grafana]
name=grafana
baseurl=https://rpm.grafana.com
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://rpm.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt

# install grafana
yum install -y grafana

# Start grafana，and check its running status
systemctl start grafana-server
systemctl status grafana-server

# grafana listens on port 3000 by default, Users can edit /etc/grafana/grafana.ini to modify the configuration

# Acccess grafana，using the default credentials admin / admin，you will be promoted to change the password on the first login.
http://{instance-ip}:3000/

# If you need public access, please configure an Application Load Balancer (ALB) on your own.
# When configuring the ALB, modify the Idle timeout to 1800 to avoid the following error during large data queries (when a single API call exceeds 60 seconds)：
# "a padding to disable MSIE and Chrome friendly error page"

Step 2: Authorize the EC2 where Grafana is located to access Athena

Prerequisites:

  • You have deployed Grafana on EC2.

  • EC2 has been configured with an IAM Instance Profile. You must record the corresponding role ARN of the Instance Profile.

Follow these steps:

  1. Access IAM Management Console.

  2. Search for "AthenaPublicAccessRole" and choose it to access the details page. Record the role ARN, which will be used later.

  3. Choose the Trust relationships tab.

  4. Choose Edit trust policy.

  5. Choose Add next to Add a principal.

  6. Select IAM Roles from the Principal type dropdown list.

  7. Enter the role ARN that you recorded in Step 2.

  8. Choose Add principal.

  9. Choose update policy.

Step 3: Install Amazon Athena plugins

Prerequisites:

  • Grafana is installed.

  • Grafana is accessible over the public network.

Follow these steps:

  1. Access the Grafana console.

  2. Select Administration from the left navigation pane, and then choose Plugins.

  3. Select All in the State section on the right side.

  4. In the search box, enter Athena and choose the Amazon Athena to access the details page.

  5. Choose Install on the page and wait for the plugin installation to complete.

Step 4: Create service accounts

Follow these steps:

  1. Access the Grafana console.

  2. Select Administration from the left navigation pane, and then choose Service accounts.

  3. Select Add service account.

  4. Enter a display name. For example, "johndoe".

  5. Select the role as Admin.

  6. Choose Create.

  7. Choose Add service account token.

  8. Choose Generate token.

  9. Choose Copy to clipboard and close.

  10. Save and record this token, which will be used when you must create a pipeline.

OpenSSL 1.1 Installation

Centralized Logging with OpenSearch uses Fluent Bit as the log agent, which requires OpenSSL 1.1 or later. You can install the dependency according to your operating system (OS). It is recommended to make your own AMI with OpenSSL 1.1 installed.

Important

Important If your OS is not listed in the following sections, you can follow the official installation guide to install OpenSSL.

Amazon Linux 2

sudo yum install openssl11

Ubuntu

22.04 

ln -s /usr/lib/x86_64-linux-gnu/libsasl2.so.2 /usr/lib/libsasl2.so.3
ln -s /snap/core18/current/usr/lib/x86_64-linux-gnu/libssl.so.1.1 /usr/lib/libssl.so.1.1
ln -s /snap/core18/current/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 /usr/lib/libcrypto.so.1.1

20.04 

ln -s /usr/lib/x86_64-linux-gnu/libsasl2.so.2 /usr/lib/libsasl2.so.3

18.04 

ln -s /usr/lib/x86_64-linux-gnu/libsasl2.so.2 /usr/lib/libsasl2.so.3

Debian

GNU/10 

ln -s /usr/lib/x86_64-linux-gnu/libsasl2.so.2 /usr/lib/libsasl2.so.3

GNU/11 

ln -s /usr/lib/x86_64-linux-gnu/libsasl2.so.2 /usr/lib/libsasl2.so.3

Red Hat Enterprise Linux

8.X

OpenSSL 1.1 is installed by default.

7.X 

sudo su -

yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm

systemctl enable amazon-ssm-agent
systemctl start amazon-ssm-agent

yum install -y wget perl unzip gcc zlib-devel
mkdir /tmp/openssl
cd /tmp/openssl
wget https://www.openssl.org/source/openssl-1.1.1s.tar.gz
tar xzvf openssl-1.1.1s.tar.gz
cd openssl-1.1.1s
./config --prefix=/usr/local/openssl11 --openssldir=/usr/local/openssl11 shared zlib
make
make install

echo /usr/local/openssl11/lib/ >> /etc/ld.so.conf
ldconfig

SUSE Linux Enterprise Server

15

OpenSSL 1.1 is installed by default.

Create Instance Group for CentOS 7

Note

Note: CentOS Linux 7 will reach end of life (EOL) on June 30, 2024. It is not an OS well tested with this solution. Consider this guide as a reference only.

  1. Log in to your CentOS 7 machine and install SSM Agent manually. 

    sudo yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm
sudo systemctl enable amazon-ssm-agent
sudo systemctl start amazon-ssm-agent

  2. Go to the Instance Group panel of Centralized Logging with OpenSearch console, create Instance Group, select the CentOS 7 machine, choose Install log agent, and wait for its status to be offline.

  3. Log in to CentOS 7 and install fluent-bit 1.9.3 manually. 

    export RELEASE_URL=${FLUENT_BIT_PACKAGES_URL:-https://packages.fluentbit.io}
export RELEASE_KEY=${FLUENT_BIT_PACKAGES_KEY:-https://packages.fluentbit.io/fluentbit.key}

sudo rpm --import $RELEASE_KEY
cat << EOF | sudo tee /etc/yum.repos.d/fluent-bit.repo
[fluent-bit]
name = Fluent Bit
baseurl = $RELEASE_URL/centos/VERSION_ARCH_SUBSTR
gpgcheck=1
repo_gpgcheck=1
gpgkey=$RELEASE_KEY
enabled=1
EOF
sudo sed -i 's|VERSION_ARCH_SUBSTR|\$releasever/\$basearch/|g' /etc/yum.repos.d/fluent-bit.repo
sudo yum install -y fluent-bit-1.9.3-1

# Modify the configuration file
sudo sed -i 's/ExecStart.*/ExecStart=\/opt\/fluent-bit\/bin\/fluent-bit -c \/opt\/fluent-bit\/etc\/fluent-bit.conf/g' /usr/lib/systemd/system/fluent-bit.service
sudo systemctl daemon-reload
sudo systemctl enable fluent-bit
sudo systemctl start fluent-bit

  4. Go back to the Instance Groups panel of the Centralized Logging with OpenSearch console and wait for the CentOS 7 machine status to be Online and proceed to create the instance group.

Upload SSL Certificate to IAM

Upload the SSL certificate by running the AWS CLI command upload-server-certificate similar to the following: 

aws iam upload-server-certificate --path /cloudfront/ \
--server-certificate-name YourCertificate \
--certificate-body file://Certificate.pem \
--certificate-chain file://CertificateChain.pem \
--private-key file://PrivateKey.pem

Replace the file names and Your Certificate with the names for your uploaded files and certificate. Specify the file:// prefix in the certificate-body, certificate-chain, and private key parameters in the API request. Otherwise, the request fails with a MalformedCertificate: Unknown error message.

Note

You must specify a path using the --path option. The path must begin with /cloudfront and must include a trailing slash (for example, /cloudfront/test/).

After the certificate is uploaded, the AWS command upload-server-certificate returns metadata for the uploaded certificate, including the certificate's Amazon Resource Name (ARN), friendly name, identifier (ID), and expiration date.

To view the uploaded certificate, run the AWS CLI command list-server-certificates: 

aws iam list-server-certificates

For more information, see uploading a server certificate to IAM.

Fix version `GLIBC_2.25' not found issue

This error is caused by the old version of glibc. Centralized Logging with OpenSearch with a version later than 1.2 requires glibc-2.25 or above. So you must upgrade the existing version in EC2 first. The upgrade command for different kinds of OS is shown as follows:

Important

We strongly recommend you run the commands with environments first. Any upgrade failure may cause severe loss.

Redhat 7.9

For Redhat 7.9, the whole process will take about 2 hours, and at least 10 GB storage is needed. 

# install library
yum install -y gcc gcc-c++ m4 python3 bison  fontconfig-devel  libXpm-devel texinfo bzip2 wget
echo /usr/local/lib  >> /etc/ld.so.conf

# create tmp directory
mkdir -p /tmp/library
cd /tmp/library

# install gmp-6.1.0
wget https://ftp.gnu.org/gnu/gmp/gmp-6.1.0.tar.bz2
tar xjvf gmp-6.1.0.tar.bz2
cd gmp-6.1.0
./configure --prefix=/usr/local
make && make install
ldconfig
cd ..

# install mpfr-3.1.4
wget https://gcc.gnu.org/pub/gcc/infrastructure/mpfr-3.1.4.tar.bz2
tar xjvf mpfr-3.1.4.tar.bz2
cd mpfr-3.1.4
./configure --with-gmp=/usr/local --prefix=/usr/local
make && make install
ldconfig
cd ..

# install mpc-1.0.3
wget https://gcc.gnu.org/pub/gcc/infrastructure/mpc-1.0.3.tar.gz
tar xzvf mpc-1.0.3.tar.gz
cd mpc-1.0.3
./configure --prefix=/usr/local
make && make install
ldconfig
cd ..

# install gcc-9.3.0
wget https://ftp.gnu.org/gnu/gcc/gcc-9.3.0/gcc-9.3.0.tar.gz
tar xzvf gcc-9.3.0.tar.gz
cd gcc-9.3.0
mkdir build
cd build/
../configure --enable-checking=release --enable-language=c,c++ --disable-multilib --prefix=/usr
make -j4 && make install
ldconfig
cd ../..

# install make-4.3
wget https://ftp.gnu.org/gnu/make/make-4.3.tar.gz
tar xzvf make-4.3.tar.gz
cd make-4.3
mkdir build
cd build
../configure --prefix=/usr
make && make install
cd ../..

# install glibc-2.31
wget https://ftp.gnu.org/gnu/glibc/glibc-2.31.tar.gz
tar xzvf glibc-2.31.tar.gz
cd glibc-2.31
mkdir build
cd build/
../configure  --prefix=/usr --disable-profile --enable-add-ons --with-headers=/usr/include --with-binutils=/usr/bin --disable-sanity-checks --disable-werror
make all && make install
make localedata/install-locales

# clean tmp directory
cd /tmp
rm -rf /tmp/library

Ubuntu 22 

sudo ln -s /snap/core20/1623/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
sudo ln -s /snap/core20/1623/usr/lib/x86_64-linux-gnu/libssl.so.1.1 /usr/lib/x86_64-linux-gnu/libssl.so.1.1
sudo ln -s /usr/lib/x86_64-linux-gnu/libsasl2.so.2 /usr/lib/libsasl2.so.3

Amazon Linux 2023 

sudo su -
 
yum install -y wget perl unzip gcc zlib-devel
mkdir /tmp/openssl
cd /tmp/openssl
wget https://www.openssl.org/source/openssl-1.1.1s.tar.gz
tar xzvf openssl-1.1.1s.tar.gz
cd openssl-1.1.1s
./config --prefix=/usr/local/openssl11 --openssldir=/usr/local/openssl11 shared zlib
make
make install

echo /usr/local/openssl11/lib/ >> /etc/ld.so.conf
ldconfig