Troubleshooting for RDS Proxy
Following, you can find troubleshooting ideas for some common RDS Proxy issues and information on CloudWatch logs for RDS Proxy.
In the RDS Proxy logs, each entry is prefixed with the name of the associated proxy
endpoint. This name can be the name that you specified for a user-defined endpoint. Or, it can
be the special name default
for the default endpoint of a proxy that performs
read/write requests. For more information about proxy endpoints, see Working with Amazon RDS Proxy endpoints.
Verifying connectivity for a proxy
You can use the following commands to verify that all components such as the proxy, database, and compute instances in the connection can communicate with the each other.
Examine the proxy itself using the describe-db-proxies command. Also examine the associated target group using the describe-db-proxy-target-groups command. Check that the details of the targets match the RDS DB instance that you intend to associate with the proxy. Use commands such as the following.
aws rds describe-db-proxies --db-proxy-name $DB_PROXY_NAME aws rds describe-db-proxy-target-groups --db-proxy-name $DB_PROXY_NAME
To confirm that the proxy can connect to the underlying database, examine the targets specified in the target groups using the describe-db-proxy-targets command. Use a command such as the following.
aws rds describe-db-proxy-targets --db-proxy-name $DB_PROXY_NAME
The output of the
describe-db-proxy-targets command
includes a TargetHealth
field. You can examine the fields State
,
Reason
, and Description
inside TargetHealth
to check if the proxy can
communicate with the underlying DB instance.
-
A
State
value ofAVAILABLE
indicates that the proxy can connect to the DB instance. -
A
State
value ofUNAVAILABLE
indicates a temporary or permanent connection problem. In this case, examine theReason
andDescription
fields. For example, ifReason
has a value ofPENDING_PROXY_CAPACITY
, try connecting again after the proxy finishes its scaling operation. IfReason
has a value ofUNREACHABLE
,CONNECTION_FAILED
, orAUTH_FAILURE
, use the explanation from theDescription
field to help you diagnose the issue. -
The
State
field might have a value ofREGISTERING
for a brief time before changing toAVAILABLE
orUNAVAILABLE
.
If the following Netcat command (nc
) is successful, you can access the proxy endpoint from the
EC2 instance or other system where you're logged in. This command reports failure if you're not in
the same VPC as the proxy and the associated database. You might be able to log directly in to the database
without being in the same VPC. However, you can't log into the proxy unless you're in the same
VPC.
nc -zx
MySQL_proxy_endpoint
3306 nc -zxPostgreSQL_proxy_endpoint
5432
You can use the following commands to make sure that your EC2 instance has the required properties. In particular, the VPC for the EC2 instance must be the same as the VPC for the that the proxy connects to.
aws ec2 describe-instances --instance-ids
your_ec2_instance_id
Examine the Secrets Manager secrets used for the proxy.
aws secretsmanager list-secrets aws secretsmanager get-secret-value --secret-id
your_secret_id
Make sure that the SecretString
field displayed by
get-secret-value
is encoded as a JSON string that includes the
username
and password
fields. The following example shows the
format of the SecretString
field.
{
"ARN": "some_arn
",
"Name": "some_name
",
"VersionId": "some_version_id",
"SecretString": '{"username":"some_username","password":"some_password
"}',
"VersionStages": [ "some_stage
" ],
"CreatedDate": some_timestamp
}
Common issues and solutions
This section describes some common issues and potential solutions when using RDS Proxy.
After running the aws rds describe-db-proxy-targets
CLI command, if the
TargetHealth
description states Proxy does not have any registered
credentials
, verify the following:
-
There are credentials registered for the user to access the proxy.
-
The IAM role to access the Secrets Manager secret used by the proxy is valid.
You might encounter the following RDS events while creating or connecting to a DB proxy.
Category | RDS event ID | Description |
---|---|---|
failure |
RDS-EVENT-0243 | RDS couldn't provision capacity for the proxy because there aren't enough IP addresses available in your subnets. To fix the issue, make sure that your subnets have the minimum number of unused IP addresses. To determine the recommended number for your instance class, see Planning for IP address capacity. |
failure |
RDS-EVENT-0275 |
RDS throttled some connections to DB proxy |
You might encounter the following issues while creating a new proxy or connecting to a proxy.
Error | Causes or workarounds |
---|---|
|
Select an existing IAM role instead of choosing to create a new one. |
You might encounter the following issues while connecting to a MySQL proxy.
Error | Causes or workarounds |
---|---|
ERROR 1040 (HY000): Connections rate limit exceeded (
|
The rate of connection requests from the client to the proxy has exceeded the limit. |
ERROR 1040 (HY000): IAM authentication rate limit exceeded
|
The number of simultaneous requests with IAM authentication from the client to the proxy has exceeded the limit. |
ERROR 1040 (HY000): Number simultaneous connections exceeded (
|
The number of simultaneous connection requests from the client to the proxy exceeded the limit. |
|
The Secrets Manager secret used by the proxy doesn't match the user name and password of an existing database user. Either update the credentials in the Secrets Manager secret, or make sure the database user exists and has the same password as in the secret. |
ERROR 1105 (HY000): Unknown error
|
An unknown error occurred. |
ERROR 1231 (42000): Variable ''character_set_client'' can't be set to the value of
|
The value set for the |
ERROR 3159 (HY000): This RDS Proxy requires TLS connections.
|
You enabled the setting Require Transport Layer Security in the proxy but
your connection included the parameter
|
ERROR 2026 (HY000): SSL connection error: Internal Server
|
The TLS handshake to the proxy failed. Some possible reasons include the following:
|
ERROR 9501 (HY000): Timed-out waiting to acquire database connection
|
The proxy timed-out waiting to acquire a database connection. Some possible reasons include the following:
|
You might encounter the following issues while connecting to a PostgreSQL proxy.
Error | Cause | Solution |
---|---|---|
|
The user tried to connect to the database using IAM authentication with the setting
|
The user needs to connect to the database using the minimum setting of
|
|
The user enabled the option Require Transport Layer Security but tried to
connect with |
To fix this error, do one of the following:
|
|
This error might be due to the following reasons:
|
To fix this error, do the following:
|
|
There is no Secrets Manager secret for this role. |
Add a Secrets Manager secret for this role. For more information, see Setting up AWS Identity and Access Management (IAM) policies for RDS Proxy. |
|
The database client being used to connect to the proxy is using an authentication mechanism not currently supported by the proxy. |
If you're not using IAM authentication, use the MD5 or SCRAM password authentication. |
|
The database client being used to connect to the proxy isn't sending a user name when trying to establish a connection. |
Make sure to define a user name when setting up a connection to the proxy using the PostgreSQL client of your choice. |
|
The PostgreSQL client used to connect to the proxy uses a protocol older than 3.0. |
Use a newer PostgreSQL client that supports the 3.0 messaging protocol. If you're using the PostgreSQL |
|
The PostgreSQL client used to connect to the proxy is trying to use the streaming replication mode, which isn't currently supported by RDS Proxy. |
Turn off the streaming replication mode in the PostgreSQL client being used to connect. |
|
Through the startup message, the PostgreSQL client used to connect to the proxy is requesting an option that isn't currently supported by RDS Proxy. |
Turn off the option being shown as not supported from the message above in the PostgreSQL client being used to connect. |
|
The number of simultaneous requests with IAM authentication from the client to the proxy has exceeded the limit. |
Reduce the rate in which connections using IAM authentication from a PostgreSQL client are established. |
|
The number of simultaneous connection requests from the client to the proxy exceeded the limit. |
Reduce the number of active connections from PostgreSQL clients to this RDS proxy. |
|
The rate of connection requests from the client to the proxy has exceeded the limit. |
Reduce the rate in which connections from a PostgreSQL client are established. |
|
The password for this role doesn't match the Secrets Manager secret. |
Check the secret for this role in Secrets Manager to see if the password is the same as what's being used in your PostgreSQL client. |
|
There is a problem with the IAM token used for IAM authentication. |
Generate a new authentication token and use it in a new connection. |
|
A client tried to connect using IAM authentication, but SSL wasn't enabled. |
Enable SSL in the PostgreSQL client. |
|
An unknown error occurred. |
Reach out to AWS Support to investigate the issue. |
|
The proxy timed-out waiting to acquire a database connection. Some possible reasons include the following:
|
Possible solutions are the following:
|
|
The database connection established from the proxy returned an error. |
The solution depends on the specific database error. One example is:
|