The default Amazon MSK configuration - Amazon Managed Streaming for Apache Kafka

The default Amazon MSK configuration

When you create an MSK cluster and don't specify a custom MSK configuration, Amazon MSK creates and uses a default configuration with the values shown in the following table. For properties that aren't in this table, Amazon MSK uses the defaults associated with your version of Apache Kafka. For a list of these default values, see Apache Kafka Configuration.

Default configuration values
Name Description Default value
allow.everyone.if.no.acl.found If no resource patterns match a specific resource, the resource has no associated ACLs. In this case, if you set this property ito true, everyone ican access the resource, not just the super users. true
auto.create.topics.enable Enables autocreation of a topic on the server. false
auto.leader.rebalance.enable Enables auto leader balancing. A background thread checks and initiates leader balance at regular intervals as if necessary. true
default.replication.factor Default replication factors for automatically created topics. 3 for 3-AZ clusters, 2 for 2-AZ clusters

local.retention.bytes

The maximum size of local log segments for a partition before it deletes the old segments. If you don't set this value, the value in log.retention.bytes is used. The effective value should always be less than or equal to the log.retention.bytes value. The default value of -2 indicates that there is no limit on local retention.This corresponds to the retention.ms/bytes setting of -1. The properties local.retention.ms and local.retention.bytes are similar to log.retention. This configuration is used to determine how long the log segmentsshould remain in local storage. Existing log.retention.* configurations are retention configurations for the topic partition. This includes both local and remote storage. Valid values: integers in [-2; +Inf]

-2 for unlimited

local.retention.ms

The number of milliseconds to retain the local log segment before it gets deleted. If you don't set this value, the value in log.retention.ms is used. The effective value should always be less than or equal to the log.retention.bytes value. The default value of -2 indicates that there is no limit on local retention. This corresponds to the retention.ms/bytes setting of -1.

The values local.retention.ms and local.retention.bytes are similar to log.retention. This configuration is used to determine how long the log segments should remain in local storage. Existing log.retention.* configurations are retention configurations for the topic partition. This includes both local and remote storage. Valid values: > 0
-2 for unlimited
min.insync.replicas

When a producer sets the value of acks to "all" (or "-1"), the value in min.insync.replicas specifies the minimum number of replicas that must acknowledge a write for the write to be considered successful. If this value doesn't meet this minimum, the producer raises an exception (either NotEnoughReplicas or NotEnoughReplicasAfterAppend).

When you use the values in min.insync.replicas and acks together, you can enforce greater durability guarantees. For example, you might create a topic with a replication factor of 3, set min.insync.replicas to 2, and produce with acks of "all". This ensures that the producer raises an exception if a majority of replicas don't receive a write.

2 for 3-AZ clusters, 1 for 2-AZ clusters
num.io.threads Number of threads that the server uses to produce requests, which may include disk I/O. 8
num.network.threads Number of threads that the server uses to receive requests from the network andsend responses to the network. 5
num.partitions Default number of log partitions per topic. 1
num.replica.fetchers Number of fetcher threads used to replicate messages from a source broker.If you increase this value, you can increase the degree of I/O parallelism in the follower broker. 2

remote.log.msk.disable.policy

Used with remote.storage.enable to disable tiered storage. Set this policy to Delete, to indicate that data in tiered storage is deleted when you set remote.storage.enable to false.

DELETE

remote.storage.enable

Enables tiered (remote) storage for a topic if set to true. Disables topic level tiered storage if set to false and remote.log.msk.disable.policy is set to Delete. When you disable tiered storage, you delete data from remote storage. When you disable tiered storage for a topic, you can't enable it again. false
replica.lag.time.max.ms If a follower hasn't sent any fetch requests or hasn't consumed up to the leader's log end offset for at least this number of milliseconds, the leader removes the follower from the ISR. 30000

retention.ms

Mandatory field. Minimum time is 3 days. There is no default because the setting is mandatory.

Amazon MSK uses the retention.ms value with local.retention.ms to determine when data moves from local to tiered storage. The local.retention.ms value specifies when to move data from local to tiered storage. The retention.ms value specifies when to remove data ifrom tiered storage (that is, removed from the cluster). Valid values: integers in [-1; +Inf]

Minimum 259,200,000 milliseconds (3 days). -1 for infinite retention.
socket.receive.buffer.bytes The SO_RCVBUF buffer of the socket sever sockets. If the value is -1, the OS default is used. 102400
socket.request.max.bytes Maximum number of bytes in a socket request. 104857600
socket.send.buffer.bytes The SO_SNDBUF buffer of the socket sever sockets. If the value is -1, the OS default is used. 102400
unclean.leader.election.enable Indicates if you want replicas not in the ISR set to serve as leader as a last resort, even though this might result in data loss. true
zookeeper.session.timeout.ms

The Apache ZooKeeper session timeout in milliseconds.

18000
zookeeper.set.acl The set client to use secure ACLs. false

For information on how to specify custom configuration values, see Custom MSK configurations.