Class S3CrtDirectBufferPoolConfiguration
Multipart uploads, and all downloads except GetObject requests that set partNumber, use the pool.
Single-part uploads don't. Each request that uses the pool holds at least one full part of pool memory while it runs, so a
1 KiB download still uses an 8 MiB part with the default part size.
Downloads that use S3CrtResponseTransformerProvider.toBlockingInputStreamFromDirectBufferPool() hold pool memory
until the application reads or closes the stream, so a slow reader can delay other requests on the client. A starting
size is the number of concurrent downloads that use the pool times (initialReadBufferSizeInBytes plus one part
size), plus headroom for the other downloads and multipart uploads on the client. Each download can have one requested
part beyond its read window. If the pool is too small, some downloads can stop receiving data until others finish. The
SDK does not time out a download that stops after its stream is returned. The application can close the stream to end
it.
The default initialReadBufferSizeInBytes, 10 times the part size (80 MiB with the default part size), can
limit the throughput of each download on high-bandwidth instances. Raising it can increase throughput per download, but
each download can then hold more pool memory.
Start with auto(), which lets CRT size the pool. Use elastic(long, long) to set the ceiling
yourself. An explicit ceiling, set through elastic(long, long) or through maxNativeMemoryLimitInBytes,
must fit within 80% of the JVM direct-memory limit (-XX:MaxDirectMemorySize), or client creation fails. An
elastic(long, long) ceiling must also not exceed maxNativeMemoryLimitInBytes when that is set. For
auto(), a ceiling set through CRT's AWS_CRT_S3_MEMORY_LIMIT_IN_MB or
AWS_CRT_S3_MEMORY_LIMIT_IN_GIB environment variable is also explicit. An auto() pool without an
explicit ceiling derives its ceiling from the throughput target. If that ceiling doesn't fit within the direct-memory
limit, it's lowered to fit and CRT logs a warning, instead of failing client creation.
With a pool, an explicitly set minimumPartSizeInBytes is used as the exact upload part size and is never
raised. In this case, a known-size upload above the multipart threshold fails before sending data if that part size is
below S3's 5 MiB minimum for non-final parts or the object needs more than 10,000 parts. Without an explicit part size,
a known-size upload that needs larger parts than the pool can provide fails before sending data. This doesn't apply
to uploads from a file path, such as putObject(PutObjectRequest, Path) or Transfer Manager
uploadFile. For those, CRT can raise the part size as it does without a pool.
-
Method Summary
Modifier and TypeMethodDescriptionauto()Creates a configuration for a pool whose memory limit is automatically determined by CRT.elastic(long minimumSizeInBytes, long memoryLimitInBytes) Creates a configuration for a pool that grows on demand to its ceiling and releases unused capacity above its floor when the client is idle.booleaninthashCode()Returns the configured elastic pool ceiling.Returns the configured elastic pool floor.toString()
-
Method Details
-
auto
Creates a configuration for a pool whose memory limit is automatically determined by CRT.Unless a ceiling is set through
maxNativeMemoryLimitInBytesor CRT'sAWS_CRT_S3_MEMORY_LIMIT_IN_MBorAWS_CRT_S3_MEMORY_LIMIT_IN_GIBenvironment variable, CRT derives the ceiling from the client's throughput target. The pool allocates 16 parts (128 MiB with the default part size) when the client is created, or the whole pool if the ceiling is smaller, and keeps them while idle. Above that, it grows with demand and releases unused memory when the client is idle. The derived ceiling can be reduced to fit the JVM direct-memory limit. Useelastic(long, long)for a known number of concurrent downloads that use the pool and need an explicit ceiling.- Returns:
- an automatically sized pool configuration
-
elastic
public static S3CrtDirectBufferPoolConfiguration elastic(long minimumSizeInBytes, long memoryLimitInBytes) Creates a configuration for a pool that grows on demand to its ceiling and releases unused capacity above its floor when the client is idle.The pool keeps its floor allocated from client creation, even while idle. The floor is rounded down to blocks of 16 parts (128 MiB with the default part size). A floor below one block keeps no memory allocated unless it reaches the ceiling, in which case the pool keeps its whole capacity allocated. The ceiling is
memoryLimitInBytesrounded down to whole parts.- Parameters:
minimumSizeInBytes- requested pool floor in bytesmemoryLimitInBytes- requested pool ceiling in bytes- Returns:
- an elastic pool configuration
- Throws:
IllegalArgumentException- ifminimumSizeInBytesis negative,memoryLimitInBytesis not positive, orminimumSizeInBytesexceedsmemoryLimitInBytes
-
memoryLimitInBytes
-
minimumSizeInBytes
-
equals
-
hashCode
-
toString
-