本文為英文版的機器翻譯版本，如內容有任何歧義或不一致之處，概以英文版為準。
搭配目錄儲存貯體使用多部分上傳
您可以使用分段上載流程將單一物件上載為一組零件。每個組件都是物件資料的接續部分。您可依任何順序分別上傳這些物件組件。若任何組件的傳輸失敗，您可再次傳輸該組件，而不會影響其他組件。當物件的所有組件都全部上傳完後，Amazon S3 會將這些組件組合起來建立該物件。一般而言，當物件大小達到 100 MB 時，應考慮使用分段上傳，而不是以單次操作上傳物件。
使用分段上傳具備下列優勢：
-
改善輸送量 - 您可平行上傳各組件以改進輸送量。
-
從任何網路問題中快速復原 — 較小的零件尺寸可將因網路錯誤而重新啟動失敗上傳的影響降到最低。
-
暫停及繼續上傳物件 - 您可在一段時間內上傳物件組件。啟動分段上傳之後，就沒有到期日。您必須明確完成或中止分段上傳。
-
在您知道最終物件大小前開始上傳 - 您可在建立物件的同時上傳物件。
我們建議您以下列方式使用分段上傳：
當您使用分段上傳將物件上傳到目錄儲存貯體中的 Amazon S3 Express One Zone Zone 儲存類別時，多部分上傳程序類似於使用多部分上傳將物件上傳到一般用途儲存貯體的程序。不過，還是有一些顯著的差異。
如需使用分段上傳將物件上傳到 S3 Express One 區域的相關資訊，請參閱下列主題。
多部分上傳過程
多部分上傳是一個三個步驟的過程：
-
您啟動上傳。
-
您上傳物件的零件。
-
上傳完所有零件之後，即可完成多部分上傳。
收到完整的分段上傳請求後，Amazon S3 會從上傳的零件建構物件，然後您就可以像存取儲存貯體中任何其他物件一樣存取物件。
啟動分段上傳
當您傳送要求要啟動分段上傳時，Amazon S3 會傳回具有上傳 ID 的回應，其為分段上傳的唯一識別符。每次上傳分段各組件、列出各組件、完成上傳或中止上傳時，都必須納入此上傳 ID。
組件上傳
上傳某個分段組件時，除了上傳 ID 之外，還必須指定組件編號。當您透過 S3 Express One Zone 使用多部分上傳時，多部分零件編號必須是連續的零件編號。如果您嘗試完成具有非連續零件編號的多部分上傳請求，則會產生 HTTP 400 Bad Request (無效的零件訂單) 錯誤。
零件編號可唯一識別零件及其在您要上載之物件中的位置。如果您使用與先前上載零件相同的零件編號來上載新零件，則先前上載的零件會遭到覆寫。
每次上傳一個組件時，Amazon S3 會在其回應中傳回 企業標籤 (ETag) 標頭。您必須記錄每個上傳組件的組件編號與 ETag 值。所有物件零件上載的 ETag 值將保持不變，但每個零件都會被指派不同的零件編號。後續的要求中必須包含這些值，才能完成分段上傳。
Amazon S3 會自動加密上傳到 S3 儲存貯體的所有新物件。進行分段上傳時，若您未在請求中指定加密資訊，所上傳分段的加密設定會設為目的地儲存貯體的預設加密組態。Amazon S3 儲存貯體的預設加密組態一律為啟用狀態，且最低限度設定為使用 Amazon S3 受管金鑰 (SSE-S3) 的伺服器端加密。對於目錄值區，僅支援 SSE-S3。如需詳細資訊，請參閱 使用 Amazon S3 受管金鑰 (SSE-S3) 的伺服器端加密。
完成分段上傳
當您完成分段上傳時，Amazon S3 會根據零件編號以遞增順序串連零件來建立物件。成功完成請求之後，這些片段就不再存在。
您完整的分段上傳請求必須包含上傳 ID 以及零件編號及其對應 ETag 值的清單。Amazon S3 回應包含的 ETag 可識別獨特的物件資料組合。這個 ETag 不是物件資料的 MD5 雜湊。
分段上傳清單
您可列出特定分段上傳的組件或所有進行中之分段上傳。列出組件操作會傳回特定分段上傳之已上傳組件的資訊。Amazon S3 會為每項列出的組件要求，傳回指定分段上傳組件的資訊，上限為 1,000 個組件。若分段上傳中有超過 1,000 個分段，您必須使用分頁來擷取所有分段。
傳回的零件清單不包含尚未完成上傳的零件。使用列出分段上傳操作，即可取得正在進行中的分段上傳清單。
進行中的分段上傳是您已啟動但尚未完成或已中止的上傳。每個要求最多可傳回 1,000 個分段上傳。若正在進行超過 1,000 個的分段上傳，您必須另行傳送請求以擷取剩餘的分段上傳。傳回的清單僅用於進行驗證。傳送完成分段上傳請求時，請不要使用此清單的結果。而是在上傳 Amazon S3 傳回的組件與相對應之 ETag 值時，保有您自己的組件編號清單。
如需有關分段上傳清單的詳細資訊，請參閱 Amazon 簡單儲存服務 API 參考ListParts中的。
使用分段上傳操作的檢查總和
當您上傳物件時，可以指定檢查總和演算法來檢查物件完整性。目錄儲存貯體不支援 MD5。您可以指定下列其中一個安全雜湊演算法 (SHA) 或循環冗餘檢查 (CRC) 資料完整性檢查演算法：
-
CRC32
-
CRC32C
-
SHA-1
-
SHA-256
您可以使用 Amazon S3 REST API 或AWS開發套件，透過使GetObject用或擷取個別零件的總和檢查值。HeadObject如果想要擷取分段上傳 (仍在進行中) 之個別部分的總和檢查值，您可以使用 ListParts。
使用上述總和檢查碼演算法時，多組件零件編號必須使用連續的零件編號。如果您嘗試使用不連續的零件編號完成多部分上傳請求，Amazon S3 會產生 HTTP 400 Bad Request (零件訂單無效) 錯誤。
如需如何使用分段物件檢查總和的詳細資訊，請參閱「檢查物件完整性」。
並行分段上傳操作
在分散式開發環境中，您的應用程式可以同時在同一物件上啟動多個更新。例如，您的應用程式可能會使用相同的物件索引鍵來啟動數個分段上傳。然後針對這些每一個上傳，應用程式會上傳各組件，並對 Amazon S3 傳送完成上傳要求，以建立物件。對於 S3 Express One Zone 來說，物件建立時間是指分段上傳的完成日期。
在您啟動多部分上傳然後完成該上傳之間，Amazon S3 收到的另一個請求可能會優先考慮。例如，假設您使用金鑰名稱largevideo.mp4起始多部分上傳。在您完成上傳之前，另一項作業會刪除largevideo.mp4金鑰。在這種情況下，完整的多部分上傳回應可能表示您在largevideo.mp4沒有看到物件的情況下成功建立物件。
分段上傳和定價
啟動分段上傳之後，Amazon S3 就會保留所有分段，直到您完成或中止上傳為止。在其整個生命週期內，您都要支付此分段上傳及其相關組件的儲存體、頻寬與要求之費用。如果您中止分段上傳，Amazon S3 會刪除上傳成品和您上傳的任何零件，而且不會再支付這些成品的費用。無論指定的儲存空間類別為何，刪除不完整的分段上傳都不會收取提前刪除費用。如需定價的詳細資訊，請參閱 Amazon S3 定價。
如果完整的多部分上載請求未成功發送，則對象部件將被「組裝」，並且不會創建對象。系統會向您收取與已上傳部分相關的所有儲存空間費用。請務必完成多部分上載以建立物件，或中止多部分上載以移除所有上載的零件。
您必須完成或中止所有進行中的多部分上傳，才能刪除目錄值區。目錄儲存貯體不支援 S3 生命週期組態。如果需要，您可以列出活動中的分段上傳內容，然後中止上傳，然後刪除值區。
多部分上傳 API 操作和權限
若要允許存取目錄值區上的物件管理 API 作業，請在值區政策或 AWS Identity and Access Management (IAM) 身分型政策中授與s3express:CreateSession權限。
您必須要有必要許可，才可使用分段上傳操作。您可以使用儲存貯體政策或 IAM 身分型政策授與 IAM 主體許可以執行這些操作。下表列出各種分段上傳操作所需的許可。
您可以透過元素識別多部分上傳的初始器。Initiator如果初始器是AWS 帳戶，則此元素會提供與元Owner素相同的資訊。若啟動者是 IAM 使用者，此元素會提供使用者 ARN 與顯示名稱。
| 動作 |
所需的許可 |
|
建立分段上傳
|
若要建立分段上傳，您必須被允許對目錄值區執行s3express:CreateSession動作。
|
|
啟動多部分上傳
|
若要啟動分段上傳，您必須被允許對目錄值區執行s3express:CreateSession動作。
|
| 上傳零件 |
若要上載零件，您必須被允許對目錄值區執行s3express:CreateSession動作。
若要讓初始器上傳零件，值區擁有者必須允許初始器對目錄值區執行s3express:CreateSession動作。
|
| 上傳零件 (副本) |
若要上載零件，您必須被允許對目錄值區執行s3express:CreateSession動作。
啟動者若要分段上傳該物件，儲存貯體擁有者必須允許啟動者對物件執行 s3express:CreateSession 動作。
|
| 完成分段上傳 |
若要完成分段上傳，您必須被允許對目錄值區執行s3express:CreateSession動作。
若要讓啟動者完成分段上傳，值區擁有者必須允許初始器對物件執行s3express:CreateSession動作。
|
| 中止分段上傳 |
若要中止多部分上載，您必須被允許執行動作s3express:CreateSession。
若要讓啟動器中止多部分上載，必須授與啟動器明確允許存取權，才能執行動作。s3express:CreateSession
|
| 清單零件 |
若要列出分段上傳中的零件，您必須被允許對目錄值區執行s3express:CreateSession動作。
|
| 列出進行中的分段上傳 |
若要列出儲存貯體進行中的分段上傳作業，您必須被允許對該值區執行s3:ListBucketMultipartUploads動作。
|
多部分上傳的 API 操作支持
Amazon 簡單儲存服務 API 參考中的以下各節說明用於多部分上傳的 Amazon S3 REST API 操作。
範例
若要使用分段上傳將物件上傳至目錄儲存貯體中的 S3 Express One Zone 區域，請參閱下列範例。
建立分段上傳
下列範例將使用和來建立多部分上傳作業。AWS SDK for Java 2.x AWS SDK for Python (Boto3)
- SDK for Java 2.x
-
/**
* This method creates a multipart upload request that generates a unique upload ID that is used to track
* all the upload parts
*
* @param s3
* @param bucketName - for example, 'doc-example-bucket--use1-az4--x-s3'
* @param key
* @return
*/
private static String createMultipartUpload(S3Client s3, String bucketName, String key) {
CreateMultipartUploadRequest createMultipartUploadRequest = CreateMultipartUploadRequest.builder()
.bucket(bucketName)
.key(key)
.build();
String uploadId = null;
try {
CreateMultipartUploadResponse response = s3.createMultipartUpload(createMultipartUploadRequest);
uploadId = response.uploadId();
}
catch (S3Exception e) {
System.err.println(e.awsErrorDetails().errorMessage());
System.exit(1);
}
return uploadId;
- SDK for Python
-
def create_multipart_upload(s3_client, bucket_name, key_name):
'''
Create a multipart upload to a directory bucket
:param s3_client: boto3 S3 client
:param bucket_name: The destination bucket for the multipart upload
:param key_name: The key name for the object to be uploaded
:return: The UploadId for the multipart upload if created successfully, else None
'''
try:
mpu = s3_client.create_multipart_upload(Bucket = bucket_name, Key = key_name)
return mpu['UploadId']
except ClientError as e:
logging.error(e)
return None
上傳分段上傳的部分
下列範例說明如何將單一物件分割成多個部分，然後使用適用於 Java 2.x 的 SDK 和 Python SDK 將這些部分上傳至目錄儲存貯體。
- SDK for Java 2.x
-
/**
* This method creates part requests and uploads individual parts to S3 and then returns all the completed parts
*
* @param s3
* @param bucketName
* @param key
* @param uploadId
* @throws IOException
*/
private static ListCompletedPartmultipartUpload(S3Client s3, String bucketName, String key, String uploadId, String filePath) throws IOException {
int partNumber = 1;
ListCompletedPart completedParts = new ArrayList<>();
ByteBuffer bb = ByteBuffer.allocate(1024 * 1024 * 5); // 5 MB byte buffer
// read the local file, breakdown into chunks and process
try (RandomAccessFile file = new RandomAccessFile(filePath, "r")) {
long fileSize = file.length();
int position = 0;
while (position < fileSize) {
file.seek(position);
int read = file.getChannel().read(bb);
bb.flip(); // Swap position and limit before reading from the buffer.
UploadPartRequest uploadPartRequest = UploadPartRequest.builder()
.bucket(bucketName)
.key(key)
.uploadId(uploadId)
.partNumber(partNumber)
.build();
UploadPartResponse partResponse = s3.uploadPart(
uploadPartRequest,
RequestBody.fromByteBuffer(bb));
CompletedPart part = CompletedPart.builder()
.partNumber(partNumber)
.eTag(partResponse.eTag())
.build();
completedParts.add(part);
bb.clear();
position += read;
partNumber++;
}
}
catch (IOException e) {
throw e;
}
return completedParts;
}
- SDK for Python
-
def multipart_upload(s3_client, bucket_name, key_name, mpu_id, part_size):
'''
Break up a file into multiple parts and upload those parts to a directory bucket
:param s3_client: boto3 S3 client
:param bucket_name: Destination bucket for the multipart upload
:param key_name: Key name for object to be uploaded and for the local file that's being uploaded
:param mpu_id: The UploadId returned from the create_multipart_upload call
:param part_size: The size parts that the object will be broken into, in bytes.
Minimum 5 MiB, Maximum 5 GiB. There is no minimum size for the last part of your multipart upload.
:return: part_list for the multipart upload if all parts are uploaded successfully, else None
'''
part_list = []
try:
with open(key_name, 'rb') as file:
part_counter = 1
while True:
file_part = file.read(part_size)
if not len(file_part):
break
upload_part = s3_client.upload_part(
Bucket = bucket_name,
Key = key_name,
UploadId = mpu_id,
Body = file_part,
PartNumber = part_counter
)
part_list.append({'PartNumber': part_counter, 'ETag': upload_part['ETag']})
part_counter += 1
except ClientError as e:
logging.error(e)
return None
return part_list
完成分段上傳
下列範例說明如何使用適用於 Java 2.x 的 SDK 和適用於 Python 的 SDK 來完成多部分上傳作業。
- SDK for Java 2.x
-
/**
* This method completes the multipart upload request by collating all the upload parts
* @param s3
* @param bucketName - for example, 'doc-example-bucket--usw2-az1--x-s3'
* @param key
* @param uploadId
* @param uploadParts
*/
private static void completeMultipartUpload(S3Client s3, String bucketName, String key, String uploadId, ListCompletedPart uploadParts) {
CompletedMultipartUpload completedMultipartUpload = CompletedMultipartUpload.builder()
.parts(uploadParts)
.build();
CompleteMultipartUploadRequest completeMultipartUploadRequest =
CompleteMultipartUploadRequest.builder()
.bucket(bucketName)
.key(key)
.uploadId(uploadId)
.multipartUpload(completedMultipartUpload)
.build();
s3.completeMultipartUpload(completeMultipartUploadRequest);
}
public static void multipartUploadTest(S3Client s3, String bucketName, String key, String localFilePath) {
System.out.println("Starting multipart upload for: " + key);
try {
String uploadId = createMultipartUpload(s3, bucketName, key);
System.out.println(uploadId);
ListCompletedPart parts = multipartUpload(s3, bucketName, key, uploadId, localFilePath);
completeMultipartUpload(s3, bucketName, key, uploadId, parts);
System.out.println("Multipart upload completed for: " + key);
}
catch (Exception e) {
System.err.println(e.getMessage());
System.exit(1);
}
}
- SDK for Python
-
def complete_multipart_upload(s3_client, bucket_name, key_name, mpu_id, part_list):
'''
Completes a multipart upload to a directory bucket
:param s3_client: boto3 S3 client
:param bucket_name: The destination bucket for the multipart upload
:param key_name: The key name for the object to be uploaded
:param mpu_id: The UploadId returned from the create_multipart_upload call
:param part_list: The list of uploaded part numbers with their associated ETags
:return: True if the multipart upload was completed successfully, else False
'''
try:
s3_client.complete_multipart_upload(
Bucket = bucket_name,
Key = key_name,
UploadId = mpu_id,
MultipartUpload = {
'Parts': part_list
}
)
except ClientError as e:
logging.error(e)
return False
return True
if __name__ == '__main__':
MB = 1024 ** 2
region = 'us-west-2'
bucket_name = 'BUCKET_NAME'
key_name = 'OBJECT_NAME'
part_size = 10 * MB
s3_client = boto3.client('s3', region_name = region)
mpu_id = create_multipart_upload(s3_client, bucket_name, key_name)
if mpu_id is not None:
part_list = multipart_upload(s3_client, bucket_name, key_name, mpu_id, part_size)
if part_list is not None:
if complete_multipart_upload(s3_client, bucket_name, key_name, mpu_id, part_list):
print (f'{key_name} successfully uploaded through a ultipart upload to {bucket_name}')
else:
print (f'Could not upload {key_name} hrough a multipart upload to {bucket_name}')
中止分段上傳
下列範例說明如何使用適用於 Java 2.x 的 SDK 和適用於 Python 的 SDK 中止多部分上傳作業。
- SDK for Java 2.x
-
public static void abortMultiPartUploads( S3Client s3, String bucketName ) {
try {
ListMultipartUploadsRequest listMultipartUploadsRequest = ListMultipartUploadsRequest.builder()
.bucket(bucketName)
.build();
ListMultipartUploadsResponse response = s3.listMultipartUploads(listMultipartUploadsRequest);
ListMultipartUpload uploads = response.uploads();
AbortMultipartUploadRequest abortMultipartUploadRequest;
for (MultipartUpload upload: uploads) {
abortMultipartUploadRequest = AbortMultipartUploadRequest.builder()
.bucket(bucketName)
.key(upload.key())
.uploadId(upload.uploadId())
.build();
s3.abortMultipartUpload(abortMultipartUploadRequest);
}
}
catch (S3Exception e) {
System.err.println(e.getMessage());
System.exit(1);
}
}
- SDK for Python
-
import logging
import boto3
from botocore.exceptions import ClientError
def abort_multipart_upload(s3_client, bucket_name, key_name, upload_id):
'''
Aborts a partial multipart upload in a directory bucket.
:param s3_client: boto3 S3 client
:param bucket_name: Bucket where the multipart upload was initiated - for example, 'doc-example-bucket--usw2-az1--x-s3'
:param key_name: Name of the object for which the multipart upload needs to be aborted
:param upload_id: Multipart upload ID for the multipart upload to be aborted
:return: True if the multipart upload was successfully aborted, False if not
'''
try:
s3_client.abort_multipart_upload(
Bucket = bucket_name,
Key = key_name,
UploadId = upload_id
)
except ClientError as e:
logging.error(e)
return False
return True
if __name__ == '__main__':
region = 'us-west-2'
bucket_name = 'BUCKET_NAME'
key_name = 'KEY_NAME'
upload_id = 'UPLOAD_ID'
s3_client = boto3.client('s3', region_name = region)
if abort_multipart_upload(s3_client, bucket_name, key_name, upload_id):
print (f'Multipart upload for object {key_name} in {bucket_name} bucket has been aborted')
else:
print (f'Unable to abort multipart upload for object {key_name} in {bucket_name} bucket')
建立分段上傳複製操作
下列範例說明如何使用多部分上傳，透過使用適用於 Java 2.x 的 SDK 和 Python SDK，以程式設計方式將物件從一個值區複製到另一個值區。
- SDK for Java 2.x
-
/**
* This method creates a multipart upload request that generates a unique upload ID that is used to track
* all the upload parts.
*
* @param s3
* @param bucketName
* @param key
* @return
*/
private static String createMultipartUpload(S3Client s3, String bucketName, String key) {
CreateMultipartUploadRequest createMultipartUploadRequest = CreateMultipartUploadRequest.builder()
.bucket(bucketName)
.key(key)
.build();
String uploadId = null;
try {
CreateMultipartUploadResponse response = s3.createMultipartUpload(createMultipartUploadRequest);
uploadId = response.uploadId();
} catch (S3Exception e) {
System.err.println(e.awsErrorDetails().errorMessage());
System.exit(1);
}
return uploadId;
}
/**
* Creates copy parts based on source object size and copies over individual parts
*
* @param s3
* @param sourceBucket
* @param sourceKey
* @param destnBucket
* @param destnKey
* @param uploadId
* @return
* @throws IOException
*/
public static ListCompletedPart multipartUploadCopy(S3Client s3, String sourceBucket, String sourceKey, String destnBucket, String destnKey, String uploadId) throws IOException {
// Get the object size to track the end of the copy operation.
HeadObjectRequest headObjectRequest = HeadObjectRequest
.builder()
.bucket(sourceBucket)
.key(sourceKey)
.build();
HeadObjectResponse response = s3.headObject(headObjectRequest);
Long objectSize = response.contentLength();
System.out.println("Source Object size: " + objectSize);
// Copy the object using 20 MB parts.
long partSize = 20 * 1024 * 1024;
long bytePosition = 0;
int partNum = 1;
ListCompletedPart completedParts = new ArrayList<>();
while (bytePosition < objectSize) {
// The last part might be smaller than partSize, so check to make sure
// that lastByte isn't beyond the end of the object.
long lastByte = Math.min(bytePosition + partSize - 1, objectSize - 1);
System.out.println("part no: " + partNum + ", bytePosition: " + bytePosition + ", lastByte: " + lastByte);
// Copy this part.
UploadPartCopyRequest req = UploadPartCopyRequest.builder()
.uploadId(uploadId)
.sourceBucket(sourceBucket)
.sourceKey(sourceKey)
.destinationBucket(destnBucket)
.destinationKey(destnKey)
.copySourceRange("bytes="+bytePosition+"-"+lastByte)
.partNumber(partNum)
.build();
UploadPartCopyResponse res = s3.uploadPartCopy(req);
CompletedPart part = CompletedPart.builder()
.partNumber(partNum)
.eTag(res.copyPartResult().eTag())
.build();
completedParts.add(part);
partNum++;
bytePosition += partSize;
}
return completedParts;
}
public static void multipartCopyUploadTest(S3Client s3, String srcBucket, String srcKey, String destnBucket, String destnKey) {
System.out.println("Starting multipart copy for: " + srcKey);
try {
String uploadId = createMultipartUpload(s3, destnBucket, destnKey);
System.out.println(uploadId);
ListCompletedPart parts = multipartUploadCopy(s3, srcBucket, srcKey,destnBucket, destnKey, uploadId);
completeMultipartUpload(s3, destnBucket, destnKey, uploadId, parts);
System.out.println("Multipart copy completed for: " + srcKey);
} catch (Exception e) {
System.err.println(e.getMessage());
System.exit(1);
}
}
- SDK for Python
-
import logging
import boto3
from botocore.exceptions import ClientError
def head_object(s3_client, bucket_name, key_name):
'''
Returns metadata for an object in a directory bucket
:param s3_client: boto3 S3 client
:param bucket_name: Bucket that contains the object to query for metadata
:param key_name: Key name to query for metadata
:return: Metadata for the specified object if successful, else None
'''
try:
response = s3_client.head_object(
Bucket = bucket_name,
Key = key_name
)
return response
except ClientError as e:
logging.error(e)
return None
def create_multipart_upload(s3_client, bucket_name, key_name):
'''
Create a multipart upload to a directory bucket
:param s3_client: boto3 S3 client
:param bucket_name: Destination bucket for the multipart upload
:param key_name: Key name of the object to be uploaded
:return: UploadId for the multipart upload if created successfully, else None
'''
try:
mpu = s3_client.create_multipart_upload(Bucket = bucket_name, Key = key_name)
return mpu['UploadId']
except ClientError as e:
logging.error(e)
return None
def multipart_copy_upload(s3_client, source_bucket_name, key_name, target_bucket_name, mpu_id, part_size):
'''
Copy an object in a directory bucket to another bucket in multiple parts of a specified size
:param s3_client: boto3 S3 client
:param source_bucket_name: Bucket where the source object exists
:param key_name: Key name of the object to be copied
:param target_bucket_name: Destination bucket for copied object
:param mpu_id: The UploadId returned from the create_multipart_upload call
:param part_size: The size parts that the object will be broken into, in bytes.
Minimum 5 MiB, Maximum 5 GiB. There is no minimum size for the last part of your multipart upload.
:return: part_list for the multipart copy if all parts are copied successfully, else None
'''
part_list = []
copy_source = {
'Bucket': source_bucket_name,
'Key': key_name
}
try:
part_counter = 1
object_size = head_object(s3_client, source_bucket_name, key_name)
if object_size is not None:
object_size = object_size['ContentLength']
while (part_counter - 1) * part_size <object_size:
bytes_start = (part_counter - 1) * part_size
bytes_end = (part_counter * part_size) - 1
upload_copy_part = s3_client.upload_part_copy (
Bucket = target_bucket_name,
CopySource = copy_source,
CopySourceRange = f'bytes={bytes_start}-{bytes_end}',
Key = key_name,
PartNumber = part_counter,
UploadId = mpu_id
)
part_list.append({'PartNumber': part_counter, 'ETag': upload_copy_part['CopyPartResult']['ETag']})
part_counter += 1
except ClientError as e:
logging.error(e)
return None
return part_list
def complete_multipart_upload(s3_client, bucket_name, key_name, mpu_id, part_list):
'''
Completes a multipart upload to a directory bucket
:param s3_client: boto3 S3 client
:param bucket_name: Destination bucket for the multipart upload
:param key_name: Key name of the object to be uploaded
:param mpu_id: The UploadId returned from the create_multipart_upload call
:param part_list: List of uploaded part numbers with associated ETags
:return: True if the multipart upload was completed successfully, else False
'''
try:
s3_client.complete_multipart_upload(
Bucket = bucket_name,
Key = key_name,
UploadId = mpu_id,
MultipartUpload = {
'Parts': part_list
}
)
except ClientError as e:
logging.error(e)
return False
return True
if __name__ == '__main__':
MB = 1024 ** 2
region = 'us-west-2'
source_bucket_name = 'SOURCE_BUCKET_NAME'
target_bucket_name = 'TARGET_BUCKET_NAME'
key_name = 'KEY_NAME'
part_size = 10 * MB
s3_client = boto3.client('s3', region_name = region)
mpu_id = create_multipart_upload(s3_client, target_bucket_name, key_name)
if mpu_id is not None:
part_list = multipart_copy_upload(s3_client, source_bucket_name, key_name, target_bucket_name, mpu_id, part_size)
if part_list is not None:
if complete_multipart_upload(s3_client, target_bucket_name, key_name, mpu_id, part_list):
print (f'{key_name} successfully copied through multipart copy from {source_bucket_name} to {target_bucket_name}')
else:
print (f'Could not copy {key_name} through multipart copy from {source_bucket_name} to {target_bucket_name}')
列出進行中的多部分上傳
下列範例說明如何使用適用於 Java 2.x 的 SDK 和 Python SDK，列出進行中 (不完整) 的多部分上傳。
- SDK for Java 2.x
-
public static void listMultiPartUploads( S3Client s3, String bucketName) {
try {
ListMultipartUploadsRequest listMultipartUploadsRequest = ListMultipartUploadsRequest.builder()
.bucket(bucketName)
.build();
ListMultipartUploadsResponse response = s3.listMultipartUploads(listMultipartUploadsRequest);
List MultipartUpload uploads = response.uploads();
for (MultipartUpload upload: uploads) {
System.out.println("Upload in progress: Key = \"" + upload.key() + "\", id = " + upload.uploadId());
}
}
catch (S3Exception e) {
System.err.println(e.getMessage());
System.exit(1);
}
}
- SDK for Python
-
import logging
import boto3
from botocore.exceptions import ClientError
def list_multipart_uploads(s3_client, bucket_name):
'''
List any incomplete multipart uploads in a directory bucket in e specified gion
:param s3_client: boto3 S3 client
:param bucket_name: Bucket to check for incomplete multipart uploads
:return: List of incomplete multipart uploads if there are any, None if not
'''
try:
response = s3_client.list_multipart_uploads(Bucket = bucket_name)
if 'Uploads' in response.keys():
return response['Uploads']
else:
return None
except ClientError as e:
logging.error(e)
if __name__ == '__main__':
bucket_name = 'BUCKET_NAME'
region = 'us-west-2'
s3_client = boto3.client('s3', region_name = region)
multipart_uploads = list_multipart_uploads(s3_client, bucket_name)
if multipart_uploads is not None:
print (f'There are {len(multipart_uploads)} ncomplete multipart uploads for {bucket_name}')
else:
print (f'There are no incomplete multipart uploads for {bucket_name}')
列出分段上傳的部分
下列範例說明如何使用適用於 Java 2.x 的 SDK 和 Python SDK 來列出多部分上傳的各個部分。
- SDK for Java 2.x
-
public static void listMultiPartUploadsParts( S3Client s3, String bucketName, String objKey, String uploadID) {
try {
ListPartsRequest listPartsRequest = ListPartsRequest.builder()
.bucket(bucketName)
.uploadId(uploadID)
.key(objKey)
.build();
ListPartsResponse response = s3.listParts(listPartsRequest);
ListPart parts = response.parts();
for (Part part: parts) {
System.out.println("Upload in progress: Part number = \"" + part.partNumber() + "\", etag = " + part.eTag());
}
}
catch (S3Exception e) {
System.err.println(e.getMessage());
System.exit(1);
}
}
- SDK for Python
-
import logging
import boto3
from botocore.exceptions import ClientError
def list_parts(s3_client, bucket_name, key_name, upload_id):
'''
Lists the parts that have been uploaded for a specific multipart upload to a directory bucket.
:param s3_client: boto3 S3 client
:param bucket_name: Bucket that multipart uploads parts have been uploaded to
:param key_name: Name of the object that has parts uploaded
:param upload_id: Multipart upload ID that the parts are associated with
:return: List of parts associated with the specified multipart upload, None if there are no parts
'''
parts_list = []
next_part_marker = ''
continuation_flag = True
try:
while continuation_flag:
if next_part_marker == '':
response = s3_client.list_parts(
Bucket = bucket_name,
Key = key_name,
UploadId = upload_id
)
else:
response = s3_client.list_parts(
Bucket = bucket_name,
Key = key_name,
UploadId = upload_id,
NextPartMarker = next_part_marker
)
if 'Parts' in response:
for part in response['Parts']:
parts_list.append(part)
if response['IsTruncated']:
next_part_marker = response['NextPartNumberMarker']
else:
continuation_flag = False
else:
continuation_flag = False
return parts_list
except ClientError as e:
logging.error(e)
return None
if __name__ == '__main__':
region = 'us-west-2'
bucket_name = 'BUCKET_NAME'
key_name = 'KEY_NAME'
upload_id = 'UPLOAD_ID'
s3_client = boto3.client('s3', region_name = region)
parts_list = list_parts(s3_client, bucket_name, key_name, upload_id)
if parts_list is not None:
print (f'{key_name} has {len(parts_list)} parts uploaded to {bucket_name}')
else:
print (f'There are no multipart uploads with that upload ID for {bucket_name} bucket')
