Troubleshooting data verification issues

By default, AWS DataSync verifies the integrity of your data at the end of a transfer. Use the following information to help you diagnose common verification errors and warnings, such as files being modified or deleted before DataSync finishes verifying your data.

With verification issues, many times it helps to review your CloudWatch logs (or task reports) in addition to the task execution error that you're seeing. DataSync provides JSON-structured logs for Enhanced mode tasks, while Basic mode tasks have unstructured logs.

There are mismatches between a file's contents

When your task execution finishes, you see the following error:

Transfer and verification completed. Verification detected mismatches. Files with mismatches are listed in Cloud Watch Logs

In your CloudWatch logs, you might notice failed verifications for contents that differ between the source and destination locations. This can happen if files are modified during your transfer.

For example, the following logs shows that file1.txt has different mtime, srcHash, and dstHash values:

Basic mode log example

[NOTICE] Verification failed <> /directory1/directory2/file1.txt
[NOTICE] /directory1/directory2/file1.txt   srcMeta: type=R mode=0755 uid=65534 gid=65534 size=534528 atime=1633100003/684349800 mtime=1602647222/222919600 extAttrsHash=0
[NOTICE]   srcHash: 0c506c26bd1e43bd3ac346734f1a9c16c4ad100d1b43c2903772ca894fd24e44
[NOTICE] /directory1/directory2/file1.txt   dstMeta: type=R mode=0755 uid=65534 gid=65534 size=511001 atime=1633100003/684349800 mtime=1633106855/859227500 extAttrsHash=0
[NOTICE]   dstHash: dbd798929f11a7c0201e97f7a61191a83b4e010a449dfc79fbb8233801067c46

In DataSync, mtime represents the last time a file was written to before preparation. When verifying transfers, DataSync compares mtime values between source and destination locations. A verification failure like this occurs if the mtime for a file isn't the same for both locations. The differences between srcHash and dstHash indicate the file's contents don't match at both locations.

Actions to take

Do the following:

1. Use an epoch time converter to determine whether the source or destination file or object was modified more recently. This can help identify which version is current.

2. To avoid this error again, schedule your task to run during a maintenance window when there's no activity at your source and destination.

There's a mismatch between a file's SMB metadata

When your task execution finishes, you see the following error:

Transfer and verification completed. Verification detected mismatches. Files with mismatches are listed in Cloud Watch Logs

When transferring between storage systems that support the Server Message Block (SMB) protocol, you might see this error when a file's extended SMB attributes don't match between source and destination.

For example, the following logs show that file1.txt has a different extAttrsHash value between locations, indicating the file contents are identical but extended attributes weren't set at the destination:

Basic mode log example

[NOTICE] Verification failed <> /directory1/directory2/file1.txt
[NOTICE] /directory1/directory2/file1.txt   srcMeta: type=R mode=0755 uid=65534 gid=65534 size=1469752 atime=1631354985/174924200 mtime=1536995541/986211400 extAttrsHash=2272191894
[NOTICE]   srcHash: 38571d42b646ac8f4034b7518636b37dd0899c6fc03cdaa8369be6e81a1a2bb5
[NOTICE] /directory1/directory2/file1.txt   dstMeta: type=R mode=0755 uid=65534 gid=65534 size=1469752 atime=1631354985/174924200 mtime=1536995541/986211400 extAttrsHash=3051150340
[NOTICE]   dstHash: 38571d42b646ac8f4034b7518636b37dd0899c6fc03cdaa8369be6e81a1a2bb5

You might also see a related error message about extended attributes:

[ERROR] Deferred error: WriteFileExtAttr2 failed to setextattrlist(filename="/directory1/directory2/file1.txt"): Input/output error

Action to take

This error typically occurs when there are insufficient permissions to copy access control lists (ACLs) to the destination. To resolve this issue, review the following configuration guides based on your destination type:

• Required permissions with FSx for Windows File Server file systems

• Required permissions with FSx for ONTAP file systems that use SMB

Files to transfer are no longer at source location

When your task execution finishes, you see the following error:

Transfer and verification completed. Selected files transferred except for files skipped due to errors. If no skipped files are listed in Cloud Watch Logs, please contact AWS Support for further assistance.

In your logs, you might see errors indicating that files aren't at the source location. This can happen if files (such as file1.dll and file2.dll) are deleted after preparation but before DataSync transfers them:

Basic mode log example

[ERROR] Failed to open source file /file1.dll: No such file or directory
[ERROR] Failed to open source file /file2.dll: No such file or directory

Action to take

To avoid these situations, schedule your task to run when there's no activity at the source location.

For example, you can run your task during a maintenance window when users and applications aren't actively working with that location.

In some cases, you might not see logs associated with this error. If that happens, contact AWS Support Center.

DataSync can't verify destination data

When your task execution finishes, you see the following error:

Transfer and verification completed. Verification detected mismatches. Files with mismatches are listed in Cloud Watch Logs

In your logs, you might notice that DataSync can't verify certain folders or files in the destination location. These errors can look like this:

Basic mode log example

[ERROR] Failed to read metadata for destination file /directory1/directory2/file1.txt: No such file or directory

For files, you might see verification failures like this:

Basic mode log example

[NOTICE] Verification failed <> /directory1/directory2/file1.txt
[NOTICE] /directory1/directory2/file1.txt   srcMeta: type=R mode=0755 uid=65534 gid=65534 size=61533 atime=1633099987/747713800 mtime=1536995631/894267700 extAttrsHash=232104771
[NOTICE]   srcHash: 1426fe40f669a7d36cca1b5329983df31a9aeff8eb9fe3ac885f26de2f8fff6b
[NOTICE] /directory1/directory2/file1.txt   dstMeta: type=R mode=0755 uid=65534 gid=65534 size=0 atime=0/0 mtime=0/0 extAttrsHash=0
[NOTICE]   dstHash: 0000000000000000000000000000000000000000000000000000000000000000

Action to take

These logs indicate that destination data was deleted after the transfer but before verification. (Logs look similar when data is uploaded to a source location during the same time frame.)

To avoid these situations, schedule your task to run when there's no activity at the destination location.

For example, you can run your task during a maintenance window when users and applications aren't actively working with that location.

DataSync can't read object metadata

When your task execution finishes, you see the following error:

Transfer and verification completed. Selected files transferred except for files skipped due to errors. If no skipped files are listed in Cloud Watch Logs, please contact AWS Support for further assistance.

In your logs, you might notice that DataSync can't read file1.png because of a failed Amazon S3 HeadObject request. DataSync makes HeadObject requests with S3 locations during task preparation and verification.

Basic mode log example

[WARN] Failed to read metadata for file /file1.png: S3 Head Object Failed

Actions to take

To fix this issue, verify whether DataSync has the right level of permissions to work with your S3 bucket:

• Make sure that the IAM role that DataSync uses to access your Amazon S3 locations allows the s3:GetObject permission. For more information, see Required permissions.

• If your S3 bucket uses server-side encryption, make sure that DataSync is allowed to access the objects in that bucket. For more information, see Accessing S3 buckets using server-side encryption.

There's a mismatch in an object's system-defined metadata

When your Enhanced mode task execution between S3 buckets finishes, you see the following error:

Verification failed due to a difference in metadata

You might notice in your logs a mismatch in an object’s Amazon S3 system-defined metadata. In this particular example, the source object doesn't have Content-Type metadata but the destination object does. This happened because the destination S3 bucket automatically applied "ContentType": "application/octet-stream" metadata to the object when DataSync transferred it there.

Enhanced mode log example

{
    "Action": "VERIFY",
    "Source": {
        "LocationId": "loc-0b3017fc4ba4a2d8d",
        "RelativePath": "encoding/content-null",
        "Metadata": {
            "Type": "Object",
            "ContentSize": 24,
            "LastModified": "2024-12-23T15:48:15Z",
            "S3": {
                "SystemMetadata": {
                    "ETag": "\"68b9c323bb846841ee491481f576ed4a\""
                },
                "UserMetadata": {},
                "Tags": {}
            }
        }
    },
    "Destination": {
        "LocationId": "loc-abcdef01234567890",
        "RelativePath": "encoding/content-null",
        "Metadata": {
            "Type": "Object",
            "ContentSize": 24,
            "LastModified": "2024-12-23T16:00:03Z",
            "S3": {
                "SystemMetadata": {
                    "ContentType": "application/octet-stream",
                    "ETag": "\"68b9c323bb846841ee491481f576ed4a\""
                },
                "UserMetadata": {
                    "file-mtime": "1734968895000"
                },
                "Tags": {}
            }
        }
    },
    "TransferType": "CONTENT_AND_METADATA",
    "ErrorCode": "MetadataDiffers",
    "ErrorDetail": "Verification failed due to a difference in metadata"
}

Action to take

To avoid this error, update your source location objects to include the Content-Type metadata property.

Understanding data verification duration

DataSync verification includes an SHA256 checksum on file content and an exact comparison of file metadata between locations. How long verification takes depends on several factors, including the number of files or objects involved, the size of the data in the storage systems, and the performance of these systems.

Action to take

Given the factors that can affect verification time, you shouldn't have to do anything. However, if your task execution seems stuck with a verifying status, contact AWS Support Center.