Understanding import jobs

After creating a data store in AWS HealthImaging, you must import your medical imaging data from your Amazon S3 input bucket into your data store to create image sets. You can use the AWS Management Console, AWS CLI, and AWS SDKs to start, describe, and list import jobs.

The following diagram provides an overview of how HealthImaging imports DICOM data into a data store and transforms it into image sets. Import job processing results are stored in the Amazon S3 output bucket (outputS3Uri) and image sets are stored in the AWS HealthImaging data store.

Keep the following points in mind when importing your medical imaging files from Amazon S3 into an AWS HealthImaging data store:

• Specific SOP classes and transfer syntaxes are supported for import jobs. For more information, see DICOM support.

• Length constraints apply to specific DICOM elements during import. To ensure a successful import job, verify that your medical imaging data does not exceed the length constraints. For more information, see DICOM element constraints.

• A pixel data verification check is performed at the beginning of import jobs. For more information, see Pixel data verification.

• There are endpoints, quotas, and throttling limits associated with HealthImaging import actions. For more information, see Endpoints and quotas and Throttling limits.

• For each import job, processing results are stored at the outputS3Uri location. The processing results are organized as a job-output-manifest.json file and SUCCESS and FAILURE folders.

  Note

  You can include up to 10,000 nested folders for a single import job.

  • The job-output-manifest.json file contains jobSummary output and additional details about the processed data. The following example shows output from a job-output-manifest.json file.

    {
        "jobSummary": {
            "jobId": "09876543210987654321098765432109",
            "datastoreId": "12345678901234567890123456789012",
            "inputS3Uri": "s3://medical-imaging-dicom-input/dicom_input/",
            "outputS3Uri": "s3://medical-imaging-output/job_output/12345678901234567890123456789012-DicomImport-09876543210987654321098765432109/",
            "successOutputS3Uri": "s3://medical-imaging-output/job_output/12345678901234567890123456789012-DicomImport-09876543210987654321098765432109/SUCCESS/",
            "failureOutputS3Uri": "s3://medical-imaging-output/job_output/12345678901234567890123456789012-DicomImport-09876543210987654321098765432109/FAILURE/",
            "numberOfScannedFiles": 5,
            "numberOfImportedFiles": 3,
            "numberOfFilesWithCustomerError": 2,
            "numberOfFilesWithServerError": 0,
            "numberOfGeneratedImageSets": 2,
            "imageSetsSummary": [{
                    "imageSetId": "12345612345612345678907890789012",
                    "numberOfMatchedSOPInstances": 2
                },
                {
                    "imageSetId": "12345612345612345678917891789012",
                    "numberOfMatchedSOPInstances": 1
                }
            ]
        }
    }
    

  • The SUCCESS folder holds the success.ndjson file containing results of all imaging files that imported successfully. The following example shows output from a success.ndjson file.

    {"inputFile":"dicomInputFolder/1.3.51.5145.5142.20010109.1105620.1.0.1.dcm","importResponse":{"imageSetId":"12345612345612345678907890789012"}}
    {"inputFile":"dicomInputFolder/1.3.51.5145.5142.20010109.1105630.1.0.1.dcm","importResponse":{"imageSetId":"12345612345612345678917891789012"}}
    

  • The FAILURE folder holds the failure.ndjson file containing results of all imaging files that did not import successfully. The following example shows output from a failure.ndjson file.

    {"inputFile":"dicom_input/invalidDicomFile1.dcm","exception":{"exceptionType":"ValidationException","message":"DICOM attribute TransferSyntaxUID does not exist"}}
    {"inputFile":"dicom_input/invalidDicomFile2.dcm","exception":{"exceptionType":"ValidationException","message":"DICOM attributes does not exist"}}
    

• Import jobs are retained in the list of jobs for 90 days and then archived.