Why did my import fail? - Amazon HealthLake

Why did my import fail?

A successful import job will generate a folder with output inputFileName.ndjson files, however individual records can fail to import. When this happens, a second FAILURE folder will be generated with a manifest of records that failed to be imported. The job output location to access the manifest file is JobProperties.JobOutputDataConfig.S3Configuration.S3Uri.

This manifest file contains details about the job output such as location of all successful responses (successOutput.successOutputS3Uri), the location of all failed responses (failureOutput.failureOutputS3Uri) and additional job metrics. The contents of manifest file are parsable programatically.

To analyze why an import job failed use the DescribeFHIRImportJob API to analyze the JobProperties. The following is recommended:

  • If the status is FAILED and a message is present, the failures are related to job parameters such as input data size or number of input files being beyond HealthLake quotas.

  • If the import job status is FAILED and a message is not present, go to the job output location to access the manifest file, Manifest.json.

For each input file, there is failure output file with input file name for any resource that fails to import.The responses contain line number (lineId) corresponding to the location of input data, FHIR response object (UpdateResourceResponse) and status code (statusCode) of the response.

A sample output file would look like this:

{"lineId":3, UpdateResourceResponse:{"jsonBlob":{"resourceType":"OperationOutcome","issue":[{"severity":"error","code":"processing","diagnostics":"1 validation error detected: Value 'Patient123' at 'resourceType' failed to satisfy constraint: Member must satisfy regular expression pattern: [A-Za-z]{1,256}"}]}, "statusCode":400} {"lineId":5, UpdateResourceResponse:{"jsonBlob":{"resourceType":"OperationOutcome","issue":[{"severity":"error","code":"processing","diagnostics":"This property must be an simple value, not a com.google.gson.JsonArray","location":["/EffectEvidenceSynthesis/name"]},{"severity":"error","code":"processing","diagnostics":"Unrecognised property '@telecom'","location":["/EffectEvidenceSynthesis"]},{"severity":"error","code":"processing","diagnostics":"Unrecognised property '@gender'","location":["/EffectEvidenceSynthesis"]},{"severity":"error","code":"processing","diagnostics":"Unrecognised property '@birthDate'","location":["/EffectEvidenceSynthesis"]},{"severity":"error","code":"processing","diagnostics":"Unrecognised property '@address'","location":["/EffectEvidenceSynthesis"]},{"severity":"error","code":"processing","diagnostics":"Unrecognised property '@maritalStatus'","location":["/EffectEvidenceSynthesis"]},{"severity":"error","code":"processing","diagnostics":"Unrecognised property '@multipleBirthBoolean'","location":["/EffectEvidenceSynthesis"]},{"severity":"error","code":"processing","diagnostics":"Unrecognised property '@communication'","location":["/EffectEvidenceSynthesis"]},{"severity":"warning","code":"processing","diagnostics":"Name should be usable as an identifier for the module by machine processing applications such as code generation [name.matches('[A-Z]([A-Za-z0-9_]){0,254}')]","location":["EffectEvidenceSynthesis"]},{"severity":"error","code":"processing","diagnostics":"Profile http://hl7.org/fhir/StructureDefinition/EffectEvidenceSynthesis, Element 'EffectEvidenceSynthesis.status': minimum required = 1, but only found 0","location":["EffectEvidenceSynthesis"]},{"severity":"error","code":"processing","diagnostics":"Profile http://hl7.org/fhir/StructureDefinition/EffectEvidenceSynthesis, Element 'EffectEvidenceSynthesis.population': minimum required = 1, but only found 0","location":["EffectEvidenceSynthesis"]},{"severity":"error","code":"processing","diagnostics":"Profile http://hl7.org/fhir/StructureDefinition/EffectEvidenceSynthesis, Element 'EffectEvidenceSynthesis.exposure': minimum required = 1, but only found 0","location":["EffectEvidenceSynthesis"]},{"severity":"error","code":"processing","diagnostics":"Profile http://hl7.org/fhir/StructureDefinition/EffectEvidenceSynthesis, Element 'EffectEvidenceSynthesis.exposureAlternative': minimum required = 1, but only found 0","location":["EffectEvidenceSynthesis"]},{"severity":"error","code":"processing","diagnostics":"Profile http://hl7.org/fhir/StructureDefinition/EffectEvidenceSynthesis, Element 'EffectEvidenceSynthesis.outcome': minimum required = 1, but only found 0","location":["EffectEvidenceSynthesis"]},{"severity":"information","code":"processing","diagnostics":"Unknown extension http://synthetichealth.github.io/synthea/disability-adjusted-life-years","location":["EffectEvidenceSynthesis.extension[3]"]},{"severity":"information","code":"processing","diagnostics":"Unknown extension http://synthetichealth.github.io/synthea/quality-adjusted-life-years","location":["EffectEvidenceSynthesis.extension[4]"]}]}, "statusCode":400} {"lineId":7, UpdateResourceResponse:{"jsonBlob":{"resourceType":"OperationOutcome","issue":[{"severity":"error","code":"processing","diagnostics":"2 validation errors detected: Value at 'resourceId' failed to satisfy constraint: Member must satisfy regular expression pattern: [A-Za-z0-9-.]{1,64}; Value at 'resourceId' failed to satisfy constraint: Member must have length greater than or equal to 1"}]}, "statusCode":400} {"lineId":9, UpdateResourceResponse:{"jsonBlob":{"resourceType":"OperationOutcome","issue":[{"severity":"error","code":"processing","diagnostics":"Missing required id field in resource json"}]}, "statusCode":400} {"lineId":15, UpdateResourceResponse:{"jsonBlob":{"resourceType":"OperationOutcome","issue":[{"severity":"error","code":"processing","diagnostics":"Invalid JSON found in input file"}]}, "statusCode":400}

The example shows that there were failures on line 3, 4, 7, 9, 15 from the corresponding input lines from input file. For each of those lines, the explanations are as follows:

  • On Line 3, the response explains that resourceType provided in line 3 of input file is invalid.

  • On Line 5, the response explains that there is a FHIR validation error in line 5 of input file.

  • On Line 7, the response explains that there is a validation issue with resourceId provided as input.

  • On Line 9, the response explains that input file must contain a valid resource id.

  • On line 15, the response of input file is that the file is not in a valid JSON format.