AWS Documentation » Amazon Personalize » Developer Guide » Preparing and Importing Data » Datasets and Schemas

Datasets and Schemas

Amazon Personalize recognizes three types of historical datasets. Each type has an associated schema with a name key whose value matches the dataset type. The three types are:

• Users – This dataset is intended to provide metadata about your users. This includes information such as age, gender, and loyalty membership, among others, which can be important signals in personalization systems.

• Items – This dataset is intended to provide metadata about your items. This includes information such as price, SKU type, and availability, among others.

• Interactions – This dataset is intended to provide historical interaction data between users and items.

The Users and Items dataset types are known as metadata types and are only used by certain recipes. For more information, see Using Predefined Recipes. For metadata datasets, all strings, except for USER_ID and ITEM_ID, must be marked as categorical in the schema, as shown in the following examples.

Note

A dataset group can contain only one of each kind of dataset.

Each dataset has a set of required fields and reserved keywords, as shown in the following table.

Dataset Type	Required Fields	Reserved Keywords
Users	USER_ID 1 metadata field
Items	ITEM_ID 1 metadata field
Interactions	USER_ID ITEM_ID TIMESTAMP	EVENT_TYPE EVENT_VALUE

Before you add a dataset to Amazon Personalize, you're required to define a schema for the dataset. Schemas in Amazon Personalize are defined in the Avro format. For more information, see Apache Avro. The schema fields can be in any order but must match the order of the corresponding column headers in the imported data file.

Note

The Users and Items schemas must have at least one metadata field. There is a limit of 5 metadata fields per dataset/schema. A metadata field is a field of type "string" with a "categorical" attribute, or any non-string type. The required fields and reserved keywords are not considered metadata.

The following example shows an Interactions schema. The EVENT_TYPE and EVENT_VALUE fields are optional, and are reserved keywords recognized by Amazon Personalize.

{
  "type": "record",
  "name": "Interactions",
  "namespace": "com.amazonaws.personalize.schema",
  "fields": [
      {
          "name": "USER_ID",
          "type": "string"
      },
      {
          "name": "ITEM_ID",
          "type": "string"
      },
      {
          "name": "EVENT_TYPE",
          "type": "string"
      },
      {
          "name": "EVENT_VALUE",
          "type": "string"
      },
      {
          "name": "TIMESTAMP",
          "type": "long"
      }
  ],
  "version": "1.0"
}

The following example shows a Users schema in Avro format. Only the USER_ID field is required. The AGE and GENDER fields are metadata.

{
  "type": "record",
  "name": "Users",
  "namespace": "com.amazonaws.personalize.schema",
  "fields": [
      {
          "name": "USER_ID",
          "type": "string"
      },
      {
          "name": "AGE",
          "type": "int"
      },
      {
          "name": "GENDER",
          "type": "string",
          "categorical": true
      }
  ],
  "version": "1.0"
}

The following example shows an Items schema. Only the ITEM_ID field is required.

{
  "type": "record",
  "name": "Items",
  "namespace": "com.amazonaws.personalize.schema",
  "fields": [
      {
          "name": "ITEM_ID",
          "type": "string"
      },
      {
          "name": "GENRE",
          "type": "string",
          "categorical": true
      }
  ],
  "version": "1.0"
}

If you are using the AWS console, you create a new schema when you create a dataset for your input data. You can also choose an existing schema.

If you are using the AWS CLI, see Step 1: Import Training Data for an example.

Create a Schema Using the AWS Python SDK

1. Define the Avro format schema that you want to use.

2. Save the schema in a JSON file in the default Python folder.

3. Create the schema using the following code.

   import boto3
   
   personalize = boto3.client('personalize')
   
   with open('schema.json') as f:
       createSchemaResponse = personalize.create_schema(
           name = 'YourSchema',
           schema = f.read()
       )
   
   schema_arn = createSchemaResponse['schemaArn']
   
   print('Schema ARN:' + schema_arn )

4. Amazon Personalize returns the ARN of the new schema. Store it for later use.

Amazon Personalize provides operations for managing schemas. For example, you can use the ListSchemas API to get a list of the schemas that you have created.

After you have created a schema, use it with datasets that match the schema. For more information, see Formating Your Input Data.