Step 4: Load data into HDFS

In this step, you will copy a data file into Hadoop Distributed File System (HDFS), and then create an external Hive table that maps to the data file.

Download the sample data

1. Download the sample data archive (features.zip):

   wget https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/samples/features.zip

2. Extract the features.txt file from the archive:

   unzip features.zip

3. View the first few lines of the features.txt file:

   head features.txt

   The result should look similar to this:

   1535908|Big Run|Stream|WV|38.6370428|-80.8595469|794
   875609|Constable Hook|Cape|NJ|40.657881|-74.0990309|7
   1217998|Gooseberry Island|Island|RI|41.4534361|-71.3253284|10
   26603|Boone Moore Spring|Spring|AZ|34.0895692|-111.410065|3681
   1506738|Missouri Flat|Flat|WA|46.7634987|-117.0346113|2605
   1181348|Minnow Run|Stream|PA|40.0820178|-79.3800349|1558
   1288759|Hunting Creek|Stream|TN|36.343969|-83.8029682|1024
   533060|Big Charles Bayou|Bay|LA|29.6046517|-91.9828654|0
   829689|Greenwood Creek|Stream|NE|41.596086|-103.0499296|3671
   541692|Button Willow Island|Island|LA|31.9579389|-93.0648847|98

   The features.txt file contains a subset of data from the United States Board on Geographic Names (http://geonames.usgs.gov/domestic/download_data.htm). The fields in each line represent the following:

   • Feature ID (unique identifier)

   • Name

   • Class (lake; forest; stream; and so on)

   • State

   • Latitude (degrees)

   • Longitude (degrees)

   • Height (in feet)

4. At the command prompt, enter the following command:

   hive

   The command prompt changes to this: hive>

5. Enter the following HiveQL statement to create a native Hive table:

   CREATE TABLE hive_features
       (feature_id             BIGINT,
       feature_name            STRING ,
       feature_class           STRING ,
       state_alpha             STRING,
       prim_lat_dec            DOUBLE ,
       prim_long_dec           DOUBLE ,
       elev_in_ft              BIGINT)
       ROW FORMAT DELIMITED
       FIELDS TERMINATED BY '|'
       LINES TERMINATED BY '\n';

6. Enter the following HiveQL statement to load the table with data:

   LOAD DATA
   LOCAL
   INPATH './features.txt'
   OVERWRITE
   INTO TABLE hive_features;

7. You now have a native Hive table populated with data from the features.txt file. To verify, enter the following HiveQL statement:

   SELECT state_alpha, COUNT(*)
   FROM hive_features
   GROUP BY state_alpha;

   The output should show a list of states and the number of geographic features in each.

Next step

Step 5: Copy data to DynamoDB