Using Apache Iceberg tables
Athena supports read, time travel, write, and DDL queries for Apache Iceberg tables that use the Apache Parquet format for data and the AWS Glue catalog for their metastore.
Apache Iceberg
For more information about Apache Iceberg, see https://iceberg.apache.org/
Considerations and limitations
Athena support for Iceberg tables has the following limitations:
-
Tables with AWS Glue catalog only – Only Iceberg tables created against the AWS Glue catalog based on specifications defined by the open source glue catalog implementation
are supported from Athena. -
Table locking support by AWS Glue only – Unlike the open source Glue catalog implementation, which supports plug-in custom locking, Athena supports AWS Glue optimistic locking only. Using Athena to modify an Iceberg table with any other lock implementation will cause potential data loss and break transactions.
-
Supported file formats – Iceberg file format support in Athena depends on the Athena engine version, as shown in the following table.
Athena engine version Parquet ORC Avro 2 Yes No No 3 Yes Yes Yes -
Iceberg v2 tables – Athena only creates and operates on Iceberg v2 tables. For the difference between v1 and v2 tables, see Format version changes
in the Apache Iceberg documentation. -
Display of time types without time zone – The time and timestamp without time zone types are displayed in UTC. If the time zone is unspecified in a filter expression on a time column, UTC is used.
-
Timestamp related data precision – Although Iceberg supports microsecond precision for the timestamp data type, Athena supports only millisecond precision for timestamps in both reads and writes. For data in time related columns that is rewritten during manual compaction operations, Athena retains only millisecond precision.
-
Unsupported operations – The following Athena operations are not supported for Iceberg tables.
-
Views – Use
CREATE VIEW
to create Athena views as described in Working with views. If you are interested in using the Iceberg view specificationto create views, contact athena-feedback@amazon.com . -
TTF management commands not supported in AWS Lake Formation – Although you can use Lake Formation to manage read access permissions for TransactionTable Formats (TTFs) like Apache Iceberg, Apache Hudi, and Linux Foundation Delta Lake, you cannot use Lake Formation to manage permissions for operations like
VACUUM
,MERGE
,UPDATE
orOPTIMIZE
with these table formats. For more information about Lake Formation integration with Athena, see Using AWS Lake Formation with Amazon Athena in the AWS Lake Formation Developer Guide. -
Partitioning by nested fields – Partitioning by nested fields is not supported. Attempting to do so produces the message
NOT_SUPPORTED: Partitioning by nested field is unsupported:
column_name
.nested_field_name
. -
Skipping S3 Glacier objects not supported – If objects in the Apache Iceberg table are in an Amazon S3 Glacier storage class, setting the
read_restored_glacier_objects
table property tofalse
has no effect.For example, suppose you issue the following command:
ALTER TABLE
table_name
SET TBLPROPERTIES ('read_restored_glacier_objects' = 'false')For Iceberg and Delta Lake tables, the command produces the error
Unsupported table property key: read_restored_glacier_objects
. For Hudi tables, theALTER TABLE
command does not produce an error, but Amazon S3 Glacier objects are still not skipped. RunningSELECT
queries after theALTER TABLE
command continues to return all objects.
If you would like Athena to support a particular feature, send feedback to athena-feedback@amazon.com
Topics
- Creating Iceberg tables
- Managing Iceberg tables
- Querying Iceberg table metadata
- Evolving Iceberg table schema
- Querying Iceberg table data and performing time travel
- Updating Iceberg table data
- Optimizing Iceberg tables
- Supported data types for Iceberg tables in Athena
- Other Athena operations on Iceberg tables
- Additional resources