AWS Athena Query Caching

Dyfan Jones


noctua now supports caching. This was originally inspired by pyathenato reduce the cost of using AWS Athena. noctua however has a different caching method and utilities local caching in R environments instead of using AWS list_query_executions. This is down to dbClearResult clearing AWS S3’s AWS Athena output when caching is disabled.

Caching benefits

By caching queries the performance of repeat queries is significantly improved. This is because the query is no longer sent to AWS Athena. Instead the query ID, of the repeating query, is taken from the R environment and the result is returned from AWS S3.

Here we can see a performance increase of x10 with repeat query execution.

Caching weakness

The weakness in caching occurs when the underlying data is updated. The cache will still only retrieve the previous query ID. This means that the new updated data won’t be return when the caching is enabled:

Sadly the cached query didn’t pick up the new data from iris.

Cache memory

The caching method in noctua will remember previous query ids within each R session, even if you stop and start caching in noctua_options.

We can see that res1 and res3 utilise the same QueryID, even tho caching was stopped and started.

Clear down cache

To clear down the cache, just set the parameter: clear_cache within noctua_options to TRUE