R Tools for Eurostat Open Data

This rOpenGov R package provides tools to access Eurostat database, which you can also browse on-line for the data sets and documentation. For contact information and source code, see the package website.

Installation

Release version (CRAN):

install.packages("eurostat")

Development version (Github):

library(devtools)
install_github("ropengov/eurostat")

Overall, the eurostat package includes the following functions:

clean_eurostat_cache    Clean Eurostat Cache
cut_to_classes          Cuts the Values Column into Classes and
                        Polishes the Labels
dic_order               Order of Variable Levels from Eurostat
                        Dictionary.
eu_countries            Countries and Country Codes
eurostat-package        R Tools for Eurostat open data
eurostat_geodata_60_2016
                        Geospatial data of Europe from Gisco in 1:60
                        million scale from year 2016
eurotime2date           Date Conversion from Eurostat Time Format
eurotime2num            Conversion of Eurostat Time Format to Numeric
get_bibentry            Create A Data Bibliography
get_eurostat            Read Eurostat Data
get_eurostat_dic        Download Eurostat Dictionary
get_eurostat_geospatial
                        Download Geospatial Data from GISGO
get_eurostat_json       Get Data from Eurostat API in JSON
get_eurostat_raw        Download Data from Eurostat Database
get_eurostat_toc        Download Table of Contents of Eurostat Data
                        Sets
harmonize_country_code
                        Harmonize Country Code
label_eurostat          Get Eurostat Codes
search_eurostat         Grep Datasets Titles from Eurostat
tgs00026                Auxiliary Data

Finding data

Function get_eurostat_toc() downloads a table of contents of eurostat datasets. The values in column ‘code’ should be used to download a selected dataset.

# Load the package
library(eurostat)
library(rvest)

# Get Eurostat data listing
toc <- get_eurostat_toc()

# Check the first items
library(knitr)
kable(head(toc))

With search_eurostat() you can search the table of contents for particular patterns, e.g. all datasets related to passenger transport. The kable function to produces nice markdown output. Note that with the type argument of this function you could restrict the search to for instance datasets or tables.

# info about passengers
kable(head(search_eurostat("passenger transport")))

Codes for the dataset can be searched also from the Eurostat database. The Eurostat database gives codes in the Data Navigation Tree after every dataset in parenthesis.

Downloading data

The package supports two of the Eurostats download methods: the bulk download facility and the Web Services’ JSON API. The bulk download facility is the fastest method to download whole datasets. It is also often the only way as the JSON API has limitation of maximum 50 sub-indicators at a time and whole datasets usually exceeds that. To download only a small section of the dataset the JSON API is faster, as it allows to make a data selection before downloading.

A user does not usually have to bother with methods, as both are used via main function get_eurostat(). If only the table id is given, the whole table is downloaded from the bulk download facility. If also filters are defined the JSON API is used.

Here an example of indicator ‘Modal split of passenger transport’. This is the percentage share of each mode of transport in total inland transport, expressed in passenger-kilometres (pkm) based on transport by passenger cars, buses and coaches, and trains. All data should be based on movements on national territory, regardless of the nationality of the vehicle. However, the data collection is not harmonized at the EU level.

Pick and print the id of the data set to download:

# For the original data, see
# http://ec.europa.eu/eurostat/tgm/table.do?tab=table&init=1&plugin=1&language=en&pcode=tsdtr210
id <- search_eurostat("Modal split of passenger transport", 
                         type = "table")$code[1]
print(id)

[1] “t2020_rk310”

Get the whole corresponding table. As the table is annual data, it is more convient to use a numeric time variable than use the default date format:

dat <- get_eurostat(id, time_format = "num")

Investigate the structure of the downloaded data set:

str(dat)

## Classes 'tbl_df', 'tbl' and 'data.frame':    2587 obs. of  5 variables:
##  $ unit   : Factor w/ 1 level "PC": 1 1 1 1 1 1 1 1 1 1 ...
##  $ vehicle: Factor w/ 3 levels "BUS_TOT","CAR",..: 1 1 1 1 1 1 1 1 1 1 ...
##  $ geo    : Factor w/ 34 levels "AT","BE","CH",..: 1 2 3 4 5 6 7 8 9 10 ...
##  $ time   : num  1990 1990 1990 1990 1990 1990 1990 1990 1990 1990 ...
##  $ values : num  11 10.6 3.7 9.1 11.3 32.4 14.9 13.5 6 24.8 ...

kable(head(dat))

Or you can get only a part of the dataset by defining filters argument. It should be named list, where names corresponds to variable names (lower case) and values are vectors of codes corresponding desidered series (upper case). For time variable, in addition to a time, also a sinceTimePeriod and a lastTimePeriod can be used.

dat2 <- get_eurostat(id, filters = list(geo = c("EU28", "FI"), lastTimePeriod=1), time_format = "num")
kable(dat2)

datl2 <- get_eurostat(id, filters = list(geo = c("EU28", "FI"), 
                                         lastTimePeriod = 1), 
                      type = "label", time_format = "num")
kable(head(datl2))

datl <- label_eurostat(dat)
kable(head(datl))

label_eurostat_vars(names(datl))

levels(datl$vehicle)

Selecting and modifying data

data(efta_countries)
kable(efta_countries)

dat_eu12 <- subset(datl, geo == "European Union (current composition)" & time == 2012)
kable(dat_eu12, row.names = FALSE)

library("tidyr")
dat_eu_0012 <- subset(dat, geo == "EU28" & time %in% 2000:2012)
dat_eu_0012_wide <- spread(dat_eu_0012, vehicle, values)
kable(subset(dat_eu_0012_wide, select = -geo), row.names = FALSE)

dat_trains <- subset(datl, geo %in% c("Austria", "Belgium", "Finland", "Sweden")
                     & time %in% 2000:2012 
                     & vehicle == "Trains")

dat_trains_wide <- spread(dat_trains, geo, values) 
kable(subset(dat_trains_wide, select = -vehicle), row.names = FALSE)

Visualization

Visualizing train passenger data with ggplot2:

library(ggplot2)
p <- ggplot(dat_trains, aes(x = time, y = values, colour = geo)) 
p <- p + geom_line()
print(p)

Triangle plot

Triangle plot is handy for visualizing data sets with three variables.

library(tidyr)
library(plotrix)
library(eurostat)
library(dplyr)
library(tidyr)

# Some cleaning of the data is required
 energy3 <- get_eurostat("nrg_114a") %>%
 label_eurostat(dat) %>%
 filter(time == "2013-01-01") %>%
 mutate(geo = gsub(geo, pattern=" \\(.*", replacement="")) %>%
 select(product, geo, values) %>%
 group_by(product, geo) %>%
 summarise(svalue = sum(values)) %>%
 group_by(geo) %>%
 mutate(tvalue = sum(svalue),
 svalue = svalue/sum(svalue)) %>%
 filter(tvalue > 1000) %>% # only large countries
 spread(product, svalue)
 
# Triangle plot
 par(cex=0.75, mar=c(0,0,0,0))
 positions <- plotrix::triax.plot(as.matrix(energy3[, c(3,5,4)]),
                     show.grid = TRUE,
                     label.points= FALSE, point.labels = energy3$geo,
                     col.axis="gray50", col.grid="gray90",
                     pch = 19, cex.axis=0.8, cex.ticks=0.7, col="grey50")

 # Larger labels
 ind <- which(energy3$geo %in%  c("Norway", "Iceland","Denmark","Estonia", "Turkey", "Italy", "Finland"))
 df <- data.frame(positions$xypos, geo = energy3$geo)
 points(df$x[ind], df$y[ind], cex=2, col="red", pch=19)
 text(df$x[ind], df$y[ind], df$geo[ind], adj = c(0.5,-1), cex=1.5)

Further examples

For further examples, see the package homepage.

Citations and related work

citation("eurostat")

## 
## Kindly cite the eurostat R package as follows:
## 
##   (C) Leo Lahti, Janne Huovari, Markus Kainu, Przemyslaw Biecek.
##   Retrieval and analysis of Eurostat open data with the eurostat
##   package. R Journal 9(1):385-392, 2017. Version 3.3.55 Package
##   URL: http://ropengov.github.io/eurostat Manuscript URL:
##   https://journal.r-project.org/archive/2017/RJ-2017-019/index.html
## 
## A BibTeX entry for LaTeX users is
## 
##   @Misc{,
##     title = {eurostat R package},
##     author = {Leo Lahti and Janne Huovari and Markus Kainu and Przemyslaw Biecek},
##     journal = {R Journal},
##     volume = {9},
##     number = {1},
##     pages = {385-392},
##     year = {2017},
##     url = {https://journal.r-project.org/archive/2017/RJ-2017-019/index.html},
##     note = {Version 3.3.55},
##   }

Version info

This tutorial was created with

sessionInfo()

## R version 3.5.2 (2018-12-20)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 19.04
## 
## Matrix products: default
## BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.8.0
## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.8.0
## 
## locale:
##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
##  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
##  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
##  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
##  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] dplyr_0.8.3     plotrix_3.7-6   ggplot2_3.2.1   tidyr_1.0.0    
## [5] rvest_0.3.4     xml2_1.2.2      eurostat_3.3.55 knitr_1.25     
## 
## loaded via a namespace (and not attached):
##  [1] tidyselect_0.2.5   xfun_0.9           purrr_0.3.2       
##  [4] sf_0.8-0           lattice_0.20-38    colorspace_1.4-1  
##  [7] vctrs_0.2.0        generics_0.0.2     htmltools_0.3.6   
## [10] yaml_2.2.0.9999    rlang_0.4.0        e1071_1.7-2       
## [13] pillar_1.4.2       withr_2.1.2        glue_1.3.1        
## [16] DBI_1.0.0          sp_1.3-1           RColorBrewer_1.1-2
## [19] lifecycle_0.1.0    plyr_1.8.4         stringr_1.4.0     
## [22] munsell_0.5.0      gtable_0.3.0       evaluate_0.14     
## [25] labeling_0.3       class_7.3-15       highr_0.8         
## [28] broom_0.5.2        Rcpp_1.0.2         KernSmooth_2.23-15
## [31] readr_1.3.1        backports_1.1.5    scales_1.0.0      
## [34] classInt_0.4-1     jsonlite_1.6       countrycode_1.1.0 
## [37] hms_0.5.1          digest_0.6.21      stringi_1.4.3     
## [40] grid_3.5.2         bibtex_0.4.2       tools_3.5.2       
## [43] magrittr_1.5       lazyeval_0.2.2     tibble_2.1.3      
## [46] RefManageR_1.2.12  crayon_1.3.4       pkgconfig_2.0.3   
## [49] zeallot_0.1.0      ellipsis_0.3.0     lubridate_1.7.4   
## [52] assertthat_0.2.1   rmarkdown_1.14     httr_1.4.1        
## [55] R6_2.4.0           units_0.6-4        nlme_3.1-141      
## [58] compiler_3.5.2