Introduction to roadoi

What’s returned?

The client supports API version 2. According to the Unpaywall Data Format, the following variables with the following definitions are returned:

The columns best_oa_location and oa_locations are list-columns that contain useful metadata about the OA sources found by Unpaywall These are

There at least two ways to simplify these list-columns.

To get the full-text links from the list-column best_oa_location, you may want to use purrr::map_chr().

library(dplyr)

## Warning: package 'dplyr' was built under R version 3.5.1

## 
## Attaching package: 'dplyr'

## The following objects are masked from 'package:stats':
## 
##     filter, lag

## The following objects are masked from 'package:base':
## 
##     intersect, setdiff, setequal, union

roadoi::oadoi_fetch(dois = c("10.1186/s12864-016-2566-9",
                             "10.1103/physreve.88.012814"), 
                    email = "najko.jahn@gmail.com") %>%
  dplyr::mutate(
    urls = purrr::map(best_oa_location, "url") %>% 
                  purrr::map_if(purrr::is_empty, ~ NA_character_) %>% 
                  purrr::flatten_chr()
                ) %>%
  .$urls

## [1] "https://bmcgenomics.biomedcentral.com/track/pdf/10.1186/s12864-016-2566-9?site=bmcgenomics.biomedcentral.com"
## [2] "https://link.aps.org/accepted/10.1103/PhysRevE.88.012814"

If you want to gather all full-text links and to explore where these links are hosted, simplify the list-column oa_locations with tidyr::unnest():

library(dplyr)
roadoi::oadoi_fetch(dois = c("10.1186/s12864-016-2566-9",
                             "10.1103/physreve.88.012814"), 
                    email = "najko.jahn@gmail.com") %>%
  tidyr::unnest(oa_locations) %>% 
  dplyr::mutate(
    hostname = purrr::map(url, httr::parse_url) %>% 
                  purrr::map_chr(., "hostname", .null = NA_integer_)
                ) %>% 
  dplyr::mutate(hostname = gsub("www.", "", hostname)) %>%
  dplyr::group_by(hostname) %>%
  dplyr::summarize(hosts = n())

## # A tibble: 7 x 2
##   hostname                      hosts
##   <chr>                         <int>
## 1 arxiv.org                         1
## 2 bmcgenomics.biomedcentral.com     1
## 3 doi.org                           1
## 4 europepmc.org                     1
## 5 link.aps.org                      1
## 6 ncbi.nlm.nih.gov                  1
## 7 pub.uni-bielefeld.de              2

Note that fields to be returned might change according to the Unpaywall API specs

Any API restrictions?

There are no API restrictions. However, providing your email address when using this client is required by Unpaywall. Set email address in your .Rprofile file with the option roadoi_email when you are too tired to type in your email address every time you want to call Unpaywall.

options(roadoi_email = "najko.jahn@gmail.com")

Keeping track of crawling

To follow your API call, and to estimate the time until completion, use the .progress parameter inherited from plyr to display a progress bar.

roadoi::oadoi_fetch(dois = c("10.1186/s12864-016-2566-9",
                             "10.1103/physreve.88.012814"), 
                    email = "najko.jahn@gmail.com", 
                    .progress = "text")

## 
  |                                                                       
  |                                                                 |   0%
  |                                                                       
  |================================                                 |  50%
  |                                                                       
  |=================================================================| 100%

## # A tibble: 2 x 16
##   doi          best_oa_location oa_locations  data_standard is_oa genre   
##   <chr>        <list>           <list>                <int> <lgl> <chr>   
## 1 10.1186/s12… <tibble [1 × 9]> <tibble [6 ×…             2 TRUE  journal…
## 2 10.1103/phy… <tibble [1 × 9]> <tibble [2 ×…             2 TRUE  journal…
## # ... with 10 more variables: journal_is_oa <lgl>,
## #   journal_is_in_doaj <lgl>, journal_issns <chr>, journal_name <chr>,
## #   publisher <chr>, title <chr>, year <chr>, updated <chr>,
## #   non_compliant <list>, authors <list>

Catching errors

Unpaywall is a reliable API. However, this client follows Hadley Wickham’s Best practices for writing an API package and throws an error when the API does not return valid JSON or is not available. To catch these errors, you may want to use purrr’s safely() function

random_dois <-  c("ldld", "10.1038/ng.3260", "§dldl  ")
my_data <- purrr::map(random_dois, 
              .f = purrr::safely(function(x) roadoi::oadoi_fetch(x, email ="najko.jahn@gmail.com")))

## Request failed [404]. Retrying in 1 seconds...
## Request failed [404]. Retrying in 1 seconds...

## Warning: Unpaywall request failed [404]
## 'ldld' is an invalid doi.  See http://doi.org/ldld

## Request failed [404]. Retrying in 1.5 seconds...

## Request failed [404]. Retrying in 1.2 seconds...

## Warning: Unpaywall request failed [404]
## '§dldl' is an invalid doi.  See http://doi.org/§dldl

# return results as data.frame
purrr::map_df(my_data, "result")

## # A tibble: 1 x 16
##   doi        best_oa_location oa_locations   data_standard is_oa genre    
##   <chr>      <list>           <list>                 <int> <lgl> <chr>    
## 1 10.1038/n… <tibble [1 × 9]> <tibble [1 × …             2 TRUE  journal-…
## # ... with 10 more variables: journal_is_oa <lgl>,
## #   journal_is_in_doaj <lgl>, journal_issns <chr>, journal_name <chr>,
## #   publisher <chr>, title <chr>, year <chr>, updated <chr>,
## #   non_compliant <list>, authors <list>

#show errors
purrr::map(my_data, "error")

## [[1]]
## NULL
## 
## [[2]]
## NULL
## 
## [[3]]
## NULL

Gathering DOIs representing scholarly publications

DOIs have become essential for referencing scholarly publications, and thus many digital libraries and institutional databases keep track of these persistent identifiers. For the sake of this vignette, instead of starting with a pre-defined set of publications originating from these sources, we simply generate a random sample of 100 DOIs registered with Crossref by using the rcrossref package.

library(dplyr)
library(rcrossref)
# get a random sample of DOIs and metadata describing these works
random_dois <- rcrossref::cr_r(sample = 100) %>%
  rcrossref::cr_works() %>%
  .$data
random_dois

## # A tibble: 100 x 31
##    container.title    created  deposited doi    indexed issn  issue issued
##    <chr>              <chr>    <chr>     <chr>  <chr>   <chr> <chr> <chr> 
##  1 The Journal of As… 2006-04… 2017-04-… 10.23… 2018-0… 0021… 2     1976-…
##  2 Food Additives an… 2002-07… 2016-12-… 10.10… 2018-0… 0265… 9     2000-…
##  3 American Journal … 2016-04… 2017-06-… 10.10… 2018-0… 0002… 1     2016-…
##  4 Contemporary Poli… 2008-04… 2016-12-… 10.10… 2018-0… 1356… 1     1999-…
##  5 Journal of the Ne… 2017-12… 2017-12-… 10.10… 2018-0… 0022… <NA>  2017-…
##  6 Dissolution Techn… 2015-01… 2017-04-… 10.14… 2018-0… 1521… 2     2009  
##  7 The Astronomical … 2002-07… 2007-02-… 10.10… 2018-0… 0004… <NA>  1950-…
##  8 Journal of Thorac… 2008-05… 2017-06-… 10.10… 2018-0… 1556… 3     2007-…
##  9 Sedimentation Eng… 2013-05… 2017-02-… 10.10… 2018-0… <NA>  <NA>  2008-…
## 10 OECD Economic Out… 2009-07… 2017-07-… 10.17… 2018-0… 0474… <NA>  2006-…
## # ... with 90 more rows, and 23 more variables: member <chr>, page <chr>,
## #   prefix <chr>, publisher <chr>, reference.count <chr>, score <chr>,
## #   source <chr>, title <chr>, type <chr>, url <chr>, volume <chr>,
## #   author <list>, link <list>, alternative.id <chr>, update.policy <chr>,
## #   assertion <list>, license <list>, isbn <chr>, archive <chr>,
## #   subject <chr>, subtitle <chr>, funder <list>, abstract <chr>

Let’s see when these random publications were published

random_dois %>%
  # convert to years
  mutate(issued, issued = lubridate::parse_date_time(issued, c('y', 'ymd', 'ym'))) %>%
  mutate(issued, issued = lubridate::year(issued)) %>%
  group_by(issued) %>%
  summarize(pubs = n()) %>%
  arrange(desc(pubs))

## # A tibble: 46 x 2
##    issued  pubs
##     <dbl> <int>
##  1     NA     7
##  2   2009     5
##  3   1985     4
##  4   1991     4
##  5   2006     4
##  6   2008     4
##  7   2011     4
##  8   2012     4
##  9   2014     4
## 10   2015     4
## # ... with 36 more rows

and of what type they are

random_dois %>%
  group_by(type) %>%
  summarize(pubs = n()) %>%
  arrange(desc(pubs))

## # A tibble: 8 x 2
##   type                 pubs
##   <chr>               <int>
## 1 journal-article        70
## 2 book-chapter           11
## 3 component               5
## 4 proceedings-article     5
## 5 report                  4
## 6 book                    2
## 7 reference-entry         2
## 8 journal-issue           1

Calling Unpaywall

Now let’s call Unpaywall. We are capturing possible errors.

oa_df <- purrr::map(random_dois$doi, .f = purrr::safely(
  function(x) roadoi::oadoi_fetch(x, email = "najko.jahn@gmail.com")
  )) %>%
  purrr::map_df("result")

and merge the resulting information about open access full-text links with parts of our Crossref metadata-set

my_df <- random_dois %>%
  select(doi, type) %>% 
  left_join(oa_df, by = c("doi" = "doi"))

Reporting

After gathering the data, reporting with R is very straightforward. You can even generate dynamic reports using R Markdown and related packages, thus making your study reproducible and transparent for others.

To display how many full-text links were found and which sources were used in a nicely formatted markdown-table using the knitr-package:

my_df %>%
  group_by(is_oa) %>%
  summarise(Articles = n()) %>%
  mutate(Proportion = Articles / sum(Articles)) %>%
  arrange(desc(Articles)) %>%
  knitr::kable()

How did Unpaywall find those Open Access full-texts, which were characterized as best matches, and how are these OA types distributed over publication types?

my_df %>%
  filter(is_oa == TRUE) %>%
  tidyr::unnest(best_oa_location) %>% 
  group_by(evidence, type) %>%
  summarise(Articles = n()) %>%
  arrange(desc(Articles)) %>%
  knitr::kable()

More examples

For more examples, see Piwowar et al. 2018.² Together with the article, they shared their analysis of Unpaywall Data as R Markdown supplement.