arXiv is a repository of electronic preprints for computer science, mathematics, physics, quantitative biology, quantitative finance, and statistics. The aRxiv package provides an R interface to the arXiv API.
Note that the arXiv API does not require an API key.
You can install the aRxiv package via CRAN:
devtools::install_github() to get the (more recent) version
install.packages("devtools") library(devtools) install_github("ropensci/aRxiv")
arxiv_search() to search arXiv,
arxiv_count() to get a simple count of manuscripts matching a
arxiv_open() to open the abstract pages for a set of
We'll get to the details in a moment. For now, let's look at a few examples.
Suppose we wanted to identify all arXiv manuscripts with “
Hall” as an author. It is best to first get a count, so that we have
a sense of how many records the search will return. (Peter Hall is
“among the world's most prolific and highly cited authors in both probability and statistics.”)
We first use
library() to load the aRxiv package and then
arxiv_count() to get the count.
library(aRxiv) arxiv_count('au:"Peter Hall"')
##  51
au: part indicates to search the author field; we use double
quotes to search for a phrase.
To obtain the actual records matching the query, use
rec <- arxiv_search('au:"Peter Hall"') nrow(rec)
##  10
The default is to grab no more than 10 records; this limit can be
changed with the
limit argument. But note that the arXiv API will
not let you download more than 50,000 or so records, and even in that
case it's best to do so in batches; more on this below.
Also note that the result of
arxiv_search() has an attribute
"total_results" containing the total count of search results; this
is the same as what
##  51
The following will get us all 51 records.
rec <- arxiv_search('au:"Peter Hall"', limit=50) nrow(rec)
##  50
arxiv_search() returns a data frame with each row being a single
manuscript. The columns are the different fields (e.g.,
abstract, etc.). Fields like
contain multiple items will be a single character string with the
multiple items separated by a vertical bar (
We might be interested in a more restrictive search, such as for Peter
Hall's arXiv manuscripts that have
deconvolution in the title. We
ti: to search the title field, and combine the two with
deconv <- arxiv_search('au:"Peter Hall" AND ti:deconvolution') nrow(deconv)
##  4
Let's display just the authors and title for the results.
deconv[, c('title', 'authors')]
## title ## 1 A ridge-parameter approach to deconvolution ## 2 On deconvolution with repeated measurements ## 3 Estimation of distributions, moments and quantiles in deconvolution\n problems ## 4 Kernel methods and minimum contrast estimators for empirical\n deconvolution ## authors ## 1 Peter Hall|Alexander Meister ## 2 Aurore Delaigle|Peter Hall|Alexander Meister ## 3 Peter Hall|Soumendra N. Lahiri ## 4 Aurore Delaigle|Peter Hall
We can open the abstract pages for these 4 manuscripts
arxiv_open(). It takes, as input, the output of
The two basic arguments to
character string representing the search, and
id_list, a list of
arXiv manuscript identifiers.
queryis provided, manuscripts matching that query are returned.
id_listis provided, manuscripts in the list are returned.
querywill be returned.
query may be a single character string or a vector of character
strings. If it is a vector, the elements are pasted together with
id_list may be a vector of character strings or a single
comma-separated character string.
Generally, one would ignore
id_list and focus on forming the
argument. The aRxiv package includes a dataset
lists the terms (like
au) that you can use.
## term description ## 1 ti Title ## 2 au Author ## 3 abs Abstract ## 4 co Comment ## 5 jr Journal Reference ## 6 cat Subject Category ## 7 rn Report Number ## 8 all All of the above ## 9 submittedDate Date/time of initial submission, as YYYYMMDDHHMM ## 10 lastUpdatedDate Date/time of last update, as YYYYMMDDHHMM
Use a colon (
:) to separate the query term from the actual query.
Multiple queries can be combined with
##  14883
arxiv_count('au:Peter OR au:Hall')
##  14883
arxiv_count('au:Peter AND au:Hall')
##  72
arxiv_count('au:Hall ANDNOT au:Peter')
##  1294
It appears that in the author field (and many other fields) you must search full words, and that wild cards not allowed.
arxiv_count('au:P* AND au:Hall')
##  0
arxiv_count('au:P AND au:Hall')
##  568
##  37
arXiv has a set of 127 subject classifications,
searchable with the prefix
cat:. The aRxiv package contains a
arxiv_cats containing the abbreviations and descriptions.
Here are the statistics categories.
## abbreviation description ## 1 stat.AP Statistics - Applications ## 2 stat.CO Statistics - Computation ## 3 stat.ML Statistics - Machine Learning ## 4 stat.ME Statistics - Methodology ## 5 stat.TH Statistics - Theory
To search these categories, you need to include either the full term
or use the
##  0
##  3367
##  17901
submittedDate (date/time of first submission) and
lastUpdatedDate (date/time of last revision) are particularly
useful for limiting a search with many results, so that you may
combine multiple searches together, each within some window of time,
to get the full results.
The date/time information is of the form
2007-10-18 12:25:34. You can use
for a wildcard for the times. For example, to get all manuscripts
with initial submission on 2007-10-18:
##  196
But you can't use the wildcard within the dates.
##  0
To get a count of all manuscripts with original submission in 2007,
use a date range, like
[from_date TO to_date]. (If you give a partial
date, it's treated as the earliest date/time that matches, and the
range appears to be up to but not including the second date/time.)
arxiv_count('submittedDate:[2007 TO 2008]')
##  55749
The output of
arxiv_search() is a data frame with the following
res <- arxiv_search('au:"Terry Speed"') names(res)
##  "id" "submitted" "updated" ##  "title" "abstract" "authors" ##  "affiliations" "link_abstract" "link_pdf" ##  "link_doi" "comment" "journal_ref" ##  "doi" "primary_category" "categories"
The columns are described in the help file for
A few short notes:
categoriesmay contain multiple items, separated by a vertical bar (
categoriescolumn may contain not just the aRxiv categories (e.g.,
stat.AP) but also codes for the Mathematical Subject Classification (MSC) (e.g., 14J60) and the ACM Computing Classification System (e.g., F.2.2). These are not searchable with
cat:but are searchable with a general search.
##  0
##  367
arxiv_search() function has two arguments for sorting the results,
sort_by (taking values
provided, these sorting arguments are ignored and the results are
presented according to the order in
Here's an example, to sort the results by the date the manuscripts were last updated, in descending order.
res <- arxiv_search('au:"Terry Speed"', sort_by="updated", ascending=FALSE) res$updated
##  "2012-01-31 05:54:46" "2008-06-27 08:25:01"
The arXiv metadata has a number of limitations, the key issue being that it is author-supplied and so not necessarily consistent between records.
Authors' names may vary between records (e.g., T. P. Speed vs. Terry Speed vs. Terence P. Speed). Further, arXiv provides no ability to distinguish multiple individuals with the same name (c.f., ORCID).
Authors' institutional affiliations are mostly missing. The arXiv submission form does not include an affiliation field; affiliations are entered within the author field, in parentheses. The metadata instructions may not be widely read.
There are no key words; you are stuck with searching the free text in the titles and abstracts.
Subject classifications are provided by the authors and may be incomplete or inappropriate.
Care should be taken to avoid multiple requests to the arXiv API in a short period of time. The arXiv API user manual states:
In cases where the API needs to be called multiple times in a row, we encourage you to play nice and incorporate a 3 second delay in your code.
The aRxiv package institutes a delay between requests, with the time
period for the delay configurable with the R option
"aRxiv_delay" (in seconds). The default is 3 seconds.
To reduce the delay to 1 second, use:
Don't do searches in parallel (e.g., via the parallel package). You may be locked out from the arXiv API.
The arXiv API returns only complete records (including the entire abstracts); searches returning large numbers of records can be very slow.
It's best to use
arxiv_search(), so that you have
a sense of how many records you will receive. If the count is large,
you may wish to refine your query.
arXiv has a hard limit of around 50,000 records; for a query that
matches more than 50,000 manuscripts, there is no way to receive the
full results. The simplest solution to this problem is to break the
query into smaller pieces, for example using slices of time, with a
range of dates for
limit argument to
arxiv_search() (with default
limits the number of records to be returned. If you wish to receive
more than 10 records, you must specify a larger limit (e.g.,
To avoid accidental searches that may return a very large number of
arxiv_search() uses an R option,
aRxiv_toomany (with a
default of 15,000), and refuses to attempt a search that will return
results above that limit.
Even for searches that return a moderate number of records (say
2,000), it may be best to make the requests in batches: Use a smaller
value for the
limit argument (say 100), and make multiple requests
with different offsets, indicated with the
start argument, for the
initial record to return.
This is done automatically with the
batchsize argument to
arxiv_search(). A search is split into multiple calls, with no more
batchsize records to be returned by each, and then the results