Introduction to jqr

jq is a lightweight and flexible command-line JSON processor, written in C. It's super fast, and very flexible. jq gives you the ability to index into, parse, and do calculations on JSON data. You can cut up and filter JSON data. You can change JSON key names and values. jq lets you do conditionals and comparisons, and write your own custom functions to operate on JSON data.

You can convert JSON into an R list or other R data structure, and proceed with data parsing, but why not do your JSON parsing on the actual JSON if it's easy enough? That's where jq comes in. Doing your data manipulations on the actual JSON makes it easy to pass data to downstream processes that expect JSON.

If you already familiar with jq by using it on the command line you can use the exact same commands with jqr. If you've never used jq, jqr makes jq easy to learn with a domain specific language - and you can learn the actual jq syntax as you go and apply it on the command line outside of R.

NSE vs. SE

Many functions in jqr have NSE (non-standard evaluation) as well as SE (standard evaluation) versions, where the NSE version for sorting an array is sortj() whereas the SE version is sortj_(). Some functions only have one version, and behave under SE rules.

When you pass JSON into a function as the first parameter (like ad('["a","b","c"]')) rather than piping it in (like '["a","b","c"]' %>% ad), jq() is not executed. Rather you get back an object of class jqr that holds the data you passed in and the query. To execute the query on the data, run jq(), e.g., like jq(ad('["a","b","c"]')) or ad('["a","b","c"]') %>% jq().

When piping JSON to DSL functions jq() is executed on the last DSL function used.

jqr API

There's low and high level (or DSL [domain specific language]) interfaces in jqr.

jqr low level interface

The low level and high level interfaces are unified via the function jq(). You can access the low leve interface by using jq() directly, passing a JSON string as the first parameter, the program (query) as the second, and the flags as the third (by default no flags are passed).

For example, a JSON string could be '{"a": 7, "b": 4}', and the program could be ., resulting in

{
    "a": 7,
    "b": 4
}

The program passed is exactly the same as you'd pass on the command line. Because this is a simple replication of the command line in R, there is a higher level interface, or DSL, to make it easier to use jq. Nonetheless, the low level interface is important as some jq veterans may not want to deal with a DSL, and you may need to drop down to the low level interface if the DSL doesn't work for some reason.

jqr DSL

The jqr DSL uses a suite of functions to construct queries that are executed internally with jq() after the last piped command. We use some logic to determine whether the function call is the last in a series of pipes, and if so, we run jq() on the JSON string and program/query passed.

You don't have to use pipes - they are optional. Though they do make things easier in that you can build up queries easily, just as you would with jq, or any other tools, on the command line.

Load jqr

library("jqr")

Utility functions

Peek

'{"a": 7}' %>% do(.a + 1) %>% peek
#> <jq query>
#>   query: .a + 1
'[8,3,null,6]' %>% sortj %>% peek
#> <jq query>
#>   query: sort

String

'{"a": 7}' %>% do(.a + 1) %>% string
#> [1] "{\"a\": 7}"
'[8,3,null,6]' %>% sortj %>% string
#> [1] "[8,3,null,6]"

Combine

x <- '{"foo": 5, "bar": 7}' %>% select(a = .foo)
combine(x)
#> {
#>     "a": 5
#> }

index

x <- '[{"message": "hello", "name": "jenn"}, {"message": "world", "name": "beth"}]'
x %>% index()
#> [
#>     {
#>         "message": "hello",
#>         "name": "jenn"
#>     },
#>     {
#>         "message": "world",
#>         "name": "beth"
#>     }
#> ]

sort

Note the function name is sortj to avoid collision with base::sort. In addition, a number of other functions in this package that conflict with base R functions have a j on the end.

'[8,3,null,6]' %>% sortj
#> [
#>     null,
#>     3,
#>     6,
#>     8
#> ]

sort in reverse order

'[1,2,3,4]' %>% reverse
#> [
#>     4,
#>     3,
#>     2,
#>     1
#> ]

join

'["a","b,c,d","e"]' %>% join
#> "a, b,c,d, e"
'["a","b,c,d","e"]' %>% join(`;`)
#> "a; b,c,d; e"

starts- and ends-with

'["fo", "foo", "barfoo", "foobar", "barfoob"]' %>% index %>% endswith(foo)
#> [
#>     false,
#>     true,
#>     true,
#>     false,
#>     false
#> ]
'["fo", "foo", "barfoo", "foobar", "barfoob"]' %>% index %>% startswith(foo)
#> [
#>     false,
#>     true,
#>     false,
#>     true,
#>     false
#> ]

contains

'"foobar"' %>% contains("bar")
#> true

unique

'[1,2,5,3,5,3,1,3]' %>% uniquej
#> [
#>     1,
#>     2,
#>     3,
#>     5
#> ]

data types

Get type information for each element

'[0, false, [], {}, null, "hello"]' %>% types
#> [
#>     "number",
#>     "boolean",
#>     "array",
#>     "object",
#>     "null",
#>     "string"
#> ]
'[0, false, [], {}, null, "hello", true, [1,2,3]]' %>% types
#> [
#>     "number",
#>     "boolean",
#>     "array",
#>     "object",
#>     "null",
#>     "string",
#>     "boolean",
#>     "array"
#> ]

Select elements by type

'[0, false, [], {}, null, "hello"]' %>% index() %>% type(booleans)
#> false

keys

Get keys

str <- '{"foo": 5, "bar": 7}'
str %>% keys()
#> [
#>     "bar",
#>     "foo"
#> ]

Delete by key name

str %>% del(bar)
#> {
#>     "foo": 5
#> }
str %>% del(foo)
#> {
#>     "bar": 7
#> }

Check for key existence

str3 <- '[[0,1], ["a","b","c"]]'
str3 %>% haskey(2)
#> [
#>     false,
#>     true
#> ]
str3 %>% haskey(1,2)
#> [
#>     true,
#>     false,
#>     true,
#>     true
#> ]

select

Select variables by name, and rename

'{"foo": 5, "bar": 7}' %>% select(a = .foo)
#> {
#>     "a": 5
#> }

More complicated select(), using the included dataset commits

commits %>%
  index() %>%
  select(sha = .sha, name = .commit.committer.name)
#> [
#>     {
#>         "sha": [
#>             "110e009996e1359d25b8e99e71f83b96e5870790"
#>         ],
#>         "name": [
#>             "Nicolas Williams"
#>         ]
#>     },
#>     {
#>         "sha": [
#>             "7b6a018dff623a4f13f6bcd52c7c56d9b4a4165f"
#>         ],
#>         "name": [
#>             "Nicolas Williams"
#>         ]
#>     },
#>     {
#>         "sha": [
#>             "a50e548cc5313c187483bc8fb1b95e1798e8ef65"
#>         ],
#>         "name": [
#>             "Nicolas Williams"
#>         ]
#>     },
#>     {
#>         "sha": [
#>             "4b258f7d31b34ff5d45fba431169e7fd4c995283"
#>         ],
#>         "name": [
#>             "Nicolas Williams"
#>         ]
#>     },
#>     {
#>         "sha": [
#>             "d1cb8ee0ad3ddf03a37394bfa899cfd3ddd007c5"
#>         ],
#>         "name": [
#>             "Nicolas Williams"
#>         ]
#>     }
#> ]

maths

Maths comparisons

'[5,4,2,7]' %>% index() %>% do(. < 4)
#> [
#>     false,
#>     false,
#>     true,
#>     false
#> ]
'[5,4,2,7]' %>% index() %>% do(. > 4)
#> [
#>     true,
#>     false,
#>     false,
#>     true
#> ]
'[5,4,2,7]' %>% index() %>% do(. <= 4)
#> [
#>     false,
#>     true,
#>     true,
#>     false
#> ]
'[5,4,2,7]' %>% index() %>% do(. >= 4)
#> [
#>     true,
#>     true,
#>     false,
#>     true
#> ]
'[5,4,2,7]' %>% index() %>% do(. == 4)
#> [
#>     false,
#>     true,
#>     false,
#>     false
#> ]
'[5,4,2,7]' %>% index() %>% do(. != 4)
#> [
#>     true,
#>     false,
#>     true,
#>     true
#> ]

sqrt

'9' %>% sqrtj
#> 3

floor

'3.14159' %>% floorj
#> 3

find minimum

'[5,4,2,7]' %>% minj
#> 2
'[{"foo":1, "bar":14}, {"foo":2, "bar":3}]' %>% minj
#> {
#>     "foo": 2,
#>     "bar": 3
#> }
'[{"foo":1, "bar":14}, {"foo":2, "bar":3}]' %>% minj(foo)
#> {
#>     "foo": 1,
#>     "bar": 14
#> }
'[{"foo":1, "bar":14}, {"foo":2, "bar":3}]' %>% minj(bar)
#> {
#>     "foo": 2,
#>     "bar": 3
#> }

find maximum

'[5,4,2,7]' %>% maxj
#> 7
'[{"foo":1, "bar":14}, {"foo":2, "bar":3}]' %>% maxj
#> {
#>     "foo": 1,
#>     "bar": 14
#> }
'[{"foo":1, "bar":14}, {"foo":2, "bar":3}]' %>% maxj(foo)
#> {
#>     "foo": 2,
#>     "bar": 3
#> }
'[{"foo":1, "bar":14}, {"foo":2, "bar":3}]' %>% maxj(bar)
#> {
#>     "foo": 1,
#>     "bar": 14
#> }