Getting Started

Emanuel Rodriguez

2019-04-10

Install

You can install CDECRetrieve the usual way,

# for stable version
install.packages("CDECRetrieve")

# for development version
devtools::install_github("flowwest/CDECRetrieve")

Intro

The goal for CDECRetrieve is to create a workflow for R users using CDEC data, we believe that a well defined workflow is easier to automate and less prone to error (or easier to catch errors). In order to do this we create “services” out of different endpoints available through the CDEC site. A lot ideas in developing the package came from using dataRetrieval from USGS and the NOAA CDO api.

Exploring Locations

We start by first exploring locations of interest. The CDEC site provides a web form with a lot of options,

cdec station search

cdec station search

The pakcage exposes this functionallity through cdec_stations(). Although it doesn’t (currently) map all options in the web form it does so for the most used, namely, station id, nearby city, river basin, hydro area and county. At least one of the parameters must be supplied, and combination of these can be supplied to refine the search.

library(CDECRetrieve)

cdec_stations(station_id = "kwk") # return metadata for KWK
#> # A tibble: 1 x 9
#>   station_id name  river_basin county longitude latitude elevation operator
#>   <chr>      <chr> <chr>       <chr>      <dbl>    <dbl>     <int> <chr>   
#> 1 kwk        sacr… sacramento… shasta     -122.     40.6       596 US Geol…
#> # … with 1 more variable: state <chr>

# show all locations near san francisco, this returns a set of 
# CDEC station that are near San Francisco
cdec_stations(nearby_city = "san francisco")
#> # A tibble: 3 x 9
#>   station_id name  river_basin county longitude latitude elevation operator
#>   <chr>      <chr> <chr>       <chr>      <dbl>    <dbl>     <int> <chr>   
#> 1 cx2        dail… sf bay      san f…     -122.     37.8         0 CA Dept…
#> 2 sfn        san … sf bay      san f…     -122.     37.8       150 Nationa…
#> 3 ggt        gold… sf bay      san f…     -122.     37.8         0 Nationa…
#> # … with 1 more variable: state <chr>

# show all location in the sf bay river basin
cdec_stations(river_basin = "sf bay")
#> # A tibble: 24 x 9
#>    station_id name  river_basin county longitude latitude elevation
#>    <chr>      <chr> <chr>       <chr>      <dbl>    <dbl> <chr>    
#>  1 hml        moun… sf bay      santa…     -122.     37.3 4,206    
#>  2 okm        oakl… sf bay      alame…     -122.     37.8 30       
#>  3 snn        san … sf bay      san m…     -122.     37.6 456      
#>  4 cx2        dail… sf bay      san f…     -122.     37.8 0        
#>  5 sfn        san … sf bay      san f…     -122.     37.8 150      
#>  6 sff        san … sf bay      san m…     -122.     37.6 8        
#>  7 spb        san … sf bay      contr…     -122.     37.9 330      
#>  8 rwc        redw… sf bay      none …    -1000.    100.0 31       
#>  9 vsb        vall… sf bay      alame…     -122.     37.6 635      
#> 10 lfy        lafa… sf bay      contr…     -122.     37.9 465      
#> # … with 14 more rows, and 2 more variables: operator <chr>, state <chr>

# show all station in Tehama county
cdec_stations(county = "tehama")
#> # A tibble: 44 x 9
#>    station_id name  river_basin county longitude latitude elevation
#>    <chr>      <chr> <chr>       <chr>      <dbl>    <dbl> <chr>    
#>  1 blb        blac… stony cr    tehama     -122.     39.8 426      
#>  2 ctn        cott… cottonwood… tehama     -123.     40.3 3,400    
#>  3 sbb        sacr… sacto vly … tehama     -122.     40.3 186      
#>  4 dch        deer… sacramento… tehama     -121.     40.3 50       
#>  5 sh1        shee… sacramento… tehama     -123.     39.5 6,500    
#>  6 vno        sacr… sacramento… tehama     -122.     39.9 185      
#>  7 bsf        sacr… sacramento… tehama     -122.     40.4 360      
#>  8 bnd        sacr… sacramento… tehama     -122.     40.3 286      
#>  9 teh        sacr… sacramento… tehama     -122.     40.0 213      
#> 10 vin        sacr… sacramento… tehama     -122.     39.9 185      
#> # … with 34 more rows, and 2 more variables: operator <chr>, state <chr>

Since we are simply exploring for locations of interest, it may be useful to map these for visual inspection. CDECRetrieve provides a simple function to do exactly this map_stations().

library(magrittr)
library(leaflet)

cdec_stations(county = "tehama") %>% 
  map_stations()

The same can be done with leaflet functions

d <- cdec_stations(county = "tehama")
leaflet(d) %>% 
  addTiles() %>% 
  addCircleMarkers(label=~station_id) #psk is way off here

Exploring Datasets within a Station

After exploring stations in a desired location. We can start focusing on the datasets available at the locations.

station <- "sha"
cdec_datasets("sha")
#> # A tibble: 21 x 6
#>    sensor_number sensor_name    sensor_units duration start      end       
#>            <int> <chr>          <chr>        <chr>    <date>     <date>    
#>  1             2 precipitation… inches       daily    2003-10-01 2019-04-10
#>  2             2 precipitation… inches       monthly  1953-10-01 2019-04-10
#>  3             6 reservoir ele… feet         daily    1985-01-01 2019-04-10
#>  4             6 reservoir ele… feet         hourly   1993-12-09 2019-04-10
#>  5             8 full natural … cfs          daily    1987-05-31 2019-04-10
#>  6            15 reservoir sto… af           daily    1985-01-01 2019-04-10
#>  7            15 reservoir sto… af           hourly   1994-06-24 2019-04-10
#>  8            15 reservoir sto… af           monthly  1953-10-01 2019-04-10
#>  9            22 reservoir sto… af           daily    1993-10-03 2019-04-10
#> 10            23 reservoir out… cfs          daily    1987-01-05 2019-04-10
#> # … with 11 more rows

Since all of these functions return a tidy dataframe we can make use of the dplyr to filter, mutate and explore. Here we look for datasets in Shasta that report a storage

library(magrittr)

cdec_datasets("sha") %>% 
  dplyr::filter(grepl("storage", sensor_name))
#> # A tibble: 5 x 6
#>   sensor_number sensor_name     sensor_units duration start      end       
#>           <int> <chr>           <chr>        <chr>    <date>     <date>    
#> 1            15 reservoir stor… af           daily    1985-01-01 2019-04-10
#> 2            15 reservoir stor… af           hourly   1994-06-24 2019-04-10
#> 3            15 reservoir stor… af           monthly  1953-10-01 2019-04-10
#> 4            22 reservoir stor… af           daily    1993-10-03 2019-04-10
#> 5            94 reservoir top … af           daily    2000-10-24 2019-04-10

Take note of the sensor number, and duration, these will be needed for querying data in the next section.

Query Data

Now that we have a location, parameter of interest and duration we can start to query for actual data.

sha_storage_daily <- cdec_query(station = "sha", sensor_num = "15", 
                                dur_code = "d", start_date = "2018-01-01", 
                                end_date = Sys.Date())

sha_storage_daily
#> # A tibble: 465 x 5
#>    agency_cd location_id datetime            parameter_cd parameter_value
#>    <chr>     <chr>       <dttm>              <chr>                  <dbl>
#>  1 CDEC      SHA         2018-01-01 00:00:00 15                   3203249
#>  2 CDEC      SHA         2018-01-02 00:00:00 15                   3202064
#>  3 CDEC      SHA         2018-01-03 00:00:00 15                   3203723
#>  4 CDEC      SHA         2018-01-04 00:00:00 15                   3206566
#>  5 CDEC      SHA         2018-01-05 00:00:00 15                   3210358
#>  6 CDEC      SHA         2018-01-06 00:00:00 15                   3215097
#>  7 CDEC      SHA         2018-01-07 00:00:00 15                   3217003
#>  8 CDEC      SHA         2018-01-08 00:00:00 15                   3229391
#>  9 CDEC      SHA         2018-01-09 00:00:00 15                   3237014
#> 10 CDEC      SHA         2018-01-10 00:00:00 15                   3242032
#> # … with 455 more rows

Once again the the data is in a tidy form.

Plot

We can plot with ggplot2

library(ggplot2)

sha_storage_daily %>% 
  ggplot(aes(datetime, parameter_value)) + geom_line()