Zoltr for Project Owners

Welcome to the zoltr vignette for project owners and forecasters. You should read this if you are interested in creating and managing your own zoltardata.com projects using this package to access them via the Zoltar API. Building on the Getting Started vignette, this one covers deleting and uploading forecasts. Note that before starting you should:

Create an account on zoltardata.com.
Create a ‘sandbox’ project with one timezero for testing.
Create a model in that project.
Edit your .Renviron file:
- USERNAME and PASSWORD: should match your account settings
- PROJECT_NAME and MODEL_NAME: should match your project and model
- TIMEZERO_DATE: should match your project
- FORECAST_CSV_FILE: the path of a forecast data file in the format documented in Zoltar documentation

Connect to the host and authenticate

To access your project, you’ll first need to authenticate via the zoltar_authenticate() function. Pass it the username and password for your account. Notes:

Be careful to store and use your username and password so that they’re not accessible to others. A good practice is to put them them in your .Renviron file and then use Sys.getenv() to retrieve them, as outlined in Use Environment variables and as done below.
The Zoltar service uses a “token”-based scheme for authentication. These tokens have a five minute expiration for security, which requires re-authentication after that period of time. The zoltr library takes care of re-authenticating as needed by passing your username and password back to the server to get another token. Note that the connection object returned by the new_connection function stores a token internally, so be careful if saving that object into a file.
Below you will see the host shown as 127.0.0.1:8000 - this is a temporary one that was used to generate this documentation.

library(zoltr)
conn <- new_connection()
zoltar_authenticate(conn, Sys.getenv("USERNAME"), Sys.getenv("PASSWORD"))
conn

Get the sandbox project and model work with and then list its forecasts (if any)

Get the ID of the project named by PROJECT_NAME and then get the ID of that project’s model named MODEL_NAME. From the model ID, call the forecasts() function to get a data.frame of that model’s forecasts.

the_projects <- projects(conn)
project_id <- the_projects[the_projects$name == Sys.getenv("PROJECT_NAME"), 'id']  # integer(0) if not found, which is an invalid project ID
the_models <- models(conn, project_id)
model_id <- the_models[the_models$name == Sys.getenv("MODEL_NAME"), 'id']  # integer(0) if not found
the_forecasts <- forecasts(conn, model_id)
str(the_forecasts)
#> 'data.frame':    1 obs. of  4 variables:
#>  $ id               : int 214
#>  $ url              : chr "http://127.0.0.1:8000/api/forecast/214/"
#>  $ timezero_date    : Date, format: "2017-01-17"
#>  $ data_version_date: Date, format: NA

Delete the model’s existing forecast for TIMEZERO_DATE, if any, first printing its info

Get the forecast for the timezero date specified by TIMEZERO_DATE and then call delete_forecast(), if one was found.

timezero_date_str <- Sys.getenv("TIMEZERO_DATE")
timezero_date <- as.Date(timezero_date_str, format = "%Y%m%d")  # YYYYMMDD
existing_forecast_id <- the_forecasts[the_forecasts$timezero_date == timezero_date, 'id']

the_forecast_info <- forecast_info(conn, existing_forecast_id)  # `Not Found (HTTP 404)` this if forecast doesn't exist 
str(the_forecast_info)
#> List of 6
#>  $ id            : int 214
#>  $ url           : chr "http://127.0.0.1:8000/api/forecast/214/"
#>  $ forecast_model: chr "http://127.0.0.1:8000/api/model/1/"
#>  $ csv_filename  : chr "EW1-KoTsarima-2017-01-17-small.csv"
#>  $ time_zero     :List of 2
#>   ..$ timezero_date    : chr "20170117"
#>   ..$ data_version_date: NULL
#>  $ forecast_data : chr "http://127.0.0.1:8000/api/forecast/214/data/"

delete_forecast(conn, the_forecast_info$id)

Upload a forecast

Now let’s upload the forecast data file FORECAST_CSV_FILE for TIMEZERO_DATE via the upload_forecast() function.

Keep in mind that Zoltar queues long operations like forecast uploading, which keeps the site responsive, but makes the Zoltar API a little more complicated. Rather than having the upload_forecast() function block until the upload is done, you instead get a quick response in the form of an UploadFileJob ID that you can pass to the upload_info() function to check its status to find out when the upload is done (or failed). (This is called polling the host to ask the status.) Here we poll every second using a helper function.

busy_poll_upload_file_job <- function(upload_file_job_id) {
    cat(paste0("polling for status change. upload_file_job: ", upload_file_job_id, "\n"))
    while (TRUE) {
        status <- upload_info(conn, upload_file_job_id)$status
        cat(paste0(status, "\n"))
        if (status == "FAILED") {
            cat(paste0("x failed\n"))
            break
        }
        if (status == "SUCCESS") {
            break
        }
        Sys.sleep(1)
    }
}

forecast_csv_file <- Sys.getenv("FORECAST_CSV_FILE")
upload_file_job_id <- upload_forecast(conn, model_id, timezero_date_str, forecast_csv_file)
busy_poll_upload_file_job(upload_file_job_id)
#> polling for status change. upload_file_job: 194
#> QUEUED
#> SUCCESS

And here’s the upload’s final information:

the_upload_info <- upload_info(conn, upload_file_job_id)
str(the_upload_info)
#> List of 10
#>  $ id             : int 194
#>  $ url            : chr "http://127.0.0.1:8000/api/uploadfilejob/194/"
#>  $ status         : chr "SUCCESS"
#>  $ user           : chr "http://127.0.0.1:8000/api/user/3/"
#>  $ created_at     : Date[1:1], format: "2019-05-30"
#>  $ updated_at     : Date[1:1], format: "2019-05-30"
#>  $ failure_message: chr ""
#>  $ filename       : chr "EW1-KoTsarima-2017-01-17-small.csv"
#>  $ input_json     :List of 2
#>   ..$ forecast_model_pk: int 1
#>   ..$ timezero_pk      : int 2
#>  $ output_json    :List of 1
#>   ..$ forecast_pk: int 215

Print information about the newly-uploaded forecast

Hopefully you’ll see some number of “QUEUED” entries followed by a “SUCCESS” one. Now let’s get the new forecast’s information.

Due to there being multiple kinds of files you might upload to Zoltar (including project truth, project template, and model forecasts), the API returns information about uploaded files in the $output_json field of successful upload job’s info. For newly-uploaded forecasts, that field contains a $forecast_pk field.


new_forecast_id <- upload_info(conn, upload_file_job_id)$output_json$forecast_pk
the_forecast_info <- forecast_info(conn, new_forecast_id)
str(the_forecast_info)
#> List of 6
#>  $ id            : int 215
#>  $ url           : chr "http://127.0.0.1:8000/api/forecast/215/"
#>  $ forecast_model: chr "http://127.0.0.1:8000/api/model/1/"
#>  $ csv_filename  : chr "EW1-KoTsarima-2017-01-17-small.csv"
#>  $ time_zero     :List of 2
#>   ..$ timezero_date    : chr "20170117"
#>   ..$ data_version_date: NULL
#>  $ forecast_data : chr "http://127.0.0.1:8000/api/forecast/215/data/"

Look at the model’s forecast list to see the new forecast

Finally, let’s again examine the model’s forecasts and notice the new one is there.

the_forecasts <- forecasts(conn, model_id)
str(the_forecasts)
#> 'data.frame':    1 obs. of  4 variables:
#>  $ id               : int 215
#>  $ url              : chr "http://127.0.0.1:8000/api/forecast/215/"
#>  $ timezero_date    : Date, format: "2017-01-17"
#>  $ data_version_date: Date, format: NA

Project Owners

Matthew Cornell

2019-05-28