Accessing macroeconomic data through R

2022-02-14

Introduction

This notebook provides brief guidance on a number R packages that allow accessing economic databases directly through R. These packages take advantage of the increasing number of data providers making their databases accessible through APIs and the availability of R packages for making web requests (for instance, httr or jsonlite). This note covers the public sector data providers (e.g., international organizations, central banks or similar) primarily and leaves out the commercial ones (e.g., Bloomberg, Datastream).

db.nomics

db.nomics describes itself as the “world’s economic database” and it may deserve this description. At the time of writing, db.nomics aggregated datasets from more than 80 sources and provided access to them through a single API. Effectively, by using db.nomics you can ignore many of the R packages for other sources that I cover below, given that it already provides a unified interface to many of these datasets.

The team behind db.nomics provides an R package rdbnomics for accessing their database. I will show below how to get started with this package, while further details can be found on this page or in the vignette that you can access by typing vignette("rdbnomics") in the R console. The package is available through CRAN.

You can quickly check the list of available providers and retrieve the list of datasets (I recommend doing the latter by specifying the provider in the function call):

# load the package
library("rdbnomics")

# retrieve the list of providers (including the provider code)
providers <- rdb_providers()

# retrieve the list of datasets
datasets <- rdb_datasets(provider_code = "IMF", 
                         simplify = TRUE)

# retrive the list of data series (can take some time)
series <- rdb_series(provider_code = "IMF", 
                     dataset_code = "WEO:2021-04", 
                     simplify = TRUE, 
                     verbose = FALSE)

head(series)

rdb is the main function for sending data requests. You can download the series using only the ids argument (referring to an ID of a series). To do this, you need to combine the provider, dataset, and series codes into a single string as follows "provider_code/dataset_code/series_code". Alternatively, these parameters can be fed into the function separately using their own arguments. The following code shows two alternative ways of downloading the same data series using rdb.

# download the constant price GDP per capita series for Uzbekistan using ids or mask arguments

# by using the ids argument
df1 <- rdb(ids = "IMF/WEO:2021-04/UZB.NGDPRPC.national_currency")

# by using separate arguments
df2 <- rdb(provider_code = "IMF", 
           dataset_code = "WEO:2021-04", 
           mask = "UZB.NGDPRPC.national_currency")

# plot
ggplot(df1, aes(x = period, y = value, group = 1)) + 
    geom_line()

You can also use the dimensions argument of rdb to specify one or several dimensions (i.e., variable names) from the dataset (you will need to look up how exactly the variables are named to use this option).

# download the constant price GDP per capita series using the dimensions argument
df <- rdb(provider_code = "IMF", 
          dataset_code = "WEO:2021-04", 
          dimensions = list("weo-country" = "KAZ", 
                            "weo-subject" = "NGDPRPC", 
                            "unit" = "national_currency"))

Another option is to use the query argument to search for a series within a given dataset.

# download the data using a search term
df <- rdb(provider_code = "IMF", 
          dataset_code = "WEO:2021-04", 
          query = "Kazakhstan gross domestic product per capita constant prices national currency")
head(df)

Arguments of rdb accept vectors of strings allowing downloading multiple data series at once. The mask argument also has a convenient notation for combining the data request parameters.

# by using the ids argument and a vector of strings
df <- rdb(ids = c("IMF/WEO:2021-04/KAZ.PCPIPCH.pcent_change",
                  "IMF/WEO:2021-04/UZB.PCPIPCH.pcent_change"))

# by using the mask argument
    # download the series for two countries only
    df <- rdb(provider_code = "IMF", 
              dataset_code = "WEO:2021-04", 
              mask = "KAZ+UZB.PCPIPCH.pcent_change")
    
    # download the series for all countries
    df <- rdb(provider_code = "IMF", 
              dataset_code = "WEO:2021-04", 
              mask = ".PCPIPCH.pcent_change")
    
    # download all the data from the dataset for a given country
    df <- rdb(provider_code = "IMF", 
              dataset_code = "WEO:2021-04", 
              mask = "KAZ.")
    
    # download all the data from the dataset for two countries
    df <- rdb(provider_code = "IMF", 
              dataset_code = "WEO:2021-04", 
              mask = "KAZ+UZB.")

IMF

imfr is a convenient package for directly accessing the datasets provided by IMF.

# load the package
library("imfr")

You can use the imf_ids function to look up the IDs of available datasets:

# look up the ids of datasets
data_ids <- imf_ids(return_raw = FALSE)
rmarkdown::paged_table(data_ids)

Then, use the imf_codelist function to look up the codes for a specific database. The main information you need from this call is the name of an object listing all indicators within the dataset.

# look up the codes for a database
codelist <- imf_codelist(database_id = "BOP")
head(codelist)

You can then use the name of that object to download the list of all indicators with the imf_codes function:

# look up the list of indicators
codes <- imf_codes(codelist = "CL_INDICATOR_BOP")
rmarkdown::paged_table(codes)

Finally, use the imf_data function to download the data using the database and indicator IDs:

# download the data from IMF
df <- imf_data(database_id = "BOP", 
               indicator = "BMGS_BP6_USD", 
               country = "KZ",              # use the two-letter ISO code
               start = 1995, 
               end = current_year(), 
               freq = "A",
               print_url = FALSE)
# plot
ggplot(df, aes(x = year, y = BMGS_BP6_USD, group = 1)) + 
    geom_line() + 
    scale_x_discrete(breaks = seq(df$year[1], df$year[nrow(df)], by = 5))

You can also feed vectors of strings either to indicator or country arguments. Use "all" in the country argument to download the data for all countries.

# downloading two data series for two countries
df <- imf_data(database_id = "BOP", 
               indicator = "BMGS_BP6_USD", 
               country = c("KZ", "RU"), 
               start = 1995, 
               end = current_year(), 
               print_url = FALSE)

head(df)

While the imfr package does allow easily accessing the IMF API, the same IMF data appears better structured and organized when accessed using the db.nomics package.

World Bank

There are several R packages available for downloading the World Bank data. I will review WDI and wbstats.

WDI

The package already contains a data object with the list of available indicators:

# load the package
library("WDI")

# list of indicators
rmarkdown::paged_table(as.data.frame(WDI_data$series))

This local list of indicators can be updated to the latest version using the WDI_cache() function. If necessary, the output of this function can then be fed into the function for downloading data:

# update the list of indicators
new_list <- WDIcache()

# check if the updated list is identical to the one in the package
identical(new_list$series, WDI_data$series) 

[1] FALSE

The main function for downloading the data is WDI:

# download data from the WDI database
df <- WDI(country = "KZ", 
          indicator = "NY.GDP.PCAP.KN", 
          start = 1991, 
          end = 2020,
          extra = FALSE, 
          cache = new_list)

# plot
ggplot(df, aes(x = year, y = NY.GDP.PCAP.KN, group = 1)) + 
    geom_line()

The packages also comes with a useful search function (the function supports regular expressions)

WDIsearch("gdp.*capita.*constant.*LCU")

                      indicator                            name 
               "NY.GDP.PCAP.KN" "GDP per capita (constant LCU)" 

wbstats

This package is quite similar to WDI. It similarly comes with a data object that contains the list of available indicators:

# load the package
library(wbstats)

# the list of indicators
wb_cachelist$indicators

This data object can similarly be updated to the latest version and further fed into the function for downloading data.

# update the list of indicators
new_list <- wb_cache()

wb_data is the function for downloading the data:

# download data from the WDI database
df <- wb_data(indicator = "NY.GDP.PCAP.KN", 
              country = "KZ", 
              start_date = 1991, 
              end_date = 2020, 
              cache = new_list, 
              return_wide = FALSE)

rmarkdown::paged_table(df)

The package also has a convenient function for searching the relevant data series (that also supports regular expressions):

wb_search("gdp.*capita.*constant LCU")

Compared to the WDI::WDI() function, wbstats::wb_data appears to provide more flexibility and convenient options. For instance, the return_wide argument becomes very handy when downloading multiple series at once:

# download two indicators for two countries and transform to wide format
df <- wb_data(indicator = c("NY.GDP.PCAP.KN", "NY.GDP.PCAP.KD.ZG"), 
              country = c("KZ", "UZ"), 
              start_date = 1991, 
              end_date = 2020, 
              cache = new_list, 
              return_wide = TRUE)

rmarkdown::paged_table(df)

In addition, mrv (“most recent value”) and mrnev (“most recent non-empty value”) arguments instructing the function to download the latest available data are also very useful:

# addtional useful arguments of wb_data
df <- wb_data(indicator = "NY.GDP.PCAP.CD", 
              # mrv = 1,                       # number of most recent observations
              # gapfill = TRUE,                # fill in missing values by carrying forward the latest available value
              mrnev = 1)                       # number of most recent non-empty observations

head(df, 10)

I found both WDI and wbstats easy to use but wbstats provides few more convenient functionalities within its main function for downloading data (e.g., for dealing with missing observations and filling the gaps).

OECD

Similar to the packages above, the OECD package has with a function for downloading and looking up the list of available data series:

# load the package
library(OECD)

# look up the list of indicators
oecd_data_list <- get_datasets()
head(oecd_data_list, 10)

You can use the ID of a dataset to look up its structure using the get_data_structure function:

data_structure <- get_data_structure(dataset = "SNA_TABLE1")

These data structure objects are the list of data frames each documenting different dimensions of the dataset (e.g., the list of available indicators, frequency, unit of measure and others). Elements of this information will be necessary for downloading the actual data. get_dataset() is the main function for downloading the data series and you will need to use its filter argument to indicate specific dimensions to customize the data request. You can run this function with no input to the filter argument and the whole dataset will be downloaded in this case. But, let’s say, you want to download only data for Russian GDP (from the “Quarterly National Accounts” dataset) between 2000 and 2020:

# create the list of filters
data_filter <- list("country" = "RUS", 
                    "transaction" = "B1_GA", 
                    "measure" = "V")

# download the data
df <- get_dataset(dataset = "SNA_TABLE1", 
                  filter = data_filter,
                  start_time = 2000, 
                  end_time = 2020)

head(df)

Some datasets allow filtering using additional dimensions (e.g., frequency):

# create the list of filters
data_filter <- list("country" = "RUS", 
                    "transaction" = "B1_GE",
                    "measure" = "LNBQRSA", 
                    "frequency" = "Q")

# download the data
df <- get_dataset(dataset = "QNA", 
                  filter = data_filter,
                  start_time = 2005, 
                  end_time = 2020)

Ordering of filters implied by the dataset should be respected when creating the customized list of filters. For instance, specifying the variable name (B1_GE) in the list above after the frequency element will result in an error, since the list of filters will not correspond to the structure of the QNA dataset. In addition, some filters may be mandatory, while others are optional. In the example above, leaving out the indicator name (B1_GE) but retaining the other filters will result in an error. At the same time, leaving out the measure and frequency filters (LNBQRSA and Q) allows downloading the data series but with all the available measures and frequencies.

Figuring out the relevant filters and their correct ordering through the downloaded data structure object is not straightforward. An easier way to understand which filters are used to produce a given series is to look up the SDMX expression of the series (or of the group of data series) on the OECD website after applying the necessary filters. The SDMX expression equivalent to the list of filters above looks like this (go to Export -> SDMX (XML) -> SDMX DATA URL on this page to look up this expression):

https://stats.oecd.org/restsdmx/sdmx.ashx/GetData/QNA/RUS.B1_GE.LNBQRSA.Q/all?startTime=2005-Q1&endTime=2021-Q3

Searching for datasets using a search term can be done using the search_dataset() function:

search_dataset("QNA", oecd_data_list)

UN Comtrade

UN Comtrade is one of few sources of detailed data on international trade flows. There are several options for accessing this data and I will review comtradr and tradestatistics packages for doing that. UN Comtrade API page also shows how to download their data using a user-defined R function.

comtradr

ct_search is the main function in this package for downloading data:

# load the package
library(comtradr)

# download the data
df <- ct_search(reporters = "Kazakhstan", 
                partners = "All", 
                trade_direction = "exports",
                freq = "annual", 
                start_date = 2016, 
                end_date = 2020)

# plot
ggplot(df[df$partner == "World", ], aes(x = period, y = trade_value_usd/10^6, group = 1)) + 
    geom_line()

One limitation of the UN Comtrade API is that one can download no more than five consecutive years or months of data in a single request if either the reporters or partners argument includes "all" (see here for more details). But it’s not all that hopeless. If you need to download the data for more than five years (or all the available years) at once, then you will need to apply the restrictions of up to five elements to either the reporters or partners arguments (i.e., they should be a character vector of country names of length five or fewer). For instance, this query will work and download data from 1995 (earliest available) to 2020 for a single reporting country and five of its trading partners:

df <- ct_search(reporters = "Kazakhstan", 
                partners = c("Russian Federation", "Ukraine", "Belarus", "Uzbekistan", "Armenia"), 
                trade_direction = "exports",
                freq = "annual")

On the other hand, this query will return an error, because the partners argument includes a character vector of length greater than five):

df <- ct_search(reporters = "Kazakhstan", 
                partners = c("Russian Federation", "Ukraine", "Belarus", "Uzbekistan", "Armenia", "Azerbaijan"), 
                trade_direction = "exports",
                freq = "annual")

Making a request with multiple reporters and partners works well too:

df <- ct_search(reporters = c("Kazakhstan", "Uzbekistan"), 
                partners = c("Russian Federation", "Ukraine", "Belarus", "Armenia"), 
                trade_direction = "exports",
                freq = "annual")

tradestatistics

The tradestatistics package is a good alternative to comtrader for accessing the UN Comtrade data. The package’s author makes the same UN Comtrade data available through their own infrastructure with the added benefits of the data being provided in the “tidy” format, with additional processing, extra utility functions and fewer usage limitations on the API. ots_create_tidy_data is the main function for downloading the data. The function does not have an option for indicating the direction of trade and it downloads both export and import data.

# load the package
library(tradestatistics)

# download the data
df <- ots_create_tidy_data(years = 1995:2019, 
                           reporters = "kaz", 
                           partners = c("rus", "ukr", "blr", "uzb", "arm"), 
                           commodities = "all", 
                           table = "yrpc")

It’s possible to query data for multiple reporter countries at the same time:

# data at aggregated level (year - reporter - partner)
df <- ots_create_tidy_data(years = 2018:2019, 
                           reporters = c("kaz", "uzb"),
                           partners = c("rus", "ukr", "blr", "arm"),
                           commodities = "all", 
                           table = "yrp")

head(df)

The package provides access to a number of tables through the API. You can list them by calling the ots_tables object in the console.

# look up the li
ots_tables

Eurostat

There is a very convenient eurostat package for downloading data from Eurostat. You can use the get_eurostat_toc function for looking up the list of all the datasets and the search_eurostat function for searching datasets based on a search term.

# load the package
library("eurostat")

# get the list of all eurostat datasets
df <- get_eurostat_toc()
head(df)

Searching for relevant datasets using a search term:

# search for relevant datasets
res <- search_eurostat("exports", type = "table")
head(res)

After locating a necessary dataset, you will need its ID to download the data using the get_eurostat function:

# download the data
df <- get_eurostat(id = "ei_bsco_m",
                   time_format = "date",
                   filters = "none",
                   type = "code",
                   select_time = NULL,
                   cache = TRUE,
                   update_cache = FALSE,
                   cache_dir = NULL,
                   compress_file = TRUE,
                   stringsAsFactors = FALSE,
                   keepFlags = FALSE)

The above request downloads the whole dataset. You can use the filters argument to filter the dataset down to a smaller set:

# from the dataset on business/consumer surveys, download the consumer confidence indicator (indic = "BS-CSMI"), seasonally adjusted (s_adj = "SA") and only for Austria (geo = "ATQ")
df <- get_eurostat(id = "ei_bsco_m",
                   time_format = "date",
                   filters = list(indic = "BS-CSMCI",
                                  s_adj = "SA",
                                  geo = "AT"))

# plot
df <- df[!is.na(df$values), ]
ggplot(df, aes(x = time, y = values, group = 1)) + geom_line()

One very useful function in this package is label_eurostat. The function allows to quickly download the labels associated with the data table (i.e., the one that was initially downloaded using get_eurostat():

# get the data labels
labeled_df <- label_eurostat(df)
head(labeled_df)

The package also provides a useful function for downloading the geospatial data from Eurostat (get_eurostat_geospatial):

# get the data on the NEET (neither in employment nor education and training) rate by NUTS 2 regions
df <- get_eurostat(id = "edat_lfse_22", time_format = "raw") %>%
    dplyr::filter(sex == "T", 
                  age == "Y15-24", 
                  time == 2020, 
                  nchar(geo) == 4) %>%
    dplyr::mutate(cat = cut_to_classes(values, 
                                       style = "equal"))

# download the geospatial data
data_geo <- get_eurostat_geospatial(resolution = "20", 
                                    nuts_level = "2", 
                                    year = 2021)

# merge
data <- dplyr::inner_join(data_geo, df)
# plot
ggplot(data = data) +
  geom_sf(aes(fill = cat), color = "dim grey", size = 0.1) +
  scale_fill_brewer(palette = "Blues") +
  guides(fill = guide_legend(reverse = TRUE, title = "")) +
  labs(title = "Young people (age 15-24) neither in employment nor in\n education and training (NEET), 2020, percent",
       caption = "Source: Eurostat.") +
  theme_minimal() +
  theme(legend.position = c(.87, .75), 
        title = element_text(size = 9), 
        legend.text = element_text(size = 8), 
        legend.background = element_blank(),
        plot.caption = element_text(hjust = 0, size = 9)) + 
  coord_sf(xlim = c(-10, 45), ylim = c(35, 70))

ECB

There is an ecb package for downloading data from the ECB Statistical Data Warehouse. The list of available datasets can be downloaded using the get_dataflows function but the retrieved data frame is quite high level and identification of specific data series will require going through the ECB’s data website itself. The package doesn’t have a function for looking up and searching for specific series (which looks to be the result of such functionality missing in the API) itself.

# load the package
library(ecb)

# retrieve the list of datasets (referred to as "data flows")
head(get_dataflows())

Data series can be downloaded using the get_data function (I looked up the series key or ID on this page):

# download the data
df <- get_data(key = "CISS.D.U2.Z0Z.4F.EC.SS_CI.IDX", 
               filter = list(startPeriod = "2000-01-01", 
                             endPeriod = "2020-12-31", 
                             # updateAfter = "2000-01-01", 
                             # firstNObservations = 12, 
                             # lastNObservations = 12, 
                             detail = "full"))

# plot
ggplot(df, aes(x = as.Date(obstime), y = obsvalue)) + 
    geom_line() + 
    labs(title = "Composite Indicator of Systemic Streess, Euro area, index", 
         x = NULL, 
         y = NULL)

FRED

There are a few R packages for downloading data from FRED (e.g., fredr, FredR, alfred). I will briefly review only fredr, which appears to be quite comprehensive and it is still maintained (at the time of writing). To use the FRED API, you need to obtain the FRED API key that allows to access the database. fredr is the main function for downloading the data series.

# load the package
library(fredr)

# set the API key (the alternative is to add "FRED_API_KEY=you_api_key_here" to your .Renviron file)
# fredr_set_key("your_api_key_here")

# download the total US non-farm employment series
df <- fredr(series_id = "PAYEMS",
            observation_start = as.Date("1990-01-01"),
            observation_end = as.Date("2022-01-01"))

# plot
ggplot(df, aes(x = date, y = value)) + 
    geom_line() + 
    labs(title = "Employees on non-farm payrolls (thousand)")

The fredr (and the FRED API) allows downloading only one series at a time. One option for dealing with this limitation is to apply the fredr function to a vector of data series names:

# load the `purrr` package to access the `map_*` family of functions
library(purrr)

# vector of data series names
data_series <- c("IPDCONGD", "IPNCONGD")

# download the data using the `map_dfr` function
df <- map_dfr(data_series, 
              fredr, 
              observation_start = as.Date("1990-01-01"),
              observation_end = as.Date("2022-01-01"))

# plot
ggplot(df, aes(x = date, y = value, color = series_id)) + 
    geom_line() + 
    scale_color_manual(labels = c("Industrial production: durable consumer goods", 
                                  "Industrial production: non-durable consumer goods"), 
                       values = c("blue", "red")) +
    labs(title = "Industrial production",
         subtitle = "Durable and non-durable consumer goods, 2017=100") +
    theme(legend.position = "bottom", 
          legend.title = element_blank())

The package also provides a convenient function for searching through all the FRED series.

df <- fredr_series_search_text(search_text = "consumer price index", 
                               filter_variable = "seasonal_adjustment", 
                               order_by = "popularity", 
                               limit = 5)

# have a look
head(df)

What if there is no an R package?

Sometimes an API for accessing the data is available but there is no an easy to use R (API wrapper) package for taking advantage of the API quickly. In this case, the solution may be to use a number of utility packages for making web requests through R. In the brief example below, I make a request to the API of https://exchangeratesapi.io/, a service providing exchange rates data. This example follows the structure of the API of https://exchangeratesapi.io/ and details of a request to other services may differ.

# load the packages
library(httr)           # tools for working with HTTP
library(jsonlite)       # JSON parser

# identify the base URL 
base_url <- "http://api.exchangeratesapi.io/v1/"

# your access key
access_key <- "you_access_key"

# additional directory information (necessary for exchangeratesapi.io)
path <- "latest"

# build the URL for making a request (syntax of the API for other services will very likely differ)
api_url <- paste(base_url, path, "?", "access_key=", access_key, sep = "")

# make the GET request and check the status of the request
df1 <- httr::GET(url = api_url)
httr::http_status(df1)

# the alternative is to directly download and parse the json file
df2 <- jsonlite::fromJSON(txt = api_url)
df2$success

# create a data frame with the exchange rates from df1

# what is the base currency
df2$base

# convert the downloaded data (in 'raw' format) to characters (i.e., JSON) and parse the JSON
rates <- fromJSON(rawToChar(df1$content))

# simplify the lists of lists structure
rates <- as.data.frame(sapply(rates$rates, `[`))
head(rates)

$category
[1] "Success"

$reason
[1] "OK"

$message
[1] "Success: (200) OK"

[1] TRUE
[1] "EUR"