--- title: "Working with the survey catalogue" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Working with the survey catalogue} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r, include = FALSE} knitr::opts_chunk$set(collapse = TRUE, comment = "#>") ``` ```{r setup} library(knbsdata) ``` This vignette covers finding a survey, reading its documentation and understanding what you are and are not allowed to do with it. Every code chunk that contacts the archive is shown but not run, so that building this vignette never depends on a live service. ## What is supported `knbs_sources()` reports what the package knows about each source and how far it has been verified. Nothing here makes a request: ```{r} knbs_sources(operation = "search_surveys") ``` The survey catalogue is the Kenya National Data Archive, which exposes a JSON API. It is registered as `experimental`: the endpoints are verified and covered by tests, but the archive does not document the API, so it could change without notice. If it does, the package raises a `knbs_error_schema_changed` rather than returning something plausible. ## Finding a survey ```{r, eval = FALSE} studies <- knbs_search_surveys("household budget") studies ``` You can filter by reference year, by producer and by access type: ```{r, eval = FALSE} knbs_search_surveys(year = c(2015, 2020)) knbs_search_surveys(access = "public_use", limit = 10) ``` Three columns come back as `NA` from a search: `acronym`, `geographic_coverage` and `unit_of_analysis`. They live in the study record rather than the catalogue index, and filling them in would mean one extra request per row. `knbs_survey_metadata()` retrieves them. ## Reading a study's documentation ```{r, eval = FALSE} meta <- knbs_survey_metadata("KEN-KNBS-KIHBS-2015-2016-v01") meta ``` The printed record shows the title, reference period, producer, geographic coverage, unit of analysis and access type. The full descriptive sections are in `meta$sections`; `include` narrows them: ```{r, eval = FALSE} knbs_survey_metadata("KEN-KNBS-KIHBS-2015-2016-v01", include = c("scope", "access")) ``` These records are DDI documents, in which sampling, data collection and processing all live in a single `method` section, so asking for any one of those returns that section. You do not have to know the exact identifier. Give it something recognisable and it will search the catalogue; if more than one study matches you get an error listing the candidates rather than a silently chosen "best" match. ## Searching variables Variable metadata are available without downloading any microdata: ```{r, eval = FALSE} knbs_variables("KEN-KNBS-KIHBS-2015-2016-v01", search = "expenditure") knbs_variables("KEN-KNBS-KIHBS-2015-2016-v01", file = "HH_Information") ``` `search` matches variable names and labels, and matches nothing else. If a concept you expect is missing, it is worth listing all the variables and looking at the labels the study actually uses before concluding the search is broken. Question text, universe, concept and value labels each cost one request per variable, so they are empty unless you ask: ```{r, eval = FALSE} knbs_variables( "KEN-KNBS-KIHBS-2015-2016-v01", search = "county", details = TRUE ) ``` That is capped at 100 variables. Narrow the selection first rather than raising the cap. Some studies are catalogued with no variable documentation at all — including, at the time of writing, the most recent one. That returns zero rows, which is an answer rather than a failure. ## Files, and what you may do with them ```{r, eval = FALSE} knbs_survey_files("KEN-KNBS-KIHBS-2015-2016-v01") ``` Two columns matter most: - `access_type` describes the archive's terms. `public_use` means public-use files exist. It does **not** mean they can be downloaded without an account. - `downloadable` describes what this package will do. At present it is `FALSE` for every file from every supported source. This lists **data files only**. Questionnaires, reports and technical documents are not covered: the archive publishes no machine-readable listing of them, so retrieving them would mean parsing a web page, which this package does not do. Use the `catalogue_url` column from `knbs_search_surveys()` to open the study on the archive's own site, where the documentation is linked. ## Downloading microdata You cannot do this through `knbsdata`, and that is deliberate: ```{r, eval = FALSE} knbs_download_microdata("KEN-KNBS-KIHBS-2015-2016-v01") #> Error: Microdata from "kenada" cannot be downloaded programmatically. #> x Every download requires an account with the archive. #> i Register and request access at . #> i knbsdata does not automate sign-in, licence acceptance or approval workflows. ``` The archive puts every download behind a login. The package does not automate sign-in, licence acceptance or approval workflows, so the function exists purely to fail clearly and point you at the official process. Complete that process, download the files yourself, and read them with `knbs_read_microdata()`, which reads Stata, SPSS and Excel files and the data inside a `.zip`, keeping value labels. This matters for a subtler reason too. The gate is signalled by an HTTP `Refresh` header rather than a redirect status, so a client that only checks the status code sees `200 OK` and an empty body — and could report "no data" where the honest answer is "you need an account". The package treats that as an authentication error. ## Open microdata: the FinAccess surveys Some surveys' microdata are published for anyone to download. The FinAccess household survey, Kenya's survey of financial inclusion by the Central Bank of Kenya, KNBS and FSD Kenya, is one: six rounds from 2006 to 2024, plus two waves of the FinAccess MSE Tracker Survey of micro and small enterprises. The archive catalogues these studies (and a 2019 round the portal has no files for), but lists no data files or variables for any of them, and gates its downloads behind an account as it does everywhere. The FinAccess portal publishes the files themselves, so they have their own functions. ```{r, eval = FALSE} # The catalogue ships with the package: listing it contacts nothing knbs_finaccess_files(content = "data") # Download and read the 2024 household survey (Stata, 160 MB) fa <- knbs_read_microdata(knbs_download_finaccess("hh2024-stata", "data")) knbs_cite(fa) ``` The portal is a web page linking files on Google Drive, so the package keeps its own catalogue of them — names, sizes and SHA-256 checksums, every file opened and checked by hand — and holds each download to it: a file that has changed since is refused before anything large moves, and a transfer that arrives damaged is not kept. Running the same line again finds the file already downloaded, confirms its checksum and reads it without a request. Each file gets a sidecar recording where it came from and the producers' terms. The portal states no licence; FSD Kenya's terms for the 2013 survey restrict the data to research and forbid redistribution without written agreement, so assume nothing wider. `knbs_read_microdata()` keeps Stata and SPSS value labels as `haven::labelled()` columns, and each variable's question text in its `"label"` attribute. The 2024 file has 20,871 respondents and 3,816 variables, so `col_select` is worth knowing: ```{r, eval = FALSE} fa <- knbs_read_microdata("data/2024_Finaccess_Publicdata.dta", col_select = c("county", "A13", "A18")) attr(fa$A18, "label") # "A18. AGE OF RESPONDENT" table(haven::as_factor(fa$A13)) # value labels as factor levels # Respondents' counties as census codes, ready to join census tables fa$geo_code <- knbs_county_code(haven::as_factor(fa$county)) ``` ## Handling errors Every failure carries a specific condition class, so you can react to the particular problem rather than matching on message text: ```{r, eval = FALSE} result <- tryCatch( knbs_survey_metadata("Kenya"), knbs_error_ambiguous_resource = function(cnd) { message("Several studies match; using the first.") knbs_survey_metadata(cnd$matches[[1]]) }, knbs_error_resource_not_found = function(cnd) NULL, knbs_error_source_unavailable = function(cnd) NULL ) ``` The classes you are most likely to want are `knbs_error_ambiguous_resource` (which carries the candidates in `$matches`), `knbs_error_resource_not_found`, `knbs_error_authentication_required` (which carries `$access_url`), `knbs_error_source_unavailable` and `knbs_error_schema_changed`. All of them inherit from `knbs_error`, so you can catch that instead to handle any package failure at once. One practical note: catch `error`, not `condition`. httr2 signals a benign `httr2_perform` condition on every request, so a `tryCatch(..., condition = )` handler will catch that first and never see your error. ## Citing what you used ```{r, eval = FALSE} knbs_cite(meta) ``` Where the archive states a required citation, that text is used and the package adds the retrieval date and its own version. `knbs_metadata()` shows the full provenance record behind any object. ## Caching Catalogue responses are cached for 24 hours by default: ```{r, eval = FALSE} knbs_cache_info() knbs_search_surveys("household budget", refresh = TRUE) # bypass the cache knbs_cache_clear() ``` Cache entries are checksummed and keyed partly on the package's parser version, so upgrading the package invalidates anything a new parser would read differently.