---
title: "Getting started with knbsdata"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Getting started with knbsdata}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
```

```{r setup}
library(knbsdata)
```

`knbsdata` is a client for official statistics published by the Kenya National
Bureau of Statistics. It covers six source families: **survey and census
study discovery**, **publications** (statistical releases, reports and
workbooks, downloadable with verification), **census tables** as tidy
observations, **monthly consumer prices** from 2009 onwards, **the economy as
SDMX dataflows** (GDP, trade, county finance and 320 more), and **geographic
reference codes** for county-level joins.

Code that contacts the archive is shown but not run here, so building this
vignette never depends on a live service.

## What is and is not in this version

```{r}
knbs_sources()
```

Eight sources are verified and usable: `kenada`, the Kenya National Data
Archive (surveys); `finaccess`, the FinAccess portal (open survey
microdata); `knbs_publications`, the KNBS website's media library;
`knbs_indicators`, the KNBS census data service (2019 census tables,
CC BY 4.0); `knbs_cpi`, the CPI Explorer (monthly consumer prices from
2009); `knbs_opendata`, the Data Portal (323 SDMX dataflows: GDP, trade,
county finance and more); `opendataforafrica`, the Knoema-operated
republisher that fills gaps no first-party portal serves as data — Gross
County Product above all; and `dhs_program`, the DHS Program's API for the
Kenya Demographic and Health Survey indicators, by county from 2014. A
source the audit has not verified stays at
stability `discovery` and is never contacted — there is nothing there to
call. The `source-status` vignette says exactly what is and is not covered,
and why.

## Five ideas worth knowing

### 1. Everything carries its provenance

Objects returned by the package know where they came from.

```{r, eval = FALSE}
meta <- knbs_survey_metadata("KEN-KNBS-KIHBS-2015-2016-v01")
knbs_metadata(meta)
```

That record holds the source, the retrieval timestamp, the release date, the
access conditions and the package version — everything needed to say later what
exactly was used.

### 2. Citations come from the data, not from you

```{r, eval = FALSE}
knbs_cite(meta)
knbs_cite(meta, style = "bibtex")
```

Where the archive states a required citation, that text is used verbatim and the
package adds only the retrieval information. It cannot cite a release the object
does not have.

### 3. Access conditions are data, not decoration

Two different questions, two different columns:

- `access_type` is what the **archive** permits. `public_use` means public-use
  files exist.
- `downloadable` is what **this package** will do. For survey microdata it is
  `FALSE` throughout; publications, by contrast, are served openly and can be
  downloaded.

Those differ because the archive requires an account for every download. The
package does not automate sign-in, licence acceptance or approval workflows, so
`knbs_download_microdata()` explains the official route and stops. That is a
deliberate boundary, not a missing feature.

### 4. Failure is explicit

Every failure carries a condition class, and none of them return partial data
dressed up as a result:

```{r, eval = FALSE}
tryCatch(
  knbs_survey_metadata("Kenya"),
  knbs_error_ambiguous_resource = function(cnd) cnd$matches
)
```

Catch `error`, or a specific `knbs_error_*` class — not `condition`, because
httr2 signals a benign condition on every request.

### 5. Everything is a tibble that speaks tidyverse

Every data-carrying result is a tidy tibble — one row per observation, built
on vctrs — so dplyr verbs work directly and keep the provenance attached:
filter, mutate, arrange, join, and `knbs_metadata()` still answers on the
result. Aggregations (`summarise`) and reshapes (`pivot_wider`) return plain
tibbles, because a derived table is your data, not the source's;
`tibble::as_tibble()` drops the wrapper whenever you want out.

```{r, eval = FALSE}
library(dplyr)
library(tidyr)

knbs_get_indicator("jhbzkjc") |>                    # Gross County Product
  filter(table_code == "T3", period == 2020) |>
  slice_max(value, n = 5)

knbs_abstract_table("2.2") |>                       # years across, like print
  select(industry, period, value) |>
  pivot_wider(names_from = period, values_from = value)
```

## A first session

Not sure where something lives? Search everything at once — indicators,
Statistical Abstract tables, surveys and publications — and each result
comes with the call that retrieves it:

```{r, eval = FALSE}
hits <- knbs_search("consumer price index")
hits[, c("resource_type", "resource_id", "title", "get")]
#>   resource_type resource_id   title                            get
#> 1 indicator     CPI_ALL_ITEMS Consumer Price Index, all items  knbs_get_indicator("CPI_ALL_ITEMS", source = "knbs_cpi")
#> 2 publication   17052         Kenya-Consumer-Price-Indices-…   knbs_download_publication("17052", destination = ".")
#> ...

eval(parse(text = hits$get[[1]]))   # or copy the call
```

Words match by stem ("index" finds "Indices"), and CPI, GDP, GCP, KDHS
and a few other acronyms are understood both ways. The first search of the
day lists the publications library, which takes a few seconds; after that
searches are instant.

Or go to one catalogue directly:

```{r, eval = FALSE}
# 1. Find something
studies <- knbs_search_surveys("household budget")

# 2. Read its record
meta <- knbs_survey_metadata(studies$survey_id[[1]])

# 3. See what variables it documents
knbs_variables(meta$resource_id, search = "expenditure")

# 4. See what files exist, and on what terms
knbs_survey_files(meta$resource_id)

# 5. Record what you used
knbs_cite(meta)
```

Publications work the same way, and these files *can* be downloaded — they are
served openly by the KNBS website:

```{r, eval = FALSE}
pubs <- knbs_publications("economic survey", year = 2026)
knbs_download_publication(pubs$publication_id[[1]], "data/publications")
```

The download is verified (a web page arriving where a document was expected is
refused), checksummed with SHA-256, and given a `<file>.metadata.json`
provenance sidecar.

Indicators come back as tidy observations — consumer prices, the national
accounts, Gross County Product, and census tables with geography down to
sub-county:

```{r, eval = FALSE}
cpi <- knbs_get_indicator("CPI_ALL_ITEMS", start = "2020-01")
gdp <- knbs_get_indicator("DF_NAT_GDPCP")            # GDP by activity, SDMX
knbs_get_indicator("DF_PF_CG_REV")                   # county govt revenue
gcp <- knbs_get_indicator("jhbzkjc")                 # Gross County Product

knbs_indicators(topic = "Population")
pop <- knbs_get_indicator("POPCEN0001", geo_level = "county")

# The matching reference codes, for census joins
knbs_geographies(level = "county")
```

## Configuration

Options are ordinary R options under the `knbsdata.` prefix:

```{r}
knbs_options()
```

```{r, eval = FALSE}
knbs_options(timeout = 60, retries = 5)
```

## Caching

Catalogue responses are cached for 24 hours, keyed on the source, the operation,
the normalised query and the package's parser version — so upgrading the package
invalidates anything a new parser would read differently.

```{r, eval = FALSE}
knbs_cache_info()
knbs_search_surveys("household budget", refresh = TRUE)  # ignore the cache
knbs_cache_clear()
```

Cached values are checksummed. An entry that fails its checksum is discarded and
refetched rather than returned.

## Where to next

- `vignette("survey-catalogue")` — the survey workflow in depth
- `vignette("census-indicators")` — indicators: census tables and consumer prices
- `vignette("county-data")` — county codes, vintages and safe joins
- `vignette("source-status")` — what each source supports, and the limitations
  of this release