ABCD Dictionary Search

Semantic search over the ABCD Study data dictionary. Type a phrase like "screen time on weekends" or "BMI" — the app returns the variables in the dictionary whose labels mean roughly the same thing, ranked by cosine similarity.

Live demo →
Source on GitHub →

What it is

A small R Shiny app with a Python search backend, bridged by reticulate. It is designed to deploy on the shinyapps.io free tier under tight resource limits (1 GB bundle, 1 GB RAM, short startup window). To keep the bundle and startup time small, the heavy lifting happens offline at build time, not at runtime:

  • Corpus embeddings are pre-computed once with MiniLM-L6-v2, L2-normalized, and saved as float16 NumPy arrays.
  • The MiniLM model is quantized to ONNX int8 (~23 MB) so it runs on plain onnxruntime with no torch.
  • The dictionary table is stored as Parquet (smaller and faster than the source CSV) and read by nanoparquet in R.

At runtime, each search is a tokenize → ONNX inference → single matmul → top-K sort. Typical query latency is well under 100 ms.

Current build configuration

config.yml at the repo root is the single source of truth for the build pipeline (setup.sh, python/build_embeddings.py) and the R app (app.R). Switching ABCD releases or embedding models is a one-line edit followed by ./setup.sh. The values below are read directly from the live file:

# config.yml — single source of truth for setup.sh and python/build_embeddings.py.
#
# Edit this file to point the build pipeline at a different dictionary release,
# embed from a different column, or swap the embedding model. Both setup.sh
# and python/build_embeddings.py read these values at run-time.

# Python toolchain --------------------------------------------------------
python_version: "3.12"       # required interpreter version (X.Y)
venv_dir: python_env         # virtualenv directory (relative to repo root)

# Embedding model ---------------------------------------------------------
model:
  repo_id: sentence-transformers/all-MiniLM-L6-v2
  onnx_hub_path: onnx/model_quint8_avx2.onnx   # AVX2 is universal on Linux x86_64
  max_seq_len: 128
  batch_size: 64

# Dictionary --------------------------------------------------------------
# The parquet filename is the single source of truth. Its stem (filename
# without the .parquet suffix) is reused to locate the source CSVs under
# data/:
#   <stem>.csv                  -> converted to <stem>.parquet for the R UI
#   <stem>_minimal.csv          -> embeddings_imag.npy   + metadata_imag.npz
#   <stem>_minimal_noimag.csv   -> embeddings_noimag.npy + metadata_noimag.npz
# To point the pipeline at a new release, edit only `parquet:` below.
dictionary:
  parquet: dd-abcd-7_0.parquet

  # Column whose text content is encoded into embeddings.
  text_column: label
  # Column kept alongside each embedding as metadata.
  metadata_column: domain

app.R reads the same file at startup and refuses to launch if data/embeddings/manifest.txt (written by the build) disagrees with dictionary.parquet — the UI cannot silently display the wrong dictionary version.

Where to go next

  • Using the app — The guided tour, search, filters, row details, CSV export, copying variable names, and the NBDCtools handoff.
  • How it works — Architecture diagram, file layout, and a walkthrough of a single search from keystroke to results table.
  • Local development — setup.sh, run.sh, and config.yml — what they do and how to extend them when source data or dependencies change.
  • Deployment — deploy.sh walkthrough, what the shinyapps.io manifest does, and how Python is provisioned on the server.
  • Troubleshooting — Real failures we hit during deployment and how we fixed them — useful reading if you're adapting this to your own reticulate-based app.