fielddash

Documentation: https://lambdageo.github.io/fielddash/

Schema-driven dashboards for field data collection (Epicollect5), built directly from the form schema. Every question is identified by Epicollect's stable ref, and its input type (radio, checkbox, integer, location, ...) automatically dictates the filter, chart, and map representation. Modifying, adding, or reordering questions in the form does not break the dashboard, and setting up a new fieldwork survey requires only a YAML configuration file.

Ready-to-use pages:

• Overview: summary indicators, map, submissions over time, collectors.
• Highlights: curated charts configured in your YAML.
• Questions: all questions grouped by section, with search, text-field toggle, and dynamic cross-tabulation.
• Data: searchable table with CSV and Excel export.

Installation

Install it in an isolated environment rather than globally.

One environment per project (recommended). Works everywhere and is required for deployment and custom pages:

mkdir my-survey && cd my-survey
python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install fielddash
fielddash init .

Nothing installed first. pipx or uv can run init once; then create the environment from the generated requirements.txt:

pipx run fielddash init my-survey       # or: uvx fielddash init my-survey
cd my-survey
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

Command line only. If you never edit code in the project, pipx install fielddash (or uv tool install fielddash) is enough for init, fields and run. The streamlit_app.py generated by --deploy imports fielddash, so it needs the per-project environment above.

For development: git clone https://github.com/LambdaGeo/fielddash && cd fielddash && pip install -e ".[dev]".

Quickstart

The example project lives in the repository; it is not part of the PyPI package.

git clone https://github.com/LambdaGeo/fielddash && cd fielddash
python -m venv .venv && source .venv/bin/activate
pip install -e .
fielddash run examples/waste/project.yaml

The example uses anonymized household survey data on solid waste management (Itaqui-Bacanga, São Luís – MA) and showcases a custom project page (Recycling).

Setting Up a New Field Project

1. Scaffold the project folder:

   fielddash init my-survey            # project.yaml, requirements.txt, .env.example, .gitignore
   fielddash init my-survey --deploy   # + streamlit_app.py, secrets example
   fielddash init my-survey --source json   # offline project reading data/ (git-ignored)

   Then cp .env.example .env and fill in the project slug and the Epicollect app credentials (generated under Apps in your project's administration area):

   PROJECT_MY_SURVEY=project-slug
   MY_SURVEY_CLIENT_ID=...
   MY_SURVEY_CLIENT_SECRET=...

   Public projects do not require credentials: simply omit credentials in the config.

2. The generated project.yaml starts minimal (or write it by hand):

   title: "My Field Survey"
   source:
     type: epicollect
     project: ${PROJECT_MY_SURVEY}    # slug, or the literal value
     credentials: MY_SURVEY

3. Inspect the form fields to select aliases, filters, and highlights:

   fielddash fields project.yaml

4. Complete the configuration (all keys below are optional) and run fielddash run project.yaml:

   subtitle: "Research team, institution..."
   fields:            # alias -> ref suffix, column, or question label
     neighborhood: "5401ce"
     waste_dest: "48c35e"
   types:             # force type (e.g. treat free text as category)
     neighborhood: category
   ignore: [created_by, "3401cb"]   # hide fields (e.g. collector email)
   filters: [created_at, neighborhood]
   highlights:
     - {field: waste_dest, by: neighborhood, title: "Waste destination by neighborhood"}
   sections:          # tabs on the Questions page (default: form groups)
     - {title: "Profile", fields: [age, gender]}
   map: {field: location, popup: [neighborhood]}
   extensions: [pages]              # .py files or directories with custom pages
   timezone: America/Fortaleza
   cache_minutes: 5

   (Portuguese keys from early versions — titulo, fonte, campos, ... — still load but are deprecated and emit a warning.)

Running fielddash run folder/ will scan all .yaml files in the directory and present a project selector in the sidebar. To work offline or test without internet access, use source: {type: json, data: ..., schema: ...} (as shown in examples/waste/project.yaml).

Using in a Custom Streamlit Script / Deployment

The dashboard can also be invoked as a Python function inside your own Streamlit app:

# streamlit_app.py
import fielddash

fielddash.dashboard("projects/")          # or "projects/waste.yaml"

Relative paths are resolved relative to the current directory or the script folder. Run with:

streamlit run streamlit_app.py

Streamlit Community Cloud

1. Create the project with the deploy files and try it locally:

   mkdir my-survey && cd my-survey
   python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
   pip install fielddash
   fielddash init . --deploy
   cp .env.example .env          # fill in, then edit project.yaml
   streamlit run streamlit_app.py

2. Push the folder to a GitHub repository. .env and .streamlit/secrets.toml are git-ignored: credentials never go to the repo.
3. On share.streamlit.io choose Create app, pick the repository and branch, and set streamlit_app.py as the main file.
4. Under Advanced settings → Secrets (later: Settings → Secrets) paste the content of .streamlit/secrets.toml.example, filled in:

   FIELD_ACCESS_PIN = "choose-a-code"
   PROJECT_MY_SURVEY = "project-slug"
   MY_SURVEY_CLIENT_ID = "..."
   MY_SURVEY_CLIENT_SECRET = "..."

5. Deploy, then share the app link, or https://<your-app>.streamlit.app/?token=<code> to skip the login screen.

Notes:

• Use source: {type: epicollect} for the cloud. A type: json project reads data/, which is git-ignored on purpose (it may hold personal data), so it only works locally.
• Without FIELD_ACCESS_PIN the app is public: anyone with the link sees the data.

fielddash checks environment variables and .env first, then falls back to st.secrets (also resolving ${VAR} references in YAML). The data cache is shared among server sessions, so Epicollect receives at most one request per project every cache_minutes.

Restricting access

A deployed Streamlit app is public by default. Add fielddash.require_access() before fielddash.dashboard(...) (fielddash init --deploy already does) and set FIELD_ACCESS_PIN in the secrets: visitors then see a login screen, or can use a link ending in ?token=<code>. Without the variable the call does nothing.

Self-Hosted Server

fielddash run projects/ --server.port 8501 --server.headless true

(e.g., as a systemd service behind an Nginx reverse proxy with HTTPS).

Custom Pages

A project-specific page is a standard .py file listed in extensions:

from fielddash.core.registry import page
from fielddash.ui.charts import render_field


@page("Recycling", order=40)
def render(ctx):
    ds = ctx.dataset
    render_field(ds, ctx.df, ds.field("recycle_habit"), by=ds.field("neighborhood"))

• ctx.df provides the data with all active sidebar filters applied.
• ds.field("alias") resolves the field (column, type, options) dynamically by its stable schema ref.
• Additional data sources (e.g. KoboToolbox, CSV) can be registered with @source("type").

Project Structure

fielddash/
  cli.py            CLI commands `fielddash init`, `run` and `fields`
  scaffold.py       templates and writer behind `fielddash init`
  access.py         `fielddash.require_access()`: optional PIN/token gate
  web.py            `fielddash.dashboard()`: config, cache, filters, navigation
  app.py            Streamlit entry point used by `fielddash run`
  core/schema.py    Epicollect schema parser -> list[Field], column naming, reconciliation
  core/normalize.py raw entries -> typed series (ordered categories, lists, coords, datetimes)
  core/config.py    YAML config parser (+ ${VAR} resolution from .env and secrets)
  sources/          data source plugins (@source): epicollect (API, pagination), json
  ui/               dynamic filters, charts, and maps based on field types
  views/            default pages (@page): Overview, Highlights, Questions, Data
examples/           example projects with anonymized datasets
tests/              pytest test suite (schema, index shifting, AppTest integration)
docs/               MkDocs site (`mkdocs serve`); docs/history/ holds the original development plan

Running Tests

pytest

Building the Documentation

pip install -e ".[docs]"
mkdocs serve            # http://127.0.0.1:8000
mkdocs build --strict   # what CI runs

Releasing (maintainers)

1. Bump the version in pyproject.toml, fielddash/__init__.py and CITATION.cff.
2. Publish a GitHub Release: .github/workflows/publish.yml builds the package and uploads it to PyPI through Trusted Publishing (no token stored).

One-time setup on pypi.org → Publishing → Add a new pending publisher: project fielddash, owner LambdaGeo, repository fielddash, workflow publish.yml. PyPI never lets a published version be overwritten, so check with python -m build && twine check dist/* (and optionally TestPyPI) first.

Authors

• Sergio Costa (@LambdaGeo) — Universidade Federal do Maranhão (UFMA)
• André Moura Lima (@AndreMouraL) — Universidade Federal do Maranhão (UFMA)
• Ricardo Luvizotto Santos — Universidade Federal do Maranhão (UFMA)
• Marianna Basso Jorge — Universidade Federal do Maranhão (UFMA)

License

MIT — see LICENSE.