dgp-server is the aiohttp backend for interactively previewing and publishing
dgp pipelines. A browser posts a dgpConfig,
which describes a source file, the taxonomy it maps to, and the mapping between
them. The server runs dgp's analyzers until the config is complete, sends the
completed config back, and streams sample rows from each processing stage over
Server-Sent Events. With publish.allowed: true, it also writes the result to a
SQL table named after the taxonomy.
It is consumed as a library by dgp-app.
dgp-app's server_dgp.py subclasses DgpServer and runs it on 127.0.0.1:5001
behind its Flask front end. Deployments such as datacity-importer (app.datacity.org.il)
build on the dgp-app image and supply their own taxonomies/.
browser ──POST /api/config──▶ dgp-server ──writes──▶ $BASE_PATH/<uid>/config.yaml
◀─────── {uid} ──────
──GET /api/events/<uid> (SSE)──▶ analyze loop (dgp) → c, e messages
flow: load → [0] → transform → [1] → enrich → [2] → publish → [3]
◀── r / n / d messages per stage, then "close"
| Doc | Covers |
|---|---|
| docs/architecture.md | routes, the /config → /events protocol, the multi-phase analyze loop, ResultsPoster stages 0–3, SSE message shapes, extension points |
| docs/taxonomies.md | the index.yaml format, column types, header mappings, taxonomy modules, _common_, how mappings and column types refresh from the DB (and dgp-app's overrides) |
| docs/dgp-config.md | every dgpConfig key, mapping entries, normalize (wide-to-long), constants, an annotated muni-budgets example, generating configs programmatically |
| docs/publishing.md | which publisher runs where, publish_flow, table naming, primary keys, _source and clear_by_source, append vs update mode |
| docs/releasing.md | packaging, VERSION, deploy_pip.sh, the dgp pin, how dgp-app picks up releases |
| docs/development.md | running locally, exercising the pipeline in-process, tests (none), known issues |
pip install -e ../dgp -e . aiopg 'sqlalchemy<2'
cd <dir containing taxonomies/index.yaml>
BASE_PATH=/tmp/dgp DATABASE_URL=postgresql://... python -m dgp_server.server # http://127.0.0.1:8000/apiSee docs/development.md for details.