diff --git a/README.md b/README.md index f3e78ff..bf4a077 100644 --- a/README.md +++ b/README.md @@ -2,6 +2,9 @@ [中文说明](README.zh-CN.md) +[![Public source checks](https://github.com/fooklook/pgsync-studio/actions/workflows/ci.yml/badge.svg)](https://github.com/fooklook/pgsync-studio/actions/workflows/ci.yml) +[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE) + A web console and document assembly engine for synchronizing PostgreSQL data into Elasticsearch. Configure connections, join related tables across databases, build a full baseline, and keep the resulting documents up to date. @@ -26,6 +29,14 @@ The current Web UI is in Simplified Chinese; the public setup documentation is b Studio connections support PostgreSQL. The upstream command-line code is retained; its wider backend support is not a compatibility claim for the Studio interface. +## A quick look + +![Studio task list with the stopped synthetic book-catalog job](docs/images/demo-jobs.jpg) + +The UI currently uses Simplified Chinese. This screenshot shows a stopped, +synthetic configuration preview. Follow the [book-catalog walkthrough](docs/demo.md) +to run the real baseline and cross-database incremental checks with Compose. + ## Quick start Install Docker with Compose v2, then run these commands from the repository root: @@ -117,3 +128,14 @@ consumer watermarks are also checked. Maintenance and replica routing are opt-in The repository contains no environment backups, production task definitions, internal deployment pipelines, or private repository history. + +## Feedback and upstream + +Use [Studio issues](https://github.com/fooklook/pgsync-studio/issues) for this +extension's setup, UI, document assembly, and maintenance behavior. Include a +minimal schema with synthetic data and sanitized logs; see [CONTRIBUTING.md](CONTRIBUTING.md). + +PGSync Studio credits [PGSync](https://github.com/toluaina/pgsync) for its engine +foundation. The Studio-specific runners and operating model are independently +maintained. Changes proposed upstream are reviewed by the upstream maintainers; +see [UPSTREAM.md](UPSTREAM.md) for the relationship and compatibility boundaries. diff --git a/README.zh-CN.md b/README.zh-CN.md index d3acf5c..f712284 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -2,6 +2,9 @@ [English](README.md) +[![自动检查](https://github.com/fooklook/pgsync-studio/actions/workflows/ci.yml/badge.svg)](https://github.com/fooklook/pgsync-studio/actions/workflows/ci.yml) +[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE) + PGSync Studio 是基于 [PGSync](https://github.com/toluaina/pgsync) 扩展的 PostgreSQL → Elasticsearch 同步平台,提供 Web 配置台、跨库文档组装、 存量与增量同步、故障恢复和运行诊断。 @@ -20,6 +23,13 @@ PostgreSQL → Elasticsearch 同步平台,提供 Web 配置台、跨库文档 Studio 数据源目前支持 PostgreSQL;保留的上游 CLI 与 Studio 的能力边界不同。 +## 界面预览 + +![使用虚构图书配置的任务列表,任务处于停止状态](docs/images/demo-jobs.jpg) + +截图展示独立的虚构配置预览,任务处于停止状态。 +实际存量和跨库增量同步的体验步骤见[图书示例演示](docs/demo.md)(英文),也可按下方命令启动 Compose。 + ## 本地体验 准备 Docker 和 Compose v2。在本目录执行: @@ -77,3 +87,13 @@ python tools/check_public_export.py 上述单元测试无需真实数据库;其余上游集成测试需要独立测试服务, 参见 [贡献指南](CONTRIBUTING.md)。 + +## 反馈与上游协作 + +本扩展的安装、界面、文档组装和维护问题,请提交到 +[Studio Issues](https://github.com/fooklook/pgsync-studio/issues)。 +建议附上虚构的最小表结构、数据和脱敏日志,便于复现。 + +Studio 的引擎基础来自 [PGSync](https://github.com/toluaina/pgsync), +Studio 扩展的运行器和运维机制由本项目独立维护。 +上游贡献由上游维护者审核;关系和兼容边界见 [UPSTREAM.md](UPSTREAM.md)。 diff --git a/docs/demo.md b/docs/demo.md new file mode 100644 index 0000000..db2321c --- /dev/null +++ b/docs/demo.md @@ -0,0 +1,101 @@ +# Book-catalog walkthrough + +This disposable Compose example joins PostgreSQL rows into Elasticsearch +documents. It uses synthetic books, authors, and view counters; the same example +runs in the repository's [end-to-end CI](https://github.com/fooklook/pgsync-studio/actions/workflows/ci.yml). + +## What it demonstrates + +The `book-catalog-demo` job reads `books` and `authors` from `catalog`, and +`book_metrics` from the separate `analytics` database. Both databases live in the +demo PostgreSQL container. The Elasticsearch index is `book_catalog_v1`. + +```mermaid +flowchart LR + Books["catalog.public.books"] --> Document["book_catalog_v1 document"] + Authors["catalog.public.authors"] -->|"books.author_id = authors.id"| Document + Metrics["analytics.public.book_metrics"] -->|"books.id = book_metrics.book_id"| Document +``` + +The job first builds a baseline, then uses native change listeners and durable +outbox events to rebuild affected root documents. The automatic check changes a +counter in the second database and a title in the root table, then waits for the +corresponding Elasticsearch values and checks that runner/listener threads are alive. + +## Start the example + +From the repository root, with Docker and Compose v2 installed: + +```sh +cp .env.example .env +# Set DEMO_POSTGRES_PASSWORD in .env before the first start. +docker compose up --build -d --wait +docker compose exec studio python examples/studio/bootstrap_demo.py +``` + +In PowerShell, replace the first command with `Copy-Item .env.example .env`. +Open and select **任务列表** (task list). +Use **加载** (load) to inspect a job's tables and relationships. The current UI +is in Simplified Chinese; the source and setup documentation are available in English. + +![Task configuration using the synthetic book-catalog schema](images/demo-config.jpg) + +This screenshot was captured from a local configuration preview using the +checked-in demo schema. Its task is stopped and it does not show live +synchronization results or performance measurements. + +## Verify synchronization + +Run the same check used by CI: + +```sh +docker compose exec studio python tools/smoke_demo.py +``` + +Expected checkpoints: + +```text +PASS baseline assembly +PASS cross-database metric update +PASS root-table update +PASS actual runner and listener threads +``` + +The script updates only the synthetic Compose example. It increments the first +book's view counter and changes its title, so these values can differ on later runs. +It rejects jobs whose sources do not match the example's container and databases. + +For a manual update: + +```sh +docker compose exec postgres psql -U studio -d analytics -c "UPDATE book_metrics SET views = views + 1, updated_at = now() WHERE book_id = 1;" +``` + +In **ES 自定义查询** (custom ES query), select `book_catalog_v1` and submit: + +```json +{"query": {"ids": {"values": ["1"]}}, "size": 1} +``` + +The returned document should contain the changed `book_metrics.views` value. +Synchronization is eventually consistent; this example is a functional check, +not a latency or throughput benchmark. + +## Stop and explore further + +```sh +docker compose down +``` + +Named volumes preserve this example's data and configuration. The SQL initializer +runs only on the first creation of the PostgreSQL data volume. Changing `.env` +later does not change an existing PostgreSQL role's password. + +Replica reads, cleanup, and partition maintenance are disabled in this example. +They require an appropriate topology and deliberate configuration; see +[deployment and configuration](deployment.md). Protect administrative access +before using a shared environment; see [SECURITY.md](../SECURITY.md). + +For Studio feedback, use [the project's issue tracker](https://github.com/fooklook/pgsync-studio/issues) +with a minimal synthetic reproduction. See [UPSTREAM.md](../UPSTREAM.md) for how +this independent extension relates to PGSync. diff --git a/docs/images/demo-config.jpg b/docs/images/demo-config.jpg new file mode 100644 index 0000000..2f92207 Binary files /dev/null and b/docs/images/demo-config.jpg differ diff --git a/docs/images/demo-jobs.jpg b/docs/images/demo-jobs.jpg new file mode 100644 index 0000000..c64a508 Binary files /dev/null and b/docs/images/demo-jobs.jpg differ