Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@

[中文说明](README.zh-CN.md)

[![Public source checks](https://github.com/fooklook/pgsync-studio/actions/workflows/ci.yml/badge.svg)](https://github.com/fooklook/pgsync-studio/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)

A web console and document assembly engine for synchronizing PostgreSQL data
into Elasticsearch. Configure connections, join related tables across databases,
build a full baseline, and keep the resulting documents up to date.
Expand All @@ -26,6 +29,14 @@ The current Web UI is in Simplified Chinese; the public setup documentation is b
Studio connections support PostgreSQL. The upstream command-line code is retained;
its wider backend support is not a compatibility claim for the Studio interface.

## A quick look

![Studio task list with the stopped synthetic book-catalog job](docs/images/demo-jobs.jpg)

The UI currently uses Simplified Chinese. This screenshot shows a stopped,
synthetic configuration preview. Follow the [book-catalog walkthrough](docs/demo.md)
to run the real baseline and cross-database incremental checks with Compose.

## Quick start

Install Docker with Compose v2, then run these commands from the repository root:
Expand Down Expand Up @@ -117,3 +128,14 @@ consumer watermarks are also checked. Maintenance and replica routing are opt-in

The repository contains no environment backups, production task definitions,
internal deployment pipelines, or private repository history.

## Feedback and upstream

Use [Studio issues](https://github.com/fooklook/pgsync-studio/issues) for this
extension's setup, UI, document assembly, and maintenance behavior. Include a
minimal schema with synthetic data and sanitized logs; see [CONTRIBUTING.md](CONTRIBUTING.md).

PGSync Studio credits [PGSync](https://github.com/toluaina/pgsync) for its engine
foundation. The Studio-specific runners and operating model are independently
maintained. Changes proposed upstream are reviewed by the upstream maintainers;
see [UPSTREAM.md](UPSTREAM.md) for the relationship and compatibility boundaries.
20 changes: 20 additions & 0 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@

[English](README.md)

[![自动检查](https://github.com/fooklook/pgsync-studio/actions/workflows/ci.yml/badge.svg)](https://github.com/fooklook/pgsync-studio/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)

PGSync Studio 是基于 [PGSync](https://github.com/toluaina/pgsync) 扩展的
PostgreSQL → Elasticsearch 同步平台,提供 Web 配置台、跨库文档组装、
存量与增量同步、故障恢复和运行诊断。
Expand All @@ -20,6 +23,13 @@ PostgreSQL → Elasticsearch 同步平台,提供 Web 配置台、跨库文档

Studio 数据源目前支持 PostgreSQL;保留的上游 CLI 与 Studio 的能力边界不同。

## 界面预览

![使用虚构图书配置的任务列表,任务处于停止状态](docs/images/demo-jobs.jpg)

截图展示独立的虚构配置预览,任务处于停止状态。
实际存量和跨库增量同步的体验步骤见[图书示例演示](docs/demo.md)(英文),也可按下方命令启动 Compose。

## 本地体验

准备 Docker 和 Compose v2。在本目录执行:
Expand Down Expand Up @@ -77,3 +87,13 @@ python tools/check_public_export.py

上述单元测试无需真实数据库;其余上游集成测试需要独立测试服务,
参见 [贡献指南](CONTRIBUTING.md)。

## 反馈与上游协作

本扩展的安装、界面、文档组装和维护问题,请提交到
[Studio Issues](https://github.com/fooklook/pgsync-studio/issues)。
建议附上虚构的最小表结构、数据和脱敏日志,便于复现。

Studio 的引擎基础来自 [PGSync](https://github.com/toluaina/pgsync),
Studio 扩展的运行器和运维机制由本项目独立维护。
上游贡献由上游维护者审核;关系和兼容边界见 [UPSTREAM.md](UPSTREAM.md)。
101 changes: 101 additions & 0 deletions docs/demo.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
# Book-catalog walkthrough

This disposable Compose example joins PostgreSQL rows into Elasticsearch
documents. It uses synthetic books, authors, and view counters; the same example
runs in the repository's [end-to-end CI](https://github.com/fooklook/pgsync-studio/actions/workflows/ci.yml).

## What it demonstrates

The `book-catalog-demo` job reads `books` and `authors` from `catalog`, and
`book_metrics` from the separate `analytics` database. Both databases live in the
demo PostgreSQL container. The Elasticsearch index is `book_catalog_v1`.

```mermaid
flowchart LR
Books["catalog.public.books"] --> Document["book_catalog_v1 document"]
Authors["catalog.public.authors"] -->|"books.author_id = authors.id"| Document
Metrics["analytics.public.book_metrics"] -->|"books.id = book_metrics.book_id"| Document
```

The job first builds a baseline, then uses native change listeners and durable
outbox events to rebuild affected root documents. The automatic check changes a
counter in the second database and a title in the root table, then waits for the
corresponding Elasticsearch values and checks that runner/listener threads are alive.

## Start the example

From the repository root, with Docker and Compose v2 installed:

```sh
cp .env.example .env
# Set DEMO_POSTGRES_PASSWORD in .env before the first start.
docker compose up --build -d --wait
docker compose exec studio python examples/studio/bootstrap_demo.py
```

In PowerShell, replace the first command with `Copy-Item .env.example .env`.
Open <http://127.0.0.1:8180> and select **任务列表** (task list).
Use **加载** (load) to inspect a job's tables and relationships. The current UI
is in Simplified Chinese; the source and setup documentation are available in English.

![Task configuration using the synthetic book-catalog schema](images/demo-config.jpg)

This screenshot was captured from a local configuration preview using the
checked-in demo schema. Its task is stopped and it does not show live
synchronization results or performance measurements.

## Verify synchronization

Run the same check used by CI:

```sh
docker compose exec studio python tools/smoke_demo.py
```

Expected checkpoints:

```text
PASS baseline assembly
PASS cross-database metric update
PASS root-table update
PASS actual runner and listener threads
```

The script updates only the synthetic Compose example. It increments the first
book's view counter and changes its title, so these values can differ on later runs.
It rejects jobs whose sources do not match the example's container and databases.

For a manual update:

```sh
docker compose exec postgres psql -U studio -d analytics -c "UPDATE book_metrics SET views = views + 1, updated_at = now() WHERE book_id = 1;"
```

In **ES 自定义查询** (custom ES query), select `book_catalog_v1` and submit:

```json
{"query": {"ids": {"values": ["1"]}}, "size": 1}
```

The returned document should contain the changed `book_metrics.views` value.
Synchronization is eventually consistent; this example is a functional check,
not a latency or throughput benchmark.

## Stop and explore further

```sh
docker compose down
```

Named volumes preserve this example's data and configuration. The SQL initializer
runs only on the first creation of the PostgreSQL data volume. Changing `.env`
later does not change an existing PostgreSQL role's password.

Replica reads, cleanup, and partition maintenance are disabled in this example.
They require an appropriate topology and deliberate configuration; see
[deployment and configuration](deployment.md). Protect administrative access
before using a shared environment; see [SECURITY.md](../SECURITY.md).

For Studio feedback, use [the project's issue tracker](https://github.com/fooklook/pgsync-studio/issues)
with a minimal synthetic reproduction. See [UPSTREAM.md](../UPSTREAM.md) for how
this independent extension relates to PGSync.
Binary file added docs/images/demo-config.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/demo-jobs.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading