A distributed background job processing system built in Go, with persistent job storage, concurrent workers, and asynchronous task execution.
Server receives HTTP request from client to perform a job and validates it. A valid job request is stored in the database with a PENDING status. Workers claim jobs and complete them. The system is fault-tolerant and resistant. Workers can complete multiple tasks in parallel.
When the server is run, it starts by applying all database migrations. Then, it creates the ConnectRPC Service Handlers generated by Buf and serves the RPCs on a Go HTTP Server. The procedures then interact with the database using the jackc/pgx library and return the response to the client.
The client is a collection of generated Connect-Go Clients which are instantiated at startup time as Go HTTP Clients.
The communication protocol and between the client and the server is defined using protobuf with the Buf CLI tool to generate the structs. Run buf lint to check the current .proto files agains the linter and buf generate to compile them and generate the Go code they describe. Use just buf-breaking to check for breaking changes between the current changes and the previous ones. The buf workspace is defined in buf.yaml and the specifications for generating the code is in buf.gen.yaml.
The application uses the ConnectRPC protocol for communication between the client and the server. This is done using the Connect-Go library which allows to define type-safe Remote Procedure Calls using our protobuf definition. We use the protovalidate library to ensure the invariants are respected on all the fields passed between client and server.
This project uses a PostgreSQL database inside of a docker container. The database can be spun up using docker compose up and lives on port 5432. Once the database is running, you can connect to it by using the just db-connect command.
Migrations to the database are applied using the golang-migrate library. The server always tries to migrate up to the latest schema on startup. Migrations can also be applied or reversed manually using the CLI tool by running just db-migrate <up|down> [amount]. Leaving amount blank will apply or reverse all migrations. To create a new migration, run just db-create <name>. See documentation for more details.
Database errors are translated in the service into useful ConnectRPC Errors, abstracting database information away.
Centralised system configuration is found in the config folder. This provides a single place where you can change properties of the system.
Current properties are:
- Server port on localhost
- just : Open-source command runner
- golang-migrate : Open-source database schema migration tool
- Buf CLI : Open-source protobuf toolchain
- Docker Compose : Define and run multi-container applications
- Connect-Go : Open-source library for connectRPC
- Lefthook : Fast and simple pre-commit and pre-push manager
- Service/Repository architecture
- Testing