Frank!Framework Insights

The Frank!Framework insights application is an open-source tool designed to provide in-depth insights into the development and release lifecycle of the Frank!Framework.

Purpose of the Application

Frank!Framework Insights provides users, contributors, and maintainers with a centralized overview of the development activities surrounding the Frank!Framework. Instead of manually gathering information from different sources, this tool collects, analyzes, and visualizes key data to make the entire release lifecycle transparent.

The key goal is to offer detailed insights into releases at every stage.

Past Releases
Analyze the composition of previous releases. Understand which issues were fixed, view the final statistics, and identify vulnerabilities.

Current Release
Track the real-time progress of the release currently in development. The dashboard provides insights into the stability and progress by visualizing the current development roadmap with updates about the progress.

Future Releases (Roadmap)
Look ahead at the project's direction. The tool visualizes the roadmap based on GitHub Projects, showing planned features and epics for (upcoming) releases.

In-depth Release Analysis

For any given release, the tool provides deep-dive analytics by processing a wide range of data points and visualizing the relationships between them. This includes analyzing issue attributes (e.g., bug, feature, priority, and labels), development data such as associated branches and pull requests, and planning elements like milestones and their completion status.

By analyzing the ratios and connections between these elements, users can gain a much deeper understanding of the work involved in a release, identify potential risks, and track the overall health of the development process.

Security Vulnerability Scanning

The application integrates Trivy to automatically scan Frank!Framework release artifacts for known vulnerabilities (CVEs). It provides detailed security information including severity levels, CVSS scores, and vulnerability trends across releases, helping maintainers and users make informed decisions about release security.

System Architecture

The backend is split into two independently deployable Spring Boot services that share one PostgreSQL database:

Service	Responsibility	Port
insights-data-import	Gathers data from external sources (GitHub API, Trivy) and writes it to the database. Runs on a schedule and on a GitHub release webhook. Exposes no read API.	8081
insights-webapp	Reads from the database and serves it through the REST API and the bundled Angular single page application. Never writes imported data.	8080

A third module, insights-common, holds what both need: the JPA entities and repositories, the Flyway migrations, the object mapper and the shared HTTP client infrastructure. It is a plain library jar, not a runnable application.

Splitting the two means the import job, which is long running, memory hungry and runs Trivy and Maven as subprocesses, can be restarted, scaled or taken down without touching the site that users are looking at.

Currently, the application primarily uses the GitHub API to retrieve data about Frank!Framework's repository. However, the architecture is designed to be extensible, meaning other external data sources can be integrated in a similar way in the future. This allows the application to be scaled with new integrations as needed.

Before changing anything substantial, read docs/ARCHITECTURE.md. It covers how the modules fit together, how data gets in, and most importantly, the business rules and decisions behind the insights. Those cannot be worked out from the code alone.

Project Structure

The application is a Maven multi-module project with an integrated Angular frontend:

insights/
├── docker/
│   ├── Dockerfile                       # One file, two build targets: webapp and data-import
│   └── scripts/                         # Container startup scripts
├── insights-common/                     # Shared library (no main class)
│   └── src/main/
│       ├── java/                        # JPA entities, repositories, mapper, HTTP clients
│       └── resources/
│           ├── db/migration/            # Flyway migrations
│           └── insights-common.properties
├── insights-data-import/                # Gathers data and writes it to the database
│   └── src/main/
│       ├── java/                        # GitHub GraphQL client, *InjectionService, Trivy scanning
│       └── resources/
│           ├── graphql-documents/       # GitHub GraphQL queries
│           └── application*.properties
├── insights-webapp/                     # Serves the data: REST API + frontend
│   └── src/main/
│       ├── java/                        # Controllers, *QueryService, security, rate limiting
│       ├── resources/application*.properties
│       └── frontend/                    # Angular frontend application
│           ├── src/                     # Frontend source code
│           ├── cypress/                 # E2E tests
│           ├── package.json             # Frontend dependencies (pnpm)
│           └── angular.json             # Angular configuration
├── pom.xml                              # Parent POM / reactor
├── docker-compose.yaml                  # Local Docker setup (database + both services)
└── pnpm-lock.yaml                       # pnpm lock file

Within a domain package such as org.frankframework.insights.release, the classes are divided over the modules by what they do: the entity and repository live in insights-common, everything that writes (ReleaseInjectionService, the GitHub DTOs) in insights-data-import, and everything that reads (ReleaseController, ReleaseQueryService, ReleaseResponse) in insights-webapp.

Build Process:

insights-common is built first and both services depend on it
While building insights-webapp, Maven triggers pnpm to install frontend dependencies
Maven triggers pnpm to build the Angular application
The built frontend is packaged as static resources inside the insights-webapp JAR
The webapp serves both the API and the frontend; insights-data-import is packaged as a separate JAR

Quick Local Setup with Docker

For a fast and easy setup, you can use Docker Compose to run the entire stack: the database, the import service and the web application.

Ensure you have Docker Desktop installed, as it includes Docker and Docker Compose.

Clone the repository:

git clone https://github.com/frankframework/insights.git
cd insights

Fill in your GitHub token, project id, webhook secret and OAuth client in the application-local.properties of each module. Both services run with the local Spring profile.
Build the JARs. The images copy them out of the target directories, so Maven has to run first:
```
./mvnw clean package -DskipTests
```
Start everything. The --build flag forces a rebuild of the images so you are running the latest code, and -d runs the containers in the background:
```
docker compose up -d --build
```

What	Where
Application (API + frontend)	`http://localhost:8080`
Import service health	`http://localhost:8081/actuator/health`
GitHub release webhook	`http://localhost:8081/api/webhooks/github`
PostgreSQL	`localhost:5432`

Both services run their own Flyway migrations against the shared database. Flyway locks the schema history table while migrating, so it does not matter which of the two starts first.

Note: Trivy and Maven are baked into the insights-data-import image only (see docker/Dockerfile), so no additional installation or path configuration is required. The insights-webapp image is a plain JRE image and does not carry them.

Running only one of the services

Because the two are independent, you can start just the part you need:

docker compose up -d insights-webapp        # site only, serves whatever is already in the database
docker compose up -d insights-data-import   # importer only

Database Seeding

If you prefer to start with a clean database and fetch real data from GitHub, configure your GitHub API token in insights-data-import/src/main/resources/application-local.properties, which also sets data.fetch-enabled=true. To browse the application with mock data instead, run the webapp with the local-seed Spring profile, which loads db/e2e/R__Seed_Data.sql into an in-memory database.

Please note that not all releases in the mock data set have detailed content. For a full example of a release with associated issues and pull requests, check release v9.0.1.

Manual Development Setup

For active development, a manual setup provides more granular control over the individual components. This setup automatically uses the local Spring profile for local development configuration.

Prerequisites

For a manual setup, you will need:

Git - Version control system
Java Development Kit (JDK 25) - Required for the backend (java.version in the parent POM)
Node.js (version 24) - Required for the frontend
pnpm (version 10.33.0) - Package manager (npm install -g pnpm)
PostgreSQL - Database instance
Trivy - Security vulnerability scanner (Installation guide)
IDE - Recommended: IntelliJ IDEA, WebStorm, VS Code, or Eclipse

Note on Maven: A separate installation of Apache Maven is not required. The project includes the Maven Wrapper (mvnw), which automatically downloads and uses the correct Maven version.

Steps

Clone the Repository

git clone https://github.com/frankframework/insights.git
cd insights

Open the Project

Open the repository root in your Java IDE. It will detect it as a Maven project with three modules. There are two runnable applications:

Module	Main class	Profile files
`insights-webapp`	`org.frankframework.insights.InsightsWebappApplication`	`insights-webapp/src/main/resources/application-local.properties`
`insights-data-import`	`org.frankframework.insights.InsightsDataImportApplication`	`insights-data-import/src/main/resources/application-local.properties`

Settings that both share (Flyway, JPA, actuator, GitHub URLs) live in insights-common/src/main/resources/insights-common.properties, which both applications pull in through spring.config.import. Anything you set in a module's own properties file wins over it.

Create & Configure the Database

Create a single, empty PostgreSQL database. Both services use the same one. Then set the datasource properties in both application-local.properties files:
```
spring.datasource.url=jdbc:postgresql://localhost:5432/your_database_name
spring.datasource.username=your_username
spring.datasource.password=your_password
```
Configure the Import Service (insights-data-import/src/main/resources/application-local.properties)
- GitHub API access: create a GitHub Personal Access Token (PAT) with the read:org and project permissions (official guide), then set:
```
github.graphql.secret=YOUR_PERSONAL_ACCESS_TOKEN_HERE
github.graphql.project-id=YOUR_GITHUB_PROJECT_ID_HERE
```
- Initial data injection: to populate the database with GitHub data on startup, set:
```
data.fetch-enabled=true
```
  After the first successful run you can set this to false to avoid refetching on every start. The daily job at midnight runs regardless.
- Trivy path: point at your locally installed Trivy executable:
```
trivy.path=C:/Program Files/trivy/trivy.exe
```
  Note: only needed for a manual setup. The Docker image ships Trivy on the PATH.

Configure the Web Application (insights-webapp/src/main/resources/application-local.properties)

Set the GitHub OAuth app used to log users in:

spring.security.oauth2.client.registration.github.client-id=YOUR_CLIENT_ID
spring.security.oauth2.client.registration.github.client-secret=YOUR_CLIENT_SECRET

Frontend Development (Optional)

The frontend is built by Maven as part of insights-webapp. For active frontend development with live reloading:
- Navigate to the frontend directory:
```
cd insights-webapp/src/main/frontend
```
- Install dependencies:
```
pnpm install
```
- Start the development server:
```
ng serve
```
The frontend will be available at http://localhost:4200 with live reloading enabled.

Configuration Reference

Settings both services share live in insights-common/src/main/resources/insights-common.properties (Flyway, JPA, actuator, GitHub URLs, and the branch and label filters). Both services import it via spring.config.import; anything a module sets itself wins over it.

The application-local.properties files are tracked in Git with placeholder values. Fill in your own credentials locally, but never commit real secrets to them.

Properties you need to set

Property	Service	What it does
`spring.datasource.url` / `.username` / `.password`	both	The shared database. Both point at the same one.
`data.fetch-enabled`	data-import	Master switch for all GitHub fetching. `false` disables startup, nightly and webhook refreshes.
`github.graphql.secret`	data-import	GitHub PAT, needs `read:org` and `project`
`github.graphql.project-id`	data-import	The GitHub Project the roadmap is built from
`insights.webhook.secret`	data-import	Shared secret for GitHub webhook HMAC signatures
`trivy.path`	data-import	Path to the Trivy executable. Not needed in Docker, the image ships it on the `PATH`.
`release.archive.directory`	data-import	Where downloaded release zips are cached. Must be persistent storage.
`trivy.scan.workspace` / `trivy.db.cache` / `maven.local-repo`	data-import	Scratch space, Trivy DB cache, and the Maven repo used to pre-cache dependencies
`spring.security.oauth2.client.registration.github.client-id` / `.client-secret`	webapp	GitHub OAuth app, used to log users in
`cors.allowed.origins[n]`	webapp	Allowed browser origins. Credentials are allowed, so this can never be `*`.
`frankframework.security.csrf.secure`	webapp	`secure` flag on the CSRF cookie. `false` locally so plain HTTP works, `true` in production.

Two settings in insights-common.properties are business rules, not just config github.graphql.branch-protection-regexes and github.graphql.includedLabels. See docs/ARCHITECTURE.md before changing either.

Environment variables (the `prod` profile)

Variable	Service
`DATABASE_HOST`, `DATABASE_PORT`, `DATABASE_NAME`, `DATABASE_USERNAME`, `DATABASE_PASSWORD`	both
`GITHUB_API_SECRET`, `GITHUB_PROJECT_ID`, `INSIGHTS_WEBHOOK_SECRET`	data-import
`GITHUB_OAUTH_CLIENT_ID`, `GITHUB_OAUTH_CLIENT_SECRET`	webapp
`SERVER_PORT` (optional), `JAVA_OPTS` (optional)	both

There is no default Spring profile. A service started without one has no datasource and will not boot, always pass local, local-seed or prod.

Building & Testing

Building the Application

Maven builds the three modules in one reactor: insights-common first, then the two services. The Angular frontend is built into the insights-webapp JAR.

Full Build with Tests:

./mvnw clean package "-Dspring.profiles.active=local-seed"

or

./mvnw clean install "-Dspring.profiles.active=local-seed"

This command:

Builds and installs insights-common
Builds insights-data-import and runs its tests
Installs frontend dependencies using pnpm and builds the Angular application
Compiles insights-webapp, runs its tests and the E2E tests
Produces two executable JARs:
- insights-webapp/target/insights-webapp-<version>.jar (API + frontend as static resources)
- insights-data-import/target/insights-data-import-<version>.jar

Automated Testing: Running mvn package or mvn install with the local-seed profile automatically executes:

Backend Tests: JUnit 5 unit and integration tests with Mockito, per module

End-to-End Tests: Cypress tests using Testcontainers (in insights-webapp)

Important: The Cypress E2E tests require the local-seed Spring profile to seed the database with test data. This is because the E2E tests verify the complete user interface and workflows, which require actual data to be present in the database (releases, issues, pull requests, etc.). Without seeded data, the tests would have nothing to interact with and would fail. The tests run in a containerized environment via Testcontainers, so no additional setup or running application is required.

Note: Frontend unit tests (Jasmine/Karma) are not run during the Maven build. To run them separately, see Running Tests Individually.

Quick Build (Skip Tests):

For faster iteration during development, you can skip tests:

./mvnw clean package -DskipTests

Building a Single Module:

-am ("also make") builds the modules the requested one depends on:

./mvnw clean package -pl insights-data-import -am        # importer + common
./mvnw clean package -pl insights-webapp -am             # webapp + common (builds the frontend)

To skip the pnpm steps while working on the backend only, add -Dexec.skip=true.

Running Tests Individually

All Backend Tests:

./mvnw test

One Module's Tests:

./mvnw test -pl insights-data-import
./mvnw test -pl insights-webapp

Frontend Tests Only:

cd insights-webapp/src/main/frontend
pnpm test

E2E Tests Interactively:

cd insights-webapp/src/main/frontend
pnpm run cypress:open  # Interactive mode with UI
pnpm run cypress:run   # Headless mode

Running the Application

The two services are started separately. The web application works on its own against whatever is already in the database; you only need the import service when you want to refresh that data.

Option 1 - Run the JARs:

java -jar insights-webapp/target/insights-webapp-*.jar
java -jar insights-data-import/target/insights-data-import-*.jar

Option 2 - IDE: Start InsightsWebappApplication and/or InsightsDataImportApplication directly from your IDE with the local profile active.

Option 3 - Maven:

./mvnw spring-boot:run -pl insights-webapp
./mvnw spring-boot:run -pl insights-data-import

The web application will be available at http://localhost:8080 and the import service at http://localhost:8081. If you're running the frontend development server separately, it will be at http://localhost:4200 and proxy API calls to the backend.

CI/CD & Quality Assurance

The project uses GitHub Actions to run automated workflows for every pull request and merge to the master branch. These workflows ensure code quality, security, and stability before changes are merged.

Continuous Integration Pipeline

The CI pipeline (ci.yaml) runs the following checks on every pull request:

Code Linting
- Backend: Checkstyle for Java code style enforcement (mvn checkstyle:check, fails the build)
- Frontend: ESLint for TypeScript/JavaScript code quality
Code Formatting
- Spotless (palantir-java-format) is configured in the parent POM but has no lifecycle binding, so it does not run in CI and never fails a build. Run it yourself: mvn spotless:apply to format, mvn spotless:check to verify.
Automated Testing
- Backend unit and integration tests (JUnit 5)
- Frontend unit tests (Jasmine/Karma)
- End-to-end tests (Cypress via Testcontainers)
Build Verification
- Full Maven build with pnpm frontend integration
- Validates that the application can be packaged successfully

Continuous Deployment

Docker Image Creation
On every merge to master, two Docker images are built from docker/Dockerfile (one target each) and pushed to the GitHub Container Registry:

ghcr.io/frankframework/insights-webapp: API and frontend, plain JRE image
ghcr.io/frankframework/insights-data-import: importer, includes Trivy and Maven for vulnerability scanning

Each image is pushed with three tags: 0.0.<run number> (the immutable build, matching the Maven revision baked into the JAR), latest and master. Pin production deployments to the versioned tag; latest always points at the most recent master build.

Both images are signed with cosign, once per digest, so every tag on that digest is covered. The local docker compose build reuses the same two names with a :local tag, so a locally built image never shadows a pulled release.

Security

The project includes multiple security analysis tools:

Trivy: Scans Frank!Framework release artifacts for CVEs (included in Docker, requires local installation for development)
SonarQube: Tracks code quality metrics, code smells, and security issues

Performance Testing

Stress Tests
Stress tests can be triggered manually via GitHub Actions (stress-tests.yaml) to test the latest version on master. These tests push the system to its limits by simulating high traffic or data load to measure performance, identify bottlenecks, and ensure the application remains stable and responsive under pressure.

Contributing

This is an open-source project, and contributions are highly welcome! We follow the overarching contribution guidelines of the Frank!Framework organization.

Code of Conduct
All contributors are expected to adhere to our Code of Conduct.

Contribution Guidelines
For general guidelines like commit messages and pull request procedures, see the main CONTRIBUTING.md.

Project Structure
The project is a Maven multi-module monorepo (insights-common, insights-data-import, insights-webapp) with an integrated Angular frontend in insights-webapp/src/main/frontend. When working on the frontend, always use pnpm as the package manager (not npm or yarn).

When adding backend code, put it in the module that matches what it does: shared entities and repositories in insights-common, anything that writes imported data in insights-data-import, anything that serves data to the frontend in insights-webapp. The two services must not depend on each other, and Flyway migrations always belong in insights-common. The reasoning behind those rules is in docs/ARCHITECTURE.md.

If you add an API endpoint, add its row to the endpoint table in docs/ARCHITECTURE.md there is no OpenAPI spec, so that table is the API documentation.

Code Conventions

Frontend: Must adhere to the Frank!Framework Frontend Conventions
Backend: Document public classes and methods with Javadoc. Follow Checkstyle and Spotless formatting rules

Quality Requirements
Before submitting a pull request:

Ensure all CI/CD pipeline checks pass (linting, testing, building)
Clearly explain what you changed and why in your commit messages and pull request description
For details on running tests locally, see the Building & Testing section

How to Contribute

Report bugs or suggest features by creating an issue
Fork the repository and submit a pull request with your improvements
Improve documentation or add examples

License

This project is licensed under the Apache 2.0 License. See the LICENSE file for the full terms.

Navigation Menu

Uh oh!

Repository files navigation

Frank!Framework Insights

Purpose of the Application

In-depth Release Analysis

Security Vulnerability Scanning

System Architecture

Project Structure

Quick Local Setup with Docker

Running only one of the services

Database Seeding

Manual Development Setup

Prerequisites

Steps

Configuration Reference

Properties you need to set

Environment variables (the `prod` profile)

Building & Testing

Building the Application

Running Tests Individually

Running the Application

CI/CD & Quality Assurance

Continuous Integration Pipeline

Continuous Deployment

Security

Performance Testing

Contributing

License

About

Releases

Packages

Used by

Contributors

Languages

Name		Name	Last commit message	Last commit date
.github		.github
.mvn/wrapper		.mvn/wrapper
docker		docker
docs		docs
insights-common		insights-common
insights-data-import		insights-data-import
insights-webapp		insights-webapp
.dockerignore		.dockerignore
.editorconfig		.editorconfig
.gitattributes		.gitattributes
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
checkstyle.xml		checkstyle.xml
docker-compose.yaml		docker-compose.yaml
mvnw		mvnw
mvnw.cmd		mvnw.cmd
pnpm-lock.yaml		pnpm-lock.yaml
pnpm-workspace.yaml		pnpm-workspace.yaml
pom.xml		pom.xml

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

Frank!Framework Insights

Purpose of the Application

In-depth Release Analysis

Security Vulnerability Scanning

System Architecture

Project Structure

Quick Local Setup with Docker

Running only one of the services

Database Seeding

Manual Development Setup

Prerequisites

Steps

Configuration Reference

Properties you need to set

Environment variables (the prod profile)

Building & Testing

Building the Application

Running Tests Individually

Running the Application

CI/CD & Quality Assurance

Continuous Integration Pipeline

Continuous Deployment

Security

Performance Testing

Contributing

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages

Environment variables (the `prod` profile)