[Workload]: multi-tier

[mrhillsman]

Workload Name

multi-tier

Workload Description

A linked three-VM application stack — web frontend (nginx reverse proxy), application server (Python/Flask or Go HTTP service), and database (PostgreSQL) — with correlated request traffic flowing through all three tiers. Produces cross-VM request traces, per-tier latency breakdowns, and inter-service dependency signals that no single-concern workload can generate.

This is the only workload that creates correlated signals across VMs. The existing workloads produce independent signals: CPU pressure on one VM, disk I/O on another, network throughput between a pair. But APM and distributed tracing partners need to see a request enter the frontend, propagate to the app server, query the database, and return — with latency attribution at each hop. That's the signal their products are built to visualize, and it doesn't exist today without multi-tier.

The architecture mirrors the canonical three-tier application that most enterprises run in VMs: a web tier handling HTTP, an app tier running business logic, and a data tier running a relational database. This is the workload pattern that partners encounter most often when customers migrate from VMware to OpenShift Virtualization.

Tooling and Packages

Web VM (frontend):

• Tool: nginx configured as a reverse proxy to the app tier
• RPM packages: nginx
• systemd service: nginx.service with upstream pointing to app-tier Kubernetes Service
• Exposes port 80 via Kubernetes Service (entry point for load generation)

App VM (application server):

• Tool: Python Flask app or lightweight Go HTTP server
• RPM packages: python3, python3-flask, python3-psycopg2 (or a single Go binary with no deps)
• Behavior: receives requests from web tier, executes a parameterized SQL query against the database tier, returns a JSON response with per-tier timing
• systemd service: virtwork-app.service
• Exposes port 8080 via Kubernetes Service

Database VM (data tier):

• Tool: PostgreSQL (reuses patterns from existing database workload)
• RPM packages: postgresql-server, postgresql
• systemd service: postgresql.service
• Seeded with a pgbench-style schema for realistic query patterns
• Exposes port 5432 via Kubernetes Service

Load generator (built into web VM or separate client):

• Tool: hey or wrk2 targeting the web tier's nginx
• systemd service: virtwork-multi-tier-load.service
• Configurable RPS, concurrency, request mix

VM Count Model

Multiple coordinated VMs (describe in additional context)

Three VMs minimum per instance:

• 1 web VM (nginx reverse proxy + load generator)
• 1 app VM (Flask/Go application server)
• 1 database VM (PostgreSQL)

The load generator runs on the web VM to keep the VM count at three. Alternatively, it could be a fourth VM for cleaner separation.

Required Resources

• Persistent storage (DataVolume)
• Kubernetes Service (for inter-VM communication)
• Kubernetes Secret (for credentials or config)
• Additional CPU/memory beyond defaults
• GPU or special device passthrough

Three Kubernetes Services (web:80, app:8080, db:5432) for inter-tier communication. One Secret for database credentials shared between app and database VMs. One DataVolume for the database VM's persistent storage.

Cloud-Init Details

Database VM:

packages:
  - postgresql-server
  - postgresql
write_files:
  - path: /usr/local/bin/virtwork-db-setup.sh
    permissions: '0755'
    content: |
      #!/bin/bash
      set -euo pipefail
      postgresql-setup --initdb
      sed -i "s/#listen_addresses.*/listen_addresses = '*'/" /var/lib/pgsql/data/postgresql.conf
      echo "host all all 0.0.0.0/0 md5" >> /var/lib/pgsql/data/pg_hba.conf
      systemctl enable --now postgresql
      sudo -u postgres psql -c "CREATE USER appuser WITH PASSWORD '<from-secret>';"
      sudo -u postgres createdb -O appuser appdb
      sudo -u postgres pgbench -i -s 10 -U appuser appdb
runcmd:
  - /usr/local/bin/virtwork-db-setup.sh

App VM:

packages:
  - python3
  - python3-flask
  - python3-psycopg2
write_files:
  - path: /opt/virtwork/app.py
    content: |
      import time, os, json
      from flask import Flask, jsonify
      import psycopg2
      app = Flask(__name__)
      DB_HOST = os.environ.get("DB_HOST", "<db-service>")
      DB_USER = os.environ.get("DB_USER", "appuser")
      DB_PASS = os.environ.get("DB_PASS", "<from-secret>")

      @app.route("/api/v1/accounts/<int:aid>")
      def get_account(aid):
          t0 = time.monotonic()
          conn = psycopg2.connect(host=DB_HOST, dbname="appdb",
                                  user=DB_USER, password=DB_PASS)
          cur = conn.cursor()
          cur.execute("SELECT abalance FROM pgbench_accounts WHERE aid = %s", (aid,))
          row = cur.fetchone()
          db_ms = (time.monotonic() - t0) * 1000
          cur.close(); conn.close()
          return jsonify({"aid": aid, "balance": row[0] if row else None,
                          "db_latency_ms": round(db_ms, 2),
                          "tier": "app", "timestamp": time.time()})

      @app.route("/health")
      def health():
          return jsonify({"status": "ok"})

      if __name__ == "__main__":
          app.run(host="0.0.0.0", port=8080)
  - path: /etc/systemd/system/virtwork-app.service
    content: |
      [Unit]
      Description=Virtwork multi-tier application server
      After=network-online.target
      Wants=network-online.target
      [Service]
      Type=simple
      Environment=DB_HOST=<db-service>
      Environment=DB_USER=appuser
      Environment=DB_PASS=<from-secret>
      ExecStart=/usr/bin/python3 /opt/virtwork/app.py
      Restart=always
      RestartSec=5
      [Install]
      WantedBy=multi-user.target
runcmd:
  - systemctl enable --now virtwork-app.service

Web VM:

packages:
  - nginx
write_files:
  - path: /etc/nginx/conf.d/virtwork-proxy.conf
    content: |
      upstream app_backend {
          server <app-service>:8080;
      }
      server {
          listen 80;
          location / {
              proxy_pass http://app_backend;
              proxy_set_header Host $host;
              proxy_set_header X-Real-IP $remote_addr;
              proxy_set_header X-Request-Start "t=${msec}";
          }
          location /nginx_status {
              stub_status on;
          }
      }
  - path: /etc/systemd/system/virtwork-multi-tier-load.service
    content: |
      [Unit]
      Description=Virtwork multi-tier load generator
      After=nginx.service
      Wants=nginx.service
      [Service]
      Type=simple
      ExecStartPre=/bin/sleep 30
      ExecStart=/usr/local/bin/hey -z 0 -c 20 -q 50 http://localhost/api/v1/accounts/1
      Restart=always
      RestartSec=10
      [Install]
      WantedBy=multi-user.target
runcmd:
  - curl -Lo /usr/local/bin/hey https://hey-release.s3.us-east-2.amazonaws.com/hey_linux_amd64
  - chmod +x /usr/local/bin/hey
  - systemctl enable --now nginx
  - systemctl enable --now virtwork-multi-tier-load.service

Use Case

• APM/Distributed tracing partners (Dynatrace, Datadog APM, New Relic, Elastic APM, Instana): The primary audience. These products exist to trace requests across service boundaries and attribute latency to each tier. A request flowing web→app→database across three VMs is the minimum viable distributed trace. Without cross-VM request correlation, APM partners can only validate single-service instrumentation — they can't prove their product correctly stitches together a distributed transaction in a KubeVirt environment.
• Service mesh partners (Istio-based, Linkerd integrations): Need multi-tier traffic to validate that sidecar proxies correctly handle VM-to-VM traffic, mTLS between tiers, traffic shifting, and fault injection at the service boundary. A single-VM workload doesn't exercise service mesh features.
• Observability platform partners (Grafana Labs, Chronosphere): Need correlated metrics, logs, and traces across services to validate unified observability dashboards. The multi-tier workload produces per-tier latency metrics (nginx upstream time, Flask handler time, PostgreSQL query time) that should appear correlated in the partner's UI.
• Load balancer / Ingress partners (F5, HAProxy, Nginx Plus): Need multi-tier traffic to validate request routing, health checks, and load distribution across VM backends. The nginx frontend is the natural insertion point for partner load balancer products.
• VMware-to-OpenShift migration validation: The three-tier architecture is the canonical enterprise VM workload being migrated from VMware to OpenShift Virtualization. Partners helping customers with this migration (MTV, consulting partners) need a reproducible three-tier workload to validate that the migrated application performs equivalently on KubeVirt.

Additional Context

• Implementation approach: This is the most complex workload in virtwork's catalog. The recommended implementation extends MultiVMWorkload with three roles (web, app, database) via RoleDistribution() []RoleSpec. Each role gets distinct cloud-init via UserdataForRole(role, namespace). The orchestrator creates VMs in dependency order: database first (needs to be ready before app connects), then app (needs to be ready before web proxies), then web (with load generator starting after a delay).
• Startup ordering: The ExecStartPre=/bin/sleep 30 on the load generator is a simple approach. A more robust approach would be a readiness check loop (until curl -sf http://localhost/health; do sleep 5; done). The database and app VMs use cloud-init runcmd ordering which naturally sequences setup.
• Service discovery: Each tier connects to the next via Kubernetes Service DNS (<service-name>.<namespace>.svc.cluster.local). This exercises the same DNS resolution path that the dns-stress workload tests, but in a realistic application context.
• Scaling: The initial implementation uses one VM per tier (3 VMs total). Future enhancement: --vm-count 2 could create 2 web + 2 app + 1 database (5 VMs), modeling a scaled-out application tier with load balancing. This significantly increases complexity but is the pattern that load balancer partners need.
• Per-tier latency visibility: The Flask app returns db_latency_ms in the response body. nginx can be configured to log $upstream_response_time. Combined with the load generator's overall latency, this gives three-tier latency attribution without external instrumentation — the baseline that partner APM products should match or improve upon.
• Composability: This workload pairs naturally with:
  • chaos-network — inject latency between app and database tiers to create cascading latency signals
  • chaos-process — kill the app process and observe how the web tier handles upstream failures
  • metrics-emitter — add Prometheus metrics alongside the application metrics for monitoring validation