Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,16 @@ TRAINING_HUB_SITE_HOSTING_LOCATION=Ashburn, Virginia, USA
MARKETGUARD_DB_DRIVER=mariadb
MARKETGUARD_CACHE_TTL_SECONDS=60
MARKETGUARD_STALE_IF_ERROR_SECONDS=300
# Refresh the Lowest BIN and Bazaar snapshots on a timer instead of inside client
# requests. Leave this on: with it off, the first caller after the cache expires
# pays for the full Hypixel fetch and every other request waits behind it.
MARKETGUARD_BACKGROUND_REFRESH_ENABLED=true
# 0 = follow MARKETGUARD_CACHE_TTL_SECONDS
MARKETGUARD_BACKGROUND_REFRESH_INTERVAL_SECONDS=0
MARKETGUARD_BACKGROUND_REFRESH_RETRY_SECONDS=15
# Extra snapshot age /api/v1/ready still reports as 200. Hypixel only regenerates
# these datasets about once a minute, so a slightly-over-TTL snapshot is normal.
MARKETGUARD_READINESS_FRESH_GRACE_SECONDS=120
MARKETGUARD_HISTORY_RETENTION_DAYS=45
MARKETGUARD_LOWESTBIN_RATE_LIMIT_PER_MINUTE=30
MARKETGUARD_HYPIXEL_API_KEY=
Expand Down
37 changes: 37 additions & 0 deletions DEPLOY.md
Original file line number Diff line number Diff line change
Expand Up @@ -268,6 +268,7 @@ Important notes:
- keep `TRAINING_HUB_TRUSTED_PROXIES=127.0.0.1` unless you know you need more
- `/docs`, `/redoc`, and `/openapi.json` should stay disabled publicly unless you intentionally expose them
- keep `WEB_CONCURRENCY=1` while `/api/v1/players` uses its process-local per-IP limiter; Redis shares cached responses but not rate-limit state
- keep `MARKETGUARD_BACKGROUND_REFRESH_ENABLED=true`; the single worker relies on snapshots being refreshed off the request path
- use Admin Analytics > Metrics to inspect Player QUERY cache hit rate, response time, active upstream loads, and upstream failures before raising `MARKETGUARD_PLAYERS_MAX_UPSTREAM_CONCURRENCY`

## 9) Run Preflight
Expand Down Expand Up @@ -491,6 +492,39 @@ bash scripts/preflight.sh

Fix the exact value the script reports before retrying the deploy.

### Uptime monitor reports the API offline or flapping

The public API keeps its Hypixel snapshots warm with a background refresher. If
uptime checks flap, work through this in order.

1. Point the monitor at `https://<host>/api/v1/ready` and accept `200` and `206`
as up. `206` means "serving slightly stale data", not "down"; only `503`
means no usable snapshot. Do not monitor `/api/v1/health` - it is
intentionally `403` from public networks.
2. Confirm the refresher started:

```bash
docker compose logs scamscreener-api | grep "background refresher"
```

You should see `Started MarketGuard lowestbin background refresher` and the
same for `bazaar`. If instead you see `MarketGuard background refresh is
disabled`, `MARKETGUARD_BACKGROUND_REFRESH_ENABLED` is set to `false` -
with it off, refreshes fall back into client requests and a single Uvicorn
worker goes unresponsive on every cache expiry.
3. Check for repeated `background refresh failed` warnings. Those are upstream
Hypixel or storage errors: the last good snapshot is still served (marked
stale) and the refresher retries every
`MARKETGUARD_BACKGROUND_REFRESH_RETRY_SECONDS`.
4. If snapshots are consistently just past their TTL, raise
`MARKETGUARD_READINESS_FRESH_GRACE_SECONDS` rather than lowering
`MARKETGUARD_CACHE_TTL_SECONDS`; Hypixel only regenerates these datasets
about once per minute, so fetching more often buys nothing and burns quota.

Behind Cloudflare, also make sure the monitor is not being served a cached
response: `/api/v1/ready` sends `Cache-Control: no-store`, so a cached `200`
there means a proxy rule is overriding origin cache headers.

### Compose starts but the app is unhealthy

Check:
Expand All @@ -505,6 +539,9 @@ Typical causes:
- missing SMTP values while admin MFA or password-reset mail is enabled
- mismatched domain settings
- missing external-auth provider credentials when that flow is enabled
- the API container failing its healthcheck because the event loop is busy; with
background refresh enabled this should not happen, so capture
`docker compose logs scamscreener-api` before restarting

## Final Expected State

Expand Down
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -238,6 +238,10 @@ The production topology is Compose-first. Running a single `docker run` containe
- `MARKETGUARD_DB_REQUIRE_TLS` and `MARKETGUARD_DB_SSL_CA` enable verified external MariaDB TLS; the managed internal compose database defaults to plain internal transport
- `MARKETGUARD_CACHE_TTL_SECONDS` default `60`
- `MARKETGUARD_STALE_IF_ERROR_SECONDS` default `300`
- `MARKETGUARD_BACKGROUND_REFRESH_ENABLED` default `true`; refreshes the Lowest BIN and Bazaar snapshots on a timer so client requests never pay for a Hypixel fetch
- `MARKETGUARD_BACKGROUND_REFRESH_INTERVAL_SECONDS` default `0`, meaning "follow `MARKETGUARD_CACHE_TTL_SECONDS`"
- `MARKETGUARD_BACKGROUND_REFRESH_RETRY_SECONDS` default `15`; retry delay after a failed background refresh
- `MARKETGUARD_READINESS_FRESH_GRACE_SECONDS` default `120`; extra snapshot age `/api/v1/ready` still reports as `200` instead of `206`
- `MARKETGUARD_HISTORY_RETENTION_DAYS` default `45`
- `MARKETGUARD_LOWESTBIN_RATE_LIMIT_PER_MINUTE` default `30`
- `MARKETGUARD_HYPIXEL_API_KEY` required for `/api/v1/players`; keep it only in the deployment secret environment
Expand Down
40 changes: 40 additions & 0 deletions app/main.py
Original file line number Diff line number Diff line change
@@ -1,14 +1,53 @@
from __future__ import annotations

import contextlib
import logging

from fastapi import FastAPI

from .marketguard_api.config import MarketGuardSettings
from .marketguard_api.player_service import PlayerService
from .marketguard_api.refresher import build_refresh_supervisor
from .marketguard_api.routes import register_marketguard_routes
from .marketguard_api.service import BazaarService, LowestBinService
from .training_hub.config.settings import TrainingHubSettings
from .training_hub.main import create_training_hub_app, install_filtered_openapi

logger = logging.getLogger(__name__)


def _install_marketguard_refreshers(app: FastAPI) -> None:
"""Run the snapshot refreshers alongside the Training Hub lifespan.

The combined entrypoint reuses the hub's lifespan, so the refreshers are
layered on top of it rather than replacing it. Without them the MarketGuard
routes would fall back to refreshing inside client requests.
"""
supervisor = build_refresh_supervisor(
getattr(app.state, "marketguard_settings", None),
lowestbin_service=getattr(app.state, "marketguard_service", None),
bazaar_service=getattr(app.state, "marketguard_bazaar_service", None),
)
app.state.marketguard_refresh_supervisor = supervisor
if supervisor is None:
return

hub_lifespan = app.router.lifespan_context

@contextlib.asynccontextmanager
async def combined_lifespan(scoped_app: FastAPI):
async with hub_lifespan(scoped_app):
await supervisor.start()
try:
yield
finally:
try:
await supervisor.stop()
except Exception:
logger.exception("Could not stop the MarketGuard snapshot refreshers during shutdown.")

app.router.lifespan_context = combined_lifespan


def create_app(
training_hub_settings: TrainingHubSettings | None = None,
Expand All @@ -25,6 +64,7 @@ def create_app(
bazaar_service=marketguard_bazaar_service,
player_service=marketguard_player_service,
)
_install_marketguard_refreshers(app)
install_filtered_openapi(
app,
route_filter=lambda route: str(route.endpoint.__module__) == "app.marketguard_api.routes",
Expand Down
34 changes: 34 additions & 0 deletions app/marketguard_api/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,10 @@ class MarketGuardSettings:
snapshot_retries: int = 3
cache_ttl_seconds: int = 60
stale_if_error_seconds: int = 300
background_refresh_enabled: bool = True
background_refresh_interval_seconds: int = 0
background_refresh_retry_seconds: int = 15
readiness_fresh_grace_seconds: int = 120
history_retention_days: int = 45
lowestbin_rate_limit_per_minute: int = 30
players_rate_limit_per_minute: int = 3
Expand All @@ -123,6 +127,28 @@ class MarketGuardSettings:
redis_key_prefix: str = "marketguard:response"
api_docs_enabled: bool = True

@property
def effective_background_refresh_interval_seconds(self) -> int:
"""Refresh cadence of the background snapshot refresher.

Defaults to the cache TTL so a snapshot is replaced right about when it
would otherwise go stale, which keeps every request on a warm cache.
"""
if self.background_refresh_interval_seconds > 0:
return self.background_refresh_interval_seconds
return max(5, int(self.cache_ttl_seconds))

@property
def readiness_fresh_seconds(self) -> int:
"""Snapshot age still reported as fully ready by ``/api/v1/ready``.

Hypixel only regenerates the auction house roughly once per minute, so a
snapshot slightly older than the cache TTL is normal operation, not a
degraded service. Without the grace window every uptime probe that
requires HTTP 200 flaps between ready and degraded.
"""
return int(self.cache_ttl_seconds) + int(self.readiness_fresh_grace_seconds)

@classmethod
def from_env(cls) -> "MarketGuardSettings":
base_dir = Path(__file__).resolve().parents[2]
Expand Down Expand Up @@ -167,6 +193,10 @@ def from_env(cls) -> "MarketGuardSettings":
snapshot_retries=_env_int("MARKETGUARD_SNAPSHOT_RETRIES", 3, 1, 10),
cache_ttl_seconds=_env_int("MARKETGUARD_CACHE_TTL_SECONDS", 60, 5, 900),
stale_if_error_seconds=_env_int("MARKETGUARD_STALE_IF_ERROR_SECONDS", 300, 5, 3600),
background_refresh_enabled=_env_bool("MARKETGUARD_BACKGROUND_REFRESH_ENABLED", True),
background_refresh_interval_seconds=_env_int("MARKETGUARD_BACKGROUND_REFRESH_INTERVAL_SECONDS", 0, 0, 900),
background_refresh_retry_seconds=_env_int("MARKETGUARD_BACKGROUND_REFRESH_RETRY_SECONDS", 15, 5, 300),
readiness_fresh_grace_seconds=_env_int("MARKETGUARD_READINESS_FRESH_GRACE_SECONDS", 120, 0, 3600),
history_retention_days=_env_int("MARKETGUARD_HISTORY_RETENTION_DAYS", 45, 31, 365),
lowestbin_rate_limit_per_minute=_env_int("MARKETGUARD_LOWESTBIN_RATE_LIMIT_PER_MINUTE", 30, 0, 600),
players_rate_limit_per_minute=_env_int("MARKETGUARD_PLAYERS_RATE_LIMIT_PER_MINUTE", 3, 0, 60),
Expand All @@ -185,6 +215,10 @@ def from_env(cls) -> "MarketGuardSettings":
)
if settings.stale_if_error_seconds < settings.cache_ttl_seconds:
raise ValueError("MARKETGUARD_STALE_IF_ERROR_SECONDS must be greater than or equal to CACHE_TTL_SECONDS.")
if settings.readiness_fresh_grace_seconds > settings.stale_if_error_seconds:
raise ValueError(
"MARKETGUARD_READINESS_FRESH_GRACE_SECONDS must not exceed MARKETGUARD_STALE_IF_ERROR_SECONDS."
Comment on lines +218 to +220

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Bound the full readiness window by the stale cutoff

For allowed settings such as a 300-second cache TTL, 300-second stale cutoff, and 300-second grace, this validation passes even though readiness_fresh_seconds becomes 600 seconds. At snapshot age 400, /api/v1/ready therefore returns 200 while the data services refuse to serve that snapshot after an upstream failure and return 503. Compare cache_ttl_seconds + readiness_fresh_grace_seconds against the stale cutoff, or cap the readiness threshold, so unusable data is never reported as healthy.

Useful? React with 👍 / 👎.

)
if settings.redis_enabled and not settings.redis_url:
raise ValueError("MARKETGUARD_REDIS_ENABLED requires a usable Redis URL or host/port configuration.")
return settings
14 changes: 14 additions & 0 deletions app/marketguard_api/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
from .config import MarketGuardSettings
from .player_service import PlayerService
from .rate_limit import InMemoryRateLimiter
from .refresher import build_refresh_supervisor
from .routes import register_marketguard_routes
from .service import BazaarService, LowestBinService
from .storage import MarketGuardStorage
Expand Down Expand Up @@ -47,15 +48,27 @@ def create_marketguard_app(
runtime_bazaar_service = bazaar_service or BazaarService(runtime_settings, storage=storage)
runtime_player_service = player_service or PlayerService(runtime_settings)
persistent_api_metrics = PersistentApiMetricsRecorder(runtime_settings.database_url)
refresh_supervisor = build_refresh_supervisor(
runtime_settings,
lowestbin_service=lowestbin_service,
bazaar_service=runtime_bazaar_service,
)
docs_url = "/docs" if runtime_settings.api_docs_enabled else None
redoc_url = "/redoc" if runtime_settings.api_docs_enabled else None
openapi_url = "/openapi.json" if runtime_settings.api_docs_enabled else None

@asynccontextmanager
async def app_lifespan(_: FastAPI):
if refresh_supervisor is not None:
await refresh_supervisor.start()
try:
yield
finally:
if refresh_supervisor is not None:
try:
await refresh_supervisor.stop()
except Exception:
logger.exception("Could not stop the MarketGuard snapshot refreshers during shutdown.")
try:
await run_in_threadpool(persistent_api_metrics.flush)
except Exception:
Expand Down Expand Up @@ -85,6 +98,7 @@ async def app_lifespan(_: FastAPI):
app.state.marketguard_response_cache = runtime_response_cache
app.state.live_api_metrics = LiveApiRequestMetrics()
app.state.persistent_api_metrics = persistent_api_metrics
app.state.marketguard_refresh_supervisor = refresh_supervisor

def _require_internal_observability_access(request: Request) -> None:
if _request_originates_from_internal_network(request, runtime_settings.trusted_proxies):
Expand Down
Loading