diff --git a/elastic-stack-setup/elastic-stack-setup-02-environment-watch/elastic-stack-rotate-api-key-environment-watch.md b/elastic-stack-setup/elastic-stack-setup-02-environment-watch/elastic-stack-rotate-api-key-environment-watch.md new file mode 100644 index 0000000..e120679 --- /dev/null +++ b/elastic-stack-setup/elastic-stack-setup-02-environment-watch/elastic-stack-rotate-api-key-environment-watch.md @@ -0,0 +1,232 @@ +# Rotate an Elasticsearch API Key using the Relativity Server CLI + +The `rotate-api-key` command creates a new Elasticsearch API key for the specified cluster, persists it to the Relativity Secret Store, and invalidates the old key. Run this command periodically to rotate expiring keys or as part of a scheduled security practice. + +> [!NOTE] +> It is recommended to run the CLI from the Primary SQL Server. + +> This guide assumes the Relativity Server bundle was extracted to `C:\Server.Bundle.x.y.z` or a similar directory chosen by the user. + +## Prerequisites + +- The Server-bundle zip file has been downloaded and extracted to `C:\Server.Bundle.x.y.z` +- Access to the Relativity Secret Store (Whitelisted for Secret Store access. Please see [here](https://help.relativity.com/Server2025/Content/System_Guides/Secret_Store/Secret_Store.htm#Configuringclients) for information on whitelisting.) +- Elasticsearch is running and accessible +- The initial Environment Watch setup has been completed. See [Set up Environment Watch using the Relativity Server CLI](./elastic-stack-setup-02-environment-watch.md) + +## Options + +| Flag | Short alias | Description | Default | +|------|-------------|-------------|---------| +| `--cluster ` | `-c` | Target Elasticsearch cluster. Valid values: `rel-cluster-infrawatch`, `rel-cluster-datagrid` | Prompted interactively | +| `--quiet` | | Suppress confirmation and expiry prompts; auto-confirms with the default 180-day expiry. Use together with `--cluster` for fully unattended execution | `false` | +| `--dryrun` | | Preview what would happen without making any changes to Elasticsearch or the Secret Store | `false` | + +## Usage + +### Interactive + +Running `rotate-api-key` without any flags launches an interactive session. The CLI prompts you to select a target cluster. + +``` +C:\Server.Bundle.x.y.z\relsvr.exe rotate-api-key + +Relativity Server CLI - 102.1.18 +Copyright (c) 2026, Relativity ODA LLC + +Select the Elasticsearch cluster to rotate the API key for: + +> rel-cluster-infrawatch + rel-cluster-datagrid +``` + +After selecting a cluster, the CLI displays the current API key ID and the number of days remaining before expiry, then asks you to confirm the rotation. Entering `n` aborts with no changes made: + +``` +Key rotation aborted. No changes were made. +``` + +Entering `y` continues to prompt for a validity period in days, then performs the rotation. + +**InfraWatch cluster:** + +``` +Current API key ID: g9tMUp8BVmBEyCHxvRon (name: rel-infrawatch) +Days remaining before expiry: 179 +Rotate API key for cluster 'rel-cluster-infrawatch'? This will invalidate the current key. [y/n] (n): y +How many days should the new key be valid? (180): 180 + +API key rotation completed ----------------------------------------- 100% + +Successfully rotated the Elasticsearch API key and persisted the new key to the Secret Store. +``` + +**DataGrid cluster:** + +``` +Current API key ID: idtNUp8BVmBEyCHxYC02 (name: rel-datagrid) +Days remaining before expiry: 179 +Rotate API key for cluster 'rel-cluster-datagrid'? This will invalidate the current key. [y/n] (n): y +How many days should the new key be valid? (180): 180 + +API key rotation completed ----------------------------------------- 100% + +Successfully rotated the Elasticsearch API key and persisted the new key to the Secret Store. +``` + +### Rotate with a pre-selected cluster + +Use `--cluster` to target a specific cluster directly, skipping the cluster selection menu. The CLI still displays the current key information and asks for confirmation before rotating. + +``` +C:\Server.Bundle.x.y.z\relsvr.exe rotate-api-key --cluster rel-cluster-infrawatch + +Relativity Server CLI - 102.1.18 +Copyright (c) 2026, Relativity ODA LLC + +Current API key ID: GUgcV58BbZgP437js08h (name: rel-infrawatch) +Days remaining before expiry: 179 +Rotate API key for cluster 'rel-cluster-infrawatch'? This will invalidate the current key. [y/n] (n): y +How many days should the new key be valid? (180): 180 + +API key rotation completed ----------------------------------------- 100% + +Successfully rotated the Elasticsearch API key and persisted the new key to the Secret Store. +``` + +``` +C:\Server.Bundle.x.y.z\relsvr.exe rotate-api-key --cluster rel-cluster-datagrid + +Relativity Server CLI - 102.1.18 +Copyright (c) 2026, Relativity ODA LLC + +Current API key ID: gUkdV58BbZgP437jHQNQ (name: rel-datagrid) +Days remaining before expiry: 179 +Rotate API key for cluster 'rel-cluster-datagrid'? This will invalidate the current key. [y/n] (n): y +How many days should the new key be valid? (180): 180 + +API key rotation completed ----------------------------------------- 100% + +Successfully rotated the Elasticsearch API key and persisted the new key to the Secret Store. +``` + +Use the `-c` short alias to achieve the same result: + +``` +C:\Server.Bundle.x.y.z\relsvr.exe rotate-api-key -c rel-cluster-infrawatch + +Relativity Server CLI - 102.1.18 +Copyright (c) 2026, Relativity ODA LLC + +Current API key ID: GUgcV58BbZgP437js08h (name: rel-infrawatch) +Days remaining before expiry: 179 +Rotate API key for cluster 'rel-cluster-infrawatch'? This will invalidate the current key. [y/n] (n): y +How many days should the new key be valid? (180): 180 + +API key rotation completed ----------------------------------------- 100% + +Successfully rotated the Elasticsearch API key and persisted the new key to the Secret Store. +``` + +### Quiet mode (automated / scripted rotation) + +Combining `--quiet` with `--cluster` suppresses all prompts, auto-confirms the rotation, and uses the default 180-day expiry. This is suitable for scheduled or unattended scripts. + +``` +C:\Server.Bundle.x.y.z\relsvr.exe rotate-api-key --quiet --cluster rel-cluster-infrawatch + +Relativity Server CLI - 102.1.18 +Copyright (c) 2026, Relativity ODA LLC + +Current API key ID: 5UkeV58BbZgP437jfiJ8 (name: rel-infrawatch) +Days remaining before expiry: 179 + +API key rotation completed ----------------------------------------- 100% + +Successfully rotated the Elasticsearch API key and persisted the new key to the Secret Store. +``` + +``` +C:\Server.Bundle.x.y.z\relsvr.exe rotate-api-key --quiet --cluster rel-cluster-datagrid + +Relativity Server CLI - 102.1.18 +Copyright (c) 2026, Relativity ODA LLC + +Current API key ID: gUkdV58BbZgP437jHQNQ (name: rel-datagrid) +Days remaining before expiry: 179 + +API key rotation completed ----------------------------------------- 100% + +Successfully rotated the Elasticsearch API key and persisted the new key to the Secret Store. +``` + +### Dry run + +Use `--dryrun` to simulate the rotation without making any changes. The CLI displays the current key information, accepts the same prompts as a normal rotation, then confirms the simulation without writing to Elasticsearch or the Secret Store. + +``` +C:\Server.Bundle.x.y.z\relsvr.exe rotate-api-key --cluster rel-cluster-infrawatch --dryrun + +Relativity Server CLI - 102.1.18 +Copyright (c) 2026, Relativity ODA LLC + +Current API key ID: 5UkeV58BbZgP437jfiJ8 (name: rel-infrawatch) +Days remaining before expiry: 179 +Rotate API key for cluster 'rel-cluster-infrawatch'? This will invalidate the current key. [y/n] (n): y +How many days should the new key be valid? (180): 180 +DryRun mode: No changes were committed. The API key rotation was simulated successfully. +``` + +### Invalid cluster value + +If an unrecognized value is passed to `--cluster`, the CLI rejects it immediately and lists the valid options. + +``` +C:\Server.Bundle.x.y.z\relsvr.exe rotate-api-key --cluster infrawatch + +Relativity Server CLI - 102.1.18 +Copyright (c) 2026, Relativity ODA LLC + +Invalid --cluster value 'infrawatch'. Valid values are: rel-cluster-infrawatch, rel-cluster-datagrid +``` + +## Verify the rotation + +### Kibana API keys + +1. In Kibana, navigate to **Stack Management** > **Security** > **API keys**. +2. Confirm a new key for the rotated cluster appears at the top of the list with a recent creation timestamp and an expiry approximately 180 days in the future. + +![Kibana API keys page showing newly rotated rel-infrawatch and rel-datagrid keys](../../resources/RotateApiKey-Kibana-After.png) + +### Secret Store + +To confirm the new API key was persisted, read the secret for the rotated cluster using the Secret Store client. To compare values, read the secret both before and after running `rotate-api-key` — the `api-key` value should differ between the two reads. + +- **InfraWatch secret path:** `/database/elasticsearch/clusters/rel-cluster-infrawatch/security/api-keys/rel-infrawatch` +- **DataGrid secret path:** `/database/elasticsearch/clusters/rel-cluster-datagrid/security/api-keys/rel-datagrid` + +### Elasticsearch Dev Tools (optional) + +To confirm that the old key has been invalidated and the new key is active, query the Elasticsearch security API in Kibana Dev Tools using the key ID. + +1. In Kibana, navigate to **Dev Tools** > **Console**. +2. Run the following query, replacing `` with the ID of the key to inspect: + + ``` + GET /_security/api_key?id= + ``` + +3. Verify the results: + - The old key shows `"invalidated": true`. + - The new key shows `"invalidated": false`. + +**Old key — invalidated:** + +![Old API key showing invalidated: true in Elasticsearch Dev Tools](../../resources/RotateApiKey-InfraWatch-DevTools-OldKey.png) + +**New key — active:** + +![New API key showing invalidated: false in Elasticsearch Dev Tools](../../resources/RotateApiKey-InfraWatch-DevTools-NewKey.png) + +Refer to the [Troubleshooting Guide](../troubleshooting/relativity-server-cli.md) if you encounter any issues. diff --git a/elastic-stack-setup/elastic-stack-setup-03-audit.md b/elastic-stack-setup/elastic-stack-setup-03-audit.md index adfb75f..bfe39af 100644 --- a/elastic-stack-setup/elastic-stack-setup-03-audit.md +++ b/elastic-stack-setup/elastic-stack-setup-03-audit.md @@ -191,3 +191,17 @@ Follow these steps to set up Data Grid Audit using the Relativity Server CLI. Al - Recent audit events are visible and populating (new user actions appear within a few minutes). - No error banners or "Elasticsearch connection failed" messages are displayed. - Audit search returns results without errors. + +## API Key Expiry and Rotation + +The Elasticsearch API key created during setup is valid for 180 days by default. Rotate the key before it expires to avoid authentication failures on the Audit tab. + +To rotate the DataGrid API key, use the `rotate-api-key` command from the Relativity Server CLI: + +``` +C:\Server.Bundle.x.y.z\relsvr.exe rotate-api-key --cluster rel-cluster-datagrid +``` + +See [Rotate an Elasticsearch API Key using the Relativity Server CLI](./elastic-stack-setup-02-environment-watch/elastic-stack-rotate-api-key-environment-watch.md) for full instructions, including quiet mode for scripted rotation and how to verify the new key in Kibana and the Secret Store. + +Refer to the [Data Grid Audit Troubleshooting Guide](./troubleshooting/datagrid-audit-troubleshooting.md) if the Audit tab does not load data after rotating the key. diff --git a/elastic-stack-setup/troubleshooting/datagrid-audit-troubleshooting.md b/elastic-stack-setup/troubleshooting/datagrid-audit-troubleshooting.md new file mode 100644 index 0000000..fe86306 --- /dev/null +++ b/elastic-stack-setup/troubleshooting/datagrid-audit-troubleshooting.md @@ -0,0 +1,39 @@ +# Data Grid Audit Troubleshooting + +This document provides troubleshooting guidance for common issues encountered with the Data Grid Audit tab in Relativity after initial setup or API key rotation. + +## Audit Tab Does Not Load After API Key Rotation + +### Symptoms + +After rotating the DataGrid cluster API key using the `rotate-api-key` command, the Audit tab may display one of the following errors: + +- **"Authentication failed for user to access Elasticsearch. Please check the permissions for the user."** — appears immediately after key rotation, before Relativity services have been restarted. +- **"This chart did not return any results."** — appears after restarting Relativity services, while Elasticsearch is still applying the new key. + +### Cause + +When the API key is rotated, the new key is persisted to the Relativity Secret Store. Relativity services must be restarted to read the updated key from the Secret Store. In some cases, the Elasticsearch service also requires a restart to fully accept connections using the new key. + +### Resolution + +1. After rotating the DataGrid API key, navigate to **Audit** > **Audit** in Relativity. + + If the page shows *"Authentication failed for user to access Elasticsearch. Please check the permissions for the user."*, proceed to step 2. + +2. Open **Services** (`services.msc`) and restart the following Relativity services on **all servers** in the Relativity instance: + - `kCura Edds Agent Manager` + - `kCura Edds Web Processing Manager` + - `kCura Service Host Manager` + +3. Refresh the Audit tab in the browser. The error may change to *"This chart did not return any results."* — this indicates Relativity can now reach Elasticsearch but data has not yet loaded. + +4. On the Elasticsearch server, open **Services** and restart the **Elasticsearch** service. + +5. Wait **1–5 minutes** for Elasticsearch to fully restart and for Relativity to re-establish the connection. + +6. Reload the Relativity UI and navigate back to the **Audit** tab. Audit data should now display correctly. + +> [!NOTE] +> If the Audit tab still does not load after following these steps, verify that the new API key was successfully persisted to the Secret Store. See [Rotate an Elasticsearch API Key using the Relativity Server CLI](../elastic-stack-setup-02-environment-watch/elastic-stack-rotate-api-key-environment-watch.md#verify-the-rotation) for Secret Store and Kibana verification steps. + diff --git a/elastic-stack-setup/troubleshooting/monitoring-agent-and-otel-collector.md b/elastic-stack-setup/troubleshooting/monitoring-agent-and-otel-collector.md index a58c1ed..574cb7b 100644 --- a/elastic-stack-setup/troubleshooting/monitoring-agent-and-otel-collector.md +++ b/elastic-stack-setup/troubleshooting/monitoring-agent-and-otel-collector.md @@ -234,6 +234,31 @@ If the above steps do not resolve the issue, verify the following access and con For service account requirements and troubleshooting, see [Environment_Watch_Installer](../elastic-stack-setup-02-environment-watch/ew-01-install-monitoring-agents.md) +## Environment Watch Not Working After API Key Rotation + +If the InfraWatch cluster API key has been newly created or rotated using the `rotate-api-key` command and Environment Watch dashboards are no longer updating or showing data, follow these steps. + +See [Rotate an Elasticsearch API Key using the Relativity Server CLI](../elastic-stack-setup-02-environment-watch/elastic-stack-rotate-api-key-environment-watch.md) for instructions on rotating the API key. + +1. Restart the **Relativity Environment Watch** Windows service on each monitored server: + ```powershell + Restart-Service -Name "Relativity Environment Watch" + ``` + +2. Wait **2–5 minutes** for the service to reinitialize and begin sending data. + +3. Open Kibana and check the **Monitoring Agents** dashboard to confirm hosts are reporting. If data is still not appearing, proceed to step 4. + +4. If the issue persists, restart the Elastic Stack services on the Elasticsearch server: + 1. **Elasticsearch** — open **Services** (`services.msc`) and restart the Elasticsearch service. + 2. **Kibana** — restart the Kibana service. + 3. **APM Server** — restart the APM Server service. + +5. Wait a few minutes and reload the Kibana dashboards. Environment Watch data should resume populating. + +> [!NOTE] +> If data still does not appear after restarting all services, verify that the rotated API key was successfully persisted to the Secret Store. See the [Verify the rotation](../elastic-stack-setup-02-environment-watch/elastic-stack-rotate-api-key-environment-watch.md#verify-the-rotation) section for Secret Store and Kibana verification steps. + ## Installer and Service Errors This section covers issues related to the Environment Watch installer and the underlying Windows services it manages. diff --git a/resources/RotateApiKey-InfraWatch-DevTools-NewKey.png b/resources/RotateApiKey-InfraWatch-DevTools-NewKey.png new file mode 100644 index 0000000..d7a7f9a Binary files /dev/null and b/resources/RotateApiKey-InfraWatch-DevTools-NewKey.png differ diff --git a/resources/RotateApiKey-InfraWatch-DevTools-OldKey.png b/resources/RotateApiKey-InfraWatch-DevTools-OldKey.png new file mode 100644 index 0000000..2e2bf0f Binary files /dev/null and b/resources/RotateApiKey-InfraWatch-DevTools-OldKey.png differ diff --git a/resources/RotateApiKey-Kibana-After.png b/resources/RotateApiKey-Kibana-After.png new file mode 100644 index 0000000..7a16caf Binary files /dev/null and b/resources/RotateApiKey-Kibana-After.png differ