CDAP-21261 : Add retry logging for GCP Secret Manager SDK calls

[sahusanket]

Summary

Adds retry observability to CloudSecretManagerClient by registering a GAX BaseApiTracerFactory on SecretManagerServiceSettings.

Motivation

In #16227, GAX retries were enabled on CloudSecretManagerClient RPCs (updateSecret, getSecret, addSecretVersion) to handle transient errors and rate limiting (RESOURCE_EXHAUSTED / HTTP 429). However, GAX performs retries silently (only logging at java.util.logging FINEST), leaving no visibility in CDAP logs when Secret Manager quota limits are hit—either when a retry eventually recovers or when all retry attempts are exhausted.

Changes

Registers a BaseApiTracer on SecretManagerServiceSettings to log retry events at appropriate severity levels without polluting normal operational logs:

• WARN on RESOURCE_EXHAUSTED retry (attemptFailed): Logs when a Secret Manager call hits a quota limit and is being retried, providing early visibility for alerting and capacity planning even if a subsequent retry succeeds.
• DEBUG on transient 5XX retry (attemptFailed): Logs non-quota transient failures (UNAVAILABLE, INTERNAL, UNKNOWN, DEADLINE_EXCEEDED) at DEBUG level to avoid log noise.
• ERROR on retry exhaustion (attemptFailedRetriesExhausted): Logs the RPC method name and failure cause when all configured retry attempts are exhausted.

[gemini-code-assist]

Code Review

This pull request configures a custom tracer factory for the GCP Secret Manager client to log transient failures and exhausted retries. Feedback focuses on improving the logging implementation: first, by avoiding logging the full stack trace for transient quota limit warnings to prevent log flooding, and second, by removing a redundant and potentially unsafe call to error.getMessage() when logging exhausted retries.

[gemini-code-assist]

Logging the full stack trace (error) at WARN level for every transient quota limit hit can severely flood the logs, especially under high load when quota limits are frequently exceeded. Since this is a transient failure that is being retried, it is better to log a concise warning message with the exception's string representation and avoid printing the entire stack trace. The full stack trace will still be logged at ERROR level if all retry attempts are exhausted.

Suggested change

	LOG.warn("GCP Secret Manager quota limit hit during {}, retrying in {} ms.",
	spanName.getMethodName(), delay.toMillis(), error);
	LOG.warn("GCP Secret Manager quota limit hit during {}, retrying in {} ms. Error: {}",
	spanName.getMethodName(), delay.toMillis(), error.toString());

[gemini-code-assist]

Passing error.getMessage() as a separate placeholder is redundant because the error object is already passed as the last argument, which automatically logs the exception's class, message, and full stack trace. Additionally, if error is ever null, calling error.getMessage() will throw a NullPointerException. Simplifying the log statement avoids redundancy and potential NPEs.

Suggested change

	LOG.error("GCP Secret Manager retries exhausted for {}: {}",
	spanName.getMethodName(), error.getMessage(), error);
	LOG.error("GCP Secret Manager retries exhausted for {}",
	spanName.getMethodName(), error);

[vsethi09]

nit: GCP Secret Manager API request quota limit exceeded during...

[vsethi09]

nit: Retries for GCP Secret Manager API Request exhausted for..

[sonarqubecloud]

Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud