checks: report APIC database collection errors

[monrog2]

Summary

• prevent apic_database_size_check from reporting PASS when legacy mitmocounters files are missing, empty, unavailable, or malformed
• retry each legacy counter read up to three times with a one-second delay to tolerate transient /debug population
• capture and report the final command error instead of suppressing stderr
• return structured ERROR details while preserving any oversized object-count findings already collected
• move filtering and numeric top-four sorting into Python so shell pipelines cannot mask collection failures
• add regression coverage for persistent and transient failures, partial collection, malformed rows, sorting, and the 1,500,000 threshold

Why

On affected pre-6.1(3a) releases, a missing mitmocounters/mo file caused cat to fail while the final sort process returned success. The check then received no rows and incorrectly reported PASS. Live 6.0(8e) testing also showed that the dynamic /debug files can be transiently unavailable even though they are subsequently readable by the same techsupport user.

Validation

• focused tests: 17 passed
• full local suite: 923 passed
• internal Python 2.7 job: passed
• internal Python 3.8 job: passed
• live integration pipeline fix: Handle missing tver in SUP HW Rev check #168 produced 16 debug bundles
• APIC 6.0(8e) bundle: all four legacy counter reads succeeded and the database-size check passed

The overall integration job remains red due to unrelated existing fabric/check failures.

Fixes #327

[monrog2]

Live validation findings on APIC 6.0(8e):

• The updated check correctly stopped the previous false PASS and reported all four legacy DME counter reads as collection errors.
• The generated debug bundle showed each direct cat /debug/<apic>/.../mitmocounters/mo returning exit status 1 within the same run.
• Follow-up checks confirmed the paths are readable cross-APIC as both admin and the same techsupport account used by integration, ruling out a persistent path or permission problem.
• The debug files are dynamically populated and were readable on retry, indicating transient /debug availability.

I am updating this PR to retry each legacy counter read a bounded number of times, capture the final command error instead of discarding stderr, and retain ERROR behavior when all attempts fail.

[monrog2]

Integration validation update:

• Pipeline fix: Handle missing tver in SUP HW Rev check #168 ran commit 029d4d4 with debug-bundle collection enabled.
• Python 2.7 and Python 3.8 jobs passed.
• The integration job archived 16 .tgz bundles.
• On the same APIC 6.0(8e fabric that previously returned exit status 1 for all four reads, the new bundle shows all four cat .../mitmocounters/mo 2>&1 commands succeeding on their first attempt.
• apic_database_size_check.json reports ruleStatus: passed; no retry warnings were required in this run.

This validates the successful collection path in the live environment. Unit coverage separately verifies fail-once/succeed-on-retry and persistent-failure behavior. The overall integration job remains red for unrelated existing fabric/check failures.

[monrog2]

Integration Test Summary

Pipeline executed against seven current/target version combinations.

• Python 2.7 and Python 3.8 unit-test jobs passed.
• All seven integration jobs completed but reported failures from existing fabric validation findings.
• The APIC Database Size check produced:
  • 106 passes
  • 10 non-passing results
  • 1 run without a result artifact

The retry logic was exercised in six runs. Each retried an unavailable counter file three times, reported the persistent collection error, and preserved valid data collected from other DME services.

Other non-passing results consisted of:

• Two malformed counter-data responses
• One API timeout
• One legitimate database-size threshold violation

No integration run encountered an empty successful response, so that specific retry path remains covered by unit tests. Overall, the results confirm the original retry behavior works as expected and does not discard successfully collected findings.