Added check for certificate expiration issue

[asraf-khan]

Detection logic:

1. Find the fault using below query
   'faultInst.json?query-target-filter=or(eq(faultInst.code,"F4501"),eq(faultInst.code,"F4502"),eq(faultInst.code,"F4503"),eq(faultInst.code,"F3081"),eq(faultInst.code,"F3082"),eq(faultInst.code,"F4617"),eq(faultInst.code,"F4752"),eq(faultInst.code,"F4753"))'

2. if one fault F4501, F3081, F4617, F4752 found and state is “raised” or “soaking”
   Result = Manual & recommended_action ="Renew the certificate(s) before it expires to avoid service disruption."

3. if one fault F4502, F4503, F3082, F4753 and state is “raised” or “soaking”
   Result = FAIL_O & recommended_action="Renew the certificate(s) immediately to restore functionality."

4. if faults found as combination (e.g. expiring F3081 & expired F4502 ) means
   Result = FAIL_O & recommended_action="Renew expired certificate(s) immediately. For certificate(s) approaching expiry, renew before they expire to avoid service disruption."

Faults            Short Description                    Introduced         First build
F3081, F3082      SAML encryption cert expiring/expired     2018-02-22        3.1(2f)
F4501, F4502      KeyRing cert expiring/expired             2023-09-13        6.0(4c)
F4503, F4617      TP cert expired/expiring                  2024-04-22        6.1(1e)
F4752,F4753 Factory cert expiring/expired 6.1(5e)

For Factory certificate status if cversion < 6.1(5e), check via cli command "acidiag verifyapic"

Fixes #408

[asraf-khan]

Attached log for reference.

APIC Log :

APIC Run log - SSL Cert expiry issue.docx

Pytest Log:

Pytest log - SSL Cert expiry issue.docx

[asraf-khan]

Attaching latest APIC & Pytest logs

APIC Logs:

APIC Run log - SSL Cert expiry issue.docx

Pytest Logs:

Pytest log - SSL Cert expiry issue.docx

[thjonson]

This is great — thanks for adding fault-based cert expiration detection. One
gap I noticed: this approach only surfaces an issue once ACI has already
raised the fault (F4501/F4502/etc.), which means the earliest signal is
whatever threshold ACI's fault logic uses internally, and it doesn't cover
node SSL certs the way pkiFabricNodeSSLCertificate does.

As a complementary check (not a replacement), I've been running a script
that queries pkiFabricNodeSSLCertificate directly (joined with
fabricNode for name/model/serial/role) and computes days-to-expiry per
node, independent of whether a fault has fired yet:

• EXPIRED — already past validityNotAfter
• CRITICAL — expires within ~90 days
• WARN — expires within ~365 days (well ahead of any fault threshold)
• Flags certs not chained to "Cisco Manufacturing CA"
• Flags notAfter in year 2099, which matches FN-72339 (renewed-SUDI
  signature) — not something the fault codes above catch, since it isn't a
  fault condition, just a suspicious cert attribute worth a manual review.

Since fabric node SSL cert expiry/re-validation failures are a distinct
failure mode from the SAML/KeyRing/TP/Factory certs this PR covers (nodes
stay Inactive on reboot if their SSL cert fails re-validation), would it
make sense to add this as an additional check in this same PR, or should it
be a separate follow-on check? Happy to share the reference script/logic if
useful.

[asraf-khan]

@thjonson Thanks for sharing your suggesstions.

In this PR itself added validation for leaf/spine certificate expiration using MO's "pkiFabricNodeSSLCertificate" irrespective of faults and alert the customer if it's expired and expiring (expires within ~30 days) which matched the days with expired & expiring fault codes and we maintain same with faults expiring & MO's based expiring.

can you please provide details for below item ? how to verify ? what is the impact of this customer will face ? what is the recommended_action we have to provide to customer if they face this ?

Flags notAfter in year 2099, which matches FN-72339 (renewed-SUDI
signature) — not something the fault codes above catch, since it isn't a
fault condition, just a suspicious cert attribute worth a manual review.

[asraf-khan]

@monrog2 this PR still hold validation for APIC Side cert expiry validation. Only leaf/spine cert validation has been reverted.

[thjonson]

Quick clarification first: looking at the current diff, I don't see the
leaf/spine pkiFabricNodeSSLCertificate check in the PR right now —
certificate_expiration_check() currently only queries faultInst for
F4501/F4502 (KeyRing), F4503/F4617 (TP), F3081/F3082 (SAML), and F4752/
F4753 (Factory), gated by fault_min_versions. I think this lines up with
the "Removing leaf/spine certs check from validation" commit — sounds like
the pkiFabricNodeSSLCertificate-based leaf/spine check you described got
reverted before I saw your reply. So just flagging that the gap I raised
(no direct MO polling, and fault-based detection is version-gated to when
each fault code was introduced) still applies to the PR as it stands today
— let me know if that leaf/spine check is coming back in a follow-up.

On to FN-72339 specifically:

I hadn't realized your earlier changes had polled
pkiFabricNodeSSLCertificate via moquery directly rather than relying on
the fault codes exclusively — that's a meaningful advantage worth keeping
in mind for a future pass, since it also covers ACI versions before
F4501/F4502/etc. were introduced (e.g. pre-3.1(2f) for the SAML fault,
pre-6.0(4c) for KeyRing, etc.), where fault-based detection can't fire
because the fault doesn't exist yet on that code train. Polling the object
directly sidesteps that version-gating problem entirely.

How to verify FN-72339

FN-72339 hits switches manufactured after June 2020 (dual-cert ACT2 chip)
that get downgraded to 4.1(1i)/14.1(1i). On an affected node, moquery -c pkiFabricNodeSSLCertificate shows two tells together:

1. validityNotAfter = a date in 2099 (bug CSCvv44023 — the switch
   derives the wrong cert as server.crt, which carries a 2099 expiry as
   part of its structure)
2. message = a parse error along the lines of "failure to parse the
   subject line"

My script checks for these independently, as flags rather than a single
hard rule, because either one alone doesn't necessarily mean broken:

• !2099 fires when validityNotAfter contains "2099"
• !msg fires whenever the message attribute is non-empty (any parse/
  validation error text)
• !issuer fires when the cert issuer doesn't chain to "Cisco
  Manufacturing CA" (a broader integrity check, not FN-72339-specific)

The reason I don't treat a lone !2099 as an automatic FAIL: a
legitimately-renewed SUDI cert can also carry a far-future notAfter as
part of a normal renewal, so 2099 by itself is "worth a look," not
"broken." !2099 combined with !msg (or !issuer) is the real
FN-72339 signature — that combination is what indicates the switch has an
actual identity/discovery problem, not just an unusual-but-valid cert.

Recommended action / remediation

• If !2099 alone (no !msg, chains to Cisco Manufacturing CA): informational only, no action required — likely a legitimately renewed cert.
• If !2099 + !msg (and/or !issuer): flag as CRITICAL / do not upgrade or reboot this node yet. Per FN-72339, the fix is to get the node onto 4.1(1j)/14.1(1j) or later before the next reload, then let it do a clean reload so it re-derives the certificate correctly from the ACT2 chip. If the node is already stuck (failed discovery / already rebooted into this state), open a TAC case referencing FN-72339 / CSCvv44023 rather than attempting to force cert regeneration manually.

Let me know if it'd help to see the actual bash logic — it's about 10 lines, happy to paste it inline here.

[Priyanka-Patil14]

Using severity attribute to classify expired vs expiring is fragile. Better to check the fault code directly against the known expired/expiring sets.

[Priyanka-Patil14]

No test covers the case where fault API returns expired certs (has_critical=True) AND SSH to a controller fails (has_error=True) at the same time. In that scenario the code returns ERROR and hides the confirmed FAIL_O finding.

[Priyanka-Patil14]

No test covers lc: soaking state - the detection spec says soaking faults must also be caught but all fixtures only use raised.