scep: size the signer key DER buffer from the key

[yosuke-wolfssl]

Problem

rsa_key_to_der (src/scep/scep_client.c) allocated a fixed 2048-byte buffer for wc_RsaKeyToDer, while every other serializer in the tree sizes from the key (keygen.c, csr.c, ca_issue.c). rsa_make accepts 2048|3072|4096 and the CLI advertises --key-type rsa:BITS, so --key-type rsa:4096 --proto scep died with a bare WOLFCERT_ERR_CRYPTO before sending a byte — an advertised configuration dead on arrival.

Key	PKCS#1 DER	Fits 2048 B?
RSA-2048	1191 B	yes
RSA-3072	1769 B	yes
RSA-4096	2348 B	no — BUFFER_E

The finding text also claims RSA-3072 overflows. It does not; only 4096 was broken.

Fix (src/scep/scep_client.c)

• Capacity derives from the key as bits + 2048, the same idiom keygen.c:229 and csr.c:368 already use. It falls back to 4096 bits because rsa_priv_decode leaves rsa_bits at 0 for a key loaded from DER/PEM, so the fallback is load-bearing rather than decorative.
• WOLFCERT_ERR_WC replaces the bare WOLFCERT_ERR_CRYPTO on the wc_RsaKeyToDer failure path, so a future sizing failure is diagnosable.

The helper is the single choke point for all six SCEP serialization sites, so the blocking and non-blocking session entry points are both covered by the one change.

Closes f-8011.

Test harness

check_rsa4096 in tests/integration/test_scep_roundtrip.c enrolls an RSA-4096 key over SCEP and verifies the issued certificate chains to the CA that answered, through a WOLFSSL_CERT_MANAGER — not a PEM-marker match. It follows the surrounding check_* helpers' fallthrough-rc and free-on-every-path discipline.

Verification

• Full suite 26/26; clean under -Werror.
• ASan + UBSan clean, 26/26 (Darwin, no LeakSanitizer).
• Negative control: the new check fails with rc=-9 against the unfixed helper.
• skoll review: 0 findings, APPROVE.

Not in this PR

rsa_priv_decode never sets rsa_bits, unlike ecc_priv_decode which repopulates curve_id. Fixing that properly means dropping the ? : 4096 guard from all three sites and failing when wc_RsaEncryptSize() returns <= 0 — dropping the guard alone reintroduces this bug. It is unfiled and wants its own finding, alongside 8014 (same class, ca_issue.c).

[Copilot]

Pull request overview

Fixes SCEP RSA-4096 private-key DER serialization and adds regression coverage.

Changes:

• Sizes RSA DER buffers dynamically.
• Improves serialization error reporting.
• Adds RSA-4096 enrollment and certificate-chain verification.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
tests/integration/test_scep_roundtrip.c	Adds RSA-4096 SCEP enrollment and chain verification coverage.
src/scep/scep_client.c	Dynamically sizes RSA DER buffers and improves diagnostics.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #18

Scan targets checked: wolfcert-bugs, wolfcert-src

No new issues found in the changed files. ✅