Replace global RSA methods with per-context callbacks

[mtrojnar]

Pull Request Type

• Bug fix
• New feature
• Code style / formatting / renaming
• Refactoring (no functional or API changes)
• Build / CI related changes
• Documentation
• Other (please describe):

Related Issue

Fixes #672

Current Behavior

On OpenSSL 3.x, PKCS11_get_private_key() registers libp11's custom RSA
EVP_PKEY_METHOD process-wide with EVP_PKEY_meth_add0().

That registration changes method selection for unrelated software RSA keys.
Provider-backed keys created before the PKCS#11 key can subsequently fail in
operations such as X509_sign() with:

error:03000093:digital envelope routines:default_check:command not supported

Software RSA keys created afterward can also silently fall back to legacy
implementations instead of remaining backed by the default provider.

The registration also crosses an architecture boundary: the libp11 library
initializes deprecated OpenSSL method integration that belongs to the ENGINE
component.

New Behavior

PKCS11_get_private_key() no longer registers an RSA EVP_PKEY_METHOD
globally. Loading a PKCS#11 key therefore does not alter method selection for
unrelated keys.

A new per-context callback API lets integrations customize each returned key:

PKCS11_CTX_set_pkey_callback(ctx, callback_type, callback, user_data);

The callback receives borrowed PKCS11_KEY and EVP_PKEY objects and may
customize the returned key in place. The callback must not free either
object; an error aborts key retrieval and libp11 releases the EVP_PKEY.

The ENGINE registers this callback and attaches its method dispatch only to
private keys loaded by that ENGINE. This preserves native RSA-PSS and
software-PSS-over-raw-RSA behavior without changing process-wide method
selection.

Provider key loading and dispatch remain unchanged; the provider does not
need the callback or an additional keydata handoff.

Scope of Changes

• Add and export PKCS11_CTX_set_pkey_callback() and the
  PKCS11_PKEY_CALLBACK_GET_PRIVATE_KEY event.
• Preserve callback registration when UTIL_CTX creates or reloads its
  internal libp11 context.
• Remove process-wide RSA EVP_PKEY_METHOD registration and its global state
  from the libp11 library.
• Select ENGINE PKEY methods per loaded private key from eng_* code.
• Add regression coverage for unrelated provider-backed software RSA keys,
  callback-selected ENGINE dispatch, and ENGINE signing on both raw-RSA-only
  and native-PSS-only tokens.

This adds a public libp11 API symbol and callback type. No existing public
symbol or function signature is removed.

Testing

• Existing tests
• New tests added
• Manual testing

Built with OpenSSL 3.6.3 using strict configuration and ran:

./configure --enable-strict
make
make check

Result: 51 total, 46 passed, 5 skipped due to unavailable ML-DSA/ML-KEM
features, 0 failed.

The software-key regression test verifies both an existing and a newly
created default-provider RSA key after PKCS11_get_private_key() loads a
token key.

The ENGINE scenario does not pass an ENGINE to EVP_PKEY_CTX_new(), so
success depends on the key-loading callback. It runs against raw-RSA-only and
native-PSS-only SoftHSM configurations with explicit SHA-256, MGF1, and
salt-length parameters.

Additional Notes

ENGINE-specific APIs remain confined to eng_* sources. No provider-specific
callback or keydata handoff is introduced, preserving the provider-only
OpenSSL 4 architecture.

License Declaration

• I hereby agree to license my contribution under the project's license.

[olszomal]

Looks good to me.