Keep PKCS5S2 usable on a FIPS-restricted JCE, and name the key wrapping property when the runtime has no RSA-OAEP (#1058)

Author: vharseko

opendj-doc-generated-ref/src/main/asciidoc/admin-guide/chap-import-export.adoc

@@ -12,7 +12,7 @@
   information: "Portions copyright [year] [name of copyright owner]".
 
   Copyright 2017 ForgeRock AS.
-  Portions Copyright 2025 3A Systems LLC.
+  Portions Copyright 2025-2026 3A Systems LLC.
 ////
 
 :figure-caption!:
@@ -228,9 +228,9 @@ replace: initials
 initials: AAA
 
 $ ldifmodify \
- --sourceLDIF generated.ldif \
- --changesLDIF changes.ldif \
- --targetLDIF new.ldif
+ --outputLDIF new.ldif \
+ generated.ldif \
+ changes.ldif
 ----
 Notice that the resulting new LDIF file is likely to be about the same size as the source LDIF file.
 

opendj-doc-generated-ref/src/main/asciidoc/install-guide/chap-install.adoc

@@ -789,6 +789,60 @@ At this point you can use OpenDJ directory server, or you can perform additional
 
 ====
 
+[#install-fips]
+.To Install OpenDJ Directory Server on a FIPS 140 Java Runtime
+====
+OpenDJ ships the Bouncy Castle FIPS provider. The `setup` command and the server register it themselves when the server key store is a BCFKS key store (`--useBcfksKeystore`), and the server registers it at start whenever the `org.openidentityplatform.opendj.fips.register` Java system property is `true`, for example through the `OPENDJ_JAVA_ARGS` environment variable. With this provider in its default mode the default configuration works as it is. The approved-only mode of the provider (the `org.bouncycastle.fips.approved_only` Java system property), which OpenDJ does not turn on, is not covered here.
+
+The crypto manager wraps the secret keys it shares with the other servers of a replication topology with each server's public key. The transformation it uses, the `key-wrapping-transformation` property of the crypto manager, is `RSA/ECB/OAEPWITHSHA-1ANDMGF1PADDING` by default: RSA-OAEP, the key transport scheme of NIST SP 800-56B, which Bouncy Castle FIPS provides. A Java runtime whose cryptography comes from a `SunPKCS11` provider alone, such as `SunPKCS11-NSS-FIPS` on a Linux system in FIPS mode, provides no RSA-OAEP at all: its only RSA cipher is `RSA/ECB/PKCS1Padding`, which NIST SP 800-131A Rev. 2 disallows for key transport. On such a runtime the server refuses to start with the default transformation, `setup` says so, and the choice of another one is yours to make: OpenDJ does not make it for you. If you make it, keep in mind that every server of a replication topology must use the same transformation, since each server unwraps what the others wrapped.
+
+. Install the server without starting it:
++
+
+[source, console]
+----
+$ ./setup --cli --doNotStart \
+ --hostname opendj.example.com \
+ --ldapPort 1389 \
+ --adminConnectorPort 4444 \
+ --rootUserDN "cn=Directory Manager" \
+ --rootUserPassword password \
+ --baseDN dc=example,dc=com \
+ --acceptLicense \
+ --no-prompt
+----
+
+. Set the transformation in the server configuration file, which the `dsconfig` command cannot change while the server is stopped:
++
+
+[source, console]
+----
+$ cat changes.ldif
+dn: cn=Crypto Manager,cn=config
+changetype: modify
+replace: ds-cfg-key-wrapping-transformation
+ds-cfg-key-wrapping-transformation: RSA/ECB/PKCS1Padding
+
+$ ldifmodify \
+ --outputLDIF /path/to/opendj/config/config.ldif.new \
+ /path/to/opendj/config/config.ldif \
+ changes.ldif
+
+$ mv /path/to/opendj/config/config.ldif.new /path/to/opendj/config/config.ldif
+----
+
+. Start the server:
++
+
+[source, console]
+----
+$ start-ds
+----
++
+Once the server runs, the `dsconfig set-crypto-manager-prop` command changes the property, and refuses a transformation the runtime does not support.
+
+====
+
 [#pdb-to-je]
 .To Move Data from a PDB Backend to a JE Backend
 ====
@@ -933,7 +987,7 @@ replace: ds-cfg-java-class
 ds-cfg-java-class: org.opends.server.backends.jeb.JEBackend
 EOF
 
-./bin/ldifmodify --targetLDIF "$LOC"/config/config.ldif.$$ --sourceLDIF "$LOC"/config/config.ldif --changesLDIF /tmp/changes_$$
+./bin/ldifmodify --outputLDIF "$LOC"/config/config.ldif.$$ "$LOC"/config/config.ldif /tmp/changes_$$
 if test $? -ne 0
 then
   echo "Modifications failed. Restoring the original configuration"

opendj-server-legacy/src/main/java/org/opends/quicksetup/installer/Installer.java

@@ -130,6 +130,7 @@
 import org.opends.server.backends.task.TaskState;
 import org.opends.server.tools.BackendTypeHelper;
 import org.opends.server.tools.BackendTypeHelper.BackendTypeUIAdapter;
+import org.opends.server.tools.ConfigureDS;
 import org.opends.server.types.HostPort;
 import org.opends.server.util.CertificateManager;
 import org.opends.server.util.CollectionUtils;
@@ -1363,6 +1364,13 @@ public void abort()
     };
     invokeLongOperation(thread);
     notifyListeners(getFormattedDoneWithLineBreak());
+    // Given here rather than by ConfigureDS, whose output the listeners do not see while it runs.
+    final LocalizableMessage keyWrappingWarning = ConfigureDS.unsupportedKeyWrappingTransformationWarning();
+    if (keyWrappingWarning != null)
+    {
+      notifyListeners(getFormattedWarning(keyWrappingWarning));
+      notifyListeners(getLineBreak());
+    }
     checkAbort();
     configureCertificate(sec);
   }

opendj-server-legacy/src/main/java/org/opends/server/crypto/CryptoManagerImpl.java

@@ -392,6 +392,9 @@ public boolean isConfigurationChangeAcceptable(
                         requestedKeyWrappingTransformation));
         isAcceptable = false;
       }
+      else if (!isKeyWrappingTransformationSupported(requestedKeyWrappingTransformation, unacceptableReasons)) {
+        isAcceptable = false;
+      }
       else {
         try {
           /* Note that the TrustStoreBackend not available at initial,
@@ -430,6 +433,29 @@ public boolean isConfigurationChangeAcceptable(
     return isAcceptable;
   }
 
+  /**
+   * Checks that this Java runtime provides the key wrapping transformation. Only a refusal here
+   * names the key-wrapping-transformation property: the wrap which follows it also needs an MD5
+   * digest and a 1024-bit RSA key, and changing the property does not help when one of those is
+   * what the runtime refuses.
+   */
+  private static boolean isKeyWrappingTransformationSupported(
+      final String transformation, final List<LocalizableMessage> unacceptableReasons)
+  {
+    try
+    {
+      Cipher.getInstance(transformation);
+      return true;
+    }
+    catch (GeneralSecurityException ex)
+    {
+      logger.traceException(ex);
+      unacceptableReasons.add(
+          ERR_CRYPTOMGR_KEY_WRAPPING_TRANSFORMATION_UNSUPPORTED.get(transformation, getExceptionMessage(ex)));
+      return false;
+    }
+  }
+
   @Override
   public ConfigChangeResult applyConfigurationChange(CryptoManagerCfg cfg)
   {

opendj-server-legacy/src/main/java/org/opends/server/extensions/AbstractPBKDF2PasswordStorageScheme.java

@@ -82,7 +82,8 @@ public void initializePasswordStorageScheme(PBKDF2PasswordStorageSchemeCfg confi
         }
         catch (NoSuchAlgorithmException e)
         {
-            throw new InitializationException(null);
+            throw new InitializationException(
+                ERR_PWSCHEME_CANNOT_INITIALIZE_MESSAGE_DIGEST.get(getMessageDigestAlgorithm(), e), e);
         }
 
         this.config = configuration;

opendj-server-legacy/src/main/java/org/opends/server/extensions/ExtensionsConstants.java

@@ -13,6 +13,7 @@
  *
  * Copyright 2006-2008 Sun Microsystems, Inc.
  * Portions copyright 2013-2016 ForgeRock AS.
+ * Portions Copyright 2026 3A Systems, LLC.
  */
 package org.opends.server.extensions;
 
@@ -153,11 +154,6 @@ public class ExtensionsConstants
   public static final String MESSAGE_DIGEST_ALGORITHM_PBKDF2_HMAC_SHA512 =
           "PBKDF2WithHmacSHA512";
 
-  /**
-   * The name of the pseudo-random number generator using SHA-1.
-   */
-  public static final String SECURE_PRNG_SHA1 = "SHA1PRNG";
-
 
 
   /**

opendj-server-legacy/src/main/java/org/opends/server/extensions/PKCS5S2PasswordStorageScheme.java

@@ -13,6 +13,7 @@
  *
  * Copyright 2014-2016 ForgeRock AS.
  * Portions Copyright 2014 Emidio Stani & Andrea Stani
+ * Portions Copyright 2026 3A Systems, LLC.
  */
 package org.opends.server.extensions;
 
@@ -85,13 +86,15 @@ public void initializePasswordStorageScheme(PKCS5S2PasswordStorageSchemeCfg conf
   {
     try
     {
-      random = SecureRandom.getInstance(SECURE_PRNG_SHA1);
+      // The provider's default random source: a FIPS-restricted JCE registers no SHA1PRNG.
+      random = new SecureRandom();
       // Just try to verify if the algorithm is supported
       SecretKeyFactory.getInstance(MESSAGE_DIGEST_ALGORITHM_PBKDF2);
     }
     catch (NoSuchAlgorithmException e)
     {
-      throw new InitializationException(null);
+      throw new InitializationException(
+          ERR_PWSCHEME_CANNOT_INITIALIZE_MESSAGE_DIGEST.get(MESSAGE_DIGEST_ALGORITHM_PBKDF2, e), e);
     }
   }
 
@@ -246,8 +249,7 @@ private static byte[] encodeWithRandomSalt(ByteString plaintext, byte[] saltByte
   {
     try
     {
-      final SecureRandom random = SecureRandom.getInstance(SECURE_PRNG_SHA1);
-      return encodeWithRandomSalt(plaintext, saltBytes, random);
+      return encodeWithRandomSalt(plaintext, saltBytes, new SecureRandom());
     }
     catch (DirectoryException e)
     {

opendj-server-legacy/src/main/java/org/opends/server/tools/ConfigureDS.java

@@ -1208,45 +1208,36 @@ private void updateCryptoManagerCertNickname() throws ConfigureDSException
    */
   private void updateCryptoCipher() throws ConfigureDSException
   {
-    final CryptoManagerCfgDefn cryptoManager = CryptoManagerCfgDefn.getInstance();
-    final StringPropertyDefinition prop = cryptoManager.getKeyWrappingTransformationPropertyDefinition();
-    String defaultCipher = null;
-
-    final DefaultBehaviorProvider<?> p = prop.getDefaultBehaviorProvider();
-    if (p instanceof DefinedDefaultBehaviorProvider)
-    {
-      final Collection<?> defaultValues = ((DefinedDefaultBehaviorProvider<?>) p).getDefaultValues();
-      if (!defaultValues.isEmpty())
-      {
-        defaultCipher = defaultValues.iterator().next().toString();
-      }
-    }
-
+    final String defaultCipher = defaultKeyWrappingTransformation();
     if (defaultCipher != null)
     {
-      // Check that the default cipher is supported by the JVM.
+      final String cipher;
       try
       {
-        Cipher.getInstance(defaultCipher);
+        cipher = supportedKeyWrappingTransformation(defaultCipher);
       }
       catch (final GeneralSecurityException ex)
       {
-        // The cipher is not supported: try to find an alternative one.
-        final String alternativeCipher = getAlternativeCipher();
-        if (alternativeCipher != null)
+        // The default stays, and the server will refuse to start with it: there is no secure
+        // transformation to fall back to (#776), so the administrator has to choose one. Under
+        // setup this stream reaches the setup log only, and the installer gives the warning
+        // itself (see unsupportedKeyWrappingTransformationWarning()).
+        printWrappedText(err, unsupportedKeyWrappingTransformationWarning(defaultCipher, ex));
+        return;
+      }
+      if (!cipher.equals(defaultCipher))
+      {
+        try
         {
-          try
-          {
-            updateConfigEntryWithAttribute(
-                DN_CRYPTO_MANAGER,
-                ATTR_CRYPTO_CIPHER_KEY_WRAPPING_TRANSFORMATION,
-                CoreSchema.getDirectoryStringSyntax(),
-                alternativeCipher);
-          }
-          catch (final Exception e)
-          {
-            throw new ConfigureDSException(e, ERR_CONFIGDS_CANNOT_UPDATE_CRYPTO_MANAGER.get(e));
-          }
+          updateConfigEntryWithAttribute(
+              DN_CRYPTO_MANAGER,
+              ATTR_CRYPTO_CIPHER_KEY_WRAPPING_TRANSFORMATION,
+              CoreSchema.getDirectoryStringSyntax(),
+              cipher);
+        }
+        catch (final Exception e)
+        {
+          throw new ConfigureDSException(e, ERR_CONFIGDS_CANNOT_UPDATE_CRYPTO_MANAGER.get(e));
         }
       }
     }
@@ -1326,6 +1317,83 @@ private Entry removeAttribute(Entry entry, String attrName)
     return duplicateEntry;
   }
 
+  /**
+   * Returns the warning to give when this Java runtime supports neither the default key wrapping
+   * transformation of the crypto manager nor an alternative to it: the server will then refuse to
+   * start until the administrator sets one. The installer calls this itself, since what
+   * {@code configMain} writes while it runs under setup reaches the setup log only.
+   *
+   * @return The warning, or {@code null} when the runtime supports a transformation.
+   */
+  public static LocalizableMessage unsupportedKeyWrappingTransformationWarning()
+  {
+    final String defaultCipher = defaultKeyWrappingTransformation();
+    if (defaultCipher == null)
+    {
+      return null;
+    }
+    try
+    {
+      supportedKeyWrappingTransformation(defaultCipher);
+      return null;
+    }
+    catch (final GeneralSecurityException ex)
+    {
+      return unsupportedKeyWrappingTransformationWarning(defaultCipher, ex);
+    }
+  }
+
+  private static LocalizableMessage unsupportedKeyWrappingTransformationWarning(
+      final String defaultCipher, final GeneralSecurityException ex)
+  {
+    return WARN_CONFIGDS_KEY_WRAPPING_TRANSFORMATION_UNSUPPORTED.get(defaultCipher, ex.getMessage());
+  }
+
+  /** Returns the default key wrapping transformation of the crypto manager, or {@code null}. */
+  private static String defaultKeyWrappingTransformation()
+  {
+    final StringPropertyDefinition prop =
+        CryptoManagerCfgDefn.getInstance().getKeyWrappingTransformationPropertyDefinition();
+    final DefaultBehaviorProvider<?> p = prop.getDefaultBehaviorProvider();
+    if (p instanceof DefinedDefaultBehaviorProvider)
+    {
+      final Collection<?> defaultValues = ((DefinedDefaultBehaviorProvider<?>) p).getDefaultValues();
+      if (!defaultValues.isEmpty())
+      {
+        return defaultValues.iterator().next().toString();
+      }
+    }
+    return null;
+  }
+
+  /**
+   * Returns the key wrapping transformation this Java runtime supports: the default one when it
+   * does, otherwise the OAEP alternative of {@link #getAlternativeCipher()}.
+   *
+   * @param defaultCipher
+   *          The default key wrapping transformation of the crypto manager.
+   * @return The transformation to configure.
+   * @throws GeneralSecurityException
+   *           If the runtime supports neither, with the reason the default one is not.
+   */
+  static String supportedKeyWrappingTransformation(final String defaultCipher) throws GeneralSecurityException
+  {
+    try
+    {
+      Cipher.getInstance(defaultCipher);
+      return defaultCipher;
+    }
+    catch (final GeneralSecurityException ex)
+    {
+      final String alternativeCipher = getAlternativeCipher();
+      if (alternativeCipher == null)
+      {
+        throw ex;
+      }
+      return alternativeCipher;
+    }
+  }
+
   /**
    * Returns a cipher that is supported by the JVM we are running at.
    * Returns <CODE>null</CODE> if no alternative cipher could be found.

opendj-server-legacy/src/messages/org/opends/messages/core.properties

@@ -1376,3 +1376,7 @@ WARN_CRYPTOMGR_SSL_CERT_NICKNAME_LOOKUP_FAILED_765=The certificate nicknames now
  up in the trust store used for server to server communication: %s. A nickname that \
  trust store does not hold is only reported once the server has been restarted with it, \
  so check the nicknames against the trust store before restarting the server
+ERR_CRYPTOMGR_KEY_WRAPPING_TRANSFORMATION_UNSUPPORTED_766=This Java runtime \
+ does not provide the key wrapping transformation %s: %s. The \
+ key-wrapping-transformation property of the crypto manager must name a \
+ transformation which the runtime supports

opendj-server-legacy/src/messages/org/opends/messages/tool.properties

@@ -2634,6 +2634,11 @@ ERR_FILE_NOT_FULLY_READABLE_20015=Could not completely read file '%s'
 SUPPLEMENT_DESCRIPTION_BACKEND_TOOL_SUBCMD_LIST_INDEX_STATUS_20016=\
   <xinclude:include href="variablelist-backendstat-index-status.xml" />
 INFO_DESCRIPTION_DEFAULT_ADD_20017=Legacy argument for ForgeRock OpenDJ compatibility.
+WARN_CONFIGDS_KEY_WRAPPING_TRANSFORMATION_UNSUPPORTED_20018=This Java runtime \
+ supports neither the default key wrapping transformation %s nor an alternative \
+ to it: %s. The server will not start until the key-wrapping-transformation \
+ property of the crypto manager names a transformation which the runtime \
+ supports; set it in config/config.ldif before starting the server
 INFO_LDAP_CONN_PROMPT_SECURITY_LDAP=LDAP
 INFO_LDAP_CONN_PROMPT_SECURITY_USE_SSL=LDAP with SSL
 INFO_LDAP_CONN_PROMPT_SECURITY_USE_START_TLS=LDAP with StartTLS