Unbound fails to start in rawhide with tls-cert-bundle: /etc/pki/tls/certs/ca-bundle.crt

[flo-renaud]

Describe the bug
The FreeIPA project is internally using unbound and has CI tests running on rawhide.
One of our tests started failing this week with unbound-1.23.1-2.fc43.x86_64.

To reproduce
Steps to reproduce the behavior:

1. Install a rawhide machine and enable the copr repo @freeipa/freeipa-master-nightly: dnf copr enable -y @freeipa/freeipa-master-nightly
2. Install ipa-server-dns and freeipa-server-encrypted-dns package: dnf install -y freeipa-server-dns freeipa-server-encrypted-dns
3. Install IPA server with ipa-server-install -n ipa.test -r IPA.TEST -p Secret123 -a Secret123 -U --setup-dns --forwarder 10.11.5.160 --auto-reverse --dns-over-tls --dot-forwarder '1.1.1.1#cloudflare-dns.com'

Expected behavior
The installation should succeed. Instead, the installation fails in a step restarting the unbound service:

# ipa-server-install -n ipa.test -r IPA.TEST -p Secret123 -a Secret123 -U --setup-dns --forwarder 10.11.5.160 --auto-reverse --dns-over-tls --dot-forwarder '1.1.1.1#cloudflare-dns.com'

The log file for this installation can be found in /var/log/ipaserver-install.log
==============================================================================
This program will set up the IPA Server.
Version 4.13.0.dev202507310711+git
[...]
Applying LDAP updates
Upgrading IPA:. Estimated time: 1 minute 30 seconds
  [1/10]: stopping directory server
  [2/10]: saving configuration
  [3/10]: disabling listeners
  [4/10]: enabling DS global lock
  [5/10]: disabling Schema Compat
  [6/10]: starting directory server
  [7/10]: upgrading server
  [8/10]: stopping directory server
  [9/10]: restoring configuration
  [10/10]: starting directory server
Done.
Restarting the KDC
Request certificate for DNS over TLS, using IPA CA
dnssec-validation yes
Configuring DNS (named)
  [1/12]: generating rndc key file
  [2/12]: adding DNS container
  [3/12]: setting up our zone
  [4/12]: setting up our own record
  [5/12]: setting up records for other masters
  [6/12]: adding NS record to the zones
  [7/12]: setting up kerberos principal
  [8/12]: setting up LDAPI autobind
  [9/12]: setting up named.conf
created new /etc/named.conf
created named user config '/etc/named/ipa-ext.conf'
created named user config '/etc/named/ipa-options-ext.conf'
created named user config '/etc/named/ipa-logging-ext.conf'
  [10/12]: setting up server configuration
  [11/12]: configuring named to start on boot
  [12/12]: changing resolv.conf to point to ourselves
Done configuring DNS (named).
Setting up DNS over TLS
CalledProcessError(Command ['/bin/systemctl', 'restart', 'unbound.service'] returned non-zero exit status 1: 'Job for unbound.service failed because the control process exited with error code.\nSee "systemctl status unbound.service" and "journalctl -xeu unbound.service" for details.\n')
The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information

The logs show:

# systemctl status unbound.service
× unbound.service - Unbound recursive Domain Name Server
     Loaded: loaded (/usr/lib/systemd/system/unbound.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Mon 2025-08-04 03:25:50 EDT; 12min ago
 Invocation: dad4e98b10b44cc38e3f8a5ad066a5b7
    Process: 9722 ExecStartPre=/usr/sbin/unbound-checkconf (code=exited, status=0/SUCCESS)
    Process: 9723 ExecStart=/usr/sbin/unbound -d $UNBOUND_OPTIONS (code=exited, status=1/FAILURE)
   Main PID: 9723 (code=exited, status=1/FAILURE)
   Mem peak: 2.7M
        CPU: 16ms

Aug 04 03:25:50 server.ipa.test systemd[1]: Starting unbound.service - Unbound recursive Domain Name Server...
Aug 04 03:25:50 server.ipa.test unbound-checkconf[9722]: unbound-checkconf: no errors in /etc/unbound/unbound.conf
Aug 04 03:25:50 server.ipa.test unbound[9723]: Aug 04 03:25:50 unbound[9723:0] error: error in SSL_CTX verify crypto error:80000002:system library::No such file or directory
Aug 04 03:25:50 server.ipa.test unbound[9723]: Aug 04 03:25:50 unbound[9723:0] error: and additionally crypto error:10000080:BIO routines::no such file
Aug 04 03:25:50 server.ipa.test unbound[9723]: Aug 04 03:25:50 unbound[9723:0] error: and additionally crypto error:05880020:x509 certificate routines::BIO lib
Aug 04 03:25:50 server.ipa.test unbound[9723]: Aug 04 03:25:50 unbound[9723:0] fatal error: could not set up connect SSL_CTX
Aug 04 03:25:50 server.ipa.test systemd[1]: unbound.service: Main process exited, code=exited, status=1/FAILURE
Aug 04 03:25:50 server.ipa.test systemd[1]: unbound.service: Failed with result 'exit-code'.
Aug 04 03:25:50 server.ipa.test systemd[1]: Failed to start unbound.service - Unbound recursive Domain Name Server.

FreeIPA installer configures unbound with the following file:

# cat /etc/unbound/conf.d/zzz-ipa.conf 
server:
    tls-cert-bundle: /etc/pki/tls/certs/ca-bundle.crt
    tls-upstream: yes
    interface: 127.0.0.55
    log-servfail: yes
    # module-config: "iterator"
forward-zone:
    name: "."
    forward-tls-upstream: yes
    forward-first: no
    forward-addr: 1.1.1.1#cloudflare-dns.com

System:

• Unbound version: unbound-1.23.1-2.fc43.x86_64
• OS: rawhide
• unbound -V output:

Additional information
Issue reported in freeipa project at https://pagure.io/freeipa/issue/9838

We suspect that the issue is related to rawhide dropping cert.pem file (https://fedoraproject.org/wiki/Changes/dropingOfCertPemFile and https://bugzilla.redhat.com/show_bug.cgi?id=2360110)

[wcawijngaards]

The error that is pertinent is: error in SSL_CTX verify crypto error:80000002:system library::No such file or directory, for /etc/pki/tls/certs/ca-bundle.crt. It looks like the file does not exist.

We do not create the unbound package that you used. That is done by the linux distribution. But this is more about the cert pem files.

On another system, I can see that ca-bundle.crt is a symbolic link to another file. Likely the removal of the cert.pem file may have also removed this. We also do not create the cert.pem or ca-bundle files. They are supplied by the user and configured as trusted certificate authorities for the TLS connection, but we do not maintain that cert file as part of the source code distribution.

[flo-renaud]

Is it possible to provide a directory containing certificates to the "tls-cert-bundle: " parameter in unbound conf file, instead of a single file?

[wcawijngaards]

From the link to the certpemfile changes, it says you are encouraged to use /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem instead. That would be tls-cert-bundle: "/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem". Unbound wants a nonNULL cert bundle, then it sets up tls peer verification. Perhaps with tls-system-cert: yes and then without tls-cert-bundle, it would load the default locations using the faster implementation.