Try to set CFLAGS and LIBS by passing them to ./configure. With the settings that you need, eg. CFLAGS=-I/usr/include/openssl11 LIBS=-L/usr/lib/locationofopenssl11.

About pkg-config. I do not see how that would fix things, apart from not wanting to depend on it. How would configure run that pkg-config command, since it does not know about openssl11 or anything. But configure it limited in the placing of the openssl include and library directories, perhaps we could have separate flags to set these. One flag for the include and one flag for the lib directory. Having the directories in the same place is what everything else does, really, in /usr/local, /pkg, /sfw and also some macos open source package systems as well /usr/sfw and other places. I think it is your usecase of having multiple openssl versions, all installed in the same install prefix of /usr that is the difference here.

Hi @wcawijngaards. I tried your suggestions, but it's not working. First, I set:

CFLAGS=-I/usr/include/openssl11 LDFLAGS="-L/usr/lib64/openssl11 -lssl -lcrypto"

And ran configure. It failed. Next, I kept those flags, but I also set the flag --with-ssl=/usr/include/openssl11, but it failed again, with the same error:

checking for SSL... configure: error: Cannot find the SSL libraries in /usr/include/openssl11

I think the issue seems to be on line 9293 of the configure script. It tries:

if test -f "$dir/include/openssl/ssl.h"; then
    found_ssl="yes";

So it's trying to look for ssl.h in /usr/include/openssl11/include/openssl/ssl.h, whereas the file is actually in /usr/include/openssl11/openssl/ssl.h. I think if the CFLAGS have provided a path for the openssl header files, then the test in the configure script should look for openssl/ssl.h (relative) rather than an absolute path.

No, it won't work. The code has includes in the C code that include openssl/header.h and this is openssl, not openssl11. So this is not going to compile with any setting of CFLAGS or changes to the configure script to detect differently. Also, I think that is seriously broken placement for the other openssl library. There is really no way to use it exept to recode the source to #include the headers from the openssl library differently.

I do not think I should do that, and that the #include of openssl/header.h is the correct method. I wonder what program can even use that openssl11 place, unless they have hardcoded the directory. Or they use a -I for the include/openssl en then include the header.h without the openssl/ in front. But I think that is not really the right way for the sourcecode. I could still do it but would end up with every #include twice for openssl, depending on where it installed the headers.

If you want to compile with a different openssl and want it to 'just work now', compile openssl yourself, so you can specify the install prefix. And then use a ./config --prefix=/usr/local or --prefix=/opt/local for example, then make, make install. That would put the openssl version in the /usr/local or /opt/local directory. Then for unbound use --with-ssl=/usr/local or --with-ssl=/opt/local and that should work fine and use that openssl version with unbound.

If I had a single server, with all the build tools, I could compile openssl 1.1 on it, and install it all under a single prefix, such as /opt/openssl1.1. But this approach does not scale. I don't want to install build tools on every server out there, and compile openssl on every one of them. That's why we have RPMs (or other binary packages).

The next issue is FHS. On most Linux distributions, there is a standard for organising files. So include files all go into /usr/include and library files go into /usr/lib or /usr/lib64. Thus, any package that installs files must adhere to this. The openssl11 package for CentOS does just this. It places all the include files in /usr/include/openssl11/openssl. The library files go into /usr/lib64/openssl11. This approach allows for multiple versions of openssl to be installed.

If I write some C program using openssl, I will have code like this:

#include <openssl/ssl.h>

When compiling, I can link it against either openssl 1.0 or 1.1. I just need to tell the compiler which paths to look for the header and library files, for example:

gcc -o mysslapp mysslapp.c -I/usr/include/openssl11 -L/usr/lib64/openssl11 -lssl -lcrypto

or

gcc -o mysslapp mysslapp.c $(pkg-config --cflags --libs openssl11)

The entire issue I see here is that the configure script is very rigid. When asked to compile against openssl, it is looking for openssl in a very specific set of hard-coded locations, and it's making rigid assumptions about paths, with no consideration for the Linux FHS. In comparison, BIND's configure script happily finds openssl 1.1, and generates the appropriate config.h file. Compiling BIND correctly links it against openssl 1.1.

Thank you for the detailed explanation. In the fix commit the ./configure can have --with-ssl=/usr/include/openssl11 as an option. In then should detect the header file in /usr/include/openssl11/openssl/ssl.h and conclude that the library is in /usr/lib64/openssl11 and hopefully that fixes the issue?

There is a similar fix in Unbound, where --with-ssl=/usr/include/openssl11 should work for this.

I checked out the latest NSD code from github, and tried to compile it. I got:

# autoreconf
# ./configure --with-ssl=/usr/include/openssl11
...
...
checking for SSL... sed: -e expression #1, char 11: extra characters after command
sed: -e expression #1, char 11: extra characters after command
configure: error: Could not find openssl lib file, /libssl.so,a, pass like "/usr/local" or "/usr/include/openssl11"

The sed expression was wrong, this commit fixes it c81891c
Otherwise it looks like the code is continuing in the correct direction (so far).

Thanks @wcawijngaards! That fixed it. Now NSD compiles correctly against openssl 1.1, and has TLS 1.3 support too.