Build Windows releases with openssl; automate external libraries

External libraries for Windows are now built automatically in the qpdf/external-libs repository and include openssl in addition to zlib and jpeg. Use these, and update the Windows build to build with the openssl crypto provider by default. We leave the native crypto provider enabled in case there is a problem with openssl and also to continue to exercise that code.

Build Windows releases with openssl; automate external libraries
External libraries for Windows are now built automatically in the qpdf/external-libs repository and include openssl in addition to zlib and jpeg. Use these, and update the Windows build to build with the openssl crypto provider by default. We leave the native crypto provider enabled in case there is a problem with openssl and also to continue to exercise that code.
Jay Berkenbilt
1 parent 026330eb
Showing 17 changed files with 167 additions and 60 deletions
.github/workflows/main.yml
ChangeLog
README-maintainer
README-what-to-download.md
README-windows.md
README.md
TODO
autofiles.sums
build-scripts/build-mac
build-scripts/build-windows
build-scripts/download-external-libs
build-scripts/make-distfiles
config-mingw
config-msvc
configure
configure.ac
copy_dlls
@@ -35,6 +35,11 @@ jobs:
         with:
           name: distfiles
           path: distfiles.zip
+      - name: 'Upload external libs'
+        uses: actions/upload-artifact@v1
+        with:
+          name: external-libs
+          path: external-libs-dist
   Linux:
     runs-on: ubuntu-latest
     steps:
@@ -64,6 +69,11 @@ jobs:
         uses: actions/download-artifact@v2
         with:
           name: distfiles
+      - name: 'Download external libs'
+        uses: actions/download-artifact@v2
+        with:
+          name: external-libs
+          path: .
       - name: 'Build, test, generate binary distributions'
         shell: cmd
         run: build-scripts/build-windows.bat ${{ matrix.wordsize }} ${{ matrix.tool }}
@@ -81,6 +91,11 @@ jobs:
         uses: actions/download-artifact@v2
         with:
           name: distfiles
+      - name: 'Download external libs'
+        uses: actions/download-artifact@v2
+        with:
+          name: external-libs
+          path: .
       - name: 'Mac build and test'
         run: build-scripts/build-mac
   AppImage:
+2020-10-25  Jay Berkenbilt  <ejb@ql.org>
+
+	* Official Windows releases are now built using the openssl crypto
+	provider. The native provider is still available for selection at
+	runtime using the QPDF_CRYPTO_PROVIDER environment variable.
+
 2020-10-23  Jay Berkenbilt  <ejb@ql.org>
 	* Bug fix: when concatenating content streams, insert a newline if
@@ -326,20 +326,6 @@ To construct a source distribution from a pristine checkout,
 make build_manual
 make distclean
-To create a source release of external libs, do an export from the
-version control system into a directory called `qpdf-external-libs`
-and just make a zip file of the result called
-`qpdf-external-libs-src.zip`. See the README.txt file there for
-information on creating binary external libs releases. Run this from
-the external-libs repository:
-
-git archive --prefix=external-libs/ HEAD . | (cd /tmp; tar xf -)
-cd /tmp
-zip -r qpdf-external-libs-src.zip external-libs
-
-When releasing on sourceforge, `external-libs` distributions go in
-`external-libs/yyyymmdd`, and qpdf distributions go in `qpdf/vvv`.
-
 For local iteration on the AppImage generation, it works to just
 ./build-scripts/build-appimage and get the resulting AppImage from
 the distribution directory. You can also pass -e SKIP_TESTS=1
 To build from source for Linux or other UNIX/UNIX-like systems, it is generally sufficient to download just the source `qpdf-<version>.tar.gz` file.
-Virtually all Linux distributions include packages for qpdf. If you'd like to run the latest version of qpdf as an [AppImage](https://appimage.org/), you can download `qpdf-<version>-x86_64.AppImage`. This is a self-contained executable that you make symlink `qpdf` to and run on most reasonably recent Linux distributions. See README-appimage.md in the qpdf source distribution for additional details, or run the AppImage with the `--ai-usage` argument to get help specific to the AppImage.
+Windows Binaries
-For Windows, there are several additional files that you might want to download.
+You can download Windiows binaries that are statically linked with qpdf's external dependencies and use the OpenSSL crypto provider. There are several options:
-* `qpdf-<version>-bin-mingw32.zip`
+* `qpdf-<version>-bin-mingw32.zip` - 32-bit executables that should work on basically any Windows system, including 64-bit systems. The 32-bit executables are capable of handling files larger than 2 GB. If you just want to use the qpdf command line program or use the qpdf DLL's C-language interface, you can download this file.  You can also download this version if you are using MINGW's gcc and want to program using the C++ interface.
-   If you just want to use the qpdf command line program or use the qpdf DLL's C-language interface, you can download this file.  You can also download this version if you are using MINGW's gcc and want to program using the C++ interface.
+* `qpdf-<version>-bin-mingw64.zip` - A 64-bit version built with mingw.  Use this for 64-bit Windows systems.  The 32-bit version will also work on Windows 64-bit. Both the 32-bit and the 64-bit version support files over 2 GB in size, but you may find it easier to integrate this with your own software if you use the 64-bit version.
-* `qpdf-<version>-bin-mingw64.zip`
+* `qpdf-<version>-bin-msvc32.zip` - If you want to program using qpdf's C++ interface and you are using a recent version of Microsoft Visual C++ in 32-bit mode, you can download this file.
-   A 64-bit version built with mingw.  Use this for 64-bit Windows systems.  The 32-bit version will also work on Windows 64-bit. Both the 32-bit and the 64-bit version support files over 2 GB in size, but you may find it easier to integrate this with your own software if you use the 64-bit version.
+* `qpdf-<version>-bin-msvc64.zip` - If you want to program using qpdf's C++ interface and you are using a recent version of Microsoft Visual C++ in 64-bit mode, you can download this file.
-* `qpdf-<version>-bin-msvc32.zip`
+Linux Binaries
-  If you want to program using qpdf's C++ interface and you are using Microsoft Visual C++ 2015 in 32-bit mode, you can download this file.
+Virtually all Linux distributions include packages for qpdf. There is also a PPA for Ubuntu at https://launchpad.net/~qpdf/+archive/ubuntu/qpdf that includes the latest version of qpdf for recent versions of Ubuntu. However, there are some downloads available for Linux as well.
-* `qpdf-<version>-bin-msvc64.zip`
+* `qpdf-<version>-x86_64.AppImage` - If you'd like to run the latest version of qpdf as an [AppImage](https://appimage.org/), you can download this. This is a self-contained executable that you make symlink `qpdf` to and run on most reasonably recent Linux distributions. See README-appimage.md in the qpdf source distribution for additional details, or run the AppImage with the `--ai-usage` argument to get help specific to the AppImage.
-  If you want to program using qpdf's C++ interface and you are using Microsoft Visual C++ 2015 in 64-bit mode, you can download this file.
+* `qpdf-<version>-bin-linux-x86_64.zip` - This is a (nearly) stand-alone Linux binary, built using an Ubuntu LTS release. It contains the qpdf executables and shared libraries as well as dependent shared libraries that would not typically be present on a minimal system. This can be used to include qpdf in a minimal environment such as a docker container. It is also known to work as a layer in AWS Lambda and was initially created for that purpose.
-* `qpdf-external-libs-bin.zip`
-
-  If you want to build qpdf for Windows yourself with either MINGW or MSVC 2015, you can download this file and extract it inside the qpdf source distribution.  Please refer to README-windows.md in the qpdf source distribution for additional details.  Note that you need the 2017-08-21 version or later to be able to build qpdf 7.0 or newer. Generally grab the `external-libs` distribution that was the latest version at the time of the release of whichever version of qpdf you are building.
-
-* `qpdf-external-libs-src.zip`
-
-  If you want to build the external libraries on your own (for Windows or anything else), you can download this archive. In addition to including an unmodified distribution `zlib` and the `jpeg` library, it includes a `README` file and some scripts to help you build it for Windows. You will also have to provide those.
-
-If you want to build on Windows, please see also README-windows.md in the qpdf source distribution.
+Windows Build Support
+If you are building on Windows and want to use pre-built external static libraries, you should obtain current versions from https://github.com/qpdf/external-libs/releases. The `external-libs` directory contains older versions that will not work with qpdf versions >= 10.0.2. Please see README-windows.md in the qpdf source distribution.
@@ -32,11 +32,11 @@ Image comparison tests are disabled by default, but it is possible to run them o
 # External Libraries
-In order to build qpdf, you must have a copy of `zlib` and the `jpeg` library. The easy way to get it is to download the external libs from the qpdf download area. There are packages called `external-libs-bin.zip` and `external-libs-src.zip`. If you are building with MSVC 2015 or MINGW with MSYS2, you can just extract the `qpdf-external-libs-bin.zip` zip file into the top-level qpdf source tree. Note that you need the 2017-08-21 version (at least) to build qpdf 7.0 or greater since this includes jpeg. Passing `--enable-external-libs` to `./configure` (which is done automatically if you follow the instructions below) is sufficient to find them.
+In order to build qpdf, you must have a copy of `zlib` and the `jpeg` library. You can download [prebuilt static external libraries from the qpdf/external-libs github repository](https://github.com/qpdf/external-libs/releases). These include `zlib`, `jpeg`, and `openssl` libraries. There are packages called `external-libs-bin.zip` and `external-libs-src.zip`. If you are building with a recent MSVC or MINGW with MSYS2, you can just extract the `qpdf-external-libs-bin.zip` zip file into the top-level qpdf source tree. Note that you need the 2020-10-24 version (at least) to build qpdf 10.0.2 or greater since this includes openssl. Passing `--enable-external-libs` to `./configure` (which is done automatically if you follow the instructions below) is sufficient to find them.
 You can also obtain `zlib` and `jpeg` directly on your own and install them. If you are using mingw, you can just set `CPPFLAGS`, `LDFLAGS`, and `LIBS` when you run ./configure so that it can find the header files and libraries. If you are building with MSVC and you want to do this, it probably won't work because `./configure` doesn't know how to interpret `LDFLAGS` and `LIBS` properly for MSVC (though qpdf's own build system does). In this case, you can probably get away with cheating by passing `--enable-external-libs` to `./configure` and then just editing `CPPFLAGS`, `LDFLAGS`, `LIBS` in the generated autoconf.mk file. Note that you should use UNIX-like syntax (`-I`, `-L`, `-l`) even though this is not what cl takes on the command line. qpdf's build rules will fix it.
-You can also download `qpdf-external-libs-src.zip` and follow the instructions in the README.txt there for how to build external libs.
+External libraries are built using GitHub Actions from the [qpdf/external-libs](https://github.com/qpdf/external-libs) repository.
 # Building from version control
@@ -36,7 +36,7 @@ Depending on which crypto providers are enabled, then [GnuTLS](https://www.gnutl
 # Licensing terms of embedded software
-QPDF makes use of zlib and jpeg libraries for its functionality. These packages can be downloaded separately from their own download locations, or they can be downloaded in the external-libs area of the qpdf download site. If the optional GnuTLS crypto provider is enabled, then GnuTLS is also required.
+QPDF makes use of zlib and jpeg libraries for its functionality. These packages can be downloaded separately from their own download locations. If the optional GnuTLS or OpenSSL crypto providers are enabled, then GnuTLS and/or OpenSSL are also required.
 Please see the [NOTICE](NOTICE.md) file for information on licenses of embedded software.
@@ -8,8 +8,6 @@ Candidates for upcoming release
     * #446: recognize edited QDF files
     * #436: parsing of document with form xobject
-* See if we can work in Windows Build/External Libraries (below)
-
 * QPDFObjectHandle::pipeContentStreams calls finish() after each
   stream. In some code paths, Pl_Concatenate is used, which suppresses
   that, but in other code paths, it's not used, and the library relies
@@ -94,20 +92,62 @@ GitHub Actions
   but I'd rather not depend on them. Keep an eye open for this coming
   to GitHub Actions.
-Windows Build/External Libraries
-================================
+External Libraries
+==================
+
+Current state (10.0.2):
+
+* qpdf/external-libs repository builds external-libs on a schedule.
+  It detects and downloads the latest versions of zlib, jpeg, and
+  openssl and creates source and binary distribution zip files in an
+  artifact called "distribution".
+
+* Releases in qpdf/external-libs are made manually. They contain
+  qpdf-external-libs-{bin,src}.zip.
+
+* The qpdf build finds the latest non-prerelease release and downloads
+  the qpdf-external-libs-*.zip files from the releases in the setup
+  stage.
+
+* To upgrade to a new version of external-libs, create a new release
+  of qpdf/external-libs (see README-maintainer in external-libs) from
+  the distribution artifact of the most recent successful build after
+  ensuring that it works.
+
+Desired state:
+
+* The qpdf/external-libs repository should create release candidates.
+  Ideally, every scheduled run would make its zip files available.
+  A personal access token with actions:read scope for the
+  qpdf/external-libs repository is required to download the artifact
+  from an action run, and qpdf/qpdf's secrets.GITHUB_TOKEN doesn't
+  have this access. As an alternative, we could have a draft release
+  in qpdf/external-libs that the qpdf/external-libs build could update
+  with each candidate.
+
+* Scheduled runs of the qpdf build in the qpdf/qpdf repository (not a
+  fork or pull request) could download external-libs from the release
+  candidate area instead of the latest stable release. Pushes to the
+  build branch should still use the latest release so it always
+  matches the main branch.
+
+* Periodically, we would create a release of external-libs from the
+  release candidate zip files. This could be done safely because we
+  know the latest qpdf works with it. This could be done at least
+  before every release of qpdf, but potientially it could be done at
+  other times, such as when a new dependency version is available or
+  after some period of time.
-* Migrate external library build code to a separate repository.
+Other notes:
-* Automate downloading and building latest versions of external
-  libraries. Add openssl.
+* The external-libs branch in qpdf/qpdf was never documented. We might
+  be able to get away with deleting it.
-* Build external libraries on a schedule and create releases
-  periodically or when they change. See if we can get rid of the
-  external-libs branch in qpdf/qpdf.
+* See README-maintainer in qpdf/external-libs for information on
+  creating a release. This could be at least partially scripted in a
+  way that works for the qpdf/qpdf repository as well since they are
+  very similar.
-* Update the Windows build so that it uses current versions of
-  external libraries and openssl as its crypto provider.
 ABI Changes
 ===========
-cc3c7947646412e7c3152c3ef238226ede1c2199328a38df93debee26184b087  configure.ac
+1c8c57be01a03bc85ca49293d0212026e0e9b28a73567bda8e27462abc3ffd1a  configure.ac
 d3f9ee6f6f0846888d9a10fd3dad2e4b1258be84205426cf04d7cef02d61dad7  aclocal.m4
 cf2c764639c4c94abc183a0976eca6ae500b80790ea25e3d0af97b23587363b7  libqpdf/qpdf/qpdf-config.h.in
 5297971a0ef90bcd5563eb3f7127a032bb76d3ae2af7258bf13479caf8983a60  m4/ax_cxx_compile_stdcxx.m4
 #!/bin/bash
 set -ex
-curl -L https://github.com/qpdf/qpdf/raw/external-libs/jpegsrc.v9c.tar.gz -o jpegsrc.v9c.tar.gz
-tar xzf jpegsrc.v9c.tar.gz
-cd jpeg-9c
+cd $(dirname $0)/..
+unzip qpdf-external-libs-src.zip
+tar xzf external-libs-src/jpegsrc*
+cd jpeg-*
 ./configure
 make -k
 sudo make install
@@ -23,14 +23,13 @@ fi
 if [ -f distfiles/distfiles.zip ]; then
     unzip distfiles/distfiles.zip
 fi
-curl -L https://github.com/qpdf/qpdf/raw/external-libs/qpdf-external-libs-bin.zip -o qpdf-external-libs-bin.zip
 unzip qpdf-external-libs-bin.zip
 cwd=`pwd`
 PATH=$cwd/libqpdf/build:$PATH
 installdir=install-$tool$wordsize
 rm -rf $installdir
-./config-$tool --enable-show-failed-test-output --disable-crypto-gnutls --disable-crypto-openssl
+./config-$tool --enable-show-failed-test-output
 make -j$(nproc) -k
 make -k check
 make install
+#!/usr/bin/env python3
+import json
+import os
+import requests
+import sys
+from zipfile import ZipFile
+from operator import itemgetter
+
+
+def warn(*args, **kwargs):
+    print(*args, **kwargs, file=sys.stderr)
+
+
+def download_file(url, local_filename):
+    # From https://stackoverflow.com/questions/16694907/
+    # download-large-file-in-python-with-requests
+    if os.path.exists(local_filename):
+        warn('Using existing', local_filename)
+        return
+    warn('Downloading', local_filename)
+    with requests.get(url, stream=True) as r:
+        r.raise_for_status()
+        with open(local_filename, 'wb') as f:
+            for chunk in r.iter_content(chunk_size=8192):
+                f.write(chunk)
+    return local_filename
+
+
+bin_name = 'qpdf-external-libs-bin.zip'
+src_name = 'qpdf-external-libs-src.zip'
+dir_name = 'external-libs-dist'
+os.makedirs(dir_name, exist_ok=True)
+
+r = requests.get(
+    'https://api.github.com/repos/qpdf/external-libs/releases')
+releases = json.loads(r.text)
+by_tag = sorted(
+    [(r['tag_name'], r) for r in releases
+     if r['prerelease'] is False],
+    reverse=True)
+latest = by_tag[0][1]
+bin_url = None
+src_url = None
+for i in latest['assets']:
+    if i['name'] == bin_name:
+        bin_url = i['browser_download_url']
+    elif i['name'] == src_name:
+        src_url = i['browser_download_url']
+print(bin_url)
+download_file(bin_url, f'{dir_name}/{bin_name}')
+download_file(src_url, f'{dir_name}/{src_name}')
+
+print('\n** external library information **')
+with ZipFile(f'{dir_name}/{src_name}') as z1:
+    with z1.open('external-libs-src/versions') as z2:
+        print(z2.read().decode())
 #!/bin/bash
+cd $(dirname $0)/..
 set -ex
 sudo apt-get update
 sudo apt-get -y install \
@@ -6,3 +7,4 @@ sudo apt-get -y install \
    docbook-xsl fop xsltproc libxml2-utils inkscape imagemagick
 ./configure --enable-doc-maintenance
 make -j$(nproc) distfiles.zip
+build-scripts/download-external-libs
@@ -6,4 +6,4 @@ if g++ -v 2&gt;&amp;1 | grep Target: | grep -q x86_64; then
 else
     wordsize=32
 fi
-./configure --disable-test-compare-images --enable-external-libs --enable-werror --with-buildrules=mingw ${1+"$@"}
+./configure --disable-test-compare-images --enable-external-libs --enable-werror --with-buildrules=mingw ${1+"$@"} --enable-crypto-openssl --disable-crypto-gnutls
@@ -6,4 +6,4 @@ if cl 2&gt;&amp;1 | grep -q &#39;for x64&#39;; then
 else
     wordsize=32
 fi
-CC=cl CXX="cl -TP -GR" ./configure --disable-test-compare-images --enable-external-libs --enable-werror --with-buildrules=msvc ${1+"$@"}
+CC=cl CXX="cl -TP -GR" ./configure --disable-test-compare-images --enable-external-libs --enable-werror --with-buildrules=msvc ${1+"$@"} --enable-crypto-openssl --disable-crypto-gnutls
@@ -17829,6 +17829,9 @@ $as_echo &quot;#define USE_CRYPTO_NATIVE 1&quot; &gt;&gt;confdefs.h
 fi
+if test "$USE_EXTERNAL_LIBS" = "1"; then
+  OPENSSL_FOUND=1
+else
 pkg_failed=no
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for pc_openssl" >&5
@@ -17900,6 +17903,7 @@ else
 $as_echo "yes" >&6; }
 	OPENSSL_FOUND=1
 fi
+fi
 if test "$OPENSSL_FOUND" = "0"; then :
   ac_fn_c_check_header_mongrel "$LINENO" "openssl/evp.h" "ac_cv_header_openssl_evp_h" "$ac_includes_default"
@@ -18601,7 +18605,7 @@ if test &quot;$USE_EXTERNAL_LIBS&quot; = &quot;1&quot;; then
    # much trouble getting it to work with a different compiler.
    CPPFLAGS="$CPPFLAGS -Iexternal-libs/include"
    LDFLAGS="$LDFLAGS -Lexternal-libs/lib-$BUILDRULES$WINDOWS_WORDSIZE"
-   LIBS="$LIBS -lz -ljpeg"
+   LIBS="$LIBS -lz -ljpeg -lssl -lcrypto -lmsvcrt -lws2_32 -lshell32 -ladvapi32 -lgdi32 -luser32 -lcrypt32"
 fi
 cat >confcache <<\_ACEOF
@@ -608,8 +608,12 @@ dnl If the openssl provider is not explicitly disabled, enable it if
 dnl openssl is available. If the openssl provider is explicitly
 dnl disabled, do not link with openssl even if present.
-PKG_CHECK_MODULES([pc_openssl], [openssl >= 1.1.0],
-                  [OPENSSL_FOUND=1], [OPENSSL_FOUND=0])
+if test "$USE_EXTERNAL_LIBS" = "1"; then
+  OPENSSL_FOUND=1
+else
+  PKG_CHECK_MODULES([pc_openssl], [openssl >= 1.1.0],
+                    [OPENSSL_FOUND=1], [OPENSSL_FOUND=0])
+fi
 dnl Override pkg-config if headers and libraries are present.
 AS_IF([test "$OPENSSL_FOUND" = "0"],
@@ -958,7 +962,7 @@ if test &quot;$USE_EXTERNAL_LIBS&quot; = &quot;1&quot;; then
    # much trouble getting it to work with a different compiler.
    CPPFLAGS="$CPPFLAGS -Iexternal-libs/include"
    LDFLAGS="$LDFLAGS -Lexternal-libs/lib-$BUILDRULES$WINDOWS_WORDSIZE"
-   LIBS="$LIBS -lz -ljpeg"
+   LIBS="$LIBS -lz -ljpeg -lssl -lcrypto -lmsvcrt -lws2_32 -lshell32 -ladvapi32 -lgdi32 -luser32 -lcrypt32"
 fi
 AC_OUTPUT
@@ -98,6 +98,7 @@ sub get_dlls
             my $dll = $1;
             $dll =~ tr/A-Z/a-z/;
             next if $dll =~ m/^(kernel32|user32|msvcrt|advapi32)\.dll$/;
+            next if $dll =~ m/^(api-ms-win.*|ws2_32|crypt32|bcrypt)\.dll$/;
             push(@result, $dll);
         }
         elsif (m/^Magic.*\((PE.+?)\)/)