Update docs for crypto providers

Jay Berkenbilt
1 parent 70b8c41f
Showing 4 changed files with 298 additions and 5 deletions
ChangeLog
README.md
TODO
manual/qpdf-manual.xml
+2019-11-05  Jay Berkenbilt  <ejb@ql.org>
+
+	* Add support for pluggable crypto providers, enabling multiple
+	implementations of the cryptographic functions needed by qpdf.
+	This feature was added by request of Red Hat, which recognized the
+	use of qpdf's native crypto implementations as a potential
+	security liability, preferring instead to get all crypto
+	functionality from a third-party library that receives a lot of
+	scrutiny. However it was also important to me to not impose any
+	unnecessary third party depdendencies on my users or packagers,
+	some of which build qpdf for lots of environments, some of which
+	may not easily support gnutls. Starting in qpdf 9.1.0, it is be
+	possible to build qpdf with both the native and gnutls crypto
+	providers or with either in isolation. In support of this feature,
+	new classes QPDFCryptoProvider and QPDFCryptoImpl have been added
+	to the public interface. See QPDFCryptoImpl.hh for details about
+	adding your own crypto provider and QPDFCryptoProvider.hh for
+	details about choosing which one is used. Note that selection of
+	crypto providers is invisible to anyone who doesn't explicitly
+	care. Neither end users nor developers have to be concerned about
+	it.
+
+	* The environment variable QPDF_CRYPTO_PROVIDER can be used to
+	override qpdf's default choice of crypto provider. The
+	--show-crypto flag to the qpdf CLI can be used to present a list
+	of supported crypto providers with the default provider always
+	listed first.
+
+	* Add gnutls crypto provider. Thanks to Zdenek Dohnal for
+	contributing the code that I ultimately used in the gnutls crypto
+	provider and for engaging in an extended discussion about this
+	feature. Fixes #218.
+
 2019-10-22  Jay Berkenbilt  <ejb@ql.org>
 	* Incorporate changes from Masamichi Hosoda <trueroad@trueroad.jp>
@@ -31,12 +31,33 @@ QPDF requires a C++ compiler that supports C++-11.
 QPDF depends on the external libraries [zlib](http://www.zlib.net/) and [jpeg](http://www.ijg.org/files/). The [libjpeg-turbo](https://libjpeg-turbo.org/) library is also known to work since it is compatible with the regular jpeg library, and QPDF doesn't use any interfaces that aren't present in the straight jpeg8 API. These are part of every Linux distribution and are readily available. Download information appears in the documentation. For Windows, you can download pre-built binary versions of these libraries for some compilers; see [README-windows.md](README-windows.md) for additional details.
+If the optional gnutls crypto provider is enabled, then gnutls is also required. This is discussed more in `Crypto providers` below.
+
 # Licensing terms of embedded software
-QPDF makes use of zlib and jpeg libraries for its functionality. These packages can be downloaded separately from their own download locations, or they can be downloaded in the external-libs area of the qpdf download site.
+QPDF makes use of zlib and jpeg libraries for its functionality. These packages can be downloaded separately from their own download locations, or they can be downloaded in the external-libs area of the qpdf download site. If the optional gnutls crypto provider is enabled, then gnutls is also required.
 Please see the [NOTICE](NOTICE.md) file for information on licenses of embedded software.
+# Crypto providers
+
+As of version 9.1.0, qpdf can use different crypto implementations. These can be selected at compile time or at runtime. The native crypto implementations that were used in all versions prior to 9.1.0 are still present and enabled by default.
+
+Initially, the following providers are available:
+* `native`: a native implementation where all the source is embedded in qpdf and no external dependencies are required
+* `gnutls`: an implementation that uses the gnutls library to provide cyrpto; causes libqpdf to link with the gnutls library
+
+The default behavior is for ./configure to discover which other crypto providers can be supported based on available external libraries, to build all available crypto providers, and to use an external provider as the default over the native one. This behavior can be changed with the following flags to ./configure:
+
+* `--enable-crypto-x` -- (where `x` is a supported crypto provider): enable the `x` crypto provider, requiring any external dependencies it needs
+* `--disable-crypto-x` -- disable the `x` provider, and do not link against its dependencies even if they are available
+* `--with-default-crypto=x` -- make `x` the default provider even if a higher priority one is available
+* `--disable-implicit-crypto` -- only build crypto providers that are explicitly requested with an `--enable-crypto-x` option
+
+For example, if you want to guarantee that the gnutls crypto provider is used, you could run ./configure with `--enable-crypto-gnutls --disable-implicit-crypto`.
+
+Please see the section on cyrpto providers in the manual for more details.
+
 # Building from a pristine checkout
 When building qpdf from a pristine checkout from version control, generated documentation files are not present. You may either generate them (by passing `--enable-doc-maintenance` to `./configure` and satisfying the extra build-time dependencies) or obtain them from a released source package, which includes them. If you want to grab just the files that are in the source distribution but not in the repository, extract a source distribution in a temporary directory, and run `make CLEAN=1 distfiles.zip`. This will create a file called `distfiles.zip`, which can you can extract in a checkout of the source repository. This step is optional unless you are running make install and want the html and PDF versions of the documentation to be installed.
@@ -59,7 +80,7 @@ QPDF is known to build and pass its test suite with mingw (latest version tested
 # Additional Notes on Build
-QPDF's build system, inspired by [abuild](http://www.abuild.org), can optionally use its own built-in rules rather than using libtool and obeying the compiler specified with configure.  This can be enabled by passing `--with-buildrules=buildrules` where buildrules corresponds to one of the `.mk` files (other than `rules.mk`) in the make directory. This should never be necessary on a UNIX system, but may be necessary on a Windows system.  See [README-windows.md](README-windows.md) for details.
+QPDF's build system, inspired by [abuild](http://www.qbilt.org/abuild), can optionally use its own built-in rules rather than using libtool and obeying the compiler specified with configure.  This can be enabled by passing `--with-buildrules=buildrules` where buildrules corresponds to one of the `.mk` files (other than `rules.mk`) in the make directory. This should never be necessary on a UNIX system, but may be necessary on a Windows system.  See [README-windows.md](README-windows.md) for details.
 The QPDF package provides some executables and a software library.  A user manual can be found in the "doc" directory.  The docbook sources to the user manual can be found in the `manual` directory.
@@ -272,10 +272,13 @@ I find it useful to make reference to them in this list
    to add any dependencies on threading libraries.
  * Study what's required to support savable forms that can be saved by
-   Adobe Reader.  Does this require actually signing the document with
-   an Adobe private key?  Search for "Digital signatures" in the PDF
+   Adobe Reader. Does this require actually signing the document with
+   an Adobe private key? Search for "Digital signatures" in the PDF
    spec, and look at ~/Q/pdf-collection/form-with-full-save.pdf, which
-   came from Adobe's example site.
+   came from Adobe's example site. See also
+   ../misc/digital-sign-from-trueroad/. If digital signatures are
+   implemented, update the docs on crytpo providers, which mention
+   that this may happen in the future.
  * See if we can avoid preserving unreferenced objects in object
    streams even when preserving the object streams.
@@ -242,6 +242,224 @@ make
     top-level <filename>Makefile</filename>.
    </para>
   </sect1>
+  <sect1 id="ref.crypto">
+   <title>Crypto Providers</title>
+   <para>
+    Starting with qpdf 9.1.0, the qpdf library can be built with
+    multiple implementations of providers of cryptographic functions,
+    which we refer to as &ldquo;crypto providers.&rdquo; At the time
+    of writing, a crypto implementation must provide MD5 and SHA2
+    (256, 384, and 512-bit) hashes and RC4 and AES256 with and without
+    CBC encryption. In the future, if digital signature is added to
+    qpdf, there may be additional requirements beyond this.
+   </para>
+   <para>
+    Starting with qpdf version 9.1.0, the available implementations
+    are <literal>native</literal> and <literal>gnutls</literal>.
+    Additional implementations may be added if needed. It is also
+    possible for a developer to provide their own implementation
+    without modifying the qpdf library.
+   </para>
+   <sect2 id="ref.crypto.build">
+    <title>Build Support For Cyrpto Providers</title>
+    <para>
+     When building with qpdf's build system, crypto providers can be
+     enabled at build time using various
+     <command>./configure</command> options. The default behavior is
+     for <command>./configure</command> to discover which crypto
+     providers can be supported based on available external libraries,
+     to build all available crypto providers, and to use an external
+     provider as the default over the native one. This behavior can be
+     changed with the following flags to
+     <command>./configure</command>:
+     <itemizedlist>
+      <listitem>
+       <para>
+        <option>--enable-crypto-<replaceable>x</replaceable></option>
+        (where <replaceable>x</replaceable> is a supported crypto
+        provider): enable the <replaceable>x</replaceable> crypto
+        provider, requiring any external dependencies it needs
+       </para>
+      </listitem>
+      <listitem>
+       <para>
+        <option>--disable-crypto-<replaceable>x</replaceable></option>:
+        disable the <replaceable>x</replaceable> provider, and do not
+        link against its dependencies even if they are available
+       </para>
+      </listitem>
+      <listitem>
+       <para>
+        <option>--with-default-crypto=<replaceable>x</replaceable></option>:
+        make <replaceable>x</replaceable> the default provider even if
+        a higher priority one is available
+       </para>
+      </listitem>
+      <listitem>
+       <para>
+        <option>--disable-implicit-crypto</option>: only build crypto
+        providers that are explicitly requested with an
+        <option>--enable-crypto-<replaceable>x</replaceable></option>
+        option
+       </para>
+      </listitem>
+     </itemizedlist>
+    </para>
+    <para>
+     For example, if you want to guarantee that the gnutls crypto
+     provider is used and that the native provider is not built, you
+     could run <command>./configure --enable-crypto-gnutls
+     --disable-implicit-crypto</command>.
+    </para>
+    <para>
+     If you build qpdf using your own build system, in order for qpdf
+     to work at all, you need to enable at least one crypto provider.
+     The file <filename>include/qpdf/qpdf-config.h.in</filename>
+     provides macros <literal>DEFAULT_CRYPTO</literal>, whose value
+     must be a string naming the default crypto provider, and various
+     symbols starting with <literal>USE_CRYPTO_</literal>, at least
+     one of which has to be enabled. Additionally, you must compile
+     the source files that implement a crypto provider. To get a list
+     of those files, look at <filename>libqpdf/build.mk</filename>. If
+     you want to omit a particular crypto provider, as long as its
+     <literal>USE_CRYPTO_</literal> symbol is undefined, you can
+     completely ignore the source files that belong to a particular
+     crypto provider. Additionally, crypto providers may have their
+     own external dependencies that can be omitted if the crypto
+     provider is not used. For example, if you are building qpdf
+     yourself and are using an environment that does not support
+     gnutls, you can ensure that <literal>USE_CRYPTO_NATIVE</literal>
+     is defined, <literal>USE_CRYPTO_GNUTLS</literal> is not defined,
+     and <literal>DEFAULT_CRYPTO</literal> is defined to
+     <literal>"native"</literal>. Then you must include the source
+     files used in the native implementation, some of which were added
+     or renamed from earlier versions, to your build, and you can
+     ignore <filename>QPDFCrypto_gnutls.cc</filename>. Always consult
+     <filename>libqpdf/build.mk</filename> to get the list of source
+     files you need to build.
+    </para>
+   </sect2>
+   <sect2 id="ref.crypto.runtime">
+    <title>Runtime Cryto Provider Selection</title>
+    <para>
+     You can use the <option>--show-crypto</option> option to
+     <command>qpdf</command> to get a list of available crypto
+     providers. The default provider is always listed first, and the
+     rest are listed in lexical order. Each crypto provider is listed
+     on a line by itself with no other text, enabling the output of
+     this command to be used easily in scripts.
+    </para>
+    <para>
+     You can override which crypto provider is used by setting the
+     <literal>QPDF_CRYPTO_PROVIDER</literal> environment variable.
+     There are few reasons to ever do this, but you might want to do
+     it if you were explicitly trying to compare behavior of two
+     different crypto providers while testing performance or
+     reproducing a bug. It could also be useful for people who are
+     implementing their own crypto providers.
+    </para>
+   </sect2>
+   <sect2 id="ref.crypto.develop">
+    <title>Cryto Provider Information for Developers</title>
+    <para>
+     If you are writing code that uses libqpdf and you want to force a
+     certain crypto provider to be used, you can call the method
+     <function>QPDFCryptoProvider::setDefaultProvider</function>. The
+     argument is the name of a built-in or developer-supplied
+     provider. To add your own crypto provider, you have to create a
+     class derived from <classname>QPDFCryptoImpl</classname> and
+     register it with <classname>QPDFCryptoProvider</classname>. For
+     additional information, see comments in
+     <filename>include/qpdf/QPDFCryptoImpl.hh</filename>.
+    </para>
+   </sect2>
+   <sect2 id="ref.crypto.design">
+    <title>Crypto Provider Design Notes</title>
+    <para>
+     This section describes a few bits of rationale for why the crypto
+     provider interface was set up the way it was. You don't need to
+     know any of this information, but it's provided for the record
+     and in case it's interesting.
+    </para>
+    <para>
+     As a general rule, I want to avoid as much as possible including
+     large blocks of code that are conditionally compiled such that,
+     in most builds, some code is never built. This is dangerous
+     because it makes it very easy for invalid code to creep in
+     unnoticed. As such, I want it to be possible to build qpdf with
+     all available crypto providers, and this is the way I build qpdf
+     for local development. At the same time, if a particular packager
+     feels that it is a security liability for qpdf to use crypto
+     functionality from other than a library that gets considerable
+     scrutiny for this specific purpose (such as gnutls, openssl, or
+     nettle), then I want to give that packager the ability to
+     completely disable qpdf's native implementation. Or if someone
+     wants to avoid adding a dependency on one of the external crypto
+     providers, I don't want the availability of the provider to
+     impose additional external dependencies within that environment.
+     Both of these are situations that I know to be true for some
+     users of qpdf.
+    </para>
+    <para>
+     I want registration and selection of crypto providers to be
+     thread-safe, and I want it to work deterministically for a
+     developer to provide their own crypto provider and be able to set
+     it up as the default. This was the primary motivation behind
+     requiring C++-11 as doing so enabled me to exploit the guaranteed
+     thread safety of local block static initialization. The
+     <classname>QPDFCryptoProvider</classname> class uses a singleton
+     pattern with thread-safe initialization to create the singleton
+     instance of <classname>QPDFCryptoProvider</classname> and exposes
+     only static methods in its public interface. In this way, if a
+     developer wants to call any
+     <classname>QPDFCryptoProvider</classname> methods, the library
+     guarantees the <classname>QPDFCryptoProvider</classname> is fully
+     initialized and all built-in crypto providers are registered.
+     Making <classname>QPDFCryptoProvider</classname> actually know
+     about all the built-in providers may seem a bit sad at first, but
+     this choice makes it extremely clear exactly what the
+     initialization behavior is. There's no question about provider
+     implementations automatically registering themselves in a
+     nondeterministic order. It also means that implementations do not
+     need to know anything about the provider interface, which makes
+     them easier to test in isolation. Another advantage of this
+     approach is that a developer who wants to develop their own
+     crypto provider can do so in complete isolation from the qpdf
+     library and, with just two calls, can make qpdf use their
+     provider in their application. If they decided to contribute
+     their code, plugging it into the qpdf library would require a
+     very small change to qpdf's source code.
+    </para>
+    <para>
+     The decision to make the crypto provider selectable at runtime
+     was one I struggled with a little, but I decided to do it for
+     various reasons. Allowing an end user to switch crypto providers
+     easily could be very useful for reproducing a potential bug. If a
+     user reports a bug that some cryptographic thing is broken, I can
+     easily ask that person to try with the
+     <literal>QPDF_CRYPTO_PROVIDER</literal> variable set to different
+     values. The same could apply in the event of a performance
+     problem. This also makes it easier for qpdf's own test suite to
+     exercise code with different providers without having to make
+     every program that links with qpdf aware of the possibility of
+     multiple providers. In qpdf's continuous integration environment,
+     the entire test suite is run for each supported crypto provider.
+     This is made simple by being able to select the provider using an
+     environment variable.
+    </para>
+    <para>
+     Finally, making crypto providers selectable in this way establish
+     a pattern that I may follow again in the future for stream filter
+     providers. One could imagine a future enhancement where someone
+     could provide their own implementations for basic filters like
+     <literal>/FlateDecode</literal> or for other filters that qpdf
+     doesn't support. Implementing the registration functions and
+     internal storage of registered providers was also easier using
+     C++-11's functional interfaces, which was another reason to
+     require C++-11 at this time.
+    </para>
+   </sect2>
+  </sect1>
   <sect1 id="ref.packaging">
    <title>Notes for Packagers</title>
    <para>
@@ -250,6 +468,13 @@ make
     <itemizedlist>
      <listitem>
       <para>
+       Make sure you are getting the intended behavior with regard to
+       crypto providers. Read <xref linkend="ref.crypto.build"/> for
+       details.
+      </para>
+     </listitem>
+     <listitem>
+      <para>
        Passing <option>--enable-show-failed-test-output</option> to
        <command>./configure</command> will cause any failed test
        output to be written to the console. This can be very useful
@@ -387,6 +612,17 @@ make
       </listitem>
      </varlistentry>
      <varlistentry>
+      <term><option>--show-crypto</option></term>
+      <listitem>
+       <para>
+        Show a list of available crypto providers, each on a line by
+        itself. The default provider is always listed first. See <xref
+        linkend="ref.crypto"/> for more information about crypto
+        providers.
+       </para>
+      </listitem>
+     </varlistentry>
+     <varlistentry>
       <term><option>--completion-bash</option></term>
       <listitem>
        <para>