The basic idea is to read galleries incrementally, but there are some
complications especially related to progress counting--if we don't read a
gallery we don't know how many templates are stored in it since gallery
formats aren't nice enough to provide headers with that information.
One solution to the progress counting problem is to measure progress based
on the position of a file pointer in the gallery file (i.e. measure the
current position in the gallery file, divide by the total size of the
gallery file). This is supported by expanding the Gallery API to include a
totalSize method indicating the total size of the gallery file (or total
number of templates if that is known), and then as templates are read,
their position is stored in metadata (using the "p" key).
Several galleries are updated to respect readBlockSize, and also to store
position data in read templates.
Support for filtering out already enrolled templates in read-mode was
maintained by making the filtering an online process (part of the
enrollment pipeline) rather than a batch process done before
enrollment-proper starts.