Peter M. Groen / openbr

Preliminary work on not loading complete galleries during enrollment

Preliminary support for evaluating a comparison matrix on disk

Only verification mode evaluation is supported. The main issue is to avoid
loading potentially very large similarity and mask matrices into main
memory.

The basic idea is to read galleries incrementally, but there are some
complications especially related to progress counting--if we don't read a
gallery we don't know how many templates are stored in it since gallery
formats aren't nice enough to provide headers with that information.

One solution to the progress counting problem is to measure progress based
on the position of a file pointer in the gallery file (i.e. measure the
current position in the gallery file, divide by the total size of the
gallery file). This is supported by expanding the Gallery API to include a
totalSize method indicating the total size of the gallery file (or total
number of templates if that is known), and then as templates are read,
their position is stored in metadata (using the "p" key).

Several galleries are updated to respect readBlockSize, and also to store
position data in read templates.

Support for filtering out already enrolled templates in read-mode was
maintained by making the filtering an online process (part of the
enrollment pipeline) rather than a batch process done before
enrollment-proper starts.

We have no particular need to mess with internal whitespace to deal with
this issue.

QTextStream is apparently somewhat better behaved about these things than
QFile.

Better support for dealing with large galleries

For sufficiently large matrices, attempting to write out a single blob
will fail.

For sufficiently large matrices, reading as a single blob will fail

Incrementally write a file instead of accumulating all lines of a text
file, then outputting them in the destructor. This avoids maintaining a
full copy of all metadata being written to disk.

ALso, respect readBlockSize in these galleries when reading incrementally.

Address #188 (gallery names not propagated to mtxOutput)

Remove NoneTransform, it is a strict duplicate of IdentityTransform

This avoids warnings about project(file,file) shadowing
project(template,template).

Introduce transforms which operate solely on metadata

Stream based comparison

Also, minor stylistic/clarity tweaks for cascade training

Introduces MetadataTransform and UntrainableMetadataTransform, which
provide a project(file,file) interface for defining operations that only
affect metadata, and do not touch matrices.

Modify a number of transforms to inherit from these interfaces instead of
UntrainableTransform or UntrainableMetaTransform

Instead, we just prepend/append to transform lists as needed. Maybe this
is less confusing.

Move emptyRead to a higher visibility scope, and make it handle caching
metadata in addition to reading galleries. Caching still leverages
memGallery since re-implementing a cache just for FileLists would largely
be redundant.

generalized -evalDetection

Split downloading the MEDS dataset to a separate file

downloading all the datasets in downloadDatasets.sh is time consuming, and
MEDS is the only dataset used in make test, as well as
evalFaceRecognition-MEDS.sh. Splitting MEDS to a searate file simplifies
the process of validating an OpenBR build.