OpenSystemsDevelopment / qpdf

Create unresolved objects only for objects in the xref table (except during
parsing of the xref table). Do not add indirect nulls into the the object
cache as the result of a cache miss during a call to getObject except
during parsing or creation/updating from JSON. To support this behaviour,
add new private methods getObjectForParser and getObjectForJSON.

As a result of this change, dangling references are treated as direct nulls
rather than indirect nulls.

Prepare for treating indirect references differently depending on whether
we are parsing a PDF file (in which case reference to objects not in the
xref table are null even if they are in the object cache) or whether parse
from user code (in which case an indirect reference can refer to a user
created object).

Avoid unnecessary rescanning of lines and repositioning of input file.
Limit max size of tokens.

In FUTURE make various QPDFObjectHandle methods const

Throw damagedFile if max_warnings is exceeded. Change qpdf_fuzzer warnings limit to
limit to 500.

Run getAllPages as sanity check and throw an exception if too many
warnings are generated or no pages are found.

If reconstruct_xref generates more than 1000 warnings give up because the
file is so severely damaged that there is very little point continuing.

Check that xref table is not empty after recovery. Empty xref tables
disable other sanity checks.

Fixes oss-fuzz 70055

Previous test case was lost in #1221. Test file was created from
object-stream.pdf by adding a reference to itself into object stream 1 0.

As a rule, we should avoid conditional compilation is it always causes
code paths that are sometimes not even seen lexically by the compiler.
Also, we want the actual code being fuzzed to be as close as possible
to the real code. Conditional compilation is suitable to handle
underlying system differences.

Instead, favor configuration using callbacks or other methods that can
be triggered in the places where they need to be exercised.

Invalid entries are created when objects in the stream do not have
an existing xref entry.

Ensure objects with impossibly large ids are ignored.

Add extra fuzz test case and amend memory limit for Pl_DCT.

In QPDF::read_xrefEntry add buffer overflow test for first eol character.
Overlong f1 or f2 entries consisting only of zeros could cause a buffer
overflow.

Add fuzz testcase 69913.

Refine handling of severely damaged files

Also add new fuzz test case.

Refactor QPDF::parse_xrefEntry

QPDF::processXRefStream

Change the processed Index array to a vector of <first object, number of
entries> pairs.

Add closure damaged to create damagedPDF exceptions.

Tune processing of subsections.

Tune pointer arithmetic.

Apply temporary fix to deal with fuzz case 68915.

(Error is an integer overflow which would immediately cause a runtime error
as a result of a call to QInitCQIntC::to_size.)

Create set without creation of an intermediate vector.

Move reading of the entry from read_xrefTable to parse_xrefEntry.

Split parse_xrefEntry into two new methods read_xrefEntry and
read_bad_xrefEntry. read_xrefEntry is optimised for reading
correct entries. To handle incorrect entries it calls read_bad_xrefEntry,
which is largely unchanged from parse_xrefEntry.

Fix two errors introduced in #1110 and #1112. Since
#1110, encountering the invalid indirect reference #1110
-2147483648 n R produces an integer underflow which, if
 undetected, immediately trigger a logic error. Since
 #1112, object -1 0 R may be incorrectly identified as
 an earlier generation of itself and deleted,
 invalidating a live iterator.