OpenSystemsDevelopment / qpdf

To be encapsulated later.

Also, when recovering trailer from xref streams, pick the last valid
trailer encountered rather than the first.

Fix handling of hybrid reference files in QPDF::read_xrefTable

Fix QPDF::recoverStreamLength

Tidy QPDF_Stream

1. Make class final
2. Pass og parameter by value
3. Properly initialize qpdf and og

Also, tweak QPDF::replaceObject to allow stream replacement without
violating the requirement that streams must always be indirect objects.

Also, remove QPDF::reserveStream as it does not do what the name implies
and having this as a separate methods does not aid code readability.

Ensure the the recovered stream end is not part of a different object.

Test file is bad24.pdf with stream 4 'endstream' corrupted.

QPDF::read_xrefTable ignores type 0 entries for objects in a section if an
associates XRefStm has an entry for the same object.

The spec states:

When the conforming reader searches for an object, if an entry is not
found in any given standard cross-reference section, the search shall
proceed to a cross-reference stream specified by the XRefStm entry
before looking in the previous cross-reference section,

If a deleted entry is found in a section, the XRefStm is not searched
according to the standard.

Create unresolved objects only for objects in the xref table (except during
parsing of the xref table). Do not add indirect nulls into the the object
cache as the result of a cache miss during a call to getObject except
during parsing or creation/updating from JSON. To support this behaviour,
add new private methods getObjectForParser and getObjectForJSON.

As a result of this change, dangling references are treated as direct nulls
rather than indirect nulls.

Prepare for treating indirect references differently depending on whether
we are parsing a PDF file (in which case reference to objects not in the
xref table are null even if they are in the object cache) or whether parse
from user code (in which case an indirect reference can refer to a user
created object).

Avoid unnecessary rescanning of lines and repositioning of input file.
Limit max size of tokens.

In FUTURE make various QPDFObjectHandle methods const

Throw damagedFile if max_warnings is exceeded. Change qpdf_fuzzer warnings limit to
limit to 500.

Run getAllPages as sanity check and throw an exception if too many
warnings are generated or no pages are found.

If reconstruct_xref generates more than 1000 warnings give up because the
file is so severely damaged that there is very little point continuing.

Check that xref table is not empty after recovery. Empty xref tables
disable other sanity checks.

Fixes oss-fuzz 70055

Previous test case was lost in #1221. Test file was created from
object-stream.pdf by adding a reference to itself into object stream 1 0.

As a rule, we should avoid conditional compilation is it always causes
code paths that are sometimes not even seen lexically by the compiler.
Also, we want the actual code being fuzzed to be as close as possible
to the real code. Conditional compilation is suitable to handle
underlying system differences.

Instead, favor configuration using callbacks or other methods that can
be triggered in the places where they need to be exercised.

Invalid entries are created when objects in the stream do not have
an existing xref entry.

Ensure objects with impossibly large ids are ignored.

Add extra fuzz test case and amend memory limit for Pl_DCT.

In QPDF::read_xrefEntry add buffer overflow test for first eol character.
Overlong f1 or f2 entries consisting only of zeros could cause a buffer
overflow.

Add fuzz testcase 69913.