Commit b8f20fe34c1e91e312e904cf828cb164dac2326e
Committed by
GitHub
GitHub
Merge pull request #998 from m-holger/todo
Add content table to TODO file
Showing
4 changed files
with
701 additions
and
845 deletions
.git-blame-ignore-revs
TODO deleted
| 1 | -Always | |
| 2 | -====== | |
| 3 | - | |
| 4 | -* Evaluate issues tagged with `next` and `bug`. Remember to check | |
| 5 | - discussions and pull requests in addition to regular issues. | |
| 6 | -* When close to release, make sure external-libs is building and | |
| 7 | - follow instructions in ../external-libs/README | |
| 8 | - | |
| 9 | -Next | |
| 10 | -==== | |
| 11 | - | |
| 12 | -* Fix #874 -- make args in --encrypt to match the json and make | |
| 13 | - positional fill in the gaps | |
| 14 | -* Maybe fix #553 -- use file times for attachments | |
| 15 | -* std::string_view transition -- work being done by m-holger | |
| 16 | -* Break ground on "Document-level work" -- TODO-pages.md lives on a | |
| 17 | - separate branch. | |
| 18 | -* Standard for CLI and Job JSON support for JSON-based command-line | |
| 19 | - arguments. Come up with a standard way of supporting command-line | |
| 20 | - arguments that take JSON specifications of things so that | |
| 21 | - * there is a predictable way to indicate whether an argument is a | |
| 22 | - file or a JSON blob | |
| 23 | - * with QPDFJob JSON, make sure it is possible to directly include | |
| 24 | - the JSON rather than having to stringify a JSON blob | |
| 25 | - * One option might be to prepend file:// to a filename or otherwise | |
| 26 | - to take a JSON blob. We could have that as a particular type of | |
| 27 | - argument that would behave properly for both job JSON and CLI. | |
| 28 | - | |
| 29 | - | |
| 30 | -Possible future JSON enhancements | |
| 31 | -================================= | |
| 32 | - | |
| 33 | -* Consider not including unreferenced objects and trimming the trailer | |
| 34 | - in the same way that QPDFWriter does (except don't remove `/ID`). | |
| 35 | - This means excluding the linearization dictionary and hint stream, | |
| 36 | - the encryption dictionary, all keys from trailer that are removed by | |
| 37 | - QPDFWriter::getTrimmedTrailer except `/ID`, any object streams, and | |
| 38 | - the xref stream as long as all those objects are unreferenced. (They | |
| 39 | - always should be, but there could be some bizarre case of someone | |
| 40 | - creating a PDF file that has an indirect reference to one of those, | |
| 41 | - in which case we need to preserve it.) If this is done, make | |
| 42 | - `--preserve-unreferenced` preserve unreference objects and also | |
| 43 | - those extra keys. Search for "linear" and "trailer" in json.rst to | |
| 44 | - update the various places in the documentation that discuss this. | |
| 45 | - Also update the help for --json and --preserve-unreferenced. | |
| 46 | - | |
| 47 | -* Add to JSON output the information available from a few additional | |
| 48 | - informational options: | |
| 49 | - | |
| 50 | - * --check: add but maybe not by default? | |
| 51 | - | |
| 52 | - * --show-linearization: add but maybe not by default? Also figure | |
| 53 | - out whether warnings reported for some of the PDF specs (1.7) are | |
| 54 | - qpdf problems. This may not be worth adding in the first | |
| 55 | - increment. | |
| 56 | - | |
| 57 | - * --show-xref: add | |
| 58 | - | |
| 59 | -* Consider having --check, --show-encryption, etc., just select the | |
| 60 | - right keys when in json mode. I don't think I want check on by | |
| 61 | - default, so that might be different. | |
| 62 | - | |
| 63 | -* Consider having warnings be included in the json in a "warnings" key | |
| 64 | - in json mode. | |
| 65 | - | |
| 66 | -QPDFJob | |
| 67 | -======= | |
| 68 | - | |
| 69 | -Here are some ideas for QPDFJob that didn't make it into 10.6. Not all | |
| 70 | -of these are necessarily good -- just things to consider. | |
| 71 | - | |
| 72 | -* How do we chain jobs? The idea would be that the input and/or output | |
| 73 | - of a QPDFJob could be a QPDF object rather than a file. For input, | |
| 74 | - it's pretty easy. For output, none of the output-specific options | |
| 75 | - (encrypt, compress-streams, objects-streams, etc.) would have any | |
| 76 | - affect, so we would have to treat this like inspect for error | |
| 77 | - checking. The QPDF object in the state where it's ready to be sent | |
| 78 | - off to QPDFWriter would be used as the input to the next QPDFJob. | |
| 79 | - For the job json, I think we can have the output be an identifier | |
| 80 | - that can be used as the input for another QPDFJob. For a json file, | |
| 81 | - we could the top level detect if it's an array with the convention | |
| 82 | - that exactly one has an output, or we could have a subkey with other | |
| 83 | - job definitions or something. Ideally, any input | |
| 84 | - (copy-attachments-from, pages, etc.) could use a QPDF object. It | |
| 85 | - wouldn't surprise me if this exposes bugs in qpdf around foreign | |
| 86 | - streams as this has been a relatively fragile area before. | |
| 87 | - | |
| 88 | -Documentation | |
| 89 | -============= | |
| 90 | - | |
| 91 | -* Do a full pass through the documentation. | |
| 92 | - | |
| 93 | - * Make sure `qpdf` is consistent. Use QPDF when just referring to | |
| 94 | - the package. | |
| 95 | - * Make sure markup is consistent | |
| 96 | - * Autogenerate where possible | |
| 97 | - * Consider which parts might be good candidates for moving to the | |
| 98 | - wiki. | |
| 99 | - | |
| 100 | -* Commit 'Manual - enable line wrapping in table cells' from | |
| 101 | - Mon Jan 17 12:22:35 2022 +0000 enables table cell wrapping. See if | |
| 102 | - this can be incorporated directly into sphinx_rtd_theme and the | |
| 103 | - workaround can be removed. | |
| 104 | - | |
| 105 | -* When possible, update the debian package to include docs again. See | |
| 106 | - https://bugs.debian.org/1004159 for details. | |
| 107 | - | |
| 108 | -Document-level work | |
| 109 | -=================== | |
| 110 | - | |
| 111 | -* Ideas here may by superseded by #593. | |
| 112 | - | |
| 113 | -* QPDFPageCopier -- object for moving pages around within files or | |
| 114 | - between files and performing various transformations. Reread/rewrite | |
| 115 | - _page-selection in the manual if needed. | |
| 116 | - | |
| 117 | - * Handle all the stuff of pages and split-pages | |
| 118 | - * Do n-up, booklet, collation | |
| 119 | - * Look through cli and see what else...flatten-*? | |
| 120 | - * See comments in QPDFPageDocumentHelper.hh for addPage -- search | |
| 121 | - for "a future version". | |
| 122 | - * Make it efficient for bulk operations | |
| 123 | - * Make certain doc-level features selectable | |
| 124 | - * qpdf.cc should do all its page operations, including | |
| 125 | - overlay/underlay, splitting, and merging, using this | |
| 126 | - * There should also be example code | |
| 127 | - | |
| 128 | -* After doc-level checks are in, call --check on the output files in | |
| 129 | - the "Copy Annotations" tests. | |
| 130 | - | |
| 131 | -* Document-level checks. For example, for forms, make sure all form | |
| 132 | - fields point to an annotation on exactly one page as well as that | |
| 133 | - all widget annotations are associated with a form field. Hook this | |
| 134 | - into QPDFPageCopier as well as the doc helpers. Make sure it is | |
| 135 | - called from --check. | |
| 136 | - | |
| 137 | -* See also issues tagged with "pages". Include closed issues. | |
| 138 | - | |
| 139 | -* Add flags to CLI to select which document-level options to | |
| 140 | - preserve or not preserve. We will probably need a pair of mutually | |
| 141 | - exclusive, repeatable options with a way to specify all, none, only | |
| 142 | - {x,y}, or all but {x,y}. | |
| 143 | - | |
| 144 | -* If a page contains a reference a file attachment annotation, when | |
| 145 | - that page is copied, if the file attachment appears in the top-level | |
| 146 | - EmbeddedFiles tree, that entry should be preserved in the | |
| 147 | - destination file. Otherwise, we probably will require the use of | |
| 148 | - --copy-attachments-from to preserve these. What will the strategy be | |
| 149 | - for deduplicating in the automatic case? | |
| 150 | - | |
| 151 | -Text Appearance Streams | |
| 152 | -======================= | |
| 153 | - | |
| 154 | -This is a list of known issues with text appearance streams and things | |
| 155 | -we might do about it. | |
| 156 | - | |
| 157 | -* For variable text, the spec says to pull any resources from /DR that | |
| 158 | - are referenced in /DA but if the resource dictionary already has | |
| 159 | - that resource, just use the one that's there. The current code looks | |
| 160 | - only for /Tf and adds it if needed. We might want to instead merge | |
| 161 | - /DR with resources and then remove anything that's unreferenced. We | |
| 162 | - have all the code required for that in ResourceFinder except | |
| 163 | - TfFinder also gets the font size, which ResourceFinder doesn't do. | |
| 164 | - | |
| 165 | -* There are things we are missing because we don't look at font | |
| 166 | - metrics. The code from TextBuilder (work) has almost everything in | |
| 167 | - it that is required. Once we have knowledge of character widths, we | |
| 168 | - can support quadding and multiline text fields (/Ff 4096), and we | |
| 169 | - can potentially squeeze text to fit into a field. For multiline, | |
| 170 | - first squeeze vertically down to the font height, then squeeze | |
| 171 | - horizontally with Tz. For single line, squeeze horizontally with Tz. | |
| 172 | - If we use Tz, issue a warning. | |
| 173 | - | |
| 174 | -* When mapping characters to widths, we will need to care about | |
| 175 | - character encoding. For built-in fonts, we can create a map from | |
| 176 | - Unicode code point to width and then go from the font's encoding to | |
| 177 | - unicode to the width. See misc/character-encoding/ (not on github) | |
| 178 | - and font metric information for the 14 standard fonts in my local | |
| 179 | - pdf-spec directory. | |
| 180 | - | |
| 181 | -* Once we know about character widths, we can correctly support | |
| 182 | - auto-sized variable text fields (0 Tf). If this is fixed, search for | |
| 183 | - "auto-sized" in cli.rst. | |
| 184 | - | |
| 185 | -Fuzz Errors | |
| 186 | -=========== | |
| 187 | - | |
| 188 | -* https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=<N> | |
| 189 | - | |
| 190 | -* Ignoring these: | |
| 191 | - * Out of memory in dct: 35001, 32516 | |
| 192 | - | |
| 193 | -External Libraries | |
| 194 | -================== | |
| 195 | - | |
| 196 | -Current state (10.0.2): | |
| 197 | - | |
| 198 | -* qpdf/external-libs repository builds external-libs on a schedule. | |
| 199 | - It detects and downloads the latest versions of zlib, jpeg, and | |
| 200 | - openssl and creates source and binary distribution zip files in an | |
| 201 | - artifact called "distribution". | |
| 202 | - | |
| 203 | -* Releases in qpdf/external-libs are made manually. They contain | |
| 204 | - qpdf-external-libs-{bin,src}.zip. | |
| 205 | - | |
| 206 | -* The qpdf build finds the latest non-prerelease release and downloads | |
| 207 | - the qpdf-external-libs-*.zip files from the releases in the setup | |
| 208 | - stage. | |
| 209 | - | |
| 210 | -* To upgrade to a new version of external-libs, create a new release | |
| 211 | - of qpdf/external-libs (see README-maintainer in external-libs) from | |
| 212 | - the distribution artifact of the most recent successful build after | |
| 213 | - ensuring that it works. | |
| 214 | - | |
| 215 | -Desired state: | |
| 216 | - | |
| 217 | -* The qpdf/external-libs repository should create release candidates. | |
| 218 | - Ideally, every scheduled run would make its zip files available. A | |
| 219 | - personal access token with actions:read scope for the | |
| 220 | - qpdf/external-libs repository is required to download the artifact | |
| 221 | - from an action run, and qpdf/qpdf's secrets.GITHUB_TOKEN doesn't | |
| 222 | - have this access. We could create a service account for this | |
| 223 | - purpose. As an alternative, we could have a draft release in | |
| 224 | - qpdf/external-libs that the qpdf/external-libs build could update | |
| 225 | - with each candidate. It may also be possible to solve this by | |
| 226 | - developing a simple GitHub app. | |
| 227 | - | |
| 228 | -* Scheduled runs of the qpdf build in the qpdf/qpdf repository (not a | |
| 229 | - fork or pull request) could download external-libs from the release | |
| 230 | - candidate area instead of the latest stable release. Pushes to the | |
| 231 | - build branch should still use the latest release so it always | |
| 232 | - matches the main branch. | |
| 233 | - | |
| 234 | -* Periodically, we would create a release of external-libs from the | |
| 235 | - release candidate zip files. This could be done safely because we | |
| 236 | - know the latest qpdf works with it. This could be done at least | |
| 237 | - before every release of qpdf, but potentially it could be done at | |
| 238 | - other times, such as when a new dependency version is available or | |
| 239 | - after some period of time. | |
| 240 | - | |
| 241 | -Other notes: | |
| 242 | - | |
| 243 | -* The external-libs branch in qpdf/qpdf was never documented. We might | |
| 244 | - be able to get away with deleting it. | |
| 245 | - | |
| 246 | -* See README-maintainer in qpdf/external-libs for information on | |
| 247 | - creating a release. This could be at least partially scripted in a | |
| 248 | - way that works for the qpdf/qpdf repository as well since they are | |
| 249 | - very similar. | |
| 250 | - | |
| 251 | -ABI Changes | |
| 252 | -=========== | |
| 253 | - | |
| 254 | -This is a list of changes to make next time there is an ABI change. | |
| 255 | -Comments appear in the code prefixed by "ABI". | |
| 256 | - | |
| 257 | -Always: | |
| 258 | -* Search for ABI in source and header files | |
| 259 | -* Search for "[[deprecated" to find deprecated APIs that can be removed | |
| 260 | -* Search for issues, pull requests, and discussions with the "abi" label | |
| 261 | -* Check discussion "qpdf X planning" where X is the next major | |
| 262 | - version. This should be tagged `abi` | |
| 263 | - | |
| 264 | -For qpdf 12, see https://github.com/qpdf/qpdf/discussions/785 | |
| 265 | - | |
| 266 | -C++ Version Changes | |
| 267 | -=================== | |
| 268 | - | |
| 269 | -Use | |
| 270 | -// C++NN: ... | |
| 271 | -to mark places in the code that should be updated when we require at | |
| 272 | -least that version of C++. | |
| 273 | - | |
| 274 | -Page splitting/merging | |
| 275 | -====================== | |
| 276 | - | |
| 277 | - * Update page splitting and merging to handle document-level | |
| 278 | - constructs with page impact such as interactive forms and article | |
| 279 | - threading. Check keys in the document catalog for others, such as | |
| 280 | - outlines, page labels, thumbnails, and zones. For threads, | |
| 281 | - Subramanyam provided a test file; see ../misc/article-threads.pdf. | |
| 282 | - Email Q-Count: 431864 from 2009-11-03. | |
| 283 | - | |
| 284 | - * bookmarks (outlines) 12.3.3 | |
| 285 | - * support bookmarks when merging | |
| 286 | - * prune bookmarks that don't point to a surviving page when merging | |
| 287 | - or splitting | |
| 288 | - * make sure conflicting named destinations work possibly test by | |
| 289 | - including the same file by two paths in a merge | |
| 290 | - * see also comments in issue 343 | |
| 291 | - | |
| 292 | - Note: original implementation of bookmark preservation for split | |
| 293 | - pages caused a very high performance hit. The problem was | |
| 294 | - introduced in 313ba081265f69ac9a0324f9fe87087c72918191 and reverted | |
| 295 | - in the commit that adds this paragraph. The revert includes marking | |
| 296 | - a few tests cases as $td->EXPECT_FAILURE. When properly coded, the | |
| 297 | - test cases will need to be adjusted to only include the parts of | |
| 298 | - the outlines that are actually copied. The tests in question are | |
| 299 | - "split page with outlines". When implementing properly, ensure that | |
| 300 | - the performance is not adversely affected by timing split-pages on | |
| 301 | - a large file with complex outlines such as the PDF specification. | |
| 302 | - | |
| 303 | - When pruning outlines, keep all outlines in the hierarchy that are | |
| 304 | - above an outline for a page we care about. If one of the ancestor | |
| 305 | - outlines points to a non-existent page, clear its dest. If an | |
| 306 | - outline does not have any children that point to pages in the | |
| 307 | - document, just omit it. | |
| 308 | - | |
| 309 | - Possible strategy: | |
| 310 | - * resolve all named destinations to explicit destinations | |
| 311 | - * concatenate top-level outlines | |
| 312 | - * prune outlines whose dests don't point to a valid page | |
| 313 | - * recompute all /Count fields | |
| 314 | - | |
| 315 | - Test files | |
| 316 | - * page-labels-and-outlines.pdf: old file with both page labels and | |
| 317 | - outlines. All destinations are explicit destinations. Each page | |
| 318 | - has Potato and a number. All titles are feline names. | |
| 319 | - * outlines-with-actions.pdf: mixture of explicit destinations, | |
| 320 | - named destinations, goto actions with explicit destinations, and | |
| 321 | - goto actions with named destinations; uses /Dests key in names | |
| 322 | - dictionary. Each page has Salad and a number. All titles are | |
| 323 | - silly words. One destination is an indirect object. | |
| 324 | - * outlines-with-old-root-dests.pdf: like outlines-with-actions | |
| 325 | - except it uses the PDF-1.1 /Dests dictionary for named | |
| 326 | - destinations, and each page has Soup and a number. Also pages are | |
| 327 | - numbered with upper-case Roman numerals starting with 0. All | |
| 328 | - titles are silly words preceded by a bullet. | |
| 329 | - | |
| 330 | - If outline handling is significantly improved, see | |
| 331 | - ../misc/bad-outlines/bad-outlines.pdf and email: | |
| 332 | - https://mail.google.com/mail/u/0/#search/rfc822msgid%3A02aa01d3d013%249f766990%24de633cb0%24%40mono.hr) | |
| 333 | - | |
| 334 | - * Form fields: should be similar to outlines. | |
| 335 | - | |
| 336 | -Analytics | |
| 337 | -========= | |
| 338 | - | |
| 339 | -Consider features that make it easier to detect certain patterns in | |
| 340 | -PDF files. The information below could be computed using an external | |
| 341 | -program that reads the existing json, but if it's useful enough, we | |
| 342 | -could add it directly to the json output. | |
| 343 | - | |
| 344 | - * Add to "pages" in the json: | |
| 345 | - * "inheritsresources": bool; whether there are any inherited | |
| 346 | - attributes from ancestor page tree nodes | |
| 347 | - * "sharedresources": a list of indirect objects that are | |
| 348 | - "/Resources" dictionaries or "XObject" resource dictionary subkeys | |
| 349 | - of either the page itself or of any form XObject referenced by the | |
| 350 | - page. | |
| 351 | - | |
| 352 | - * Add to "objectinfo" in json: "directpagerefcount": the number of | |
| 353 | - pages that directly reference this object (i.e., you can find an | |
| 354 | - indirect reference to the object in the page dictionary without | |
| 355 | - traversing over any indirect objects) | |
| 356 | - | |
| 357 | -General | |
| 358 | -======= | |
| 359 | - | |
| 360 | -NOTE: Some items in this list refer to files in my personal home | |
| 361 | -directory or that are otherwise not publicly accessible. This includes | |
| 362 | -things sent to me by email that are specifically not public. Even so, | |
| 363 | -I find it useful to make reference to them in this list. | |
| 364 | - | |
| 365 | -* Consider enabling code scanning on GitHub. | |
| 366 | - | |
| 367 | -* Add an option --ignore-encryption to ignore encryption information | |
| 368 | - and treat encrypted files as if they weren't encrypted. This should | |
| 369 | - make it possible to solve #598 (--show-encryption without a | |
| 370 | - password). We'll need to make sure we don't try to filter any | |
| 371 | - streams in this mode. Ideally we should be able to combine this with | |
| 372 | - --json so we can look at the raw encrypted strings and streams if we | |
| 373 | - want to, though be sure to document that the resulting JSON won't be | |
| 374 | - convertible back to a valid PDF. Since providing the password may | |
| 375 | - reveal additional details, --show-encryption could potentially retry | |
| 376 | - with this option if the first time doesn't work. Then, with the file | |
| 377 | - open, we can read the encryption dictionary normally. If this is | |
| 378 | - done, search for "raw, encrypted" in json.rst. | |
| 379 | - | |
| 380 | -* In libtests, separate executables that need the object library | |
| 381 | - from those that strictly use public API. Move as many of the test | |
| 382 | - drivers from the qpdf directory into the latter category as long | |
| 383 | - as doing so isn't too troublesome from a coverage standpoint. | |
| 384 | - | |
| 385 | -* Consider generating a non-flat pages tree before creating output to | |
| 386 | - better handle files with lots of pages. If there are more than 256 | |
| 387 | - pages, add a second layer with the second layer nodes having no more | |
| 388 | - than 256 nodes and being as evenly sizes as possible. Don't worry | |
| 389 | - about the case of more than 65,536 pages. If the top node has more | |
| 390 | - than 256 children, we'll live with it. This is only safe if all | |
| 391 | - intermediate page nodes have only /Kids, /Parent, /Type, and /Count. | |
| 392 | - | |
| 393 | -* Look at https://bestpractices.coreinfrastructure.org/en | |
| 394 | - | |
| 395 | -* Consider adding fuzzer code for JSON | |
| 396 | - | |
| 397 | -* Rework tests so that nothing is written into the source directory. | |
| 398 | - Ideally then the entire build could be done with a read-only | |
| 399 | - source tree. | |
| 400 | - | |
| 401 | -* Large file tests fail with linux32 before and after cmake. This was | |
| 402 | - first noticed after 10.6.3. I don't think it's worth fixing. | |
| 403 | - | |
| 404 | -* Consider updating the fuzzer with code that exercises | |
| 405 | - copyAnnotations, file attachments, and name and number trees. Check | |
| 406 | - fuzzer coverage. | |
| 407 | - | |
| 408 | -* Add code for creation of a file attachment annotation. It should | |
| 409 | - also be possible to create a widget annotation and a form field. | |
| 410 | - Update the pdf-attach-file.cc example with new APIs when ready. | |
| 411 | - | |
| 412 | -* Flattening of form XObjects seems like something that would be | |
| 413 | - useful in the library. We are seeing more cases of completely valid | |
| 414 | - PDF files with form XObjects that cause problems in other software. | |
| 415 | - Flattening of form XObjects could be a useful way to work around | |
| 416 | - those issues or to prepare files for additional processing, making | |
| 417 | - it possible for users of the qpdf library to not be concerned about | |
| 418 | - form XObjects. This could be done recursively; i.e., we could have a | |
| 419 | - method to embed a form XObject into whatever contains it, whether | |
| 420 | - that is a form XObject or a page. This would require more | |
| 421 | - significant interpretation of the content stream. We would need a | |
| 422 | - test file in which the placement of the form XObject has to be in | |
| 423 | - the right place, e.g., the form XObject partially obscures earlier | |
| 424 | - code and is partially obscured by later code. Keys in the resource | |
| 425 | - dictionary may need to be changed -- create test cases with lots of | |
| 426 | - duplicated/overlapping keys. | |
| 427 | - | |
| 428 | -* Part of closed_file_input_source.cc is disabled on Windows because | |
| 429 | - of odd failures. It might be worth investigating so we can fully | |
| 430 | - exercise this in the test suite. That said, ClosedFileInputSource | |
| 431 | - is exercised elsewhere in qpdf's test suite, so this is not that | |
| 432 | - pressing. | |
| 433 | - | |
| 434 | -* If possible, consider adding CCITT3, CCITT4, or any other easy | |
| 435 | - filters. For some reference code that we probably can't use but may | |
| 436 | - be handy anyway, see | |
| 437 | - http://partners.adobe.com/public/developer/ps/sdk/index_archive.html | |
| 438 | - | |
| 439 | -* If possible, support the following types of broken files: | |
| 440 | - | |
| 441 | - - Files that have no whitespace token after "endobj" such that | |
| 442 | - endobj collides with the start of the next object | |
| 443 | - | |
| 444 | - - See ../misc/broken-files | |
| 445 | - | |
| 446 | - - See ../misc/bad-files-issue-476. This directory contains a | |
| 447 | - snapshot of the google doc and linked PDF files from issue #476. | |
| 448 | - Please see the issue for details. | |
| 449 | - | |
| 450 | -* Additional form features | |
| 451 | - * set value from CLI? Specify title, and provide way to | |
| 452 | - disambiguate, probably by giving objgen of field | |
| 453 | - | |
| 454 | -* Pl_TIFFPredictor is pretty slow. | |
| 455 | - | |
| 456 | -* Support for handling file names with Unicode characters in Windows | |
| 457 | - is incomplete. qpdf seems to support them okay from a functionality | |
| 458 | - standpoint, and the right thing happens if you pass in UTF-8 | |
| 459 | - encoded filenames to QPDF library routines in Windows (they are | |
| 460 | - converted internally to wchar_t*), but file names are encoded in | |
| 461 | - UTF-8 on output, which doesn't produce nice error messages or | |
| 462 | - output on Windows in some cases. | |
| 463 | - | |
| 464 | -* If we ever wanted to do anything more with character encoding, see | |
| 465 | - ../misc/character-encoding/, which includes machine-readable dump | |
| 466 | - of table D.2 in the ISO-32000 PDF spec. This shows the mapping | |
| 467 | - between Unicode, StandardEncoding, WinAnsiEncoding, | |
| 468 | - MacRomanEncoding, and PDFDocEncoding. | |
| 469 | - | |
| 470 | -* Some test cases on bad files fail because qpdf is unable to find | |
| 471 | - the root dictionary when it fails to read the trailer. Recovery | |
| 472 | - could find the root dictionary and even the info dictionary in | |
| 473 | - other ways. In particular, issue-202.pdf can be opened by evince, | |
| 474 | - and there's no real reason that qpdf couldn't be made to be able to | |
| 475 | - recover that file as well. | |
| 476 | - | |
| 477 | -* Audit every place where qpdf allocates memory to see whether there | |
| 478 | - are cases where malicious inputs could cause qpdf to attempt to | |
| 479 | - grab very large amounts of memory. Certainly there are cases like | |
| 480 | - this, such as if a very highly compressed, very large image stream | |
| 481 | - is requested in a buffer. Hopefully normal input to output | |
| 482 | - filtering doesn't ever try to do this. QPDFWriter should be checked | |
| 483 | - carefully too. See also bugs/private/from-email-663916/ | |
| 484 | - | |
| 485 | -* Interactive form modification: | |
| 486 | - https://github.com/qpdf/qpdf/issues/213 contains a good discussion | |
| 487 | - of some ideas for adding methods to modify annotations and form | |
| 488 | - fields if we want to make it easier to support modifications to | |
| 489 | - interactive forms. Some of the ideas have been implemented, and | |
| 490 | - some of the probably never will be implemented, but it's worth a | |
| 491 | - read if there is an intention to work on this. In the issue, search | |
| 492 | - for "Regarding write functionality", and read that comment and the | |
| 493 | - responses to it. | |
| 494 | - | |
| 495 | -* Look at ~/Q/pdf-collection/forms-from-appian/ | |
| 496 | - | |
| 497 | -* When decrypting files with /R=6, hash_V5 is called more than once | |
| 498 | - with the same inputs. Caching the results or refactoring to reduce | |
| 499 | - the number of identical calls could improve performance for | |
| 500 | - workloads that involve processing large numbers of small files. | |
| 501 | - | |
| 502 | -* Consider adding a method to balance the pages tree. It would call | |
| 503 | - pushInheritedAttributesToPage, construct a pages tree from scratch, | |
| 504 | - and replace the /Pages key of the root dictionary with the new | |
| 505 | - tree. | |
| 506 | - | |
| 507 | -* Study what's required to support savable forms that can be saved by | |
| 508 | - Adobe Reader. Does this require actually signing the document with | |
| 509 | - an Adobe private key? Search for "Digital signatures" in the PDF | |
| 510 | - spec, and look at ~/Q/pdf-collection/form-with-full-save.pdf, which | |
| 511 | - came from Adobe's example site. See also | |
| 512 | - ../misc/digital-sign-from-trueroad/ and | |
| 513 | - ../misc/digital-signatures/digitally-signed-pdf-xfa.pdf. If digital | |
| 514 | - signatures are implemented, update the docs on crypto providers, | |
| 515 | - which mention that this may happen in the future. | |
| 516 | - | |
| 517 | -* Qpdf does not honor /EFF when adding new file attachments. When it | |
| 518 | - encrypts, it never generates streams with explicit crypt filters. | |
| 519 | - Prior to 10.2, there was an incorrect attempt to treat /EFF as a | |
| 520 | - default value for decrypting file attachment streams, but it is not | |
| 521 | - supposed to mean that. Instead, it is intended for conforming | |
| 522 | - writers to obey this when adding new attachments. Qpdf is not a | |
| 523 | - conforming writer in that respect. | |
| 524 | - | |
| 525 | -* The whole xref handling code in the QPDF object allows the same | |
| 526 | - object with more than one generation to coexist, but a lot of logic | |
| 527 | - assumes this isn't the case. Anything that creates mappings only | |
| 528 | - with the object number and not the generation is this way, | |
| 529 | - including most of the interaction between QPDFWriter and QPDF. If | |
| 530 | - we wanted to allow the same object with more than one generation to | |
| 531 | - coexist, which I'm not sure is allowed, we could fix this by | |
| 532 | - changing xref_table. Alternatively, we could detect and disallow | |
| 533 | - that case. In fact, it appears that Adobe reader and other PDF | |
| 534 | - viewing software silently ignores objects of this type, so this is | |
| 535 | - probably not a big deal. | |
| 536 | - | |
| 537 | -* From a suggestion in bug 3152169, consider having an option to | |
| 538 | - re-encode inline images with an ASCII encoding. | |
| 539 | - | |
| 540 | -* From github issue 2, provide more in-depth output for examining | |
| 541 | - hint stream contents. Consider adding on option to provide a | |
| 542 | - human-readable dump of linearization hint tables. This should | |
| 543 | - include improving the 'overflow reading bit stream' message as | |
| 544 | - reported in issue #2. There are multiple calls to stopOnError in | |
| 545 | - the linearization checking code. Ideally, these should not | |
| 546 | - terminate checking. It would require re-acquiring an understanding | |
| 547 | - of all that code to make the checks more robust. In particular, | |
| 548 | - it's hard to look at the code and quickly determine what is a true | |
| 549 | - logic error and what could happen because of malformed user input. | |
| 550 | - See also ../misc/linearization-errors. | |
| 551 | - | |
| 552 | -* If I ever decide to make appearance stream-generation aware of | |
| 553 | - fonts or font metrics, see email from Tobias with Message-ID | |
| 554 | - <5C3C9C6C.8000102@thax.hardliners.org> dated 2019-01-14. | |
| 555 | - | |
| 556 | -* Look at places in the code where object traversal is being done and, | |
| 557 | - where possible, try to avoid it entirely or at least avoid ever | |
| 558 | - traversing the same objects multiple times. | |
| 559 | - | |
| 560 | ----------------------------------------------------------------------- | |
| 561 | - | |
| 562 | -HISTORICAL NOTES | |
| 563 | - | |
| 564 | -Performance | |
| 565 | -=========== | |
| 566 | - | |
| 567 | -As described in https://github.com/qpdf/qpdf/issues/401, there was | |
| 568 | -great performance degradation between qpdf 7.1.1 and 9.1.1. Doing a | |
| 569 | -bisect between dac65a21fb4fa5f871e31c314280b75adde89a6c and | |
| 570 | -release-qpdf-7.1.1, I found several commits that damaged performance. | |
| 571 | -I fixed some of them to improve performance by about 70% (as measured | |
| 572 | -by saying that old times were 170% of new times). The remaining | |
| 573 | -commits that broke performance either can't be correct because they | |
| 574 | -would re-introduce an old bug or aren't worth correcting because of | |
| 575 | -the high value they offer relative to a relatively low penalty. For | |
| 576 | -historical reference, here are the commits. The numbers are the time | |
| 577 | -in seconds on the machine I happened to be using of splitting the | |
| 578 | -first 100 pages of PDF32000_2008.pdf 20 times and taking an average | |
| 579 | -duration. | |
| 580 | - | |
| 581 | -Commits that broke performance: | |
| 582 | - | |
| 583 | -* d0e99f195a987c483bbb6c5449cf39bee34e08a1 -- object description and | |
| 584 | - context: 0.39 -> 0.45 | |
| 585 | -* a01359189b32c60c2d55b039f7aefd6c3ce0ebde (minus 313ba08) -- fix | |
| 586 | - dangling references: 0.55 -> 0.6 | |
| 587 | -* e5f504b6c5dc34337cc0b316b4a7b1fca7e614b1 -- sparse array: 0.6 -> 0.62 | |
| 588 | - | |
| 589 | -Other intermediate steps that were previously fixed: | |
| 590 | - | |
| 591 | -* 313ba081265f69ac9a0324f9fe87087c72918191 -- copy outlines into | |
| 592 | - split: 0.55 -> 4.0 | |
| 593 | -* a01359189b32c60c2d55b039f7aefd6c3ce0ebde -- fix dangling references: | |
| 594 | - 4.0 -> 9.0 | |
| 595 | - | |
| 596 | -This commit fixed the awful problem introduced in 313ba081: | |
| 597 | - | |
| 598 | -* a5a016cdd26a8e5c99e5f019bc30d1bdf6c050a2 -- revert outline | |
| 599 | - preservation: 9.0 -> 0.6 | |
| 600 | - | |
| 601 | -Note that the fix dangling references commit had a much worse impact | |
| 602 | -prior to removing the outline preservation, so I also measured its | |
| 603 | -impact in isolation. | |
| 604 | - | |
| 605 | -A few important lessons (in README-maintainer) | |
| 606 | - | |
| 607 | -* Indirection through PointerHolder<Members> is expensive, and should | |
| 608 | - not be used for things that are created and destroyed frequently | |
| 609 | - such as QPDFObjectHandle and QPDFObject. | |
| 610 | -* Traversal of objects is expensive and should be avoided where | |
| 611 | - possible. | |
| 612 | - | |
| 613 | -Also, it turns out that PointerHolder is more performant than | |
| 614 | -std::shared_ptr. (This was true at the time but subsequent | |
| 615 | -implementations of std::shared_ptr became much more efficient.) | |
| 616 | - | |
| 617 | -QPDFPagesTree | |
| 618 | -============= | |
| 619 | - | |
| 620 | -On a few occasions, I have considered implementing a QPDFPagesTree | |
| 621 | -object that would allow the document's original page tree structure to | |
| 622 | -be preserved. See comments at the top QPDF_pages.cc for why this was | |
| 623 | -abandoned. | |
| 624 | - | |
| 625 | -Partial work is in refs/attic/QPDFPagesTree. QPDFPageTree is mostly | |
| 626 | -implemented and mostly tested. There are not enough cases of different | |
| 627 | -kinds of operations (pclm, linearize, json, etc.) with non-flat pages | |
| 628 | -trees. Insertion is not implemented. Insertion is potentially complex | |
| 629 | -because of the issue of inherited objects. We will have to call | |
| 630 | -pushInheritedAttributesToPage before adding any pages to the pages | |
| 631 | -tree. The test suite is failing on that branch. | |
| 632 | - | |
| 633 | -Some parts of page tree repair are silent (no warnings). All page tree | |
| 634 | -repair should warn. The reason is that page tree repair will change | |
| 635 | -object numbers, and knowing that is important when working with JSON | |
| 636 | -output. | |
| 637 | - | |
| 638 | -If we were to do this, we would still need keep a pages cache for | |
| 639 | -efficient insertion. There's no reason we can't keep a vector of page | |
| 640 | -objects up to date and just do a traversal the first time we do | |
| 641 | -getAllPages just like we do now. The difference is that we would not | |
| 642 | -flatten the pages tree. It would be useful to go through QPDF_pages | |
| 643 | -and reimplement everything without calling flattenPagesTree. Then we | |
| 644 | -can remove flattenPagesTree, which is private. That said, with the | |
| 645 | -addition of creating non-flat pages trees, there is really no reason | |
| 646 | -not to flatten the pages tree for internal use. | |
| 647 | - | |
| 648 | -In its current state, QPDFPagesTree does not proactively fix /Type or | |
| 649 | -correct page objects that are used multiple times. You have to | |
| 650 | -traverse the pages tree to trigger this operation. It would be nice if | |
| 651 | -we would do that somewhere but not do it more often than necessary so | |
| 652 | -isPagesObject and isPageObject are reliable and can be made more | |
| 653 | -reliable. Maybe add a validate or repair function? It should also make | |
| 654 | -sure /Count and /Parent are correct. | |
| 655 | - | |
| 656 | -Rejected Ideas | |
| 657 | -============== | |
| 658 | - | |
| 659 | -* Investigate whether there is a way to automate the memory checker | |
| 660 | - tests for Windows. | |
| 661 | - | |
| 662 | -* Provide support in QPDFWriter for writing incremental updates. | |
| 663 | - Provide support in qpdf for preserving incremental updates. The | |
| 664 | - goal should be that QDF mode should be fully functional for files | |
| 665 | - with incremental updates including fix_qdf. | |
| 666 | - | |
| 667 | - Note that there's nothing that says an indirect object in one | |
| 668 | - update can't refer to an object that doesn't appear until a later | |
| 669 | - update. This means that QPDF has to treat indirect null objects | |
| 670 | - differently from how it does now. QPDF drops indirect null objects | |
| 671 | - that appear as members of arrays or dictionaries. For arrays, it's | |
| 672 | - handled in QPDFWriter where we make indirect nulls direct. This is | |
| 673 | - in a single if block, and nothing else in the code cares about it. | |
| 674 | - We could just remove that if block and not break anything except a | |
| 675 | - few test cases that exercise the current behavior. For | |
| 676 | - dictionaries, it's more complicated. In this case, | |
| 677 | - QPDF_Dictionary::getKeys() ignores all keys with null values, and | |
| 678 | - hasKey() returns false for keys that have null values. We would | |
| 679 | - probably want to make QPDF_Dictionary able to handle the special | |
| 680 | - case of keys that are indirect nulls and basically never have it | |
| 681 | - drop any keys that are indirect objects. | |
| 682 | - | |
| 683 | - If we make a change to have qpdf preserve indirect references to | |
| 684 | - null objects, we have to note this in ChangeLog and in the release | |
| 685 | - notes since this will change output files. We did this before when | |
| 686 | - we stopped flattening scalar references, so this is probably not a | |
| 687 | - big deal. We also have to make sure that the testing for this | |
| 688 | - handles non-trivial cases of the targets of indirect nulls being | |
| 689 | - replaced by real objects in an update. I'm not sure how this plays | |
| 690 | - with linearization, if at all. For cases where incremental updates | |
| 691 | - are not being preserved as incremental updates and where the data | |
| 692 | - is being folded in (as is always the case with qpdf now), none of | |
| 693 | - this should make any difference in the actual semantics of the | |
| 694 | - files. | |
| 695 | - | |
| 696 | -* The second xref stream for linearized files has to be padded only | |
| 697 | - because we need file_size as computed in pass 1 to be accurate. If | |
| 698 | - we were not allowing writing to a pipe, we could seek back to the | |
| 699 | - beginning and fill in the value of /L in the linearization | |
| 700 | - dictionary as an optimization to alleviate the need for this | |
| 701 | - padding. Doing so would require us to pad the /L value | |
| 702 | - individually and also to save the file descriptor and determine | |
| 703 | - whether it's seekable. This is probably not worth bothering with. | |
| 704 | - | |
| 705 | -* Based on an idea suggested by user "Atom Smasher", consider | |
| 706 | - providing some mechanism to recover earlier versions of a file | |
| 707 | - embedded prior to appended sections. | |
| 708 | - | |
| 709 | -* Consider creating a sanitizer to make it easier for people to send | |
| 710 | - broken files. Now that we have json mode, this is probably no | |
| 711 | - longer worth doing. Here is the previous idea, possibly implemented | |
| 712 | - by making it possible to run the lexer (tokenizer) over a whole | |
| 713 | - file. Make it possible to replace all strings in a file lexically | |
| 714 | - even on badly broken files. Ideally this should work files that are | |
| 715 | - lacking xref, have broken links, duplicated dictionary keys, syntax | |
| 716 | - errors, etc., and ideally it should work with encrypted files if | |
| 717 | - possible. This should go through the streams and strings and | |
| 718 | - replace them with fixed or random characters, preferably, but not | |
| 719 | - necessarily, in a manner that works with fonts. One possibility | |
| 720 | - would be to detect whether a string contains characters with normal | |
| 721 | - encoding, and if so, use 0x41. If the string uses character maps, | |
| 722 | - use 0x01. The output should otherwise be unrelated to the input. | |
| 723 | - This could be built after the filtering and tokenizer rewrite and | |
| 724 | - should be done in a manner that takes advantage of the other | |
| 725 | - lexical features. This sanitizer should also clear metadata and | |
| 726 | - replace images. If I ever do this, the file from issue #494 would | |
| 727 | - be a great one to look at. | |
| 728 | - | |
| 729 | -* Here are some notes about having stream data providers modify | |
| 730 | - stream dictionaries. I had wanted to add this functionality to make | |
| 731 | - it more efficient to create stream data providers that may | |
| 732 | - dynamically decide what kind of filters to use and that may end up | |
| 733 | - modifying the dictionary conditionally depending on the original | |
| 734 | - stream data. Ultimately I decided not to implement this feature. | |
| 735 | - This paragraph describes why. | |
| 736 | - | |
| 737 | - * When writing, the way objects are placed into the queue for | |
| 738 | - writing strongly precludes creation of any new indirect objects, | |
| 739 | - or even changing which indirect objects are referenced from which | |
| 740 | - other objects, because we sometimes write as we are traversing | |
| 741 | - and enqueuing objects. For non-linearized files, there is a risk | |
| 742 | - that an indirect object that used to be referenced would no | |
| 743 | - longer be referenced, and whether it was already written to the | |
| 744 | - output file would be based on an accident of where it was | |
| 745 | - encountered when traversing the object structure. For linearized | |
| 746 | - files, the situation is considerably worse. We decide which | |
| 747 | - section of the file to write an object to based on a mapping of | |
| 748 | - which objects are used by which other objects. Changing this | |
| 749 | - mapping could cause an object to appear in the wrong section, to | |
| 750 | - be written even though it is unreferenced, or to be entirely | |
| 751 | - omitted since, during linearization, we don't enqueue new objects | |
| 752 | - as we traverse for writing. | |
| 753 | - | |
| 754 | - * There are several places in QPDFWriter that query a stream's | |
| 755 | - dictionary in order to prepare for writing or to make decisions | |
| 756 | - about certain aspects of the writing process. If the stream data | |
| 757 | - provider has the chance to modify the dictionary, every piece of | |
| 758 | - code that gets stream data would have to be aware of this. This | |
| 759 | - would potentially include end user code. For example, any code | |
| 760 | - that called getDict() on a stream before installing a stream data | |
| 761 | - provider and expected that dictionary to be valid would | |
| 762 | - potentially be broken. As implemented right now, you must perform | |
| 763 | - any modifications on the dictionary in advance and provided | |
| 764 | - /Filter and /DecodeParms at the time you installed the stream | |
| 765 | - data provider. This means that some computations would have to be | |
| 766 | - done more than once, but for linearized files, stream data | |
| 767 | - providers are already called more than once. If the work done by | |
| 768 | - a stream data provider is especially expensive, it can implement | |
| 769 | - its own cache. | |
| 770 | - | |
| 771 | - The example examples/pdf-custom-filter.cc demonstrates the use of | |
| 772 | - custom stream filters. This includes a custom pipeline, a custom | |
| 773 | - stream filter, as well as modification of a stream's dictionary to | |
| 774 | - include creation of a new stream that is referenced from | |
| 775 | - /DecodeParms. | |
| 776 | - | |
| 777 | -* Removal of raw QPDF* from the API. Discussions in #747 and #754. | |
| 778 | - This is a summary of the arguments I put forth in #754. The idea was | |
| 779 | - to make QPDF::QPDF() private and require all QPDF objects to be | |
| 780 | - shared pointers created with QPDF::create(). This would enable us to | |
| 781 | - have QPDFObjectHandle::getOwningQPDF() return a std::weak_ptr<QPDF>. | |
| 782 | - Prior to #726 (QPDFObject/QPDFValue split, released in qpdf 11.0.0), | |
| 783 | - getOwningQPDF() could return an invalid pointer if the owning QPDF | |
| 784 | - disappeared, but this is no longer the case, which removes the main | |
| 785 | - motivation. QPDF 11 added QPDF::create() anyway though. | |
| 786 | - | |
| 787 | - Removing raw QPDF* would look something like this. Note that you | |
| 788 | - can't use std::make_shared<T> unless T has a public constructor. | |
| 789 | - | |
| 790 | - QPDF_POINTER_TRANSITION = 0 -- no warnings around calling the QPDF constructor | |
| 791 | - QPDF_POINTER_TRANSITION = 1 -- calls to QPDF() are deprecated, but QPDF is still available so code can be backward compatible and use std::make_shared<QPDF> | |
| 792 | - QPDF_POINTER_TRANSITION = 2 -- the QPDF constructor is private; all calls to std::make_shared<QPDF> have to be replaced with QPDF::create | |
| 793 | - | |
| 794 | - If we were to do this, we'd have to look at each use of QPDF* in the | |
| 795 | - interface and decide whether to use a std::shared_ptr or a | |
| 796 | - std::weak_ptr. The answer would almost always be to use a | |
| 797 | - std::weak_ptr, which means we'd have to take the extra step of | |
| 798 | - calling lock(), and it means there would be lots of code changes | |
| 799 | - cause people would have to pass weak pointers instead of raw | |
| 800 | - pointers around, and those have to be constructed and locked. | |
| 801 | - Passing std::shared_ptr around leaves the possibility of creating | |
| 802 | - circular references. It seems to be too much trouble in the library | |
| 803 | - and too much toil for library users to be worth the small benefit of | |
| 804 | - not having to call resetObjGen in QPDF's destructor. | |
| 805 | - | |
| 806 | -* Fix Multiple Direct Object Parent Issue | |
| 807 | - | |
| 808 | - This idea was rejected because it would be complicated to implement | |
| 809 | - and would likely have a high performance cost to fix what is not | |
| 810 | - really that big of a problem in practice. | |
| 811 | - | |
| 812 | - It is possible for a QPDFObjectHandle for a direct object to be | |
| 813 | - contained inside of multiple QPDFObjectHandle objects or even | |
| 814 | - replicated across multiple QPDF objects. This creates a potentially | |
| 815 | - confusing and unintentional aliasing of direct objects. There are | |
| 816 | - known cases in the qpdf library where this happens including page | |
| 817 | - splitting and merging (particularly with page labels, and possibly | |
| 818 | - with other cases), and also with unsafeShallowCopy. Disallowing this | |
| 819 | - would incur a significant performance penalty and is probably not | |
| 820 | - worth doing. If we were to do it, here are some ideas. | |
| 821 | - | |
| 822 | - * Add std::weak_ptr<QPDFObject> parent to QPDFObject. When adding a | |
| 823 | - direct object to an array or dictionary, set its parent. When | |
| 824 | - removing it, clear the parent pointer. The parent pointer would | |
| 825 | - always be null for indirect objects, so the parent pointer, which | |
| 826 | - would reside in QPDFObject, would have to be managed by | |
| 827 | - QPDFObjectHandle. This is because QPDFObject can't tell the | |
| 828 | - difference between a resolved indirect object and a direct object. | |
| 829 | - | |
| 830 | - * Phase 1: When a direct object that already has a parent is added | |
| 831 | - to a dictionary or array, issue a warning. There would need to be | |
| 832 | - unsafe add methods used by unsafeShallowCopy. These would add but | |
| 833 | - not modify the parent pointer. | |
| 834 | - | |
| 835 | - * Phase 2: In the next major release, make the multiple parent case | |
| 836 | - an error. Require people to create a copy. The unsafe operations | |
| 837 | - would still have to be permitted. | |
| 838 | - | |
| 839 | - This approach would allow an object to be moved from one object to | |
| 840 | - another by removing it, which returns the now orphaned object, and | |
| 841 | - then inserting it somewhere else. It also doesn't break the pattern | |
| 842 | - of adding a direct object to something and subsequently mutating it. | |
| 843 | - It just prevents the same object from being added to more than one | |
| 844 | - thing. |
TODO.md
0 → 100644
| 1 | +Contents | |
| 2 | +======== | |
| 3 | + | |
| 4 | +- [Always](#always) | |
| 5 | +- [Next](#always) | |
| 6 | +- [Possible future JSON enhancements](#possible-future-json-enhancements) | |
| 7 | +- [QPDFJob](#qpdfjob) | |
| 8 | +- [Documentation](#documentation) | |
| 9 | +- [Document-level work](#document-level-work) | |
| 10 | +- [Text Appearance Streams](#text-appearance-streams) | |
| 11 | +- [Fuzz Errors](#fuzz-errors) | |
| 12 | +- [External Libraries](#external-libraries) | |
| 13 | +- [ABI Changes](#abi-changes) | |
| 14 | +- [C++ Version Changes](#c-version-changes) | |
| 15 | +- [Page splitting/merging](#page-splittingmerging) | |
| 16 | +- [Analytics](#analytics) | |
| 17 | +- [General](#general) | |
| 18 | + | |
| 19 | +- [HISTORICAL NOTES](#historical-notes) | |
| 20 | + | |
| 21 | +Always | |
| 22 | +====== | |
| 23 | + | |
| 24 | +* Evaluate issues tagged with `next` and `bug`. Remember to check discussions and pull requests in | |
| 25 | + addition to regular issues. | |
| 26 | +* When close to release, make sure external-libs is building and follow instructions in | |
| 27 | + ../external-libs/README | |
| 28 | + | |
| 29 | +Next | |
| 30 | +==== | |
| 31 | + | |
| 32 | +* Fix #874 -- make args in --encrypt to match the json and make positional fill in the gaps | |
| 33 | +* Maybe fix #553 -- use file times for attachments | |
| 34 | +* std::string_view transition -- work being done by m-holger | |
| 35 | +* Break ground on "Document-level work" -- TODO-pages.md lives on a separate branch. | |
| 36 | +* Standard for CLI and Job JSON support for JSON-based command-line arguments. Come up with a | |
| 37 | + standard way of supporting command-line arguments that take JSON specifications of things so that | |
| 38 | + * there is a predictable way to indicate whether an argument is a file or a JSON blob | |
| 39 | + * with QPDFJob JSON, make sure it is possible to directly include the JSON rather than having to | |
| 40 | + stringify a JSON blob | |
| 41 | + * One option might be to prepend file:// to a filename or otherwise to take a JSON blob. We could | |
| 42 | + have that as a particular type of argument that would behave properly for both job JSON and CLI. | |
| 43 | + | |
| 44 | +Possible future JSON enhancements | |
| 45 | +================================= | |
| 46 | + | |
| 47 | +* Consider not including unreferenced objects and trimming the trailer in the same way that | |
| 48 | + QPDFWriter does (except don't remove `/ID`). This means excluding the linearization dictionary and | |
| 49 | + hint stream, the encryption dictionary, all keys from trailer that are removed by | |
| 50 | + QPDFWriter::getTrimmedTrailer except `/ID`, any object streams, and the xref stream as long as all | |
| 51 | + those objects are unreferenced. (They always should be, but there could be some bizarre case of | |
| 52 | + someone creating a PDF file that has an indirect reference to one of those, in which case we need | |
| 53 | + to preserve it.) If this is done, make `--preserve-unreferenced` preserve unreference objects and | |
| 54 | + also those extra keys. Search for "linear" and "trailer" in json.rst to update the various places | |
| 55 | + in the documentation that discuss this. Also update the help for --json and | |
| 56 | + --preserve-unreferenced. | |
| 57 | + | |
| 58 | +* Add to JSON output the information available from a few additional informational options: | |
| 59 | + | |
| 60 | + * --check: add but maybe not by default? | |
| 61 | + | |
| 62 | + * --show-linearization: add but maybe not by default? Also figure out whether warnings reported | |
| 63 | + for some of the PDF specs (1.7) are qpdf problems. This may not be worth adding in the first | |
| 64 | + increment. | |
| 65 | + | |
| 66 | + * --show-xref: add | |
| 67 | + | |
| 68 | +* Consider having --check, --show-encryption, etc., just select the right keys when in json mode. I | |
| 69 | + don't think I want check on by default, so that might be different. | |
| 70 | + | |
| 71 | +* Consider having warnings be included in the json in a "warnings" key in json mode. | |
| 72 | + | |
| 73 | +QPDFJob | |
| 74 | +======= | |
| 75 | + | |
| 76 | +Here are some ideas for QPDFJob that didn't make it into 10.6. Not all of these are necessarily | |
| 77 | +good -- just things to consider. | |
| 78 | + | |
| 79 | +* How do we chain jobs? The idea would be that the input and/or output of a QPDFJob could be a QPDF | |
| 80 | + object rather than a file. For input, it's pretty easy. For output, none of the output-specific | |
| 81 | + options (encrypt, compress-streams, objects-streams, etc.) would have any affect, so we would have | |
| 82 | + to treat this like inspect for error checking. The QPDF object in the state where it's ready to be | |
| 83 | + sent off to QPDFWriter would be used as the input to the next QPDFJob. For the job json, I think | |
| 84 | + we can have the output be an identifier that can be used as the input for another QPDFJob. For a | |
| 85 | + json file, we could the top level detect if it's an array with the convention that exactly one has | |
| 86 | + an output, or we could have a subkey with other job definitions or something. Ideally, any input | |
| 87 | + (copy-attachments-from, pages, etc.) could use a QPDF object. It wouldn't surprise me if this | |
| 88 | + exposes bugs in qpdf around foreign streams as this has been a relatively fragile area before. | |
| 89 | + | |
| 90 | +Documentation | |
| 91 | +============= | |
| 92 | + | |
| 93 | +* Do a full pass through the documentation. | |
| 94 | + | |
| 95 | + * Make sure `qpdf` is consistent. Use QPDF when just referring to the package. | |
| 96 | + * Make sure markup is consistent | |
| 97 | + * Autogenerate where possible | |
| 98 | + * Consider which parts might be good candidates for moving to the wiki. | |
| 99 | + | |
| 100 | +* Commit 'Manual - enable line wrapping in table cells' from Mon Jan 17 12:22:35 2022 +0000 enables | |
| 101 | + table cell wrapping. See if this can be incorporated directly into sphinx_rtd_theme and the | |
| 102 | + workaround can be removed. | |
| 103 | + | |
| 104 | +* When possible, update the debian package to include docs again. See | |
| 105 | + https://bugs.debian.org/1004159 for details. | |
| 106 | + | |
| 107 | +Document-level work | |
| 108 | +=================== | |
| 109 | + | |
| 110 | +* Ideas here may by superseded by #593. | |
| 111 | + | |
| 112 | +* QPDFPageCopier -- object for moving pages around within files or between files and performing | |
| 113 | + various transformations. Reread/rewrite | |
| 114 | + _page-selection in the manual if needed. | |
| 115 | + | |
| 116 | + * Handle all the stuff of pages and split-pages | |
| 117 | + * Do n-up, booklet, collation | |
| 118 | + * Look through cli and see what else...flatten-*? | |
| 119 | + * See comments in QPDFPageDocumentHelper.hh for addPage -- search for "a future version". | |
| 120 | + * Make it efficient for bulk operations | |
| 121 | + * Make certain doc-level features selectable | |
| 122 | + * qpdf.cc should do all its page operations, including overlay/underlay, splitting, and merging, | |
| 123 | + using this | |
| 124 | + * There should also be example code | |
| 125 | + | |
| 126 | +* After doc-level checks are in, call --check on the output files in the "Copy Annotations" tests. | |
| 127 | + | |
| 128 | +* Document-level checks. For example, for forms, make sure all form fields point to an annotation on | |
| 129 | + exactly one page as well as that all widget annotations are associated with a form field. Hook | |
| 130 | + this into QPDFPageCopier as well as the doc helpers. Make sure it is called from --check. | |
| 131 | + | |
| 132 | +* See also issues tagged with "pages". Include closed issues. | |
| 133 | + | |
| 134 | +* Add flags to CLI to select which document-level options to preserve or not preserve. We will | |
| 135 | + probably need a pair of mutually exclusive, repeatable options with a way to specify all, none, | |
| 136 | + only {x,y}, or all but {x,y}. | |
| 137 | + | |
| 138 | +* If a page contains a reference a file attachment annotation, when that page is copied, if the file | |
| 139 | + attachment appears in the top-level EmbeddedFiles tree, that entry should be preserved in the | |
| 140 | + destination file. Otherwise, we probably will require the use of --copy-attachments-from to | |
| 141 | + preserve these. What will the strategy be for deduplicating in the automatic case? | |
| 142 | + | |
| 143 | +Text Appearance Streams | |
| 144 | +======================= | |
| 145 | + | |
| 146 | +This is a list of known issues with text appearance streams and things we might do about it. | |
| 147 | + | |
| 148 | +* For variable text, the spec says to pull any resources from /DR that are referenced in /DA but if | |
| 149 | + the resource dictionary already has that resource, just use the one that's there. The current code | |
| 150 | + looks only for /Tf and adds it if needed. We might want to instead merge /DR with resources and | |
| 151 | + then remove anything that's unreferenced. We have all the code required for that in ResourceFinder | |
| 152 | + except TfFinder also gets the font size, which ResourceFinder doesn't do. | |
| 153 | + | |
| 154 | +* There are things we are missing because we don't look at font metrics. The code from TextBuilder | |
| 155 | + (work) has almost everything in it that is required. Once we have knowledge of character widths, | |
| 156 | + we can support quadding and multiline text fields (/Ff 4096), and we can potentially squeeze text | |
| 157 | + to fit into a field. For multiline, first squeeze vertically down to the font height, then squeeze | |
| 158 | + horizontally with Tz. For single line, squeeze horizontally with Tz. If we use Tz, issue a | |
| 159 | + warning. | |
| 160 | + | |
| 161 | +* When mapping characters to widths, we will need to care about character encoding. For built-in | |
| 162 | + fonts, we can create a map from Unicode code point to width and then go from the font's encoding | |
| 163 | + to unicode to the width. See misc/character-encoding/ (not on github) | |
| 164 | + and font metric information for the 14 standard fonts in my local pdf-spec directory. | |
| 165 | + | |
| 166 | +* Once we know about character widths, we can correctly support auto-sized variable text fields | |
| 167 | + (0 Tf). If this is fixed, search for "auto-sized" in cli.rst. | |
| 168 | + | |
| 169 | +Fuzz Errors | |
| 170 | +=========== | |
| 171 | + | |
| 172 | +* https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=<N> | |
| 173 | + | |
| 174 | +* Ignoring these: | |
| 175 | + * Out of memory in dct: 35001, 32516 | |
| 176 | + | |
| 177 | +External Libraries | |
| 178 | +================== | |
| 179 | + | |
| 180 | +Current state (10.0.2): | |
| 181 | + | |
| 182 | +* qpdf/external-libs repository builds external-libs on a schedule. It detects and downloads the | |
| 183 | + latest versions of zlib, jpeg, and openssl and creates source and binary distribution zip files in | |
| 184 | + an artifact called "distribution". | |
| 185 | + | |
| 186 | +* Releases in qpdf/external-libs are made manually. They contain qpdf-external-libs-{bin,src}.zip. | |
| 187 | + | |
| 188 | +* The qpdf build finds the latest non-prerelease release and downloads the qpdf-external-libs-*.zip | |
| 189 | + files from the releases in the setup stage. | |
| 190 | + | |
| 191 | +* To upgrade to a new version of external-libs, create a new release of qpdf/external-libs (see | |
| 192 | + README-maintainer in external-libs) from the distribution artifact of the most recent successful | |
| 193 | + build after ensuring that it works. | |
| 194 | + | |
| 195 | +Desired state: | |
| 196 | + | |
| 197 | +* The qpdf/external-libs repository should create release candidates. Ideally, every scheduled run | |
| 198 | + would make its zip files available. A personal access token with actions:read scope for the | |
| 199 | + qpdf/external-libs repository is required to download the artifact from an action run, and | |
| 200 | + qpdf/qpdf's secrets.GITHUB_TOKEN doesn't have this access. We could create a service account for | |
| 201 | + this purpose. As an alternative, we could have a draft release in qpdf/external-libs that the | |
| 202 | + qpdf/external-libs build could update with each candidate. It may also be possible to solve this | |
| 203 | + by developing a simple GitHub app. | |
| 204 | + | |
| 205 | +* Scheduled runs of the qpdf build in the qpdf/qpdf repository (not a fork or pull request) could | |
| 206 | + download external-libs from the release candidate area instead of the latest stable release. | |
| 207 | + Pushes to the build branch should still use the latest release so it always matches the main | |
| 208 | + branch. | |
| 209 | + | |
| 210 | +* Periodically, we would create a release of external-libs from the release candidate zip files. | |
| 211 | + This could be done safely because we know the latest qpdf works with it. This could be done at | |
| 212 | + least before every release of qpdf, but potentially it could be done at other times, such as when | |
| 213 | + a new dependency version is available or after some period of time. | |
| 214 | + | |
| 215 | +Other notes: | |
| 216 | + | |
| 217 | +* The external-libs branch in qpdf/qpdf was never documented. We might be able to get away with | |
| 218 | + deleting it. | |
| 219 | + | |
| 220 | +* See README-maintainer in qpdf/external-libs for information on creating a release. This could be | |
| 221 | + at least partially scripted in a way that works for the qpdf/qpdf repository as well since they | |
| 222 | + are very similar. | |
| 223 | + | |
| 224 | +ABI Changes | |
| 225 | +=========== | |
| 226 | + | |
| 227 | +This is a list of changes to make next time there is an ABI change. Comments appear in the code | |
| 228 | +prefixed by "ABI". | |
| 229 | + | |
| 230 | +Always: | |
| 231 | +* Search for ABI in source and header files | |
| 232 | +* Search for "[[deprecated" to find deprecated APIs that can be removed | |
| 233 | +* Search for issues, pull requests, and discussions with the "abi" label | |
| 234 | +* Check discussion "qpdf X planning" where X is the next major version. This should be tagged `abi` | |
| 235 | + | |
| 236 | +For qpdf 12, see https://github.com/qpdf/qpdf/discussions/785 | |
| 237 | + | |
| 238 | +C++ Version Changes | |
| 239 | +=================== | |
| 240 | + | |
| 241 | +Use | |
| 242 | +``` | |
| 243 | +// C++NN: ... | |
| 244 | +``` | |
| 245 | +to mark places in the code that should be updated when we require at least that version of C++. | |
| 246 | + | |
| 247 | +Page splitting/merging | |
| 248 | +====================== | |
| 249 | + | |
| 250 | +* Update page splitting and merging to handle document-level constructs with page impact such as | |
| 251 | + interactive forms and article threading. Check keys in the document catalog for others, such as | |
| 252 | + outlines, page labels, thumbnails, and zones. For threads, Subramanyam provided a test file; see | |
| 253 | + ../misc/article-threads.pdf. Email Q-Count: 431864 from 2009-11-03. | |
| 254 | + | |
| 255 | +* bookmarks (outlines) 12.3.3 | |
| 256 | + * support bookmarks when merging | |
| 257 | + * prune bookmarks that don't point to a surviving page when merging or splitting | |
| 258 | + * make sure conflicting named destinations work possibly test by including the same file by two | |
| 259 | + paths in a merge | |
| 260 | + * see also comments in issue 343 | |
| 261 | + | |
| 262 | + Note: original implementation of bookmark preservation for split pages caused a very high | |
| 263 | + performance hit. The problem was introduced in 313ba081265f69ac9a0324f9fe87087c72918191 and | |
| 264 | + reverted in the commit that adds this paragraph. The revert includes marking a few tests cases as | |
| 265 | + $td->EXPECT_FAILURE. When properly coded, the test cases will need to be adjusted to only include | |
| 266 | + the parts of the outlines that are actually copied. The tests in question are | |
| 267 | + "split page with outlines". When implementing properly, ensure that the performance is not | |
| 268 | + adversely affected by timing split-pages on a large file with complex outlines such as the PDF | |
| 269 | + specification. | |
| 270 | + | |
| 271 | + When pruning outlines, keep all outlines in the hierarchy that are above an outline for a page we | |
| 272 | + care about. If one of the ancestor outlines points to a non-existent page, clear its dest. If an | |
| 273 | + outline does not have any children that point to pages in the document, just omit it. | |
| 274 | + | |
| 275 | + Possible strategy: | |
| 276 | + * resolve all named destinations to explicit destinations | |
| 277 | + * concatenate top-level outlines | |
| 278 | + * prune outlines whose dests don't point to a valid page | |
| 279 | + * recompute all /Count fields | |
| 280 | + | |
| 281 | + Test files | |
| 282 | + * page-labels-and-outlines.pdf: old file with both page labels and outlines. All destinations are | |
| 283 | + explicit destinations. Each page has Potato and a number. All titles are feline names. | |
| 284 | + * outlines-with-actions.pdf: mixture of explicit destinations, named destinations, goto actions | |
| 285 | + with explicit destinations, and goto actions with named destinations; uses /Dests key in names | |
| 286 | + dictionary. Each page has Salad and a number. All titles are silly words. One destination is an | |
| 287 | + indirect object. | |
| 288 | + * outlines-with-old-root-dests.pdf: like outlines-with-actions except it uses the PDF-1.1 /Dests | |
| 289 | + dictionary for named destinations, and each page has Soup and a number. Also pages are numbered | |
| 290 | + with upper-case Roman numerals starting with 0. All titles are silly words preceded by a bullet. | |
| 291 | + | |
| 292 | + If outline handling is significantly improved, see ../misc/bad-outlines/bad-outlines.pdf and | |
| 293 | + email: | |
| 294 | + https://mail.google.com/mail/u/0/#search/rfc822msgid%3A02aa01d3d013%249f766990%24de633cb0%24%40mono.hr) | |
| 295 | + | |
| 296 | +* Form fields: should be similar to outlines. | |
| 297 | + | |
| 298 | +Analytics | |
| 299 | +========= | |
| 300 | + | |
| 301 | +Consider features that make it easier to detect certain patterns in PDF files. The information below | |
| 302 | +could be computed using an external program that reads the existing json, but if it's useful enough, | |
| 303 | +we could add it directly to the json output. | |
| 304 | + | |
| 305 | +* Add to "pages" in the json: | |
| 306 | + * "inheritsresources": bool; whether there are any inherited attributes from ancestor page tree | |
| 307 | + nodes | |
| 308 | + * "sharedresources": a list of indirect objects that are | |
| 309 | + "/Resources" dictionaries or "XObject" resource dictionary subkeys of either the page itself or | |
| 310 | + of any form XObject referenced by the page. | |
| 311 | + | |
| 312 | +* Add to "objectinfo" in json: "directpagerefcount": the number of pages that directly reference | |
| 313 | + this object (i.e., you can find an indirect reference to the object in the page dictionary without | |
| 314 | + traversing over any indirect objects) | |
| 315 | + | |
| 316 | +General | |
| 317 | +======= | |
| 318 | + | |
| 319 | +NOTE: Some items in this list refer to files in my personal home directory or that are otherwise not | |
| 320 | +publicly accessible. This includes things sent to me by email that are specifically not public. Even | |
| 321 | +so, I find it useful to make reference to them in this list. | |
| 322 | + | |
| 323 | +* Consider enabling code scanning on GitHub. | |
| 324 | + | |
| 325 | +* Add an option --ignore-encryption to ignore encryption information and treat encrypted files as if | |
| 326 | + they weren't encrypted. This should make it possible to solve #598 (--show-encryption without a | |
| 327 | + password). We'll need to make sure we don't try to filter any streams in this mode. Ideally we | |
| 328 | + should be able to combine this with --json so we can look at the raw encrypted strings and streams | |
| 329 | + if we want to, though be sure to document that the resulting JSON won't be convertible back to a | |
| 330 | + valid PDF. Since providing the password may reveal additional details, --show-encryption could | |
| 331 | + potentially retry with this option if the first time doesn't work. Then, with the file open, we | |
| 332 | + can read the encryption dictionary normally. If this is done, search for "raw, encrypted" in | |
| 333 | + json.rst. | |
| 334 | + | |
| 335 | +* In libtests, separate executables that need the object library from those that strictly use public | |
| 336 | + API. Move as many of the test drivers from the qpdf directory into the latter category as long as | |
| 337 | + doing so isn't too troublesome from a coverage standpoint. | |
| 338 | + | |
| 339 | +* Consider generating a non-flat pages tree before creating output to better handle files with lots | |
| 340 | + of pages. If there are more than 256 pages, add a second layer with the second layer nodes having | |
| 341 | + no more than 256 nodes and being as evenly sizes as possible. Don't worry about the case of more | |
| 342 | + than 65,536 pages. If the top node has more than 256 children, we'll live with it. This is only | |
| 343 | + safe if all intermediate page nodes have only /Kids, /Parent, /Type, and /Count. | |
| 344 | + | |
| 345 | +* Look at https://bestpractices.coreinfrastructure.org/en | |
| 346 | + | |
| 347 | +* Consider adding fuzzer code for JSON | |
| 348 | + | |
| 349 | +* Rework tests so that nothing is written into the source directory. Ideally then the entire build | |
| 350 | + could be done with a read-only source tree. | |
| 351 | + | |
| 352 | +* Large file tests fail with linux32 before and after cmake. This was first noticed after 10.6.3. I | |
| 353 | + don't think it's worth fixing. | |
| 354 | + | |
| 355 | +* Consider updating the fuzzer with code that exercises copyAnnotations, file attachments, and name | |
| 356 | + and number trees. Check fuzzer coverage. | |
| 357 | + | |
| 358 | +* Add code for creation of a file attachment annotation. It should also be possible to create a | |
| 359 | + widget annotation and a form field. Update the pdf-attach-file.cc example with new APIs when | |
| 360 | + ready. | |
| 361 | + | |
| 362 | +* Flattening of form XObjects seems like something that would be useful in the library. We are | |
| 363 | + seeing more cases of completely valid PDF files with form XObjects that cause problems in other | |
| 364 | + software. Flattening of form XObjects could be a useful way to work around those issues or to | |
| 365 | + prepare files for additional processing, making it possible for users of the qpdf library to not | |
| 366 | + be concerned about form XObjects. This could be done recursively; i.e., we could have a method to | |
| 367 | + embed a form XObject into whatever contains it, whether that is a form XObject or a page. This | |
| 368 | + would require more significant interpretation of the content stream. We would need a test file in | |
| 369 | + which the placement of the form XObject has to be in the right place, e.g., the form XObject | |
| 370 | + partially obscures earlier code and is partially obscured by later code. Keys in the resource | |
| 371 | + dictionary may need to be changed -- create test cases with lots of duplicated/overlapping keys. | |
| 372 | + | |
| 373 | +* Part of closed_file_input_source.cc is disabled on Windows because of odd failures. It might be | |
| 374 | + worth investigating so we can fully exercise this in the test suite. That said, | |
| 375 | + ClosedFileInputSource is exercised elsewhere in qpdf's test suite, so this is not that pressing. | |
| 376 | + | |
| 377 | +* If possible, consider adding CCITT3, CCITT4, or any other easy filters. For some reference code | |
| 378 | + that we probably can't use but may be handy anyway, see | |
| 379 | + http://partners.adobe.com/public/developer/ps/sdk/index_archive.html | |
| 380 | + | |
| 381 | +* If possible, support the following types of broken files: | |
| 382 | + | |
| 383 | + - Files that have no whitespace token after "endobj" such that endobj collides with the start of | |
| 384 | + the next object | |
| 385 | + | |
| 386 | + - See ../misc/broken-files | |
| 387 | + | |
| 388 | + - See ../misc/bad-files-issue-476. This directory contains a snapshot of the google doc and linked | |
| 389 | + PDF files from issue #476. Please see the issue for details. | |
| 390 | + | |
| 391 | +* Additional form features | |
| 392 | + * set value from CLI? Specify title, and provide way to disambiguate, probably by giving objgen of | |
| 393 | + field | |
| 394 | + | |
| 395 | +* Pl_TIFFPredictor is pretty slow. | |
| 396 | + | |
| 397 | +* Support for handling file names with Unicode characters in Windows is incomplete. qpdf seems to | |
| 398 | + support them okay from a functionality standpoint, and the right thing happens if you pass in | |
| 399 | + UTF-8 encoded filenames to QPDF library routines in Windows (they are converted internally to | |
| 400 | + wchar_t*), but file names are encoded in UTF-8 on output, which doesn't produce nice error | |
| 401 | + messages or output on Windows in some cases. | |
| 402 | + | |
| 403 | +* If we ever wanted to do anything more with character encoding, see ../misc/character-encoding/, | |
| 404 | + which includes machine-readable dump of table D.2 in the ISO-32000 PDF spec. This shows the | |
| 405 | + mapping between Unicode, StandardEncoding, WinAnsiEncoding, MacRomanEncoding, and PDFDocEncoding. | |
| 406 | + | |
| 407 | +* Some test cases on bad files fail because qpdf is unable to find the root dictionary when it fails | |
| 408 | + to read the trailer. Recovery could find the root dictionary and even the info dictionary in other | |
| 409 | + ways. In particular, issue-202.pdf can be opened by evince, and there's no real reason that qpdf | |
| 410 | + couldn't be made to be able to recover that file as well. | |
| 411 | + | |
| 412 | +* Audit every place where qpdf allocates memory to see whether there are cases where malicious | |
| 413 | + inputs could cause qpdf to attempt to grab very large amounts of memory. Certainly there are cases | |
| 414 | + like this, such as if a very highly compressed, very large image stream is requested in a buffer. | |
| 415 | + Hopefully normal input to output filtering doesn't ever try to do this. QPDFWriter should be | |
| 416 | + checked carefully too. See also bugs/private/from-email-663916/ | |
| 417 | + | |
| 418 | +* Interactive form modification: | |
| 419 | + https://github.com/qpdf/qpdf/issues/213 contains a good discussion of some ideas for adding | |
| 420 | + methods to modify annotations and form fields if we want to make it easier to support | |
| 421 | + modifications to interactive forms. Some of the ideas have been implemented, and some of the | |
| 422 | + probably never will be implemented, but it's worth a read if there is an intention to work on | |
| 423 | + this. In the issue, search for "Regarding write functionality", and read that comment and the | |
| 424 | + responses to it. | |
| 425 | + | |
| 426 | +* Look at ~/Q/pdf-collection/forms-from-appian/ | |
| 427 | + | |
| 428 | +* When decrypting files with /R=6, hash_V5 is called more than once with the same inputs. Caching | |
| 429 | + the results or refactoring to reduce the number of identical calls could improve performance for | |
| 430 | + workloads that involve processing large numbers of small files. | |
| 431 | + | |
| 432 | +* Consider adding a method to balance the pages tree. It would call pushInheritedAttributesToPage, | |
| 433 | + construct a pages tree from scratch, and replace the /Pages key of the root dictionary with the | |
| 434 | + new tree. | |
| 435 | + | |
| 436 | +* Study what's required to support savable forms that can be saved by Adobe Reader. Does this | |
| 437 | + require actually signing the document with an Adobe private key? Search for "Digital signatures" | |
| 438 | + in the PDF spec, and look at ~/Q/pdf-collection/form-with-full-save.pdf, which came from Adobe's | |
| 439 | + example site. See also ../misc/digital-sign-from-trueroad/ and | |
| 440 | + ../misc/digital-signatures/digitally-signed-pdf-xfa.pdf. If digital signatures are implemented, | |
| 441 | + update the docs on crypto providers, which mention that this may happen in the future. | |
| 442 | + | |
| 443 | +* Qpdf does not honor /EFF when adding new file attachments. When it encrypts, it never generates | |
| 444 | + streams with explicit crypt filters. Prior to 10.2, there was an incorrect attempt to treat /EFF | |
| 445 | + as a default value for decrypting file attachment streams, but it is not supposed to mean that. | |
| 446 | + Instead, it is intended for conforming writers to obey this when adding new attachments. Qpdf is | |
| 447 | + not a conforming writer in that respect. | |
| 448 | + | |
| 449 | +* The whole xref handling code in the QPDF object allows the same object with more than one | |
| 450 | + generation to coexist, but a lot of logic assumes this isn't the case. Anything that creates | |
| 451 | + mappings only with the object number and not the generation is this way, including most of the | |
| 452 | + interaction between QPDFWriter and QPDF. If we wanted to allow the same object with more than one | |
| 453 | + generation to coexist, which I'm not sure is allowed, we could fix this by changing xref_table. | |
| 454 | + Alternatively, we could detect and disallow that case. In fact, it appears that Adobe reader and | |
| 455 | + other PDF viewing software silently ignores objects of this type, so this is probably not a big | |
| 456 | + deal. | |
| 457 | + | |
| 458 | +* From a suggestion in bug 3152169, consider having an option to re-encode inline images with an | |
| 459 | + ASCII encoding. | |
| 460 | + | |
| 461 | +* From github issue 2, provide more in-depth output for examining hint stream contents. Consider | |
| 462 | + adding on option to provide a human-readable dump of linearization hint tables. This should | |
| 463 | + include improving the 'overflow reading bit stream' message as reported in issue #2. There are | |
| 464 | + multiple calls to stopOnError in the linearization checking code. Ideally, these should not | |
| 465 | + terminate checking. It would require re-acquiring an understanding of all that code to make the | |
| 466 | + checks more robust. In particular, it's hard to look at the code and quickly determine what is a | |
| 467 | + true logic error and what could happen because of malformed user input. See also | |
| 468 | + ../misc/linearization-errors. | |
| 469 | + | |
| 470 | +* If I ever decide to make appearance stream-generation aware of fonts or font metrics, see email | |
| 471 | + from Tobias with Message-ID | |
| 472 | + <5C3C9C6C.8000102@thax.hardliners.org> dated 2019-01-14. | |
| 473 | + | |
| 474 | +* Look at places in the code where object traversal is being done and, where possible, try to avoid | |
| 475 | + it entirely or at least avoid ever traversing the same objects multiple times. | |
| 476 | + | |
| 477 | +---------------------------------------------------------------------- | |
| 478 | + | |
| 479 | +### HISTORICAL NOTES | |
| 480 | + | |
| 481 | +* [Performance](#performance) | |
| 482 | +* [QPDFPagesTree](#qpdfpagestree) | |
| 483 | +* [Rejected Ideas](#rejected-ideas) | |
| 484 | + | |
| 485 | +Performance | |
| 486 | +=========== | |
| 487 | + | |
| 488 | +As described in https://github.com/qpdf/qpdf/issues/401, there was great performance degradation | |
| 489 | +between qpdf 7.1.1 and 9.1.1. Doing a bisect between dac65a21fb4fa5f871e31c314280b75adde89a6c and | |
| 490 | +release-qpdf-7.1.1, I found several commits that damaged performance. I fixed some of them to | |
| 491 | +improve performance by about 70% (as measured by saying that old times were 170% of new times). The | |
| 492 | +remaining commits that broke performance either can't be correct because they would re-introduce an | |
| 493 | +old bug or aren't worth correcting because of the high value they offer relative to a relatively low | |
| 494 | +penalty. For historical reference, here are the commits. The numbers are the time in seconds on the | |
| 495 | +machine I happened to be using of splitting the first 100 pages of PDF32000_2008.pdf 20 times and | |
| 496 | +taking an average duration. | |
| 497 | + | |
| 498 | +Commits that broke performance: | |
| 499 | + | |
| 500 | +* d0e99f195a987c483bbb6c5449cf39bee34e08a1 -- object description and context: 0.39 -> 0.45 | |
| 501 | +* a01359189b32c60c2d55b039f7aefd6c3ce0ebde (minus 313ba08) -- fix dangling references: 0.55 -> 0.6 | |
| 502 | +* e5f504b6c5dc34337cc0b316b4a7b1fca7e614b1 -- sparse array: 0.6 -> 0.62 | |
| 503 | + | |
| 504 | +Other intermediate steps that were previously fixed: | |
| 505 | + | |
| 506 | +* 313ba081265f69ac9a0324f9fe87087c72918191 -- copy outlines into split: 0.55 -> 4.0 | |
| 507 | +* a01359189b32c60c2d55b039f7aefd6c3ce0ebde -- fix dangling references: | |
| 508 | + 4.0 -> 9.0 | |
| 509 | + | |
| 510 | +This commit fixed the awful problem introduced in 313ba081: | |
| 511 | + | |
| 512 | +* a5a016cdd26a8e5c99e5f019bc30d1bdf6c050a2 -- revert outline preservation: 9.0 -> 0.6 | |
| 513 | + | |
| 514 | +Note that the fix dangling references commit had a much worse impact prior to removing the outline | |
| 515 | +preservation, so I also measured its impact in isolation. | |
| 516 | + | |
| 517 | +A few important lessons (in README-maintainer) | |
| 518 | + | |
| 519 | +* Indirection through PointerHolder<Members> is expensive, and should not be used for things that | |
| 520 | + are created and destroyed frequently such as QPDFObjectHandle and QPDFObject. | |
| 521 | +* Traversal of objects is expensive and should be avoided where possible. | |
| 522 | + | |
| 523 | +Also, it turns out that PointerHolder is more performant than std::shared_ptr. (This was true at the | |
| 524 | +time but subsequent implementations of std::shared_ptr became much more efficient.) | |
| 525 | + | |
| 526 | +QPDFPagesTree | |
| 527 | +============= | |
| 528 | + | |
| 529 | +On a few occasions, I have considered implementing a QPDFPagesTree object that would allow the | |
| 530 | +document's original page tree structure to be preserved. See comments at the top QPDF_pages.cc for | |
| 531 | +why this was abandoned. | |
| 532 | + | |
| 533 | +Partial work is in refs/attic/QPDFPagesTree. QPDFPageTree is mostly implemented and mostly tested. | |
| 534 | +There are not enough cases of different kinds of operations (pclm, linearize, json, etc.) with | |
| 535 | +non-flat pages trees. Insertion is not implemented. Insertion is potentially complex because of the | |
| 536 | +issue of inherited objects. We will have to call pushInheritedAttributesToPage before adding any | |
| 537 | +pages to the pages tree. The test suite is failing on that branch. | |
| 538 | + | |
| 539 | +Some parts of page tree repair are silent (no warnings). All page tree repair should warn. The | |
| 540 | +reason is that page tree repair will change object numbers, and knowing that is important when | |
| 541 | +working with JSON output. | |
| 542 | + | |
| 543 | +If we were to do this, we would still need keep a pages cache for efficient insertion. There's no | |
| 544 | +reason we can't keep a vector of page objects up to date and just do a traversal the first time we | |
| 545 | +do getAllPages just like we do now. The difference is that we would not flatten the pages tree. It | |
| 546 | +would be useful to go through QPDF_pages and reimplement everything without calling | |
| 547 | +flattenPagesTree. Then we can remove flattenPagesTree, which is private. That said, with the | |
| 548 | +addition of creating non-flat pages trees, there is really no reason not to flatten the pages tree | |
| 549 | +for internal use. | |
| 550 | + | |
| 551 | +In its current state, QPDFPagesTree does not proactively fix /Type or correct page objects that are | |
| 552 | +used multiple times. You have to traverse the pages tree to trigger this operation. It would be nice | |
| 553 | +if we would do that somewhere but not do it more often than necessary so isPagesObject and | |
| 554 | +isPageObject are reliable and can be made more reliable. Maybe add a validate or repair function? It | |
| 555 | +should also make sure /Count and /Parent are correct. | |
| 556 | + | |
| 557 | +Rejected Ideas | |
| 558 | +============== | |
| 559 | + | |
| 560 | +* Investigate whether there is a way to automate the memory checker tests for Windows. | |
| 561 | + | |
| 562 | +* Provide support in QPDFWriter for writing incremental updates. Provide support in qpdf for | |
| 563 | + preserving incremental updates. The goal should be that QDF mode should be fully functional for | |
| 564 | + files with incremental updates including fix_qdf. | |
| 565 | + | |
| 566 | + Note that there's nothing that says an indirect object in one update can't refer to an object that | |
| 567 | + doesn't appear until a later update. This means that QPDF has to treat indirect null objects | |
| 568 | + differently from how it does now. QPDF drops indirect null objects that appear as members of | |
| 569 | + arrays or dictionaries. For arrays, it's handled in QPDFWriter where we make indirect nulls | |
| 570 | + direct. This is in a single if block, and nothing else in the code cares about it. We could just | |
| 571 | + remove that if block and not break anything except a few test cases that exercise the current | |
| 572 | + behavior. For dictionaries, it's more complicated. In this case, QPDF_Dictionary::getKeys() | |
| 573 | + ignores all keys with null values, and hasKey() returns false for keys that have null values. We | |
| 574 | + would probably want to make QPDF_Dictionary able to handle the special case of keys that are | |
| 575 | + indirect nulls and basically never have it drop any keys that are indirect objects. | |
| 576 | + | |
| 577 | + If we make a change to have qpdf preserve indirect references to null objects, we have to note | |
| 578 | + this in ChangeLog and in the release notes since this will change output files. We did this before | |
| 579 | + when we stopped flattening scalar references, so this is probably not a big deal. We also have to | |
| 580 | + make sure that the testing for this handles non-trivial cases of the targets of indirect nulls | |
| 581 | + being replaced by real objects in an update. I'm not sure how this plays with linearization, if at | |
| 582 | + all. For cases where incremental updates are not being preserved as incremental updates and where | |
| 583 | + the data is being folded in (as is always the case with qpdf now), none of this should make any | |
| 584 | + difference in the actual semantics of the files. | |
| 585 | + | |
| 586 | +* The second xref stream for linearized files has to be padded only because we need file_size as | |
| 587 | + computed in pass 1 to be accurate. If we were not allowing writing to a pipe, we could seek back | |
| 588 | + to the beginning and fill in the value of /L in the linearization dictionary as an optimization to | |
| 589 | + alleviate the need for this padding. Doing so would require us to pad the /L value individually | |
| 590 | + and also to save the file descriptor and determine whether it's seekable. This is probably not | |
| 591 | + worth bothering with. | |
| 592 | + | |
| 593 | +* Based on an idea suggested by user "Atom Smasher", consider providing some mechanism to recover | |
| 594 | + earlier versions of a file embedded prior to appended sections. | |
| 595 | + | |
| 596 | +* Consider creating a sanitizer to make it easier for people to send broken files. Now that we have | |
| 597 | + json mode, this is probably no longer worth doing. Here is the previous idea, possibly implemented | |
| 598 | + by making it possible to run the lexer (tokenizer) over a whole file. Make it possible to replace | |
| 599 | + all strings in a file lexically even on badly broken files. Ideally this should work files that | |
| 600 | + are lacking xref, have broken links, duplicated dictionary keys, syntax errors, etc., and ideally | |
| 601 | + it should work with encrypted files if possible. This should go through the streams and strings | |
| 602 | + and replace them with fixed or random characters, preferably, but not necessarily, in a manner | |
| 603 | + that works with fonts. One possibility would be to detect whether a string contains characters | |
| 604 | + with normal encoding, and if so, use 0x41. If the string uses character maps, use 0x01. The output | |
| 605 | + should otherwise be unrelated to the input. This could be built after the filtering and tokenizer | |
| 606 | + rewrite and should be done in a manner that takes advantage of the other lexical features. This | |
| 607 | + sanitizer should also clear metadata and replace images. If I ever do this, the file from issue | |
| 608 | + #494 would be a great one to look at. | |
| 609 | + | |
| 610 | +* Here are some notes about having stream data providers modify stream dictionaries. I had wanted to | |
| 611 | + add this functionality to make it more efficient to create stream data providers that may | |
| 612 | + dynamically decide what kind of filters to use and that may end up modifying the dictionary | |
| 613 | + conditionally depending on the original stream data. Ultimately I decided not to implement this | |
| 614 | + feature. This paragraph describes why. | |
| 615 | + | |
| 616 | + * When writing, the way objects are placed into the queue for writing strongly precludes creation | |
| 617 | + of any new indirect objects, or even changing which indirect objects are referenced from which | |
| 618 | + other objects, because we sometimes write as we are traversing and enqueuing objects. For | |
| 619 | + non-linearized files, there is a risk that an indirect object that used to be referenced would | |
| 620 | + no longer be referenced, and whether it was already written to the output file would be based on | |
| 621 | + an accident of where it was encountered when traversing the object structure. For linearized | |
| 622 | + files, the situation is considerably worse. We decide which section of the file to write an | |
| 623 | + object to based on a mapping of which objects are used by which other objects. Changing this | |
| 624 | + mapping could cause an object to appear in the wrong section, to be written even though it is | |
| 625 | + unreferenced, or to be entirely omitted since, during linearization, we don't enqueue new | |
| 626 | + objects as we traverse for writing. | |
| 627 | + | |
| 628 | + * There are several places in QPDFWriter that query a stream's dictionary in order to prepare for | |
| 629 | + writing or to make decisions about certain aspects of the writing process. If the stream data | |
| 630 | + provider has the chance to modify the dictionary, every piece of code that gets stream data | |
| 631 | + would have to be aware of this. This would potentially include end user code. For example, any | |
| 632 | + code that called getDict() on a stream before installing a stream data provider and expected | |
| 633 | + that dictionary to be valid would potentially be broken. As implemented right now, you must | |
| 634 | + perform any modifications on the dictionary in advance and provided /Filter and /DecodeParms at | |
| 635 | + the time you installed the stream data provider. This means that some computations would have to | |
| 636 | + be done more than once, but for linearized files, stream data providers are already called more | |
| 637 | + than once. If the work done by a stream data provider is especially expensive, it can implement | |
| 638 | + its own cache. | |
| 639 | + | |
| 640 | + The example examples/pdf-custom-filter.cc demonstrates the use of custom stream filters. This | |
| 641 | + includes a custom pipeline, a custom stream filter, as well as modification of a stream's | |
| 642 | + dictionary to include creation of a new stream that is referenced from /DecodeParms. | |
| 643 | + | |
| 644 | +* Removal of raw QPDF* from the API. Discussions in #747 and #754. This is a summary of the | |
| 645 | + arguments I put forth in #754. The idea was to make QPDF::QPDF() private and require all QPDF | |
| 646 | + objects to be shared pointers created with QPDF::create(). This would enable us to have | |
| 647 | + QPDFObjectHandle::getOwningQPDF() return a std::weak_ptr<QPDF>. Prior to #726 ( | |
| 648 | + QPDFObject/QPDFValue split, released in qpdf 11.0.0), getOwningQPDF() could return an invalid | |
| 649 | + pointer if the owning QPDF disappeared, but this is no longer the case, which removes the main | |
| 650 | + motivation. QPDF 11 added QPDF::create() anyway though. | |
| 651 | + | |
| 652 | + Removing raw QPDF* would look something like this. Note that you can't use std::make_shared<T> | |
| 653 | + unless T has a public constructor. | |
| 654 | + | |
| 655 | + QPDF_POINTER_TRANSITION = 0 -- no warnings around calling the QPDF constructor | |
| 656 | + QPDF_POINTER_TRANSITION = 1 -- calls to QPDF() are deprecated, but QPDF is still available so code | |
| 657 | + can be backward compatible and use std::make_shared<QPDF> | |
| 658 | + QPDF_POINTER_TRANSITION = 2 -- the QPDF constructor is private; all calls to | |
| 659 | + std::make_shared<QPDF> have to be replaced with QPDF::create | |
| 660 | + | |
| 661 | + If we were to do this, we'd have to look at each use of QPDF* in the interface and decide whether | |
| 662 | + to use a std::shared_ptr or a std::weak_ptr. The answer would almost always be to use a std:: | |
| 663 | + weak_ptr, which means we'd have to take the extra step of calling lock(), and it means there would | |
| 664 | + be lots of code changes cause people would have to pass weak pointers instead of raw pointers | |
| 665 | + around, and those have to be constructed and locked. Passing std::shared_ptr around leaves the | |
| 666 | + possibility of creating circular references. It seems to be too much trouble in the library and | |
| 667 | + too much toil for library users to be worth the small benefit of not having to call resetObjGen in | |
| 668 | + QPDF's destructor. | |
| 669 | + | |
| 670 | +* Fix Multiple Direct Object Parent Issue | |
| 671 | + | |
| 672 | + This idea was rejected because it would be complicated to implement and would likely have a high | |
| 673 | + performance cost to fix what is not really that big of a problem in practice. | |
| 674 | + | |
| 675 | + It is possible for a QPDFObjectHandle for a direct object to be contained inside of multiple | |
| 676 | + QPDFObjectHandle objects or even replicated across multiple QPDF objects. This creates a | |
| 677 | + potentially confusing and unintentional aliasing of direct objects. There are known cases in the | |
| 678 | + qpdf library where this happens including page splitting and merging (particularly with page | |
| 679 | + labels, and possibly with other cases), and also with unsafeShallowCopy. Disallowing this would | |
| 680 | + incur a significant performance penalty and is probably not worth doing. If we were to do it, here | |
| 681 | + are some ideas. | |
| 682 | + | |
| 683 | + * Add std::weak_ptr<QPDFObject> parent to QPDFObject. When adding a direct object to an array or | |
| 684 | + dictionary, set its parent. When removing it, clear the parent pointer. The parent pointer would | |
| 685 | + always be null for indirect objects, so the parent pointer, which would reside in QPDFObject, | |
| 686 | + would have to be managed by QPDFObjectHandle. This is because QPDFObject can't tell the | |
| 687 | + difference between a resolved indirect object and a direct object. | |
| 688 | + | |
| 689 | + * Phase 1: When a direct object that already has a parent is added to a dictionary or array, issue | |
| 690 | + a warning. There would need to be unsafe add methods used by unsafeShallowCopy. These would add | |
| 691 | + but not modify the parent pointer. | |
| 692 | + | |
| 693 | + * Phase 2: In the next major release, make the multiple parent case an error. Require people to | |
| 694 | + create a copy. The unsafe operations would still have to be permitted. | |
| 695 | + | |
| 696 | + This approach would allow an object to be moved from one object to another by removing it, which | |
| 697 | + returns the now orphaned object, and then inserting it somewhere else. It also doesn't break the | |
| 698 | + pattern of adding a direct object to something and subsequently mutating it. It just prevents the | |
| 699 | + same object from being added to more than one thing. | ... | ... |
appimage/build-appimage
| ... | ... | @@ -137,7 +137,7 @@ for i in appdir/usr/share/doc/qpdf; do |
| 137 | 137 | cp $top/LICENSE.txt $i |
| 138 | 138 | cp $top/Artistic-2.0 $i/Artistic-LICENSE.txt |
| 139 | 139 | cp $top/ChangeLog $i/README-ChangeLog |
| 140 | - cp $top/TODO $i/README-todo | |
| 140 | + cp $top/TODO.md $i/README-todo.md | |
| 141 | 141 | done |
| 142 | 142 | |
| 143 | 143 | # The following lines are experimental (for debugging; and to test | ... | ... |