Commit fc30e75122f6bef20ac8d676ac7cb8822feecf3e

Authored by Jay Berkenbilt
Committed by GitHub
2 parents ddc86451 09365372

Merge pull request #987 from m-holger/maintainer

Rename README-maintainer to README-maintainer.md and add table of content
README-maintainer renamed to README-maintainer.md
1   -ROUTINE DEVELOPMENT
  1 +# Maintainer Notes
  2 +
  3 +## Contents
  4 +
  5 +* [ROUTINE DEVELOPMENT](#routine-development)
  6 +* [VERSIONS](#versions)
  7 +* [CHECKING DOCS ON readthedocs](#checking-docs-on-readthedocs)
  8 +* [GOOGLE OSS-FUZZ](#google-oss-fuzz)
  9 +* [CODING RULES](#coding-rules)
  10 +* [HOW TO ADD A COMMAND-LINE ARGUMENT](#how-to-add-a-command-line-argument)
  11 +* [RELEASE PREPARATION](#release-preparation)
  12 +* [CREATING A RELEASE](#creating-a-release)
  13 +* [RUNNING pikepdf's TEST SUITE](#running-pikepdfs-test-suite)
  14 +* [OTHER NOTES](#other-notes)
  15 +* [DEPRECATION](#deprecation)
  16 +* [LOCAL WINDOWS TESTING PROCEDURE](#local-windows-testing-procedure)
  17 +* [DOCS ON readthedocs.org](#docs-on-readthedocsorg)
  18 +* [CMAKE notes](#cmake-notes)
  19 +* [ABI checks](#abi-checks)
  20 +* [CODE FORMATTING](#code-formatting)
  21 +
  22 +
  23 +## ROUTINE DEVELOPMENT
2 24  
3 25 **Remember to check pull requests as well as issues in github.**
4 26  
5 27 Default:
6 28  
  29 +```
7 30 cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=1 \
8 31 -DMAINTAINER_MODE=1 -DBUILD_STATIC_LIBS=0 \
9 32 -DCMAKE_BUILD_TYPE=RelWithDebInfo ..
  33 +```
10 34  
11 35 Debugging:
12 36  
  37 +```
13 38 cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=1 \
14 39 -DMAINTAINER_MODE=1 -DBUILD_SHARED_LIBS=0 \
15 40 -DCMAKE_BUILD_TYPE=Debug ..
  41 +```
16 42  
17 43 Profiling:
18 44  
  45 +```
19 46 CFLAGS=-pg LDFLAGS=-pg \
20 47 cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=1 \
21 48 -DMAINTAINER_MODE=1 -DBUILD_SHARED_LIBS=0 \
22 49 -DCMAKE_BUILD_TYPE=Debug ..
  50 +```
23 51  
24 52 Then run `gprof gmon.out`. Note that gmon.out is not cumulative.
25 53  
26 54 Memory checks:
27 55  
  56 +```
28 57 CFLAGS="-fsanitize=address -fsanitize=undefined" \
29 58 CXXFLAGS="-fsanitize=address -fsanitize=undefined" \
30 59 LDFLAGS="-fsanitize=address -fsanitize=undefined" \
... ... @@ -32,21 +61,23 @@ CFLAGS="-fsanitize=address -fsanitize=undefined" \
32 61 cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=1 \
33 62 -DMAINTAINER_MODE=1 -DBUILD_SHARED_LIBS=0 \
34 63 -DCMAKE_BUILD_TYPE=Debug ..
  64 +```
35 65  
36 66 Windows:
37 67  
  68 +```
38 69 ../cmake-win {mingw|msvc} maint
  70 +```
39 71  
40 72 See ./build-scripts for other ways to run the build for different
41 73 configurations.
42 74  
43   -
44   -VERSIONS
  75 +## VERSIONS
45 76  
46 77 * The version number on the main branch is whatever the version would
47 78 be if the top of the branch were released. If the most recent
48 79 release is version a.b.c, then
49   -
  80 +
50 81 * If there are any ABI-breaking changes since the last release,
51 82 main's version is a+1.0.0
52 83 * Else if there is any new public API, main's version is a.b+1.0
... ... @@ -65,15 +96,13 @@ VERSIONS
65 96 there or the changes can be merged back, depending on the amount of
66 97 drift.
67 98  
68   -
69   -CHECKING DOCS ON readthedocs
  99 +## CHECKING DOCS ON readthedocs
70 100  
71 101 To check docs on readthedocs.io without running all of CI, push to the
72 102 doc-check branch. Then visit https://qpdf.readthedocs.io/en/doc-check/
73 103 Building docs from pull requests is also enabled.
74 104  
75   -
76   -GOOGLE OSS-FUZZ
  105 +## GOOGLE OSS-FUZZ
77 106  
78 107 * See ../misc/fuzz (not in repo) for unfixed, downloaded fuzz test cases
79 108  
... ... @@ -87,26 +116,28 @@ GOOGLE OSS-FUZZ
87 116  
88 117 * To test locally, see https://github.com/google/oss-fuzz/tree/master/docs/,
89 118 especially new_project_guide.md. Summary:
90   -
  119 +
91 120 Clone the oss-fuzz project. From the root directory of the repository:
92   -
  121 +
  122 + ```
93 123 python3 infra/helper.py build_image --pull qpdf
94 124 python3 infra/helper.py build_fuzzers [ --sanitizer memory|undefined|address ] qpdf [path-to-qpdf-source]
95 125 python3 infra/helper.py check_build qpdf
96 126 python3 infra/helper.py build_fuzzers --sanitizer coverage qpdf
97 127 python3 infra/helper.py coverage qpdf
98   -
  128 + ```
  129 +
99 130 To reproduce a test case, build with the correct sanitizer, then run
100   -
  131 +
101 132 python3 infra/helper.py reproduce qpdf <specific-fuzzer> testcase
102   -
  133 +
103 134 where fuzzer is the fuzzer used in the crash.
104   -
  135 +
105 136 The fuzzer is in build/out/qpdf. It can be run with a directory as
106 137 an argument to run against files in a directory. You can use
107   -
  138 +
108 139 qpdf_fuzzer -merge=1 cur new >& /dev/null&
109   -
  140 +
110 141 to add any files from new into cur if they increase coverage. You
111 142 need to do this with the coverage build (the one with
112 143 --sanitizer coverage)
... ... @@ -120,28 +151,27 @@ GOOGLE OSS-FUZZ
120 151 * Latest corpus:
121 152 gs://qpdf-backup.clusterfuzz-external.appspot.com/corpus/libFuzzer/qpdf_fuzzer/latest.zip
122 153  
  154 +## CODING RULES
123 155  
124   -CODING RULES
125   -
126   -* Code is formatted with clang-format >= 15. See .clang-format and the
  156 +* Code is formatted with clang-format-16. See .clang-format and the
127 157 "Code Formatting" section in manual/contributing.rst for details.
128 158 See also "CODE FORMATTING" below.
129 159  
130 160 * Use std::to_string instead of QUtil::int_to_string et al
131 161  
132 162 * Use of assert:
133   -
  163 +
134 164 * Test code: #include <qpdf/assert_test.h> first.
135 165 * Debug code: #include <qpdf/assert_debug.h> first and use
136 166 qpdf_assert_debug instead of assert.
137   -
  167 +
138 168 These rules are enforced by the check-assert test. This practices
139 169 serves to
140   -
  170 +
141 171 * remind us that assert in release code disappears and so should only
142 172 be used for debugging; when doing so use a Debug build
143 173 configuration
144   -
  174 +
145 175 * protect us from using assert in test code without explicitly
146 176 removing the NDEBUG definition, since that would cause the assert
147 177 not to actually be testing anything in non-Debug build
... ... @@ -190,7 +220,7 @@ CODING RULES
190 220 dynamic_cast with those, but doing it anyway may help with some
191 221 strange cases for mingw or with some code generators that may
192 222 systematically do this for other reasons.
193   -
  223 +
194 224 IMPORTANT NOTE ABOUT QPDF_DLL_CLASS: On mingw, the vtable for a
195 225 class with some virtual methods and no pure virtual methods seems
196 226 often (always?) not to be generated if the destructor is inline or
... ... @@ -199,9 +229,11 @@ CODING RULES
199 229 methods, you must declare the destructor in the header without
200 230 `= default` and provide a non-inline implementation in the source
201 231 file. Add this comment to the implementation:
202   -
  232 +
  233 + ```cpp
203 234 // Must be explicit and not inline -- see QPDF_DLL_CLASS in
204 235 // README-maintainer
  236 + ```
205 237  
206 238 * Put private member variables in std::shared_ptr<Members> for all
207 239 public classes. Remember to use QPDF_DLL on ~Members(). Exception:
... ... @@ -220,8 +252,7 @@ CODING RULES
220 252 * Avoid attaching too much metadata to objects and object handles
221 253 since those have to get copied around a lot.
222 254  
223   -
224   -HOW TO ADD A COMMAND-LINE ARGUMENT
  255 +## HOW TO ADD A COMMAND-LINE ARGUMENT
225 256  
226 257 Quick reminder:
227 258  
... ... @@ -278,18 +309,17 @@ When done, the following should happen:
278 309 * The job JSON file should have a new key in the schema corresponding
279 310 to the new option
280 311  
281   -
282   -RELEASE PREPARATION
  312 +## RELEASE PREPARATION
283 313  
284 314 * Each year, update copyright notices. This will find all relevant
285 315 places (assuming current copyright is from last year):
286   -
  316 +
287 317 git --no-pager grep -i -n -P "copyright.*$(expr $(date +%Y) - 1).*berkenbilt"
288   -
  318 +
289 319 Also update the copyright in these places:
290 320 * debian package -- search for copyright.*berkenbilt in debian/copyright
291 321 * qtest-driver, TestDriver.pm in qtest source
292   -
  322 +
293 323 Copyright last updated: 2023.
294 324  
295 325 * Take a look at "External Libraries" in TODO to see if we need to
... ... @@ -311,24 +341,24 @@ RELEASE PREPARATION
311 341 * Check work `qpdf` project for private issues
312 342  
313 343 * Make sure the code is formatted.
314   -
  344 +
315 345 ./format-code
316 346  
317 347 * Run a spelling checker over the source code to catch errors in
318 348 variable names, strings, and comments.
319   -
  349 +
320 350 ./spell-check
321   -
  351 +
322 352 This uses cspell. Install with `npm install -g cspell`. The output
323 353 of cspell is suitable for use with `M-x grep` in emacs. Add
324 354 exceptions to cSpell.json.
325 355  
326 356 * If needed, run large file and image comparison tests by setting
327 357 these environment variables:
328   -
  358 +
329 359 QPDF_LARGE_FILE_TEST_PATH=/full/path
330 360 QPDF_TEST_COMPARE_IMAGES=1
331   -
  361 +
332 362 For Windows, use a Windows style path, not an MSYS path for large files.
333 363  
334 364 * If any interfaces were added or changed, check C API to see whether
... ... @@ -340,12 +370,12 @@ RELEASE PREPARATION
340 370  
341 371 * Double check versions and shared library details. They should
342 372 already be up to date in the code.
343   -
  373 +
344 374 * Make sure version numbers are consistent in the following locations:
345 375 * CMakeLists.txt
346 376 * include/qpdf/DLL.h
347 377 * manual/conf.py
348   -
  378 +
349 379 `make_dist` verifies this consistency, and CI fails if they are
350 380 inconsistent.
351 381  
... ... @@ -362,12 +392,12 @@ RELEASE PREPARATION
362 392 testing, do performance testing.
363 393  
364 394 * Test for performance and binary compatibility:
365   -
  395 +
366 396 ./abi-perf-test v<old> @
367   -
  397 +
368 398 Prefix with SKIP_PERF=1 to skip performance test.
369 399 Prefix with SKIP_TESTS=1 to skip test suite run.
370   -
  400 +
371 401 See "ABI checks" for details about the process.
372 402 End state:
373 403 * /tmp/check-abi/perf contains the performance comparison
... ... @@ -389,8 +419,7 @@ env PKG_CONFIG_PATH=/tmp/inst/lib/pkgconfig \
389 419 CMAKE_PREFIX_PATH=/tmp/inst \
390 420 ./pkg-test/run-all
391 421  
392   -
393   -CREATING A RELEASE
  422 +## CREATING A RELEASE
394 423  
395 424 * Push to main. This will create an artifact called distribution
396 425 which will contain all the distribution files. Download these,
... ... @@ -442,7 +471,9 @@ git push qpdf @:stable
442 471 * Create a github release after pushing the tag. `gcurl` is an alias
443 472 that includes the auth token.
444 473  
  474 +```
445 475 # Create release
  476 +
446 477 GITHUB_TOKEN=$(qdata-show cred github-token)
447 478 function gcurl() { curl -H "Authorization: token $GITHUB_TOKEN" ${1+"$@"}; }
448 479  
... ... @@ -458,6 +489,7 @@ for i in *; do
458 489 mime=$(file -b --mime-type $i)
459 490 gcurl -H "Content-Type: $mime" --data-binary @$i "$upload_url?name=$i"
460 491 done
  492 +```
461 493  
462 494 If needed, go onto github and make any manual updates such as
463 495 indicating a pre-release, adding release notes, etc.
... ... @@ -470,8 +502,10 @@ This is qpdf version x.y.z. (Brief description)
470 502 For a full list of changes from previous releases, please see the [release notes](https://qpdf.readthedocs.io/en/stable/release-notes.html). See also [README-what-to-download](./README-what-to-download.md) for details about the available source and binary distributions.
471 503 ```
472 504  
  505 +```
473 506 # Publish release
474 507 gcurl -XPOST $url -d'{"draft": false}'
  508 +```
475 509  
476 510 * Upload files to sourceforge.
477 511  
... ... @@ -486,14 +520,14 @@ rsync -vrlcO ./ jay_berkenbilt,qpdf@frs.sourceforge.net:/home/frs/project/q/qp/q
486 520  
487 521 * Email the qpdf-announce list.
488 522  
489   -
490   -RUNNING pikepdf's TEST SUITE
  523 +## RUNNING pikepdf's TEST SUITE
491 524  
492 525 We run pikepdf's test suite from CI. These instructions show how to do
493 526 it manually.
494 527  
495 528 Do this in a separate shell.
496 529  
  530 +```
497 531 cd ...qpdf-source-tree...
498 532 export QPDF_SOURCE_TREE=$PWD
499 533 export QPDF_BUILD_LIBDIR=$QPDF_SOURCE_TREE/build/libqpdf
... ... @@ -510,10 +544,12 @@ python3 -m pip install &#39;.[test]&#39;
510 544 rehash
511 545 python3 -m pip install .
512 546 pytest -n auto
  547 +```
513 548  
514 549 If there are failures, use git bisect to figure out where the failure
515 550 was introduced. For example, set up a work area like this:
516 551  
  552 +```
517 553 rm -rf /tmp/z
518 554 mkdir /tmp/z
519 555 cd /tmp/z
... ... @@ -541,12 +577,12 @@ python3 -m pip install .
541 577 pytest -n auto
542 578 EOF
543 579 chmod +x /tmp/check
  580 +```
544 581  
545 582 Then in /tmp/z/qpdf, run git bisect. Use /tmp/check at each stage to
546 583 test whether it's a good or bad commit.
547 584  
548   -
549   -OTHER NOTES
  585 +## OTHER NOTES
550 586  
551 587 For local iteration on the AppImage generation, it works to just
552 588 ./build-scripts/build-appimage and get the resulting AppImage from the
... ... @@ -558,9 +594,11 @@ Use -ti -e RUN_SHELL=1 to run a shell instead of the build script.
558 594  
559 595 To iterate on the scripts directly in the source tree, you can run
560 596  
  597 +```
561 598 docker build -t qpdfbuild appimage
562 599 docker run --privileged --rm -ti -e SKIP_TESTS=1 -e RUN_SHELL=1 \
563 600 -v $PWD/..:/tmp/build ${1+"$@"} qpdfbuild
  601 +```
564 602  
565 603 This will put you at a shell prompt inside the container with your
566 604 current directory set to the top of the source tree and your uid equal
... ... @@ -569,12 +607,13 @@ to the owner of the parent directory source tree.
569 607 Note: this will leave some extra files (like .bash_history) in the
570 608 parent directory of the source tree. You will want to clean those up.
571 609  
572   -DEPRECATION
  610 +## DEPRECATION
573 611  
574 612 This is a reminder of how to use and test deprecation.
575 613  
576 614 To temporarily disable deprecation warnings for testing:
577 615  
  616 +```cpp
578 617 #ifdef _MSC_VER
579 618 # pragma warning(disable : 4996)
580 619 #endif
... ... @@ -586,13 +625,15 @@ To temporarily disable deprecation warnings for testing:
586 625 #if (defined(__GNUC__) || defined(__clang__))
587 626 # pragma GCC diagnostic pop
588 627 #endif
  628 +```
589 629  
590 630 To declare something as deprecated:
591 631  
  632 +```cpp
592 633 [[deprecated("explanation")]]
  634 +```
593 635  
594   -
595   -LOCAL WINDOWS TESTING PROCEDURE
  636 +## LOCAL WINDOWS TESTING PROCEDURE
596 637  
597 638 This is what I do for routine testing on Windows.
598 639  
... ... @@ -612,8 +653,7 @@ This is what I do for routine testing on Windows.
612 653  
613 654 * Test with mingw: `ctest --verbose -C RelWithDebInfo`
614 655  
615   -
616   -DOCS ON readthedocs.org
  656 +## DOCS ON readthedocs.org
617 657  
618 658 * Registered for an account at readthedocs.org with my github account
619 659 * Project page: https://readthedocs.org/projects/qpdf/
... ... @@ -640,8 +680,7 @@ following branching strategy to support docs:
640 680 The release process includes updating the approach branches and
641 681 activating versions.
642 682  
643   -
644   -CMAKE notes
  683 +## CMAKE notes
645 684  
646 685 To verify the various cmake options and their interactions, several
647 686 manual tests were done:
... ... @@ -662,8 +701,7 @@ We are using RelWithDebInfo for mingw and other non-Windows builds but
662 701 Release for MSVC. There are linker warnings if MSVC is built with
663 702 RelWithDebInfo when using external-libs.
664 703  
665   -
666   -ABI checks
  704 +## ABI checks
667 705  
668 706 Until the conversion of the build to cmake, we relied on running the
669 707 test suite with old executables and a new library. When QPDFJob was
... ... @@ -698,8 +736,7 @@ steps. See comments in check_abi for additional notes. Running
698 736 "check_abi check-sizes" is run by ctest on Linux when CHECK_SIZES is
699 737 on.
700 738  
701   -
702   -CODE FORMATTING
  739 +## CODE FORMATTING
703 740  
704 741 * Emacs doesn't indent breaking strings concatenated with + over
705 742 lines but clang-format does. It's clearer with clang-format. To
... ... @@ -707,18 +744,22 @@ CODE FORMATTING
707 744 that builds the concatenated string.
708 745  
709 746 * With
710   -
  747 +
  748 + ```cpp
711 749 long_function(long_function(
712 750 args)
713 751  
  752 + ```
  753 +
714 754 clang-format anchors relative to the first function, and emacs
715 755 anchors relative to the second function. Use
716   -
  756 +
  757 + ```cpp
717 758 long_function(
718 759 // line-break
719 760 long_function(
720 761 args)
721   -
  762 + ```
722 763 to resolve.
723 764  
724 765 In the revision control history, there is a commit around April 3,
... ...