Peter M. Groen / oletools

Checks output on all data in tests/test-data dir

Samples were created by me using Office 2010 on a Windows 7 machine,
password is "encrypted", contents is trivial (no links, macros, ...)

Encryption should be the CryptoAPI RC4 Encryption (see [MS-OFFCRYPTO])

Sofar, most tools (like msodde) do not complain if file is encrypted, but
cannot inspect it either. That gives the user a false sense of security
which is dangerous. Raise error to make the situation clear. But only
raise when iterating records (stream names are still reliable and useful.

Sometimes there just is no filename

Error was probably not noticeable since None was returned which usually
also evaluates to False

Shame on me, I did not realize when creating ooxml and record_base that
oleid already does some of the work of basic file classification.

Added comments that this should one day be remedied.

Right now, olevba[3] does not complain at all but cannot do its job.
That could give the user a false sense of security.

Return new return code so condition can easily be tested

This should avoid repeated opening and parsing of the same file if oleid
is used by other tools

- add docstrings
- break lines at 80
- remove irregular whitespace
- correct bare except
- replace extra self.encrypted and self.macros
  with helper function get_indicator
- correct a few variable names and one function name
- make absolute import for scripts little less hacky

Fix false-positive with detect_vba_macros

Unified logging with json option

Common has been added by other commits, name is just as good as "utils"

The log_helper was refactored to fallback to the root logger when
logging is enabled. Now, instead of each logger having its own
level and handler, we reuse the level of the root logger by
setting the level of the loggers to logging.NOTSET. We also keep
the default propagation behavior so instead of having a custom
handler for each logger, they will keep the NullHandler and when
logging they will use the handler of the root logger. This also
allowed us to set the JSON formatter only for the root logger,
since it's the only one actually logging things.

The base class for loggers was also removed because the way it
was used it would replace the base class for every logger, which
would affect modules importing oletools files. We now use an
adapter that is only valid for loggers created by the log_helper.

The hook for exceptions now wraps any hook previosly set instead
of replacing it altogether and this is only done when calling
modules as script.

Finally, the log_at_current_level() function was replaced by a
print_str() function defined in the adapter that will print
instead of logging when not using JSON so we can keep the old
behavior.

We only use it to initialize a field, but since it depends
on an external module, it's better to avoid use it and
prevent things from breaking if that changes.

Oletools always use the NullHandler and a high log level,
so there's no need for an extra function.

This class was only necessary for Python 2.6, but as the support
for that version was dropped it's not needed anymore.

Unittests worked on Travis and from IDEs like PyCharm, but from
command line I had to change log helper test root dir to one level up
and unset PYTHONPATH

The OutputCapture was a slightly hacky way to get output from scripts without
using subprocess. However, keeping it working started requiring reloads since
module's logs are global variables which were re-initialized which caused
trouble). That required reload of all submodules, so more hacks are needed
to keep the initial hack "alive". Therefore I am glad it is not needed any more
and can be removed.

ooxml was changed to use our custom log helper
so that it can also output JSON messages whenever
a caller that uses the -j flag has JSON output
set.

Some msodde tests call the main() function from the module
which tries to enable logging, causing lots of problems
when running multiple tests due to the singleton dependency
of the logging modules.

Since what they want to test is whether msodde is finding
links, we can use the process_file() function directly
instead and check its output.

msodde was changed to use the log helper, which adds
better support for JSON output. Since we also changed
ooxml, now we have full JSON output for any level
(including debug) and for ooxml messages.

Adds a log helper module that can be used by all modules and
handles logging in general, avoiding code duplication. It also
handles JSON output formatting by logging all messages in JSON
format, thus making it easy for modules to support the `-j` flag.

The module works by overwriting the default logging class to
extend its behavior (adding a function to always log, useful for
banners and other stuff). The JSON format uses a special formatter
that will output JSON from a string or a dictionary.

…suspicious (issue #358)