provide monitoring of parallel and distributed jobs

[mmckerns]

provide monitoring of parallel and distributed jobs (branched from ticket #94; see the notes there)

[mmckerns]

repeated from ticket #94:

added solver.id and monitoring of solver.id in r296

need to modify "log_reader" to use solver.id

[mmckerns]

use python's "logging" module?

[mmckerns]

also look at pyre's "journal"

[mmckerns]

NOTES:

Logging requirements from mystic's monitors:

1) Capture current parameter vector "x", and cost "F(x)" after each optimizer
iteration, and also after each function evaluation.  Default should be to write
to a monitor object.  Alternately log to stdout, to a file, a database, or serialized
and sent across a socket.

2) Loggers should be configurable. They should read all input, but allow
application of a filter, such as writing every 10th tuple (x,F(x)) to the log.
Each tuple (x, F(x)) may be composed of one to N points. Hence, x may
be a 1-D or 2-D array, while F(x) may be a float or a 1-D array of floats.

3) Loggers may also be required to capture gradient " x' ", Hessian " x'' ",
or Hessian product "Hp".  Gradient and Hessian product are one dimension
larger than "x", while Hessian is two dimensions larger than "x".

4) Each logger will monitor from one to M solvers, where solvers may
be running sequentially or in parallel.

5) Logs should be dynamically accessible (i.e during a calculation).

6) Logs should be both human readable and easily parsed by a log reader
(for plotting, or future calculations).


================================================
'Desirable' feature summary for python's "logging" module:

Simple example of logger using a 'handler' and a 'formatter':
 - http://docs.python.org/library/logging.html#simple-examples

Handlers attach to a logger:
 >>> my_logger = logger.getLogger('MyLogger')
 >>> handler = logging.StreamHandler()
 >>> my_logger.addHandler(handler)

There are ~14 handlers, some of which send instances across the network:
 - http://docs.python.org/library/logging.html#useful-handlers
 - http://docs.python.org/library/logging.html#handler-objects

Can toggle on/off the handling of exceptions raised during logging:
 - http://docs.python.org/library/logging.html#exceptions-raised-during-logging

Example of logging across the network with a SocketHandler:
 - http://docs.python.org/library/logging.html#sending-and-receiving-logging-events-across-a-network

Default messages are sent as strings, however passing arbitrary objects are allowed.
For example, a pickled object is the default for the SocketHandler.

Queries to a logging instance to return which channels are active:
 >>> logger.isEnabledFor(logging.DEBUG)

Logger and Handler include "addFilter" and "removeFilter" methods to provide
more 'advanced' configuration (beyond 'is channel.severity >= message.severity?).
Severity of Loggers and Handlers are both independently configurable:
 - http://docs.python.org/library/logging.html#logging.Logger.addFilter
 - http://docs.python.org/library/logging.html#filter-objects

Additional logging configuration available with "disable":
 - http://docs.python.org/library/logging.html#logging.disable

Loggers are built in a hierarchical structure, based on the logger's name.  While a
good idea, the implementation may cause some configuration issues:
 - http://docs.python.org/library/logging.html#logging.getLogger
 - http://docs.python.org/library/logging.html#loggers

Can toggle on/off message passing up the Logger hierarchy:
 - http://docs.python.org/library/logging.html#logging.Logger.propagate