Items for Kripa

OK, first off: I previously thought that my relegator code was written such that it would only work with tf2.0.  This is apparently not the case, and Kripa is able to run it on her machine with tf1.13.

Over the next two weeks, I'm going to rework the code to run with tf2.0, primarily because I would like to use eager execution.  This will make it easier to build custom loss functions that can have batch-dependent features.

For the time being, Kripa will work with tf1.13 and the old code.  I'd like her to do the following:

• run (lots of) fits with the regressor.  Fits should be run with 10k train events, noise=0.2, angle=0.9.  Signal fractions should take three values per decade (log spaced) from 0.001 up to 0.5.  (That last value need not respect log spacing.)  Run 25 fits for each value combination.
• For each fit, output important result parameters to a master results file (see the 'write_results' loop in master_moons.py).  You can make the file a pickle if you want to.
• Add to the output file arrays of the decision function values and the resulting significance values so that you can make the plots in the next bullet.
• Make plots of significance vs decision value threshold for each signal fraction.  It would be nice to do this as a "band" that includes the results of all fits for a given signal fraction.

For the time being, let's use $s / \sqrt(s+b)$ as our significance estimator/figure of merit.

RELEGATOR update

Most of the code for investigating the relegator nn classifier on the moons+mass dataset is up and running.  The plan is to check increase in analysis significance, s/sqrt(s+b), for various architecture/loss function combos and various signal fractions.

Arch/loss combos:

• regressor with binary CE loss
• regressor with binary CE + significance loss
• binary nn with categorical CE loss
• binary nn with categorical CE + significance loss
• relegator with modified categorical CE (no penalty for relegation class)
• relegator mod CCE + signif loss

It would be ideal to test each on the same generated dataset, but this might take a lot more coding.  Perhaps some way to gen the dataset and THEN pass it into master_moons.py???

For the time being, we'll take the statistical approach --> run each MANY times, histogram the increases in signif.

So, next step is to add CCE+signif loss function capabilities to regressor and nn_binary, and to make the 5th model in the list above.  This means that we'll have a total of SIX models to test.