Relegation classifier thoughts

Thinking about investigating a NN classifier for separating signal from a pernicious background.  Typical approach is to train classifier, and then use decision cut value that give the best analysis power, often interpreted from ROC AUC.  I suspect, though, that the NN could train differently if we "build in" that we want to optimize analysis power.

So, my idea is to investigate a binary classification problem with a NN that predicts probabilities for THREE classes: signal, background, and RELEGATION.  The idea is that events which are too difficult to correctly characterize will be placed in the relegation class.  The penalty for doing so is that the loss function will contain a term(s) that wants to keep S/sqrt(S+B) as high as possible.

For multiclass classification, we would use the categorical cross-entropy loss function:

$$\[\mathcal{L} = \displaystyle\sum_{c=1}^{M} y_{o,c} \ln (p_{o,c})\]$$

We will add to this a term:

A potential problem is that the total S and B can only be accurately calculated once per epoch.  BUT the tuning of the network depends on the change in S and B (derivative for backprop.).  These derivatives can be calculated on a per-event basis.