two-class classify with uq

Resources and high-level API for a two-class classification with UQ workflow.

Implements a two-class classification workflow with a bespoke uncertainty quantification tuning method. [muyskens2021star] describes this method and its application to a star-galaxy image separation problem.

do_classify_uq() is a high-level api for executing a two-class classification workflow with the uncertainty quantification. It calls the maker APIs MuyGPyS.examples.classify.make_classifier() and MuyGPyS.examples.classify.make_multivariate_classifier() to create and train models, and performs the inference using the functions classify_two_class_uq(), make_masks(), and train_two_class_interval(). do_uq() takes the true labels of the test data and the surrgoate_prediction and masks outputs to report the statistics of the confidence intervals associated with each supplied objective function.

MuyGPyS.examples.two_class_classify_uq.classify_two_class_uq(surrogate, test_features, train_features, train_nbrs_lookup, train_labels)[source]

Simultaneously predicts the surrogate means and variances for each test item under the assumption of binary classification.

Parameters:
  • surrogate (Union[MuyGPS, MultivariateMuyGPS]) – Surrogate regressor.

  • test_features (ndarray) – Test observations of shape (test_count, feature_count).

  • train_features (ndarray) – Train observations of shape (train_count, feature_count).

  • train_nbrs_lookup (NN_Wrapper) – Trained nearest neighbor query data structure.

  • train_labels (ndarray) – One-hot encoding of class labels for all training data of shape (train_count, class_count).

Return type:

Tuple[ndarray, ndarray, Dict[str, float]]

Returns:

  • means – The surrogate predictions for each test observation of shape (test_count, 2).

  • variances – The posterior variances for each test observation of shape (test_count,)

  • timing – Timing for the subroutines of this function.

MuyGPyS.examples.two_class_classify_uq.do_classify_uq(test_features, train_features, train_labels, nn_count=30, opt_batch_count=200, uq_batch_count=500, loss_fn=<MuyGPyS.optimize.loss.LossFn object>, opt_fn=<MuyGPyS.optimize.chassis.OptimizeFn object>, uq_objectives=[<function <lambda>>, <function <lambda>>, <function <lambda>>, <function <lambda>>, <function <lambda>>], k_kwargs={}, nn_kwargs={}, opt_kwargs={}, verbose=False)[source]

Convenience function for initializing a model and performing two-class surrogate classification, while tuning uncertainty quantification.

Performs the classification workflow with uncertainty quantification tuning as described in [muyskens2021star].

Expected parameters include keyword argument dicts specifying kernel parameters and nearest neighbor parameters. See the docstrings of the appropriate functions for specifics.

Example

>>> import numpy as np
>>> from MuyGPyS.examples.regress import do_classify_uq, do_uq
>>> train_features, train_responses = make_train()  # stand-in function
>>> test_features, test_responses = make_test()  # stand-in function
>>> nn_kwargs = {"nn_method": "exact", "algorithm": "ball_tree"}
>>> k_kwargs = {
...     "kernel": RBF(
...         deformation=Isotropy(
...             metric=F2,
...             length_scale=Parameter(0.5, (0.01, 1)),
...         ),
...     )
...     "noise": HomoscedasticNoise(1e-5),
... }
>>> muygps, nbrs_lookup, surrogate_predictions = do_classify_uq(
...         test_features,
...         train_features,
...         train_responses,
...         nn_count=30,
...         batch_count=200,
...         loss_fn=cross_entropy_fn,
...         opt_fn=Bayes_optimize,
...         k_kwargs=k_kwargs,
...         nn_kwargs=nn_kwargs,
...         verbose=False,
... )
>>> accuracy, uq = do_uq(surrogate_predictions, test_responses, masks)
>>> print(f"obtained accuracy {accuracy}")
obtained accuracy: 0.973...
>>> print(f"obtained mask uq : \n{uq}")
obtained mask uq :
[[8.21000000e+02 8.53836784e-01 9.87144569e-01]
[8.59000000e+02 8.55646100e-01 9.87528717e-01]
[1.03500000e+03 8.66666667e-01 9.88845510e-01]
[1.03500000e+03 8.66666667e-01 9.88845510e-01]
[5.80000000e+01 6.72413793e-01 9.77972239e-01]]
Parameters:
  • test_features (ndarray) – A matrix of shape (test_count, feature_count) whose rows consist of observation vectors of the test data.

  • train_features (ndarray) – A matrix of shape (train_count, feature_count) whose rows consist of observation vectors of the train data.

  • train_labels (ndarray) – A matrix of shape (train_count, response_count) whose rows consist of label vectors for the training data.

  • nn_count (int) – The number of nearest neighbors to employ.

  • opt_batch_count (int) – The batch size for hyperparameter optimization.

  • uq_batch_count (int) – The batch size for uncertainty quantification calibration.

  • loss_fn (LossFn) – The loss functor to use in hyperparameter optimization. Ignored if all of the parameters specified by k_kwargs are fixed.

  • opt_fn (OptimizeFn) – The optimization functor to use in hyperparameter optimization. Ignored if all of the parameters specified by argument k_kwargs are fixed.

  • uq_objectives (Union[List[Callable], Tuple[Callable, ...]]) – list(Callable) List of objective_count`functions taking four arguments: bit masks `alpha and beta - the type 1 and type 2 error counts at each grid location, respectively - and the numbers of correctly and incorrectly classified training examples. Used to tune the scale parameter \(\sigma^2\) for setting confidence intervals. See MuyGPyS.examples.classify.example_lambdas for examples.

  • k_kwargs (Dict) – Parameters for the kernel, possibly including kernel type, deformation function, noise and scale hyperparameter specifications, and specifications for kernel hyperparameters. If all of the hyperparameters are fixed or are not given optimization bounds, no optimization will occur.

  • nn_kwargs (Dict) – Parameters for the nearest neighbors wrapper. See MuyGPyS.neighbors.NN_Wrapper for the supported methods and their parameters.

  • opt_kwargs (Dict) – Parameters for the wrapped optimizer. See the docs of the corresponding library for supported parameters.

  • verbose (bool) – If True, print summary statistics.

Return type:

Tuple[MuyGPS, NN_Wrapper, ndarray, ndarray]

Returns:

  • muygps – A (possibly trained) MuyGPs object.

  • nbrs_lookup – A data structure supporting nearest neighbor queries into train_features.

  • surrogate_predictions – A matrix of shape (test_count, response_count) whose rows indicate the surrogate predictions of the model. The predicted classes are given by the indices of the largest elements of each row.

  • masks – A matrix of shape (objective_count, test_count) whose rows consist of index masks into the training set. Each True index includes 0.0 within the associated prediction’s confidence interval.

MuyGPyS.examples.two_class_classify_uq.do_uq(surrogate_predictions, test_labels, masks)[source]

Convenience function performing uncertainty quantification given predicted labels and ground truth for a given set of confidence interval scales.

Parameters:
  • predictions – A matrix of shape (test_count, class_count) whose rows consist of the surrogate predictions.

  • test_labels (ndarray) – A matrix of shape (test_count, class_count) listing the true one-hot encodings of each test observation’s class.

  • masks (ndarray) – A matrix of shape (objective_count, test_count) whose rows consist of index masks into the training set. Each True index includes 0.0 within the associated prediction’s confidence interval.

Return type:

Tuple[float, ndarray]

Returns:

  • accuracy – The accuracy over all of the test data.

  • uq – A matrix of shape (objective_count, 3) listing the uncertainty quantification associated with each input mask (i.e. each objective function). The first column is the total number of ambiguous samples, i.e. those whose confidence interval contains the mid_value, usually 0.0. The second column is the accuracy of the ambiguous samples. The third column is the accuracy of the unambiguous samples.

MuyGPyS.examples.two_class_classify_uq.make_masks(predictions, cutoffs, variances, mid_value)[source]

Compute boolean masks over all of the test data indicating which test indices are considered ambiguous

Parameters:
  • predictions (ndarray) – A matrix of shape (test_count, class_count) whose rows consist of the surrogate predictions.

  • cutoffs (ndarray) – A vector of shape (objective_count,) indicating the confidence interval scale parameter \(\sigma^2\) that minimizes each of the considered objective function.

  • variances (ndarray) – A vector of shape (test_count, 1) indicating the diagonal posterior variance of each test item.

  • mid_value (float) – The discriminating value determining absolute uncertainty. Usually 0.0 or 0.5.

Return type:

ndarray

Returns:

A matrix of shape (objective_count, test_count) whose rows consist of index masks into the training set. Each True index includes mid_value within the associated prediction’s confidence interval.

MuyGPyS.examples.two_class_classify_uq.train_two_class_interval(surrogate, batch_indices, batch_nn_indices, train_features, train_responses, train_labels, objective_fns)[source]

For 2-class classification problems, get estimate of the confidence interval scaling parameter.

Parameters:
  • surrogate (MuyGPS) – Surrogate regressor.

  • batch_indices (ndarray) – Batch observation indices of shape (batch_count).

  • batch_nn_indices (ndarray) – Indices of the nearest neighbors of shape (batch_count, nn_count).

  • train – The full training data matrix of shape (train_count, feature_count).

  • train_responses (ndarray) – One-hot encoding of class labels for all training data of shape (train_count, class_count).

  • train_labels (ndarray) – List of class labels for all training data of shape (train_count,).

  • objective_fns (Union[List[Callable], Tuple[Callable, ...]]) – A collection of objective_count functions taking the four arguments bit masks alpha and beta - the type 1 and type 2 error counts at each grid location, respectively - and the numbers of correctly and incorrectly classified training examples. Each objective function effervesces a cutoff value to calibrate UQ for class decision-making.

Return type:

ndarray

Returns:

A vector of shape (objective_count) indicating the confidence interval scale parameter that minimizes each considered objective function.