Collection of methods for testing and calibrating recall for SAM Models. More...

Collaboration diagram for SAM Tests:

Namespaces
	SAM.SAM_Core.svi_ratio

Functions
def	SAM.SAM_Core.SAMTesting.deep_getsizeof (o, ids)
	Method to calculate the size of an object `o` in bytes. More...

def	SAM.SAM_Core.SAMTesting.calibrateModelRecall (thisModel)
	Logic to initialise calibration of model recall in order to recognize known from unknown instances. More...

def	SAM.SAM_Core.SAMTesting.calibrateSingleModelRecall (thisModel)
	Perform calibration for single model implementations. More...

def	SAM.SAM_Core.SAMTesting.calibrateMultipleModelRecall (thisModel)
	Perform calibration for multiple model implementations. More...

def	SAM.SAM_Core.SAMTesting.formatDataFunc (Ydata)
	Utility function to format data for testing. More...

def	SAM.SAM_Core.SAMTesting.singleRecall (thisModel, testInstance, verbose, visualiseInfo=None, optimise=100)
	Method that performs classification for single model implementations. More...

def	SAM.SAM_Core.SAMTesting.multipleRecall_noCalib (thisModel, testInstance, verbose, visualiseInfo=None, optimise=True)
	Method that performs classification for uncalibrated multiple model implementations. More...

def	SAM.SAM_Core.SAMTesting.multipleRecall (thisModel, testInstance, verbose, visualiseInfo=None, optimise=100)
	Method that performs classification for calibrated multiple model implementations. More...

def	SAM.SAM_Core.SAMTesting.wait_watching_stdout (ar, dt=1, truncate=1000)
	Monitoring function that logs to stdout the logging output of multiple threads. More...

def	SAM.SAM_Core.SAMTesting.testSegment (thisModel, Ysample, verbose, visualiseInfo=None, optimise=100)
	Utility function to test a sample. More...

def	SAM.SAM_Core.SAMTesting.segmentTesting (thisModel, Ysample, Lnum, verbose, label, serialMode=False, optimise=100, calibrate=False)
	Method to test multiple samples at a time. More...

def	SAM.SAM_Core.SAMTesting.testSegments (thisModel, Ysample, Lnum, verbose, label, serialMode=False)
	Function to test segments and return a confusion matrix. More...

def	SAM.SAM_Core.SAMTesting.calculateVarianceThreshold (segIntersections, mk, muk, vk, vuk)
	Method to decide on the approach to be used for setting variance thresholds and method of thresholding. More...

def	SAM.SAM_Core.SAMTesting.calculateData (textLabels, confMatrix, numItems=None)
	Calculate the normalised confusion matrix. More...

def	SAM.SAM_Core.SAMTesting.combineClassifications (thisModel, labels, likelihoods)
	Combine multiple classifications into a single classification. More...

Detailed Description

Collection of methods for testing and calibrating recall for SAM Models.

Function Documentation

◆ calculateData()

def SAM.SAM_Core.SAMTesting.calculateData	(	textLabels,
		confMatrix,
		numItems = `None`
	)

Calculate the normalised confusion matrix.

Parameters

textLabels	List of classifications.
confMatrix	Confusion matrix to normalise.
numItems	Total number of items tested.

Returns: Normalised confusion matrix and overall percentage correct.

Definition at line 843 of file SAMTesting.py.

◆ calculateVarianceThreshold()

def SAM.SAM_Core.SAMTesting.calculateVarianceThreshold	(	segIntersections,
		mk,
		muk,
		vk,
		vuk
	)

Method to decide on the approach to be used for setting variance thresholds and method of thresholding.

Parameters

segIntersections	Number of gaussian intersections.
mk	Means of known.
muk	Means of unknown.
vk	Variances of known.
vuk	Variances of unknown.

Returns: List of threshold variances and method of thresholding.

Definition at line 791 of file SAMTesting.py.

◆ calibrateModelRecall()

def SAM.SAM_Core.SAMTesting.calibrateModelRecall ( thisModel )

Logic to initialise calibration of model recall in order to recognize known from unknown instances.

Parameters

thisModel SAMObject model to calibrate.

Returns: None

Definition at line 87 of file SAMTesting.py.

◆ calibrateMultipleModelRecall()

def SAM.SAM_Core.SAMTesting.calibrateMultipleModelRecall ( thisModel )

Perform calibration for multiple model implementations.

In contrast with calibrateSingleModelRecall, in this method known and unknown are calibrated according to measures of familiarity between all model classes. The familiarity of each class with each other class and with itself are then used to perform a Bayesian decision depending on the resulting familiarity when testing a new instance.

        SAMObject model to calibrate.

Returns: None

Definition at line 208 of file SAMTesting.py.

◆ calibrateSingleModelRecall()

def SAM.SAM_Core.SAMTesting.calibrateSingleModelRecall ( thisModel )

Perform calibration for single model implementations.

This method either uses the bhattacharyya distance to perform calibration of known and unknown or uses histograms to use the histogram distribution of known and unknown labels in the training data to carry out the classification. This method depends on the following parameters present in config.ini.
1) useMaxDistance : False or True. This enables the use of bhattacharyya distance method to recognise known and unknown.
2) calibrateUnknown : True or False. This turns on or off the calibration of the model for known and unknown inputs.
3) noBins : Integer number of bins to be used for the histogram method if calibrateUnknown is True and useMaxDistance is False.
4) method : String indicating the method used when histograms are used for calibration. When using histograms, the multi-dimensional probability of known and unknown are both calculated using the histogram. sumProb then performs a decision based on the largest sum after summing the probabilities of known and unknown independently. mulProb performs a decision based on the largest sum after multiplying the probabilities of known and unknown independently.

Parameters

thisModel SAMObject model to calibrate.

Returns: None

Definition at line 113 of file SAMTesting.py.

◆ combineClassifications()

def SAM.SAM_Core.SAMTesting.combineClassifications	(	thisModel,
		labels,
		likelihoods
	)

Combine multiple classifications into a single classification.

Parameters

thisModel	SAMObject model.
labels	List of labels for classifications.
likelihoods	List of likelihoods.

Returns: Label with the highest likelihood together with the normalised likelihood.

Definition at line 883 of file SAMTesting.py.

◆ deep_getsizeof()

def SAM.SAM_Core.SAMTesting.deep_getsizeof	(	o,
		ids
	)

Method to calculate the size of an object o in bytes.

Parameters

o	Object to calculate the size of.
ids	Set of ids to not consider when calculating the size of the object.

Returns: Size of object in bytes.

Definition at line 52 of file SAMTesting.py.

◆ formatDataFunc()

def SAM.SAM_Core.SAMTesting.formatDataFunc ( Ydata )

Utility function to format data for testing.

Parameters

Ydata Data to format for testing.

Returns: Formatted data for testing.

Definition at line 304 of file SAMTesting.py.

◆ multipleRecall()

def SAM.SAM_Core.SAMTesting.multipleRecall	(	thisModel,
		testInstance,
		verbose,
		visualiseInfo = `None`,
		optimise = `100`
	)

Method that performs classification for calibrated multiple model implementations.

Parameters

thisModel	SAMObject model to recall from.
testInstance	Novel feature vector to test.
verbose	Enable or disable logging to stdout.
visualiseInfo	None to disable plotting and plotObject to display plot of recall.
optimise	Number of optimisation iterations to perform during recall.

Returns: Classification label and calibrated familiarity values.

Definition at line 469 of file SAMTesting.py.

◆ multipleRecall_noCalib()

def SAM.SAM_Core.SAMTesting.multipleRecall_noCalib	(	thisModel,
		testInstance,
		verbose,
		visualiseInfo = `None`,
		optimise = `True`
	)

Method that performs classification for uncalibrated multiple model implementations.

Parameters

thisModel	SAMObject model to recall from.
testInstance	Novel feature vector to test.
verbose	Enable or disable logging to stdout.
visualiseInfo	None to disable plotting and plotObject to display plot of recall.
optimise	Number of optimisation iterations to perform during recall.

Returns: Classification label and raw familiarity values.

Definition at line 432 of file SAMTesting.py.

◆ segmentTesting()

def SAM.SAM_Core.SAMTesting.segmentTesting	(	thisModel,
		Ysample,
		Lnum,
		verbose,
		label,
		serialMode = `False`,
		optimise = `100`,
		calibrate = `False`
	)

Method to test multiple samples at a time.

Parameters

thisModel	SAMObject model to recall from.
Ysample	Novel feature vector to test.
Lnum	Ground truth labels to compare with.
verbose	Enable or disable logging to stdout.
label	Label for the current segments being tested.
serialMode	Boolean to test serially or in parallel.
optimise	Number of optimisation iterations to perform during recall.
calibrate	Indicate calibration mode when True which requires a different return.

Returns: labelList, confMatrix, ret, variancesKnown, variancesUnknown if calibrate is True. labelList, confMatrix, labelComparisonDict if calibrate is False.; labelList List of classification labels; confMatrix Numpy array with the confusion matrix; ret Classification object; variancesKnown Variances returned during calibration for known training instances; variancesUnknown Variances returned during calibration for unknown training instances; labelComparisonDict Dictionary with two items 'original' and 'results'.

Definition at line 584 of file SAMTesting.py.

◆ singleRecall()

def SAM.SAM_Core.SAMTesting.singleRecall	(	thisModel,
		testInstance,
		verbose,
		visualiseInfo = `None`,
		optimise = `100`
	)

Method that performs classification for single model implementations.

This method returns the classification label of a test instance by calculating the predictive mean and variance of the backwards mapping and subsequently decides whether the test instance is first known or unknown and if known its most probable classification label.

Parameters

thisModel	SAMObject model to recall from.
testInstance	Novel feature vector to test.
verbose	Enable or disable logging to stdout.
visualiseInfo	None to disable plotting and plotObject to display plot of recall.
optimise	Number of optimisation iterations to perform during recall.

Returns: Classification label and variance if calibrateUnknown is set to False in the config file. Otherwise returns classification label and normalised classification probability.

Definition at line 326 of file SAMTesting.py.

◆ testSegment()

def SAM.SAM_Core.SAMTesting.testSegment	(	thisModel,
		Ysample,
		verbose,
		visualiseInfo = `None`,
		optimise = `100`
	)

Utility function to test a sample.

This model determines the type of model being used for the testing and directs the query to the appropriate function.

Parameters

thisModel	SAMObject model to recall from.
Ysample	Novel feature vector to test.
verbose	Enable or disable logging to stdout.
visualiseInfo	`None` to disable plotting and plotObject to display plot of recall.
optimise	Number of optimisation iterations to perform during recall.

Returns: Classification result containing a list with the classification string and a measure of the familiarity or probability of the recall.

Definition at line 551 of file SAMTesting.py.

◆ testSegments()

def SAM.SAM_Core.SAMTesting.testSegments	(	thisModel,
		Ysample,
		Lnum,
		verbose,
		label,
		serialMode = `False`
	)

Function to test segments and return a confusion matrix.

Parameters

thisModel	SAMObject model to recall from.
Ysample	Novel feature vector to test.
Lnum	Ground truth labels to compare with.
verbose	Enable or disable logging to stdout.
label	Label for the current segments being tested.
serialMode	Boolean to test serially or in parallel.

Returns: Confusion matrix, confusion labels, list of possible labels and dictionary with results and comparison or truth and classification.

Definition at line 768 of file SAMTesting.py.

◆ wait_watching_stdout()

def SAM.SAM_Core.SAMTesting.wait_watching_stdout	(	ar,
		dt = `1`,
		truncate = `1000`
	)

Monitoring function that logs to stdout the logging output of multiple threads.

Parameters

ar	Thread to log.
dt	Integer delta time between readings of ar.stdout.
truncate	Integer limit on the number of lines returned by the threads.

Returns: None

Definition at line 521 of file SAMTesting.py.

Namespaces

Functions

Detailed Description

Function Documentation

◆ calculateData()

◆ calculateVarianceThreshold()

◆ calibrateModelRecall()

◆ calibrateMultipleModelRecall()

◆ calibrateSingleModelRecall()

◆ combineClassifications()

◆ deep_getsizeof()

◆ formatDataFunc()

◆ multipleRecall()

◆ multipleRecall_noCalib()

◆ segmentTesting()

◆ singleRecall()

◆ testSegment()

◆ testSegments()

◆ wait_watching_stdout()