playground_metrics.map_metric

Usage

The MeanAveragePrecisionMetric class is responsible for mAP computation. To do so it relies on a MatchEngine whose role is to associate detection to corresponding ground truth in order to assess whether a particular detection counts as a False Positive or True Positive. For more information on the matching algorithms provided, please see Matching.

By default, MeanAveragePrecisionMetric expects either bounding boxes or polygons but this may be overridden by providing a user-instantiated MatchEngineBase subclasses (If no MatchEngineBase is provided, a default MatchEngineIoU is created upon instantiation). More information on other possible MatchEngineBase, please see playground_metrics.match_detections.

Note

  • The update() method takes an entire image/tile worth of inputs.

  • The following behaviour are to be expected in case of empty input:

    • No detection and no ground truth keep the mAP constant

    • No detection with ground truths decreases the mAP (they are assumed to be False Negative)

    • Detections with no ground truth decreases the mAP (they are assumed to be False Positive)

API doc

Implement the public interface to compute mAP from a set of detections and ground truths.

For more in formation on how the metric is computed, see: ../content/map_metric.

If one wants to integrate the module into a framework to use it as a validation metric, the MeanAveragePrecisionMetric class described below should be wrappped accordingly to follow the framework convention.

class playground_metrics.map_metric.MeanAveragePrecisionMetric(threshold=None, match_algorithm=None, label_mean_area=None, trim_invalid_geometry=False, autocorrect_invalid_geometry=False, match_engine=None)[source]

Bases: object

Implement an API to compute mAP.

It gives three methods:

  • update(detections, ground_truths) which accumulates TP, FP and FN over examples

  • compute() which computes mAP and AP per label from accumulated values

  • reset() which resets accumulated values to their initial values to start mAP computation from scratch

See also

Information on how mAP, AP, precision and recall are computed may be found in ../content/map_metric.

Parameters:

  • threshold (float) – Optional, default to 0.5. Similarity threshold for which we consider a valid match between detection and ground truth.

  • match_algorithm (str) – Optional, default to ‘coco’. ‘xview’ or ‘coco’ to choose the matching algorithm (c.f. Matching) or ‘non-unitary’ to use non-unitary matching.

  • label_mean_area (dict) – Optional, default to None. A dictionary containing the mean area for each label in the dataset, if given, it is used to match with iIoU instead of IoU (c.f. Instance IoU).

  • trim_invalid_geometry (bool) – Optional, default to False. If set to True conversion will ignore invalid geometries and leave them out of mAP computations. This means that the function will work on arrays where work_array.shape[0] <= input_array.shape[0]. If set to False, an invalid geometry will raise an InvalidGeometryError.

  • autocorrect_invalid_geometry (Bool) – Optional, default to False. Whether to attempt correcting a faulty geometry to form a valid one. If set to True and the autocorrect attempt is unsuccessful, it falls back to the behaviour defined in trim_invalid_geometry.

  • match_engine (MatchEngineBase) – Optional, default to MatchEngineIoU. If provided matching will be done using the provided match_engine instead of the default one. Note that the threshold and match_algorithm provided parameters will be overridden by those provided in the match_engine.

Warning

When using non-unitary matching, the AP per class and the mAP are ill-defined and must be taken with a grain of salt.

Warns:

  • UserWarning – If match_algorithm is ‘non-unitary’ to warn that mAP and AP per class values are ill-defined.

  • RuntimeWarning – If a match_engine is provided and its threshold or match_algorithm attribute differs from those provided as arguments to the constructor.

Note

  • Polygon auto-correction only corrects self-crossing exterior rings, in which case it creates one Polygon out of every simple ring which might be extracted from the original Polygon exterior.

  • Polygon auto-correction will systematically fail on Polygons with at least one inner ring.

mAP

The mAP computed by compute() from accumulated values

Type:

float

average_precision_per_class

The AP for each label as constructed by compute() from accumulated values

Type:

defaultdict

precision_per_class

The precision for each label as constructed by compute() from accumulated values

Type:

defaultdict

recall_per_class

The recall for each label as constructed by compute() from accumulated values

Type:

defaultdict

number_true_detection_per_class

The number of detection matched to a ground truth as constructed by compute() from accumulated values

Type:

defaultdict

number_false_detection_per_class

The number of detection not matched to a ground truth as constructed by compute() from accumulated value

Type:

defaultdict

number_found_ground_truth_per_class

The number of ground truth matched to a detection as constructed by compute() from accumulated values

Type:

defaultdict

number_missed_ground_truth_per_class

The number of ground truth not matched to a detection as constructed by compute() from accumulated values

Type:

defaultdict

match_engine

The match_engine object used to match detections and ground truths. If none where provided in the constructor call, it defaults to MatchEngineIoU.

Type:

MatchEngineBase

property threshold

The IoU threshold by self.match_engine or None if self.match_engine doesn’t use any threshold.

Type:

float

property ground_truth_labels

The set of unique label accumulated up to this point.

Type:

set

update(detections, ground_truths)[source]

Accumulate values necessary to compute mAP with detections and ground truths of a single image.

Parameters:

  • detections (ndarray, list) –

    A ndarray of detections stored as:

    • Bounding boxes for a given class where each row is a detection stored as: [x_min, y_min, x_max, y_max, confidence, label]

    • Polygons for a given class where each row is a detection stored as: [[[outer_ring], [inner_rings]], confidence, label]

    • Points for a given class where each row is a detection stored as: [x, y, confidence, label]

  • ground_truths (ndarray,list) –

    A ndarray of ground truth stored as:

    • Bounding boxes for a given class where each row is a ground truth stored as: [x_min, y_min, x_max, y_max, label]

    • Polygons for a given class where each row is a ground truth stored as: [[[outer_ring], [inner_rings]], label]

    • Points for a given class where each row is a ground truth stored as: [x, y, label]

Raises:

KeyError – If self.label_mean_area is not None but a label is missing

The input detections and ground truths are allowed to be points in the documentation. This is to allow the use of a custom MatchEngine for points, however, the default MatchEngineIoU works on intersection-over-union which is incompatible with points. More information on input geometrical types can be found in playground_metrics.match_detections.

Note

The labels provided in the input arrays can theoretically be any hashable type, however, only numeric types, strings and tuples are officially supported.

compute()[source]

Compute the mAP according to the accumulated values.

Moreover it sets the value for the following attributes:

Returns:

The Mean Average Precision metric

Return type:

float

reset()[source]

Reset all intermediate and return values to their initial value.

If reset() is not called in-between two compute() call, the values returned by compute() will take into account the entire prediction stack, not just the predictions in-between the two compute() calls.