`tsgm.metrics.metrics`¶

Module Contents¶

class Metric[source]¶

Bases: abc.ABC

Helper class that provides a standard way to create an ABC using inheritance.

class DistanceMetric(statistics: list, discrepancy: Callable)[source]¶

Bases: Metric

Metric that measures similarity between synthetic and real time series

Parameters:

statistics (list) – A list of summary statistics (callable)
discrepancy (Callable) – Discrepancy function, measures the distance between the vectors of summary statistics.

stats(X: tsgm.types.Tensor) → tsgm.types.Tensor[source]¶

Parameters:: X (tsgm.types.Tensor.) – A time series dataset.
Returns:: a tensor with calculated summary statistics.

discrepancy(stats1: tsgm.types.Tensor, stats2: tsgm.types.Tensor) → float[source]¶

Parameters:

stats1 (tsgm.types.Tensor.) – A vector of summary statistics.
stats2 (tsgm.types.Tensor.) – A vector of summary statistics.

Returns:

the distance between two vectors calculated by self._discrepancy.

__call__(D1: tsgm.dataset.DatasetOrTensor, D2: tsgm.dataset.DatasetOrTensor) → float[source]¶

Parameters:

D1 (tsgm.dataset.DatasetOrTensor.) – A time series dataset.
D2 (tsgm.dataset.DatasetOrTensor.) – A time series dataset.

Returns:

similarity metric between D1 & D2.

class ConsistencyMetric(evaluators: List)[source]¶

Bases: Metric

Predictive consistency metric measures whether a set of evaluators yield consistent results on real and synthetic data.

Parameters:: evaluators (list) – A list of evaluators (each item should implement method .evaluate(D))

__call__(D1: tsgm.dataset.DatasetOrTensor, D2: tsgm.dataset.DatasetOrTensor, D_test: tsgm.dataset.DatasetOrTensor) → float[source]¶

Parameters:

D1 (tsgm.dataset.DatasetOrTensor.) – A time series dataset.
D2 (tsgm.dataset.DatasetOrTensor.) – A time series dataset.

Returns:

consistency metric between D1 & D2.

class BaseDownstreamEvaluator[source]¶

Bases: abc.ABC

Helper class that provides a standard way to create an ABC using inheritance.

class DownstreamPerformanceMetric(evaluator: BaseDownstreamEvaluator)[source]¶

Bases: Metric

The downstream performance metric evaluates the performance of a model on a downstream task. It returns performance gains achieved with the addition of synthetic data.

Parameters:: evaluator (BaseDownstreamEvaluator) – An evaluator, should implement method .evaluate(D)

__call__(D1: tsgm.dataset.DatasetOrTensor, D2: tsgm.dataset.DatasetOrTensor, D_test: tsgm.dataset.DatasetOrTensor | None, return_std: bool = False) → float[source]¶

Parameters:

D1 (tsgm.dataset.DatasetOrTensor.) – A time series dataset.
D2 (tsgm.dataset.DatasetOrTensor.) – A time series dataset.

Returns:

downstream performance metric between D1 & D2.

class PrivacyMembershipInferenceMetric(attacker: Any, metric: Callable | None = None)[source]¶

Bases: Metric

The metric that measures the possibility of membership inference attacks.

Parameters:

attacker (Callable) – An attacker, one class classififier (OCC) that implements methods .fit and .predict
metric – Measures quality of attacker (precision by default)

__call__(d_tr: tsgm.dataset.Dataset, d_syn: tsgm.dataset.Dataset, d_test: tsgm.dataset.Dataset) → float[source]¶

Parameters:

d_tr (tsgm.dataset.DatasetOrTensor.) – Training dataset (the dataset that was used to produce d_dyn).
d_syn (tsgm.dataset.DatasetOrTensor.) – Training dataset (the dataset that was used to produce d_dyn).
d_test (tsgm.dataset.DatasetOrTensor.) – Training dataset (the dataset that was used to produce d_dyn).

Returns:

how well the attacker can distinguish d_tr & d_test when it is trained on d_syn.

class MMDMetric(kernel: Callable = tsgm.utils.mmd.exp_quad_kernel)[source]¶

Bases: Metric

This metric calculated MMD between real and synthetic samples

Args:

d (tsgm.dataset.DatasetOrTensor): The input dataset or tensor.

Returns:

float: The computed spectral entropy.

Example:

>>> metric = MMDMetric(kernel)
>>> dataset, synth_dataset = tsgm.dataset.Dataset(...), tsgm.dataset.Dataset(...)
>>> result = metric(dataset)
>>> print(result)

class DiscriminativeMetric[source]¶

Bases: Metric

The DiscriminativeMetric measures the discriminative performance of a model in distinguishing between synthetic and real datasets.

This metric evaluates a discriminative model by training it on a combination of synthetic and real datasets and assessing its performance on a test set.

Parameters:

d_hist (tsgm.dataset.DatasetOrTensor) – Real dataset.
d_syn (tsgm.dataset.DatasetOrTensor) – Synthetic dataset.
model (T.Callable) – Discriminative model to be evaluated.
test_size (T.Union[float, int]) – Proportion of the dataset to include in the test split or the absolute number of test samples.
n_epochs (int) – Number of training epochs for the model.
metric (T.Optional[T.Callable]) – Optional evaluation metric to use (default: accuracy).
random_seed (T.Optional[int]) – Optional random seed for reproducibility.

Returns:

Discriminative performance metric.

Return type:

float

Example:¶

>>> from my_module import DiscriminativeMetric, MyDiscriminativeModel
>>> import tsgm.dataset
>>> import numpy as np
>>> import sklearn
>>>
>>> # Create real and synthetic datasets
>>> real_dataset = tsgm.dataset.Dataset(...)  # Replace ... with appropriate arguments
>>> synthetic_dataset = tsgm.dataset.Dataset(...)  # Replace ... with appropriate arguments
>>>
>>> # Create a discriminative model
>>> model = MyDiscriminativeModel()  # Replace with the actual discriminative model class
>>>
>>> # Create and use the DiscriminativeMetric
>>> metric = DiscriminativeMetric()
>>> result = metric(real_dataset, synthetic_dataset, model, test_size=0.2, n_epochs=10)
>>> print(result)

class EntropyMetric[source]¶

Bases: Metric

Calculates the spectral entropy of a dataset or tensor.

This metric measures the randomness or disorder in a dataset or tensor using spectral entropy, which is a measure of the distribution of energy in the frequency domain.

Args:

d (tsgm.dataset.DatasetOrTensor): The input dataset or tensor.

Returns:

float: The computed spectral entropy.

Example:

>>> metric = EntropyMetric()
>>> dataset = tsgm.dataset.Dataset(...)
>>> result = metric(dataset)
>>> print(result)

__call__(d: tsgm.dataset.DatasetOrTensor) → float[source]¶

Calculate the spectral entropy of the input dataset or tensor.

Args:: d (tsgm.dataset.DatasetOrTensor): The input dataset or tensor.
Returns:: float: The computed spectral entropy.

class DemographicParityMetric[source]¶

Bases: Metric

Measuring demographic parity between two datasets.

This metric assesses the disparity in the distributions of a target variable among different groups in two datasets. By default, it uses the Kolmogorov-Smirnov statistic to quantify the maximum vertical deviation between the cumulative distribution functions of the target variable for the historical and synthetic data within each group.

Args:

d_hist (tsgm.dataset.DatasetOrTensor): The historical input dataset or tensor. groups_hist (TensorLike): The group assignments for the historical data. d_synth (tsgm.dataset.DatasetOrTensor): The synthetic input dataset or tensor. groups_synth (TensorLike): The group assignments for the synthetic data. metric (callable, optional): The metric used to compare the target variable distributions within each group.

Default is the Kolmogorov-Smirnov statistic.

Returns:

dict: A dictionary mapping each group to the computed demographic parity metric.

Example:

>>> metric = DemographicParityMetric()
>>> dataset_hist = tsgm.dataset.Dataset(...)
>>> dataset_synth = tsgm.dataset.Dataset(...)
>>> groups_hist = [0, 1, 0, 1, 1, 0]
>>> groups_synth = [1, 1, 0, 0, 0, 1]
>>> result = metric(dataset_hist, groups_hist, dataset_synth, groups_synth)
>>> print(result)

__call__(d_hist: tsgm.dataset.DatasetOrTensor, groups_hist: tensorflow.python.types.core.TensorLike, d_synth: tsgm.dataset.DatasetOrTensor, groups_synth: tensorflow.python.types.core.TensorLike, metric: Callable = _DEFAULT_KS_METRIC) → Dict[source]¶

Calculate the demographic parity metric for the input datasets.

Args:: d_hist (tsgm.dataset.DatasetOrTensor): The historical input dataset or tensor. groups_hist (TensorLike): The group assignments for the historical data. d_synth (tsgm.dataset.DatasetOrTensor): The synthetic input dataset or tensor. groups_synth (TensorLike): The group assignments for the synthetic data. metric (callable, optional): The metric used to compare the target variable distributions within each group.

Default is the Kolmogorov-Smirnov statistic.
Returns:: dict: A dictionary mapping each group to the computed demographic parity metric.

tsgm.metrics.metrics¶

Module Contents¶

Example:¶

`tsgm.metrics.metrics`¶