API Reference: Evaluation

Evaluation metrics for dimensionality reduction quality assessment.

This module provides evaluation functions for assessing the quality of dimensionality reduction embeddings. It includes both ranking-based and distance-based metrics to evaluate how well the low-dimensional representation preserves the structure and relationships from the high-dimensional space.

The implementation includes adaptations from the pyDRMetrics package by Yinsheng Zhang, with optimizations for runtime efficiency and integration with the moddr pipeline.

References

  • Zhang, Y. pyDRMetrics: A Python package for dimensionality reduction quality metrics. https://github.com/zhangys11/pyDRMetrics

  • Lee, J. A., et al. “Type 1 and 2 mixtures of Kullback–Leibler divergences as cost functions in dimensionality reduction based on similarity preservation.” Neurocomputing 112 (2013): 92-108.

moddr.evaluation.evaluation.compute_continuity(cr_matrix: ndarray[tuple[int, ...], dtype[float32]], k: int) → float[source]

Compute the continuity metric from a co-ranking matrix.

Continuity measures how well the k-nearest neighbors in the high-dimensional space correspond to the k-nearest neighbors in the low-dimensional space. A value of 1 indicates perfect continuity.

This implementation is adapted from the pyDRMetrics package by Yinsheng Zhang [1].

Parameters:

  • cr_matrix (npt.NDArray[np.float32]) – The co-ranking matrix computed from ranking data.

  • k (int) – The neighborhood size parameter. Must be in range [1, n-1] where n is the size of the co-ranking matrix.

Returns:

The continuity value between 0 and 1, where 1 is perfect.

Return type:

float

Raises:

ValueError – If k is not in the valid range [1, n-1].

References

[1] Zhang, Y. pyDRMetrics: A Python package for dimensionality reduction quality metrics. https://github.com/zhangys11/pyDRMetrics Licensed under Creative Commons Attribution 4.0 International (CC BY 4.0). https://creativecommons.org/licenses/by/4.0/

moddr.evaluation.evaluation.compute_coranking_matrix(r1: ndarray[tuple[int, ...], dtype[int32]], r2: ndarray[tuple[int, ...], dtype[int32]]) → ndarray[tuple[int, ...], dtype[int32]][source]

Compute the co-ranking matrix between two ranking arrays.

This implementation is adapted from the pyDRMetrics package by Yinsheng Zhang [1]. Modifications include computing AUC of metrics only when needed for runtime efficiency, rather than automatically during co-ranking matrix computation.

Parameters:

  • r1 (npt.NDArray[np.int32]) – Ranking matrix for the original high-dimensional space. Each row contains the ranks of distances for one data point.

  • r2 (npt.NDArray[np.int32]) – Ranking matrix for the reduced low-dimensional space. Each row contains the ranks of distances for one data point. Must have the same shape as r1.

Returns:

The co-ranking matrix as a 2D array where entry (i,j) represents the count of points that are among the i nearest neighbors in the original space and j nearest neighbors in the reduced space.

Return type:

npt.NDArray[np.int32]

Raises:

ValueError – If the input ranking arrays have different shapes.

References

[1] Zhang, Y. pyDRMetrics: A Python package for dimensionality reduction quality metrics. https://github.com/zhangys11/pyDRMetrics Licensed under Creative Commons Attribution 4.0 International (CC BY 4.0). https://creativecommons.org/licenses/by/4.0/

moddr.evaluation.evaluation.compute_dist_score(embedding: EmbeddingState) → float[source]

Compute the overall distance score from distance-based metrics.

The distance score combines similarity stress and community stress difference metrics to provide an overall assessment of distance preservation quality. The score is normalized to the range [0, 1] where 1 is perfect.

Parameters:

embedding (EmbeddingState) – An EmbeddingState object containing the computed metrics ‘sim_stress’ and ‘sim_stress_com_diff’.

Returns:

The distance score as a float between 0 and 1, where 1 is perfect.

Return type:

float

Raises:

ValueError – If any of the required metrics (‘sim_stress’, ‘sim_stress_com_diff’) are missing from the embedding’s metrics dictionary.

moddr.evaluation.evaluation.compute_kruskal_stress(dists_highdim: ndarray[tuple[int, ...], dtype[float32]], dists_lowdim: ndarray[tuple[int, ...], dtype[float32]]) → float[source]

Compute the Kruskal stress metric between high-dimensional and low-dimensional distances.

Kruskal stress measures how well the low-dimensional embedding preserves the pairwise distances from the high-dimensional space. The stress is normalized by the sum of squared high-dimensional distances.

Parameters:

  • dists_highdim (npt.NDArray[np.float32]) – Array of pairwise distances in high-dimensional space. Can be either a square distance matrix or condensed distance vector.

  • dists_lowdim (npt.NDArray[np.float32]) – Array of pairwise distances in low-dimensional space. Can be either a square distance matrix or condensed distance vector. Must have the same shape as dists_highdim.

Returns:

The normalized Kruskal stress value (0 = perfect preservation, higher = worse).

Return type:

float

Raises:

ValueError – If the input arrays have different shapes or have non-zero off-diagonal elements.

moddr.evaluation.evaluation.compute_kruskal_stress_partition(dists_highdim: ndarray[tuple[int, ...], dtype[float32]], dists_lowdim: ndarray[tuple[int, ...], dtype[float32]], partition: dict[int, ndarray[tuple[int, ...], dtype[int32]]]) → float[source]

Compute the average Kruskal stress for each community in a partition.

This function computes the Kruskal stress separately for each community defined in the partition and returns the average stress across all communities with at least 2 nodes.

Parameters:

  • dists_highdim (npt.NDArray[np.float32]) – Array of pairwise distances in high-dimensional space. Must be a square distance matrix or condensed distance vector.

  • dists_lowdim (npt.NDArray[np.float32]) – Array of pairwise distances in low-dimensional space. Must be a square distance matrix or condensed distance vector. Must have the same shape as dists_highdim.

  • partition (dict[int, npt.NDArray[np.int32]]) – Dictionary mapping community IDs to arrays of node indices belonging to each community.

Returns:

The average Kruskal stress across all communities with at least 2 nodes.

Return type:

float

Raises:

ValueError – If the input arrays have different shapes or have non-zero off-diagonal elements.

moddr.evaluation.evaluation.compute_metrics(highdim_df: DataFrame, embeddings: list[EmbeddingState], target_features: list[str], fixed_k: int | None = None, ranking_metrics: bool = True, distance_metrics: bool = True, inplace: bool = False, verbose: bool = False) → list[EmbeddingState][source]

Compute comprehensive evaluation metrics for dimensionality reduction embeddings.

This function computes various quality metrics for dimensionality reduction including ranking-based metrics (trustworthiness, continuity, RNX) and distance-based metrics (stress measures). The metrics are computed for each embedding and stored in their respective metrics dictionaries.

The co-ranking matrix computation and associated metrics are adapted from the pyDRMetrics package by Yinsheng Zhang [1], with modifications to compute AUC metrics only when needed for improved runtime efficiency.

Parameters:

  • highdim_df (pd.DataFrame) – DataFrame containing the original high-dimensional data.

  • embeddings (list[EmbeddingState]) – List of EmbeddingState objects to evaluate.

  • target_features (list[str]) – List of feature names in highdim_df to use for similarity calculations.

  • fixed_k (int | None) – If specified, compute ranking metrics only for this neighborhood size. If None, compute AUC (average) over all possible k values.

  • ranking_metrics (bool) – Whether to compute ranking-based metrics (trustworthiness, continuity, RNX). Default is True.

  • distance_metrics (bool) – Whether to compute distance-based metrics (stress). Default is True.

  • inplace (bool) – Whether to modify embeddings in-place or create deep copies. Default is False (creates copies).

  • verbose (bool) – Whether to print progress information. Default is False.

Returns:

List of EmbeddingState objects with computed metrics. If inplace=False, these are deep copies of the input embeddings.

Return type:

list[EmbeddingState]

Notes

  • Community-based stress metrics require partitions to be defined in the embeddings.

  • The reference embedding for community stress differences is taken from the first embedding in the list.

References

Zhang, Y. pyDRMetrics: A Python package for dimensionality reduction quality metrics. https://github.com/zhangys11/pyDRMetrics Licensed under Creative Commons Attribution 4.0 International (CC BY 4.0). https://creativecommons.org/licenses/by/4.0/

moddr.evaluation.evaluation.compute_rank_score(embedding: EmbeddingState) → float[source]

Compute the overall rank score from ranking-based metrics.

The rank score is the average of trustworthiness, continuity, and RNX metrics. All three metrics must have been computed and stored in the embedding’s metrics dictionary.

Parameters:

embedding (EmbeddingState) – An EmbeddingState object containing the computed metrics ‘trustworthiness’, ‘continuity’, and ‘rnx’.

Returns:

The average rank score as a float between 0 and 1, where 1 is perfect.

Return type:

float

Raises:

ValueError – If any of the required metrics are missing from the embedding’s metrics dictionary.

moddr.evaluation.evaluation.compute_rnx(qnn: ndarray[tuple[int, ...], dtype[float32]], k: int) → float[source]

Compute the RNX (R_NX) metric from QNN values (see [1]).

RNX (Rank-based relative neighborhood preservation) measures the degree to which the neighborhood structure is preserved in the dimensionality reduction. A value of 1 indicates perfect preservation.

Parameters:

  • qnn (npt.NDArray[np.float32]) – Array of QNN (Quality of Nearest Neighbors) values computed from the co-ranking matrix.

  • k (int) – The neighborhood size parameter. Must be in range [1, n-1] where n is the length of the qnn array.

Returns:

The RNX value, where 1 indicates perfect neighborhood preservation.

Return type:

float

Raises:

ValueError – If k is not in the valid range [1, n-1].

Reference:

[1]: Lee, J. A., et al. “Type 1 and 2 mixtures of Kullback–Leibler divergences as cost functions in dimensionality reduction based on similarity preservation.” Neurocomputing 112 (2013): 92-108. https://doi.org/10.1016/j.neucom.2012.12.036

moddr.evaluation.evaluation.compute_total_score(embedding: EmbeddingState, balance: float = 0.5) → float[source]

Compute the overall total score by combining rank and distance scores.

The total score is a weighted average of the rank score and distance score, allowing for different emphasis on ranking-based vs distance-based metrics.

Parameters:

  • embedding (EmbeddingState) – An EmbeddingState object containing the computed metrics ‘rank_score’ and ‘distance_score’.

  • balance (float) – Weight for the rank score in the range [0, 1]. The distance score gets weight (1 - balance). Default is 0.5 for equal weighting.

Returns:

The total score as a float between 0 and 1, where 1 is perfect.

Return type:

float

Raises:

ValueError – If any of the required metrics (‘rank_score’, ‘distance_score’) are missing from the embedding’s metrics dictionary.

moddr.evaluation.evaluation.compute_trustworthiness(cr_matrix: ndarray[tuple[int, ...], dtype[float32]], k: int) → float[source]

Compute the trustworthiness metric from a co-ranking matrix.

Trustworthiness measures how well the k-nearest neighbors in the low-dimensional space correspond to the k-nearest neighbors in the high-dimensional space. A value of 1 indicates perfect trustworthiness.

This implementation is adapted from the pyDRMetrics package by Yinsheng Zhang [1].

Parameters:

  • cr_matrix (npt.NDArray[np.float32]) – The co-ranking matrix computed from ranking data.

  • k (int) – The neighborhood size parameter. Must be in range [1, n-1] where n is the size of the co-ranking matrix.

Returns:

The trustworthiness value between 0 and 1, where 1 is perfect.

Return type:

float

Raises:

ValueError – If k is not in the valid range [1, n-1].

References

[1] Zhang, Y. pyDRMetrics: A Python package for dimensionality reduction quality metrics. https://github.com/zhangys11/pyDRMetrics Licensed under Creative Commons Attribution 4.0 International (CC BY 4.0). https://creativecommons.org/licenses/by/4.0/

moddr.evaluation.evaluation.create_report(embeddings: list[EmbeddingState], metadata: bool = True, metrics: bool = True) → DataFrame[source]

Create a DataFrame report from a list of EmbeddingState objects.

This function extracts metadata and/or metrics from EmbeddingState objects and creates a structured DataFrame for analysis and comparison.

Parameters:

  • embeddings (list[EmbeddingState]) – List of EmbeddingState objects to include in the report.

  • metadata (bool) – Whether to include metadata columns in the report. Default is True.

  • metrics (bool) – Whether to include metrics columns in the report. Default is True.

Returns:

A pandas DataFrame containing the requested information from all embeddings. The ‘coranking_matrix’ metric is excluded from the output as it is not suitable for tabular representation.

Return type:

pd.DataFrame

Raises:

ValueError – If both metadata and metrics are False, or if the embeddings list is empty.

Notes

  • Each row in the DataFrame represents one embedding.

  • The ‘obj_id’ column is always included to identify embeddings.

  • The co-ranking matrix is excluded from metrics output due to size.