API Reference: Processing

Processing pipeline for modified dimensionality reduction (moddr).

This module provides the core processing functionality for the moddr package, implementing a pipeline for modified dimensionality reduction with community detection and position refinement.

Pipeline Stages: 1. Dimensionality Reduction: UMAP-based reduction to 2D space 2. Feature Similarity: Computation of pairwise similarities based on target features 3. Graph Construction: Creation of neighborhood graphs from DR results or KNN 4. Community Detection: Leiden algorithm for community identification 5. Position Refinement: Layout algorithms (MDS, Kamada-Kawai, Fruchterman-Reingold) 6. Metrics Computation: Quality assessment of the final embeddings

The module supports both individual function usage and complete pipeline execution through the run_pipeline function.

moddr.processing.processing.apply_balance_factor(embedding: EmbeddingState, modified_positions: ndarray[tuple[int, ...], dtype[float32]], balance_factor: float, inplace: bool = False, verbose: bool = False) → EmbeddingState

Apply a balance factor to blend original and modified positions.

This function creates a weighted combination of the original embedding positions and new modified positions using a balance factor.

Parameters:

• embedding (EmbeddingState) – EmbeddingState object containing the original positions.

• modified_positions (npt.NDArray[np.float32]) – Array of new positions to blend with originals.

• balance_factor (float) – Weight for the modified positions (0-1). 0 = only original positions, 1 = only modified positions.

• inplace (bool) – Whether to modify the embedding in-place or create a copy. Default is False.

• verbose (bool) – Whether to print progress information. Default is False.

Returns:

EmbeddingState object with blended positions. If inplace=False, returns a deep copy of the input embedding.

Return type:

EmbeddingState

Raises:

ValueError – If embedding has no positions, modified_positions is None, lengths don’t match, or balance_factor is not in [0, 1].

moddr.processing.processing.assign_graph_edge_weights(embedding: EmbeddingState, similarity_matrix: ndarray[tuple[int, ...], dtype[float32]], inplace: bool = False, verbose: bool = False) → EmbeddingState

Assign edge weights to a graph based on a similarity matrix.

This function sets the ‘weight’ attribute of each edge in the embedding’s graph to the corresponding similarity value from the similarity matrix.

Parameters:

• embedding (EmbeddingState) – EmbeddingState object containing the graph to modify.

• similarity_matrix (npt.NDArray[np.float32]) – Square matrix containing similarity values between nodes. Matrix should be indexed by node IDs.

• inplace (bool) – Whether to modify the embedding in-place or create a copy. Default is False.

• verbose (bool) – Whether to print progress information. Default is False.

Returns:

EmbeddingState object with updated edge weights. If inplace=False, returns a deep copy of the input embedding.

Return type:

EmbeddingState

Raises:

ValueError – If the embedding object doesn’t have a graph or if the similarity matrix is not suitable (e.g., not square or different shape than the graph).

moddr.processing.processing.community_detection_leiden(embedding: EmbeddingState, resolution_parameter: float, inplace: bool = False, verbose: bool = False) → EmbeddingState

Perform community detection using the Leiden algorithm.

This function applies the Leiden algorithm for community detection on the embedding’s graph, using edge weights and a specified resolution parameter. The detected communities are stored in the embedding’s partition attribute and as node attributes in the graph.

Uses igraph for community detection, as networkx does not support a native implementation of the Leiden algorithm.

Parameters:

• embedding (EmbeddingState) – EmbeddingState object containing the graph for community detection.

• resolution_parameter (float) – Resolution parameter controlling the granularity of communities. Higher values lead to smaller, more communities.

• inplace (bool) – Whether to modify the embedding in-place or create a copy. Default is False.

• verbose (bool) – Whether to print progress information. Default is False.

Returns:

EmbeddingState object with detected communities. If inplace=False, returns a deep copy of the input embedding.

Return type:

EmbeddingState

Raises:

ValueError – If the embedding object doesn’t have a graph or edge weights.

moddr.processing.processing.compute_boundary_subgraph(graph: Graph, subgraph: Graph) → tuple[Graph, list[Any]]

Compute the boundary subgraph by adding boundary nodes and edges.

This function identifies neighbors of subgraph nodes that lie outside the subgraph and adds them as boundary nodes with their connecting edges. Boundary nodes are those that are adjacent to the subgraph but not part of it.

Parameters:

• graph (nx.Graph) – The full graph containing all nodes and edges.

• subgraph (nx.Graph) – A subgraph extracted from the full graph.

Returns:

Tuple containing: - Extended subgraph with boundary nodes and edges added - List of boundary neighbor node IDs

Return type:

tuple[nx.Graph, list[Any]]

moddr.processing.processing.compute_community_graphs(embedding: EmbeddingState, boundary_neighbors: bool = False) → tuple[dict[int, Graph], dict[int, tuple[float, float]], dict[int, list[Any]]]

Extract subgraphs and compute centers for each community.

This function creates individual subgraphs for each community in the embedding’s partition and computes their centers based on node positions.

The center position is defined as the median of all node positions within the community instead of bounding box centers to avoid distortions from outliers.

Parameters:

• embedding (EmbeddingState) – EmbeddingState object with detected communities.

• boundary_neighbors (bool) – Whether to include boundary nodes which are adjacent to the current community. Default is False.

Returns:

Tuple containing: - Dictionary mapping community IDs to their subgraphs - Dictionary mapping community IDs to their center coordinates - Dictionary mapping community IDs to boundary neighbor lists

Return type:

tuple[dict[int, nx.Graph], dict[int, tuple[float, float]], dict[int, list[Any]]]

moddr.processing.processing.compute_distance_scaling(dists_highdim: ndarray[tuple[int, ...], dtype[float32]], dists_lowdim: ndarray[tuple[int, ...], dtype[float32]]) → float

Compute scaling factor to align high-dimensional and low-dimensional distances.

This function calculates the optimal scaling factor to multiply low-dimensional distances to best match high-dimensional distances in a least-squares sense.

Parameters:

• dists_highdim (npt.NDArray[np.float32]) – Array of pairwise distances in high-dimensional space. Can be square matrix or condensed vector.

• dists_lowdim (npt.NDArray[np.float32]) – Array of pairwise distances in low-dimensional space. Can be square matrix or condensed vector. Must match shape of dists_highdim.

Returns:

Scaling factor as a float. If high-dimensional distances are all zero, returns 1.0.

Return type:

float

Raises:

ValueError – If the input arrays have different shapes.

moddr.processing.processing.compute_fruchterman_reingold_layout(embedding: EmbeddingState, partition_subgraphs: dict[int, Graph], pairwise_sims: ndarray[tuple[int, ...], dtype[float32]], iterations: int, boundary_neighbors: dict[int, list[int]] | None = None, inplace: bool = False, verbose: bool = False) → EmbeddingState

Apply Fruchterman-Reingold layout to community subgraphs.

This function applies the Fruchterman-Reingold force-directed algorithm to each community separately, using similarity values as edge weights.

Parameters:

• embedding (EmbeddingState) – EmbeddingState object with community partition.

• partition_subgraphs (dict[int, nx.Graph]) – Dictionary of community subgraphs.

• pairwise_sims (npt.NDArray[np.float32]) – Similarity matrix for edge weights.

• iterations (int) – Number of iterations to run the algorithm.

• boundary_neighbors (dict[int, list[int]] | None) – Dictionary of boundary neighbors for each community. Default is None.

• inplace (bool) – Whether to modify the embedding in-place. Default is False.

• verbose (bool) – Whether to print progress information. Default is False.

Returns:

Modified EmbeddingState object with new positions.

Return type:

EmbeddingState

moddr.processing.processing.compute_kamada_kawai_layout(embedding: EmbeddingState, partition_subgraphs: dict[int, Graph], pairwise_dists: ndarray[tuple[int, ...], dtype[float32]], balance_factor: float = 0.5, boundary_neighbors: dict[int, list[int]] | None = None, inplace: bool = False, verbose: bool = False) → tuple[EmbeddingState, dict[int, ndarray[tuple[int, ...], dtype[float32]]]]

Apply Kamada-Kawai layout algorithm to community subgraphs.

This function applies the Kamada-Kawai force-directed algorithm to each community separately, using target distances and a balance factor to blend original and new positions.

Parameters:

• embedding (EmbeddingState) – EmbeddingState object with community partition.

• partition_subgraphs (dict[int, nx.Graph]) – Dictionary of community subgraphs.

• pairwise_dists (npt.NDArray[np.float32]) – Target distance matrix for the layout algorithm.

• balance_factor (float) – Weight for blending original and new positions (0-1). Default is 0.5.

• boundary_neighbors (dict[int, list[int]] | None) – Dictionary of boundary neighbors for each community. Default is None.

• inplace (bool) – Whether to modify the embedding in-place. Default is False.

• verbose (bool) – Whether to print progress information. Default is False.

Returns:

Tuple containing: - Modified EmbeddingState object - Array of modified positions as computed by the layout algorithm (w/o balance factor influence)

Return type:

tuple[EmbeddingState, npt.NDArray[np.float32]]

moddr.processing.processing.compute_knn_graph(df: DataFrame, n_neighbors: int = 15, mode: str = 'distance', sim_features: list[str] | None = None) → tuple[Graph, ndarray[tuple[int, ...], dtype[float32]]]

Compute a k-nearest neighbors graph from the input data.

This function creates a k-nearest neighbors graph using the specified features and assigns edge weights based on pairwise distances.

Parameters:

• df (pd.DataFrame) – Input DataFrame containing the data points.

• n_neighbors (int) – Number of nearest neighbors to connect for each node. Default is 15.

• mode (str) – Mode for the KNN graph construction. Default is “distance” (euclidean).

• sim_features (list[str] | None) – List of feature column names to use for similarity computation. If None, uses all columns. Default is None.

Returns:

Tuple containing: - NetworkX graph with KNN connections and distance-based edge weights - Array of edge weights from the KNN graph

Return type:

tuple[nx.Graph, npt.NDArray[np.float32]]

moddr.processing.processing.compute_mds_layout(embedding: EmbeddingState, partition_subgraphs: dict[int, Graph], pairwise_dists: ndarray[tuple[int, ...], dtype[float32]], balance_factor: float = 0.5, boundary_neighbors: dict[int, list[int]] | None = None, inplace: bool = False, verbose: bool = False) → tuple[EmbeddingState, dict[int, ndarray[tuple[int, ...], dtype[float32]]]]

Apply Multidimensional Scaling (MDS) layout to community subgraphs.

This function applies MDS to each community separately, using target distances to compute new positions and a balance factor to blend with original positions.

Parameters:

• embedding (EmbeddingState) – EmbeddingState object with community partition.

• partition_subgraphs (dict[int, nx.Graph]) – Dictionary of community subgraphs.

• pairwise_dists (npt.NDArray[np.float32]) – Target distance matrix for the MDS algorithm.

• balance_factor (float) – Weight for blending original and new positions (0-1). Default is 0.5.

• boundary_neighbors (dict[int, list[int]] | None) – Dictionary of boundary neighbors for each community. Default is None.

• inplace (bool) – Whether to modify the embedding in-place. Default is False.

• verbose (bool) – Whether to print progress information. Default is False.

Returns:

Tuple containing: - Modified EmbeddingState object - Array of modified positions as computed by the layout algorithm (w/o balance factor influence)

Return type:

tuple[EmbeddingState, npt.NDArray[np.float32]]

moddr.processing.processing.compute_modified_positions(embedding: EmbeddingState, layout_param: int | float, layout_method: str, targets: ndarray[tuple[int, ...], dtype[float32]], boundary_neighbors: bool = False, inplace: bool = False, verbose: bool = False) → tuple[EmbeddingState, dict[int, ndarray[tuple[int, ...], dtype[float32]]]]

Compute new positions for an embedding using various layout algorithms.

This function applies different layout algorithms (Kamada-Kawai, MDS, or Fruchterman-Reingold) to modify node positions within detected communities while taking into account the original structure.

Parameters:

• embedding (EmbeddingState) – EmbeddingState object with computed partition.

• layout_param (int | float) – Algorithm-specific parameter. For FR: iterations, for KK/MDS: balance factor.

• layout_method (str) – Layout algorithm to use (“KK”, “MDS”, or “FR”).

• targets (npt.NDArray[np.float32]) – Target distance matrix for KK/MDS algorithms. Target similarity matrix for FR algorithm.

• boundary_neighbors (bool) – Whether to include boundary neighbors between communities. Default is False.

• inplace (bool) – Whether to modify the embedding in-place. Default is False.

• verbose (bool) – Whether to print progress information. Default is False.

Returns:

Tuple containing: - Modified EmbeddingState object - Dictionary of modified positions as computed by the layout algorithm (w/o balance factor influence)

Return type:

tuple[EmbeddingState, dict[int, npt.NDArray[np.float32]]]

Raises:

ValueError – If unsupported layout method is specified or required parameters are missing or invalid.

moddr.processing.processing.compute_pairwise_dists(df: DataFrame, apply_squareform: bool = True, invert: bool = False, normalize: bool = False, no_null: bool = False, sim_features: list[str] | None = None) → ndarray[tuple[int, ...], dtype[float32]]

Compute pairwise Euclidean distances between data points.

This function calculates pairwise distances between all data points, with various options for processing and transforming the distances.

Parameters:

• df (pd.DataFrame) – Input DataFrame containing the data points.

• apply_squareform (bool) – Whether to convert condensed distance matrix to square form. If False, returns condensed form. Default is True.

• invert (bool) – Whether to invert the distances. Uses 1/d if normalize=False, or 1-d if normalize=True. Default is False.

• normalize (bool) – Whether to normalize distances to [0, 1] range using MinMaxScaler. Default is False.

• no_null (bool) – Whether to replace zero distances with a small value (1e-9) to avoid division by zero. Default is False.

• sim_features (list[str] | None) – List of feature column names to use for distance computation. If None, uses all columns. Default is None.

Returns:

Array of pairwise distances as float32. Shape depends on apply_squareform parameter.

Return type:

npt.NDArray[np.float32]

Raises:

ValueError – If the input DataFrame is empty or if the provided sim_features are not in the DataFrame.

moddr.processing.processing.dimensionality_reduction_umap(data: DataFrame, n_neighbors: int = 15, min_dist: float = 1.0, random_state: int = 0, compute_metrics: bool = False) → EmbeddingState

Perform dimensionality reduction using UMAP.

Creates an EmbeddingState object with the resulting embedding and the UMAP-generated graph structure.

Parameters:

• data (pd.DataFrame) – Input DataFrame containing the high-dimensional data.

• n_neighbors (int) – Number of nearest neighbors to consider for UMAP. Controls the balance between local and global structure. Default is 15.

• min_dist (float) – Minimum distance parameter for UMAP. Controls how tightly UMAP packs points together. Default is 1.0.

• random_state (int) – Random seed for reproducible results. Default is 0.

• compute_metrics (bool) – Whether to compute evaluation metrics for the resulting embedding. Default is False.

Returns:

EmbeddingState object containing the 2D embedding coordinates, the UMAP-generated graph, and associated metadata.

Return type:

EmbeddingState

moddr.processing.processing.run_pipeline(data: DataFrame, sim_features: list[str], dr_method: str = 'UMAP', dr_param_n_neighbors: int = 15, graph_method: str = 'DR', community_resolutions: list[float] | None = None, community_resolution_amount: int = 3, layout_method: str = 'MDS', boundary_neighbors: bool = False, layout_params: list[int] | None = None, compute_metrics: bool = True, verbose: bool = False) → list[EmbeddingState]

Run the complete moddr (modified Dimensionality Reduction) pipeline.

This function orchestrates the entire pipeline including dimensionality reduction, feature similarity computation, graph construction, community detection, and position refinement using various layout algorithms.

Parameters:

• data (pd.DataFrame) – Input DataFrame containing the high-dimensional data.

• sim_features (list[str]) – List of feature column names to use for similarity computation.

• dr_method (str) – Dimensionality reduction method. Currently only “UMAP” is supported. Default is “UMAP”.

• dr_param_n_neigbors (int) – Number of neighbors parameter for the DR method. Default is 15.

• graph_method (str) – Graph construction method. Either “DR” (use DR graph) or “KNN” (use KNN graph based on feature similarity). Default is “DR”.

• community_resolutions (list[float] | None) – List of resolution parameters for community detection. If None, automatic resolutions are computed based on the minimum and maximum edge weights. Default is None.

• community_resolution_amount (int) – Number of resolution values to use if community_resolutions is None. Has no effect if community_resolutions is set. Default is 3.

• layout_method (str) – Layout method for position refinement. Options: “MDS” (Multidimensional Scaling), “KK” (Kamada-Kawai), “FR” (Fruchterman-Reingold). Default is “MDS”.

• boundary_neighbors (bool) – Whether to include boundary edges in layout computation. Default is False.

• layout_params (list[int] | None) – Parameter values for the layout method. Iterations for FR, balance factors for MDS/KK. If None, defaults are used ([1, 10, 100, 1000] for FR, [0.2, 0.4, 0.6, 0.8, 1.0] for KK/MDS). Default is None.

• compute_metrics (bool) – Whether to compute evaluation metrics for embeddings. Default is True.

• verbose (bool) – Whether to print detailed progress information. Default is False.

Returns:

List of EmbeddingState objects containing the processed embeddings with different community resolutions and layout parameters.

Return type:

list[EmbeddingState]

Raises:

ValueError – If unsupported methods are specified or invalid parameters are provided.