petitRADTRANS.sbi.calibration ============================= .. py:module:: petitRADTRANS.sbi.calibration .. autoapi-nested-parse:: Calibration and posterior-predictive reporting for amortized SBI. Classes ------- .. autoapisummary:: petitRADTRANS.sbi.calibration.CoverageLevelReport petitRADTRANS.sbi.calibration.SimulationBasedCalibrationReport petitRADTRANS.sbi.calibration.PosteriorPredictiveReport petitRADTRANS.sbi.calibration.LocalSensitivityPointReport petitRADTRANS.sbi.calibration.LocalSensitivityReport Functions --------- .. autoapisummary:: petitRADTRANS.sbi.calibration.generate_local_sensitivity_report petitRADTRANS.sbi.calibration.local_sensitivity_report_to_payload petitRADTRANS.sbi.calibration.central_credible_interval petitRADTRANS.sbi.calibration.empirical_interval_coverage petitRADTRANS.sbi.calibration.simulation_based_calibration_report petitRADTRANS.sbi.calibration.generate_sbc_report petitRADTRANS.sbi.calibration.posterior_predictive_report_from_samples petitRADTRANS.sbi.calibration.generate_posterior_predictive_report Module Contents --------------- .. py:class:: CoverageLevelReport Coverage summary for one nominal interval level. Attributes ---------- level: Nominal central credible-interval level. empirical_coverage: Measured coverage fraction over the evaluated cases. absolute_error: Absolute difference between empirical and nominal coverage. .. py:attribute:: level :type: float .. py:attribute:: empirical_coverage :type: float .. py:attribute:: absolute_error :type: float .. py:class:: SimulationBasedCalibrationReport Rank-based SBC summary over a held-out set of observations. Attributes ---------- ranks: Integer rank of the ground-truth parameter within posterior samples for each held-out case and parameter dimension. rank_histogram_counts: Per-parameter SBC histogram counts. posterior_means: Posterior mean for each held-out case. truths: Ground-truth parameters paired with each posterior sample set. coverages: Coverage summaries for the requested nominal interval levels. mean_rank: Mean empirical rank for each parameter dimension. normalized_mean_rank_error: Absolute mean-rank error normalized by the expected average rank. metadata: Auxiliary run metadata such as split name and number of posterior draws. .. py:attribute:: ranks :type: numpy.ndarray .. py:attribute:: rank_histogram_counts :type: numpy.ndarray .. py:attribute:: posterior_means :type: numpy.ndarray .. py:attribute:: truths :type: numpy.ndarray .. py:attribute:: coverages :type: tuple[CoverageLevelReport, Ellipsis] .. py:attribute:: mean_rank :type: numpy.ndarray .. py:attribute:: normalized_mean_rank_error :type: numpy.ndarray .. py:attribute:: metadata :type: Mapping[str, Any] .. py:class:: PosteriorPredictiveReport Aggregate posterior-predictive summaries for held-out observations. Attributes ---------- observed_values: Observed values kept on the original observation scale. predictive_mean: Posterior-predictive mean curves or vectors. predictive_std: Posterior-predictive standard deviations. interval_lower, interval_upper: Central predictive interval bounds for the requested level. interval_coverage: Per-dataset fraction of observed points covered by the predictive interval. mean_absolute_error: Mean absolute deviation between predictive mean and observed values. metadata: Auxiliary metadata such as split name, number of cases, and parameter space used during predictive generation. .. py:attribute:: observed_values :type: Mapping[str, numpy.ndarray] .. py:attribute:: predictive_mean :type: Mapping[str, numpy.ndarray] .. py:attribute:: predictive_std :type: Mapping[str, numpy.ndarray] .. py:attribute:: interval_lower :type: Mapping[str, numpy.ndarray] .. py:attribute:: interval_upper :type: Mapping[str, numpy.ndarray] .. py:attribute:: interval_coverage :type: Mapping[str, float] .. py:attribute:: mean_absolute_error :type: Mapping[str, float] .. py:attribute:: mean_absolute_error_sigma :type: Mapping[str, float] .. py:attribute:: median_interval_width_over_uncertainty :type: Mapping[str, float] .. py:attribute:: crps :type: Mapping[str, float] .. py:attribute:: metadata :type: Mapping[str, Any] .. py:class:: LocalSensitivityPointReport Local linear-identifiability summary around one representative point. Attributes ---------- label: Short human-readable label for the representative point. parameters: Physical parameter vector at which the Jacobian was evaluated. finite_difference_steps: Per-parameter finite-difference step sizes used during Jacobian construction. finite_difference_schemes: Per-parameter scheme labels such as ``'central'`` or ``'forward'``. whitened_jacobian: Observation Jacobian divided by observational uncertainty, with shape ``(n_observation_values, n_parameters)``. singular_values: Singular values of the whitened Jacobian. effective_rank: Number of singular values larger than the configured relative cutoff. condition_number: Condition number inferred from the singular spectrum. fisher_matrix: Approximate Fisher information matrix ``J^T J``. fisher_covariance: Damped pseudo-inverse of the Fisher matrix. fisher_correlation: Correlation matrix derived from the approximate Fisher covariance. parameter_sensitivity_norm: Per-parameter square-root Fisher diagonal. local_sigma: Approximate local standard deviation from the Fisher covariance. metadata: Auxiliary diagnostics such as the ridge term and failed columns. .. py:attribute:: label :type: str .. py:attribute:: parameters :type: numpy.ndarray .. py:attribute:: finite_difference_steps :type: numpy.ndarray .. py:attribute:: finite_difference_schemes :type: tuple[str, Ellipsis] .. py:attribute:: whitened_jacobian :type: numpy.ndarray .. py:attribute:: singular_values :type: numpy.ndarray .. py:attribute:: effective_rank :type: int .. py:attribute:: condition_number :type: float .. py:attribute:: fisher_matrix :type: numpy.ndarray .. py:attribute:: fisher_covariance :type: numpy.ndarray .. py:attribute:: fisher_correlation :type: numpy.ndarray .. py:attribute:: parameter_sensitivity_norm :type: numpy.ndarray .. py:attribute:: local_sigma :type: numpy.ndarray .. py:attribute:: metadata :type: Mapping[str, Any] .. py:class:: LocalSensitivityReport Aggregate local information-content diagnostics for one observation. Attributes ---------- parameter_names: Parameter ordering used throughout the report. posterior_mean: Posterior mean in physical parameter space. posterior_std: Posterior standard deviation in physical parameter space. posterior_median: Posterior median in physical parameter space. posterior_iqr: Posterior interquartile range in physical parameter space. representative_points: Local sensitivity summaries evaluated at representative posterior points such as the posterior mean and highest-density sample. aggregate_local_sigma: Median local Fisher sigma across representative points. aggregate_parameter_sensitivity_norm: Median parameter sensitivity norm across representative points. posterior_to_local_sigma_ratio: Ratio between posterior standard deviation and local Fisher sigma. parameter_diagnostics: Per-parameter heuristic summary separating weak data constraints from broader-than-local posterior structure. metadata: Auxiliary metadata such as quantile levels and observation slices. .. py:attribute:: parameter_names :type: tuple[str, Ellipsis] .. py:attribute:: posterior_mean :type: numpy.ndarray .. py:attribute:: posterior_std :type: numpy.ndarray .. py:attribute:: posterior_median :type: numpy.ndarray .. py:attribute:: posterior_iqr :type: numpy.ndarray .. py:attribute:: representative_points :type: tuple[LocalSensitivityPointReport, Ellipsis] .. py:attribute:: aggregate_local_sigma :type: numpy.ndarray .. py:attribute:: aggregate_parameter_sensitivity_norm :type: numpy.ndarray .. py:attribute:: posterior_to_local_sigma_ratio :type: numpy.ndarray .. py:attribute:: parameter_diagnostics :type: Mapping[str, Mapping[str, Any]] .. py:attribute:: metadata :type: Mapping[str, Any] .. py:function:: generate_local_sensitivity_report(task: petitRADTRANS.sbi.task.SBITask, posterior_samples: Any, observation_blocks: Sequence[Any], posterior_log_probabilities: Any = None, parameter_space: str = 'physical', simulator: petitRADTRANS.sbi.simulator.RuntimeSimulator | None = None, quantile_levels: Sequence[float] = (0.1, 0.5, 0.9), finite_difference_relative_step: float = 0.001, finite_difference_std_fraction: float = 0.1, finite_difference_absolute_floor: float = 1e-05, max_step_reduction_attempts: int = 6, svd_relative_tolerance: float = 0.001, posterior_underexploited_ratio_threshold: float = 1.5, weak_sensitivity_fraction_threshold: float = 0.15, seed: int | None = None) -> LocalSensitivityReport Diagnose local physical identifiability around representative posterior points. The report evaluates a deterministic simulator Jacobian at representative posterior points, whitens it by observational uncertainty, and derives a Fisher-style local covariance approximation for each point. .. py:function:: local_sensitivity_report_to_payload(report: LocalSensitivityReport) -> dict[str, Any] Convert a local sensitivity report into a JSON-serializable payload. .. py:function:: central_credible_interval(samples: Any, level: float = 0.8) -> tuple[numpy.ndarray, numpy.ndarray] Compute a central credible interval from posterior samples. Parameters ---------- samples: Posterior samples for one observation, with shape ``(n_samples,)`` or ``(n_samples, n_dim)``. level: Nominal central interval level between 0 and 1. Returns ------- tuple[np.ndarray, np.ndarray] Lower and upper quantile bounds for each inferred parameter dimension. The returned arrays are one-dimensional even for scalar posteriors. .. py:function:: empirical_interval_coverage(samples: Any, truths: Any, level: float = 0.8) -> float Measure empirical central-interval coverage over a batch of posteriors. Parameters ---------- samples: Posterior draws for one or more held-out cases. Expected shape is ``(n_cases, n_draws, n_dim)`` or a shape that can be promoted to it. truths: Ground-truth parameters matched to each held-out case. level: Nominal central interval level between 0 and 1. Returns ------- float Fraction of truth values falling inside the requested central interval. .. py:function:: simulation_based_calibration_report(samples: Any, truths: Any, levels: Sequence[float] = (0.5, 0.8, 0.95), metadata: Mapping[str, Any] | None = None) -> SimulationBasedCalibrationReport Build an SBC report from posterior draws and matched ground truths. Parameters ---------- samples: Posterior draws for held-out cases with shape ``(n_cases, n_draws, n_dim)`` or a promotable equivalent. truths: Ground-truth parameters paired with each held-out case. levels: Coverage levels to summarize alongside the rank statistics. metadata: Optional metadata attached to the returned report. Returns ------- SimulationBasedCalibrationReport SBC summary containing ranks, histograms, posterior means, and coverage summaries. .. py:function:: generate_sbc_report(posterior: Any, dataset_reader: petitRADTRANS.sbi.dataset.NormalizedObservationDatasetReader, split: petitRADTRANS.sbi.dataset.DatasetSplit = DatasetSplit.TEST, n_posterior_samples: int = 256, batch_size: int = 32, parameter_space: str | None = None, levels: Sequence[float] = (0.5, 0.8, 0.95), max_cases: int | None = None, seed: int | None = None, data_parallel: bool | None = None) -> SimulationBasedCalibrationReport Run SBC over a dataset reader using normalized observation batches. Parameters ---------- posterior: Trained posterior estimator exposing ``encode_observation`` and ``sample_posterior``. dataset_reader: Reader yielding normalized held-out observations and matched parameter values. split: Dataset split used for the SBC evaluation. n_posterior_samples: Number of posterior draws generated per held-out case. batch_size: Reader batch size used during report generation. parameter_space: Optional parameter space override. Defaults to the posterior's own configured parameter space. levels: Coverage levels summarized in the returned report. max_cases: Optional cap on the number of held-out observations evaluated. seed: Optional base seed used to generate reproducible posterior draws. Returns ------- SimulationBasedCalibrationReport SBC summary computed from the requested dataset split. .. py:function:: posterior_predictive_report_from_samples(task: petitRADTRANS.sbi.task.SBITask, posterior_samples: Any, observation_blocks: Sequence[Any], parameter_space: str = 'physical', interval_level: float = 0.9, simulator: petitRADTRANS.sbi.simulator.RuntimeSimulator | None = None, seed: int | None = None, n_predictive_forward_model_samples: int | None = None) -> PosteriorPredictiveReport Summarize posterior-predictive draws for one observed system. Parameters ---------- task: SBI task used to map posterior samples back into physical parameter space and construct the simulator. posterior_samples: Posterior draws in the specified parameter space. observation_blocks: Original user-facing observation blocks. Reported observations remain on this raw scale. parameter_space: Coordinate system in which ``posterior_samples`` are expressed. interval_level: Central predictive interval level used for reporting. simulator: Optional simulator override. When omitted a new runtime-backed simulator is constructed from ``task``. seed: Optional seed used when instantiating the simulator. n_predictive_forward_model_samples: Number of posterior draws passed through the forward model to generate the predictive distribution. When ``None`` all ``posterior_samples`` are used. Setting this to a small value (e.g. 50–200) dramatically reduces the number of expensive petitRADTRANS calls without materially affecting the predictive summary. Returns ------- PosteriorPredictiveReport Posterior-predictive summary for the supplied observation. Notes ----- The observations in the returned report remain on the original observation scale even when the posterior itself was trained on normalized inputs. .. py:function:: generate_posterior_predictive_report(task: petitRADTRANS.sbi.task.SBITask, posterior: Any, dataset_reader: petitRADTRANS.sbi.dataset.NormalizedObservationDatasetReader, split: petitRADTRANS.sbi.dataset.DatasetSplit = DatasetSplit.TEST, n_posterior_samples: int = 256, interval_level: float = 0.9, max_cases: int | None = None, seed: int | None = None, simulator: petitRADTRANS.sbi.simulator.RuntimeSimulator | None = None, n_predictive_forward_model_samples: int | None = None, checkpoint_directory: str | pathlib.Path | None = None, data_parallel: bool | None = None) -> PosteriorPredictiveReport Generate posterior-predictive summaries for a held-out dataset split. Parameters ---------- task: SBI task defining parameter transforms and the simulator configuration. posterior: Trained posterior estimator used to sample held-out predictive draws. dataset_reader: Reader providing normalized observations and preprocessing metadata. split: Held-out split used for the predictive report. n_posterior_samples: Number of posterior draws generated per held-out observation. interval_level: Central predictive interval level reported for each dataset. max_cases: Optional cap on the number of held-out observations evaluated. seed: Optional base seed used to make predictive sampling reproducible. simulator: Optional runtime simulator override. n_predictive_forward_model_samples: Number of posterior draws passed through the forward model per held-out case. When ``None`` all ``n_posterior_samples`` draws are forwarded. Subsampling here is the primary lever for keeping the total number of petitRADTRANS calls to a manageable level when evaluating many held-out cases. checkpoint_directory: Optional directory for per-case checkpoints. When provided, each completed case is written to disk as a compressed ``.npz`` file and skipped on resume. This makes the expensive forward-model loop restartable after interruption. Returns ------- PosteriorPredictiveReport Aggregate posterior-predictive summary over the requested split. Notes ----- Observations are normalized internally for posterior encoding but compared on the original observation scale in the returned report.