Data I/O

Read summary statistics produced by the sum_stat package.

The sum_stat package stores measurements in two formats:

  • HDF5 — LS10/BGS two-point functions and stellar mass functions. See Data Formats for the full schema.

  • FITS binary tables — GAMA and COSMOS stellar mass / luminosity functions.

All angular and projected distances are stored in physical Mpc by sum_stat. This module converts everything to h-units (Mpc/h) on the way out, using the Hubble constant stored in the file’s cosmology/H0 sub-group.

Conversion rules

  • Distances: r_h    = r_Mpc * h

  • Volumes: V_h    = V_Mpc * h^3

  • Number densities: phi_h  = phi_Mpc / h^3

  • Covariances scale as the square of the primary quantity.

Examples

Read a single-stat projected correlation function file:

reader = SumStatReader.from_hdf5(
    "/path/to/sum_stat/data/twopcf/"
    "LS10_VLIM_ANY_Mstar10.0-12.0_z0.05-0.18-wp-pimax100-sys-comb.h5"
)
data = reader.wp()
rp   = data["rp"]     # (N,) array, Mpc/h
wp   = data["wp"]     # (N,) array, Mpc/h
cov  = data["cov"]    # (N, N) array, (Mpc/h)^2

Read a joint SMF+wp+ESD file:

reader = SumStatReader.from_hdf5(
    "/path/to/sum_stat/data/BGS_Mstar10.00/"
    "joint_smf_wprp_deltasigma-sys-comb.h5"
)
jt = reader.joint()
# jt["data_vector"], jt["cov"] — full joint data vector and covariance
# jt["n_bins_smf"], jt["n_bins_wp"], jt["n_bins_ds"]

Read a GAMA stellar mass function:

reader = SumStatReader.from_fits(
    "/path/to/sum_stat/data/GAMA/gama_smf_z0.060_0.100.fits"
)
data = reader.smf()

References

class hod_mod.data_io.sum_stat_reader.SumStatReader(path: str, fmt: str, _cache: dict)[source]

Bases: object

Unified reader for sum_stat HDF5 and FITS measurement files.

Do not instantiate directly; use the class methods:

  • from_hdf5() — for HDF5 files from the twopcf, lf_smf, mocks, or BGS_Mstar* directories.

  • from_fits() — for FITS files from the GAMA or COSMOS directories.

attrs() → dict[source]

File-level attributes (creation date, version).

esd() → dict[source]

Return the excess surface mass density ΔΣ(R).

Returns:

  • dict with keys

  • * rp — projected separation bin centres, Mpc/h

  • * delta_sigma — ΔΣ(R) in M_sun pc⁻²

  • * cov — covariance matrix, (M_sun pc⁻²)²

  • * attrs — raw HDF5 group attributes

classmethod from_fits(path: str) → SumStatReader[source]

Open a GAMA or COSMOS FITS SMF/LF file.

The FITS file must have an HDU-1 binary table with at least the columns log10mstar, phi, and phi_err. Units are assumed to be Mpc (not Mpc/h); no h-conversion is applied because GAMA and COSMOS files do not embed cosmological parameters. Pass the \(h\) value explicitly when calling smf() if needed.

Parameters:

path (str) – Absolute or relative path to the FITS file.

Returns:

SumStatReader

classmethod from_hdf5(path: str) → SumStatReader[source]

Open a sum_stat HDF5 file.

Parameters:

path (str) – Absolute or relative path to the HDF5 file.

Returns:

SumStatReader

h() → float | None[source]

Hubble constant h = H0/100 embedded in the file (HDF5 only).

joint() → dict[source]

Return the full joint data vector and covariance matrix.

The joint data vector has layout [φ_SMF (n_smf), w_p (n_wp), ΔΣ (n_ds)].

Returns:

  • dict with keys

  • * data_vector — (n_total,) concatenated data vector

  • * cov — (n_total, n_total) joint covariance

  • * err_jackknife — sqrt(diag(cov))

  • * mstar_centres — log10(M*/M_sun) bin centres for SMF section

  • * rp_centres — r_p bin centres [Mpc/h] for wp and ΔΣ sections

  • * n_bins_smf, n_bins_wp, n_bins_ds — section lengths

  • * attrs — raw HDF5 group attributes

joint_bgs(probes: tuple = ('wp', 'esd_hsc', 'esd_des')) → dict[source]

Extract a multi-probe sub-data-vector from the new BGS joint format.

Selects the requested probes from the full 286-element joint data vector and returns the corresponding sub-block of the joint covariance matrix, with h-unit conversion applied:

  • wp — w_p multiplied by h (Mpc → Mpc/h); covariance × h²

  • esd_* — ΔΣ left as M_sun pc⁻² (invariant); covariance unchanged

  • Cross-terms — scaled by h¹ (WP axis) × 1 (ESD axis) = h

Parameters:

probes (tuple of str) – Any subset of ('smf', 'wp', 'esd_hsc', 'esd_des', 'esd_kids', 'wtheta', 'knn'). Default: ('wp', 'esd_hsc', 'esd_des').

Returns:

  • dict with keys

  • * data_vector — (n_sel,) sub-vector in h-units

  • * cov — (n_sel, n_sel) sub-block covariance in h-units

  • * rp_wp — (30,) r_p bin centres for WP [Mpc/h] (if wp requested)

  • * rp_esd — (30,) r_p bin centres for ESD [Mpc/h] (if any esd requested)

  • * slices_out — {probe: slice} mapping probe name to its position – in the returned data_vector

  • * h — embedded Hubble constant h = H0/100

  • * attrs — raw HDF5 group attributes

list_groups() → list[source]

List available statistic types in this file.

number_density() → dict[source]

Return the galaxy number density n of the sample.

Returns:

  • dict with keys

  • * n — number density in h³ Mpc⁻³

  • * n_err — uncertainty in h³ Mpc⁻³

  • * cov — (1, 1) variance in (h³ Mpc⁻³)²

  • * estimator — 'sum(w_i / Vmax_i)'

  • * attrs — raw HDF5 group attributes

smf(h: float | None = None) → dict[source]

Return the stellar mass function Φ(M*).

For FITS files (GAMA, COSMOS) the covariance matrix is not stored; cov will be a diagonal matrix built from phi_err.

Parameters:

h (float, optional) – Hubble constant h = H0/100 for Mpc → Mpc/h conversion of number densities. Only used for FITS files; HDF5 files carry h internally.

Returns:

  • dict with keys

  • * log10mstar — log10(M*/M_sun) bin centres

  • * phi — Φ(M*) in h³ Mpc⁻³ dex⁻¹

  • * phi_err — uncertainty

  • * cov — covariance matrix, (h³ Mpc⁻³ dex⁻¹)²

wp() → dict[source]

Return the projected correlation function w_p(r_p).

Returns:

  • dict with keys

  • * rp — projected separation bin centres, Mpc/h

  • * wp — projected correlation function, Mpc/h

  • * cov — covariance matrix, (Mpc/h)²

  • * pi_max — line-of-sight integration limit, Mpc/h

  • * estimator — 'landy-szalay' or similar

  • * attrs — raw HDF5 group attributes