Evaluation¶

Metrics¶

evaluation ¶

Design quality evaluation metrics.

DesignEvaluation `dataclass` ¶

DesignEvaluation(
    n_runs, n_params, dof, efficiencies, power, vif, fds
)

Quality report card of a design (result of :func:evaluate).

Aggregates efficiencies, power, VIF and FDS into one object. Print or call :meth:summary for a human-readable card; use :meth:to_dict to persist.

Attributes:

Name	Type	Description
`n_runs, n_params, dof`	`int`	Number of runs, model parameters and residual degrees of freedom.
`efficiencies`	`dict`	Output of :func:`efficiencies`.
`power`	`Series`	Output of :func:`power_analysis` (empty for a saturated design).
`vif`	`Series`	Output of :func:`vif`.
`fds`	`DataFrame`	Output of :func:`fds_data`.

Examples:

>>> import doekit as ed
>>> ev = ed.evaluate(ed.full_factorial(3), n_region=500, seed=0)
>>> ev.d_efficiency > 90
True

a_efficiency `property` ¶

a_efficiency

A-efficiency (%).

d_efficiency `property` ¶

d_efficiency

D-efficiency (%).

g_efficiency `property` ¶

g_efficiency

G-efficiency (%).

from_dict `classmethod` ¶

from_dict(d)

Rebuild a :class:DesignEvaluation from :meth:to_dict output.

summary ¶

summary()

Return a human-readable multi-line summary of the evaluation.

to_dict ¶

to_dict()

Serialize to a JSON-safe dict (schema: doekit.DesignEvaluation/1).

alias_matrix ¶

alias_matrix(design, model=None, alias_model=None)

Alias matrix relating primary-model bias to omitted terms.

Quantifies the bias of the primary-model coefficients (X1) induced by potential omitted terms (X2). If alias_model is None, the two-factor interactions not already in the primary model are used.

Formulas

A = (X1'X1)^-1 X1'X2
E[beta_hat_1] = beta_1 + A beta_2

Parameters:

Name	Type	Description	Default
`design`	`Design`	The design to evaluate.	required
`model`	`Model`	Primary model `X1`; resolved from the design if omitted.	`None`
`alias_model`	`Model`	Model of potentially omitted terms `X2`; defaults to the 2-factor interactions absent from the primary model.	`None`

Returns:

Type	Description
`DataFrame`	Alias matrix indexed by primary terms and columned by alias terms.

Examples:

>>> import doekit as ed
>>> A = ed.alias_matrix(ed.fractional_factorial(3, generators=["C=AB"]))
>>> A.shape[0] > 0 and A.shape[1] > 0
True

efficiencies ¶

efficiencies(
    design,
    model=None,
    region=None,
    n_region=20000,
    seed=None,
)

D/A/G efficiencies (%) and scaled-prediction-variance statistics.

Computed in coded units (+/-1). Values near 100% indicate a design close to the theoretical optimum for the fitted model. Mean SPV is the I-optimality criterion (smaller is better).

Formulas

D-eff = 100 * det(X'X)^(1/p) / N (100% for an orthogonal design).
A-eff = 100 * p / (N * tr((X'X)^-1)).
G-eff = 100 * p / max(SPV) over the region, with SPV(x) = N * x'(X'X)^-1 x (general equivalence theorem: max SPV >= p).
I = mean SPV over the region; also report min/max SPV.

Parameters:

Name	Type	Description	Default
`design`	`Design`	The design to evaluate.	required
`model`	`Model`	Model whose matrix is scored; resolved from the design if omitted.	`None`
`region`	`DataFrame`	Precomputed region sample; a fresh Monte Carlo sample is drawn if omitted.	`None`
`n_region`	`int`	Region sample size when `region` is not provided.	`20000`
`seed`	`int`	Seed for the region sampling.	`None`

Returns:

Type	Description
`dict`	Keys `n_runs`, `n_params`, `dof`, `rank_deficient`, `D_efficiency`, `A_efficiency`, `G_efficiency` and `spv_min`/`spv_mean`/`spv_max`. Efficiencies are `nan` when the design is rank deficient.

Notes

D/A are clamped to <= 100% (Hadamard / AM-HM). G may slightly exceed 100% under Monte Carlo region sampling and is likewise clamped.

Examples:

>>> import doekit as ed
>>> eff = ed.efficiencies(ed.full_factorial(3), seed=0)
>>> eff["D_efficiency"] > 90 and not eff["rank_deficient"]
True

evaluate ¶

evaluate(
    design,
    model=None,
    effect_size=1.0,
    sigma=1.0,
    alpha=0.05,
    region=None,
    n_region=20000,
    seed=None,
    report=None,
)

Evaluate a design end-to-end and return a :class:DesignEvaluation.

Combines :func:efficiencies, :func:power_analysis, :func:vif and :func:fds_data. Metrics use coded factor units when Design.factors is set. Saturated designs yield an empty power series instead of raising.

Parameters:

Name	Type	Description	Default
`design`	`Design`	The design to evaluate.	required
`model`	`Model`	Model to score; resolved from the design if omitted.	`None`
`effect_size`	`float or array - like`	Anticipated effect size for the power analysis (same units as the response; non-centrality uses `effect_size / sigma`).	`1.0`
`sigma`	`float`	Noise standard deviation.	`1.0`
`alpha`	`float`	Two-sided significance level for the power analysis.	`0.05`
`region`	`DataFrame`	Precomputed region sample shared by efficiency and FDS.	`None`
`n_region`	`int`	Region sample size when `region` is not provided.	`20000`
`seed`	`int`	Seed for the region sampling.	`None`
`report`	`(None, bool, str, Path or dict)`	If not `None`, a design-quality HTML report is generated and its path is stored in `result.report_path`.	`None`

Returns:

Type	Description
`DesignEvaluation`	The full quality report card.

Notes

See the Formulas sections of :func:efficiencies, :func:power_analysis, :func:vif and :func:fds_data for the underlying equations.

Examples:

>>> import doekit as ed
>>> ev = ed.evaluate(ed.plackett_burman(6), n_region=500, seed=0)
>>> print(ev.d_efficiency > 80)
True

fds_data ¶

fds_data(
    design,
    model=None,
    region=None,
    n_region=20000,
    seed=None,
)

Data for the Fraction of Design Space (FDS) plot.

Sorts SPV over a Monte Carlo sample of the experimental region. A flat, low curve means uniform and precise prediction across the whole region (the desirable case). Rank-deficient designs return spv as nan.

Formulas

SPV(x) = N * x'(X'X)^-1 x, sorted ascending; fraction is the empirical CDF abscissa (i - 0.5) / m for i = 1..m.

Parameters:

Name	Type	Description	Default
`design`	`Design`	The design to evaluate.	required
`model`	`Model`	Model whose matrix is scored; resolved from the design if omitted.	`None`
`region`	`DataFrame`	Precomputed region sample; a fresh sample is drawn if omitted.	`None`
`n_region`	`int`	Region sample size when `region` is not provided.	`20000`
`seed`	`int`	Seed for the region sampling.	`None`

Returns:

Type	Description
`DataFrame`	Columns `fraction` (0→1) and `spv`.

Examples:

>>> import doekit as ed
>>> fds = ed.fds_data(ed.full_factorial(3), n_region=500, seed=0)
>>> list(fds.columns) == ["fraction", "spv"]
True

power_analysis ¶

power_analysis(
    design,
    model=None,
    effect_size=1.0,
    sigma=1.0,
    alpha=0.05,
)

Per-coefficient power to detect an effect of size effect_size.

Uses a two-sided t-test at level alpha with N - p residual degrees of freedom. Requires an unsaturated design (N > p).

Formulas

se_i = sigma * sqrt((X'X)^-1_ii)
non-centrality delta_i = effect_size / se_i
power from the non-central t distribution (normal fallback in the far tails when SciPy's nct is numerically unstable)

Parameters:

Name	Type	Description	Default
`design`	`Design`	The design to evaluate.	required
`model`	`Model`	Model whose terms are powered; resolved from the design if omitted.	`None`
`effect_size`	`float or array - like`	Anticipated effect size (scalar shared by all terms, or one per term).	`1.0`
`sigma`	`float`	Noise standard deviation.	`1.0`
`alpha`	`float`	Two-sided significance level.	`0.05`

Returns:

Type	Description
`Series`	Power per model term (index = term names), values in `[0, 1]`.

Raises:

Type	Description
`ValueError`	If `N <= p` (a saturated design has no residual degrees of freedom).

Examples:

>>> import doekit as ed
>>> pw = ed.power_analysis(ed.full_factorial(3), effect_size=2.0, sigma=1.0)
>>> float(pw.min()) > 0.8
True

vif ¶

vif(design, model=None)

Variance inflation factors of the model terms (intercept excluded).

VIF = 1 indicates orthogonality; values > 5-10 signal serious multicollinearity. Perfect collinearity yields inf (not a misleading finite value from a pseudoinverse).

Formulas

VIF_j = 1 / (1 - R2_j) where R2_j comes from regressing column j on the other non-constant columns (after centering/scaling).

Parameters:

Name	Type	Description	Default
`design`	`Design`	The design to evaluate.	required
`model`	`Model`	Model whose terms are checked; resolved from the design if omitted.	`None`

Returns:

Type	Description
`Series`	VIF per non-intercept term (empty if the model has no such terms).

Examples:

>>> import doekit as ed
>>> v = ed.vif(ed.full_factorial(3))
>>> float(v.max()) < 1.01
True

Plotting¶

figures_mpl ¶

Visualization helpers (matplotlib optional, lazy import).

Install with pip install doekit[plot].

acquisition_plot ¶

acquisition_plot(
    surrogate,
    best=None,
    goal="max",
    kind="ei",
    x=None,
    y=None,
    bounds=None,
    at=None,
    proposed=None,
    resolution=60,
    ax=None,
    cmap="magma",
    kappa=2.0,
    xi=0.01,
    colorbar=True,
)

Contour of an acquisition surface: where the loop wants to sample next.

Parameters:

Name	Type	Description	Default
`surrogate`	`Surrogate`		required
`best`	`float`	Best objective so far (defaults to the best of the training data).	`None`
`goal`	`('max', 'min')`		`"max"`
`kind`	`('ei', 'ucb', 'pi')`		`"ei"`
`x`		As in :func:`surrogate_surface`.	`None`
`y`		As in :func:`surrogate_surface`.	`None`
`bounds`		As in :func:`surrogate_surface`.	`None`
`at`		As in :func:`surrogate_surface`.	`None`
`resolution`		As in :func:`surrogate_surface`.	`None`
`ax`		As in :func:`surrogate_surface`.	`None`
`proposed`	`DataFrame`	Proposed runs to overlay.	`None`
`kappa`	`float`	Acquisition exploration parameters.	`2.0`
`xi`	`float`	Acquisition exploration parameters.	`2.0`
`colorbar`	`bool`		`True`

Returns:

Type	Description
`Axes`

alias_heatmap ¶

alias_heatmap(alias_df, ax=None)

Heatmap of the alias matrix (output of alias_matrix).

Parameters:

Name	Type	Description	Default
`alias_df`	`DataFrame`	The alias matrix, indexed by primary terms and columned by alias terms.	required
`ax`	`Axes`	Axes to draw on; a new figure is created if omitted.	`None`

Returns:

Type	Description
`Axes`	The axes drawn on.

calibration_plot ¶

calibration_plot(
    surrogate, levels=(0.5, 0.8, 0.95), ax=None, label=None
)

LOO interval coverage vs. nominal — the moat plot (is sigma honest?).

Points on the diagonal mean the surrogate's uncertainty is well-calibrated; below the diagonal means it is over-confident.

Returns:

Type	Description
`Axes`

convergence_plot ¶

convergence_plot(
    history, ax=None, label=None, optimum=None, marker="o"
)

Best-so-far y* per generation (are we approaching the optimum?).

Parameters:

Name	Type	Description	Default
`history`	`array - like`	Best-so-far value per generation (or a 2-column `(gen, value)`).	required
`ax`	`Axes`		`None`
`label`	`str`	Legend label (overlay multiple strategies).	`None`
`optimum`	`float`	Known optimum, drawn as a dashed reference line.	`None`
`marker`	`str`		`"o"`

Returns:

Type	Description
`Axes`

correlation_plot ¶

correlation_plot(design, model=None, ax=None)

Correlation map of the model-matrix columns (aliasing).

Parameters:

Name	Type	Description	Default
`design`	`Design`	The design whose model matrix is correlated.	required
`model`	`Model`	Model to build the matrix; taken from `design.model` if omitted.	`None`
`ax`	`Axes`	Axes to draw on; a new figure is created if omitted.	`None`

Returns:

Type	Description
`Axes`	The axes drawn on.

effects_plot ¶

effects_plot(effects, labels=None, ax=None)

Bar chart of signed effects, sorted by |effect|.

Parameters:

Name	Type	Description	Default
`effects`	`array - like`	Estimated effects.	required
`labels`	`sequence of str`	One label per effect; defaults to `e1..em`.	`None`
`ax`	`Axes`	Axes to draw on; a new figure is created if omitted.	`None`

Returns:

Type	Description
`Axes`	The axes drawn on.

fds_plot ¶

fds_plot(
    design,
    model=None,
    ax=None,
    label=None,
    surrogate=None,
    n_region=2000,
    seed=None,
    **fds_kwargs,
)

Fraction of Design Space plot: SPV vs. region fraction (0->1).

A low, flat curve indicates uniform and precise prediction across the whole region. When a surrogate is passed, the curve instead shows the sorted predictive standard deviation sigma(x) over a region cover — the optimize-intent analogue of SPV (how uniform the surrogate's uncertainty is).

Formulas

Design mode: SPV(x) = N * x'(X'X)^-1 x sorted over the region sample. Surrogate mode: sigma(x) from surrogate.predict sorted likewise.

Parameters:

Name	Type	Description	Default
`design`	`Design or DataFrame`	A :class:`Design`, or a `DataFrame` already produced by :func:`doekit.fds_data`.	required
`model`	`Model`	Model to build the matrix (only used when `design` is a `Design`).	`None`
`ax`	`Axes`	Axes to draw on; accepts repeated calls to overlay several designs.	`None`
`label`	`str`	Legend label for the curve.	`None`
`surrogate`	`Surrogate`	If given, plot sorted `sigma(x)` from the surrogate over a sampled region cover instead of the design SPV.	`None`
`n_region`	`int`	Number of region points sampled for the surrogate curve.	`2000`
`seed`	`int`	RNG seed for the surrogate region cover.	`None`
`**fds_kwargs`		Extra keyword arguments forwarded to :func:`doekit.fds_data`.	`{}`

Returns:

Type	Description
`Axes`	The axes drawn on.

Examples:

>>> import doekit as ed
>>> ax = ed.presentation.render.figures_mpl.fds_plot(
...     ed.full_factorial(3), seed=0)
>>> ax.get_xlabel()
'Fraction of design region'

half_normal_plot ¶

half_normal_plot(
    effects, labels=None, ax=None, annotate=True
)

Half-normal plot of effects: points off the line are significant.

Daniel plot for screening: inactive effects follow the half-normal reference; departures indicate active factors.

Parameters:

Name	Type	Description	Default
`effects`	`array - like`	Estimated effects (or coefficients).	required
`labels`	`sequence of str`	One label per effect; defaults to `e1..em`.	`None`
`ax`	`Axes`	Axes to draw on; a new figure is created if omitted.	`None`
`annotate`	`bool`	Whether to annotate each point with its label.	`True`

Returns:

Type	Description
`Axes`	The axes drawn on.

Examples:

>>> import doekit as ed
>>> ax = ed.presentation.render.figures_mpl.half_normal_plot([0.1, 3.0, -0.2])
>>> ax.get_ylabel()
'|Effect|'

pareto_plot ¶

pareto_plot(
    Y,
    goals=None,
    columns=None,
    x=None,
    y=None,
    ax=None,
    annotate=False,
)

Scatter of two objectives, highlighting the non-dominated (Pareto) front.

Parameters:

Name	Type	Description	Default
`Y`	`DataFrame or ndarray`	Objective values (n x m).	required
`goals`	`dict`	`{column: "max"\|"min"}`.	`None`
`columns`	`sequence of str`	Objective names (taken from a DataFrame when available).	`None`
`x`	`str or int`	Which two objectives to plot.	`None`
`y`	`str or int`	Which two objectives to plot.	`None`
`ax`	`Axes`		`None`
`annotate`	`bool`		`False`

Returns:

Type	Description
`Axes`

parity_plot ¶

parity_plot(
    surrogate, design=None, y=None, ax=None, loo=False
)

Predicted vs. observed with sigma error bars — audit the surrogate.

Parameters:

Name	Type	Description	Default
`surrogate`	`Surrogate`		required
`design`	`optional`	Data to evaluate; defaults to the surrogate's training data.	`None`
`y`	`optional`	Data to evaluate; defaults to the surrogate's training data.	`None`
`ax`	`Axes`		`None`
`loo`	`bool`	Reserved (in-sample parity by default).	`False`

Returns:

Type	Description
`Axes`

power_plot ¶

power_plot(power, ax=None, alpha_ref=0.8)

Bar chart of per-term power (output of power_analysis).

Parameters:

Name	Type	Description	Default
`power`	`Series or array - like`	Power per model term.	required
`ax`	`Axes`	Axes to draw on; a new figure is created if omitted.	`None`
`alpha_ref`	`float`	Reference power level marked with a dashed line.	`0.8`

Returns:

Type	Description
`Axes`	The axes drawn on.

Examples:

>>> import doekit as ed
>>> ev = ed.evaluate(ed.full_factorial(3), seed=0)
>>> ax = ed.presentation.render.figures_mpl.power_plot(ev.power)
>>> ax.get_xlabel()
'Power'

slice_plot ¶

slice_plot(
    surrogate,
    factor=None,
    at=None,
    bounds=None,
    resolution=120,
    ax=None,
    label=None,
)

1D slice / partial dependence of mu(x) +/- sigma along one factor.

Parameters:

Name	Type	Description	Default
`surrogate`	`Surrogate`		required
`factor`	`str`	Factor to vary (first factor by default).	`None`
`at`	`dict`	Fixed values for the other factors (midpoints by default) — e.g. the optimum, to slice through it.	`None`
`bounds`	`dict`		`None`
`resolution`	`int`		`120`
`ax`	`Axes`		`None`
`label`	`str`		`None`

Returns:

Type	Description
`Axes`

surrogate_surface ¶

surrogate_surface(
    surrogate,
    x=None,
    y=None,
    bounds=None,
    at=None,
    measured=None,
    proposed=None,
    optimum=None,
    resolution=60,
    ax=None,
    cmap="viridis",
    colorbar=True,
)

Contour of the surrogate mean mu(x) over two factors.

Marks measured points, the current optimum, and any proposed runs — the "optimum vs. the non-optima" picture.

Parameters:

Name	Type	Description	Default
`surrogate`	`Surrogate`	A fitted surrogate (`predict` + `factor_names`).	required
`x`	`str`	Factor names for the two axes (first two factors by default).	`None`
`y`	`str`	Factor names for the two axes (first two factors by default).	`None`
`bounds`	`dict`	`name -> (low, high)` overrides.	`None`
`at`	`dict`	Fixed values for the other factors (defaults to their midpoint).	`None`
`measured`	`DataFrame`	Measured design points to overlay (factor columns).	`None`
`proposed`	`DataFrame`	Proposed next runs to overlay (e.g. `proposal.added.matrix`).	`None`
`optimum`	`mapping or array`	Point to star as the current best.	`None`
`resolution`	`int`	Grid resolution per axis.	`60`
`ax`	`Axes`		`None`
`cmap`	`str`		`"viridis"`
`colorbar`	`bool`		`True`

Returns:

Type	Description
`Axes`

Evaluation¶

Metrics¶

evaluation ¶

DesignEvaluation dataclass ¶

a_efficiency property ¶

d_efficiency property ¶

g_efficiency property ¶

from_dict classmethod ¶

summary ¶

to_dict ¶

alias_matrix ¶

efficiencies ¶

evaluate ¶

fds_data ¶

power_analysis ¶

vif ¶

Plotting¶

figures_mpl ¶

acquisition_plot ¶

alias_heatmap ¶

calibration_plot ¶

convergence_plot ¶

correlation_plot ¶

effects_plot ¶

fds_plot ¶

half_normal_plot ¶

pareto_plot ¶

parity_plot ¶

power_plot ¶

slice_plot ¶

surrogate_surface ¶

DesignEvaluation `dataclass` ¶

a_efficiency `property` ¶

d_efficiency `property` ¶

g_efficiency `property` ¶

from_dict `classmethod` ¶