API Reference / explain /

shapley.py

Exact Shapley attributions for TuiML's tree models.

Classes

TreeExplainer

class explain.shapley.TreeExplainer

Exact per-prediction Shapley values for tree models, in polynomial time.

Shapley values are the only attribution satisfying efficiency, symmetry, dummy and additivity simultaneously — which is why they are the standard against which other explanations are judged. Their definition sums over all 2^F feature subsets, so computing them exactly is normally hopeless. TreeSHAP exploits a tree's structure to get the exact answer anyway, in time polynomial in depth rather than exponential in features.
Constructor 
__init__(
    self,
    model: Any,
    background: Optional[np.ndarray] = None,
    feature_names: Optional[List[str]] = None,
)

Overview

  1. Push the background data through the tree to record what fraction of it
reaches each node.
  1. For a sample, walk every root-to-leaf path once, carrying the set of
features split on so far together with the proportion of subsets in which each is present or absent.
  1. At each leaf, credit every feature on the path with its marginal
contribution.

Theory

The Shapley value of feature j for prediction f(x) is

\phi_j = \sum_{S \subseteq F \setminus \{j\}} \frac{|S|! \ (|F| - |S| - 1)!}{|F|!} \left[ f_x(S \cup \{j\}) - f_x(S) \right]

where f_x(S) is the prediction with only the features in S known and the rest marginalised over the background. Evaluating that directly costs O(2^F); carrying the subset-proportion bookkeeping down the tree instead collapses it to O(L D^2) for L leaves and depth D.

Efficiency is the property that makes the output trustworthy and testable: for every sample,

\sum_j \phi_j + \mathbb{E}[f] = f(x)

The attributions add up to the prediction exactly. This implementation is verified against a brute-force enumeration of all subsets and agrees to machine precision.

Parameters

model
Algorithm
A fitted TuiML tree model: DecisionTreeClassifier, DecisionTreeRegressor, RandomForestClassifier or RandomForestRegressor.
background
np.ndarray of shape (n_background, n_features)
Data defining the expectation a feature is marginalised over. Required, deliberately: attributions are always relative to this background, so "why is this prediction high?" only means something once "high compared to what?" is answered. Passing the training set gives the usual reading; passing a subgroup asks why this prediction differs from that subgroup instead.
feature_names
list of str
Names for the report.

Attributes

expected_value_
np.ndarray
The model's mean output over the background, one entry per output.
n_features_
int
Number of features the model was fitted on.

Notes

Complexity. O(T L D^2) per sample for a forest of T trees, run in the shared C++ kernel tuiml._cpp_ext.shapley.tree_shap and parallel over samples. A forest costs the sum of its trees, so explaining a 500-tree forest is 500 times the work of explaining one.

This is the path-dependent variant. A feature is marginalised by following the background's branch proportions at each split, which respects the correlations the tree learned. The alternative interventional variant breaks those correlations deliberately and answers a subtly different causal question; the two disagree when features are correlated, and neither is wrong — they answer different questions.

Attribution is not causation. A large \phi_j says the model used that feature, not that the feature drives the outcome in the world. A model that leaks a proxy for the label will attribute confidently to it.

References

Lundberg2017
Lundberg, S. M., & Lee, S.-I. (2017). A Unified Approach to Interpreting Model Predictions. NeurIPS, 4765-4774. :arxiv:`1705.07874`
Lundberg2020
Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, J. M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N., & Lee, S.-I. (2020). From Local Explanations to Global Understanding with Explainable AI for Trees. Nature Machine Intelligence, 2(1), 56-67. :doi:`10.1038/s42256-019-0138-9`

See Also

permutation_importance

Model-agnostic and global; far cheaper, far coarser.
partial_dependence

How a feature moves the prediction on average, rather than per sample.
python 
>>> import numpy as np
>>> from tuiml.explain import TreeExplainer
>>> from tuiml.algorithms.trees import DecisionTreeRegressor
>>> rng = np.random.default_rng(0)
>>> X = rng.normal(size=(200, 4))
>>> y = 3.0 * X[:, 0] - 2.0 * X[:, 1] + rng.normal(0, 0.1, 200)
>>> model = DecisionTreeRegressor(max_depth=5).fit(X, y)
>>> explainer = TreeExplainer(model, background=X)
>>> result = explainer.explain(X[:10])
>>> result.values.shape
(10, 4)

The attributions reconstruct the prediction exactly, which is what makes

them worth trusting:

python 
>>> reconstructed = result.values.sum(axis=1) + explainer.expected_value_[0]
>>> bool(np.allclose(reconstructed, model.predict(X[:10])))
True

Methods

explain (self, X: np.ndarray) -> Explanation

Compute Shapley values for each sample.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Samples to explain.
Returns
explanation
Explanation
values has shape (n_samples, n_features) for single-output models and (n_samples, n_features, n_outputs) otherwise. base_value holds expected_value_.
__repr__ (self) -> str

Return a readable representation of the explainer.