_images/shap_diagram.png

SHAP (SHapley Additive exPlanations) is a unified approach to explain the output of any machine learning model. SHAP connects game theory with local explanations, uniting several previous methods and representing the only possible consistent and locally accurate additive feature attribution method based on expectations (see the SHAP NIPS paper for details).

Explainers

class shap.TreeExplainer(model, **kwargs)

Uses the Tree SHAP method to explain the output of ensemble tree models.

Tree SHAP is a fast and exact (assuming the trees capture the input feature depndencies) method to estimate SHAP values for tree models and ensembles of trees. It depends on fast C++ implementations either inside the externel model package or in the local compiled C extention.

approximate_tree_shap(tree, x, x_missing, phi, condition=0, condition_feature=0)

This is a simple approximation equivelent to the Saabas method.

It is actually slow because it is in python, but that’s fine right now since it is just used for benchmark comparisons with Saabas. It would need to be added to tree_shap.h as C++ if we wanted it to be high speed.

x_missing, condition, and condition_feature are currently not used

shap_values(X, tree_limit=-1, approximate=False)

Estimate the SHAP values for a set of samples.

X
: numpy.array, pandas.DataFrame or catboost.Pool (for catboost)
A matrix of samples (# samples x # features) on which to explain the model’s output.
tree_limit
: int
Limit the number of trees used by the model. By default -1 means no limit.
approximate
: bool
Run a faster approximate version of Tree SHAP (proposed by Saabas). Only supported for XGBoost models right now.

For models with a single output this returns a matrix of SHAP values (# samples x # features). Each row sums to the difference between the model output for that sample and the expected value of the model output (which is stored as expected_value attribute of the explainer). For models with vector outputs this returns a list of such matrices, one for each output.

class shap.KernelExplainer(model, data, link=<shap.common.IdentityLink object>, **kwargs)

Uses the Kernel SHAP method to explain the output of any function.

Kernel SHAP is a method that uses a special weighted linear regression to compute the importance of each feature. The computed importance values are Shapley values from game theory and also coefficents from a local linear regression.

model
: function or iml.Model
User supplied function that takes a matrix of samples (# samples x # features) and computes a the output of the model for those samples. The output can be a vector (# samples) or a matrix (# samples x # model outputs).
data
: numpy.array or pandas.DataFrame or iml.DenseData or any scipy.sparse matrix
The background dataset to use for integrating out features. To determine the impact of a feature, that feature is set to “missing” and the change in the model output is observed. Since most models aren’t designed to handle arbitrary missing data at test time, we simulate “missing” by replacing the feature with the values it takes in the background dataset. So if the background dataset is a simple sample of all zeros, then we would approximate a feature being missing by setting it to zero. For small problems this background dataset can be the whole training set, but for larger problems consider using a single reference value or using the kmeans function to summarize the dataset. Note: for sparse case we accept any sparse matrix but convert to lil format for performance.
link
: “identity” or “logit”
A generalized linear model link to connect the feature importance values to the model output. Since the feature importance values, phi, sum up to the model output, it often makes sense to connect them to the ouput with a link function where link(outout) = sum(phi). If the model output is a probability then the LogitLink link function makes the feature importance values have log-odds units.
shap_values(X, **kwargs)

Estimate the SHAP values for a set of samples.

X
: numpy.array or pandas.DataFrame or any scipy.sparse matrix
A matrix of samples (# samples x # features) on which to explain the model’s output.
nsamples
: “auto” or int
Number of times to re-evaluate the model when explaining each prediction. More samples lead to lower variance estimates of the SHAP values.
l1_reg
: “auto” or float
The l1 regularization to use for feature selection (the estimation procedure is based on a debiased lasso). Set this to zero to remove the feature selection step before estimation.

For models with a single output this returns a matrix of SHAP values (# samples x # features). Each row sums to the difference between the model output for that sample and the expected value of the model output (which is stored as expected_value attribute of the explainer). For models with vector outputs this returns a list of such matrices, one for each output.

class shap.DeepExplainer(model, data, session=None, learning_phase_flags=None)

Meant to approximate SHAP values for deep learning models.

This is an enhanced version of the DeepLIFT algorithm (Deep SHAP) where, similar to Kernel SHAP, we approximate the conditional expectations of SHAP values using a selection of background samples. Lundberg and Lee, NIPS 2017 showed that the per node attribution rules in DeepLIFT (Shrikumar, Greenside, and Kundaje, arXiv 2017) can be chosen to approximate Shapley values. By integrating over many backgound samples DeepExplainer estimates approximate SHAP values such that they sum up to the difference between the expected model output on the passed background samples and the current model output (f(x) - E[f(x)]). Using tf.gradients to implement the backgropagation was inspired by the gradient based implementation approach proposed by Ancona et al, ICLR 2018. Note that this package does not currently use the reveal-cancel rule for ReLu units proposed in DeepLIFT.

custom_grad(op, *grads)

Passes a gradient op creation request to the correct handler.

phi_symbolic(i)

Get the SHAP value computation graph for a given model output.

run(out, model_inputs, X)

Runs the model while also setting the learning phase flags to False.

shap_values(X, ranked_outputs=None, output_rank_order='max')

Return approximate SHAP values for the model applied to the data given by X.

X
: list, numpy.array, or pandas.DataFrame
A tensor (or list of tensors) of samples (where X.shape[0] == # samples) on which to explain the model’s output.
ranked_outputs
: None or int
If ranked_outputs is None then we explain all the outputs in a multi-output model. If ranked_outputs is a positive integer then we only explain that many of the top model outputs (where “top” is determined by output_rank_order). Note that this causes a pair of values to be returned (shap_values, indexes), where shap_values is a list of numpy arrays for each of the output ranks, and indexes is a matrix that indicates for each sample which output indexes were choses as “top”.
output_rank_order
: “max”, “min”, or “max_abs”
How to order the model outputs when using ranked_outputs, either by maximum, minimum, or maximum absolute value.

For a models with a single output this returns a tensor of SHAP values with the same shape as X. For a model with multiple outputs this returns a list of SHAP value tensors, each of which are the same shape as X. If ranked_outputs is None then this list of tensors matches the number of model outputs. If ranked_outputs is a positive integer a pair is returned (shap_values, indexes), where shap_values is a list of tensors with a length of ranked_outputs, and indexes is a matrix that indicates for each sample which output indexes were chosen as “top”.

Plots

shap.summary_plot(shap_values, features=None, feature_names=None, max_display=None, plot_type='dot', color=None, axis_color='#333333', title=None, alpha=1, show=True, sort=True, color_bar=True, auto_size_plot=True, layered_violin_max_num_bins=20, class_names=None)

Create a SHAP summary plot, colored by feature values when they are provided.

shap_values
: numpy.array
Matrix of SHAP values (# samples x # features)
features
: numpy.array or pandas.DataFrame or list
Matrix of feature values (# samples x # features) or a feature_names list as shorthand
feature_names
: list
Names of the features (length # features)
max_display
: int
How many top features to include in the plot (default is 20, or 7 for interaction plots)
plot_type
: “dot” (default) or “violin”
What type of summary plot to produce
shap.dependence_plot(ind, shap_values, features, feature_names=None, display_features=None, interaction_index='auto', color='#1E88E5', axis_color='#333333', dot_size=16, alpha=1, title=None, show=True)

Create a SHAP dependence plot, colored by an interaction feature.

ind
: int
Index of the feature to plot.
shap_values
: numpy.array
Matrix of SHAP values (# samples x # features)
features
: numpy.array or pandas.DataFrame
Matrix of feature values (# samples x # features)
feature_names
: list
Names of the features (length # features)
display_features
: numpy.array or pandas.DataFrame
Matrix of feature values for visual display (such as strings instead of coded values)
interaction_index
: “auto”, None, or int
The index of the feature used to color the plot.
shap.force_plot(base_value, shap_values, features=None, feature_names=None, out_names=None, link='identity', plot_cmap='RdBu')

Visualize the given SHAP values with an additive force layout.

shap.image_plot(shap_values, x, labels=None, show=True)

Plots SHAP values for image inputs.