%%capture # execute the creation & training notebook first %run "02-01-creation_and_training.ipynb" # execute the outlier detection notebook %run "02-05-outlier_detection.ipynb"
In the outlier detection section we saw how to detect outliers in a test data set and how the outlier threshold influenced the detection results. In the rank estimation section the underlying method of rank estimation was explained.
In this section we take a look at the
.estimate_probabilities method, which calls the
ProbabilityEstimator, the basis for both outlier detection and rank estimation.
ProbabilityEstimator is used by the
RankEstimator under the hood to estimate the probability of individual data points. Let us apply the probability estimator to the modified test data set from the outlier detection example.
log_probabilities = causal_structure.estimate_probabilities(data=mod_test_data) log_probabilities
100 rows × 4 columns
The numbers that are returned are the logarithmic probability densities. These values are unbound. They can assume floating point value.
Since they are densities their individual values do not tell us much about the likelihood of a given event. Only the comparison to other probability density values for the same parameter provides some insight.
For example let us extract the 5 data points with the lowest probability for the whole graph.
We see that the modified data points 50 and 60 indeed have very low logarithmic probability densities. Much lower than any of the unmodified data points. This is why they were easily picked up by the
One more thing is noticable when looking at the logarithmic probability densities. The sum of the ‘(a)’, ‘(b|a)’ and ‘(c|a,b)’ columns seems to be close to the ‘graph’ column.
log_probability_sums = log_probabilities[['(a)', '(b|a)', '(c|a,b)']].sum(axis=1) log_probabilities_with_sum = log_probabilities.copy() log_probabilities_with_sum["sum of '(a)', '(b|a)', '(c|a,b)'"] = log_probability_sums log_probabilities_with_sum.sort_values("graph").iloc[:5]
|(a)||(b|a)||(c|a,b)||graph||sum of '(a)', '(b|a)', '(c|a,b)'|
This is not by accident. The chain rule (or product rule) of conditional probabilities tells us that the product of the (correctly conditioned) probability densities is the total probability density. The graph column represents the total logarithmic probability density and for logarithmic probability densities the product becomes a sum.
The reason why the sum and the graph column do not match exactly is that the graph also contains the probabilites of the trained regression parameters, which enter the total sum.
For further details about the
ProbabilityEstimator see the corresponding section in the core-documentation.