Insight 3 - Bayes' Theorem

-
Before giving the Bayes theorem statement is important to define the conditional probability. Once done, the Bayes theorem is easy to derive.

Conditional Probability

The conditional probability measures the likelihood of an event given that another event has occurred.
For instance, assuming that A is the event of interest and that B consists of a different event which has already occurred.
The probability of the event A to occur, given the occurrence of B, (written as) P(A | B), can be computed as detailed by the following equality.

P(A | B) = P(A ∩ B) / P(B)        P(B) > 0                               (1)

It is defined as the quotient of the probability of the joint events A and B and the probability of B.
Now consider P(B | A), which is equals to P(B ∩ A) / P(A). Since P(B ∩ A) = P(A ∩ B), then the following equality holds:

P(A ∩ B) = P(A | B) P(B) =  P(B | A) P(A) 

Replacing the joint probability of A and B in (1) we obtain the next equality which is exactly the Bayes' theorem statement.


P(A | B) = ( P(B | A) P(A) ) / P(B) 

This theorem has several interpretations, which naturally depend on the interpretation of the probability itself.
One of the many applications of the theorem is Bayesian inference, which is a method of statistical inference. Here the theorem is used to update (or just change) the degree of belief regarding a certain event in order to account the occurrence of one or more different event. For instance, considering the toss of a coin, an individual may change his opinion once he's observed that (e.g.) 3 out of 5 tosses gave heads instead of tails.
One interesting application of the theorem is also known as Bayesian spam filtering which is a technique allowing a piece of software to "guess" whether an e-mail message is spam or not. The decision is made by analysing the e-mail messages received in the past and estimating the likelihood for some particular word to appear into a spam message.
Last but not least, is important to remember that the Bayesian inference has played a crucial role during the Second World War. In particular, it helped the brilliant A. Turing and his colleagues to break the encrypted German messages computed by the Enigma machines. 

Commenti

Posta un commento

Post popolari in questo blog

Welford Algorithm

-
Immagine
The variance is the expectation of the squared deviation of a random variable form its mean.  Informally, it measures how far a set of (random) numbers are spread out from their average value. The variance is the square of the standard deviation, the second central moment of a distribution and the covariance of the random variable with itself, and it is often represented by Var(X), σ 2 , s 2 . One of the most widely known formula for computing the variance is: where x-bar is the mean of the sample. The definition given above can be converted into an algorithm that computed the variance and the standard deviation in two passes: 1. Compute the mean                            (O(n)) 2. Compute the square differences       (O(n)) Output the variance Even though this algorithm seems working properly, it may become too expensive on some input instances. Just consider a sampling procedu...
Continua a leggere

Research 6 - Derivation of Chebyshev's inequality and its application to prove the (weak) LLN

-
Immagine
Chebyshev's Inequality In probability theory, the Chebyshev's inequality guarantees that, for a wide class of probability distributions, no more than a certain fraction of values can be more distant than a certain value from the mean. In particular, the mentioned inequality states that no more than 1/k 2 of the distribution's values can be more than k standard deviations away from the mean. In other words, this mean that at least (1 - 1/k 2 ) of the distribution's values are within k standard deviations of the mean. The Chebyshev's inequality can be easily derived from the Markov's inequality, where the latter defines an upper bound for the probability that a non-negative random variable is greater than (or equal to) some positive integer constant. Remember the Markov's inequality where   a  > 0 and  X  is a nonnegative random variable The Chebyshev inequality follows by considering the random variable ( X - E ( X )) 2  and the constant a 2...
Continua a leggere

Research 7 -- Central Limit Theorem, LLN, and most common probability distributions

-
Immagine
Law of Large Numbers and CLT Intuitively, everyone can be convinced of the fact that the average of many measurements of the same unknown quantity tends to give a better estimate than a single measurement. The law of the large numbers (LLN) and the central limit theorem (CLT) formalise this general ideas through mathematics and random variables. Suppose X 1 , X 2 , ..., X n are independent random variables with the same underlying distribution. In this case, we say that the X i are independent and identically-distributed (or, i.i.d.). In particular, the X i have all the same mean μ and standard deviation σ. The average of the i.i.d. variables is defined as: The central limit theorem states that when an infinite number of successive random samples are taken from a population, the sampling distribution of the means of those samples will become approximately normally distributed with mean μ and standard deviation σ/ √N as the sample size becomes larger, irrespective of the sh...
Continua a leggere