Research 7 -- Central Limit Theorem, LLN, and most common probability distributions

Law of Large Numbers and CLT

Intuitively, everyone can be convinced of the fact that the average of many measurements of the same unknown quantity tends to give a better estimate than a single measurement. The law of the large numbers (LLN) and the central limit theorem (CLT) formalise this general ideas through mathematics and random variables.
Suppose X1, X2, ..., Xn are independent random variables with the same underlying distribution. In this case, we say that the Xi are independent and identically-distributed (or, i.i.d.). In particular, the Xi have all the same mean μ and standard deviation σ.
The average of the i.i.d. variables is defined as:


The central limit theorem states that when an infinite number of successive random samples are taken from a population, the sampling distribution of the means of those samples will become approximately normally distributed with mean μ and standard deviation σ/ √N as the sample size becomes larger, irrespective of the shape of the population distribution.
More formally, the central limit theorem can be defined as follows: for the sample size → ∞ the arithmetic average tends more and more closely to the expected value E(X) = μ. In other words, given a set of random variables with expected value μ and variance σ2, for every d > 0:



The proof of the above equality follows immediately from the Chebyshev inequality: Var(X) = σ2 then Var(Xn) = σ2 / n. Thus









Most Popular Distributions

All probability distributions can be classified as discrete probability distributions or as continuous probability distributions, depending on whether they define probabilities associated with discrete variables or continuous variables.
The most common discrete probability distributions are:

1) Boolean (Bernoulli): takes the value 1 with probability p and the value 0 with probability 1-p.
2) Binomial: which expresses the probability of successes in a series of independent yes/no experiments, all with the same probability of success
3) Poisson: which expresses the probability of a given number of events occurring in a fixed interval of time or space if these events occur with a known constant rate and independently of the time since the last event.
4) Hypergeometric: which expresses the probability of k successes in n draws, without replacement, from a finite population of size N that contains exactly K objects and each draw is either a success or a failure.

Continuous probability distributions differ in several ways from discrete distributions.
For instance,

- the probability that a continuous random variable assumes a particular value is zero
- they cannot be represented in tabular form
- a formula is required in order to describe them

- naturally, the probability that a random variable assumes a value between a and b is equal to the area under the density function of a and b.

The most common continuous probability distributions are:
- Normal (or Gaussian, Laplace-Gauss): often used to represent real-valued random variables whose distributions are not known.
- Student's t: is defined as any member of a family of continuous probability distributions that arises when estimating the mean of a (normally distributed) population when the sample size is small and the standard deviation is unknown.
- Chi-Squared: this is a special case of the gamma distribution and is commonly used for goodness of fit of an observed distribution to a theoretical one. For instance, it's very useful for cryptanalysis of (not strictly limited to) "old" ciphers like Caesar, Vigenère, and so on.
- F-distribution (Fisher-Snedecor): is a continuous probability distribution that arises frequently as the  null distribution of a test statistics, most notably in the analysis of the variance
- Weibull: used to model the lifetime of technical device and used to describe the particle size distribution of particles generated by grinding, mining and crushing operations


Commenti

Post popolari in questo blog

Welford Algorithm

Research 6 - Derivation of Chebyshev's inequality and its application to prove the (weak) LLN