Statistical Inference

1 Populations and Samples

1. Populations are the complete set of all possible observations or measurements that are of interest. Its size is the number of elements in the population, denoted by $N$.

2. Samples are subsets of the population selected by sampling procedures. Its size is the number of elements in the sample, denoted by $n$.

3. A sampling procedure is called biased if it produces inferences that consistently overestimate or underestimate the population parameter of interest.

2 Location Measures

1. Mean is the sum of all observations divided by the number of observations. It is denoted by $\overline{x}$. It is a measure of central tendency and sensitive to outliers.

   • For scalar data $x$: $\overline{x}=\frac{1}{n}\sum _{i=1}^n{x}_i$

   • For vector data ${\mathit{x}}_i\in {\mathbb{R}}^{d\times 1}$: $\overline{\mathit{x}}=\frac{1}{n}\sum _{i=1}^n{\mathit{x}}_i$

2. Median is the middle value of a dataset. It is denoted by $M$. It is a measure of central tendency and robust to outliers.

   • For scalar data $x$:
   $$M=\{\begin{array}{ll}{x}_{\left(\frac{n+1}{2}\right)}& \text{if }n\text{ is odd}\\ \frac{1}{2}(x_{\left(\frac{n}{2}\right)}+x_{(\frac{n}{2}+1)})& \text{if }n\text{ is even}\end{array}$$
   • For vector data ${\mathit{x}}_i\in {\mathbb{R}}^{d\times 1}$: We compute the geometric median.
   $$M=\arg \underset{\mathit{m}}{min}\sum _{i=1}^n\parallel {\mathit{x}}_i-\mathit{m}{\parallel }_2$$

3. Mode is the value that appears most frequently in a dataset. It is denoted by $Mo$. It is a measure of central tendency and robust to outliers.

   • For scalar data $x$:
   $$Mo=\arg \underset{x}{max}\sum _{i=1}^n\mathbb{I}(x_i=x)$$

   • For vector data ${\mathit{x}}_i\in {\mathbb{R}}^{d\times 1}$: We compute the mode corresponds to the point where all data points, then combine them to form a new mode vector.

   • If samples are less repetitive, we group them into bins and compute the mode of each bin.

4. Variance and Standard Deviation are measures of dispersion. They quantify the spread of the data around the mean. They are denoted by $s^2$ and $s$ (For sample) and ${\sigma }^2$ and $\sigma$ (For population).

   • For scalar data $x$:
   $$s^2=\frac{1}{n-1}\sum _{i=1}^n(x_i-\overline{x}{)}^2$$$$s=\sqrt{s^2}$$
   • For vector data ${\mathit{x}}_i\in {\mathbb{R}}^{d\times 1}$:
   $$s^2=\frac{1}{n-1}\sum _{i=1}^n\parallel {\mathit{x}}_i-\overline{\mathit{x}}{\parallel }_2^2$$$$s=\sqrt{s^2}$$
   • For population variance and standard deviation, replace $n-1$ by $n$.

   Why?

   The denominator is $n-1$ instead of $n$ because the sample variance is an unbiased estimator of the population variance. The unbiased estimator is the one that gives the correct answer on average over many samples. The sample variance is an unbiased estimator because the expected value of the sample variance is equal to the population variance.

5. Boxplot is a graphical representation of the data based on the five-number summary: minimum, first quartile, median, third quartile, and maximum. It is useful for detecting outliers and comparing distributions.

   1. Minimum: The smallest value in the dataset.
   2. First Quartile (Q1): The median of the lower half of the dataset.
   3. Median (Q2): The middle value of the dataset.
   4. Third Quartile (Q3): The median of the upper half of the dataset.
   5. Maximum: The largest value in the dataset.
   6. Interquartile Range (IQR): The range between the first and third quartiles, $IQR=Q3-Q1$.
   7. Outliers: Values that fall below $Q1-1.5\times IQR$ or above $Q3+1.5\times IQR$.
   8. Whiskers: The lines extending from the box to the minimum and maximum values, excluding outliers.

3 Correlation Measures

1. Pearson Correlation Coefficient measures the linear relationship between two variables. It ranges from -1 to 1, where:

   • 1 indicates a perfect positive linear relationship.
   • -1 indicates a perfect negative linear relationship.
   • 0 indicates no linear relationship.
   $$\rho =\frac{\sum _{i=1}^n(x_i-\overline{x})(y_i-\overline{y})}{\sqrt{\sum _{i=1}^n(x_i-\overline{x}{)}^2}\sqrt{\sum _{i=1}^n(y_i-\overline{y}{)}^2}}$$$$\rho =\frac{\text{Cov}(x,y)}{s_x{s}_y}$$

2. Multi-correlation Coefficient measures the linear relationship between multiple variables. It ranges from 0 to 1, where:

   • 1 indicates a perfect linear relationship.
   • 0 indicates no linear relationship.
   $$R^2=\frac{\sqrt{\sum _{i=1}^n({\hat{y}}_i-\overline{y}{)}^2}}{\sqrt{\sum _{i=1}^n(y_i-\overline{y}{)}^2}}$$

3. Covariance measures the relationship between two variables. It is the Pearson correlation coefficient multiplied by the standard deviations of the variables.

   • If $\text{Cov}(x,y)>0$, the variables are positively correlated.
   • If $\text{Cov}(x,y)<0$, the variables are negatively correlated.
   $$\text{Cov}(x,y)=\frac{1}{n-1}\sum _{i=1}^n(x_i-\overline{x})(y_i-\overline{y})$$

   Note

   If given two variables $\mathit{A}\in {\mathbb{R}}^{1\times n}$ and $\mathit{B}\in {\mathbb{R}}^{1\times n}$, the covariance matrix is given by:

   $$\text{Cov}(\mathit{A},\mathit{B})=\frac{1}{n-1}\sum _{i=1}^n({\mathit{A}}_i-\overline{\mathit{A}})({\mathit{B}}_i-\overline{\mathit{B}})$$

4. Covariance Matrix: The covariance matrix is a square matrix that contains the covariance between each pair of variables. It is symmetric and positive semi-definite.

   • The diagonal elements are the variances of the variables.
   • The off-diagonal elements are the covariances between the variables.
   $$\text{Cov}(\mathit{X})=\left[\begin{array}{cccc}\text{Cov}(X_1,X_1)& \text{Cov}(X_1,X_2)& \cdots & \text{Cov}(X_1,X_d)\\ \text{Cov}(X_2,X_1)& \text{Cov}(X_2,X_2)& \cdots & \text{Cov}(X_2,X_d)\\ ⋮& ⋮& \ddots & ⋮\\ \text{Cov}(X_d,X_1)& \text{Cov}(X_d,X_2)& \cdots & \text{Cov}(X_d,X_d)\end{array}\right]$$

   Note

   Let ${\mathit{x}}_{\mathbf{1}},{\mathit{x}}_{\mathbf{2}},\dots ,{\mathit{x}}_{\mathit{n}}\in {\mathbb{R}}^{d\times 1}$, (sample), then, its covariance matrix is given by:

   $$\mathrm{\Sigma }=\frac{1}{n-1}\sum _{i=1}^n({\mathit{x}}_i-\overline{\mathit{x}})({\mathit{x}}_i-\overline{\mathit{x}}{)}^T\in {\mathbb{R}}^{d\times d}.$$

   Question

   1. How to understand the covariance matrix?

   2. Linear relationship means direction + strength. dose the covaiance matrix here indicates these info?

   1. The covariance matrix $\mathrm{\Sigma }\in {\mathbb{R}}^{d\times d}$ represents the relationships between the components of the vectors. For diagonal elements, they represent the variance of each component. For off-diagonal elements, they represent the covariance between the components. The covariance matrix is symmetric and positive semi-definite. If its value is positive, it means the two components are positively correlated. If its value is negative, it means the two components are negatively correlated. If its value is zero, it means the two components are nearly independent.
   2. A linear relationship consists of direction and strength.
   3. Strength of Linear Relationship: Covariance values indicate the degree to which two dimensions are linearly related. However, these values are not normalized, so interpreting the "strength" directly can be difficult. Instead, the correlation coefficient (a normalized version of covariance) is often used to quantify strength. For ${\mathrm{\Sigma }}_{ij}$, the closer its magnitude to $\sqrt{{\mathrm{\Sigma }}_{ii}{\mathrm{\Sigma }}_{jj}}$, the stronger the linear relationship.
   4. Direction of Linear Relationship: The same to above:

   • If $\text{Cov}(x,y)>0$, the variables are positively correlated.
   • If $\text{Cov}(x,y)<0$, the variables are negatively correlated.

   Question

   If we suppose the dataset $\mathit{X}=[{\mathit{A}}_1,{\mathit{A}}_2,\dots ,{\mathit{A}}_d{]}^T\in {\mathbb{R}}^{d\times n}$, then, what is the dimension of the covariance matrix $\mathrm{\Sigma }$?

   $$\mathrm{\Sigma }=\left[\begin{array}{cccc}Var({\mathit{A}}_1,{\mathit{A}}_1)& Cov({\mathit{A}}_1,{\mathit{A}}_2)& \cdots & Cov({\mathit{A}}_1,{\mathit{A}}_d)\\ Cov({\mathit{A}}_2,{\mathit{A}}_1)& Var({\mathit{A}}_2,{\mathit{A}}_2)& \cdots & Cov({\mathit{A}}_2,{\mathit{A}}_d)\\ ⋮& ⋮& \ddots & ⋮\\ Cov({\mathit{A}}_d,{\mathit{A}}_1)& Cov({\mathit{A}}_d,{\mathit{A}}_2)& \cdots & Var({\mathit{A}}_d,{\mathit{A}}_d)\end{array}\right]$$

   The covariance matrix $\mathrm{\Sigma }\in {\mathbb{R}}^{d\times d}$.

4 Shape Measures

1. Skewness measures the asymmetry of the data distributions. It is denoted by $\gamma$.

   • If $\gamma >0$, the distribution is right-skewed.
   • If $\gamma <0$, the distribution is left-skewed.
   • If $\gamma =0$, the distribution is symmetric.
   • If $|\gamma |$ is small, it has a mild skewness.
   • If $|\gamma |$ is large, it has a severe skewness.
   $$\gamma =\frac{\frac{1}{n}\sum _{i=1}^n(x_i-\overline{x}{)}^3}{s^3}$$

   From Wikipedia:

   In probability theory and statistics, skewness is a measure of the asymmetry of the probability distribution of a real-valued random variable about its mean. The skewness value can be positive, zero, negative, or undefined.

   $${\gamma }_1:={\tilde{\mu }}_3=\mathrm{E}\left[{\left(\frac{X-\mu }{\sigma }\right)}^3\right]=\frac{{\mu }_3}{{\sigma }^3}=\frac{\mathrm{E}[(X-\mu {)}^3]}{{\left(\mathrm{E}[(X-\mu {)}^2]\right)}^{3/2}}=\frac{{\kappa }_3}{{\kappa }_2^{3/2}}$$

2. Kurtosis measures the tailedness of the data distributions. It is denoted by $\kappa$.

   • If $\kappa >3$, the distribution has heavier tails than the normal distribution (Leptokurtic).
   • If $\kappa <3$, the distribution has lighter tails than the normal distribution (Platykurtic).
   • If $\kappa =3$, the distribution has tails similar to the normal distribution (Mesokurtic).
   $$\kappa =\frac{\frac{1}{n}\sum _{i=1}^n(x_i-\overline{x}{)}^4}{s^4}$$

   From Wikipedia:

   In probability theory and statistics, kurtosis (from Greek: κυρτός, kyrtos or kurtos, meaning "curved, arching") refers to the degree of “tailedness” in the probability distribution of a real-valued random variable. Similar to skewness, kurtosis provides insight into specific characteristics of a distribution. Various methods exist for quantifying kurtosis in theoretical distributions, and corresponding techniques allow estimation based on sample data from a population. It’s important to note that different measures of kurtosis can yield varying interpretations.

   $${\kappa }_2:={\tilde{\mu }}_4=\mathrm{E}\left[{\left(\frac{X-\mu }{\sigma }\right)}^4\right]=\frac{{\mu }_4}{{\sigma }^4}=\frac{\mathrm{E}[(X-\mu {)}^4]}{{\left(\mathrm{E}[(X-\mu {)}^2]\right)}^2}=\frac{{\kappa }_4}{{\kappa }_2^2}$$