20  GLM definition

In this class, the focus is on building models that tie a vector of predictors \((\boldsymbol x_{i*})\) to a response \(y_i\). For linear regression, the mean of \(y\) was modeled as a linear combination of the predictors \(\boldsymbol x_{i*}^T \boldsymbol{\beta}\): \(\mu_i = \boldsymbol x_{i*}^T \boldsymbol{\beta}\). More generally, we might want to model a function of the mean \(\eta_i = g(\mu_i)\) as a linear combination of the predictors; \(g\) is called the link function and \(\eta_i\) the linear predictor. Pairing a link function with an EDM gives us a generalized linear model (GLM):

20.1 Definition

We define \(\{(y_i, \boldsymbol x_{i*})\}_{i = 1}^n\) as following a generalized linear model based on the exponential dispersion model \(f_{\theta, \phi}\), monotonic and differentiable link function \(g\), offset terms \(o_i \in \mathbb R\), and observation weights \(w_i > 0\) if

\[ y_i \overset{\text{ind}} \sim \text{EDM}(\mu_i, \phi_0/w_i), \quad \eta_i \equiv g(\mu_i) = o_i + \boldsymbol x^T_{i*}\boldsymbol{\beta}. \tag{20.1}\]

The offset terms \(o_i\) and observation weights \(w_i\) are both known in advance. The free parameters in a GLM are the coefficients \(\boldsymbol{\beta}\) and, possibly, the parameter \(\phi_0\) controlling the dispersion. We will see examples where \(\phi_0\) is known (e.g. Poisson regression) and those where \(\phi_0\) is unknown (e.g. linear regression).

The “default” choice for the link function \(g\) is the canonical link function

\[ g(\mu) = \dot \psi^{-1}(\mu), \]

which, given the relationship (19.2), gives \(\eta = \dot \psi^{-1}(\mu) = \theta\), i.e. the linear predictor coincides with the natural parameter. As discussed in the context of equation (19.3), \(\dot \psi^{-1}\) is a valid link function because it is monotonic and differentiable. Canonical link functions are very commonly used with GLMs because they lead to various nice properties that general GLMs do not enjoy (e.g. concave log-likelihood).

20.2 Examples

20.2.1 Example: Linear regression model

The linear regression model is a special case of a GLM, with \(\phi_0 = \sigma^2\) (unknown), \(w_i = 1\), \(o_i = 0\), and identity (canonical) link function:

\[ y_i \overset{\text{ind}}\sim N(\mu_i, \sigma^2); \quad \eta_i = \mu_i = \boldsymbol x_{i*}^T \boldsymbol{\beta}. \]

20.2.2 Example: Weighted linear regression model

If each observation \(y_i\) is the mean of \(m_i\) independent repeated observations, then we get a weighted linear regression model, with \(\phi_0 = \sigma^2\) (unknown), \(w_i = m_i\), \(o_i = 0\), and identity (canonical) link function:

\[ y_i \overset{\text{ind}}\sim N(\mu_i, {\textstyle \frac{\sigma^2}{m_i}}); \quad \eta_i = \mu_i = \boldsymbol x_{i*}^T \boldsymbol{\beta}. \]

20.2.3 Example: Ungrouped logistic regression model

The ungrouped logistic regression model is the GLM based on the Bernoulli EDM with \(\phi_0 = 1\) (known), \(w_i = 1\), \(o_i = 0\), and the canonical link function:

\[ y_i \overset{\text{ind}}\sim \text{Ber}(\mu_i); \quad \eta_i = \theta_i = \log\frac{\mu_i}{1-\mu_i} = \boldsymbol x_{i*}^T \boldsymbol{\beta}. \]

Thus the canonical link function for logistic regression is the logistic link function \(g(\mu) = \log \frac{\mu}{1-\mu}\).

20.2.4 Example: Grouped logistic regression model

Suppose \(y_i\) is a binomial proportion based on \(m_i\) trials. The grouped logistic regression model is the GLM based on the binomial EDM with \(\phi_0 = 1\) (known), \(w_i = 1/m_i\), \(o_i = 0\), and the canonical link function:

\[ m_i y_i \sim \text{Bin}(m_i, \mu_i); \quad \eta_i = \log \frac{\mu_i}{1-\mu_i} = o_i + \boldsymbol x^T_{i*}\boldsymbol{\beta}. \]

Note that a binomial proportion \(y_i\) based on \(m_i\) trials and a success probability of \(\mu_i\) can be equivalently represented as \(m_i\) independent Bernoulli draws with the same success probability \(\mu_i\). Therefore, any grouped logistic regression model can be equivalently represented as an ungrouped logistic regression model with \(\sum_{i = 1}^n m_i\) observations. We will see that, despite this equivalence, grouped logistic regression models have some useful properties that ungrouped logistic regression models do not.

20.2.5 Example: Poisson regression model

Poisson regression is the Poisson EDM with \(\phi_0 = 1\) (known), \(w_i = 1\), \(o_i = 0\), and the canonical link function:

\[ y_i \overset{\text{ind}}\sim \text{Poi}(\mu_i); \quad \eta_i = \theta_i = \log \mu_i = \boldsymbol x_{i*}^T \boldsymbol{\beta}. \]

Thus the canonical link function for Poisson regression is the log link function \(g(\mu) = \log \mu\).