Statistical Models Archives - R Frequently Asked Questions

In the least square method, the regression model is established in such a way that the sum of the squares of the vertical distances of different points (residuals) from the regression line is minimized. When the relationship between the variables is not linear (one has a non-linear regression model), one may

try to transform the data to linearize the relationship,
fit polynomial or complex spline model to the data, or
fit a non-linear regression to the data.

In the non-linear regression model, a function is specified by a set of parameters to fit the data. The non-linear least squares approach is used to estimate such parameters. In R, the nls() is used to approximate the non-linear function using a linear one and iteratively try to find the best parameter values.

Some frequently used non-linear regression models are listed in the Table below.

sr#	Name	Model
1)	Michaelis-Menten	$y=\frac{ax}{1+bx}$
2)	Two-parameter asymptotic exponential	$y=a(1-e^{-bx})$
3)	Three-parameter asymptotic exponential	$y=a-be^{-cx}$
4)	Two parameter Logistic	$y=\frac{e^{a+bx}}{1+e^{a+bx}}$
5)	Three parameter Logistic	$y=\frac{a}{1+be^{-ex}}$
6)	Weibull	$y=a-be^{-cx^d}$
7)	Gompertz	$y=e^{-be^{-cx}}$
8)	Ricker curves	$y=axe^{-bx}$
9)	Bell-Shaped	$y=a \, exp(-\|bx\|^2)$

Let fit Michaelis-Menten non-linear function to the data given below.

x <- seq(1, 10, 1)
y <- c(3.7, 7.1, 11.9, 19, 27, 38.5, 51, 67.7, 85, 102)

nls_model <- nls(y ~ a * x/(1 + b * x), start = list(a = 1, b = 1))

summary(nls_model)

#### Output
Formula: y ~ a * x/(1 + b * x)

Parameters:
   Estimate Std. Error t value Pr(>|t|)    
a  4.107257   0.226711   18.12 8.85e-08 ***
b -0.060900   0.002708  -22.49 1.62e-08 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 2.805 on 8 degrees of freedom

Number of iterations to convergence: 11 
Achieved convergence tolerance: 6.354e-06

Let plot the non-linear predicted values from 10 data points of newly generated x-values

new.data <- data.frame(x = seq(min(x), max(x), len = 10))

plot(x, y)

lines(new.data$x, predict(nls_model, newdata = new.data) )

The sum of squared residuals and the confidence interval of the chosen values of the coefficient can be obtained by issuing the commands,

sum(resid(nls_model)^2) 
# or 
print(sum(resid(nls_model)^2))

confint(nls_model) 
# or 
print(confint(nls_model))

Note that the formula for nls() does not use special coding in linear terms, factors, interactions, etc. The right-hand side in the expression of nls() computes the expected value to the left-hand side. The start argument contains the list of starting values of the parameter used in the expression and is varied by the algorithm.

Logistic Regression Models in R

No Comments

| Statistical Models

In logistic regression models, the response variable ($y$) is of categorical (binary, dichotomous) values such as 1 or 0 (TRUE/ FALSE). It measures the probability of a binary response variable based on mathematical equation relating the values of response variable with the predictor(s). The built-in function glm() can be used to perform logistic regression analysis.

The odds are used in logistic regression. If $p$ is the probability of success, the odds of in favour of success are, $\frac{p}{q}=\frac{p}{1-p}$.

Note that probability can be converted to odds and odds can also be converted to probability. However, unlike probability, odds can exceed 1. For example, if the probability of an event is 0.25, the odds in the favour of that event are $\frac{0.25}{0.75}=0.33$, but the odds against the event are $\frac{0.75}{0.25}=3$.

In built-in dataset (“mtcars“), the column (am) describes the transmission mode (automatic or manual) which is of binary value (0 or 1). The logistic regression models between the column (response variable) am and other columns (regressors) namely hp, wt, and cyl can be performed as given are:

Logistic regression with one dichotomous predictor

logmodel1 <- glm(am ~ vs, family = "binomial")
summary(logmodel1)

Logistic regression with one continuous predictor

logmodel2 <- glm(am ~ wt, family = "binomial")
summary(logmodel2)

Logistic regression with multiple predictors

logmodel3 <- glm(am ~ cyl + hp + wt, family = "binomial")
summary(logmodel3)
plot(logmodel3)

Note: in logistic regression model, both dichotomous and continuous variables can be used as predictor.

In R, the coefficients returned by logistic regression are a logit, or the log of the odds. To convert logits to odds ratio exponentiate it and to convert logits to probability use $\frac{e^\beta}{1-e^\beta}$. For example,

logmodel1 <- glm(am ~ vs, family = "binomial", data = mtcars)
logit_coef <- logmodel1$coef

exp(logmodel1$coef)
exp(logit_coef)/(1 + exp(logmodel1$coef))

Generalized Linear Models (GLM) in R

No Comments

| Data Analysis, Statistical Models

The generalized linear models (GLM) can be used when the distribution of the response variable is non-normal or when the response variable is transformed into linearity. The GLMs are flexible extensions of linear models that are used to fit the regression models to non-Gaussian data.

The basic form of a Generalized linear model is
\begin{align*}
g(\mu_i) &= X_i’ \beta \\
&= \beta_0 + \sum\limits_{j=1}^p x_{ij} \beta_j
\end{align*}
where $\mu_i=E(U_i)$ is the expected value of the response variable $Y_i$ given the predictors, $g(\cdot)$ is a smooth and monotonic link function that connects $\mu_i$ to the predictors, $X_i’=(x_{i0}, x_{i1}, \cdots, x_{ip})$ is the known vector having $i$th observations with $x_{i0}=1$, and $\beta=(\beta_0, \beta_1, \cdots, \beta_p)’$ is the unknown vector of regression coefficients.

The glm() is a function that can be used to fit a generalized linear model, using the form

mod <- glm(formula, family = gaussian, data = data.frame)

The family argument is a description of the error distribution and link function to be used in the model.

The class of generalized linear models is specified by giving a symbolic description of the linear predictor and a description of the error distribution.

Family Name	Link Functions
`binomial`	`logit` , `probit`, `cloglog`
`gaussian`	`identity`, `log`, `inverse`
`Gamma`	`identity`, `inverse`, `log`
`inverse gaussian`	$1/ \mu^2$, `identity`, `inverse`,`log`
`poisson`	`logit`, `probit`, `cloglog`, `identity`, `inverse`
`quasi`	`log`, $1/ \mu^2$, `sqrt`

Generalized Linear Model Example in R

Consider the “cars” dataset available in R.

data(cars)
head(cars)
attach(cars)
scatter.smooth(x=speed, y=dist, main = "Dist ~ Speed")

# Linear Model
lm(dist ~ speed, data = cars)
summary(lm(dist ~ speed, data = cars)

# Generalized Linear Model
glm(dist ~ speed, data=cars, family = "binomial")
plot(glm(dist ~ speed, data = cars))
summary(glm(dist ~ speed, data = cars))

Weighted Least Squares In R

No Comments

| Statistical Models

This post will discuss the implementation of Weighted Least Squares (WLS) in R. The OLS method minimizes the sum of squared residuals, while the WLS weights the square residuals. The WLS technique is used when the OLS assumption related to constant variance in the errors is violated.

The WLS technique is also known as weighted linear regression it is a generalization of ordinary least squares (OLS) and linear regression in which knowledge of the variance of observations is incorporated into the regression.

Let us perform the WLS in R.

Here we will use mtcars data set. You need to load this data set first using attach() function. For example,

attach(mtcars)

Consider the following example regarding the weighted least squares in which the reciprocal of $wt$ variable is used as weights. Here two different weighted models are performed and then check the fit of the model using anova() function.

w_model1 <- lm(mpg ~ wt + hp, data = mtcars)

w_model2 <- lm(mpg ~ wt + hp, data = mtcars, weights = 1/wt)

To check the models fit, summary statistics of the fitted model, and different diagnostic plots of the fitted model, one can use the built-in functions as,

anova(w_model1, w_model2 )

summary(w_model1)
summary(w_model2)

plot(w_model1)
plot(w_model2)

Weighted Least Squares (Model 1) — Diagnostic Plots for WLS model: Model-1

Weighted Least Squares Model-2 — Diagnostic Plots for WLS model: Model-2

Learn about Generalized Least Squares

Curvilinear Regression in R

No Comments

| Data Analysis, Statistical Models

In this post, we will learn about some basics of curvilinear regression in R.

The curvilinear regression analysis is used to determine if there is a non-linear trend exists between $X$ and $Y$.

Adding more parameters to an equation results in a better fit to the data. A quadratic and cubic equation will always have higher $R^2$ than the linear regression model. Similarly, a cubic equation will usually have higher $R^2$ than quadratic.

The logarithmic relationship can be described as follows:
$$Y=m\, log(x)++c$$
the polynomial relationship can be described as follows:
$$Y=m_1x + m_2x^2 + m_3x^3 + m_nx^n + c$$

The logarithmic example is more akin to a simple regression, whereas the polynomial example is multiple regression. Logarithmic relationships are common in the natural world; you may encounter them in many circumstances. Drawing the relationships between response and predictor variables as a scatter plot is generally a good starting point.

Consider the following data that are related in a curvilinear form,

Growth	Nutrient
2	2
9	4
11	6
12	8
13	10
14	16
17	22
19	28
17	30
18	36
20	48

Let us perform a curvilinear regression in R language.

Growth <- c(2, 9, 11, 12, 13, 14, 17, 19, 17, 18, 20)
Nutrient <- c(2, 4, 6, 8, 10, 16, 22, 28, 30, 36, 48)

data <- data <- as.data.frame(cbind(Growth, Nutrient))

ggplot(data, aes(Nutrient, Growth) ) +
  geom_point() +
  stat_smooth()

The Scatter plot shows the relationship appears to be a logarithmic one.

Let us carry out a linear regression using the lm() function by taking the $\log$ of the predictor variable rather than the basic variable itself.

data <- cbind(Growth, Nutrient)
mod <- lm(Growth~log(Nutrient, data))

summary(mod)

##
Call:
lm(formula = Growth ~ log(Nutrient), data = data)

Residuals:
    Min      1Q  Median      3Q     Max 
-2.2274 -0.9039  0.5400  0.9344  1.3097 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)    
(Intercept)     0.6914     1.0596   0.652     0.53    
log(Nutrient)   5.1014     0.3858  13.223 3.36e-07 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1.229 on 9 degrees of freedom
Multiple R-squared:  0.951,     Adjusted R-squared:  0.9456 
F-statistic: 174.8 on 1 and 9 DF,  p-value: 3.356e-07

Learn about Performing Linear Regression in R

R Frequently Asked Questions

Category: Statistical Models

Non-Linear Regression Model

Like this:

Logistic Regression Models in R

Logistic regression with one dichotomous predictor

Logistic regression with one continuous predictor

Logistic regression with multiple predictors

Like this:

Generalized Linear Models (GLM) in R

Generalized Linear Model Example in R

Like this:

Weighted Least Squares In R

Like this:

Curvilinear Regression in R

Like this:

Share this:

Like this:

Logistic regression with one dichotomous predictor

Logistic regression with one continuous predictor

Logistic regression with multiple predictors

Share this:

Like this:

Generalized Linear Model Example in R

Share this:

Like this:

Share this:

Like this:

Share this:

Like this: