Linear regression

Definitions & Key takeaways

Linear regression is a mathematical technique used to estimate the relationship between two variables. It is used when the relationship between the two variables is linear (meaning that it follows a straight line).

The linear regression equation looks like this: y = ax + b, where y is the predicted value, x is the independent variable (the one we are trying to predict), a is the slope of the line, and b is the y-intercept.

If we know the values of a and b, we can use the equation to predict the value of y for any given x. We can also use linear regression to determine how strong (or weak) the relationship between x and y actually is.

Watch video only
Let’s say you want to figure out if smoking more cigarettes leads to a lower forced expiratory volume, or FEV, which is the total amount of air, in liters per second, that a person can exhale in a single forced breath.
And let’s say that healthy men generally have an FEV of around 4 liters per second. Now, to figure out if men who smoke have a lower FEV, you might ask 100 men how many cigarettes they smoke in a day, and then measure each man’s FEV.
You could plot these measurements, or data points, on a scatterplot, with the number of cigarettes, which is the exposure, on the x-axis, and FEV, which is the outcome, on the y-axis, and where each data point represents one individual.
Typically, a linear trend line, or model, is drawn to represent the pattern of data points on the plot. Theoretically, there are lots of lines that can be drawn to represent the data points, but the best trend line is the one with the smallest amount of error, which is a measurement of how far away an individual data point is from the trend line.
Usually, we look at the squared error, which is the distance between the data point and the line, squared. For example, if a data point is very close to the trend line, then the squared error is small.
On the other hand, if a data point is very far from the trend line, then the squared error is large. If you add up all the squared error for a line, you get the total squared error, and a line with a smaller total squared error is considered a better fit for the data than a line with a larger total squared error.
Now, when two variables are linearly related, we might want to know specifically what happens to the outcome variable when the exposure variable changes.
For example, we might want to know how a person’s FEV changes if they smoke five cigarettes per day compared to if they smoke ten cigarettes per day.
To figure this out, we have to use linear regression, which is a statistical method that calculates an equation for the best fitting trend line for a set of data points.
Specifically, in this case, we would use simple linear regression, since we only have two variables—one y-variable, which is the FEV measurement, and one x-variable, which is the number of cigarettes.
This is different from multiple linear regression, where there’s one y-variable and two or more x-variables. Typically, multiple linear regression is used to control for confounding variables, or x-variables that distort the true relationship between the main exposure variable and the outcome variable.
For example, age is a confounding variable in the relationship between smoking and FEV, because older people tend to have a lower FEV than younger people.
So, let’s say you wanted to figure out how FEV changes for people who smoke more cigarettes per day, controlling for age.
In this example, there are two x-variables—the number of cigarettes a person smokes and a person’s age—so, you’d have to use multiple linear regression.
Now, there are four key assumptions that are used in linear regression, and an acronym for these assumptions is L-I-N-E, or LINE.
First, the relationship between the two variables has to be Linear, which means that the trendline drawn to represent the data points is a straight line.
Relationships that have a different type of curve, like in an exponential relationship or a U-shaped relationship, will have a low correlation coefficient because a straight trendline doesn’t match the shape of the data points.
The second assumption is that each individual in the sample was recruited Independently from other individuals in the sample.
In other words, no individuals influenced whether or not any other individual was included in the study. For example, if one person agrees to be in the study only if their friend can also be included in the study, then these two individuals would not be independent of each other and the second assumption would not be met.
Independent recruitment ensures that the people included in the study have similar characteristics to the target population, and that the results of the test can be applied to the target population - meaning it has good external validity!
Additionally, the sample population must have been recruited randomly, like if you chose 100 names randomly from a list of all names in the target population.
Like independent recruitment, random sampling is important because it ensures that the sample population approximates the target population.
The third assumption is that the errors between the observed and predicted values of y are Normally distributed around a value of x.
That might sound confusing - no worries. Let’s break it down, by reviewing the trendline, which shows a predicted y-value for a specific x-value.
For example, let’s say we have a trendline that looks like this. According to our trendline, for people that smoke 10 cigarettes per day, we predict that their FEV will be around 3 liters per second.
But in reality, some people that smoke 10 cigarettes per day will have an FEV that’s higher or lower than 3, simply because of individual differences, like how old a person is or how much a person exercises.
Now, remember that the distance a point is from the prediction line is called error, and points that are far from the line are said to have a high error.
Okay, so the third assumption says that the errors for a given point will follow a normally distributed bell curve, meaning most people will have low error, so their FEV will be clustered near the trendline; and fewer people will have high error, so their FEV will be higher or lower than the trendline.
And this is true for every value of x, or for each number of cigarettes smoked per day. The fourth assumption is that the data points must have Equal variance, which is also called homoscedasticity.
Basically, this means that the data points are equally spread out from the trendline for every value of x. If the data points are closer to the trendline on one end and further away on the other end, then the assumption of equal variance is not met and the data are heteroscedastic.
Now, going back to simple linear regression, a linear trend line is represented by the equation y-hat equals b0 plus b1x1, where y-hat is the estimated value for the outcome variable, which in this case is FEV, and x1 is the value of the exposure variable, so in this case that’s the number of cigarettes a person smokes.
b0 represents the y-intercept, and b1 represents the slope of the line. Typically, we use statistical software to calculate the linear equation, and the software will provide b0 and b1, which we can then plug into our equation.
For example, let’s say the software gives us a b0 of 4 and a b1 of negative 0.1, so the equation for the line would be y-hat equals 4 minus 0.1 times x1.
This equation tells us two crucial pieces of information. First, b0 tells us the average value of the outcome variable when the exposure variable equals zero.
For example, a b0 of 4 means that people who smoke 0 cigarettes per day have an average FEV of 4 liters per second. Second, b1 tells us the effect size, or how much the outcome variable changes for every one-unit increase in the exposure variable.
For example, a b1 of negative 0.1 means that, on average, the FEV will decrease by 0.1 liters per second for every one additional cigarette smoked per day.
In other words, if people that smoke zero cigarettes per day have an average FEV of 4, then people who smoke one cigarette per day will have an average FEV of 4 minus 0.1, or 3.9; people that smoke two cigarettes will have an average FEV of 3.9 minus 0.1, or 3.8; and so on.
We can also use this equation to predict the average FEV for a certain number of cigarettes smoked per day. For example, let’s say we want to predict what the FEV would be for a person that smokes 10 cigarettes per day.
To do this, we plug the number 10 into the x1 spot in our equation, so y-hat equals 4 minus 0.1 times 10, and then solve for y-hat.
0.1 times 10 is 1, and 4 minus 1 is 3, so we predict that a person that smokes 10 cigarettes per day will have an average FEV of 3 liters per second.
In multiple linear regression, it’s common to have 2, 3, 4, or more exposure variables, that are all included in the linear regression equation.
To keep things simple, let’s just do an example with two exposure variables. So, let’s say we want to look at the relationship between FEV and number of cigarettes, and that we want to control for age, which we think is a confounder.
So here, the exposure variables are the number of cigarettes and age. In this case, our equation will be: y-hat equals b0 plus b1x1, plus b2x2, where x1 is number of cigarettes and x2 is age.
Statistical software provides the y-intercept and slopes for each variable, so let’s say it gives us a b0 of 4, a b1 of negative 0.1, and a b2 of negative 0.02.
A b2 of negative 0.02 means that: the average FEV decreases by 0.02 liters per second for every one year increase in age.
So, let’s say we want to predict the average FEV for a 50-year-old man who smokes 15 cigarettes per day, so we plug each number into our equation and get: y-hat equals 4 minus 0.1 times 15, minus 0.02 times 50.
0.1 times 15 is 1.5, and 0.02 times 50 is 1, so the equation can be simplified to 4 minus 1.5 minus 1, which equals 1.5.
This means that a 50-year-old man who smokes 15 cigarettes a day has a predicted FEV of 1.5 liters per second. One important thing to know about linear regression is that the outcome variable always has to be continuous, like liters per second, but the exposure variable can either be continuous, like number of cigarettes smoked per day, or categorical, like if a person smokes or doesn’t smoke.
So, let’s say we want to figure out how FEV changes for people that smoke versus people that don’t smoke. The statistical software will still provide a value for b0 and b1, but in this case, b1 is slightly different.
For categorical variables, the software automatically creates a dummy variable, which is basically a binary variable where the number 1 represents the reference group, or simply, one subgroup of the exposure variable, like people that smoke.
On the flip side, 0 represents the other, non-reference subgroup of the exposure variable, so people that don’t smoke. So, let’s say the value for b0 is still 4, and the value of b1 is negative 1.2, so the linear equation is y-hat equals 4 minus 1.2 times x1.
In this case, the effect size is 1.2, which means that on average, people that smoke have an FEV 1.2 liters per second less than people that don’t smoke.
To predict the FEV for a person who smokes, we simply plug in the number 1 into the equation, because people that smoke is the reference group.
So, 1.2 times 1 is still 1.2, and 4 minus 1.2 is 2.8, so the predicted FEV for a person that smokes is 2.8 liters per second.
On the other hand, to predict the FEV for a person that doesn’t smoke, we would plug 0 into the equation, because people who don’t smoke is the non-reference group.
So, 1.2 times 0 is 0, and 4 minus 0 is still 4, so the predicted FEV for a person that doesn’t smoke is 4 liters per second.
Typically, the software will also provide the coefficient of determination, or the R-squared value, which tells you how much variation in the outcome variable is explained by the linear regression model, as opposed to natural variation.
In other words, can a person’s FEV be explained by the number of cigarettes they smoke, or is it more the result of other factors, like sex, age, or the amount they exercise.
So let’s say we get an R-squared value of 0.72. This means that 72% of the variation in FEV can be explained by the number of cigarettes smoked per day, and the remaining 28% is explained by other factors.
A high R-squared value like this one tells us that most of the variation we see between people with a high FEV and a low FEV is the result of how much they smoke, and specifically, that people with lower FEV smoke more cigarettes than people with high FEV.
Now, there are two key limitations when doing linear regression. The first limitation is that linear regression can be sensitive to outliers, or data points that are much higher or much lower than all the other data points.
For example, let’s say that one person is added to our study, and that person smokes a lot of cigarettes but also has a very high FEV.
In this case, the trendline would try to account for this new data point, but since it’s so far away from all the other data points, the slope will be much different than the slope that doesn’t include the outlier.
In linear regression, outliers can affect both the effect size calculation and predictions. The other limitation of linear regression is that we can’t extrapolate, or make conclusions, about the data points outside the range of x-values that are in the sample.
For example, let’s say the range of cigarettes smoked by people in our study is 0 to 25, so 0 is the lowest number of cigarettes smoked by people in the study and 25 is the highest number of cigarettes smoked by people in the study.
Now, we know there’s a linear relationship between the number of cigarettes smoked in this range, but we don’t know if the linear relationship between FEV and smoking continues outside the range of our sample data - say for someone who smokes 40 cigarettes a day.
Alright, as a quick recap, linear regression is a statistical method used to figure out the effect of an exposure variable, like number of cigarettes per day, on an outcome variable, like FEV, and it uses the equation y-hat equals b0 plus b1x1.
b0 represents the average value of the outcome when the exposure variable is zero, and for continuous exposure variables, b1 represents the average change in the outcome variable for every one unit increase in the exposure variable.
In categorical exposure variables, b1 represents the value of the outcome for the reference group compared to the non-reference group.
It’s important to keep in mind that linear regression works best for data that don’t have outliers, and that you can only make conclusions about the data points within the sample range.