Logistic regression

Definitions & Key takeaways

Logistic regression is a statistical method used to describe the relationship between an outcome variable and one or more exposure variables. Logistic regression can help to figure out the effect of an exposure variable (e.g. the number of cigarettes per day) on a categorical outcome variable (e.g. Having a heart attack). Note that the outcome variable is always categorical, but the exposure variables can be either categorical or quantitative.

Watch video only
Logistic regression is a type of statistical method that’s used to describe the relationship between an outcome variable and one or more exposure variables.
In logistic regression, the outcome variable is always categorical, and the exposure variables can be either categorical or quantitative.
For example, let’s say you want to figure out if smoking more cigarettes increases the chance of having a heart attack. In this case, the number of cigarettes is a quantitative exposure and whether or not a person has a heart attack is a categorical outcome.
Now, to figure this out, you might ask 200 people how many cigarettes they smoke in a day, and then follow that group of people for five years and see who has a heart attack and who doesn’t.
You could organize your data in a table like this—where the first column, or variable, is the number of cigarettes a person smokes, the second column is if they had a heart attack or not, and the rest of the columns are other characteristics, or variables, that you collected about each person, like their age, sex, and body mass index, or BMI.
Usually, for binary variables, like yes or no, we use the numbers zero and 1 to represent the two possible answers. So, for the heart attack variable, we might say that zero represents “no” and 1 represents “yes”.
We could do the same thing for sex, where zero represents females and 1 represents males. Now, let’s just look at the first two variables, so how many cigarettes they smoke and if they had a heart attack or not.
You could plot these measurements, or data points, on a scatterplot, with the number of cigarettes on the x-axis, and heart attack on the y-axis, and where each data point represents one individual.
This scatterplot might seem a little funny looking, and that’s because all of the data points are clustered on two points on the y-axis—they’re either on the zero, which represents no, or the 1, which represents yes.
This scatterplot can help us figure out how the odds of having a heart attack changes for people as they smoke more and more cigarettes.
And that’s the goal of logistic regression. Now, in statistics, probability and odds are often confused with one another, so let’s break down the difference.
The probability is the number of times an outcome happened divided by the number of times the outcome could have happened, and it’s often represented by a capital P.
So, using our data, we could figure out the probability of having a heart attack for each number of cigarettes smoked per day.
Let’s say the range for the number of cigarettes smoked is between zero and 19, so we can break up the scatterplot up into 20 different sections - and it’s 20 sections instead of just 19 because zero is also a section.
Now, to find the probability of having a heart attack in a specific section, we count up the number of people who had heart attacks in that section and divide it by the total number of people in that section.
As an example, let’s say there are 10 people in the 15-cigarette section - or in other words, there are 10 people in the study that smoke 15 cigarettes per day - and of those 10 people, 6 people had heart attacks.
So, the probability of having a heart attack for people that smoke 15 cigarettes per day is 6 over 10, or 60%. Now let’s switch gears and talk about odds.
Odds compare the probability of an outcome occurring with the probability of an outcome not occurring, so the equation for odds is P divided by 1 minus P.
So, in our example, we’d use the probability of having a heart attack - which is 60%, or 0.6 - compared with the probability of not having a heart attack, which is 1 minus 0.6, or 0.4.
If we divide both sides by 0.4 we get 1.5 compared to 1. To make it easier to interpret, let’s multiply both sides by two, so we get a 3 to 2 ratio.
In other words, out of all the people that smoke 15 cigarettes per day, there are 3 people that have heart attacks for every 2 people that don’t have heart attacks.
Okay, coming back to the scatterplot... typically, in linear regression, a linear trend line is drawn to represent the pattern of data points on the plot, and that line is represented by the equation: y-hat equals b0 plus b1x1, where y-hat is the estimated value for the outcome variable; x1 is the value of the exposure variable; b0 represents the y-intercept; and b1 represents the slope of the line.
But in logistic regression, the trend line looks a bit different. That’s because the data points for logistic regression aren’t arranged in a straight line, so a linear trend line isn’t a good fit, or representation, of the data.
Instead, the trend line for logistic regression is curved, and specifically, it’s an S-shaped curve. And the equation for this S-shaped curve is P equals e, raised to the power of b0 plus b1x1, divided by 1 plus e, raised to the power of b0 plus b1x1.
At this point, you might be wondering what trend lines have to do with probability and odds. Well, the slope of a line at a specific point represents a certain probability - so a data point on a very steep slope represents a high probability and a data point on a very shallow slope represents a low probability.
So with an S-shaped curve, the probability of an occurrence - which in this case, is the probability of having a heart attack - is different at various points along the line, which makes it pretty hard to interpret.
For example, toward the bottom of the line, the probability is low, then the toward the middle of the line, the probability is higher, and then at the top of the line, the probability is low again.
And this is a problem because typically, we want to draw a line that represents the whole sample population, so that we can make conclusions about the whole sample population.
To fix this problem, we have to transform, or change, our data points so that they’re arranged on the plot in a more linear pattern.
To do this, we have to change our data from looking at probability to looking at odds, and we actually have to go one step further and change them to the logarithm of odds, or the log-odds.
So, if the basic equation for odds is P over 1 minus P, then to change our probability equation to an odds equation, we do P over 1 minus P equals - e, raised to the power of b0 plus b1x1 - over - 1 minus e, raised to the power of b0 plus b1x1.
If you simplify this equation - which we won’t show here, just for simplicity’s sake - what you end up with is P over 1 minus P equals e, raised to the power of b0 plus b1x1.
Okay, so that’s the odds equation. And to change the odds equation to the log-odds, we just take the log of both sides.
So, we end up with log of P over 1 minus P equals b0 plus b1x1. This equation, as you might recognize, is the equation for a linear trend line.
And, in fact, if we look at the newly transformed data, we’ll see that it’s now linearly arranged on the plot. Whew!
Sometimes the math can seem a little confusing, but luckily, statistical software does most of that work for us! Basically, the software spits out values for b0 and b1, which we can then plug into our equation.
For example, let’s say the software gives us a b0 of 0.05 and a b1 of 0.2, so the equation for the line would be y-hat equals 0.05 plus 0.2 times x1.
This equation tells us two crucial pieces of information. First, b0 tells us the y-intercept, which is the log-odds of the outcome variable when the exposure variable equals zero.
So, for example, a b0 of 0.05 means that people who smoke zero cigarettes per day have a 0.05 log-odds of having a heart attack.
Since the log-odds isn’t very interpretable, usually it’s converted it back to regular odds by exponentiating the log-odds with a base of e.
So, e to the 0.05 equals 1.05, which means that the odds of having a heart attack for people that smoke zero cigarettes is 1.05.
The other thing the equation tells us is b1, which is the effect size, or how much the log-odds of the outcome variable changes for every one-unit increase in the exposure variable.
For example, a b1 of 0.2 means that, on average, the log-odds of having a heart attack will increase by 0.2 for every one additional cigarette smoked per day.
In other words, if people that smoke zero cigarettes per day have a log-odds of 0.05, people who smoke one cigarette per day will have an odds of 0.05 plus 0.2, or 0.25; people that smoke two cigarettes will have a log-odds of 0.05 plus 2 times 0.2, or 0.45; and so on.
These numbers can also be converted to regular odds by exponentiating them by a base of e. For example, e to the 0.45 equals 1.6, so people that smoke two cigarettes per day have a log-odds of 0.45 of having a heart attack, and a 1.6 odds of having a heart attack.
We can also use this equation to predict the odds of having a heart attack for any specific number of cigarettes smoked per day.
For example, let’s say we want to predict what the log-odds and odds would be for a person that smokes 10 cigarettes per day.
To do this, we plug the number 10 into the x1 spot in our equation, so the log-odds equals 0.05 plus 0.2 times 10. 0.2 times 10 is 2, and 0.05 plus 2 is 2.05, so we predict that a person that smokes 10 cigarettes per day will have a 2.05 log-odds of having a heart attack.
If we exponentiate that, we get 7.8. So, we predict that a person that smokes 10 cigarettes per day will have a 7.8 odds of having a heart attack.
Now, this was all an example of simple logistic regression, since we only had two variables—one y-variable, which is heart attack or no heart attack, and one x-variable, which is the number of cigarettes smoked per day.
This is different from multiple logistic regression, where where there’s one y-variable and two or more x-variables. Typically, multiple logistic regression is used to control for confounding variables, or x-variables that distort the true relationship between the main exposure variable and the outcome variable.
For example, age is a confounding variable in the relationship between smoking and heart attacks, because older people tend to have more heart attacks than younger people.
So, let’s say you wanted to figure out how the odds of having a heart attack changes for people who smoke more cigarettes per day, controlling for age.
In this example, there are two x-variables—the number of cigarettes a person smokes and a person’s age—so, you’d have to use multiple logistic regression.
In this case, our log-odds equation will be: log of P over 1 minus P equals b0 plus b1x1, plus b2x2, where x1 is number of cigarettes and x2 is age.
Statistical software provides the y-intercept and slopes for each variable, so let’s say it gives us a b0 of 0.05, a b1 of 0.2, and a b2 of 0.01.
A b2 of 0.01 means that: the log-odds of having a heart attack increases by 0.01 for every one year increase in age. We can also use this equation to predict the odds of having a heart attack for person who is a certain age and who smokes a certain number of cigarettes per day.
So, let’s say we want to predict the odds of having a heart attack for a 50-year-old person who smokes 15 cigarettes per day, so we plug each number into our equation and get 0.05 plus 0.2 times 15, plus 0.01 times 50.
0.2 times 15 is 3, and 0.01 times 50 is 0.5, so the equation can be simplified to 0.05 plus 3 plus 0.5, which equals 3.55.
This means that a 50-year-old person who smokes 15 cigarettes a day has a predicted log-odds of having a heart attack of 3.55.
When we exponentiate this, we get 34.8, so a 50-year-old person who smokes 15 cigarettes a day has a predicted odds of having a heart attack of 34.8, which is a very high odds!
One limitation of logistic regression is that we can’t extrapolate, or make conclusions, about the data points outside the range of x-values that are in the sample.
For example, let’s say the range of cigarettes smoked by people in our study is 0 to 19, so 0 is the lowest number of cigarettes smoked by people in the study and 19 is the highest number of cigarettes smoked by people in the study.
Now, we know there’s a relationship between heart attacks and the number of cigarettes smoked in this range, but we don’t know if there continues to be a relationship between heart attack and smoking outside the range of our sample data - say for someone who smokes 40 cigarettes a day.
Alright, as a quick recap, logistic regression is a statistical method used to figure out the effect of an exposure variable, like number of cigarettes per day, on a categorical outcome variable, like having a heart attack.
Specifically, logistic regression finds the log-odds of an outcome, and uses the equation b0 plus b1x1. b0 represents the log-odds of the outcome when the exposure variable is zero, and b1 represents the average change in the log-odds of the outcome variable for every one unit increase in the exposure variable.
To make it more interpretable, the log-odds can be transformed to odds by exponentiating the log-odds with base e. For logistic regression, it’s important to keep in mind that you can only make conclusions about the data points within the sample range.