Sample size

Definitions & Key takeaways

In statistics and quantitative research methodology, a data sample is a set of data collected and/or selected from a statistical population by a defined procedure. The elements of a sample are known as sample points, sampling units or observations.
Watch video only
Typically, the goal of a study is to explore the relationship between an exposure and an outcome in a target population.
For example, let’s say we want to find out if Medication A, can lower blood pressure better than Medication B, which is the current treatment, in people with hypertension - or high blood pressure - who live in Perth, Australia.
But the population of Perth is around 2 million people, and almost a third of the population has hypertension, so that makes our target population nearly 670 thousand people.
It would take way too much money and time to include them all in the study, so instead, we have to select just a sample of them - which becomes our sample population.
And the sample population should be selected by randomization, so that we have a high chance of including people of ages, races, and socioeconomic statuses that reflect the target population.
But how many people do we choose? Choosing too many costs more time and money, and choosing too few means that they may not adequately represent the target population.
For example, let’s say we choose 20 people as our sample population for our study, 10 are given Medication A and 10 are given Medication B, and we check their blood pressures after five years.
In the Medication A group, 5 people have a lower blood pressure, and in the Medication B group, 2 people have a lower blood pressure.
This gives us an overall relative risk of 2.5, meaning that Medication A is 2.5 times more effective than Medication B in the sample population.
But that doesn’t necessarily mean that Medication A will be 2.5 times more effective among all of the hypertensive people in Perth.
For example, maybe the sample contained all women, and Medication A happens to work really well just in women - in which case we may be overestimating this effect.
Or what if it works really well in women, but even better in men - in which case we’re underestimating the effect. To figure out the perfect sample size, we need to know five things.
First, we need to know the current response rate, or the proportion of people who respond to the current treatment. For example, let’s say that 50 out of 100 people have lower blood pressure after five years of using Medication B, then the current response rate is 50% over 5 years.
Second, we need to know the estimated difference in response rates between Medication B and Medication A, based on previous research.
For example, if Medication A had a response rate of 70% over 5 years in Berlin, Germany, then we can assume the response rates might be similar.
So, if we consider that Medication B’s response rate is 50%, we’d say that the estimated response rate of Medication A is 70%, and the estimated difference in response rates is 20%.
Third, we need to know if we want a one- sided study or two- sided study. A one-sided study could ask a question like, does Medication A have a cure rate that’s higher than Medication B’s cure rate?
Alternatively, it could ask, does Medication A have a cure rate that’s lower than Medication B’s cure rate? Basically, a one-sided study can look for a new cure rate that’s either higher or lower than the current cure rate, but not both.
In contrast, a two-sided study might ask a question like, does Medication A have a different cure rate than Medication B?
Meaning that this one study, would look for a cure rate that might be both higher or lower than the current cure rate. Overall, two-sided studies are more common, because even if we suspect that a new treatment will work better than a current treatment, there’s always a chance that opposite will be true.
Fourth, we need to know what p- value will be used in the study. The p- value represents the probability that the result of a study happened by chance.
For example, we might find that Medication A works better than Medication B, with a p-value of 0.05. This means that there’s a 5% probability that these results happened merely by chance, and not because Medication A actually works better than Medication B.
Since 5% is a very small probability, we can conclude that Medication A most likely does work better than Medication B. Typically, the researchers decide on a p-value before they start a study, and the most commonly used p-value is 0.05.
The fifth thing we need to know is the power of a study, and it refers to how well a study can detect a difference between two treatments when a difference really does exist.
Most often, the power of a study is set at 0.80 - or 80% or higher. For example, if your study has a power of 0.80, then that means that your study will find a difference between Medication A and Medication B, if a difference exists.
Now, once we’ve identified the five things, we can use a sample size chart -like these ones - to look up what sample size is needed for the study.
Each sample size chart is slightly different, depending on the p- value, power, and whether or not the test is one- tailed or two- tailed.
One thing about sample size charts is that they tell you how many people are needed in each study group. In this case, we want to figure out how many people we would need in the Medication A group and in the Medication B group.
So, let’s say we want to do a two- tailed test with a p- value of 0.05 and a power of 0.80. And let’s add that Medication B has a current response rate of 20% and that Medication A has an estimated response rate of 45%, creating an estimated difference in response rates of 25%.
So, on the appropriate sample size chart, we’d first want to look at the row that matches the current response rate - 0.20 - and then find the column that matches the difference in response rates - 0.25.
In this situation, we’d need 53 people in the Medication A group and 53 people in the Medication B group, so a total of 106 people.
On both one- tailed and two- tailed sample size charts, as the estimated difference in response rates increases, the sample size decreases.
That’s because big changes in response rates will be a lot more obvious than small changes, so to find very small difference, you end up needing a lot of people in the study.
Alright, as a quick recap, choosing the right sample size for a study is important, because having too large of a sample size is expensive and having too small of a sample size may lead to results that don’t reflect the target population.
To calculate the right sample size, researchers use five details about the study - like the current response rate, the estimated difference in response rates, whether the test is one- or two- tailed, the p- value, and power - to choose a correct sample size chart, which can then be used to find the number of people who need to be included in each treatment group.