What are the considerations that determine the nature of sample size in quantitative research?

Understanding Sample Size

A sample size calculation before beginning a study ensures that the planned number of observations will offer a reasonable chance of obtaining a clear answer at the end.64 This is of paramount importance in animal studies, in which sample size is limited by financial constraints, concerns over animal welfare, and limited laboratory space.65 For example, a groundbreaking experiment in 10 giraffes is of little value, when a sample size of 20 is needed for adequate power or precision. Similarly, why experiment on 200 chinchillas, when only 100 are adequate to test a hypothesis? Such considerations are by no means limited to basic science research. Why devote endless hours to abstracting data from 500 patient charts, when only 150 observations would suffice?

Calculating sample size is an essential first step in evaluating or planning a research study.66 Basic requirements for all sample-size calculations include (1) estimates of the smallest difference desired to be detected between the groups (minimal clinically important difference), (2) level of confidence that any difference detected is not simply due to chance (typically 95% or 99%), and (3) level of confidence that the difference detected will be as small as what was specified earlier (typically 80% or 90%), assuming that such a difference truly exists. In addition, sample-size calculations for numeric data require some estimate of the variability (variance) among observations.

Determining the minimal clinically important difference to be detected is based solely on clinical judgment. When comparing categorical data, the difference of interest is that between proportions (rate difference, seeTable 2.8); for example, an investigator may wish to know if success rates for two drugs differ by at least 20% for otitis media, but a difference of perhaps 5% may be important when treating cancer. In contrast, differences in numeric data are expressed as a difference in means; for example, a researcher may wish to know if a potentially ototoxic drug decreases mean hearing by at least 5 dB, or if a new surgical technique decreases blood loss by at least 200 mL.

Outcomes measured on a numeric scale require an estimate of variance to calculate sample size. Because variance is defined as the square of the SD, a method is needed to estimate SD to derive variance. If pilot data are available, some estimate of SD may already exist. Alternatively, one can “guess” the SD by realizing that the mean value ± 2 SD typically encompasses 95% of the observations. In other words, the SD of a set of measurements can be approximated as one fourth of the range of that set of measurements. Suppose you are interested in detecting a 200-mL difference in blood loss between two procedures, and based on your clinical experience, you expect that about 95% of the time you will see a difference that ranges from 100 mL to 500 mL. Subtracting 100 from 500 and dividing by 4 gives 100 as an estimate of SD. Squaring the SD yields 10 000, which estimates variance.

Conclusion

Sample size determination is an integral part of any well-designed scientific study. The procedure to determine sample size depends on the proposed design characteristics including the nature of the outcome of interest in the study. There exists a vast amount of literature on the topic, including several books. The modern computer environment also facilitates determination of sample size; software designed exclusively for this purpose is available. Many of the procedures depend on the normality assumption of the statistic. Modern computer-intensive statistical methods give some alternative procedures that do not depend on the normality assumption. For example, many people working in this field of study now prefer to use bootstrap procedures to derive the sample size. Uncertainty in specifying prior information of effect size has led to Bayesian approaches to sample size determination. These computer-intensive procedures seem to have several advantages over many of the conventional procedures of estimating sampling size.

Sample Size and Power

So far, we have focused mainly on the assessment of type I (or α) error in clinical research, defined as the probability of rejecting the null hypothesis when in fact the null is correct. Type II (or β) error is defined as the probability of accepting the null hypothesis when in fact it is false. In a study with type II error, results are falsely reported as negative, and thus a true difference is missed. This typically occurs when the sample size is insufficient.

This concept of a false-negative study emphasizes the importance of sample size estimation and statistical power (power is defined as 1 minus the β error). Sample size estimates should be performed before any observational or interventional study. Following are the key components of a sample size estimate for cohort studies or clinical trials and for case-control studies (that have a binary outcome):

Sample size estimate for cohort study or clinical trial

α error

β error

Incidence of outcome in unexposed subjects

Ratio of exposed to unexposed subjects

Minimum detectable relative risk

Sample size estimate for case-control study

α error

β error

Prevalence of exposure in controls

Ratio of controls to cases

Minimum detectable odds ratio

Some of these components warrant discussion. First, α error, by tradition, is set at .05, reflecting the intent to perform a study whose results will be falsely declared to be positive less than 5% of the time. Second, β error is usually set between .05 and .20, reflecting the intent to identify a true relationship at least 80% of the time when a relationship truly exists and, simultaneously, a willingness to miss finding a true relationship up to 20% of the time. This means that such a study is described as having 80% to 95% power. Third, the incidence of exposure or the prevalence of exposure can generally be estimated from the literature or pilot data. Last, the minimum detectable odds ratio or relative risk is that meant to be clinically relevant and within a biologically plausible effect of the exposure. In practice, there is a trade-off between wanting to detect as small a difference as possible and wanting to maintain a reasonable sample size (from a logistical and cost perspective).

Utilities to do the Calculations

Computing sample size from statistical theory can be a major challenge because: (a) the theoretical distributions of values are complex and asymmetrical, and (b) a different formula needs to be used for each experimental design. Methods and tables provided by Cohen (1988, 1992) and Borenstein and colleagues (2001), several books on the topic (Bausell & Li, 2002; Kraemer & Thiemann, 1987; Murphy & Myors, 2004), as well as programs such as Gpower (Erdfelder, Faul, & Buchner, 1996; Faul, Erdfelder, Lang, & Buchner, 2007) or SamplePower (Borenstein, Rothstein, Cohen, Schoenfeld, & Berlin, 2000) can be helpful, but there are many situations where those approaches are not readily applied.

Instead, it is possible to make the calculations with the aid of a series of utilities created with Microsoft Excel, each adapted to a specific experimental design. Calculations of sample size in relation to power are based on a normal approximation to the non-central t distribution (Wahlsten, 1991) for some tests, and a normal approximation to the non-central F distribution (Severo & Zelen, 1960) for more complex designs. With the utilities provided in Chapter 5, the user needs to choose the right utility for the specific application and then enter appropriate information into the yellow boxes in the spreadsheet. All other cells in the spreadsheet are protected so that they cannot be modified from the keyboard, and calculations take place behind the scenes. The user must have a good understanding of the basic concepts involved in power analysis but does not need to know calculus or matrix algebra. This is much like following a recipe in the kitchen. A person does not need a background in organic chemistry to be a good cook, but some knowledge of why onions go into the pot before the tomatoes can help to avoid embarrassing and unpalatable errors.

Sample Size

A study needs a sufficient number of participants to provide reliable conclusions about the specified outcomes and treatment effects. If the number of study participants is too low, the investigators could incorrectly conclude that there is no treatment effect.137 If the number of study participants is too large, it may delay the availability of information that is vital to patient care. In both cases resources will be wasted and participants will be exposed to unnecessary risk. A sample size calculation therefore should be conducted in the design phase of all studies. In qualitative and observational research, investigators should justify the reasons for their sampling frame (the population from which the sample is drawn).20 This should be based on the availability of interviewees or relevant data in an appropriate format, or the investigators’ expectations about the numbers of eligible participants and/or their estimates of acceptable 95% confidence intervals around the incidence of the primary outcome.138 In comparative studies, sample size calculations are based on the expected difference in the primary outcome between the groups (effect size), the variance in the effect size (for continuous variables), and the risk that the investigators are willing to accept of false positive (α; type 1 error) and false negative (β; type 2 error) findings.139,140 Effect sizes and variances can be estimated from the literature, pilot studies, statistical methods, or opinions about minimal clinically important differences.141 The planned sample size will also depend on the number of groups, number of anticipated dropouts, and statistical analyses planned. The protocol should provide sufficient information to allow replication of quantitative sample size calculations.138,140 In the anesthesia literature this is frequently not the case.142 Post hoc calculation of statistical power using the results of a trial is inappropriate; the width of the confidence intervals around the primary outcome is a better indicator of the reliability of the result.140

Sample Size

Sample size is mainly determined by the sample design, required accuracy of estimates, and resource constraints. For a particular design, sample size can be determined by the level of accuracy required, or confidence interval at a given confidence level desired (Cochran, 1977: 75–78). A pilot study can yield estimates of the parameters used in computation by providing useful information for determining sample design and sample size. An increase in sample size will reduce sampling error, although the relationship is not linear. On the other hand, a small sample is easier to manage and will introduce less nonsampling error. In statistical theory, it is sample size and not population size that determines the accuracy of estimates, unless the sample size is relatively large, say greater than 5% of the population. The resource constraints of a study enforce an upper limit on the sample size.

4.1 Basic Concepts

Power is the probability of rejecting the null hypothesis when a particular alternative hypothesis is true. Power equals one minus the probability of making a type II error. When designing studies, it is essential to consider the power because it indicates the chance of finding a significant difference when the truth is that a difference of a certain magnitude exists. A study with low power is likely to produce nonsignificant results even when meaningful differences do indeed exist. Low power to detect important differences usually results from a situation in which the study was designed with too small a sample size. Studies with low power are a waste of resources since they do not adequately address their scientific questions.

There are various approaches to sample size and power estimation. First, one often calculates power for a fixed sample size. The following is a typical question: What is the power of a study to detect a 20% reduction in the average response due to treatment when we randomize 30 participants to either a placebo or treatment group? Second, one often wishes to estimate a required sample size for a fixed power. The following is a typical question for this approach: What sample size (in each of the two groups) is required to have 80% power to detect a 20% reduction in the average response due to treatment using a randomized parallel groups design? The focus of this section is on the latter approach, namely, estimating the required sample size for a fixed power.

Sample size and power calculations are specific for a particular hypothesis test. One needs to specify a model for the data and propose a particular hypothesis test to compute power and estimate sample size. For continuous outcomes, one needs to specify the standard deviation of the outcome, the significance level of the test, and whether the test is one-sided or two-sided. Power and sample size depend on these other design factors. For example, power changes as a function of the following parameters:

Sample size (n): Power increases as the sample size increases.

Variation in outcome (σ2): Power increases as variation in outcome decreases.

Difference (effect) to be detected d: Power increases as this difference increases.

Significance level a: Power increases as the significance level increases.

One-tailed versus two-tailed tests: Power is greater in one-tailed tests than in comparable two-tailed tests.

By comparison, sample size changes as a function of the following parameters:

Power (1 – β): Sample size increase as the power increases.

Variation in outcome (σ2): Sample size increases as variation in outcome increases.

Difference (effect) to be detected δ: Sample size increases as this difference decreases.

Significance level a: Sample size increases as the significance level decreases.

One-tailed versus two-tailed tests: Sample size is smaller in one-tailed tests than in comparable two-tailed tests.

Sample Size

Sample size calculations for survival analyses are substantially different than those explained in Chapter 25 or in the first section above. The power of a time to event study depends not on the original sample size but on the number of observed events. Since an event may never happen for some subjects, we estimate the number of subjects needed working backwards from the total number of events we need at the end of the study. There are a variety of formulas that may be used, and these are outlined in survival analysis textbooks and a few papers.79,87–89 Information that will be needed to compute sample size is similar to that needed for the standard two-sample difference of means example described in Chapter 25 and earlier in this chapter. To determine sample size, we need to specify the desired power, significance level α, whether a one-tailed or two-tailed test is of interest, interim analysis plans, plus other study design information.

Sample-Size Calculations

Sample-size calculations should be linked to the statistical methods used in the analysis. Margaret Pepe has developed a rigorous methodology for sample-size calculations [2]. Here, we will provide examples of sample sizes for a continuous biomarker based on TPR and FPR using Pepe’s methodology.

For the sample-size calculation, we will determine if the TPR is above some minimally acceptable value for a given minimally acceptable FPR. The following assumptions are required for the calculations: significance level, power level, disease event rate, and the ratio of the variability of the biomarker in diseased and nondiseased patients. We assume the variances of the biomarker in diseased and nondiseased patients are equal, in order to provide us with the largest sample sizes. In addition, we assume a significance level of 5% and 80% power.

For example, suppose we are evaluating a new continuous biomarker for a disease with an event rate of 7%. The largest acceptable false positive rate is 5% (corresponds to specificity of 95%) and at that rate, the biomarker must have a true positive rate of at least 5% (TPR null), in order to be considered a useful biomarker. It is expected that the biomarker will have a TPR of at least 10% (TPR alternative). Given these assumptions, 3300 patients are required (231 diseased patients and 3069 nondiseased patients). A smaller sample size is required with a higher event rate and larger effect sizes (Fig. 2.2). Thus, 190 patients will be required for a biomarker with expected TPR rate of 40%, where the event rate of disease is 20%.

Sample Size

Sample sizes for PFE usually range from 0.5 to 10 g. Obviously, the amount of sample used must be large enough to ensure homogeneity and obtain adequate sensitivity for trace analyses. A small sample size (<5 g) is preferred so as to avoid compaction of the sample in the extraction cell. Moreover, the smaller the amount of sample used is, the greater is the cell volume that can be occupied by the extractant and hence the higher is the extractant/analyte ratio and the more markedly shifted is the partitioning equilibrium toward solubilization of the analytes, which facilitates their extraction.