When you want to compare a rate to a goal or another rate, a statistical test will tell you if there’s a difference. If there’s only a small chance there’s no difference, that’s called a statistically significant difference. Most statisticians choose 5% for that small chance, which means 1 out of 20 tests will falsely say there’s a difference when there really isn’t.
Cautions Before Testing
Percentages
A percentage is the same as a rate. They’re usually “crude” rates, which mean they’re not adjusted. However, rates from the Behavioral Risk Factor Surveillance System (BRFSS) survey have been adjusted.
Enough Events
An adjusted rates should be based on 20 or more events. If it has less, it is unreliable and should not be compared. Consider the options described in Calculating Reliable Rates to make the rate more reliable.
Similarly, a crude rate should be based on 10 or more events.
Significance Versus Importance
These tests can identify a statistically significant difference, but “significant” has a different meaning here than in everyday conversations. A difference between rates is significant if it likely did not happen by chance. If the difference is not significant, that only means we cannot say if there is a difference. It does not mean there is no difference.
A significant difference may not be an important difference. If one county’s rate is 1% higher than another county’s rate, a statistical test could tell you if that difference is significant. But it will not tell you if it’s worth acting on.
Comparing Rates Against a Target Value
Sometimes rates are compared against a chosen target, such as a goal or limit.
First, calculate the 95% confidence interval for the observed rate. See Confidence Intervals for an Age-adjusted Rate (PDF) for details. If the target value is not in the confidence interval, then the difference is statistically significant.
Simple Comparison of Two Rates
The simplest way to compare two rates or percentages is to calculate the [95% confidence interval] for each. See Calculating Reliable Rates for details. If those two intervals do not overlap, then the difference is statistically significant.
This is a convservative test. That means the chance of falsely finding a difference is smaller than 5%. More accurate tests are described below.
Process for Choosing a Test
- What type of rates do you have?
- Age-adjusted: Use the simplified confidence interval test described above.
- Crude: Go to step 2.
- Choose a test:
- Standardized ratio: Use the test described in Confidence Intervals and Significance Testing for a Standardized Ratio (PDF).
- Test of difference: Go to step 3.
- Are the rates independent? See below for the definition of independence.
- No: Use the test described in Comparing Crude Rates or Ratios, Part I: Dependent Rates (PDF).
- Yes: Use the test described in Comparing Crude Rates or Ratios, Part II: Independent Rates (PDF).
Dependent Versus Indepedent
Two rates are dependent if the value of one affects the value of the other. A common case of dependent rates is when some events are included in both. For example, a county rate and the state rate are dependent, because all cases in that county were used for both.