We often get questions about what makes a change in survey responses significant or meaningful. Understanding how your results change over time is an important part of analyzing your data and setting clear and achievable goals, so below you'll find our guidelines for determining when a change is statistically significant. To learn more about using this information to set goals for growth, check out this article.
Before jumping into the table below, we recommend that you compare the sample sizes from your two most recent surveys to ensure they are comparable. If these numbers are very different, this can cause change in your data over time since you are not comparing the same size of respondents. You can view the number of responses from previous surveys in your Panorama platform.
After identifying your sample size, use the table to determine a reasonable percentage point goal for growth at the topic or question level. Here are the approximate sample sizes needed for a given change over time to infer significance. (This table and its values were calculated by our Research team at a 95% confidence level)
| Sample Size (# of respondents) | Change (in % favorable) |
| 3,500+ | +/- 1% |
| 900 | +/- 2% |
| 400 | +/- 3% |
| 220 | +/- 4% |
| 140 | +/- 5% |
| 100 | +/- 6% |
| 70 | +/- 7% |
| 60 | +/- 8% |
| 50 | +/- 9% |
| 40 | +/- 10% |
Don't forget: Just because a result is statistically reliable doesn’t mean it is practically significant. We defer to your judgment on what meaningful change is, in part because it depends on your efforts. A small change, for example, might be meaningful if it results from a small effort.