Statistical Significance Calculator for Survey Results
Test whether two survey percentages or averages are really different. Z-test, Fisher's exact, Welch t-test, p-value, effect size and a plain-English verdict.
inquiry@globalvoxpopuli.com · globalvoxpopuli.com
How to use the significance calculator
- Choose what to compare: two percentages (e.g. men vs women who agree), two averages (e.g. satisfaction scores in two countries), or one percentage against a benchmark (e.g. is 54% really above half?).
- Enter each result and its base: the number of respondents it is based on.
- Pick a confidence level and whether you are testing for any difference (two-sided) or only for A being higher (one-sided).
The verdict tells you in plain words whether the difference is likely to be real. The details show the p-value, the confidence interval of the difference and the effect size, so you can judge whether a real difference is also a meaningful one.
What "statistically significant" means
A result is significant at 95% confidence when a difference at least this large would happen by chance less than 5% of the time if the two groups were really the same. That chance is the p-value. A p-value of 0.03 is significant at 95% confidence but not at 99%.
Significance is not importance. With 10,000 respondents per group, a 1-point gap can be significant and still irrelevant to the business. With 80 per group, a 12-point gap can be real and still fail the test. Read the effect size and the confidence interval alongside the verdict.
Which test does the calculator use?
| Comparison | Test | Notes |
|---|---|---|
| Two percentages | Two-proportion z-test, plus Fisher's exact test | Fisher's exact is shown for small cells, where the z-test is unreliable. |
| Two averages | Welch's t-test (or Student's t) | Welch's does not assume equal spreads, so it is the default. |
| % vs benchmark | One-sample z-test, plus exact binomial test | Use for "is it above 50%?" or "did we beat last year's norm?" |
Worked example
In a study of 340 nurses, 24% of the 180 in hospitals and 15% of the 160 in clinics report a product shortage. The difference is 9 points.
- Pooled proportion: 19.8%
- z = 0.09 ÷ √(0.198 × 0.802 × (1/180 + 1/160)) = 2.08
- Two-sided p = 0.038, significant at 95% but not at 99%
The 95% confidence interval for the difference runs from about 0.7 to 17.3 points, so the gap is real but its size is uncertain. Press Example to load it.
Common mistakes in crosstab significance testing
- Testing a subgroup against the total. The subgroup is part of the total, so the samples overlap and this test overstates significance. Compare the subgroup with everyone outside it instead.
- Testing two answers from the same people. "Brand A 45% vs brand B 38%" among the same respondents needs a paired test, not this one.
- Running dozens of tests. At 95% confidence, 1 test in 20 will be "significant" by chance. For large banner tables, use 99% or a correction such as Bonferroni.
- Ignoring weighting. Weighted bases should be divided by the design effect before testing.
Frequently asked questions
What p-value is significant?
By convention, p < 0.05 (95% confidence). Many market research tables also flag p < 0.10 (90%) as directional, and p < 0.01 (99%) as strong.
Should I use a one-sided or two-sided test?
Use two-sided unless you decided before seeing the data that only one direction matters, for example "does the new ad beat the old one?". A one-sided test halves the p-value, so choosing it after the fact inflates false positives.
How big a difference do I need for significance?
The calculator shows the smallest significant gap for your bases. With 500 per group at around 40%, it is about 6 points. To plan a study that can detect a given gap, use the A/B test sample size calculator.
Can I compare more than two groups?
Use the chi-square calculator to test a whole table at once, then this calculator for specific pairs.
Need the respondents, not just the numbers?
Global Vox Populi runs quantitative and qualitative fieldwork in 170+ countries through its own proprietary panels of consumers, B2B and IT decision-makers, physicians, nurses, patients, caregivers and payers. ISO 9001, ISO 20252 and ISO/IEC 27001 certified and HIPAA compliant, following ESOMAR and Insights Association guidelines.
Check feasibilityLink to this tool
Free to use in proposals, courses and articles. If it helped, a link back is appreciated:
<a href="https://globalvoxpopuli.com/tools/statistical-significance-calculator/">Statistical Significance Calculator</a> by Global Vox Populi