Effects of Incentive Amount and Type of Web Survey Response Rates

Up: presentation-increasing-response-rates-incentives

Effects of Incentive Amount and Type of Web Survey Response Rates

Reading: Coopersmith, Jared, Lisa Klein Vogel, Timothy Bruursema, and Kathleen Feeney. 2016. “Effects of Incentive Amount and Type of Web Survey Response Rates.” Survey Practice 9(1): 1–10. doi:10.29115/SP-2016-0002.

This study tests if different ways of giving gift card rewards can get more public school principals to complete an online survey. They tried three ideas: a bigger reward for finishing early, giving some money upfront, or offering extra money to those who hadn’t responded yet. The survey was part of the National College Ready Survey (NCRS), and all 1,680 principals were offered a $50Amazongiftcardaftercompletingit.Theexperimentscomparedextrarewardsagainstjustthe$50 post-survey reward. Only the early reward boosted responses at first, but none beat the standard $50 in the end. Here’s an easy breakdown in everyday words:

What They Did

• Why They Did It: Principals are busy, so getting them to do surveys is hard. Past research shows rewards (especially cash upfront) increase responses, but it’s unclear how well they work for online surveys. They wanted to find the best reward strategy for a web survey to use in future years.
• The Plan:
  • Who: 1,680 public school principals across the U.S., surveyed online for 15 minutes about college readiness (part of NCRS, funded by the Bill & Melinda Gates Foundation).
  • Standard Reward: Everyone was promised a $50 Amazon gift card after completing the survey.
  • Groups: Principals were randomly split into four groups (560 each, except one split further):
    • Early Reward Group (560 principals): Extra $50(total$100) if they finished within three weeks; $50 if later.
    • Upfront Reward Group (560 principals): $25giftcardwiththeinvite,plus$50 after finishing (total $75).
    • Nonresponse Reward Group (280 principals): $25giftcardofferedafter12weeksiftheyhadn’tresponded,plus$50 after finishing (total $75).
    • Standard Group (280 principals): Just the $50 post-survey reward (control group).
  • Survey Process:
    • Invites were emailed with gift card codes (or links to claim them). Non-responders got follow-up emails.
    • Some school districts blocked surveys or extra rewards, but these principals were kept in the study to avoid bias (called “intent-to-treat”).
  • What They Checked:
    • Response Rate: How many principals completed the survey, especially early vs. at the end.
    • Timing: How fast principals responded.
    • Compared groups using stats (chi-squared for response rates, t-tests for timing).
  • What They Expected:
    • Early reward would get more responses early and overall.
    • Upfront reward would boost responses more than just the $50.
    • Nonresponse reward would convince late responders.
    • Based on “leverage-salience theory,” which says rewards make people notice and value the survey.

What Happened

• Early Reward Group ($100early,$50 later vs. $50 standard):
  • At 3 Weeks: 30% responded (vs. 20% in standard group), a big boost (p < 0.05).
  • By End: 55% responded (vs. 57% in standard group), no real difference (p = 0.68).
  • Why: The $100gotmoreprincipalstoactfast,butafterthreeweeks,whenbothgroupsgot$50, the standard group caught up. Some in the early group might’ve felt discouraged after missing the $100.
• Upfront Reward Group ($25upfront+$50 after vs. $50 standard):
  • By End: 55% responded (vs. 57% in standard group), no boost (p = 0.49).
  • Why: The $25 gift card didn’t motivate more principals. Maybe they didn’t trust the Amazon code or thought they had to finish the survey to use it, making it feel like a post-survey reward.
• Nonresponse Reward Group ($25after12weeks+$50 after vs. $50 standard):
  • Overall: 57% responded (vs. 57% in standard group), no difference (p = 0.97).
  • For Late Nonresponders: After 12 weeks, 24% of 79 nonresponders who got the $25 offer finished (vs. 13% of 62 in standard group). This was a hint of a boost but not strong (p < 0.10) due to the small group.
  • Why: The $25 helped a bit with stubborn nonresponders, but too few got the offer to change the overall rate.

What It Means

• Key Points:
  • The standard $50 post-survey gift card worked best overall, getting 57% to respond.
  • Offering $100 for early completion got 10% more responses in three weeks, but the effect faded by the end.
  • Giving $25upfrontortolatenonrespondersdidn’tbeatthe$50 alone, though the nonresponse reward showed some promise for latecomers.
• Why the Early Reward Worked at First:
  • Doubling the reward ($100) grabbed attention and got busy principals to act fast.
  • But after three weeks, missing the $100might’vemadesomefeelitwasn’tworthdoingforjust$50.
• Why Upfront and Nonresponse Rewards Didn’t Work Well:
  • Amazon gift cards (not cash) might’ve seemed less valuable or confusing to claim online.
  • Principals might not have read the full invite to notice the $25 upfront.
  • The nonresponse reward helped a bit, but only a small group got it, so it didn’t change much overall.
• Tips for Online Surveys:
  • Stick with a solid post-survey reward ($50 worked here) for steady responses.
  • Use bigger rewards for quick responses if the survey is short (a few weeks), but expect a drop-off later.
  • Try nonresponse rewards for specific low-response groups, but test with more people.
  • Use cash or clear rewards (not gift cards) for upfront payments to avoid confusion.