GAO Management News

Vol. 31, No. 29, Week of April 26-30, 2004

Success Story: Gathering Data with Web-Based Surveys

It doesn't take a survey to see that GAO's approach to conducting surveys over the Internet is a success.

The last four years have seen "exponential growth" in the use of web-based surveys to gather data for GAO engagements and for various GAO mission support activities, according to ARM Managing Director Nancy Kingsbury.

In FY 2001, GAO conducted 16 online surveys involving about 2,500 respondents; in FY 2002, 35 surveys reaching about 20,000 respondents; and in FY 2003, 68 surveys involving about 40,000 respondents. During just the first six months of this year, GAO has undertaken 58 projects involving about 35,000 respondents; about 120 projects involving 70,000 respondents are expected by the end of the year. GAO's in-house capability to design and administer the surveys has saved the agency about $10.5 million in contractor costs, Kingsbury estimates.

"Other than major statistical agencies, we are one of the largest survey organizations making extensive use of web-based surveys," she said at a recent briefing on web-based surveys.

The tool that makes it all possible is GAO's Questionnaire Programming Language-a complete web questionnaire development system written by ARM's Kevin Dooley. Using a simple language to describe your questionnaire, the system automatically builds all the files needed to create a dynamic web site.

The web-based survey approach grew out of a program developed in the mid-1980s to support telephone interviews. Kevin Dooley adapted the program for use as an interactive survey instrument made available to respondents through the Internet.

Alice Feldesman and Steve Palincsar developed GAO's first on-line survey in 1999 to gather nationwide data on the status of Superfund cleanups in 1999 (GAO/RCED-99-245, July 30, 1999). Recognizing that a software package would make it easier to conduct similar surveys, a task force was formed to look at available programs, but didn't find any that provided the needed features. The task force then asked Kevin Dooley to modify his QPL program, originally designed for telephone interviews, to be used for online surveys. In 2002, GAO's Internet-based survey instrument was a finalist for the Federal CIO Council's Excellence.Gov award (see Management News, September 13-17, 1999 and February 4-8, 2002).

In recent months, QPL has been used to survey 200 NASA managers on the agency's lessons learned project and 24,000 students and faculty at the nation's three military academies on student life, among other projects. GAO's mission support offices rely on Internet-based survey for a wide variety of one-time and continuing information-gathering functions, including the annual employee feedback survey, Employee Advisory Council elections, student loan repayment applications, exit surveys, on-line course evaluations, training self-certification, and congressional feedback on products.

GAO has made its web survey software available to other organizations through its web site (/index.htm). Registered users include a number of overseas auditors general, U.S. and foreign universities, think tanks, and software companies.

Web surveys offer a number of advantages, but are not always the best way to gather data, Kingsbury said. Web surveys usually require less time to gather data and usually result in a higher response rate. They cost less to administer, since there is no need for postage and contractor data entry. Data can be also be monitored while the survey is underway. Paper surveys may be more appropriate when it is necessary to reach wider populations including people who may not have access to the Internet or where the respondent may have to combine responses from multiple locations.

GAO's QPL team is continuously looking for ways to improve the program. A recent enhancement has been the capability of producing an Internet-only supplement or CD-Rom containing survey results as well as files of tables for use in GAO printed products. Steps have also been taken to streamline survey administration, reduce the time needed to deploy a survey, and enhance data security and privacy.

The success of the program has been built on an effective partnership of ARM's web specialists and social scientists, Knowledge Services, ISTS, and GAO units, and an established process for developing and deploying Internet-based surveys, Kingsbury said.