How to design better surveys for higher ed research
A new paper illustrates the benefits of a mixed-methods approach that combines qualitative and quantitative tools.
Abstract | Full article HTML | Full article PDF
Higher education research often gets divided into two categories: qualitative research that relies on more subjective methods like interviews and focus groups, and the numbers-based quantitative research that relies on surveys and measurable records. Combining the two methods still isn’t common practice, but can yield higher-quality research on complex topics in higher education.
That is the main finding of a new paper published today in Research in Higher Education. Written by three researchers at the Pullias Center, the paper promotes a mixed methods survey design process that the authors developed while studying a the effectiveness of a college transition program at three universities.
“Using mixed method is not only important for constructing comprehensive and meaningful research findings, it can also be used to create an effective survey instrument,” said Darnell Cole, one of the paper’s authors and a Pullias Center faculty member.
“We might really have missed the mark in terms of psycho-social outcome measures added to the survey if we had not interviewed students, faculty and staff,” added Adrianna Kezar, co-director of the Pullias Center and a co-author of this paper. “And we also learned more about the program to fine tune those items as well.”
In the paper, the authors outline an eight-step mixed methods process for designing surveys, from drafting and piloting a baseline survey to conducting focus group interviews and case study research, all aimed at refining the final survey. Using this approach in their own research allowed the authors to spot and address overlooked topics with new questions, as well as add more nuance and tailoring to existing questions.
“The mixed methods model we propose for survey design is useful for other higher education researchers who seek to develop surveys that are flexible, contextualized, and responsive to observations,” said Joseph Kitchen, Pullias postdoctoral research associate and a co-author of the paper. “Our mixed-methods model is a distinct improvement over relying strictly on the assumption that survey measures are appropriate for the particular phenomenon under study, only to realize later the limitations of those measures that were assumed to be appropriate.”
Despite the many benefits of mixed-methods approach, only about 5 percent of articles published in top higher education journals use mixed methods, the authors found.
This paper comes out of the Pullias Center’s ongoing mixed-methods study of the Thompson Scholars Learning Communities at the University of Nebraska, which seeks to explore, document and better understand whether this college success and transition program’s experiences translate into greater student success.
Abstract | Full article HTML | Full article PDF
Cole, D., Kitchen, J. A. & Kezar, A. (2018). Examining a comprehensive college transition program: An account of iterative mixed methods longitudinal survey design. Research in Higher Education. https://doi.org/10.1007/s11162-018-9515-1
The research for this paper was conducted with funding from the Susan Thompson Buffett Foundation.