User Satisfaction and Best Practices: Correlation in Live Online Reference: Virtual Reference Desk Conference

User Satisfaction and Best Practices: Correlation in Live Online Reference: Virtual Reference Desk Conference
Christina Peterson, Sarah Giffen West

Peterson and West discussed this very important topic, which seems to often be too often neglected in all library services, digital or otherwise: measuring user satisfaction.

SJSU started a live online reference program in the fall of 2001 as one of two academic libraries in Q&A Café, a now-defunct Northern California multi-library-type cooperative.  [We were part of it too, and it’s evolved into the California statewide AskNow project.]  In the fall of 2002, CSU formed an academic queue cooperative separate from the public library questions in Q&A Café.  They also began to experiment with web-based chat as a platform for virtual office hours.

Peterson began this research to find out how much information literacy instruction was actually taking place in the VR environment.  She wanted to see if there was a correlation of best reference practices to student satisfaction.  Feedback on student satisfaction would also be provided to colleagues in the library.

They used RUSA’s Guidelines for Behavioral Performance of Reference and Information Science Providers, VRD’s Facets of Quality for Digital Reference Services, ACRL’s Information Literacy Competency Standards for Higher Education, and something from RUSQ.

They evaluated 428 transcripts with satisfaction surveys from January 2003 to September 2005.  Using coding criteria and a coding thesaurus, they determined that 337 of the transactions were research transactions, 65 were ready reference transactions, and 25 informational transactions.  The survey asked several questions, but they only looked at the first question: “Were you satisfied with the answer you received to your reference question?”  Answers were multiple choice: satisfied, somewhat satisfied, or not satisfied.  They looked at the inclusion of instructional behavior in the interaction, including co-browsing.  They looked at whether or not there was any follow-up in the interaction (does this answer your question?  is there anything else you need? do you have enough information to continue on your own?).  They also looked at the mirroring of emotion, whether or not the librarian responded in a positive way to the user and not a negative way.  Example:
User: This research project is driving me crazy! 
Librarian: Don’t worry.  I’m sure we will be able to find you some materials on your topic.

Results
• 68.2% of transactions named sources that would be used
• 65.2% explained the overall search strategy and sequence
• 32.9% assisted with the construction of a specific search query
• 37.9% included a follow-up statement
• 32.2% mirrored positive emotion
• 28.3% explained how to use sources
• 23.6% used co-browsing
• 4.9% explained critical evaluation of resources
• 0.2% explained ethical and legal use of information

Correlation of Best Practices with User Satisfaction
(% shown is % of students who responded with “satisfied” or “somewhat satisfied” for sessions with these particular activities)
• names sources that will be used 72.7%
• explains overall search strategy and sequence 68.5%
• follow-up 41.2%
• mirroring of positive emotion 35.6%
• assistants with construction with specific search query 34.5%
• explains how to use sources 30.9%
• uses co-browsing 26.0%
In other words, 3 out of 4 times co-browsing was used the patrons replied in the survey that they were unsatisfied.  [Okay.  Is co-browsing really improving the user’s experience?  Do we really need co-browsing?  Is it the cat’s pajamas like so many people think?]

Someone pointed out that the survey asked about the student’s satisfaction with the outcome of the experience, while the factors they’re looking at are related to the process, not the outcome. She suggested that it would be perhaps better to ask students how satisfied they were with the process.  The two are certainly related, but there is a difference.

Future Goals
They want to compare satisfaction between mirroring of emotion and missed opportunities to mirror emotion.  They also want to code for other missed opportunities (like follow-up).  Finally, they want to look at the correlation between truncated sessions (technical difficulties, user or librarian logged out before the end, user chose to receive a response via email instead) and a lack of user satisfaction.  Finally, they’d like to look at the correlation of the duration of a session and the inclusion of instruction in the session.