Evaluation of a Chat Reference Service from the User’s Perspective: Virtual Reference Desk Conference

Evaluation of a Chat Reference Service from the User’s Perspective: Virtual Reference Desk Conference
Lili Luo

Luo (a doctoral student in University of North Carolina at Chapel Hill’s Information and Library Science program) discussed the evaluation process of NCknows, North Carolina’s statewide collaborative web-based chat reference service.

Research Context: NC Knows
The NCknows service (http://www.ncknows.org) started as an 18-month pilot project in 2004, with 19 libraries participating, including public, university, government, and community college libraries.  Luo’s evaluation project included 4 components:
1) Statistics from the 24/7 reference chat application
2) Quality Control (peer reviews of chat transcripts)responses from the public librarians rated higher than those from academic librarians
3) Administration (interviews with librarians)the librarians are largely satisfied with the service
4) Use and user satisfaction (measured through surveys and follow-up interviews)
Today’s presentation focused on #4: use and user satisfaction.

Research Questions they were trying to answer with the study
1) Motivation: what motivates users to use a chat reference service?
2) Satisfaction: what is the user’s level of satisfaction with chat reference service?
3) Use: how do users use the information provided by the chat reference service?

Study Design & Methodology
1) Exit surveys: pops up for users at the end of a chat session
2) e-mail and telephone interviews conducted 2-3 weeks after users’ chat sessions(willingness to participate was asked about on the exit survey)

Results
1) They received 393 exit survey responses, and conducted 73 follow-up interviews
2) 69% of users found out about the service at the local library or through library-produced publicity materials.  12% stumbled upon the serivce through online searching.
3) 48% of respondents hasd used desk reference before, 19% had used 3-mail reference, 33% had used telephone reference and 19% had never used reference services before.
4) When asked why users chose to use NCknows as opposed to other reference options, 47% said convenience.  15% responded that they had found that other means of finding the information they needed had proved unhelpful, so they sought out professional guidance.  13% used it purely out of curiosity.  7% used it because live chat was their preferred method of communication.
5) When asked what motivated their visit to the service, 51% responded that it was to answer a work or school related question.  32% sought an answer to a question related to their personal life.  Interestingly, 3% of users were seeking information on behalf of others.
6) When asked if they would recommend the service to a friend, 87.5% responded that it would be very likely.
7) About half the users in their survey were students (K-12, undergraduate, or graduate).
8) In the follow-up interview, they asked if the user had a change to use the information that NCknows provided, and 63% responded yes, 12% responded that the information only partially satisfied their needs, and 25% responded that the information provided was not used/useful to them.

Discussion and Future Work
By analyzing the data collected in the study, several suggestions to the service were made.
1) Since half the respondents were students, marketing should be targeted to them (and to other large user groups, such as businesspeople).
2) Service hours should be expanded to 24/7.
3) Users should be encouraged to use the reference medium most appropriate for their question (not necessarily chat).  The question then arises: how can the user identify that?  Or does that responsibility fall to the librarian?
She noted some limitations of the study, such as a non-response bias (those who did not respond may have had very different answers).  There was also a problem in that the survey only came up if the user clicked on the “Exit” option; it would not come up if the user simply closed out the window.
For future work, they plan to make methodological enhancements, such as formulating an alternative subject recruiting method.  They also plan to explore the service’s value: what users actually gain from using the information provided by the service.