Chapter 2. Trying to Measure Success: A Mid-size Academic Library Grapples with Assessing a Popular Tablet Loan Project

Stephen Bollinger, Nina Exner, and Octavious Spruill

Introduction

Customer support with technology has long been a mainstay of academic libraries. Tablets are an increasingly important technology across society and learning environments, and so libraries are responding with support and services for tablets and other mobile technologies. In 2010, the year that iPads were released, North Carolina Agricultural and Technical State University’s F. D. Bluford Library purchased its first set of iPads to begin a student iPad loan pilot, which was soon joined by a faculty loan project and finally an expanded student iPad loan project. The university’s strategic plan, Preeminence 2020, with its emphasis on support for intellectual climate, excellence in the classroom, partnering for increased efficiency, and support for creativity and professional skills development in emerging technologies, motivated the library to offer then-novel tablet technologies.1 For all of these reasons, the library has steadily grown its efforts to empower students and campus staff and faculty to become familiar with emerging mobile technologies.

However, this growth of efforts to meet customers’ needs can easily outpace strategic and assessment initiatives. Particularly in mid-sized universities, implementation often takes most of the personnel time, leaving little opportunity for assessment. At Bluford Library, the mobile lending and faculty and staff mobile technologies programs have been in place for four years. It is time to assess progress and decide whether to continue the program or phase it out.

Assessment Planning and Framework

Although the Bluford Library philosophically values a full culture of assessment, its desire to assess often outstrips practicality. The library purchased twenty iPads with the intent to circulate them to students and investigate their suitability in a higher education environment. Evaluation was a central component of the original proposal, but that initial evaluation by necessity focused on managing the staff time involved in supporting the project. Measures included tracking and optimizing the amount of labor per circulation and the number and nature of student support incidents. Additionally, the focus of the program was to allow the students the flexibility to truly evaluate the devices throughout their academic life, which meant they were encouraged to use them for coursework and entertainment purposes, initially with a three-day loan period.

With the success of the original yearlong pilot, a proposal was developed to expand the project. Thirty third-generation iPads were purchased for student circulation, followed shortly by ten fourth-generation iPads specifically for faculty and staff circulation, and new supporting technology (a charge-and-sync cart and Macintosh computer for device management) was deployed. A cost-benefit analysis revealed that upgrading the supporting technology would result in both the reduction of labor costs per circulation and the ability to expand the program without requiring additional staffing. The improved system cost $2,500 at the time, but saved $16.63 (equivalent) in reduced labor costs per circulation. A mere 151 unit circulations were enough to recoup the cost (in equivalent value of labor) and give 100 percent ROI. We had more than ten times that many unit circulations.

As a result of our cost improvements, the evaluation focus of this second phase shifted to the borrower experience. Each iPad included a link to an online user survey, and an observational study of the apps (tablet software applications) each student installed was undertaken. Each school year, a summative evaluation helped maintain the focus on providing a valuable service, benefitting the university community while remaining sustainable and manageable for the library.

Finally, in the third year of the program, the first equipment losses were sustained, prompting an investigation of the circumstances. High-risk students were found to be in their first semester and near withdrawal from the university, usually withdrawing in the middle of the school year. This last factor, withdrawal from the university, is the main reason why lost equipment could not be recovered and represents major losses for the program.

Bluford Library has now come to a decision point in these programs, to assess whether to grow or scale back the program. The project demands a midstream assessment rather than a full-cycle integrated assessment because it is an ongoing program. Therefore, this analysis needs a generalizable assessment approach to organize and track what is known so far. An ideal approach seems to be applying Kaplan and Norton’s original Balanced Scorecard approach in its broadest possible application to organize the findings.2

This broad use of the Scorecard applies Kaplan and Norton’s four dimensions: the financial perspective, the internal perspective, the innovation and learning perspective, and the customer perspective. This analysis looks at how these interact and attempts to document the linkages between perspectives. The goals are drawn from departmental, library, and campus strategic plans and are assigned as closely as they can be to a scorecard dimension. The measures that can be computed are also assigned, adding impressions where no direct measures are available but with notes so that these “soft” measures are to be given less weight.

Although many different Balanced Scorecard frameworks have been created since then,3 for these purposes the original is sufficient to lend some strategic structure to the assessment. This implementation is not rigorous, and the balanced scorecard is for insight, not scholarly rigor.4 The goal is not deep scholarship but to frame whether this program has been sufficiently valuable enough to continue. This is not a perfect example of evaluative analysis, but it is a realistic one.

Therefore, the analysis below is organized according to the four dimensions of the Balanced Scorecard. Based on the classic Scorecard approach, each section opens with a goals and measures table (or “scorecard”). This scorecard table lists the goals of the program that pertain to that analytical dimension and then describes what “measures” or data and observations we have about the goal.

The Scorecard Assessment

Financial Perspective

Evaluating the financial perspective encompasses not only the direct costs of purchasing, outfitting, and maintaining the devices, but also the indirect costs of staff time to process each circulation and provide support to borrowers (see table 2.1). Additionally, late fees and fines for damaged, missing, or incompletely returned devices need to be determined. Costs in staff time, workflow burden, and replacement delays also have impacts on customer and internal perspectives, so it is important to look at these indirect costs rather than only the direct costs of a tablet.

A primary initial risk, given the significant costs of purchasing the tablets, was deciding whether to allow the devices to leave the library building. One purpose of the initial pilot was to determine what loss rates might be for a larger program with more devices. The initial one-year pilot, with twenty tablets circulated with a three-day loan period, suffered only one damaged device, ironically from a faculty loan. Lost or damaged accessories, especially charging cables, accounted for more costs than repairing or replacing tablet devices.

The tablets themselves are not the only direct cost. Decisions made about the service affect direct costs such as apps, carrying cases and covers, support equipment and software, replacement accessories, and whether staff will be assigned or hired to sustain the program. Each of these decisions directly affects the other quadrants of analysis. For instance, providing a device without apps beyond the factory default was deemed to lower the educational and learning value of the program. Unfortunately, due to the novelty of the App Store ecosystem, purchasing apps was initially not possible. So only free apps were selected and installed in the initial configuration. Customer privacy and security were also priorities of the program. To ensure privacy and secure customer usage data, the library implemented centralized iPad management. This required a syncing and management computer to fully wipe all possible customer data between uses. A Macintosh computer would be the usual, and most efficient, platform to manage and sync data; to control costs, a new Macintosh computer was not purchased for this purpose. Instead, a repurposed existing Windows computer was employed for syncing and managing the iPads.

The direct costs of the devices and support equipment, while significant, were secondary in the financial analysis to the indirect costs of staff time and labor to support circulating the devices. Measuring the device processing time, length of each check-in and check-out interaction, and the impact on other services yielded indirect costs that need to be managed if the program is going to be sustainable. Initial device processing time was forty-five minutes per circulation, requiring a semi-attended device wipe and restore that could be done only one at a time. By reducing the content and apps loaded onto each device, at the risk of decreasing the borrowers’ experience, this was lowered to thirty minutes staff time per device per circulation.

Other customer service concerns also add to indirect costs associated with the program, most notably with novice users unable to connect to the campus wireless network. Direct observation of circulation transactions combined with staff tallying iPad support questions were used to measure and evaluate staff labor costs, revealing that over 30 percent of borrowers immediately requested assistance connecting to the wireless network, creating congestion at the Access Services desk. Ultimately insights like these were used to determine that investing in direct costs, such as a sync-and-charge cart and a Macintosh computer for device management, would pay for themselves by improving the borrower experience while further reducing staff time per circulation to fifteen minutes.

The final lens of financial analysis involves program fines and fees and balancing fair assessments against the program devolving into an inexpensive iPad rental program for borrowers. Borrower surveys, along with a high percentage of circulations yielding late fines, revealed a strong desire to borrow the devices for longer than the initial three-day loan. Loan renewals were dismissed as a solution both because of the program imperative to provide loans to as many borrowers as possible and because of the desire to ensure that renewals did not generate additional staff support transactions.

While the fees were lucrative to the library, the goal of the program was to self-sustain, not generate revenue. Generated funds were used to replace the corresponding missing or damaged accessories. In 2014, however, devices were unreturned for the first time, and analysis shifted to whether fines and fees were in effect discouraging scofflaw borrowers from returning the iPads. Throughout the program, late fees were capped at approximately $120, coupled with fines equaling the replacement cost of the iPad and accessories, with the goal being the return or replacement to circulation of the device. Unfortunately, the common thread of all six incidents were students who left the university, meaning registration holds and other last-resort measures were ineffective in pressuring them to return the device or pay the fines. As of this writing, only one of the six iPads has been recovered. This has led to internal business concerns, which run counter to the early internal business enthusiasm for the program.

Internal Business Perspective

In 2010–2011, the strategic cycle prompted the creation of a new campus strategic plan, Preeminence 2020, which in turn necessitated revisions of the library strategic plan.5 New committees and task forces were designated to examine the goals and objectives of the library’s strategy and spearhead initiatives to facilitate change. Among the proposed initiatives, the library administration established specific action plans addressing the technology needs of library customers in response to two of the major goals of Preeminence 2020.

Goal one focuses on the intellectual climate of the university. The main concept of goal one is to “create an intellectual climate that encourages the creative exchange of ideas and increases the quality of the professional environment.”6 The action plan that the library constructed was to “acquire and circulate new mobile devices in support of instruction, learning and scholarship.”7 From an internal business perspective, acquiring the first set of twenty iPads was a grand step forward toward encouraging the exploration of emerging technologies and their potential for intellectual exchange and creative discovery (see table 2.2).

Goal two focuses on excellence in teaching, research, and engagement. The main concept of goal two is “commit to excellence in teaching, research, public service and engagement.”8 The action plan that the library constructed was to “Partner with DoIT [the Division of Information Technology] to enhance faculty development in innovative approaches to use of technology in the classroom.”9 In light of successes in the initial phase of the iPad program, library and DoIT staff accomplished this by partnering to increase the number of iPads as well as deploy the management computer and cart, supplied through DoIT and implemented through library services. The library expanded the program again through relationships with other campus departments, including the Division of Student Affairs and the Academy for Teaching and Learning, which generously provided iPads for student use as well as the faculty and staff program.

The main goal of the iPad checkout program is to allow customers to experience using a tablet for study, research, classes, and personal use. Policies and procedures were created to ensure a smooth circulation process of the iPad devices. Even with strict rules and regulations in place, the iPads have been very popular with faculty and students. Library statistics show 1,965 iPad circulations over a three-year period. Many students thanked the library for the iPad service and gave positive feedback. Several students, faculty, and staff actually purchased an iPad after checking one out from the library.

iPad circulation statistics (from 2011 to 2014) show that customers are using the iPad equipment. Numbers show that customers preferred checking out the iPads over other technology equipment (Amazon Kindles and Sony E-readers). Since the introduction of the iPads, data shows the other equipment experienced a 42 percent decrease in usage. Incredibly, the data shows that the iPads were circulated over 1,400 more times than the other equipment. Unequivocally, this proves that the iPad program has fulfilled strategic hopes as well as building an innovative service with great popular acclaim.

Innovation and Learning Perspective

While tablet computers predating iPads have been used in library contexts, a primary motivator in developing the iPad loan project was to investigate how these new tablets could be useful to students, faculty, and staff in a higher education setting. A primary issue around encouraging borrower innovation and learning involves the configuration of the devices and what content and apps to preload. The decision was made to leave the device open so the borrowers could install apps and to provide a sampling of apps and content that would guide novice tablet users. Simply providing access to iPads did not meet the criterion of innovation. The challenge, however, is effectively evaluating whether the iPad loan project provided value beyond novelty (see table 2.3).

Several strategies were adopted, with varying levels of success, to quantify borrowers’ experience using the tablets. Informal sampling, such as observing checkouts and asking people returning devices about their experiences, was particularly effective. More structured approaches, such as providing a link and encouragement message on the device itself to a user survey, yielded a very low response rate (less than 1 percent, despite providing a gift card raffle inducement) but also provided insights beyond in-person interactions. Finally, traditional data sources such as circulation statistics are available for analysis. However, in this context they were not useful. The devices were in constant circulation and in high demand. This popularity met one minor goal of the program, that students were gaining literacy with tablet devices. However, answering the question of whether borrowers were gaining anything else from the program required different approaches.

Oddly, the most effective evaluation of innovation and learning was a less intuitive one, a census conducted upon return of the devices of the apps installed by users. Initially a one-off measure undertaken out of curiosity, this quickly became a regular activity to monitor the program. Each time the configuration of the apps and content on the iPads is updated, a census of borrower-installed apps is undertaken during the first few weeks of circulation to determine what the borrowers find useful on the devices. To maintain borrower privacy, apps were not opened nor was user data viewed, and devices returned locked (roughly 30 percent of circulations) were unable to be analyzed. Also, devices are returned around 20 percent of the time seemingly untouched, with no apps installed and no user customizations (lock screen or backgrounds unchanged, nor any apps installed). Initial results of the June 2011 census indicated that social media apps such as Twitter and Facebook were installed by over 60 percent of borrowers. Additionally, over 50 percent of borrowers installed a note-taking or student planner app. The result of each census then informs the next “build” of apps, usually done once each semester, with the goal of establishing a virtuous cycle of improved service and utility to borrowers.

As the app ecosystem has evolved and student sophistication with tablets matured, subsequent censuses have revealed new shifts. For instance, the August 2013 census revealed that students were installing cloud services apps, like Dropbox, Google Drive, and Microsoft SkyDrive (now OneDrive), revealing consistent productivity usages for the borrowed devices. However, the January 2014 census also revealed a strong trend towards casual and social uses of the devices, with dating, chat, and instant messaging apps (primarily Instagram, Skype, Snapchat, and Tinder) appearing in the census. This, combined with the first instances of borrowers failing to return the devices, may indicate that the service is no longer as innovative or fostering learning as it appears it once did. New considerations require balancing the loss of innovation and increasing costs against the service popularity.

Customer Perspectives

In the mission statement for the library, there is special emphasis on constant improvement of services, collections, technologies, and spaces. Near the start of the iPad program, the LibQUAL+ survey was administered to customers. The “need for newer technology” was the second most common desire of the respondents (see table 2.4). Out of the 340 comments, 37 percent of students desired newer technologies. Library administration studied the results and added providing students access to new technologies as a priority to the library’s strategic plan. Thus the library sought to acquire new technologies.

The university has also set high goals for customer service through the Preeminence 2020 plan. Goals one and two of the strategic plan have mandates that promote an intellectual climate and excelling in “teaching, research, and engagement.”10 Library administration adheres to those mandates. It placed a high priority on providing quality services to its customers. The library has taken great strides in achieving those goals with space and technology redesigns, including supplying its customers with new iPad mobile technologies and mobile-friendly spaces. With the analysis of circulation data, administration is able to measure the success of the iPad checkout program.

Increasing access to new technologies and customer satisfaction with library services are key goals for library administration. After reviewing data, administration was very pleased with the positive circulation statistics of the iPads. Data showed that customers increasingly took advantage of the program during the 2012–2014 school years. The campus faculty and staff increased their iPad checkout numbers. There was a 46 percent increase with their number of iPad checkouts, moving from 46 to 67 checkouts. Students (PhD, graduate, and undergraduate) had a 5.4 percent increase in their checkouts, moving from 758 to 805 checkouts. Therefore, the statistics showed the devices were being used and enjoyed.

The library solicits faculty, staff, and student feedback concerning iPad usage. Students were randomly surveyed during a three-month period. The results were very positive, with 100 percent of the students loving the iPad device, 87 percent of the students mentioning that it was their first time using one, and all agreeing that they would return and use it again.

Interviews with circulation staff members reinforced the positive feelings of the student body. Six different staff members noted the numerous and positive comments that they received from the student body over the past year. Many of the positive comments and feedback that students gave to staff members revolved around the use of apps, playing games, portability of the device, ease of use in the classroom, using Facebook, checking e-mail, using the Internet, using Google, and so on. In addition, staff members observed that when the iPads are all checked out, they receive frequent calls day and night asking for the next available iPad. Unquestionably, the statistics and the feedback from the staff and students reveal that the iPad program is a customer favorite.

Conclusion

The iPad program has meant a lot to the F. D. Bluford Library and to A&T’s campus. Its impacts have been felt throughout the organization, and the library learned many valuable lessons from it. Strategic, technological, and structural lessons were learned. But was it a good program, and will it continue to be valuable or should it be phased out?

Its popularity and the several improvements made along the way are strong arguments in favor of keeping the program alive. But growing losses and the costs they represent have created a concern. Additionally, as the third-generation iPad ages, hardware replacement costs are daunting. Increasing this concern, students’ innovative and intellectual uses of the iPad appear to have dropped off as the tablets become less of an emerging technology and more of an established one. Despite improvements, staff time in the program is still considerable and has started to increase since equipment losses have required staff time to attempt (unsuccessfully in several cases, as noted above) to reclaim them. So there are a wide variety of arguments for and against continuing the program.

Considering the popularity of the program and the library’s dedication to customer service, if costs were not an issue the program would certainly continue. Extending the program will require purchasing current iPads capable of running the next expected versions of the iPad operating system. If fiscal realities allow for ongoing purchases, this popular program will surely be continued. However, in the current economic environment, it is difficult to be certain whether the program will be phased out or not.

Notes

1. North Carolina Agricultural and Technical State University, A&T Preeminence 2020: Embracing Our Past, Creating Our Future: Strategic Plan 2011–2020 (Greensboro: North Carolina Agricultural and Technical State University, 2011), www.ncat.edu/about/forms-pdf/strategicplan-preeminence2020.pdf.
2. Kaplan and Norton, “Balanced Scorecard.”
3. See Gavin Lawrie and Ian Cobbold, Development of the 3rd Generation Balanced Scorecard: Evolution of the Balanced Scorecard into an Effective Strategic Performance Management Tool, 2GC Working Paper (Berkshire, UK: 2GC Limited, May 2002), http://2gc.eu/files/resources/2GC-WP-Dev3rdGenBSC-090311.pdf; Mike Perkins, Anna Grey, and Helge Remmers, “What Do We Really Mean by ‘Balanced Scorecard’?” International Journal of Productivity and Performance Management 63, no. 2 (2014): 148–69.
4. Hanne Nørreklit, Lennart Nørreklit, Falconer Mitchell, and Trond Bjørnenak, “The Rise of the Balanced Scorecard! Relevance Regained?” Journal of Accounting and Organizational Change 8, no. 4 (2012): 490–510, doi:10.1108/18325911211273491.
5. North Carolina Agricultural and Technical State University, A&T Preeminence 2020.
6. Ibid., 6.
7. F. D. Bluford Library, “Library Strategic Plan,” (internal document, 10 April 2012), 2.
8. North Carolina Agricultural and Technical State University, A&T Preeminence 2020, 8.
9. F. D. Bluford Library, “Library Strategic Plan,” (internal document, 10 April 2012), 3–4.
10. North Carolina Agricultural and Technical State University, A&T Preeminence 2020, 6, 8.

About the Authors

Stephen Bollinger is currently the head of library systems and assistant professor at the F. D. Bluford Library, North Carolina Agricultural and Technical State University in Greensboro, North Carolina. He received a bachelor of arts degree in graphic design and American history from Michigan State University and a master of science in information from the School of Information at the University of Michigan, Ann Arbor. He has worked as an Internet consultant for Steelcase, Borders Group, and General Motors as well as having over ten years’ experience in library information technology. His research interests include how mobile technologies complement higher education and technological support for student and faculty researchers.

Nina Exner is a librarian/associate professor at North Carolina Agricultural and Technical State University and a PhD student at the University of North Carolina’s School of Information and Library Science. She received her MLS in 1996 from North Carolina Central University and an MA from the University of North Carolina at Greensboro. Her research interests include librarians’ publication behaviors and researcher development. Her professional interests focus on reference support and information literacy at the postgraduate level and liaison work with new and emerging researchers.

Octavious Spruill is the head of access services and facilities at North Carolina Agricultural and Technical State University’s F. D. Bluford Library. His credentials include a BS in public health education, an MLS degree from North Carolina Central University, and an associate’s degree in computer information systems from Durham Technical Community College. His research interests revolve around library space planning, evolving library policies and procedures, library safety, and crime prevention.