ltr: Vol. 49 Issue 2: p. 30
Chapter 6: Annual Review
Jill Emery
Graham Stone

Abstract

In times of fiscal constraint it is very important to make sure all resources, whether established or recent acquisitions, are providing value for money. Value for money can be calculated in a number of ways, the most obvious being cost per download. Chapter 6 of Library Technology Reports (vol. 49, no. 2) “Techniques for Electronic Resource Management” covers other factors that should be taken into account, such as renewal date, currency exchange rate, usage transferred/added titles, access queries, coverage changes, license changes, cancellation policy, pricing model, and percent price increase. It recommends compiling a report for your staff, noting that it is not enough to compile a report of raw data. Subject teams should receive a background to the usage of the resource and check indicators in order to allow them to make quick and accurate decisions on what to do next.


In times of fiscal constraint, it is very important to make sure all resources are providing value for money. Be sure to include archival collections in annual review cycles to calculate how their use is going and what may need promoting to your campus. This goes for all resources, whether established or recent acquisitions. Value for money can be calculated in a number of ways, the most obvious being cost per download. Other factors should also be taken into account, such as:

  • renewal date
  • usage
  • access queries
  • license changes
  • pricing model
  • impact factor changes and review of Eigenfactors and SNIPs
  • overlap analysis with other resources
  • percent price increase
  • currency exchange rate
  • transferred and added titles
  • coverage changes
  • cancellation policy
  • hassle of business provisions by provider (late invoicing, ongoing platform problems, etc.)
  • DRM used
  • short-term loan versus permanent purchase within your demand-driven plan
  • unpurchased titles within your demand-driven plan

A number of these factors will now be looked at in more depth.


Schedule

As noted in chapter 3, it is vital to check the notice period required for any subscription, which may be as much as three months for a cancellation notice. If possible, add this date into your ERM system or system used for tracking the administrative metadata of resources. You may also discover at the point of renewal that the price increase is greater than expected, so it may be worthwhile to review prior to having to consider a resource for cancellation.

An added complication for e-resources, e.g., aggregated resources, is that unlike “traditional” serials, renewals can take place all through the year. If they are not noted in advance or set up with a tickler in an ERM system, some possible cancellations may be missed or vital resources may cease, which will result in complaints or nonuse from users.

If you subscribe to a large number of different e-resources, it may prove easier to review them in batches, e.g., every quarter. It is important to do a review of all resources, even the ones that are seemingly widely used and readily adopted by your librarians and patrons, just to ensure the resource is performing as expected. If your financial year runs from August to July, then group the renewals into the following periods:

  • August–October
  • November–January
  • February–April
  • May–July

In order to beat the cancellation clauses, it is best to consider each quarter at least two months in advance, i.e., look at February–May in December.

Around February or March, it is also advisable to schedule a planning meeting with each subject or liaison team to discuss all e-resources that they are responsible for. Use this meeting to discuss the previous year’s usage and any comments that may have arisen about the resources. This can be of great help in the annual planning process, as “at-risk” resources can be identified at this stage and flagged for possible cancellation, greater promotion and marketing, or further review at the time of renewal.

For demand-driven plans, you want to do your analysis prior to continuing with the ongoing maintenance of the MARC/XML records in your local catalog. It may be decided that some publishers are not performing well and should be traded out for other publishers, or some subject areas may not perform as well as others. The conclusion may be that it is better to wait and re-evaluate after a couple of years instead of within a single year of usage/access.


Confirm Costs and Any New Terms and Conditions

Many vendors and a few providers will send you a statement of account many months in advance of the renewal date. However, if this has not happened, you should contact the vendor or provider at least a month in advance, once you have decided on a schedule for the review of e-resources, to request a copy of the current agreement and find out renewal pricing. If you have set up a review cycle, then this request can come as early as you need to request the information. Reviewing agreements may take some time, so make sure you allow enough time to review a new agreement in case there are significant changes from a previously signed agreement.

You may wish to contact the vendor for other pricing options, e.g., to convert up from simultaneous users to a site license, or possibly to downsize the subscription. In order to assess the need for a change in the number of concurrent users, you should consult your usage statistics and any turn-away statistics reported.

Signing up for a multiyear deal may mean that you do not have to check the costs of the resource. However, some vendors require you to sign a new license each year, and you should certainly check that the content has not altered significantly. Check the new license against your existing contract, as unwanted changes often do creep in to new agreements.

If there is a significant price jump, try to find out if it is due to an increase in content or additional functionality added to the resource. Check the price increase against the pricing given in your agreement to make sure the costs are in line with a pre-established pricing structure. This is especially important with a multiyear deal.

For a demand-driven program, you may find that a publisher you were expecting to get all content from and have full purchasing power has decided to remove or change some of the content being offered, and the platform provider should be completely transparent to libraries about what content is made readily available within your plan. This change in what is being offered may mean you switch publishers used in your demand plan or try to find another platform in which the content is still available to institutional users.


Consider Usage Statistics

There is a lot of information about usage statistics,1 and while it should never be the only reason to cancel a resource, it can certainly be very influential in deciding if a resource could be reviewed and flagged for possible cancellation, if more training is necessary, or if the number of simultaneous users should be increased or decreased.

Always make sure when negotiating a contract that the vendor can provide COUNTER-compliant usage data. A list of COUNTER-compliant vendors can be found on the COUNTER website—if the vendor is not listed there, then it is not compliant, even if it claims to be!

Project COUNTER

www.projectcounter.org

A relatively new feature of COUNTER is the differentiation between current content usage and archival content usage. This is the essential difference between the JR1 report and the JR1a report. It is well worth the time to make separate reports of the archival usage and the current content usage.

When making the decision about the number of simultaneous users, check if the vendor can give you information on turn-aways—the number of users who could not get access to the resource as they exceeded the number of simultaneous users. If turn-aways are high, you may also see overall usage drop as users become put off by the lack of access.

Many smaller vendors, or those whose primary focus is on the corporate market, are not COUNTER-compliant. If they will not agree to go through the process and you are still happy to subscribe, you should ensure that they agree to provide usage statistics—even if this is on request or monthly by e-mail. Vendors who refuse to do this are putting their subscriptions at risk.

Usage reports that record hits on a resource can be misleading and do not always reflect real usage. However, if you have both COUNTER data and hits, you may see some interesting results, especially for full-text resources, e.g., if hits are high but COUNTER stats are low, there may be an issue. From this, you could assume that although the resource is well used, the COUNTER report indicates that users did not actually find the material useful or that there could be a linking problem and that users were not getting to the full text for downloading. This could then be taken up in negotiations with the vendor; it may be a point to bargain with.


Report to Stakeholders and Get Reports from Stakeholders

It is not enough to compile a report of raw data on cost, usage, and possible license changes and expect busy staff and patrons to be able to process this information quickly. There are a number of commercial packages that can assist in the compiling of reports, such as 360 Counter from Serials Solutions, ScholarlyStats from Swets, or EBSCONET Usage Consolidation from EBSCO. However, manipulation of the data in Excel requires some intermediate understanding both of how the statistics are being captured and reported from the provider and of how to best compile the information to make sense to your subject selectors and information resource management teams.

Serials Solutions, 360 Counter

www.serialssolutions.com/management/360-counter/R

Swets, ScholarlyStats

https://www.scholarlystats.com/sstats/default.htm

EBSCONET Usage Consolidation

www2.ebsco.com/en-us/Solutions/completemanagement/Pages/CMSlearnmore.aspx

Consider also performing an overlap analysis with other resources to see if maybe the content is available from multiple resources. Two good overlap-analysis and suite-of-resource-evaluation tools are the Jisc ADAT, which provides numerous ways to evaluate databases, and the CUFTS open-source serials management system.

Jisc ADAT

www.jisc-adat.com/adat/help_guide.pl

CUFTS

http://researcher.sfu.ca/cufts

An example of a basic report for an A&I resource would include a graph of usage over time (figure 6.1) and a snapshot of key performance indicators (table 6.1). The idea behind the report is to give subject and liaison teams a background to the usage of the resource and to check key performance indicators in order to allow them to make quick and accurate decisions on what to do next.

For other resources, such as e-books, it may be more important to show a comparison between different aggregators and purchased items. This can often be difficult, as, although many vendors publish Book Report 1 (BR1) and Book Report 2 (BR2) reports, they rarely make both available. This makes comparison very difficult. In the United Kingdom, the annual SCONUL statistics return recommends that the number of title requests (BR1) be multiplied by 5.4 to estimate the number of section requests (BR2).

SCONUL Statistics

www.sconul.ac.uk/statistics

Another recent implementation in the United Kingdom is the Jisc Journal Usage Statistics Portal (JUSP), which “provides a ‘one-stop shop’ for libraries to view, download and analyse their usage reports from NESLi2 publishers.”2 There are now over 100 libraries in the United Kingdom benefitting from this resource. A major benefit is that JUSP can combine usage reports from journals that are available on multiple platforms, e.g., publisher and intermediary platforms. This can be a major headache when trying to accurately compile reports by hand, or even through commercially available packages. Another benefit of JUSP is that it can automate the removal of journal archive usage reports from current subscriptions. This makes reporting on journal value for money easier, as you want to count only what you are paying for in an individual package and not on the archive, which will have been purchased separately and therefore needs its own separate report on return on investment (see below).

Journal Usage Statistics Portal (JUSP)

http://jusp.mimas.ac.uk

Multimedia reports are just now being incorporated in the latest version of COUNTER.3 If you have a provider for streaming media and it is currently not using COUNTER, be sure to tell that provider that with release 4, it too can become COUNTER-compliant.

Reporting on return on investment of archive packages and other one-off purchases can also be built into the annual review. Although these resources may have been the result of a one-off purchase, they still need to be reviewed annually to check the return on investment in order to see how long it takes for the archive to match the subscription cost per download. To do this, you should look at the cumulative statistics since the archive was purchased rather than the annual statistics. You can then use this to work out the cost per article download over a period of time (see table 6.2).

Many libraries have created some sort of troubleshooting mechanism for tracking problems with given providers and resources. Sometimes this is a simple e-mail reporting list or web page form system, and sometimes it is a sophisticated ticketing system that allows a manager to pull statistics or reports of usage problems quarterly and annually. Whatever mechanism may be used, it is good to capture that information in any evaluation of a given resource or product and report it back to the provider and to the stakeholders. Being able to say you are cancelling a product because XX number of people were unable to access the content or resource during YY time period goes quite a way to either getting a lower price or making the case for why a resource is not being as heavily used as it once might have been.


Make Your Choice

Although it is essential to review all resources, it is often fairly straightforward to renew many resources at a glance. However, the usage graph and key performance indicators shown in figure 6.1 and table 6.1 can highlight some resources where further investigation may be needed.

It is always useful to re-assess the market even for “essential” resources. For example, the resource in figure 6.1 and table 6.1 looks like a simple renewal, but what if a rival vendor had started to host the resource on a platform that proved more popular with the users? Or a consortial deal had been announced during the previous year, resulting in potential cost savings? Either example may improve the user experience or reduce expenditure and should be investigated during the review period.


Renegotiate or Cancel

Another key issue for renegotiation is to review the new license. You will need to check the existing license against the new one to ensure that no new clauses have been introduced or that the clauses that were changed in the previous license have not reverted to the original wording.

The introduction of ONIX-PL licenses into Knowledge Base + (KB+) and subsequent compatibility with Serials Solutions 360 Resources Manager4 discussed in chapter 3 offer the ability to automatically check licenses in the future. The speedy introduction of this feature by all ERM system vendors and the adoption of ONIX-PL by publishers and vendors will make a very time-consuming part of the renewal process a lot more efficient.

Often, if a decision is made to cancel a resource or to review it for a further year, many vendors and providers will be open to negotiation. They may be prepared to drop the price for one year while you try to build usage or narrow the number of users to accommodate less use than anticipated by a site license.

Remember that contract review takes time, and be sure to allow time for the negotiation to take place. Some vendors and providers may be open to shifting your renewal period or else extending your agreement for a shorter period of time in order to avoid cancellation to see if usage can be increased. Be creative when talking to a provider or vendor, as they will often be more open to trying new things than you expect at first. For instance, if an e-book provider has recently come out with a new platform for its product, see if you can trial the new platform with previously purchased e-book content from that provider.

In addition, there may be other funds available from faculty or research offices in order to share the costs—negotiation does not have to be just with the vendor! You may need to negotiate funding with departments or other areas on campus. If this is the case, you may need to produce the reports detailed above for senior managers in the library so that they can open negotiations with faculty—if you have already done them as part of the annual review, you will be able to move things along more efficiently!


Notes
1. Angela Conyers, “Usage Statistics and Online Behaviour (2), ” in E-Resources Management Handbook, ed. Graham Stone, Rick Anderson, and Jessica Feinstein (Newbury, UK: UKSG, 2010), doi:10.1629/9552448-0-3.2.2; Peter Shepherd, “COUNTER: Current Developments and Future Plans, ” in E-Resources Management Handbook, ed. Graham Stone, Rick Anderson, and Jessica Feinstein (Newbury, UK: UKSG, 2010), doi:10.1629/9552448-0-3.23.1
2. Jisc Collections, “Librarian Tools, ” accessed December 12, 2012, www.jisc-collections.ac.uk/Librarian-Tools
3. Project Counter, “Introduction to Release 4 of the COUNTER Code of Practice for e-Resources, ” April 2012, accessed November 6, 2012, www.projectcounter.org/r4/INTROR4.pdf
4. Liam Earney, “Jisc Collections Licence Information to Be Included in Serials Solutions 360 Resource Manager, ” news release, October 15, 2012, accessed November 6, 2012, www.jisc-collections.ac.uk/News/Jisc-Collections-licences-in-360-Resource-Manager

Figures

[Figure ID: fig1]
Figure 6.1 

Sample usage graph for an A&I resource



Tables
[TableWrap ID: tbl1] Table 6.1 

Sample A&I key performance indicators


Key Performance Indicators 2005/06 2006/07 2007/08 2008/09
Metrics Total searches 509 670 937 932
Mean searches (year) 42 56 78 78
Downloads per FTE user 0.03 0.04 0.06 0.06
£ Costs Total package $1,425 $1,500 $1,525 $1,575
Cost as % of total e-resources budget 0.58% 0.61% 0.55% 0.53%
Cost per search £2.80 £2.24 £1.63 £1.69
Cost per FTE user £10.53 £10.17 £10.16 £10.16
Other Information Total FTEs 15,000 15,250 15,500 16,000
E-resources budget (total per calendar year) $245,000 $245,000 $275,000 $300,000

[TableWrap ID: tbl2] Table 6.2 

Return on investment for an archive collection


Total archive costs as a one off purchase in year 1: $17,000
Usage—article downloads (JR5) Return on investment = initial investment / Cumulative usage (year 1 + year 2, etc.), e.g., cost per article download
Year 1 1,750 $9.71
Year 2 2,500 $4.00
Year 3 4,000 $2.06


Article Categories:
  • Information Science
  • Library Science

Refbacks

  • There are currently no refbacks.


Published by ALA TechSource, an imprint of the American Library Association.
Copyright Statement | ALA Privacy Policy