COUNTER Reports Validation Tool: Requirements

IntroductionBoth COUNTER and NISO provide implementers of COUNTER Reports and the SUSHI protocol with detailed documentation and guidance on how to create compliant products; however, even with this assistance the rate of interoperability and compliance problems are still high.  These issues cost publishers and vendors collectively 100s of thousands of dollars per year in added support costs (handling service issues related to non-compliance), development costs (re-working code to address bugs and compliance issues) and potentially in loss of revenue due to the inability to deliver usage statistics in a timely manner.  For librarians, issues with COUNTER and SUSHI increase the cost to manage their e-resources and may compromise the quality of their collections due to the lack of accurate or timely usage data.The goal of this project is to create a web-based tool that will test compliance of COUNTER reports delivered in tabular format or delivered in XML via SUSHI.  Reports will be uploaded to the tool or a SUSHI harvest conducted through the tool and a series of test will be conducted on the result. Errors found will be highlighted in a report to the tester.  The tool will be made freely available on the web for all to use, though self-registration will be required to allow tracking of the tools use and improve error reporting.The tool will enable publishers, vendors and usage consolidation product owners to test their implementation of the SUSHI protocol and COUNTER reports during the development process, ensuring errors are caught and corrected before release. The result will be more efficient deployment and improved interoperability. This tool could also be used by the COUNTER audit to validate compliance to the required format specified in the COUNTER Code of Practice and the COUNTER SUSHI Implementation Profile, reducing the price of the audit and increasing the compliance rates by those publishers and vendors that use it as part of their testing.

Statements of ScopeThe following statements are intended to describe the high-level scope of the testing tool being built and are presented as a set of user stories (US).

Testing COUNTER Reports - tabular formUS1: As a librarian/publisher that has generated any standard COUNTER report in tabular (e.g. Excel) form I want to verify that the COUNTER report is formatted correctly and the data values it contains meets formatting requirements.  

“Standard Reports” means: Reports that COUNTER requires a provider to offer to be compliant (not all providers

must provide all reports).  See Table 1 in the COUNTER Code of Practice for the official list of reports).

In priority order these would be (up for discussion): Journal Report 1

Journal Report 5 Database Report 1 Book Report 1 Book Report 2 Platform Report 1 Journal Report 2 Database Report 2 Consortium Report 1 Consortium Report 2 Book Report 3 Journal Report 1 GOA Multimedia Report 1 Consortium Report 3 Book Report 4 Book Report 5 Book Report 7

“Formatted correctly” means: Report layout is exactly as specified in the Code of Practice Labels and column headings exactly match the Code of Practice, this includes

spelling, spacing and capitalization. Metric type names, when included on the report, exactly match the Code of Practice

(spelling, special characters and spacing) The report does not contain extraneous rows, such as blank rows within the body of

the report or extra rows at the end with non-report data or blank rows at the top before the headings.

Unless specified otherwise in the Code of Practice, the report does not contain items for which there was zero usage for the entire reporting period

Usage values are zero or positive numbers and do not contain empty cells unless the entire month column in blank signifying usage for that month is not available in the system (e.g. when month is a future month as when in June of 2016 a user asks for usage from January to December of 2016 -- June through December must all be blank, including total rows blank to be able to discern zero usage from data that is not yet available.)

ISxNs are valid or omitted if not available and do not contain values such as “N/A”, “not specified” or “unknown”

DOIs are omitted if not available and do not contain values such as “N/A”, “not specified” or “unknown”

Proprietary identifiers are omitted if not available and do not contain values such as “N/A”, “not specified” or “unknown”

Total rows and columns, when provided, add up correctly In Journal Report 5, years of publication back to 2000 are represented as individual

entries and the correct formatting for YOP labels used.

US2:As a librarian/publisher, if errors are found when validating my report I want to know what the errors are and where in the report they were found so that I may  efficiently report them to others or correct them.  

Testing SUSHI Harvesting of COUNTER ReportsUS3: As a librarian/publisher I want to be able to verify that a content provider’s SUSHI server (for which I have valid credentials) is operating correctly and returning properly

formatted SUSHI responses and COUNTER reports for any applicable standard reports available from that content provider.

“Operating correctly” means: The SUSHI Server doesn’t require non-standard authentication Server accepts valid request parameters COUNTER report that matches the request or proper error is returned The Server responds to error conditions with the correct error codes and messages

(automated tests the simulate errors conditions may be required).

“Properly formatted SUSHI Responses” means: Responses can be validated against the COUNTER_SUSHI Schema Error codes returned are compliant with those listed in the SUSHI Standard Responses adhere to the expectations described in the COUNTER SUSHI

Implementation Profile

“Properly formatted … COUNTER reports” means: The COUNTER XML can be validated against the release 4 versions of the

COUNTER XML schema and COUNTERElements XML schema. The testing tool should be prepared to handle multiple COUNTER Releases which requires for example different schemata for validation.

Responses adhere to the expectations described in the COUNTER SUSHI Implementation Profile (some examples follow).

Identifiers are included only when there are valid values Standard identifiers, like ISSNs and ISBNs are formatted correctly Datatypes, Categories and Metric Types are appropriate for the report and used in

valid combinations COUNTER XML does NOT include multiple-month totals or report totalsl  (each

“Period” element represents exactly one month) For Journal Report 5,

o PubYr attributes represent single years of publication in “yyyy” format, and all �years back to 2000 are included if the ReportItem had usage for any content published that year

o PubYr and PubYrFrom & PubYrTo are mutually exclusiveo PubYrFrom is less that PubYrTo and both are years in “yyyy” format.o isArchive attribute is present and set to “True” when year(s) of publication

represent content items in an archive.  IsArchive attribute may be omitted if its value would be “False”.

Unless explicitly required by the CoP, zero-usage items are excluded

“Standard reports” means: Reports that COUNTER requires a provider to offer to be compliant (not all providers

must provide all reports).  See Table 1 in the COUNTER Code of Practice for the official list of reports).

In priority order these would be (up for discussion): Journal Report 1 Journal Report 5 Database Report 1 Book Report 1 Book Report 2 Platform Report 1 Journal Report 2 Database Report 2 Consortium Report 1

Consortium Report 2 Book Report 3 Journal Report 1 GOA Multimedia Report 1 Consortium Report 3 Book Report 4 Book Report 5 Book Report 7

US4: As a librarian/publisher I want to be able to detect which reports are supported by that content provider’s SUSHI server (for which I have valid credentials).“Detect which reports” means that the testing tool would use the list of compliant reports from the COUNTER vendor registry for the Platform and run test requests for each to see if they are supported. Each report would be validated for compliance and a response of report not supported (even if a compliant error) would be considered an error for this test since the report is expected to be supported..

US5: As a librarian/publisher I want to be able to verify that a content provider’s SUSHI server (for which I have valid credentials) responds correctly to error conditions.

US6:As a librarian/publisher, if errors are found when validating my SUSHI request, I want to know what the errors are and where in the report they were found so that I may  efficiently report them to others or correct them.  

“What the errors are” means exact element and problems are enumerated and where possible expectations provided (e.g. Element <123> found X expected Y)

“Where in the report” means a meaningful position in the report with error is represented. (e.g. (e.g. Line 72, offset 1: element <123> found X expected Y)

“Know… and efficiently report them” means user should be able to review errors on-line and export or email or otherwise communicate those errors to others without having to retype.

US7 (optional): As a software developer I want to have access to open source SUSHI Client software to provide a guide and starting point for building my own client application.

“Open source” implies a project on github or similar where client code is available

“Starting point for building” implies that the SUSHI client portion of the testing tool is somewhat self-contained and can be used in other contexts.

Reporting ErrorsThe testing tool should capture and record the errors encountered during testing such that they can be viewed on screen or output as a report. Errors should be clearly articulated so that the reader can determine exactly what is wrong and where within the submitted file or SUSHI response that the error occurred.  If possible it would be nice to highlight the errors in the context of the report or SUSHI response.   (see US2 and US4).

Logging Errors

The testing tool will be used in two modes: developers will use the tool to validate their work as they develop new COUNTER

report or SUSHI implementations; and, auditors, librarians and content providers will test production version of reports and

SUSHI implementations to check for compliance.

The system should allow for logging of errors encountered; however, it is important to distinguish between developer-run-tests and tests run on production versions of COUNTER reports and SUSHI. The former will be expected to fail frequently and the latter not.

US8: As a user of the COUNTER Report Validation Tool I want the system to record if I am testing a production version of a COUNTER report or SUSHI or a version that is in-development.

US9:  As an administrator of the COUNTER Report Validation Tool, I want to be able to forward errors found in production systems to the content provider concerned and include details of the test, errors found and the contact information of the person conducting the test.

US9a: As an administrator of the COUNTER Report Validation Tool, I do not want to forward errors to the content provider concerned if the tests were being conducted on a test instance of the report/system, or if in test system or, if the tests were conducted by the staff of the content provider.

US10: As an administrator of the COUNTER Report ValidationTool I want to conduct a statistical analysis of tests run so that I can produce statistics by Platform, COUNTER Report, SUSHI vs Spreadsheet, Test version vs Production version, Success vs Failure; counting sessions, tests run, total errors encountered.

Architectural ConsiderationThe COUNTER Report Validation Tool should be accessible freely on the web, however, it should be possible to track who is using the tool.  Furthermore, the Testing Tool will be accessible from COUNTER’s new website (a WordPress based site) and from USUS (a WordPress based site.)

The new COUNTER website will also include a  Registry of Compliant Vendors and Publishers (The Registry) which will store information about compliant platforms, reports they support, admin URLs, SUSHI Server URLs, configuration notes, contact details.  The Testing Tool should interface with the Registry to simplify data entry for users running tests and to help accumulate statistics for tests run and test results at the Platform/Vendor level.

Running TestsBefore they are permitted to run tests, the user must identify themselves by providing the following:

Email (must be valid) Name Affiliation Role  (Librarian, Auditor, Developer, Content Provider) Test mode (Production System/Report, Test Version of System/Report

US11: As an administrator of the Testing Tool, want to track who is using it by requiring users to register with their email (username), name, affiliation, type of organization (library, vendor, auditor), normal role for test (librarians, auditor, developer, etc.) and self-assigned password. I further need the registration and recover of lost usernames and passwords to be completely self-service.

US11a: As an administrator of the Testing Tool, I want to require registering customers to complete a Captcha or similar to reduce the possibility of The Tool from being the target of spam or robotic use.

Once information has been provided, the user will have the option to upload a COUNTER Report, or conduct a SUSHI test.

COUNTER Report UploadThe system will prompt the user to identify:

What Why Required?

User ID/password User must identify themselves for the test. If they don’t have a login they can register.

Yes

Role for the test (one of “auditor, developer, librarian, other)

Helps with logging and reporting. Yes

Test mode (one of test-version, production-version)

Helps inform logging and reporting. Yes

COUNTER Release Version (optional)

If multiple COUNTER releases are supported by the tool, the release version is required.

(first release of tool will be Release 4)

Report (select from list of supported COUNTER reports)

Controls the flow of the testing No (if no entry, auto-detect from the file uploaded.)

Platform (option to search registered platforms, type platform name or leave blank

Ties the test to a given platform and allows confirmation that the Platform in the file is correct.

No (if no entry, auto-detect from the file uploaded.)

Browse for file to test Select file on the user’s system and upload for test

Yes

The user will then be given a file dialog to upload the file. Excel and TSV are permitted formats.

US12: As a user of the Testing Tool, I want to be able to identify the COUNTER Report I am planning to test then upload the report in Excel or TSV format.

Tests will be conducted based on the report requested.  See Appendix A.

US13: As a user of the Testing Tool, I want appropriate tests to be conducted on the file I uploaded so that I can see if the file complies.

US14: As a user of the Testing Tool, I want to be able to access the results of the test on screen or on download so that I may evaluate the results.

US15: As a user of the Testing Tool, I want to have the option to forward the test results to the contact listed for the vendor or to an email address that I enter (multiple addresses supported) and have the uploaded file included in the email as an attachment.

.

SUSHI ClientThe system will allow the user to select a Platform to test by using Registry, or type their own Platform details if not. Data elements needed for the SUSHI Client to work include,

Platform Name (Lookup/search Registry -- and auto-populate subsequent fields) Read-only notes on configuration SUSHI Server URL* Requestor ID* CustomerReference ID* Report* (select from list of valid reports for selected Platform as indicated in The

Registry. Allow entry of “Invalid Report”) Date Range* as From month-year and To month-year (default to 2 months ago)

When required data (*) is supplied, allow the user to submit the request.

US16: As a user of the COUNTER  Report Validation Tool to test a SUSHI implementation, I want to be able to leverage the SUSHI Server Registry to automatically populate the Platform Name, URL, notes on configuration, reports available so that my testing efforts are more efficient by just having to add or update data that is missing (I can overwrite any data auto-populated).

US16a: As a user of the COUNTER Report Validation Tool, I want to be able to manually enter SUSHI credentials in the event that Platform is not listed within the SUSHI Server Registry.

Testing of SUSHI Responses and Error Conditions will vary by report.  See US3 for more details on test expectations (driven by COUNTER SUSHI Implementation Profile) and section on Reporting Errors and Logging Errors.)

Efficient Modular DesignUS17: As a software developer that may be responsible for maintaining or expanding the COUNTER Report Validation Tool, I expect the code to be well organized and modular so that I may quickly diagnose issues, add new test cases or expand existing ones.

US18: As a software developer that may be responsible for maintaining or expanding the COUNTER  Report Validation Tool, I expect the code to include methods to automate

regression testing so that I can be assured my changes did not negatively affect other areas of the code

US19: As a software developer using the COUNTER Report Validation Tool I would expect the tool to provide access to underlying features as a set of microservices (provided I have a valid APIKey/credentials) so that I may directly interface with the testing framework used for my application.

User and System DocumentationUS20: As a user of the application I want to have documentation available to assist me in to use the tool effectively.

US21: As a software developer I want access to comprehensive, easy to use documentation about the application, its architectural design, how it is structured, its processing business rules, underlying schemas, underlying web services, error handling, etc. so that I may effectively maintain and extend the application as needed.

Appendix A: Sample Test Cases for COUNTER Reports in Tabular formBook Report 1

Test Name Expected Result

Check cell A1 for proper report name (case sensitive)

Book Report 1 (R4)

Check Cell B1 for proper report title (case insensitive)

Number of Successful Title Requests by Month and Title

Check Cell A4 for period covered label (case insensitive)

Period covered by Report:

Check cell A5 for correctly formatted period covered A5 contains period covered in the

form of 'yyyy-mm-dd to yyyy-mm-dd'

Check cell A6 for date run label (case insensitive) Date run:

Check cell A7 to make sure it is a date. Cell A7 contains rundate in the form 'yyyy-mm-dd'

Check cell A8 for the correct label (cell should be empty)

Check cell B8 for the correct label Publisher

Check cell C8 for the correct label Platform

Check cell D8 for the correct label Book DOI

Check cell E8 for the correct label Proprietary Identifier

Check cell F8 for the correct label ISBN

Check cell G8 for the correct label ISSN

Check cell H8 for the correct label Reporting Period Total

Check for correct date headings in cells I8:T8 First date column = first date in Period covered in A7

Check for correct date headings in cells I8:T8 Last date column = last date in Period covered in A7

Check for correct date headings in cells I8:T8 Make sure all headings in the range are dates

Check cell A9 has correct entry for Total for all Titles (case insensitive)

Total for all Titles

Check that totals in cells H9:T9 add up. Verify that totals are correct

Check that cells A10:A213 all have a title -- blank rows are not allowed.

All rows must have a title

Check the cells C10:C213 have a platform name - blank rows not allowed.

All rows must contain a platform name

Check for valid ISBNs in cells F9:F213 ISBNs are ISBN-13, have dashes and valid check-digits

Check for valid ISSNs in G9:G213 ISSNs, if present, have dashes and valid check-digits.

Check for blank count values in cells H10:H213. Blanks not allowed unless the entire column is blank.

All count cells must be non-blank

Book Report 2

Test Name Expected Result

Check cell A1 for proper report name (case sensitive) Book Report 2 (R4)

Check Cell B1 for proper report title (case insensitive) Number of Successful Section Requests by Month and Title

Check Cell B2 for section type label (case insensitive)

Section Type:

Check Cell B3 for section type value A non-empty cell

Check Cell A4 for period covered label (case insensitive)

Period covered by Report:

Check cell A5 for correctly formatted period covered A5 contains period covered in the form of 'yyyy-mm-dd to yyyy-mm-dd'

Check cell A6 for date run label (case insensitive) Date run:

Check cell A7 to make sure it is a date. Cell A7 contains rundate in the form 'yyyy-mm-dd'

Check cell A8 for the correct label (cell should be empty)

Check cell B8 for the correct label Publisher

Check cell C8 for the correct label Platform

Check cell D8 for the correct label Book DOI

Check cell E8 for the correct label Proprietary Identifier

Check cell F8 for the correct label ISBN

Check cell G8 for the correct label ISSN

Check cell H8 for the correct label Reporting Period Total

Check for correct date headings in cells I8:T8 First date column = first date in Period covered in A7

Check for correct date headings in cells I8:T8 Last date column = last date in Period covered in A7

Check for correct date headings in cells I8:T8 Make sure all headings in the range are dates

Check cell A9 has correct entry for Total for all Titles (case insensitive)

Total for all Titles

Check that totals in cells H9:T9 add up. Verify that totals are correct

Check that cells A10:A1250 all have a title -- blank rows are not allowed.

All rows must have a title

Check the cells C10:C1250 have a platform name - blank rows not allowed.

All rows must contain a platform name

Check for valid ISBNs in cells F9:F1250 ISBNs are ISBN-13, have dashes and valid check-digits

Check for valid ISSNs in G9:G1250 ISSNs, if present, have dashes and valid check-digits.

Check for blank count values in cells H10:H1250. Blanks not allowed unless the entire column is blank.

All count cells must be non-blank

Book Report 3

Test Name Expected Result

Check cell A1 for proper report name (case sensitive)

Book Report 3 (R4)

Check Cell B1 for proper report title (case insensitive)

Access Denied to Content Items by Month, Title and Category

Check Cell A4 for period covered label (case Period covered by Report:

insensitive)

Check cell A5 for correctly formatted period covered A5 contains period covered in the form of 'yyyy-mm-dd to yyyy-mm-dd'

Check cell A6 for date run label (case insensitive) Date run:

Check cell A7 to make sure it is a date. Cell A7 contains run date in the form 'yyyy-mm-dd'

Check cell A8 for the correct label (cell should be empty)

Check cell B8 for the correct label Publisher

Check cell C8 for the correct label Platform

Check cell D8 for the correct label Book DOI

Check cell E8 for the correct label Proprietary Identifier

Check cell F8 for the correct label ISBN

Check cell G8 for the correct label ISSN

Check cell H8 for the correct label Access Denied Category

Check cell I8 for the correct label Reporting Period Total

Check for correct date headings in cells J8:U8 First date column = first date in Period covered in A7

Check for correct date headings in cells J8:U8 Last date column = last date in Period covered in A7

Check for correct date headings in cells J8:U8 Make sure all headings in the range are dates

Check cell A9 has correct entry for Total for all Titles (case insensitive)

Total for all Titles

Check that cells A10:A50 all have a title -- blank rows are not allowed.

All rows must have a title

Check the cells C11:C50 have a platform name - blank rows not allowed.

All rows must contain a platform name

Check for valid ISBNs in cells F9:F50 ISBNs are ISBN-13, have dashes and valid check-digits

Check for valid ISSNs in G9:G50 ISSNs, if present, have dashes and valid check-digits.

Check for valid access denied types in cells H9:H50 All access denied types are valid

Check for blank count values in cells I10:I50. Blanks not allowed unless the entire column is blank.

All count cells must be non-blank

Journal Report 1

Test Expected Result Nature of failure

Cell A1 value Exactly equals: “Journal Report 1 (R4)” Failure

Cell B1 value Case-insensitive match to: “Number of Successful Full-Text Article Requests by Month and Journal”

Failure

Cell A4 value Case-insensitive match to: “Period covered by Report:”

Failure

Cell A5 value Contains data range in the form of: “yyyy-mm-dd to yyyy-mm-dd”From-date’s day is “01” and to-date’s day is last day of the month.

Warning

Cell A6 value Case-insensitive match to: “Date run:” Failue

Cell A7 value contains a date in the form of “yyyy-mm-dd” Warning

Cell A8 value Exactly equals: “Journal” Failure

Cell B8 value Exactly equals: “Publisher” Failure

Cell C8 value Exactly equals: “Platform” Failure

Cell D8 value Exactly equals: “Journal DOI” Failure

Cell E8 value Exactly equals: “Proprietary Identifier” without any newline character between the words.

Failure

Cell F8 value Exactly equals: “Print ISSN” Failure

Cell G8 value Exactly equals: “Online ISSN” Failure

Cell H8 value Exactly equals: “Reporting Period TotaI” Failure

Cell I8 value Exactly equals: “Reporting Period HTML” Failure

Cell J8 value Exactly equals: “Reporting Period PDF” Failure

Cells K8:<LastColumn>8 Contain headings in the format of “mmm-yyyy”Representing consecutive months.  If date formatted by Excel as a “Date”, all dates must be the first day of month represented.

Failure

Cell A9 Exactly equals: “Total for all journals” Failure

Cell B9 Is empty Failure

Cell C9 Contains name of Platform being processed Failure

Cells D9:<LastColumn>9 Equal sum of all values in column beneath (e.g. D9=SUM(D10:D<lastRow)).

Except if total for the month is blank, then the entire row must be blank.  Only ending columns can be blank (e.g. the user requested months not yet processed and those months are represented on the report -- they cannot have values.

Failure

Cells C10:C<last row of spreadsheet>

Exactly equal name of Platform in C9 with no empty cells.

Failure

Cell F10:F<lastRow> Are empty OR contain valid ISSN formatted with as nnnn-nnn?. Valid ISSN means final character passes check-digit test.

Failure

Cell G10:G<lastRow> Are empty OR contain valid ISSN formatted with as nnnn-nnn?. Valid ISSN means final character passes check-digit test.

Failure

Cells H10:<lastcolumn><lastRow>

Check column by column.  If the column total (row 9) is blankContains zero or positive integer value and not an empty cell, except in case where the column total is blank

Failure

Cells A10:A<lastRow> Are not empty.  Also test for empty rows at the end that may contain extraneous data -- if found, fail the report.

Failure

References:

COUNTER Code of Practice, COUNTER Website, URL: http://www.projectcounter.org/r4/COPR4.pdf

COUNTER-SUSHI Implementation Profile, NISO SUSHI Website, URL: http://www.niso.org/publications/rp/rp-14-2014/

SUSHI Standard, NISO SUSHI Website, URL: http://www.niso.org/standards/z39-93-2014/