+ All Categories
Home > Documents > Colis 130 Testing With Nfc Projects

Colis 130 Testing With Nfc Projects

Date post: 06-Aug-2015
Category:
Upload: k-peng-tan
View: 33 times
Download: 2 times
Share this document with a friend
Popular Tags:
11
TESTING WITHIN NFC PROJECTS ‘GETTING READY FOR LAUNCH’ SEPTEMBER 2012 WHITE PAPER BY: SANNE KETELAAR, CRISTINA VINTILA, EMER VAN DER VOORT VAN DER KLEIJ AND ANTAL VAN KOLCK (TECHNICAL), CONSULTANTS FOR UL’S TRANSACTION SECURITY UNIT. This white paper stresses the importance of test- ing within NFC projects. Important roles and responsibilities like Test Authority, Test Organi- zation and Test Execution are explained, fol- lowed by a phased approach with Component, Interoperability and Ecosystem Testing. The pa- per presents tips and tricks for testing, based on the UL vision and on our experience with test- ing and NFC projects. Each NFC project is unique. Feel free to cherry-pick the information relevant for your project. Wishing to grasp the essence of NFC projects in a few minutes? The paragraph ‘In short’ on page 10 provides the summary. Keywords: Testing, Certification, Near Field Com- munication (NFC), Trusted Service Manager (TSM), Mobile Transactions. NFC Projects NFC projects are gaining popularity. One can feel the pressure in the mobile and banking industry to bring full functioning NFC capable mobile phones to consumers, to educate them how to use these and to establish a quality service that will quickly gain sufficient market share ahead of competition. In general the first NFC based service enables consumers to pay at retailers by tapping their mobile phone to pay- ment terminals. As next steps, loyalty, voucher, transportation, access services are targeted. First, one needs to define what to target for, business wise as well as technically, and have it documented in detailed requirements. To deliver a well-thought-through business case and business requirements, to translate these into unambiguous technical requirements and a successful launch plan is not an easy task. This becomes even more challenging when one is not yet familiar with all ins and outs of NFC ecosystems. Therefore UL strongly recom- mends starting an NFC project by obtaining the necessary knowledge on NFC. Next the strategy can be defined, e.g. through a series of workshops including experts at- tendance. This first phase should result in a clear direction to project members, when prop- erly formulating all requirements, defining the planning and carrying out all other common project activities. Next, a customized Trusted Service Manager (TSM) and Wallet are procured after a vendor se- lection process including preparations like writ- ing the Request For Proposal documents and score cards. Vendors should be required to test their systems thoroughly before they are deliv- ered. Upon delivery, the test process continues. Consumer satisfaction and trust in the new ser- vice are of highest priority as these are the keys for consumer adoption. As a consequence, one needs to be absolutely confident that the new service offers a satisfying user experience and is trustworthy. This requires extensive testing to ensure functionality, performance and secu- rity of the new service. NFC is a new technol- ogy. New collaborations will have to be set up 1/11 © 2012 UL All rights reserved. May not be copied or distributed without permission. UL and the UL logo are trademarks of UL LLC.
Transcript
Page 1: Colis 130 Testing With Nfc Projects

TESTING WITHIN NFC PROJECTS

‘GETTING READY FOR LAUNCH’

SEPTEMBER 2012

WHITE PAPER BY:

SANNE KETELAAR, CRISTINA VINTILA, EMER

VAN DER VOORT VAN DER KLEIJ AND ANTAL

VAN KOLCK (TECHNICAL), CONSULTANTS FOR

UL’S TRANSACTION SECURITY UNIT.

This white paper stresses the importance of test-ing within NFC projects. Important roles and responsibilities like Test Authority, Test Organi-zation and Test Execution are explained, fol-lowed by a phased approach with Component, Interoperability and Ecosystem Testing. The pa-per presents tips and tricks for testing, based on the UL vision and on our experience with test-ing and NFC projects. Each NFC project is unique. Feel free to cherry-pick the information relevant for your project. Wishing to grasp the essence of NFC projects in a few minutes? The paragraph ‘In short’ on page 10 provides the summary.

Keywords: Testing, Certification, Near Field Com-munication (NFC), Trusted Service Manager (TSM), Mobile Transactions.

NFC Projects

NFC projects are gaining popularity. One can

feel the pressure in the mobile and banking

industry to bring full functioning NFC capable

mobile phones to consumers, to educate them

how to use these and to establish a quality

service that will quickly gain sufficient market

share ahead of competition. In general the first

NFC based service enables consumers to pay at

retailers by tapping their mobile phone to pay-

ment terminals. As next steps, loyalty, voucher,

transportation, access services are targeted.

First, one needs to define what to target for,

business wise as well as technically, and have

it documented in detailed requirements. To

deliver a well-thought-through business case

and business requirements, to translate these

into unambiguous technical requirements and

a successful launch plan is not an easy task.

This becomes even more challenging when

one is not yet familiar with all ins and outs of

NFC ecosystems. Therefore UL strongly recom-

mends starting an NFC project by obtaining the

necessary knowledge on NFC.

Next the strategy can be defined, e.g. through

a series of workshops including experts at-

tendance. This first phase should result in a

clear direction to project members, when prop-

erly formulating all requirements, defining the

planning and carrying out all other common

project activities.

Next, a customized Trusted Service Manager

(TSM) and Wallet are procured after a vendor se-

lection process including preparations like writ-

ing the Request For Proposal documents and

score cards. Vendors should be required to test

their systems thoroughly before they are deliv-

ered. Upon delivery, the test process continues.

Consumer satisfaction and trust in the new ser-

vice are of highest priority as these are the keys

for consumer adoption. As a consequence, one

needs to be absolutely confident that the new

service offers a satisfying user experience and

is trustworthy. This requires extensive testing

to ensure functionality, performance and secu-

rity of the new service. NFC is a new technol-

ogy. New collaborations will have to be set up

1/11

© 2012 UL All rights reserved. May not be copied or distributed without permission. UL and the UL logo are trademarks of UL LLC.

Page 2: Colis 130 Testing With Nfc Projects

between parties that did not interact before.

Due to these complexities it can safely be stat-

ed that testing will take up a substantial part

of the available time in the start-up of any NFC

project and requires high priority.

NFC Ecosystem

An NFC ecosystem is a complex world with mul-

tiple actors, the most prominent being mobile

network operators (MNOs) and financial insti-

tutions.

Also, wallet vendors, application providers,

handset manufacturers, TSM(s) and Secure El-

ement (SE) manufacturers are part of this eco-

system. Smooth collaboration between these

parties on commercial, technical and security

aspects is challenging due to their different na-

ture. Figure 1 depicts an example of an NFC eco-

system in which an UICC (SIM card) functions

as the Secure Element (SE). Alternative options

are embedded SEs or secure microSDs. Storing

sensitive data on a SE gives a similar security

and confidentiality level as provided by the in-

tegrated circuit on a plastic payment card

Many of the systems in an NFC ecosystem han-

dle sensitive banking and customer data: the

bank holds the customer’s payment keys and

personalisation data, while the TSM delivers

this data to the SE. The MNO handles the SE,

although without having access to the cus-

tomer’s personalisation data. At the same time,

the MNO and its TSM handle data related to the

customer’s mobile subscription, which must

not be accessible to the bank. Therefore, it is

crucial that everyone trusts that the entire eco-

system is fully functional, stable and secure. To

add to the complexity, each actor has a variety

of components (multiple handsets, SE’s, appli-

cations) and a complex back-office. Numerous

standards are applicable, leaving room for dif-

ferences in interpretation. While some stand-

ards are still under development, this increases

the chance of misalignment among different

system components even more. Adherence to

all standards - especially a consistent imple-

mentation - is essential for interoperability in

an NFC ecosystem. Therefore the role of testing

and quality assurance is a vital one to achieve a

functional, stable and secure ecosystem.

After the initial commercial launch, the NFC

ecosystem will expand rapidly through new

services from existing and new business part-

ners, new handsets and new applications. As

a consequence, a strict certification regime

and structured test management needs to be

in place not only during the project phase, but

also during the operational phase. UL has iden-

tified two critical success factors for testing in

NFC projects:

- Key Roles and Responsibilities

- A Phased Testing Approach

2/11

Figure 1 UICC based NFC Project ecosystem

© 2012 UL All rights reserved. May not be copied or distributed without permission. UL and the UL logo are trademarks of UL LLC.

IBM
Highlight
IBM
Highlight
IBM
Highlight
Page 3: Colis 130 Testing With Nfc Projects

Key Roles and Responsibilities

In any NFC ecosystem, tests are performed by

different parties and their contracted vendors,

often across country and company boundaries.

As a consequence, among these parties the

test focus is lost easily, because no-one has the

end-to-end view. A clear definition of roles and

responsibilities assures that a complete end-

to-end view is maintained throughout the test

process. UL identifies a Test Authority, Test Or-

ganization and Test Execution role.

The Test Authority is responsible for declaring

the overall test policy and determining the test

and certification scope and objectives. Once

documented, these will guide certification and

testing for compliance in the entire ecosystem.

The Test Organization manages the test and

certification activities and their results. As

many parties are involved, smart communica-

tion and alignment with all parties is essential.

Lastly the Test Execution team performs the

actual testing and certification. The feedback

provided gives clear insight into the quality

level and builds trust in the entire NFC ecosys-

tem. The roles and responsibilities of the Test

Authority, Test Organization and Test Execution

are further detailed below.

Test Authority

The total ecosystem has one crucial mission to

accomplish: to gain and guard end customers’

satisfaction. In order to achieve this goal, the

ecosystem has to prove that it provides a satis-

fying customer experience and that it handles

transaction information securely, correctly and

reliably. All parties must be able to trust that

all sensitive data in the system is secure at all

times. It is the Test Authority’s role to guard this

overall goal.

The Test Authority role should be played by the

leading party initiating the NFC ecosystem. This

could be a mobile network operator (MNO), a

financial institution or a Joint Venture. This re-

sponsibility can even be outsourced to an inde-

pendent third party, which acts on behalf of the

leading ecosystem initiator(s).

3/11

Figure 2 Roles in testing an NFC ecosystem

Tip: Planning & terminology collaboration

Complexities like confusing terminol-ogy, different planning approaches and misaligned input data easily arise due to the many and diverse parties in an NFC ecosystem. As simple as it might sound, it is essential to ensure that terminology is aligned between parties. Otherwise this can cause se-rious confusion and misunderstand-ings. Additionally, make sure that all parties agree on a strict test plan-ning right from the start. Very often, banks have very rigid testing time-lines. Therefore, careful planning will safeguard that deadlines are met. De-tail in the planning is ensuring that all parties receive needed input data/documentation in a timely manner. The needed information should be clear and available upfront.

© 2012 UL All rights reserved. May not be copied or distributed without permission. UL and the UL logo are trademarks of UL LLC.

IBM
Highlight
IBM
Highlight
IBM
Highlight
IBM
Highlight
IBM
Highlight
Page 4: Colis 130 Testing With Nfc Projects

Four key tasks characterize the role of the Test Authority:

- Establishing the test policy,- Defining certification & test processes,- Supervising risk management,- Issuing approval statements.

Establishing the test policy, determining the certification and test scope and describing the test objectives comprise the first task of the Test Authority.

The second task focuses on defining the certi-fication and test processes and requirements applicable to all existing and new ecosystem components. The goal is to ensure high qual-ity of testing across company boundaries. All actors in the ecosystem must understand well to which requirements they must conform and how they are supposed to carry out the testing. Requirements originate from several sources.

In the first place, components have to comply with many standards and (payment) scheme specifications. For example, all payment schemes require that the security of Secure Ele-ment hardware and software must be certified by EMVCo. Functionally, a Secure Element must comply with the GlobalPlatform Card specifi-cation. A TSM that deals with customer’s per-sonalisation data must be approved by the pay-ment schemes, for example via MasterCard’s Global Vendor Certification Program (GVCP). In addition, payment schemes require systems that store, process or transmit card data to be PCI compliant. Note that standards mainly tar-get individual system components.

Secondly, to cover integrated components as well, additional requirements should be de-fined. These are primarily intended to assure interoperability, security and performance throughout the entire ecosystem. Such require-ments vary from validating that the user inter-face of a wallet application functions correctly with different handsets to checking that the operating system and security controls of a par-ticular Secure Element work well with all (third party-issued) secure applications intended to run on that Secure Element. Load and perfor-

mance checks, such as simulating many con-sumers subscribing to the same service simul-taneously, also fall into this category. Last but not least, the Test Authority should verify that the required certification and tests activities including security assessments are scheduled and conducted.

The third key task is supervising that risk man-agement for the test and certification process receives adequate attention from the test or-ganization. The Test Authority will report risks on a regular basis to upper management and ensures that appropriate mitigations are put in place.

4/11

Tip: Risk Analysis

Testing is about mitigating risks. It is rec-ommended to perform a risk analysis at the beginning of the project and update it at each milestone of the project. The risk analysis separately lists the main product/system risks and the main test project risks. Both lists are created through brainstorming and interview-ing people across the project, with the purpose of making a reality check into the status of the project and identifying the weak points of the service/product targeted for commercial launch. Based on this document, the testing teams de-cide which areas require more testing, and arrange that mitigations are put in place. Upper management may de-cide to allocate more resources and/or support mitigations in other ways, for example by explicitly accepting them. Thus, the risk analysis gives the man-agement insight in the residual risks that they have to manage, especially because not all risks can be mitigated by testing due to lack of time, resources or other reasons. The risk analysis is an essential communication tool during the testing process.

© 2012 UL All rights reserved. May not be copied or distributed without permission. UL and the UL logo are trademarks of UL LLC.

IBM
Highlight
IBM
Highlight
Page 5: Colis 130 Testing With Nfc Projects

Issuing approval statements concludes the list of key tasks of the Test Authority. Without ap-proval by the Test Authority, no part of the sys-tem can go live. Two types of approval should be distinguished: - The approval for first go-live,- The approval for new entries into the live ecosystem.

‘New entries’ refers to any new or updated component or service from any new or exist-ing actors. For example new handsets, new secure elements from new vendors and new applications from new or already participating service providers. Before allowing new entries to connect to a live ecosystem, the Test Author-ity judges the test and certification reports as proof that the new components or services comply with the defined requirements. The re-sults shown in these reports need to provide the right basis to declare approval. Therefore the Test Authority has to carefully specify the test and certification requirements, balancing between thorough testing and speedy time-to-market access. Evidently, this is the most ex-posed task catching everyones’ eyes.

Test OrganizationSince a new or existing NFC ecosystem con-sists of many actors with continuous certifica-tion and testing needs, a proper test organiza-tion needs to be in place to deal with the large volume of tests. The tests can be performed at various locations by multiple parties (e.g. by a company’s own test team together with their different vendors), but there should always be a test management organization in place to align and supervise that all required tests are execut-ed by the parties involved.

Following the high level test policy, scope and objectives set by the Test Authority, the Test Organization details the test levels and test procedures in one or more Master Test Plans. Usually such a document starts with detailing the test strategy and test governance followed by an overall planning of all necessary certifica-tion and test timelines. After the test scenari-os and scripts are in place, the test execution progress is closely monitored and frequently reported. Risks are continuously assessed and mitigations proposed. While these activities are

standard for any test project, the challenge in an NFC project comes from aligning and agree-ing on a common approach among the numer-ous parties, each having its own background.

Test ExecutionFollowing all preparation driven by the Test Authority and the Test Organization, the next step is the actual Test Execution. Evidently, expertise on NFC ecosystems, on specifica-tions and standards, on technical details be-hind the services subject to market launch are not only needed by the Test Authority and the Test Organization, but also by the Test Execu-tion team. While their key tasks of writing test cases and executing these are common to any test project, judging and reporting test results, analysing issues or defects efficiently requires a thorough understanding of NFC ecosystem specifics. To improve effective use of test time, the Test Execution team will run intake tests to judge, based on the entry criteria, if (sub)systems are having a fair chance in passing the planned tests. Evidently, the team will highlight opportunities to automate testing and imple-ment these. Above all, a structured and effective test pro-cess brings benefits in terms of cost control and reduced timelines. Therefore, UL recommends – in addition to the key roles and responsibilities highlighted above – a Phased Test Approach, which is explained further below.

5/11

Tip: Clear Entry and Exit Criteria

Entry and exit criteria are a set of con-ditions that should be met to either enter a particular test phase, or close a particular test phase. Clear agree-ments should be made between all parties to reach a common under-standing on the entry and exit cri-teria for each test phase. On top of that, Final Acceptance Criteria enable objective judgment to decide when the ecosystem as a whole is ready for commercial launch.

© 2012 UL All rights reserved. May not be copied or distributed without permission. UL and the UL logo are trademarks of UL LLC.

Page 6: Colis 130 Testing With Nfc Projects

6/11

UL Phased Test Approach No matter which NFC Ecosystem is being de-veloped, the key to achieving a successful sys-tem is to use a phased test approach. Figure 3 shows the test phases proposed by UL. An NFC test project should start with Component Test-ing, in which individual components of the eco-system are tested in isolation from each other. Interoperability Testing follows, through linking system components and ensuring the newly achieved sub-system performs as expected. Af-ter having tested system components in pairs successfully, the chain of systems is expanded further upon each passed test round until a complete ecosystem has been reached. At that moment the Ecosystem Testing can start, in which the complete system is tested end-to-end. In UL’s experience, this described approach reduces testing time significantly as defects are identified earlier in the test process. This allows for less re-testing and quicker defect analysis.

The same advantage holds for security test-ing if it is conducted continuously during each test phase. Therefore in the UL Phased Test Ap-proach, Security Testing starts during the Com-ponent Testing phase and continues during the period after launch. At first, each component needs to pass individual security testing, like

EMVCo certification. In addition the entire NFC ecosystem is covered through a security analy-sis on the documented technical implementa-tion. As soon as subchains of ecosystem com-ponents are available during Interoperability Testing, security checks must be performed on these integrated components. During Ecosys-tem Testing, security tests on the final version of each integrated system component analyses the risk of security breaches during commercial operations. Insight into security is crucial be-fore concluding that an NFC ecosystem is ready for go-live. Security assessments may include several methods, such as audits and (penetra-tion) testing.

Phased Test Approach: Component TestingComponent testing is generally understood as a phase in which individual components are evaluated on their functional and security re-quirements, while simulators, stubs and drivers are used to mimic the system components that interact with the component under test. In the Component Testing phase, mobile components (SE, handsets), mobile applications (payment applications and wallets), back end and front end systems (of the MNO, bank and TSM) as well as the acceptance infrastructure are tested

Tip: Defect Management Alignment

Often each actor in an NFC ecosystem has its own defect management pro-cedures, tools and test environment. Therefore, it is essential to agree on commonly shared defect tracking tools and discuss details of the de-fect management procedures before starting. Things to include are: defect definition (blocking, major, minor and cosmetic), escalation procedure and templates of reporting defects and progress. Additionally, an overview of the necessary test environment(s) is helpful to ensure availability at differ-ent test stages and phases

Figure 3 UL Phased Test Approach

© 2012 UL All rights reserved. May not be copied or distributed without permission. UL and the UL logo are trademarks of UL LLC.

Page 7: Colis 130 Testing With Nfc Projects

independently. The use of stubs and drivers al-lows for more thorough testing, for example by showing the exact input and output of each component. Component tests can be run in par-allel in order to find defects as quickly as pos-sible.

As several components involved in the NFC ecosystem are complex systems on their own, sometimes the component test is referred to as system test. The key point is that the test pro-cess of an NFC project starts with testing each system component individually before starting Interoperability Testing.

Usually component testing is performed by the party that delivers the system component. A vendor should be required to carry out a thor-ough component test before a component is delivered. An additional acceptance test by the purchaser is recommended to verify that the system component is ready to participate in the interoperability test. The acceptance test is not intended to judge acceptance of the com-ponent from a commercial perspective, but to judge technical readiness to participate into the next test level.

Component tests should have a special empha-sis on checking the interfaces and correspond-ing messages. The test cases should cover hap-py flows, alternative flows, negative flows and related error notifications in order to achieve sufficient testing. This will assure that many defects are already filtered out before actually integrating with other system components of the ecosystem.

Another aspect of Component Testing is the important role of certification by (payment) schemes and other authorities. As explained already, the Test Authority will determine the standards that each component has to comply with. In order to show that the component un-der test really complies with a given standard, it should obtain a certificate from the relevant authority. Some examples of relevant certifi-cates have already been given. UL can provide you with a complete overview of all necessary certificates for each component, and of the way in which your component can obtain these cer-tificates.

7/11

Tip: Test Tools – Simulators

Test tools and simulators can speed up testing within NFC projects by pin-pointing defects. By setting up simula-tors around an actual system compo-nent or a small group of components, issues can quickly be analyzed. These tools can replace system components not yet available for testing. Besides, new functionalities can be tested be-fore adding new system components to an existing NFC ecosystem. A short-list of useful tools provided by UL in-cludes the following:

• Collis TSM Test Tools: powerful tools to test an actual TSM system on Global Platform compliance, load & perfor-mance; The TSM Test Tool can simulate any TSM system and can be configured according to any NFC project chosen implementation. • Collis NFC Handset Interoperability Test Bench: Simulating a POS/Check-In Check Out device and UICC in order to ensure that the Handset & Wallet software work correctly in a Visa or MasterCard payment and transit envi-ronment.• Collis Handset Test Suite SWP/HCI: validating GCF/PTCRB compliance to ETSI SWP/HCI protocols; confirming interoperability between handset and UICC • Collis EMV PVT: verifying correct per-sonalization of contact & contactless cards including Secure Elements• Aspects Spy: spy on the communica-tion between handset – UICC – con-tactless reader and measure perfor-mance.• Aspects Handset Test Platform: sim-ulating the UICC, testing the handset

© 2012 UL All rights reserved. May not be copied or distributed without permission. UL and the UL logo are trademarks of UL LLC.

Page 8: Colis 130 Testing With Nfc Projects

Actually, all relevant schemes and authorities have accredited UL laboratories for carrying out certification testing. This includes EMVCo, Mas-terCard and Visa, but also GlobalPlatform, GCF/PTCRB (for certification against ETSI standards) and Common Criteria (for security evaluations.)

For each component test, the component un-der test should have obtained the right certifi-cates before testing begins. The vendor of the component is responsible for obtaining these certificates. This reduces the risk of invalid test results and increased test timelines from re-runs of tests. In an ideal world, showing the right certificates would even be completely sufficient for a component to pass component testing. However, as already explained, stand-ards always leave some room for interpreta-tion. Moreover, many standards in the world of NFC are still under development. Therefore, UL strongly recommends carrying out additional Component Testing even for components that have obtained all relevant certificates.

Benjamin Franklin: “I did not fail the test; I just found 100 ways to do it wrong”

Phased Test Approach: Interoperability TestingWhen two or more components of the eco-system have passed the exit criteria of their Component tests, the integration of these com-ponents can start. Interoperability Testing veri-fies that, once integrated, system components work well together. While this test level is often referred to as integration testing or system in-tegration testing, in NFC projects this phase not only verifies that system x works as intended with system y, but also that all different types of system x work well with all types of system y. For example several brands, types and builds of handsets, secure elements and third party ap-plications need to work together flawlessly in order to create a positive customer experience. Going through an integration process in steps will enable finding root causes quicker, since less system components are involved. Evidently, one needs test tools, like stubs and drivers and spying tools. Stubs and drivers simulate com-ponents that are still missing. Spying tools al-low a close look at the communication between two components, in order to determine who’s at fault.

By testing pairs of components in different setups, one can quickly identify if correct mes-sages are exchanged and correct decisions are made by each component, for example when a bank TSM is connected to a MNO TSM. As a consequence, several interoperability tests are being performed. After having verified inter-operability in pairs, the integrated subsystem will be connected further, continuing interoper-ability tests. For example, adding several user interface applications and secure applets to in-tegrate with subsystems of different handsets and SEs. Eventually, this leads to a completely connected NFC ecosystem, ready to enter the next test phase.

According to our experience, one of the criti-cal complexities in NFC projects is ensuring that several business processes are migrated from batch processes into real time executed processes or at least carefully aligned with real time executed processes and the user experi-ence. The interoperability phase offers the per-fect setting to check carefully if responses given with long time intervals are properly triggering subsequent actions.

Phased Test Approach: Ecosystem TestingWhen the whole NFC system has been estab-lished, end-to-end tests are to be performed to ensure the NFC ecosystem actually serves its

8/11

Tip: Continuous Security Testing

A proper security testing process should be spread across the entire life cycle of the NFC solution, from the design until the launch phase, and continuing dur-ing operation. UL advises that security testing includes verification of the de-sign, vulnerability testing and harden-ing checks of the implemented solution and Ethical Hacking of the production environment before launch. Once the ecosystem is in production, regular and ad-hoc security tests should be per-formed to ensure the expected securi-ty level is maintained during the entire life cycle of the solution.

© 2012 UL All rights reserved. May not be copied or distributed without permission. UL and the UL logo are trademarks of UL LLC.

Page 9: Colis 130 Testing With Nfc Projects

purpose. As such, Ecosystem Testing shall be the last and final check before starting a com-mercial launch. Sometimes Ecosystem Testing is referred to as pre-deployment or end-to-end testing, because the tests are to be executed in a production-like environment. It is common to differentiate end-to-end testing into separate test activities, like a User Acceptance Test, Ethi-cal Hacking, Operational Testing and a Friendly User Test, while the latter three tests are actual-ly run in the real production environment. Such break down assures proper test attention when verifying the required user experience, security and business processes. The Ecosystem Testing phase will cover as a minimum all these tests that are detailed further below. In theory this test phase will take a short time following the concept that all requirements have been clearly decided before Component Testing started, enabling extensive testing of system compo-nents in the earlier test phases. However in re-ality end-to-end testing usually happens to be highly time-consuming because requirements are adjusted in a late phase of a project or be-cause earlier testing for several reasons missed to detect or resolve serious defects. The novelty of an NFC ecosystem leads to new insights in the due course of an NFC project, increasing the likelihood of above main risks for a delay of the Ecosystem Testing period. Therefore it is impor-tant to carefully plan time and resources for this testing phase, too.

User Acceptance TestThe objective of the User Acceptance Test is to verify that the customer experience meets the business expectations. The tests are designed from the end user perspective.

Ethical HackingDuring ethical hacking, the vulnerability is tested by attacking the system as a real hacker would do. An ethical hack involves a combina-tion of automated scanning and expert manual testing of the system to find security breaches. When engaging into penetration testing, very clear engagement rules need to be document-ed and signed to limit the actions of the tester. For example, is the tester allowed to cause a system crash, or just gain access and prove that vulnerability is present? Is the tester allowed to use social engineering to gain physical access

to the data center and steal sensitive informa-tion? Often ethical hacking takes place during the Ecosystem Testing shortly before the go-live, but it is recommended to perform ethical hacking already during earlier test levels in or-der to prevent unexpected security risks shortly before launching a service commercially.

Operational TestingOnce the testing in controlled environments has finished and the necessary certifications have been obtained, the NFC solution goes into “production”. Operational Testing targets verifying that the ecosystem functions and performs as intended in its live environment. Running this test from a business perspective ensure that business process flows like Cus-tomer Care, Administration are checked end-to-end. Have all procedures been documented in line with the actual required execution? Are all databases correctly and timely updated, for example when a consumer changes his home address? A payment scheme will request proof that a payment transaction at the point of sale terminal leads to the expected deduction of the consumer’s bank account. Sometimes this test level is referred to as Dress Rehearsals.

Friendly User TestBefore going live often a pilot or a Friendly User Test is performed: a sample group of ‘friendly users’ (for example employees of the involved bank or the MNO) will test the end product and provide feedback. This stage of testing repre-sents the beginning of the commercial roll-out, but with a smaller number of customers. After all systems are integrated and tested properly, it is time to go live. After launch, continuous up-dating and support should be provided by each party.

9/11

© 2012 UL All rights reserved. May not be copied or distributed without permission. UL and the UL logo are trademarks of UL LLC.

Page 10: Colis 130 Testing With Nfc Projects

Phased Test Approach: Post Launch TestingAfter launch, testing should not stop. Consum-ers expect to be able to use the service(s) of-fered through the NFC ecosystem 24 hours, 7 days a week. Satisfying this consumer expec-tation requires a functional, stable and secure NFC ecosystem at all times. Companies will continuously introduce new functionality to the live NFC ecosystem requiring new Compo-nent, Interoperability and Ecosystem Testing. In order to monitor, resolve and improve the existing live environment, several continuous test activities like automated regression and security testing are required. On top of that it is strongly recommended to dedicate a couple of months as an observation period immedi-ately after having launched a new service com-mercially. During this period of time, many of the teams (from ecosystem parties and their vendors) involved in the test process prior to launch should be stand-by, ready to step in for troubleshooting. This approach enables solving issues quickly, minimizing impact on customer satisfaction. Only after this observation period has ended without incidents, the solution can be accepted.

Anonymous: “Life is too short for manual test-ing”

Automated & Regression TestingOnce live with a stable, well-functioning envi-ronment, new functionalities will be added to the ecosystem. These new functionalities could either enhance existing services already used by consumers or introduce new services in the market. In both cases the new software (and hardware), e.g. bug fixes (patches) should be tested in a test environment which is identical to the live environment. Only after having en-sured that the entire NFC ecosystem continues functioning properly, the new software can be released into the operational environment. A regression test suite is prepared to cover both low-level functionality and critical user experi-ence functionality. These tests are called auto-mated regression tests, where a large part of the previously performed tests are repeated in an automated manner. Often this subset of the Ecosystem and security tests is performed before deploying a patch. In general the rule is 80% coverage of functionality in 20% of the testing volume. In many cases, most of the re-gression tests are run automatically using spe-cialized tools.

In shortNFC ecosystems in general and those used for payments in particular, are new to custom-ers. A satisfying user experience and trust in the solution are the two most important rea-sons for a consumer to start using such service. Consumer satisfaction and trust in the prod-uct/service launched in the market is created through extensive testing, ensuring the func-tionality, performance and security of the NFC ecosystem. Therefore, it is crucial that enough time is planned for testing in NFC projects. As presented in this white paper, there are many test phases and different test objectives that an NFC project has to pass, to ensure that the solu-tion meets the expectations. Furthermore, a lot of time will go into actually aligning and agree-ing upon the subjects presented in this paper, simply because many parties need to work to-gether. Bear in mind that testing in an NFC pro-ject is a continuous process. Even when the pro-ject is live, testing will continue to enable new functionalities, to resolve issues found during

Tip: Test Environments

Setting up and well maintaining test environments can reduce testing time in an NFC project. Strict release- and configuration management needs to be in place from the start of the testing process and to continue after launch, too. On top of that, having defined clear purposes for each test environ-ment brings the advantage that one can well protect and quickly improve the consumer experience. A clear ex-ample is a copy of an operational en-vironment dedicated to immediately reproduce and resolve functional, se-curity or performance issues found in production, while another test envi-ronment is used to test new services.

10/11

© 2012 UL All rights reserved. May not be copied or distributed without permission. UL and the UL logo are trademarks of UL LLC.

Page 11: Colis 130 Testing With Nfc Projects

operations quickly and to monitor security. To conclude, testing is crucial to NFC projects and commercial operations: plan sufficient time and assure the required expertise is in place.

UL offers its in-depth knowledge and expertise through the UL Mobile Test Centre, training and consultancy services. Certification services, security assessments and testing for compli-ance are UL core competences. The UL Mobile Test Centre provides tailored services, fully cus-tomized to your company needs. Mobile Pay-ments and TSM Trainings are two of UL’s excel-lent courses to get a quick start in gaining the necessary knowledge. UL directly participating hands-on in running NFC projects has proven to bring highly appreciated benefits. The UL Standardized NFC Implementation approach guides companies through business decision making, the implementation and roll out pro-cesses of NFC services. Selecting services from this approach like the Strategy workshop ena-bles companies to quick and effective decision making. At all times UL brings a pragmatic way of working into practice.

One Final Reminder…Do not be scared by the novelty and amount of technology involved. All big breaks-through had rough start-ups. Western Union internal memo, 1876: “This ‘tel-ephone’ has too many shortcomings to be se-riously considered as a means of communica-tion”

UL Transaction SecuritySparked your interest? Want to learn more? Cu-rious about a test tool demo? UL is being recog-nized as the technology partner and trusted ad-visor for a wide range of NFC implementations around the world. Contact us through [email protected] or call +31 71 581 3636.

To learn more about UL visit our website:www.ul-ts.com

About UL Transaction Security

UL is the world leader in advancing safety

with over a hundred years of history. Em-

ploying more than 10,000 professionals in

over 100 countries, UL has five distinct busi-

ness units - Product Safety, Environment,

Life & Health, Knowledge Services and Veri-

fication Services - to meet the expanding

needs of our customers and to deliver on

our public safety mission.

Through the acquisition of RFI Global, With-

am Laboratories and Collis in 2010 and 2012

respectively, UL is uniquely positioned as

the world’s number one competence center

in transaction security technology. UL acts

as your independent, trusted partner for

end-to-end transaction security services

for the mobile, payment, e-Ticketing and ID

management sectors on a global scale.

UL’s comprehensive transaction security

service line provides advisory services, ex-

pert training courses, test tools and simu-

lators, test services and certification and

security evaluation services. Our thought

leadership, close involvement with leading

industry bodies and extensive experience

enables UL to keep up with the rapid pace

of transaction innovation for years to come.

11/11

© 2012 UL All rights reserved. May not be copied or distributed without permission. UL and the UL logo are trademarks of UL LLC.


Recommended