BIO PRESENTATION PAPER

Better Software Conference June 26 – 29, 2006 Las Vegas, NV USA

W12

6/28/2006 1:45 PM

WEB SERVICES INTERFACE

DESIGN - PITFALLS AND PROVEN

TECHNIQUES

Dave Mount J-Soup Software, Inc

Dave Mount Dave is the founder and president of jSoup Software, providing technology consulting services to companies that develop software. Before starting jSoup, Dave was the senior application architect for four years at BenefitPoint, an ASP for the employee benefits industry. Dave pioneered the use of Web Services at BenefitPoint and recently has been engaged in extending BenefitPoint’s client-facing API. Dave graduated from Cornell University in 1993 with a B.S. in Applied and Engineering Physics and has been working in the software industry ever since.

1

Web Services Interface Design

Dave MountPresident

J-Soup Software, Inc.

Pitfalls and Proven Techniques

2

Objectives

● Explain why gray matter still matters● Show pitfalls of poor interface design● Provide guidelines for stronger interfaces● Discuss key elements of Web Services interface

design● Examine popular Web Services interfaces

3

A Cautionary Tale

● BenefitPoint, ASP for employee benefits industry● Aptus Connect project, expanding services● Problems

− Inconsistent naming of operations and data− Object reuse: get vs. create and update− Security holes: inconsistent authorization checks− Failure of testing tools− Interoperability issues

4

Why Gray Matter Still Matters

● Tools make it easy● Web Services is still emerging● Lots of choices that will impact customers● Interface is a contract● Potential for serious mistakes

5

Pitfalls of Poor Design

● Exposing application ugliness− Concepts and names evolve over time− Inappropriate method granularity− Ambiguous data types, inconsistent use of types− Meaningless exceptions

6

Pitfalls of Poor Design (continued)

● Bypassing application rules− Different from browser-based perspective− Lacks context of Web-page driven process− Skipping validation− Allowing access to another user's data

7

Pitfalls of Poor Design (continued)

● Ignoring interoperability− .NET, Axis, WebLogic, etc.− WS-I basic profile

● Missing the boat− WS-Security, WS-Messaging, others− SOA

8

Proven Design Techniques

● Envision the interface from perspective of user− Use consistent names based on clear concepts− Use enumerations for static values rather than internal

IDs− Structure data by concept and context− Classify exceptions and make problems explicit− Generate client stubs and test early

9

Proven Design Techniques (continued)

● Think about the future− Avoid breaking contracts− Establish a plan for versioning− Balance flexibility of operations and data types with

clarity of purpose

10

Proven Design Techniques (continued)

● Take interoperability seriously− Stick to basic XML types− Avoid constructs with limited support− Follow WS-I recommendations− Accommodate .NET

● Use work-arounds for “nillable” fields

11

Proven Design Techniques (continued)

● Perform validation− Check all values and return complete list of problems− Ensure meaningful value combinations

● Let the application enforce the rules− Create tests that attempt to violate the rules− Plug holes at the application level

12

Proven Design Techniques (continued)

● Keep users straight− Require login when important− Use WS-Security or pass session ID in header

● Provide documentation− WSDL may be human-readable but...

13

Popular Public APIs

● Google, eBay, Salesforce.com● Common features

− Well documented− Good use of enumerations− Broad support for interoperability: .NET, Java, PHP,

RoR, etc.

14

Wrap Up

● Use tools, especially the one in your head● Anticipate change● Keep an eye on standards● Get client feedback● Learn by example

15

Web Services Resources

● IBM's developerWorks – http://www.ibm.com/developerworks/

● Microsoft – http://msdn.microsoft.com/webservices/

● Web Service Interoperability Organization – http://www.ws-i.org/

● OASIS Open – http://www.oasis-open.org/

● Apache Web Services Project – http://ws.apache.org/

● jSoup Software – http://www.jsoup.com/webservices/

Web Services Interface Design Proven Techniques for Designing Web Services that are Consistent, Standards-Compliant and Secure

Dave Mount, President J-Soup Software, Inc.

April 3, 2006

Motivation Web Services interface design can seem deceptively simple. Many tools for building Web services can automatically generate a service from existing application code. While code generation accelerates development, taking design shortcuts by directly exposing application functionality or ignoring interface design altogether can lead to short- and long-term problems.

The Web services perspective is different from that of a typical application. What may be intuitive and correct in object-oriented design and programming can cause problems in Web services. Also, while the users of a browser-based application interface expect quick response times, pre-populated forms, pages with clear context and navigation, the “users” of Web services very often are other applications or batch processes. To improve Web service design, it pays to step back and consider the target audience and potential use of the services.

The pitfalls of exposing a poorly designed interface range from semantic inconsistencies that cause confusion and data integrity issues to serious compromises in application security. This guide offers a number of proven techniques for producing clean, stable, comprehensible Web services that are open to future enhancement. Follow these guidelines to avoid the pitfalls of poor Web services design.

Background Assumptions For the purposes of this paper, a basic understanding of SOAP-based Web services is assumed. A quick Web search will uncover lots of information about Web services, and a number of websites pertaining to interface design are referenced in the endnotes of this paper. The reader with some hands-on experience developing Web services is likely to get more out of this paper than a complete newbie.

Web services provide a means of integrating applications by exchanging XML-formatted messages in a defined structure. Web services consist of operations that are composed of messages, which can contain structured data. Each service has one or more modes of access, called bindings. A binding maps to a service port, which is a collection of operations. SOAP is the original and most common protocol used for packaging Web services messages and HTTP is the most commonly used transport for delivering messages.

A Web service is defined by a Web services definition language file, or WSDL, which specifies the service name and location, available bindings, ports, operations, messages and custom data types. Data types are defined using XML schema directly in the WSDL or in external files that are imported into the WSDL.

Web services standards are evolving rapidly, and many features, such as security, routing and transaction processing, are being added. Be sure to check the freshness date on this paper and be on the look-out for technology improvements.

Defining the Service The definition of the service is the primary aspect of the interface. While the WSDL defines the structure of the definition, the content of the definition is entirely up to the developer. Think of the schema that defines custom data types as a data dictionary, the source of truth regarding the semantics and meaning of application components and their fields. Add documentation to the WSDL and related XML schemas to educate consumers about the purpose of operations and the meaning of data elements and to remove ambiguity.

Web services should be grouped according to related functionality. Each service should consist of operations that work on the same major application component or that work together for a common business purpose. For example, a service might focus on order management, or it might have operations for performing product searches. Keeping the scope of each service narrow allows services to be treated as components, each with its own life-cycle, custom access rules and deployment schedule.

As with any distributed computing model, operations of a service generally should be coarse grained because the overhead of calling an operation is much higher than that of a local method call. That is, rather than defining operations that check individual fields of an entity, operations should return a subset of data that can be inspected locally. Similarly, when feasible, an action that from a browser involves multiple steps – gently leading users through option choices – in Web services should be condensed into a single step that accepts all information at once.

A good naming convention for operations provides clues that help developers quickly understand their purpose. Use a consistent naming scheme for operations, data types and fields. The following list of prefixes and their implications forms a decent basis for naming operations.

get Returns one or more specific objects. find Returns zero or more objects that match given criteria. create Creates one or more new objects. modify Changes the state of one or more objects. destroy Eliminates one or more objects. remove Eliminates an association between objects but preserves the objects

themselves. do Indicates a custom action. For example, doPriceCheck.

For data, consistent data type and field names show conceptual clarity. Often, the terminology used in applications tends to evolve as enhancements and alternate perspectives are introduced. For example, in a sales application, the variable name for the customer entity might be referenced in different areas of the code as “client,” “account,” “organization,” “lead,” etc. Although an application may use a variety of names with an implicit (if not documented) understanding that they represent the same thing, the introduction of Web services data types offers a chance to reconcile such discrepancies.

Take advantage of the constraints that can be specified in the XML schema that defines data types used by the service. Use required fields (minOccurs=1), enumerations and other restrictions as appropriate to ensure that input values conform to the rules of the application. Consider the context in which the data types are used. Operations that create objects may have different requirements than operations that modify objects. Define and use different XML data types for each context to avoid ambiguity. While this approach might seem redundant based from an object-oriented perspective where the objects “understand” their state, in terms of a Web services contract, making constraints

and allowable values explicit prevents bad data and reduces the amount of validation logic in the service implementation. Moreover, being clear and precise is more polite to the consumer of the service, who otherwise will end up dealing with a slew of validation errors.

For simple services, using the types section of the WSDL to define custom data structures may be sufficient. However, for services that share common types, create an external XML schema with its own namespace that defines the types and import the schema into WSDLs that need it. Leave the definition of message parts in the WSDL, but externalize all other data structures that have the potential for reuse. Choose namespaces that uniquely identify a service and reflect its ownership. The URL style of namespace serves this purpose best. The namespace can also include a version number, as discussed below.

For example, the service namespace could be http://ws.acme.com/catalog/v1. The namespace for types used in multiple Web services might be http://ws.acme.com/common/v1, which can be mapped to a separate package, allowing object reuse. The Axis WSDL2Java tool conveniently translates the service namespace to the Java package com.acme.ws.catalog.v1 and the common type namespace to the package com.acme.ws.common.v1. Some toolkits use a different default approach to namespace-package mapping. For the given toolkit, either adjust the namespace strategy or define a custom mapping.

Consider including a version number in the name of the service. Change is inevitable and any change to the interface will require a service to reincarnate itself. By including the version number in the service name, existing clients cannot be mistakenly left in the dust. On the other hand, services might be designed to pass version information in the SOAP header, surreptitiously translating older requests to newer implementations.

Embracing Change When a published Web service needs to change, consider the impact that the changes will have on existing clients of the Web service. If operations that are available to the outside world change, a new version of the service should be created so that systems depending on the service will continue to work. Only changes that are backward compatible can be made to a service without risk to existing consumers. The following types of changes are backward compatible:i ii

Adding new operations – Older clients simply will be unaware of and never call the new operations. Adding optional fields and loosening data types used for input – This type of change expands

behavior and allowable data, imposing no new constrains or requirements on older clients. Tightening data types used for output – Older clients will continue to handle a superset of the

allowable data. Note that if a data type is used for both input and output, incorporating a change to that data type in a Web service will require a new version.

Have a versioning strategy in place, ideally before the first version of Web services is released. A simple yet effective strategy is to include a version number in the service name and namespaces. Any changes to the Web services interface or associated schemas that are not backward compatible must be done in a new version of the WSDL or XSD documents.

Try a simple exercise. Say that Acme Co. has published the Web service CatalogServiceV1 that depends on an external schema, common-v1.xsd, for the definition of Product. Upcoming improvements require Product to include information that was not previously available through the CatalogServiceV1. Adding fields to a type used for output dictates that a new version of the service, CatalogServiceV2, be created and deployed alongside CatalogServiceV1 because existing clients are not expecting the new fields. Moreover, since version 1 of the service still depends on version 1 of the schema for common types, a new schema version must be created for version 2 of the service.

To create a new service version, it is best to copy and modify the existing service WSDL. However, new schema versions need only include definitions that have changed. WSDLs may import multiple versions of a schema and use namespace resolution to pick and choose data types. Also, types from a new schema may rely on definitions in an older schema. This is why it is important to have a consistent namespace strategy that includes version numbers.

Versioning was ignored by the authors of Web services specifications, probably for good reason, since it can become quite complicated. A number of valid approaches exist, and at this point it is left to each organization to choose a strategy and use it consistently. Be sure to include release notes with each new version and publish a sunset policy for Web service consumers that describes when support for older versions will end.

Keeping an Eye on the Side Door Web services can expose underlying applications to a number of security problems. The basic tenants of security are authentication, authorization and encryption. Beyond that are the dangers presented by hackers, which this paper does not address, although new security standards do.

OASIS Open has officially addressed the problem of authentication.iii Members of OASIS approved version 1.1 of the WS-Security standard in February of 2006. As a result, tools vendors will begin to support the standard if they are not supporting it already. In its basic form, the standard defines how to pass a Username Token in the SOAP header that contains user credentials and some additional bits of information that help prevent hacker attacks.

A common “home-grown” approach to authentication is to provide a login operation that, given valid credentials (usually a user name and password), returns a session ID for use in the header of subsequent requests. Sensitive information should always be encrypted. Typically, secure sockets layer (SSL) is used to secure HTTP-based communication, providing host and (optionally) client verification and data encryption for the entire exchange.

Once the identity of the user is known, authorization must be considered. Typically, users are restricted using role-based permissions to a subset of functionality and are usually not allowed to see or modify data that belongs to other users. Care should be taken that Web services use the same authorization rules.

A subtle security hole can develop due to the fact that many applications are written from the perspective of a Web browser, keeping users within a sensible context. The queries used to populate Web forms naturally limit the set of available data, but Web services have more of a free-form nature. Carefully check the validity of input data, provide look-up operations and summary arrays for discovering available entities, and use enumerations to define fixed allowable values. Authorization checks should be performed in a layer of the application that is accessible to both the view layer and

the Web services layer. Short of that, Web services must provide authorization checks.

Due to the automated nature of Web services, one or more new application roles might be useful. Web service that are used for database synchronization might be accessed by a “system” user with permission to view all data of an organization but not to make changes. Third-party users might have an agent role, with permission to see and manage certain data on behalf of the owner. Contrarily, avoid creating Web services super users just because an application does not support a needed role with a subset of permissions.

Getting Along with Others Web services offer the ability for true system interoperability, but interoperability is not a given. The Web Service Interoperability Organization (WS-I) has put a lot of effort into defining the variations of Web services definition that are truly compatible across platforms. The Basic Profile provides guidelines for interoperability.iv Development and testing tools such as Eclipse and Parasoft's SOA Test provide WSDL validation that includes warnings about Basic Profile violations.

Since many tools supported Web services before the Basic Profile was defined, default options may not conform to interoperability standards. For instance, by default Axis generates Web services using the rpc/encoded binding, which is not allowed by the Basic Profile. Axis can also generate rpc/literal and document/literal bindings, which are allowed. The preferred style for the binding is “document literal / wrapped” because it looks like the rpc/encoded style, with operation names and unambiguous parameter data structures. However, the operations are defined as XML documents, which conforms to interoperability standards. v

Naturally, interoperability is mostly about .NET and Java playing well together. It is best to use only the data types defined in the XML schema specification. In particular, avoid ambiguous constructs, such as Java's Collection-based types. Microsoft has published strategies for interoperability, focusing mainly on the differences between .NET- and Java-based Web services.vi

Be aware that older versions of .NET do not recognize nillable fields. .NET populates nil fields with a default value instead of leaving them empty. For example, it uses 0 for nil integers, 1/1/0001 (January 1, 1 AD) for nil dateTime fields and the first defined enumeration value for nil enumerations. To ensure interoperability, be sure to test with clients on a variety of platforms.

Conclusion It pays to think carefully about the design of Web services interfaces. The integrity of Web services hinges on the strength of the contract between the service provider and consumers. Interfaces can expose application inconsistencies, morph unexpectedly in ways that break existing clients, open security holes and otherwise violate the trust of Web service users. Alternatively, with some planning they can be a source of truth about an application and pave the way toward productivity gains and new capabilities. Keep an open mind, make informed choices, and enjoy the view from the technology treadmill.

i See the article Best Practices for Web Services Versioning on the IBM developerWorks site: http://www-

128.ibm.com/developerworks/webservices/library/ws-version/ii See the article Make minor backward-compatible changes to your Web services on the IBM developerWorks

site: http://www-128.ibm.com/developerworks/webservices/library/ws-backward.htmliii For the full WS-Security specification, see the OASIS Web Services Security Technical Committee website:

http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=wssiv See the WS-I website for a number of interoperability profiles, including the Basic Profile: http://www.ws-

i.org/deliverables/workinggroup.aspx?wg=basicprofilev For an excellent description of the various WSDL styles and their pros and cons, see this article on the IBM

developerWorks site. http://www-128.ibm.com/developerworks/webservices/library/ws-whichwsdl/vi For a full treatment of interoperability from Microsoft, see:

http://msdn.microsoft.com/vstudio/java/interop/default.aspx and http://msdn.microsoft.com/webservices/webservices/building/interop/default.aspx