Automation Enabled Modernization (AEM TM ) Organizations today face a variety of challenges that lead them to consider modernizing their legacy mainframe applications, and sooner rather than later. Such applications have typically been maintained over many years. Legacy code contains a myriad of poorly documented patches that are difficult to understand. Older programming languages use simplistic ways to segment functionality, which means that changing one part of an application can break others. These factors add up to continual spending on legacy maintenance and difficulty keeping pace with changing business needs, industry standards, and financial regulatory requirements. Other legacy challenges include limited Internet access and data sharing, obsolete technology, and spiraling costs of legacy platforms. Hardware and software vendors announce end of support for older products. Legacy programmers are retiring in increasing numbers and educational institutions are no longer teaching those older skill sets. Does any of this sound familiar? If so, you are not alone. Global awareness is rapidly growing in terms of the need for legacy modernization. Unfortunately the modernization approaches one might typically consider come with a high price tag and significant technical risk. It is little wonder that after considering the options, perhaps the most common decision is to stick with the existing applications for a while. After all, the required functionality is already in place and serving the needs of the enterprise. Maintaining the status quo can only go on so long, however, before the factors described above force companies into action. Then the question becomes: What is the best strategy for moving forward? This paper compares the alternatives, including a lesser known option that offers several advantages.
Transcript
Automation Enabled Modernization (AEMTM) Organizations today face a variety of challenges that lead them to consider modernizing their legacy mainframe applications, and sooner rather than later. Such applications have typically been maintained over many years. Legacy code contains a myriad of poorly documented patches that are difficult to understand. Older programming languages use simplistic ways to segment functionality, which means that changing one part of an application can break others. These factors add up to continual spending on legacy maintenance and difficulty keeping pace with changing business needs, industry standards, and financial regulatory requirements. Other legacy challenges include limited Internet access and data sharing, obsolete technology, and spiraling costs of legacy platforms. Hardware and software vendors announce end of support for older products. Legacy programmers are retiring in increasing numbers and educational institutions are no longer teaching those older skill sets. Does any of this sound familiar? If so, you are not alone. Global awareness is rapidly growing in terms of the need for legacy modernization. Unfortunately the modernization approaches one might typically consider come with a high price tag and significant technical risk. It is little wonder that after considering the options, perhaps the most common decision is to stick with the existing applications for a while. After all, the required functionality is already in place and serving the needs of the enterprise. Maintaining the status quo can only go on so long, however, before the factors described above force companies into action. Then the question becomes: What is the best strategy for moving forward? This paper compares the alternatives, including a lesser known option that offers several advantages.
The High Cost of Re-development A typical first instinct is to consider replacing a system the same way it was originally built, with traditional manual development. This is a tempting option for many organizations since they already have an IT organization in place to execute development projects and this is how they are used to acquiring business-specific functionality. We’re all human, so familiar approaches tend to be attractive. More than that, a re-development project has an existing system to serve as a working requirements specification, so this type of project should be easier than the original development ... right? Unfortunately, history doesn’t support that argument. One issue is that a re-development project must re-create functionality that has evolved over many years, growing far beyond the scope of the original application. Sifting manually through millions of lines of source code, multiple layers of patches and poorly documented changes is a complex, lengthy, and error-prone task. First releases of re-written systems tend to include the best-known, day-to-day functionality, but the devil is in the details. Legacy source code almost always contains decades worth of exceptional logic. Understanding, replicating and testing this detailed logic is one of the main reasons our industry is rife with stories of re-development projects that ultimately failed, years late and hugely over-budget. Package Replacement Purchasing replacement Custom Off-The-Shelf (COTS) applications can be a highly effective approach when packages are available that provide the required functionality. The results of the acquisition are predictable in the sense that costs are known and the package can be tested up-front. Packages are rarely a perfect fit with business needs, however, which often means an extensive customization effort. As with custom development, such projects tend to be time-consuming and expensive. More than that, a large percentage of corporate applications support enterprise-specific functions and business rules. Often no COTS packages are available to support such unique functionality. Code Transformation Tools Off-the-shelf tools are available for a limited set of the most common legacy programming languages. Such tools parse legacy code and convert to more modern programming platforms such as Java and .NET. Using this type of tool can dramatically reduce effort, timeframe, and cost when comparing with manual re-development, especially when the legacy code is a straightforward application of a common programming language.
Many organizations find, however, that their application portfolios contain a mix of several languages, many of which are not handled by available tools. Taking into account database definitions, data queries and updates, job control languages, presentation layers, scripting, reporting, and processing logic, it is common for even a single application to include modules written in a dozen or more languages. Off-the-shelf tools rarely cover such a diverse mix of technologies. Experience shows that every legacy renewal project presents unique challenges. Applications incorporate language extensions, unique coding styles, undocumented compiler functionality, and integration with multiple technologies. When a tool vendor designs an off-the-shelf code converter, anticipating all of these potential wrinkles ahead of time is virtually impossible. This means that organizations attempting to use such tools inevitably run into features not handled automatically, and must devise time-consuming manual workarounds. To complicate matters, the organizations’ resources are typically not experts in using the newly purchased tools, which can introduce learning curves and uncertainty over whether the tools are being used to best advantage. In many cases such tools perform relatively simplistic line-by-line translations, producing code that mimics the structure of the source programs rather than taking full advantage of the target language. A common complaint is that the output code “looks generated” and is difficult to read. This suspect level of quality increases long-term maintenance costs. In our experience, the low level of resultant code quality is the main reason organizations that have tried code conversion tools frequently move on to look for more effective solutions. To Wrap or not to Wrap? Middleware can be “wrapped” around legacy functionality to integrate existing applications into a web-based and/or service oriented architecture (SOA). The basic premise is to avoid changing the legacy code by re-hosting an application within middleware that emulates the application’s original environment. This can provide a heightened level of connectivity, exposing the application to the outside world in a more accessible way. This approach is limited, though, in the value it can provide. Presenting a monolithic legacy application as a single unified service does not provide the ideal level of granularity for a SOA environment. More than that, because the existing code is still in use, several legacy challenges are unaffected by this strategy. The organization is likely to still be faced with impending loss of software support, a shrinking pool of programmers for their languages, and the expensive ongoing challenge of attempting to keep old code up-to-date with business needs.
Automation-Enabled Modernization Trinity Millennium Group provides an alternative approach for legacy applications called Automation-Enabled Modernization (AEMTM). The defining characteristics of AEMTM are as follows:
• Provided as a service: Engaging a legacy modernization service provider means your renewal project is handled by resources who have developed code conversion expertise by performing this type of work full-time. Learning curves associated with your project are considerably reduced.
• Leverage automation to a high degree: Manual analysis and re-development is too time-consuming and expensive to be a viable solution for legacy modernization. The sheer size and complexity of a typical legacy code base is simply too overwhelming to review and re-code line by line. Automated tools can perform many code analysis and conversion tasks quickly and inexpensively. Of equal importance is the higher level of assurance that all legacy code is accounted for in the new application. Unlike people, automated tools do not mind monotonous tasks and never become bored or tired.
• Appropriate balance of automated and manual processing: As discussed above, off-the-shelf tools that attempt to fully automate the code conversion process introduce problems of their own. For example, automated variable naming and program organization algorithms can produce code that is challenging to understand during maintenance. Defining a generic solution for identifying highly effective candidates for objects and/or services from procedural legacy code is also challenging. Many conversion tasks can be performed effectively with automation, but this is not true of all tasks.
AEMTM is based on a varying level of automation depending on the task. For example, converting table definitions from one database to another can often be fully automated, while design of an overall application architecture might best be performed manually. A typical goal is to generate partial code to a 70 to 90 percent level, and then complete, polish, optimize, and unit test the code manually. Accomplishing appropriate tasks with automation reduces time, cost, and risk, while performing conceptually difficult and critical tasks manually achieves high quality results. Programs produced in this fashion are often indistinguishable from those created entirely by hand. This achieves savings during the modernization project and delivers an agile application based on highly maintainable new technology. The result is an improved ability to perform functionality updates, with decreased effort and cost over both the short and long term.
• Agile tools designed for in-house use: Delivering AEMTM as a service rather than selling a tool as a product offers another significant advantage – the tool users (that is, the full-time modernization professionals) have direct access to the people responsible for tool development and maintenance. In many cases they are one and the same. Recall the discussion above concerning the unique characteristics involved with virtually every modernization project. When the tools first encounter an unfamiliar challenge, the necessary changes are incorporated into the tool set and processing continues. The client in such a scenario is typically unaware of such process enhancements, and furthermore is not forced to devise time-consuming manual workarounds.
This service model is most effective when the automated tool suite is highly agile. Designing the tools strictly for in-house use makes this more feasible, since the emphasis can be on data-driven functionality and ease of updates, rather than pouring development resources into product support, making the tools bullet proof and usable by non-experts, and so on.
This is also a model for incremental, project-driven growth in tool suite capabilities. A client with less widely used programming languages can obtain service, although such a project may involve a “tooling up” phase to incorporate the required extensions. The good news is that an AEMTM service provider typically has prior experience with a wide variety of languages and thus is in the best position to add support for others in the least possible time.
• Incorporate conventional application development success factors: Given the presence of a legacy system to serve as a specification for required functionality, one might be tempted to define an AEMTM process as follows – use the automated tool set to generate as much code as possible, then finish the work manually using the existing application as a guide. One of the problems with such an approach is that different clients can have radically different needs regarding the to-be solution. Some wish to generate full GUI screens in the new environment, while others want the old green screens emulated to avoid retraining costs and union issues. Some clients are satisfied with a generic object-oriented application design, whereas others are moving toward a leading edge service oriented architecture. Examining legacy source code can also be an inefficient way to discover required functionality.
The lesson is that both traditional application development projects and their AEMTM counterparts benefit from properly documented requirements. The same is true of application design, definition of releases, project management, configuration management, and a host of other development best practices. Think of an AEMTM effort as an
application development project aided by automated conversion tools and the presence of legacy code as a source of information.
Comparing AEMTM with Other Modernization Approaches Each of the approaches discussed in this paper can provide value when applied in the appropriate circumstances. For instance, manual re-development may be the best approach for a small legacy application where the majority of the effort required is in understanding the target technical environment and how it should be applied for this application. Package replacement can highly effective when an available COTS package provides the right functionality for the right price and is implemented using an appropriate technology. As Figure 1 shows, however, most approaches fall short when it comes to the most common legacy challenges. For example, wrapping a large application to expose it as a service within SOA does not provide an optimal SOA configuration. Thus wrapping is depicted in Figure 1 as only a partial solution for migrating complex functionality. Package replacement, manual re-development, and off-the-shelf tools also have their shortcomings.
AEMTM provides value by addressing a broad spectrum of legacy challenges. Highly leveraged automation results in lower cost and risk than manual re-development or COTS customization. Unlike package replacement, AEMTM can be utilized to modernize organization-specific functionality. In contrast with off-the-shelf tools, AEMTM produces high value, high quality, highly maintainable source code. Wrapping a legacy application means keeping most of the associated maintenance headaches, whereas AEMTM replaces legacy code with a more agile application. Overall, AEMTM provides an answer for the most common legacy challenges. An AEMTM Case Study – Application Modernization for the FAA Like so many other organizations today, the FAA wishes to reduce application hosting costs. As one of their initiatives, the FAA hired Trinity Millennium Group to modernize their Natural / ADABAS applications. This effort involved millions of lines of source code, conversion to a Windows .NET platform, and migration of ADABAS indexed data files to a relational database. Using the AEMTM approach, our modernization consultants leverage our extensive library of automated conversion tools, and apply the human touch for critical design and re-factoring tasks. Figure 2 shows how this approach provided a fast, cost-effective migration to a high-quality result for the FAA.
FROM Conversion TO Natural procedural code & global data structures
90% Automation 10% Manual code polish
VB.NET code & SQL Server stored procedures
Natural screen maps 100% Automation ASP.NET screen definitions
ADABAS file definitions Automated DDL generation with predefined design parameters, followed by manual DDL polishing
SQL Server database table definitions
JCL 100% Automation VBScript
Batch processes defined in Control-M
Manual integration of Tidal with converted application
Batch processes defined in Tidal Scheduler
Generated code, including manual re-factoring and polishing
Automated test cases for unit, integration and regression testing
Tested and debugged code delivered to the client
Figure 2: Conversion Tasks for a Representative Project
Trinity’s AEMTM approach utilizes automated conversion to a high degree, but does so in a unique, project-specific manner. The basic idea is for Trinity’s team of transformation engineers to:
1. Identify the patterns of code that are present in the client’s legacy software for handling various types of functions;
2. Define mappings from these patterns to the corresponding artifacts that are required in the target application architecture;
3. Use Trinity’s proprietary suite of automated tools to implement these mappings;
4. Convert the associated code portions and populate a high percentage of the new application, consistent with the client-specific architecture;
5. Maintain complete, bi-directional traceability data to document precisely:
• What became of every line of legacy code;
• The source of every line of new code;
• Confirmation that the transformation completely accounts for all legacy functionality, with no extraneous functionality introduced; and
6. Use software engineering best practices to manually handle any remaining legacy functionality.
The main driving factor behind AEMTM is that automated code conversion forms a necessary and valuable part of application conversion, but automated code conversion must be:
• Applied in an application-specific manner, so patterns specific to the legacy code are identified and interpreted, and artifacts specific to the organization’s functional and architectural needs are produced;
• Combined with software engineering best practices to handle some aspects of the conversion, so that a quality result is achieved; and
• Grammar-driven to provide the flexibility to handle a wide variety of programming and scripting languages.
Figure 3 depicts the engineering process at the heart of the AEMTM approach.
With the AEMTM approach, legacy software source code is parsed into a variety of meta-data representations, which completely capture all of the detailed information in the legacy software. These representations include Abstract Syntax Trees (ASTs), symbol tables, control flow data, among other proprietary intermediate results. Discussions with the client take place to understand the target architecture and the desired characteristics of the transformed application. These project-specific characteristics drive the Advanced Transformations that are performed on the intermediate meta-data, transforming the information closer and closer to the form required for the eventual new application. This is an iterative process, utilizing a wide variety of proprietary tools, many of which tend to updated on each project to respond to specific client needs. A key success factor is that this iterative process also involves human decision making for low-volume, critically important aspects of the transformation. The result is a fast, cost-effective project (because tools handle the high-volume, repetitive tasks) and a high quality result (since software engineering best practices are applied where required). This unique blend of automation with manual intervention is provided by no other current IT application modernization approach.
This advanced software engineering approach is delivered with Trinity’s phased AEMTM approach, as depicted in Figure 4
Trinity’s AEMTM approach supports our clients’ modernization needs in a variety of ways, as described below:
1. Complete and accurate replication of existing application functionality – A modernized application must provide all of the functionality currently provided by the legacy system(s). The AEMTM re-engineering process shown in Figure 3 includes complete automated parsing of all legacy source code, ensuring that all legacy functionality is captured for transformation. The automation-enabled transformation mappings ensure consistent and accurate conversion of functionality from legacy to new functionality. The accuracy and completeness of the transformation is then validated through both:
o Rigorous, multi-phase, industry standard testing procedures; and
o Traceability reports, as per item 2 below.
2. Traceability – Traceability data is critical to confirming completeness and accuracy of an application transformation, mapping legacy code to new code, and vice versa. There is a need to be able to tell with certainty what has become of every line of legacy source code. Similarly, it is important to be able to identify the derivation of every line of newly created source code. The AEMTM approach is unique in providing complete traceability data, in detail, at the line of code level. As the automated mapping procedures are carried out, Trinity’s proprietary tools record the mappings from legacy code to the resultant modern code produced as a result. Trinity’s tools produce “side-by-side” mapping reports showing
that (a) all legacy functionality has been transformed, and (b) no unnecessary new functionality has been introduced.
3. Compliance with each client’s selected modern IT architecture – Legacy software is typically designed to be consistent with the design practices of the day, based on when the software was created. Modernized software is most valuable when it is re-architected to be consistent with modern best practices. Furthermore each client normally wishes to have the flexibility to choose the architecture design that best fits their needs, and to receive a modernized application that conforms to this architecture design. One of the keys to the unique value of the AEMTM process is that an application is transformed based on the specific characteristics of how that application is organized and programmed, rather than generically on the semantics of the programming language(s) used. Trinity’s Knowledge Mining analysis identifies the programming patterns that have been utilized within the client’s applications, and then this application-specific knowledge is used to identify the various aspects of the legacy source code. In this way, security-related code can be separated from data access, business logic, user interface interactions, and various other types of functionality. Each “aspect” of the code can then be used to populate the appropriate layers of the desired architecture. The result is a completely re-architected application, which is critical for moving forward with highly maintainable modern source code base.
4. Modern, object-oriented source code – Any newly transformed application should be organized according to current best practices of object-oriented design. Replicating the monolithic nature of the legacy source code is typically not acceptable for most client organizations, since this tends to retain current difficulties such as maintaining a fragile source code base. Transforming monolithic, procedural legacy code to object-oriented code requires the same capabilities as described above related to re-architecting the application. Trinity’s proprietary tools and procedures enable the “intention” of each code segment to be identified. This, in turn, allows well-defined object-oriented classes to be defined and implemented. This can only be achieved effectively with automation (given the thousands of classes required for a typical legacy information system), and more specifically with automation that is configured for application-specific patterns.
5. Compliance with client standards and preferences – Most of Trinity’s clients have their own standards and preferences for coding style, user interface presentation, database design, and other aspects of newly developed applications. Further, our clients require Trinity to comply with such standards and preferences. A “one-size-fits-all” solution is typically not acceptable. Just as AEMTM is able to analyze and react to the coding patterns found in the legacy code base, the “Compose” step
shown in Figure 3 is also configured on each project according to the client’s specific needs. Trinity’s proprietary AEMTM tools can be configured to produce modern application artifacts that comply with client preferences for such aspects as (for example) choice of graphical user interface components, coding styles, and database design choices.
6. High quality resultant source code – Much of the value of a newly modernized application will rest with the ability of the client to effectively maintain the software. The application should be nimble and able to respond as business needs arise. The code must be well organized, easy to understand, and structured so changes can be made locally without fear of consequences rippling through the application. The flexibility of the AEMTM approach enables the transformed source code to be properly organized, with comments retained (in the appropriate locations), and presented in the fashion expected by modern-day programmers. The result is easy to understand and highly maintainable code. Furthermore, Trinity’s proprietary repository of associated documentation artifacts enhances the ability of maintenance programmers to navigate and update new applications.
7. Rapid and cost-effective solution – It is important that application modernization projects should use automated tools where applicable to control schedule and effort / cost requirements. Large application transformations attempted as manual re-write projects tend to be significantly more expensive and time-consuming. With AEMTM, automation is used to a high degree, often with up to 80 or 90 percent of the code base converted through the use of tools. This percentage is project-specific, and is determined by the degree to which manual transformation must be used to ensure a high quality result. This ensures our clients receive timely and cost-effective services without sacrificing quality.
8. Legacy application enhancements allowed during transformation – The AEMTM process makes it possible for clients to continue to make changes to the legacy code base even while a transformation project is underway. This is critical, because it is typically not feasible to freeze the IT support for our client’s business during the period required for a transformation project. The phased re-engineering approach enables Trinity’s transformation team to manage the schedule of code processing to coordinate with client’s planned ongoing legacy application maintenance.
9. No vendor lock – Trinity produces only native source code, as if it were fully produced by hand, so our client or any third party of their choice can maintain the applications in a normal manner, with full ownership and control. We never insert proprietary routines in the newly generated source code, which would serve to tie our clients to Trinity for any
changes to the functionality represented by those routines. Following modernization, our clients are free to maintain all system functionality in any manner they choose.
10. Handling all technologies used by the legacy applications – Legacy applications often include a variety of programming languages, scripting languages, database management systems, and other notations. Trinity has been developing multi-language support for the past fifteen years of application transformation experience, and our tools are designed to handle the full range of notations for any given project. Trinity’s project experience includes over a hundred different programming languages and databases, with more being added to Trinity’s toolset on a continuous basis as client projects are conducted. This is possible due to the configurable and grammar-driven manner in which Trinity’s tools and engineering processes have been designed and developed. When a new notation is encountered, a context-free grammar is developed and the code can then be parsed and fed into the AEMTM process.
Conclusion Automation-Enabled Modernization combines the speed and cost-effectiveness of automated code conversion tools with the ability to respond to unique client requirements. In this way, AEMTM provides a rapid, cost-effective, low-risk way to leverage the tremendous investment your legacy IT assets represent.
