webMethod testing with iTKO LISA

In my previous post we saw how LISA can be used to test SSIS packages and got to know about the Database validation feature available in LISA, but why would you wish to pay for a tool like LISA to do only database testing for you. So here is how we can test services implemented using softwareAG’s webMethod.

What are webMethod /Software AG?
webMethod suite is a software for process improvement, SOA enablement, IT modernization and business and partner integration.

To ensure integration projects deliver expected business results, we have to ensure the quality of the application at every stage. iTKO LISA is built to give development and integration teams the ability to test every layer of webMethod application infrastructure from component testing to comprehensive validation, and load testing.

Brief about the project
The project involves integration between JDE ERP 811(JDE) and Maximo 4.1(MAX4). We integrated and maintained the data consistency between JDE and MAX4. The backend used in both cases was Oracle Database.

Testing webMethod Integration services:
The challenges include invoking and verifying systems across multiple technology layers, providing assurance during design and development, as well as when software updates or new interactions occur. Continuous validation is required. Have to identify potential issues and enforcing policies that may affect service levels. Many times there is a dependency on third party services and components that are not yet developed. Other available SOA testing tools were only useful for request and response testing and not for middle level testing. Many open source testing tools were useless as were dependent on WSDL (Web Service Description Language).

LISA’s coverage for webMethod:
LISA can directly interact to webMethod Integration server, Broker, webMethod BPMS and acts like a client. For Integration Testing LISA can test Web Applications/RIAs (HTTP, HTML, and Swing etc), Web Services (SOAP, WSDL, and XML), Java Components, and JDBC Database of any flavor.

For Invoke and verify
The testing process needs to be able to create the conditions that produce unexpected behaviors

Challenges for Test Team:
It was difficult for test team to create the cause. When a service is ready and is available for testing, the test team has to create the cause, the precondition to test the service and also had to create the conditions that produce unexpected behaviors in the systems, on single user basis and under load. It was difficult for the test team to manually insert record into a database, which triggers another activity within webMethods Modeler or Integration Server which launches a flow service resulting in another JMS message being sent, and then records being dropped into financial system. In order to test this example it was very difficult for the test team to create all the scenarios to validate just a single service.
A tool was needed which can interact within each layer, insert records, trigger webMethods services, transform data, publish a message and validate the received message or look into the financial system.

Solution: iTKO’s LISA:
There are several typical ways LISA can generate a chain of reactions of events within SoftwareAG integration. For instance, LISA can insert a record into database, which triggers another activity within webMethod Modeler or Integration Server. LISA can invoke and verify the Service, as well as other critical layers of the underlying application the service exposes, including Web UIs, ESB and Integration servers, Java objects, databases and mainframes. When a new service is registered over the registry LISA tests can be run to verify that the service meets structural, behavioral and performance related policies throughout their lifecycle.

To create the cause LISA step SQL Database Execution (JDBC) step was used with insert queries. LISA can even write a message to a JMS queue to start the process with the help of JNDI step. webMethod Integration Server reads the message from the queue, writes the payload of the message to the database, then sends a SOAP request to a system. LISA can validate messages, or database endpoints.

Using iTKO LISA

SSIS package testing with iTKO LISA

What are SSIS packages?

SSIS stands for SQL server Integration Services. SSIS packages can be created in Business Intelligence Development Studio (BIDS). These can be used to merge data from heterogeneous data sources into SQL server. They can be used to populate data warehouses, to clean and standardize data, and to automate administrative tasks. SQL Server Integration Services (SSIS) is a component of Microsoft SQL server 2005. Integration Services provides a platform to build data integration and workflow applications. The primary use for SSIS is data warehousing as the product features a fast and flexible tool for data Extraction, Transformation, and Loading (ETL).

Brief about the Project

A children's organisation working with communities in 48 developing countries and provides programs to over 1.5 million children and their families. Child sponsorship is there main source of funding. They have a staff of over 6000 worldwide and further volunteer force in excess of 50,000 and have regional and field offices. Each regional office operates independently using different fundraising methods within each country. New system was been developed which was been location at their Head Office which was centralized database and all the activities of regional and field offices were been routed through Head Office. Due to its diverse location and vast transaction it was impossible to implement the applications in one go, so the application was rollout in phases. It was impossible to retire the legacy system so to maintain the consistency of the data between and new application and the old legacy system SSIS packages were used.

Testing SSIS packages:

The structure of the database, their were tables from legacy system (around 200+ Tables) where the source data was available, the destination tables (around 300+ Tables) where the data will be inserted, and the audit tables or holding tables which holds the data if there are any invalid records. There were more than 500+ business validation and have more than 20 file formats through which these transactions were supposed to happen. The test conditions were right from the file naming convention, length for the file, till data in the file.

Challenges in Testing SSIS package:

To validate all these business validations, there were many challenges for testing team.

Example:

File name: XXX-YYY-CCYYMMDD-001.NNN

Where XXX will be the details of what type of transaction is there in the file, YYY will be the location code, CCYYMMDD the date format, 001 was sequential number which would be incremented in each system generated file, and NNN will be region code.

File length: There was different specified length of the file from 160 characters to 1022 characters.

Data validation: NNNnnnnnnn this type of code were been used for the children. The expected validation were NNN should be the region code, which should be same as of the file name NNN, remaining 6 characters should be valid child code which are related to the region code.

Apart from this the start and end position of each field, the different validation such as space, special characters and many other validate were involved.

When the test team was been asked to test SSIS packages, they were hardly aware of how to test it. No specific tool was available to test. And due to the huge scope of testing with limited resource it was difficult for the test team to test it manually, as to validate 500+ business conditions was very difficult and also to create the scenarios to validate all the business validations. Many brain storming session were held, people from the team were searching on the net for tool which can help them with testing and which are easy to use. There was a team in the organisation, which was working on testing web services. They were using a tool named iTKO LISA which was specifically used for SOA projects. They were been approached for any suggestion from them. All the constraints in testing were been discussed. The team suggested about the feature of LISA to directly communicate with all type database. No client software was required by the tool to communicate with any database, but just the related .jar files. A Proof of Concept was done which was a great success and later the tool iTKO LISA was used for Test Data creation, and validating the source, target and audit tables.

Solution: itKO LISA

LISA stands for Learn, Invoke, Simulate and Analyze. Used for functional, integration, business process testing of SOA projects and is a no-code software testing solution.

The test steps which are commonly used for SSIS packages testing are:

For Test Data creation:

Parse Text as Response which is used for test data creating, The data set type Read rows from Excel file was used for sample data which was used to create the file which was placed at the shared location where the scheduler was running. The package when executed used to process the file as per the business conditions. To ensure all the business validation:

LISA step SQL Database Execution (JDBC) step was used. Simple select * queries were used. For business validation different types of Assertions were used, which used to validate the error codes against the codes provided in the sample excel file.

SOA Security Testing

Soon I will be sharing my experience in SOA Security Testing.

Service Oriented Architecture

What is Service Oriented Architecture?

SOA is a business architecture where business functionality, or application logic is made available to users or consumers as shared, reusable services on an IT network.
“Services” in an SOA are modules of business or application functionality with exposed interfaces, and are invoked by messages.

What exactly the above definition means?

Business: Every business has these aspects that don't change very frequently. They
often represent a huge part of the business. We call these things the Core Business 
functions.

Example: Gas station sells gasoline in gallons.

Restaurants sells meals from a menu.


These are the core functions of both of the business mentioned above.

There are other things in a business that change very frequently. Prices, Tax rates, catalogs, new products, new customer areas etc. Indeed, business must be able to change, and change quickly, in order to survive. And yet, it is very important that those changes do not affect the core business functions.We know that software that changes frequently should be decoupled from software that changes infrequently. When it is applied to the information management of an enterprise, it is called SOA.

SOA is the practice to separate the core business functions into independent services that don't change frequently. These services are glorified functions that are called by one or more presentation programs. The presentation programs are volatile bits of software that present data to and accept data from, various users.

SOA is nothing more than separating changeable elements from unchangeable elements.

Let us see example –

In internet store-front. Customers use a browser to talk to the presentation software that displays the store’s website. The presentation software interprets the gestures of the customer and invokes services that do things like acquiring the data for the current catalog, or registering the customer’s order. The services have no idea they are talking to a website. They simply accept and return data in a standard format that the web system happens to be able to use.

Why is the separating important?
Consider that internet store-front again. It presents the user with a catalog, allows the user to move items into, and out of a shopping cart, and accepts the eventual order. The presentation of these concepts is very volatile. Marketing people are likely to want to change it frequently. For example, They may wish to present more or less descriptive data in the product list. But none of this has anything to do with the core business functions encapsulated by the services. Those services that acquire catalogs and register orders remain unchanged despite all the presentation thrashing. That’s why the separation is important. It protects the information processing assets of the business .But presentation is not the only thing that will change, but the business processes too. 

Again, consider our store-front. Perhaps our business has decided to offer fine wines as one of the products it sells. Selling alcohol requires that the age of the customer be verified. Let us say that we have a service that provides this verification. This service must be called for any order that contains alcohol products. The decision to call this service is neither a presentation decision, nor a service decision. Rather it is part of the business process for a particular kind of order. Business processes are volatile. As businesses grows they add more and more complications. Therefore we want to separate the business process from the services and from the presentation.

SOA is not about any particular technology. Rather it is a design philosophy that decouples well heeled business functions from volatile processes and presentations. It is the Model-View-Controller of enterprise software 

Value for Quality Professionals and Engineers
Software professionals know testing early and often is essential to having reliable software and reducing the risk of costly failures. However for complex Service-Oriented Architecture (SOA) applications, testing can no longer simply be a phase in the development process. 

· How is testing SOA different from testing typical applications? To avoid the unintended consequences of change in SOA applications, continuous functional and performance monitoring of the business workflow must occur. 

· SOA is by nature a heterogeneous environment, so tests must be able to invoke and verify functionality and performance at every layer of the architecture.

· In SOA, most of the business logic does not reside in the user interface, so tests must span multiple layers of the architecture and support dynamic test data. 

· Management, Policy and Testing are the key supports for an SOA governance strategy. Good SOA testing practices contribute verifiable examples of Policy that are shared throughout the management process.

· Continuous testing must happen at both build and runtime, as each layer of SOA applications can be developed and introduced into the active deployment on its own lifecycle.

· Extensibility of SOA testing is important, since all enterprise environments contain legacy applications and custom components that contain key business functionality.