esjewett.com

As might have been assumed from the my post on automated testing in SAP systems a couple months ago, I've been delving into testing in the SAP landscape. I'm beginning to put together a series of presentations and workshops on the subject, the first of which I was delighted to deliver last week.

Being the first in the series, this presentation focuses on an overview of the leading edge of the field. It would be nice to be able to emulate the sort of testing techniques we can use in Ruby in SAP BI and EPM application development. Nice, but not necessarily realistic. I can dream!

This is another in my series of notes to myself that might somehow come in handy to others, so I'll post them to the blog. This is more of a brainstorm than anything else, but some thought has gone into this area. In terms of actual testing going on in the SAP world, it probably displays my ignorance more than anything, but I'd love to be corrected. If you aren't interested in SAP or in testing methodology, then I recommend you stop reading now!

Based on what I've seen in my admittedly shortish time in the SAP world, the state of automated testing is sadly lacking. This is not to say that the facilities for testing don't exist, but perhaps that they are not adequately ergonomic or well understood. The result is that most implementations don't have a mature testing automation strategy, which is a key component of a modern approach to software development.

Unit testing

In the unit test arena we're doing alright as we have most of the standard Java capabilities on the Java development side and we have ABAP Unit on the ABAP side of the house. Though, I'll note that you've got to be a bit careful in designing your ABAP Unit tests to handle more than one failure per execution. There also isn't much of a facility for handling alternative testing strategies such as specification derived tests or behavioral testing. Part of this is due to the limitations of ABAP as a (mostly) statically typed, strictly object-oriented language. Part of this inflexibility is simply due to the fact that the necessary facilities have not been developed.

I'll also note that while Java isn't my cup of tea, apparently setting up automated unit testing using JUnit on the Java side of the SAP world isn't straightforward. The SDN blog "Test-driven development with Composite Application Framework (CE 7.1)"looks like a decent overview of the process of getting JUnit testing set up.

With SAP's focus on positioning itself as a full-featured development platform for the enterprise and the growth of agile, modern development methodologies in the enterprise, I'm surprised that unit testing and test-driven development aren't getting more attention from SAP as an integral part of the application lifecycle.

Performance

In the performance testing area a lot of work has been done to work out transactional systems under load. However, I'd argue that we need to understand the limits of this sort of testing. Usually it is focused on a single transaction or report, which is fine for testing the maximum performance of that piece of the system, but the tester must have a significant knowledge of the underlying architecture of the system in order to really test for table write contention or the performance of specialized processing logic under load.

Performance testing gets really worthwhile only when the level of knowledge of the system architecture reaches a level where a significant number of these trouble spots can be identified and tested under load in a test environment. As it happens, we're mostly doing this type of testing in production in the SAP world, which is the situation we are ostensibly trying to avoid by doing "performance testing" in a pre-production test system.

Jonathon Kohl writes knowledgeably of some of the complexities inherent in performance testing. I think we would be wise to admit the limits of performance testing in an environment as complex as most SAP implementations. This is not to say that we shouldn't do it, but simple load testing isn't getting us very far and we might be better served by focusing on replicating real system use or focusing on a more exploratory version of performance testing.

Functional/Behavior

Up until this point we've had cursory automated testing occurring in most implementations. But when we get to functional testing things start to fall apart. There is a lot of potential contention over the right place to interface into the application to do functional testing. Most MVC frameworks provide an interface into the controller level to mock view requests and model contents. Some believe that the view/interface level is a good place for this kind of testing, which makes a certain amount of sense if the framework doesn't have a clear delineation between the user interface and the application logic or won't commit to the delineation that exists.

The options for functional testing in delivered SAP systems are pretty much eCATT or an external testing tool that hooks in at the UI or API level. eCATT is not an ideal functional testing tool on some reads of "proper" functional testing but should be usable. I'll be looking into some eCATT good practices and testing strategies in a future article. There are also some options from vendors like Mercury (now HP).

It should be possible to hack together a working external automated functional testing system for your organization's SAP landscape and chosen development method, but it would be expensive to maintain across significant upgrades. For example the recent Netweaver 7.0 enhancement package 1 upgrade would probably break most browser driven testing of WebDynPro ABAP/Java applications because of the shift from full server round-trips to a more AJAX-y user interface programming approach. Multiple levels of abstraction in the test harness would be necessary to limit breakage to predictable and manageable layers of the testing stack.

I still think it is worthwhile to pursue this strategy if the built-in support for functional testing from SAP does not satisfy the needs of an organization. The automated approach done well can provide for better coverage, accuracy, and speed, while saving a bunch of un-fulfilling person-hours spent manually exercising a system while struggling to stay awake. There may be room for a vendor to build up and maintain standard APIs across versions of SAP systems and I'm sure that vendors like Mercury are working on this, at least in transactional systems. Heck, there may be room for an open source project to do the same using a browser driver like WATIR. Hmmmm...

BI and performance management applications still seem to be pretty much an open field here.

Why automation

The real power of automated functional testing starts to show up when we consider doing things that we basically don't do in SAP projects. Coupled with an automated system build process (yes, we probably need to talk about this too), automated test suites allow for

• Full nightly functional regression tests
• Running the full test suite on a contemplated change or note application
• Full regression testing after upgrading
• Behavior driven development (BDD) or business applications, where the specs are the business requirements
  
• Continuous integration for every level of the development and maintenance landscape

The potential gains in productivity and reductions in risk that can be gleaned from engaging in a disciplined testing methodology (be it focused on coverage, TDD, BDD, etc) appear large. Certainly worth more thought and study.