esjewett.com

Well, that title was going to be perfect flame-bait, but then I went all moderate and decided to write a blog that actually matters. So here's the low-down:

There's a lot of talk lately about in-memory and how it's the awesome. This is especially true in the SAP-o-sphere, primarily due to SAP's marketing might getting thrown behind Business Warehouse Accelerator (BWA) and the in-memory analytics baked into Business ByDesign.

I'm here today to throw some cold water on that pronouncement. Yes, in-memory is a great idea in a lot of situations, but it has its downsides, and it won't address a lot of the issues that people are saying it addresses. In the SAP space, I blame some of the marketing around BWA. In the rest of the internet, I'm not sure if this is even an issue.

Since I've actually done a fair amount of thinking about these issues (and as a result I troll people on Twitter about it), I thought maybe it'd be helpful if I wrote it down.

So let's get down to brass tacks:

How in-memory helps

In short: it speeds everything up.

How much? Well, let's do the math:

Your high-end server hard drive has a seek time of around 2 ms. That's 2*10^-3 seconds (thanks Google). Yes, I'm ignoring rotational latency to keep it simple.

Meanwhile, fast RAM has a latency measured in nanoseconds. Let's say 10ns to keep it simple. That's 10^-8 seconds.

So, if I remember my arithmetic (and I don't), RAM is about 2*10^5, or 200,000 times faster than hard disk access.

Keep in mind that RAM is actually faster because the CPU-memory interface usually supports faster transfer rates than the disk-CPU interface. But then, hard disks are actually faster because there are ways to drastically improve overall access performance and transfer rates (RAID, iSCSI? - not really my area). Point is, RAM helps your data access go a lot faster.

But ... er ... wait a second (or several thousand)

So here I am thinking, "Well, we're all fine and dandy then. I just put my job in RAM and it goes somewhere between 100,000 and 1,000,000 times as fast. Awesome!".

But then I remember that RAM isn't a viable backing store for some applications, like ERPs (no matter what Hasso Plattner seems to be saying) or any other application where you can't lose data, period. Yes it can act as a cache, but your writes (at least) are going to have to be transactional and will be constrained by the speed of your actual backing store, which will probably be a database on disk.

And then I see actual benchmarks meant to reflect the real world like this. For those who won't click the link, the numbers are a bit hard to read, but I'm seeing RAM doing about 10,000 database operations in the amount of time it takes a hard disk store to do about 100. That's only a 100x speedup.

Ok, now I'm back down to earth and I'm thinking, "I just put my job in RAM and I'll get maybe a 50-100x speedup but at the cost of significant volatility". (I'm also thinking that SAP's claimed performance improvements of 10x - 100x sound just about like what we'd expect.)

This is still really really good. It makes some things possible that were not possible before and it makes some things easy that used to be hard.

And finally, why in-memory doesn't matter

But really, what is the proportion of well-optimized workloads in the world? How often are people going to use in-memory as an excuse to be lazy about solving the actual underlying problems? In my experience, a lot. Already we are hearing things along the lines of, "The massive BW query on a DSO is slow? Throw the DSO into the BWA index." [Editor's note: A DSO is essentially a flat table. Also, the current version of BWA doesn't support direct indexing of DSOs, but it probably will soon, along with directly indexing ERP tables.]

Now's the part where we who know what we're doing tear these people to shreds and tell them to implement a real Information Lifecycle Management system and build a Inmon-approved data warehouse using their BW system (BW makes it relatively easy). Then that complex query on a flat table that used to take two days of runtime will run in 30 seconds.

Well, that would be one approach, but frankly most people and companies don't have the time or the organizational maturity in their IT function to pull this off. And in this world, where people have neither the time nor the business process for this sort of thing, then it starts to make sense to spend money on it, and something like BWA is a great thing in this context.

But it's not great because it's in-memory. It's great because it takes your data - that data you haven't had the time to properly build into a datawarehouse with a layered and scalable architecture, highly optimized ROLAP stores, painstakingly configured caching, and carefully crafted delta processes - and it compresses it, partitions it, and denormalizes it (where appropriate). Then, as the icing on the cake, it caches the heck out of it in memory.

Let's be clear: BW already has in-memory capabilities. Livecache is used with APO, and the OLAP cache resides in memory. The reason BWA matters is not that it is in-memory. It matters because it does the hard work for you behind the scenes, and partially because of this it is able to use architectural paradigms like column-based stores, compression, and partitioning that deliver performance improvements for certain types of queries regardless of the backing store.

In-memory is great, and fast, and should be used. But in most ways that are really important, it doesn't matter all that much.

As might have been assumed from the my post on automated testing in SAP systems a couple months ago, I've been delving into testing in the SAP landscape. I'm beginning to put together a series of presentations and workshops on the subject, the first of which I was delighted to deliver last week.

Being the first in the series, this presentation focuses on an overview of the leading edge of the field. It would be nice to be able to emulate the sort of testing techniques we can use in Ruby in SAP BI and EPM application development. Nice, but not necessarily realistic. I can dream!

This is another in my series of notes to myself that might somehow come in handy to others, so I'll post them to the blog. This is more of a brainstorm than anything else, but some thought has gone into this area. In terms of actual testing going on in the SAP world, it probably displays my ignorance more than anything, but I'd love to be corrected. If you aren't interested in SAP or in testing methodology, then I recommend you stop reading now!

Based on what I've seen in my admittedly shortish time in the SAP world, the state of automated testing is sadly lacking. This is not to say that the facilities for testing don't exist, but perhaps that they are not adequately ergonomic or well understood. The result is that most implementations don't have a mature testing automation strategy, which is a key component of a modern approach to software development.

Unit testing

In the unit test arena we're doing alright as we have most of the standard Java capabilities on the Java development side and we have ABAP Unit on the ABAP side of the house. Though, I'll note that you've got to be a bit careful in designing your ABAP Unit tests to handle more than one failure per execution. There also isn't much of a facility for handling alternative testing strategies such as specification derived tests or behavioral testing. Part of this is due to the limitations of ABAP as a (mostly) statically typed, strictly object-oriented language. Part of this inflexibility is simply due to the fact that the necessary facilities have not been developed.

I'll also note that while Java isn't my cup of tea, apparently setting up automated unit testing using JUnit on the Java side of the SAP world isn't straightforward. The SDN blog "Test-driven development with Composite Application Framework (CE 7.1)"looks like a decent overview of the process of getting JUnit testing set up.

With SAP's focus on positioning itself as a full-featured development platform for the enterprise and the growth of agile, modern development methodologies in the enterprise, I'm surprised that unit testing and test-driven development aren't getting more attention from SAP as an integral part of the application lifecycle.

Performance

In the performance testing area a lot of work has been done to work out transactional systems under load. However, I'd argue that we need to understand the limits of this sort of testing. Usually it is focused on a single transaction or report, which is fine for testing the maximum performance of that piece of the system, but the tester must have a significant knowledge of the underlying architecture of the system in order to really test for table write contention or the performance of specialized processing logic under load.

Performance testing gets really worthwhile only when the level of knowledge of the system architecture reaches a level where a significant number of these trouble spots can be identified and tested under load in a test environment. As it happens, we're mostly doing this type of testing in production in the SAP world, which is the situation we are ostensibly trying to avoid by doing "performance testing" in a pre-production test system.

Jonathon Kohl writes knowledgeably of some of the complexities inherent in performance testing. I think we would be wise to admit the limits of performance testing in an environment as complex as most SAP implementations. This is not to say that we shouldn't do it, but simple load testing isn't getting us very far and we might be better served by focusing on replicating real system use or focusing on a more exploratory version of performance testing.

Functional/Behavior

Up until this point we've had cursory automated testing occurring in most implementations. But when we get to functional testing things start to fall apart. There is a lot of potential contention over the right place to interface into the application to do functional testing. Most MVC frameworks provide an interface into the controller level to mock view requests and model contents. Some believe that the view/interface level is a good place for this kind of testing, which makes a certain amount of sense if the framework doesn't have a clear delineation between the user interface and the application logic or won't commit to the delineation that exists.

The options for functional testing in delivered SAP systems are pretty much eCATT or an external testing tool that hooks in at the UI or API level. eCATT is not an ideal functional testing tool on some reads of "proper" functional testing but should be usable. I'll be looking into some eCATT good practices and testing strategies in a future article. There are also some options from vendors like Mercury (now HP).

It should be possible to hack together a working external automated functional testing system for your organization's SAP landscape and chosen development method, but it would be expensive to maintain across significant upgrades. For example the recent Netweaver 7.0 enhancement package 1 upgrade would probably break most browser driven testing of WebDynPro ABAP/Java applications because of the shift from full server round-trips to a more AJAX-y user interface programming approach. Multiple levels of abstraction in the test harness would be necessary to limit breakage to predictable and manageable layers of the testing stack.

I still think it is worthwhile to pursue this strategy if the built-in support for functional testing from SAP does not satisfy the needs of an organization. The automated approach done well can provide for better coverage, accuracy, and speed, while saving a bunch of un-fulfilling person-hours spent manually exercising a system while struggling to stay awake. There may be room for a vendor to build up and maintain standard APIs across versions of SAP systems and I'm sure that vendors like Mercury are working on this, at least in transactional systems. Heck, there may be room for an open source project to do the same using a browser driver like WATIR. Hmmmm...

BI and performance management applications still seem to be pretty much an open field here.

Why automation

The real power of automated functional testing starts to show up when we consider doing things that we basically don't do in SAP projects. Coupled with an automated system build process (yes, we probably need to talk about this too), automated test suites allow for

• Full nightly functional regression tests
• Running the full test suite on a contemplated change or note application
• Full regression testing after upgrading
• Behavior driven development (BDD) or business applications, where the specs are the business requirements
  
• Continuous integration for every level of the development and maintenance landscape

The potential gains in productivity and reductions in risk that can be gleaned from engaging in a disciplined testing methodology (be it focused on coverage, TDD, BDD, etc) appear large. Certainly worth more thought and study.

Stephen O'Grady has a nice post clarifying his position on the telemetry opportunity in the enterprise, for open source vendors especially. "Telemetry" being here the practice of contracting with customers to send data back to the vendor, which the vendor then uses to provide some value add the to the customer. This value might take the form of better software, optimization tips tailored to the customer, or metrics comparing the customer to the full pool of customers.

Oddly enough, this might be one of those places where the enterprise is a bit ahead of the opensource world. Personally, I'm aware of SAP providing several telemetry-like services focuses on performance, maintenance, and bug/problem reporting.

It's also worth pointing out that SAP's traditional business model of partnering with customers to understand processes and develop business applications could be considered telemetry by some definitions. True, it's human-based telemetry rather than a technical system of reporting back to the vendor, but the point is the same: the customer transmits information about how they use or will use the software to the vendor while the vendor uses that information to provide a better experience to all customers.

In the open source space my impression is that we see telemetry (such as it is) focused primarily in the software quality space. Crash reporting is probably the primary place we see this, as in the ubiquitous Firefox beta crash reporter.

As O'Grady point out, software as a service companies have recognized this are as a great opportunity to develop competitive advantage. Google uses telemetry data extensively in its search product (suggestions, especially), as well as in other products like Analytics (benchmarking) to name an example in the enterprisey-er space.

In the consumer space, we're starting to see many tentative steps. Wasabe and Mint are both financial apps that allow the user to compare their activities against the aggregated activity of relevant slices of the apps' user bases. The idea is powerful, though I'm not particularly impressed by the utility of the implementations we're seeing outside of Google. It seems like apps are going to need to give up on trying to get the user to dig through this data goldmine and instead just deliver up specific nuggets, as Google does with its search suggestions.

For the second post in my pluggability series (series? seriously?), I'll return to the issue of trust, specifically what I was talking about when I said

Trust (between systems) in the enterprise has been all or nothing. With the emergence of SaaS and the integration of consumer services, this approach doesn't work anymore. Access delegation capabilities are becoming essential. OAuth is part of the answer to this issue. Standards, architectures, and UIs that can handle multiple delegation are probably the end-state solution.

Let's take this apart a bit, with the caveat that I just fiddle with security schemes and my real current interest is in the data user interface area, roughly.

There are several different types of trust between systems and I'm not going to capture them all in the discussion below, but the basic types are trusted systems (where we ask the question, "Do I, system X, trust the integrity of system Y?"), identity verification and authentication ("Do I trust that this user/system is who it says it is?"), and authorization ("Given that this user/system is identified, what access do I trust it with?").

This is all relatively straightforward in the classic situation of a user trying to log onto a system. Yes, the user is trusted. Identity is verified via a username/password pair and we protect against man-in-the-middle attacks using a VPN. Authorization is provided for on a user-by-user basis.

It's also straightforward in the circumstance where a server is attempting to communication with another server. Again, integrity is assumed because the server is known, authentication is carried out under one of a number of schemes, and authorization is on a per-system basis.

But when we're now in a world of agent systems acting on behalf of users, and these systems start to fall apart. How do I treat a system that claims it is acting on behalf of a user? Do we grant authorization appropriate to the user, or just punt on the question and let the trusted system figure it out? How can we be sure that the system is acting on behalf of the user that it says it is acting on behalf of. What if the system requesting access is one we've never heard of before? Do we trust it?

This is the problem of access delegation. (To be clear, this is not the related problem of single-sign-on.) There are a couple of general approaches to access delegation in the enterprise, neither of which access delegation:

• Punt and let the trusted system figure out the requisite authorizations - this is clearly not an acceptable approach for much sensitive data unless the authorization scheme is designed as a whole across systems, in which case we're not really talking about two separate systems and we certainly aren't looking at a pluggable architecture that might at some point become user-definable.
• Implement trust along with a single-sign-on mechanism that assumes the trusted system will vouch that user X is requesting access (usually by sending along the ID and password of the user requesting access). Authorizations assigned to user X can then be afforded to requests coming from the trusted system on behalf of that user. This is not delegation, this is impersonation - the approach is better from an authorizations standpoint and allows systems to be somewhat "pluggable" at the developer level, but it doesn't address two concerns:
  • There is no capability inherent in such an architecture for user-driven pluggability because trust is still all or nothing with IT deciding which systems are to be trusted for impersonation (as it should).
  • This type of system simply treats the requesting system the same way it treats the user itself. In many situations we may want to give a requesting system less access or different access from the user. We may want to give different requesting systems different access, even when they are requesting on behalf of the same user. For example, we might have a trusted, stand-alone internal reporting system and a less trusted prototype web-based dashboard system. Certain classes of data may simply be deemed to sensitive for prototype systems, even if the user is trusted with the general class of access. Such an architecture leaves little room for this sort of granularity except on an implementation-by-implementation basis.

So what would have to happen to allow for user-driven pluggability?

First, authorization schemes that allow for layered access controls by user and system. In some organizations, access from anonymous systems and applications on behalf of users would be allowed, and this access would be granted at a default minimal level (display only, no access to sensitive data, etc.).

Second, in all organizations a path to request and receive certification of a given application at a particular access level should be provided for. I don't have high hopes for this in a world where a large number of companies still block GMail at their corporate firewalls, but a boy can dream.

Third, standards for access delegation must become available so as to allow for automatic interoperability of delegated access systems. We still may need an open loop in an enterprise environment, as in step 2, but it shouldn't be technically required. OAuth is probably the best bet here, as all enterprise schemes appear to be pretty much vendor-specific. Hopefully the IETF push will start to propel OAuth into some enterprise vendor's products, though I'm not holding my breath.

Fourth, working out the user interface is an ongoing major issue. Single delegation is hard enough right now, but my impression is that the OAuth community is getting pretty close to nailing it down. Multiple delegation is going to be a real headache, but people are starting to think about it, and there's no question in my mind that it's where we need to get for pluggability to be really workable while maintaining (Question: why do I bother writing this stuff when someone at Google has already done it?)

This all begs for a bit of an example. Google's example (link) is a good combined consumer-web/enterprise thought experiment, but it's a target for a few years out, certainly. More simple examples done well (reporting systems signing onto data-warehousing or operational systems, for example) are hard enough to come by in the enterprise as is. Hopefully that'll be content for another post.