Downloads for Developers

The news here isn’t that the 'new king-makers', as Savio put it, look a lot like the old kingmakers: developers. The news is that management may finally be realizing it.

- Stephen O'Grady, Redmonk

Developers, developers, [...] developers

- Steve Balmer

Most software platform companies at least partially get it these days - developers drive adoption and quality among technical groups. The software that has quality developers on its side will look better to both business and technical interest groups than the same software that is dragged down by developer indifference or animosity.

These points can be debated to a certain extent.

There are plenty of non-technical power-centers that drive adoption of software platforms in the enterprise. Preferred-vendor arrangements are common and historically were often negotiated at the CIO or higher level, with little developer involvement. Further, application vendors attempt (often with good success) to sell into business units rather than IT groups.

But in both cases developers still drive quality and adoption. Business units that buy applications directly often find themselves in need of connections to other systems or extensions to the application. This means developers are involved, either via an IT group, or as outside consultants.

Meanwhile, preferred-vendor agreements are constantly undermined and even when they are successful they may very well promote homogeneity and management ease at the expense of long-term quality in people and software. In order to make good software development and vendor management decisions, one must be well aware of the world beyond a single vendor bubble.

In order to bring developers on board with a vendor's offerings, to increase general awareness, and to drive sales, vendors need to get their software into the hands of developers. In the case of open source vendors, this is mostly an issue of getting the word out, as the software itself is only a download away. But in the case of more traditional enterprise vendors this can be a complicated proposition. Most enterprise vendors now provide downloads of some version of most of their platform software. Some vendors provide downloads of much of their application software as well.

Some example download sites:

• Oracle: http://www.oracle.com/technetwork/indexes/downloads/index.html
• IBM: http://www.ibm.com/developerworks/downloads/evalmethod.html
• SAP: http://www.sdn.sap.com/irj/scn/downloads

These downloads are usually made under a fairly restrictive license and are usually not available for all parts of the application or platform software. Because of vendors' business model it is somewhat costly to provide these downloads because they appear in a format that is not the standard distribution format for the vendor's software. There are also legal costs associated with writing and maintaining the developer licenses that are applied to these downloads.

I believe that vendors have a tendency to see these sorts of downloads as an overhead cost. They are not. They are a key step in driving both sales and developer adoption, which are closely linked.

Here's how:

• Ability to prototype before purchasing is a key part of the software selection process for responsible companies.
• Today's developers will guide tomorrow's purchasing decisions.
• A healthy developer ecosystem is necessary condition for a strong third-party application ecosystem.
• A skilled, and preferably large, pool of developers is necessary for good project success rates.

In his article "The CIO is the last to know", Billy Marshall talks about the CIO of a financial services company who is surprised to find that his operations people are running Red Hat Linux. This CIO was handed a decision via bottom-up fiat. It is a story that is played out again and again in the enterprise space.

The point is not that CIOs aren't doing their jobs. It's that the decision is inevitably influenced from a different level: the level of those actually carrying out development and operations. Maybe these people aren't actually making the purchasing decisions, but they talk to the people who are. And if someone makes a purchasing decision that development and operations disagree with or are unable to execute, that person is going to hear it. And they'll probably feel it when their group's productivity falls off a cliff.

CIOs either are or should be listening to their developers' opinions. It would be wise for enterprise vendors to divert some sales attention into making sure that those developers have good opinions of their software.

The first step is getting that software into developers' hands quickly and with a minimum of developer effort.

SAP's HANA and "the Overall Confusion"

I threw together a very long response to a very long question on the SCN forums, regarding SAP's HANA application and its impact on business intelligence and datawarehousing activities. The original thread is here and I'm sure it will continue to grow. But since my response was pretty thorough and contains a ton of relevant links, I thought I would reformat it and post it here as well. In order to get a good overview of the HANA situation, I strongly recommend that anyone interested check out the following blogs and articles by several people, myself included:

• The Impact of In-Memory Technology on Data Warehousing and Business Intelligence - by Thomas Zurek (pdf)
• Infocubes and Data Store Objects ... and HANA - by Thomas Zurek
• The BW - HANA Relationship - by Thomas Zurek
• Comparing SAP BW and an Oracle DW - by Thomas Zurek (this is an old blog, but great points about using a DW toolkit vs. loosely coupled tools)
• BW path to the Data Warehouse plus the role of New Technologies - by Vitaliy Rudnytskiy
• SAP HANA: What it means for business and for your career - Vitaliy Rudnytskiy
• SAP In-memory - what is in the bag? - by Vitaliy Rudnytskiy
• What does SAP mean by "In-Memory"? - by Ethan Jewett
• Thoughts and questions about the HANA announcement - by Ethan Jewett
• The latest hot tech vs. fundamental datawarehousing tradeoffs - Remember BW? - by Ethan Jewett

Some of these blogs are using out of date terminology, which is hard to avoid since SAP seems to change its product names every 6 months. But hopefully if you read them they will give you some insight into the overall situation unfolding around HANA. With regards to DW/BI and HANA, these blogs address many of those issues as well. Now, to try answering the questions:

1. Does SAP HANA replace BI?

It's worth noting that HANA is actually a bundle of a few technologies on a specific hardware platform. It includes ETL (Sybase Replication Server and BusinessObject Data Services), Database and database-level modeling tools (ICE, or whatever it's called today), and reporting interfaces (SQL, MDX, and possibly bundled BusinessObjects BI reporting tools). So, in the sense that your question is "does anything change as far as needing to do ETL, modeling, and reporting work to develop BI solutions?", then the answer is no. If you are asking about SAP's overall strategy regarding BW, then this is open to change and I think the blogs above will give you some answers. The short answer is that I see SAP supporting both the scenario of using BW as a DW toolkit (running on top of BWA or HANA) as well as the scenario of using loosely coupled tools (HANA alone, or the database of your choice with BusinessObjects tools) for the foreseeable future. At least I hope this is the case, as I think it would be a mistake to do otherwise.

2. Will SAP continue 5-10 years down the road to support "Traditional BI"?

I hope so. If you read my last blog listed above you will see that HANA actually solves none of the traditional BI problems, and addresses only a few of them. So we still need "traditional" (read "good old hard work") approaches to address these problems.

3. What does this mean for our RDBMS, meaning Oracle?

Very interesting question. For a long time, SAP has supported competitive products to Oracle offerings. In my view, this was to give SAP and its customers options other than the major database vendors, and to give itself an out in the event that contract negotiations with a major vendor went south. So in a sense, HANA can be seen as maintaining this alternative offering. Of course, SAP says HANA is more than that, and I think they are right. Analytic DBMSes have been relatively slow catching on and as SAP's business slants more and more towards BI, the fact is that the continued use of traditional RDBMSes in BI and DW contexts has done a lot of damage by making it difficult to achieve good performance. It's a lot easier to sell fast reports than slow reports :-) So that is another driver. Personally, I don't agree with SAP's rhetoric about HANA being revolutionary or changing the industry. The technologies and approaches used in the ICE are not new, as far as I have seen. As far as changing the industry from a performance or TCO perspective, I'm reserving judgement on that until SAP releases some repeatable benchmarks against competing products. I doubt that HANA will significantly outperform competitive columnar in-memory databases like Exasol and ParAccel. If you are Oracle, you have a rejuvenated, and perhaps slightly more frightening competitor. I don't think anyone really thought that MaxDB was a danger to Oracle, but HANA holds more potential as a competitor to Exadata. Licensing discussions could get interesting.

4. Is HANA going to be adopted and implemented more quickly on the ECC side than BI side first?

Everything I have seen has indicated that SAP will be driving adoption in BI/Analytic scenarios first and then in the ECC/Business Suite scenario once everyone is satisfied with the stability of the solution. Keep in mind, the first version of HANA is still in ramp-up. SAP is usually very conservative in certifying databases to run Business Suite applications.

What is a commodity?

Lately I've seen several discussions of commoditization in the enterprise software space. Or perhaps "commoditisation", depending on which side of the Great English-language Divide on which you happened to spend your school years. Specifically the claim has been made that applications (ERP, BI, analytics) or skills (programming, project management, for example) are becoming commoditized.

I'm not going to link to any of these claims because my claim is that the word is being subject to rampant misuse and I'd rather not call out anyone specific. The misuse is widespread and I don't think it would be fair to name individuals.

This misuse is a shame. There is a lot of really useful economic and social theory around commoditization. Incorrectly labeling a market trend as "commoditization" creates the incorrect impression that these bodies of theory are applicable, and this can result in incorrect analysis.

See, for example, the insightful discussion of commoditization as a competitive strategy in this blog about Facebook's Open Compute Project by Marco Arment. When we use the term incorrectly, we poison the well from which this sort of analysis is drawn.

So, what is commoditization? Wikipedia, as usual, has a fairly good definition. To break it down, a commodity is:

1. Undifferentiated - a commodity from supplier X is basically the same as the same product from supplier Y
2. Fungible - an instance of a commodity can easily be switched for another instance of the same commodity without significant impact on the user of the commodity - in other words, a commodity has low switching costs

When a product is a commodity it will usually have a price that is determined by a market of exchange. Markets are not always efficient, and very few products are truly commodities, but there are some products that come fairly close. Wikipedia gives several examples, but here are a couple for your consideration:

• Salt - All salt is basically the same, and switching from one brand of salt to another has no noticeable impact on the user (not withstanding "premium" salts, for example sea-salt).
• Unskilled labor - The initial pay of grocery store shelf-stockers, for example, is determined primarily by market forces or minimum wage as the labor pool is considered fairly undifferentiated and the cost of hiring a different employee is fairly low in some labor markets.

So what about enterprise software? I can't think of anything that's a commodity in enterprise software. Maybe servers, as the advent of virtualization and cloud computing begins to lower switching costs (improving fungibility).

What is not a commodity in enterprise software? Lots of stuff. Here are some examples:

• ERP and BI software - Different offerings are still quite differentiated, and switching costs are astronomical because of the amount of customization required of all solutions. Additionally, cloud vendors are now creating data lock-in scenarios that can make it very difficult to migrate old data to a new solution.
• Switching from "build" to "buy" does not commoditize a market - An IT department switching from a "build" to a "buy" approach, or a vendor pushing solutions that require less customization, does not result in commoditization of a market. This is because different solutions are still differentiated based on features, performance, or ease-of-use, and because switching costs remain high. Switching costs should be lower when going from a custom solution to an "off-the-self" solution than the other way around, perhaps making the product more commodity-like, but this is a stretch. Implementation costs are still going to be quite high.
• Developers and consultants - There is lots of suspect research talking about how the best developers are some multiplier (usually 6-20X) more productive than the average developer. In fact, it's probably worse because this research tends to focus on long-term employment. Because of the time taken for on-boarding (switching costs) and the increased administration costs that come with a larger team, hiring a middling developer or consultant for your project can often make the project progress even more slowly than hiring no one at all!
• Tool-kits - Development tool-kits have a real impact on the performance of custom development. The choice to use language A versus language B for a given development project is not academic. The differentiation equation depends on your existing skill-set as well as features of the tool-kit and switching costs are high due to the need for retraining and reorientation.

Commodity theory does not apply to any of these areas. The goods are not fungible and the products are differentiated.

Any vendor who says otherwise is probably peddling a subpar product (or labor). Any IT department that believes this is probably making some bad purchasing decisions. And every purchasing department likely talks to their suppliers about how this is a commodity market ... because they are trying to negotiate a better price.

Thoughts on what's next for Apache ESME

I'm a committer on the Apache Enterprise Social Messaging Environment (Apache ESME). At least I think that's what it stands for today. We sort of looked at SAP's approach where the acronym for a product changes every year or so and we maybe went a little too far in the opposite direction, refusing to change the acronym even when we need to change the name - teen vampire romances be damned!

In any case, ESME takes a lot of its cues from Twitter, but with a focus on the needs of the enterprise. To that affect, we built in Scala (which runs on the JVM and provides easy integration with Java code), used David Pollack's super-scalable Lift framework and an actor-model designed by David, then we added features like message pools (which allow for groups of people to exchange non-public messages), the ability to post more than 140 characters, and the ability to follow not just people, but also tags and conversations.

Maybe that gives you the idea, maybe not. You can always go try it for yourself athttp://esmecloudserverapache.dickhirsch.staxapps.net/ or help out with the project - we can use help in many areas.

We've managed to do lots of other cool stuff in the context of ESME, but what I want to write a little bit about is what I think we still have ahead of us.

Distributed Twitter and federation

Talk of a non-centralized version of Twitter sprung up in earnest a couple of years ago with a post by Dave Winer, the inventor of RSS. The initial context was Twitter's regular downtime as it struggled with scaling, but the larger context quickly because the concern that we can't trust a single company to properly steward an enormous piece of communication infrastructure. The concern is basically about the Facebook-ification of Twitter. GigaOm has a prettydecent overview of the current state of the discussion.

From an enterprise perspective, this concern is even more motivated. Most enterprises still tend towards on-premise software by default and it is unclear if a messaging service is well-suited to a SaaS deployment option. Some companies, like Yammer (pure SaaS) and Status.net (open source - SaaS and on-premise options), are working on delivering a Twitter-like solution for the enterprise, but we aren't there yet.

Key requirements for a distributed Twitter service:

1. Inter-operable Federation - Status.net has worked to introduce the OStatus standard, and this is an excellent start. However, the inter-operability of this protocol is relatively untested. I'd like to see if we can make this work for ESME, but it is going to take some additions to the protocol to manage pooled messages, for example.
2. Follow any feed - Friendfeed had this capability, and ESME provides it in a bit of a different manner through our actions (though actions - Vassil's brainchild - do far more than this). I sometimes think of this capability as light-weight, or one-way, federation.
3. Real-time updates from federated data source - Not only do we need to be able to follow feeds, we need to get updates from those feeds nearly instantly. PubSubHubbub (PuSH) is probably the most wide-spread solution here, and it is the solution that OStatus uses. But PuSH has weaknesses around authorization of subscriptions to private feeds, and here it would need to be rolled up with another standard like OAuth.
4. Updates available as (protected) feeds.

Status.net seems to be the most on-the-ball with regards to these requirements, but there is a need for variety and for a tool like ESME that was built with business users in mind.

Real activities and objects as social objects

One important ability for an socially-oriented messaging system is that it makes business objects into first-class social objects. This is what John Tropea is getting at when he talks about the ability to follow conversations and tags. These objects should be first-class members of the messaging environment, supporting following and real-time updates.

We should also be able to integrate real business objects and business activities into the system as first-class objects. "I want to follow this customer", should be a desire that we support. Currently we offer a couple of ways to do this:

1. Bring an activity into the ESME system as a message, either via the API or through an action that pulls an RSS feed. This message (or rather, the conversation around it) is a first-class messaging object in ESME, so if people want to see responses to an action, then they can follow the action. For example, I have actions set up onhttp://esmecloudserverapache.dickhirsch.staxapps.net/ that pull my new Twitter messages and newly created ESME Jira tickets in to my timeline.
2. Bring an object into the ESME system as a tag, again using the API or an action. The tag then acts as the object that we can follow. We currently allow this as well, and it was used heavily by Sig Rinde in his prototype during his quite awesome prototype integration of the object-oriented business process engine (OOBPE?) Thingamy with ESME. But we could stand to have some more functionality for extracting tags and metadata from RSS feeds, allowing us to use this tag-as-object approach in a richer way.

We're currently thinking about ways to make our system more extensible and further enable the representation of business activities and objects as first-class objects in the messaging environment. We'd certainly like all the help we can get thinking about this topic.

Easier integration into other software products and environments

The capability that actions give us to pull in RSS and Atom feeds is really important. It means that ESME can integrate with systems that were not designed with ESME, or even social messaging in general, in mind. In turn, we need to improve our APIs to allow easier integration of ESME into other tools. Part of this involves doing things like providing RSS and Atom representations of timeline, probably via open standards like the Activity Streams standard.

On the somewhat more complex side, in will probably involve supporting other existing and emerging standards like allowing use of LDAP for authorization, LDAP groups for automatic pool creation, OpenSocial, PubSubHubbub for push-based feeds, using OAuth in our API, providing more semantic and linked information about data via our API, and supporting actions pulling Atom from OAuth-protected resources.

When I put it like that, it sounds like a lot, but it also sounds really exciting!

So, I'm sure I've missed a lot here, but these are just my thoughts about directions I'd like to see ESME move over the next few releases. Got ideas about where the project should go? We'd love to hear them :-)