esjewett.com

There has been some talk lately (from the Java performance maven @kohlerm and others) about JAX-RS in the context of the API of a Lift application - specifically ESME. JAX-RS is a Java annotation framework for programming RESTful web services (APIs for the non-enterprisey out there).

The real goal of the talk about JAX-RS with regards to ESME (as I see it, and I'm not the only point of view on this by a long shot), is to create an API that is as RESTful as reasonably possible, and which (for most resources) is indistinguishable from a JAX-RS implementation from the perspective of a client consuming the API.

It appears at first glance that the easiest way to achieve the goal of indistinguishability from a JAX-RS implementation is to do a JAX-RS implementation. I'm not convinced it is so straightforward.

Here are the requirements for a platform for implementing an HTTP API for ESME, in my view:

1. Access to ESME instance objects as instantiated in the Lift application (preferably we are not using the database as the way that the API and the application communicate, especially since ESME doesn't necessarily even have a DB in my understanding)
2. Respect the ESME security architecture (meaning that the API must act as a specific user)
3. Support delta/streaming collections over HTTP in addition to REST-only resources and collections

Meanwhile, I've found a few instances of people attempting to use JAX-RS in the context of a Lift application. Unfortunately, I don't think they actually meet these requirements.

Here (http://blogs.sun.com/sandoz/entry/using_scala_s_closures_with) we have a discussion of how we can implement JAX-RS provider classes in Scala instead of in Java. This is, admittedly very attractive, but I'm not seeing a way to import the entire Lift application context into these provider classes. Or rather, enough of the context to satisfy my requirements 1 & 2 above.

I had a fleeting moment of hope when I saw lift-jersey and this thread where we have James Strachan writing a Lift module that appears to allow us to use the Lift templating language within a JAX-RS implementation in Scala. However, it looks to me like this only allows using the Lift templating language, and by and large replaces the Lift request-handling stack with JAX-RS. This is sort of like what we want to do, but not really, and I think we're going to have the same struggles with integrating the Lift application itself that we would have had with the first example.

Where we'll really start running into trouble, even if we can satisfy requirements 1 & 2, is in requirement 3. In ESME, we some resource collections that need to be provided in a "delta" or "streaming" format to our clients, but also provide more orthodox RESTful HTTP interfaces. JAX-RS doesn't appear to be very friendly to the delta/streaming problem-space, so we will actually be forced to implement the streaming parts of these resources in Lift/Scala directly. This means that we can't just cordon off a portion of the URI-space of the ESME application to be served by Jersey or another JAX-RS container, outside of the context of the Lift application. The two containers would need to be very closely intertwined in the URL-space of the ESME application. In other words, we'd need to pattern-match before the request even hits Jersey and JAX-RS, which kind of defeats the purpose.

So, that little exploration was relatively fruitless, but maybe this blog will attract some comment setting me straight on how to do this.

Meanwhile, I had the thought "What's so great about annotations anyway?" My first impression is that there is nothing so great about annotations. First impressions are usually wrong, but it got me thinking along the lines that what JAX-RS does with annotations is suspiciously similar to what the Lift request dispatcher does with pattern matching. Here, let's loosely borrow an example from the lift-jersey Github page.

import javax.ws.rs.{Produces, Path, GET}
/**
* @version $Revision: 1.1 $
*/
@Path("/resourceReturningTemplateView")
class ResourceReturningTemplateView{
  @GET
  def view() = <xml_container>Some Text</xml_container>
}

So, all this says is that when we receive a GET request to the path "/resourceReturningTemplateView" in our application, return the results of the view() method, which in this case is a string of XML.

What does this look like in a native Lift? Well, ignoring the other Lift incantations that need to be done (which are really just one line in Boot.scala, a class definition, and an implicit function that converts from a NodeSeq or Elem to a LiftResponse), all that's required is something like

def dispatch: LiftRules.DispatchPF = {
  case Req("resourceReturningTemplateView" :: Nil, _, GetRequest) =>
    <xml_container>Some Text</xml_container>
}

Lift request matching appears to be pretty powerful, so I started looking at whether it can cover the use cases of JAX-RS. The upshot is that I think it does pretty much everything we need, so I'd prefer to stick with the Lift pattern matching over JAX-RS annotations for the time being. Some things may be a bit more complicated in Lift, like the @Consumes annotation and the Allow header value, or direct handling of form fields as specified in the @FormParam annotation. However, I think these are all doable and just require good patterns be developed.

Final thought: I'm pretty sure that JAX-RS isn't really an API for RESTful web services. It's an API for pattern-matching HTTP requests in a language (Java) that doesn't have native pattern-matching.

Final disclaimer: No actual code was harmed, or tested for that matter, in the writing of this blog. In other words, that code up there probably doesn't work, and that's nobody's fault but my own.

This is a quick how-to, showing how to route your Twitter feed through Yahoo! Pipes to filter and cleanse it, then consume it using ESME actions.

I've created a parametrized Yahoo! Pipe here where you can input your Twitter username and get back a feed of all mentions of "ESME", "esme", or "Esme" from your Twitter timeline.

(Pipe address: http://pipes.yahoo.com/pipes/pipe.info?_id=fJZ29qrr3RGjLPJI6icw5g)

Once the "Run Pipe" button is pressed, the filtered time-line displays. I've also removed the username from the beginning of the Tweets, for better display in ESME.

Feel free to use this pipe with your username, or "Clone" it and modify it as you like. The structure of the Pipe as it is now is available as an image here.

Once the pipe has been run, it is possible to retrieve the output as an RSS feed. Copy the link for the RSS feed for this pipe, as we will need to consume it using an ESME action.

Now navigate to your ESME server. We need to use a relatively recent version of ESME, specifically a version including the recently merged actions branch. I'm using http://esmecloudserverapache.dickhirsch.staxapps.net/ in this example.

Log in to the ESME server and create a new action. Fill in the action name as desired. For the test, use the "every N mins" test. I'm chosen 5 minutes as my interval, which means that the pipe feed will be queried every 5 mins for new entries.

For the Action, use the rss: action type, followed directly (no space) by the URL of the Pipes RSS feed that we copied earlier. Your action form should look like this.

Click the "Add" button and your recent Tweets containing the word "ESME" will start showing up on the ESME server. One neat aspect of this is that any hashtags (#esme for example) in your Twitter updates will be converted to ESME tags automatically.

This only works for public Twitter updates, as Yahoo! Pipes doesn't support authentication for feeds. You could query your private Twitter feed directly from the ESME server, embedding your Twitter username and password in the URL, but I strongly recommend against this as your username and password will be stored in the ESME server database, which you do not control.

The Enterprise Social Messaging Experiment (ESME) project grew out of a collaboration in the SAP world and is now a project in the Apache Incubator. The project itself is an interesting and inspiring demonstration of the organizing power of so-called web 2.0 tools, which allowed the project to go from an idea to an Incubator project in about half a year through the efforts of a wide-spread group of individuals, many of whom have never met. I've been on the sidelines of the project, looking on, but I've been fascinated by some conversations that have taken place around the API.

The ESME API is often described as a "REST" API or a "RESTful" API. "REST" refers to the design principle of Representational State Transfer. Wikipedia has a good overview. REST is often seen as an alternative to RPC APIs, or "Remote Procedure Call" APIs. At root, the difference as described by Wikipedia is that RPC is about telling an application to do something while REST is about changing the state of the resources of an application.

The upshot is that RPC can be thought of as interfacing in verbs, while REST can be thought of as interfacing in nouns. To grossly oversimplify, let's pretend that I'm updating my location in the application http://www.foobar.com/.

In RPC I would do something like

POST HTTP request http://www.foobar.com/api/update_location?lat=1234&long=5678

In the context of a REST API, I would do something like

PUT or POST HTTP request http://www.foobar.com/location?lat=1234&long=5678

A subtle distinction to be sure, but let's look at the difference. In the RPC version, we see the verb in the URI. There is a separate URI for every possible action we might want to take with regards to a location (update_location, get_location, create_location, delete_location). In a REST API, the resources of the program are addressed directly (that is, we send the request to the same URI that we would use to display our location in a web browser), and the "verb" that we want to apply to the resource is embedded in the HTTP request. REST is a very HTTP-oriented design approach, but it is an approach that makes sense because HTTP is a protocol for handling resources. The Wikipedia page has more on this, and may very well contradict me, as I am by no means an API expert!

To get to the point, what would an ESME REST API look like? Let's first look at the current ESME API, which is described as "REST", but which I think we can safely conclude is actually RPC.

http://code.google.com/p/esmeproject/wiki/REST_API_Documantation

Note how each API command is a verb. This is the hallmark of an RPC API. (ESME is part of a tradition here. The Twitter "REST" API is also primarily written in an RPC style, where the verb is part of the path of URI and is not assumed based on the HTTP method. The Twitter API does assume that a GET HTTP request maps to a read except when they have also have a specific "show" verb, but now I'm just nit picking.)

There is nothing wrong with this, and RPC is actually more in line with enterprise API design standards than a REST API, but I'd like to get at what a real ESME REST api would look like. I provide here the current ESME API method along with the REST equivalent that comes to mind.

I'm listing arguments here as URL-encoded, but they could easily be form-encoded, XML, JSON or all of the above. On the REST side, where a portion of the URL is in all caps, this would be substituted by the unique ID of that particular resource. This is, of course, not a well-thought-out proposal, but rather a suggestion to illustrate what a more RESTful API might look like.

One point to note is that most HTTP clients do not currently support
the "PUT" or "DELETE" methods, so these have to be simulated
through POST methods with an extra parameter. I think that because of the close mapping to resource verbs, is worth using these methods in
the specification and defining the simulation method for the entire API
separately.

The above is based on a rough object hierarchy as follows:

• ESME API instance (api/)
  • Sessions (api/sessions)
  • Users (api/users)
    • Messages posted by a user (api/users/USERID/messages)
    • Users followed by a user (api/users/USERID/followees)
    • Users following a user (api/users/USERID/followers)
    • Trackers belonging to a user (api/users/USERID/tracks)
    • Actions belonging to a user (api/users/USERID/actions)
  • Messages (api/messages)
  • Tags (api/tags)
  • Conversations (api/conversations)

Each of these bullets represents a set of objects. The resource representing an individual object lives at api/objects/OBJECTID. For example, api/sessions/SESSIONID. As much as is reasonable, one would expect to be able to GET (read), POST (create), PUT (update/amend), or DELETE (delete) any individual member of each of these object sets. Going through each of these objects to ask what it would mean to create, read, update, or delete that object may reveal holes in the existing API, some of which I have filled in above.