esjewett.com

This is a quick how-to, showing how to route your Twitter feed through Yahoo! Pipes to filter and cleanse it, then consume it using ESME actions.

I've created a parametrized Yahoo! Pipe here where you can input your Twitter username and get back a feed of all mentions of "ESME", "esme", or "Esme" from your Twitter timeline.

(Pipe address: http://pipes.yahoo.com/pipes/pipe.info?_id=fJZ29qrr3RGjLPJI6icw5g)

Once the "Run Pipe" button is pressed, the filtered time-line displays. I've also removed the username from the beginning of the Tweets, for better display in ESME.

Feel free to use this pipe with your username, or "Clone" it and modify it as you like. The structure of the Pipe as it is now is available as an image here.

Once the pipe has been run, it is possible to retrieve the output as an RSS feed. Copy the link for the RSS feed for this pipe, as we will need to consume it using an ESME action.

Now navigate to your ESME server. We need to use a relatively recent version of ESME, specifically a version including the recently merged actions branch. I'm using http://esmecloudserverapache.dickhirsch.staxapps.net/ in this example.

Log in to the ESME server and create a new action. Fill in the action name as desired. For the test, use the "every N mins" test. I'm chosen 5 minutes as my interval, which means that the pipe feed will be queried every 5 mins for new entries.

For the Action, use the rss: action type, followed directly (no space) by the URL of the Pipes RSS feed that we copied earlier. Your action form should look like this.

Click the "Add" button and your recent Tweets containing the word "ESME" will start showing up on the ESME server. One neat aspect of this is that any hashtags (#esme for example) in your Twitter updates will be converted to ESME tags automatically.

This only works for public Twitter updates, as Yahoo! Pipes doesn't support authentication for feeds. You could query your private Twitter feed directly from the ESME server, embedding your Twitter username and password in the URL, but I strongly recommend against this as your username and password will be stored in the ESME server database, which you do not control.

The Enterprise Social Messaging Experiment (ESME) project grew out of a collaboration in the SAP world and is now a project in the Apache Incubator. The project itself is an interesting and inspiring demonstration of the organizing power of so-called web 2.0 tools, which allowed the project to go from an idea to an Incubator project in about half a year through the efforts of a wide-spread group of individuals, many of whom have never met. I've been on the sidelines of the project, looking on, but I've been fascinated by some conversations that have taken place around the API.

The ESME API is often described as a "REST" API or a "RESTful" API. "REST" refers to the design principle of Representational State Transfer. Wikipedia has a good overview. REST is often seen as an alternative to RPC APIs, or "Remote Procedure Call" APIs. At root, the difference as described by Wikipedia is that RPC is about telling an application to do something while REST is about changing the state of the resources of an application.

The upshot is that RPC can be thought of as interfacing in verbs, while REST can be thought of as interfacing in nouns. To grossly oversimplify, let's pretend that I'm updating my location in the application http://www.foobar.com/.

In RPC I would do something like

POST HTTP request http://www.foobar.com/api/update_location?lat=1234&long=5678

In the context of a REST API, I would do something like

PUT or POST HTTP request http://www.foobar.com/location?lat=1234&long=5678

A subtle distinction to be sure, but let's look at the difference. In the RPC version, we see the verb in the URI. There is a separate URI for every possible action we might want to take with regards to a location (update_location, get_location, create_location, delete_location). In a REST API, the resources of the program are addressed directly (that is, we send the request to the same URI that we would use to display our location in a web browser), and the "verb" that we want to apply to the resource is embedded in the HTTP request. REST is a very HTTP-oriented design approach, but it is an approach that makes sense because HTTP is a protocol for handling resources. The Wikipedia page has more on this, and may very well contradict me, as I am by no means an API expert!

To get to the point, what would an ESME REST API look like? Let's first look at the current ESME API, which is described as "REST", but which I think we can safely conclude is actually RPC.

http://code.google.com/p/esmeproject/wiki/REST_API_Documantation

Note how each API command is a verb. This is the hallmark of an RPC API. (ESME is part of a tradition here. The Twitter "REST" API is also primarily written in an RPC style, where the verb is part of the path of URI and is not assumed based on the HTTP method. The Twitter API does assume that a GET HTTP request maps to a read except when they have also have a specific "show" verb, but now I'm just nit picking.)

There is nothing wrong with this, and RPC is actually more in line with enterprise API design standards than a REST API, but I'd like to get at what a real ESME REST api would look like. I provide here the current ESME API method along with the REST equivalent that comes to mind.

I'm listing arguments here as URL-encoded, but they could easily be form-encoded, XML, JSON or all of the above. On the REST side, where a portion of the URL is in all caps, this would be substituted by the unique ID of that particular resource. This is, of course, not a well-thought-out proposal, but rather a suggestion to illustrate what a more RESTful API might look like.

One point to note is that most HTTP clients do not currently support
the "PUT" or "DELETE" methods, so these have to be simulated
through POST methods with an extra parameter. I think that because of the close mapping to resource verbs, is worth using these methods in
the specification and defining the simulation method for the entire API
separately.

The above is based on a rough object hierarchy as follows:

• ESME API instance (api/)
  • Sessions (api/sessions)
  • Users (api/users)
    • Messages posted by a user (api/users/USERID/messages)
    • Users followed by a user (api/users/USERID/followees)
    • Users following a user (api/users/USERID/followers)
    • Trackers belonging to a user (api/users/USERID/tracks)
    • Actions belonging to a user (api/users/USERID/actions)
  • Messages (api/messages)
  • Tags (api/tags)
  • Conversations (api/conversations)

Each of these bullets represents a set of objects. The resource representing an individual object lives at api/objects/OBJECTID. For example, api/sessions/SESSIONID. As much as is reasonable, one would expect to be able to GET (read), POST (create), PUT (update/amend), or DELETE (delete) any individual member of each of these object sets. Going through each of these objects to ask what it would mean to create, read, update, or delete that object may reveal holes in the existing API, some of which I have filled in above.

Well, it's come to this. Every blog must go through a little self-centered Twitter theorizing during its journey toward fame and immortality. And it always helps to start off with a little story, right? So the story:

I was engaged in this conversation with @twhirl, who I'm pretty sure is one guy posing as a computer program. Speaks German too, I think. @twhirl also makes an application called . . . wait for it . . . Twhirl. Twhirl is a Twitter client. The conversation was about a feature request. And all of that is neither here nor there.

What is here and there is that the conversation occurred in real time. We were both engaged in it over the course of about a half-hour. But it only consisted of maybe 500 words total. Further, each of us probably spent a grand total of 30 seconds on the actual conversation and neither of us spent any time specifically waiting for a response. Lastly, I started the conversation sitting at my computer, followed a response on my phone, responded while sitting at my computer, then finished it up via SMS sitting on a city bus. Twitter was what made this all work.

To suss that out a bit, Twitter is . . .

Multi-modal

It works well on various interfaces, including the vanilla web interface, standalone widgets, SMS, and Plaxo. This is partly due to functionality built by Twitter (the company) and partly due to the API enabling third-party development.

At least as important as the fact Twitter works on various interfaces is the fact that it allows seamless switching between interfaces while in mid-conversation. It may take a bit of setup to get it just how you want it, but when it starts working it's rather astounding.

Personally, my modes are the phone (with tracking of my screen name so that I can take conversations offline), Twhirl (on the PC), and Twitterific (on the Mac). I occasionally use the web, but not terribly often.

Semi-asynchronous

An asynchronous conversation is one in which an immediate response is not required, or even expected. A face-to-face conversation is about as synchronous as it gets. It's considered rude to ignore the person you're talking to for even a few seconds. Meanwhile, a conversation carried out via snail-mail is about as asynchronous as it gets.

Twitter enables a specific level of asynchronous conversation different than anything else I've ever seen. It also enables a range of synchronicity (I maintain that's a word even if my spellcheck doesn't) that I've also never seen. It can be nearly as fast as IM and then range all the way along the spectrum to a wait of days between responses (assuming the participants remember what they're talking about). It's pretty functional at all of these levels, which is an achievement.

But Twitter has also created a new sweet-spot at the 3-to-5-minutes-between-volleys level of conversation. When you're in a conversation on Twitter, it's a conversation in slow motion where the amount of attention required can range from zero to full-on engagement. It's up to you.

Which brings me to the point that Twitter is

Robust across all attention ranges

By this I mean that a positive Twitter experience doesn't require a certain amount of attention. By removing the expectation of having to "catch up", it removes the pressure to pay attention all the time. It's not like email where I know that if I stop reading for a day I'm going to be relatively swamped. Twitter just fades into the background when you're working on something more interesting because there are no consequences for ignoring it.

The result is that Twitter is not nearly as much of a time-sink as it seems it might be. It strikes me that Twitter may actually be elevating time that would otherwise have been spent staring at desks and cubicle walls more-so than it is cannibalizing time that would have been spent on productive engagement with other tasks. Just a thought.

On the other hand, Twitter is more than capable of supporting full attention. Just follow 100 prolific and interesting Twitters and try to keep up with the conversation and relevant links that fly past you at up to 20/minute. At peak times, it's nearly impossible.

The more I think about it, the more I think these three areas are key to Twitter's utility and the unique type of communities that can arise in it and through it.

So there you have it. The obligatory Twitter blog. I'll try to make sure that never happens again.

(By the by, I'm esjewett on Twitter, and everywhere else.)

Occasionally I'm impressed that the right people are on Twitter at the right time to participate in a conversation, argument, or quip of mutual interest. These little meetings of the mind turn out to be very much in the moment, held together by mutual attention. Afterwards, the individual messages float apart and exist on their own, interspersed with bits of other conversations and status updates, disconnected from the original context. There is, to my knowledge, no place where any given conversation on Twitter can be replayed.

One of those evanescent conversations took place on Twitter today, but I've gone to the trouble to recreate it again after the fact. Hopefully I captured enough relevant bits to convey the gist.

(Read from the bottom up, click for larger version of the image. I'm @esjewett on Twitter.)

I see the sentiments expressed by @mkrigsman pop up quite often, and normally I'm not bothered by it. But I'm impressed by Michael Krigsman's level-headed analysis at his blog, so I felt the need to jump in and make a nuisance of myself.

My concern, as expressed on Twitter, is that we often (especially on the internet) attribute business decisions to some sort of ill-intent. This is especially true for decisions made by Microsoft, and increasingly Apple. Sometimes we go so far as to brand a company evil. This is a bit much in most situations. It's a questionable characterization when companies turn over private information to unjust regimes, but it's a little silly when we're talking about the delivery of iPod firmware updates. (Of course, silliness happens all the time on Twitter, and I suspect @mkrigsman was purposefully being a bit sensationalist to stimulate conversation. Mission accomplished!)

Frankly, on close examination it appears that most untoward behavior on the part of large organizations is the result of ineptitude rather than ill-intent. Most companies just can't get everything (or even most things) right. Even those that excel in a particular area, like Apple in user experience, are bound to miss every once in a while.

As always, we should try to give a charitable interpretation of actions. But where that fails, Hanlon's razor applies:

Never attribute to malice that which can be adequately explained by stupidity.

I usually like where Michael ends up his analyses, and this case was no different. We do indeed live in a complicated world.

Laura Fitton's post on tools she'd like to see for Twitter got me thinking about a related problem in aligning attention and interest, so I'd like to add to her list. The problem: It's hard, though not impossible, to align the attention a given blog/tweet/IM/email demands to the interest we have in it.

As an example, take myself and three of interests that I occasionally mention on Twitter.

The set of people that are actually interested in all three of these things is pretty minimal, and that's just a sampling of three things I mention on a regular basis.

Twitter's approach to this issue is to minimize the attention that everything receives. That seems to work pretty well for Twitter. Other systems (I'm looking at you Email) tend to maximize the attention that every message receives. Some systems make an attempt to match attention to interest and I'd like to see more of that type of work either as built-in features or as add-ons facilitated by an API.

We're starting to see some movement in this area among collaboration and knowledge management vendors. There are also tentative steps from social sites like Ma.gnolia.

Let's also remember that this is personal. The relative importance of any given piece of information is different for you than it is for me. So this approach is especially important in our personal communications software, be that software a feed reader, an email client, or an service like Twitter.