esjewett.com

Well, it's come to this. Every blog must go through a little self-centered Twitter theorizing during its journey toward fame and immortality. And it always helps to start off with a little story, right? So the story:

I was engaged in this conversation with @twhirl, who I'm pretty sure is one guy posing as a computer program. Speaks German too, I think. @twhirl also makes an application called . . . wait for it . . . Twhirl. Twhirl is a Twitter client. The conversation was about a feature request. And all of that is neither here nor there.

What is here and there is that the conversation occurred in real time. We were both engaged in it over the course of about a half-hour. But it only consisted of maybe 500 words total. Further, each of us probably spent a grand total of 30 seconds on the actual conversation and neither of us spent any time specifically waiting for a response. Lastly, I started the conversation sitting at my computer, followed a response on my phone, responded while sitting at my computer, then finished it up via SMS sitting on a city bus. Twitter was what made this all work.

To suss that out a bit, Twitter is . . .

Multi-modal

It works well on various interfaces, including the vanilla web interface, standalone widgets, SMS, and Plaxo. This is partly due to functionality built by Twitter (the company) and partly due to the API enabling third-party development.

At least as important as the fact Twitter works on various interfaces is the fact that it allows seamless switching between interfaces while in mid-conversation. It may take a bit of setup to get it just how you want it, but when it starts working it's rather astounding.

Personally, my modes are the phone (with tracking of my screen name so that I can take conversations offline), Twhirl (on the PC), and Twitterific (on the Mac). I occasionally use the web, but not terribly often.

Semi-asynchronous

An asynchronous conversation is one in which an immediate response is not required, or even expected. A face-to-face conversation is about as synchronous as it gets. It's considered rude to ignore the person you're talking to for even a few seconds. Meanwhile, a conversation carried out via snail-mail is about as asynchronous as it gets.

Twitter enables a specific level of asynchronous conversation different than anything else I've ever seen. It also enables a range of synchronicity (I maintain that's a word even if my spellcheck doesn't) that I've also never seen. It can be nearly as fast as IM and then range all the way along the spectrum to a wait of days between responses (assuming the participants remember what they're talking about). It's pretty functional at all of these levels, which is an achievement.

But Twitter has also created a new sweet-spot at the 3-to-5-minutes-between-volleys level of conversation. When you're in a conversation on Twitter, it's a conversation in slow motion where the amount of attention required can range from zero to full-on engagement. It's up to you.

Which brings me to the point that Twitter is

Robust across all attention ranges

By this I mean that a positive Twitter experience doesn't require a certain amount of attention. By removing the expectation of having to "catch up", it removes the pressure to pay attention all the time. It's not like email where I know that if I stop reading for a day I'm going to be relatively swamped. Twitter just fades into the background when you're working on something more interesting because there are no consequences for ignoring it.

The result is that Twitter is not nearly as much of a time-sink as it seems it might be. It strikes me that Twitter may actually be elevating time that would otherwise have been spent staring at desks and cubicle walls more-so than it is cannibalizing time that would have been spent on productive engagement with other tasks. Just a thought.

On the other hand, Twitter is more than capable of supporting full attention. Just follow 100 prolific and interesting Twitters and try to keep up with the conversation and relevant links that fly past you at up to 20/minute. At peak times, it's nearly impossible.

The more I think about it, the more I think these three areas are key to Twitter's utility and the unique type of communities that can arise in it and through it.

So there you have it. The obligatory Twitter blog. I'll try to make sure that never happens again.

(By the by, I'm esjewett on Twitter, and everywhere else.)

Occasionally I'm impressed that the right people are on Twitter at the right time to participate in a conversation, argument, or quip of mutual interest. These little meetings of the mind turn out to be very much in the moment, held together by mutual attention. Afterwards, the individual messages float apart and exist on their own, interspersed with bits of other conversations and status updates, disconnected from the original context. There is, to my knowledge, no place where any given conversation on Twitter can be replayed.

One of those evanescent conversations took place on Twitter today, but I've gone to the trouble to recreate it again after the fact. Hopefully I captured enough relevant bits to convey the gist.

(Read from the bottom up, click for larger version of the image. I'm @esjewett on Twitter.)

I see the sentiments expressed by @mkrigsman pop up quite often, and normally I'm not bothered by it. But I'm impressed by Michael Krigsman's level-headed analysis at his blog, so I felt the need to jump in and make a nuisance of myself.

My concern, as expressed on Twitter, is that we often (especially on the internet) attribute business decisions to some sort of ill-intent. This is especially true for decisions made by Microsoft, and increasingly Apple. Sometimes we go so far as to brand a company evil. This is a bit much in most situations. It's a questionable characterization when companies turn over private information to unjust regimes, but it's a little silly when we're talking about the delivery of iPod firmware updates. (Of course, silliness happens all the time on Twitter, and I suspect @mkrigsman was purposefully being a bit sensationalist to stimulate conversation. Mission accomplished!)

Frankly, on close examination it appears that most untoward behavior on the part of large organizations is the result of ineptitude rather than ill-intent. Most companies just can't get everything (or even most things) right. Even those that excel in a particular area, like Apple in user experience, are bound to miss every once in a while.

As always, we should try to give a charitable interpretation of actions. But where that fails, Hanlon's razor applies:

Never attribute to malice that which can be adequately explained by stupidity.

I usually like where Michael ends up his analyses, and this case was no different. We do indeed live in a complicated world.

Laura Fitton's post on tools she'd like to see for Twitter got me thinking about a related problem in aligning attention and interest, so I'd like to add to her list. The problem: It's hard, though not impossible, to align the attention a given blog/tweet/IM/email demands to the interest we have in it.

As an example, take myself and three of interests that I occasionally mention on Twitter.

The set of people that are actually interested in all three of these things is pretty minimal, and that's just a sampling of three things I mention on a regular basis.

Twitter's approach to this issue is to minimize the attention that everything receives. That seems to work pretty well for Twitter. Other systems (I'm looking at you Email) tend to maximize the attention that every message receives. Some systems make an attempt to match attention to interest and I'd like to see more of that type of work either as built-in features or as add-ons facilitated by an API.

We're starting to see some movement in this area among collaboration and knowledge management vendors. There are also tentative steps from social sites like Ma.gnolia.

Let's also remember that this is personal. The relative importance of any given piece of information is different for you than it is for me. So this approach is especially important in our personal communications software, be that software a feed reader, an email client, or an service like Twitter.

This post originally appeared as a comment on Shel Israel's Global Neighborhoods post, "An Open Letter to the Twitter Guys". I've copied it here to make it available to me in the future, and I've made some edits to connect links and bring it more into line with a blog post than a comment. In the process, it's been removed from the context of its conversation, so click back through the above links and read the thread. It's a good blog post and there are some excellent comments.

----

The only problem with the speculation on the technical underpinnings of Twitter's scaling issues going on here is that we might be (probably are?) wrong. That said, allow me to engage in some speculation!

I think in this case Shel and his technical contacts are probably on the wrong track. This conversation has already occurred once, with the base accusation that Rails can't talk to multiple databases. Once aired, the problem was solved relatively quickly and easily via a code contribution from the community: http://drnicwilliams.com/2007/04/12/magic-multi-connections-a-facility-in-rails-to-talk-to-more-than-one-database-at-a-time/

So even at the beginning, the problem was not the Rails application server architecture, but an issue of database contention. David HH's response to the criticism may have seemed a bit defensive, but at root he was correct that the best way to engage the community is to air your issues in community forums rather than try to work through your problems silently and then accuse the product of a shortcoming out of frustration during an interview. It certainly seems that the whole thing could have been handled better.

DHH also seems to be correct that at the application server level, Rails scales easily (though expensively) by simply throwing hardware at the problem. However, Twitter wasn't dealing with an application server problem and therefore wasn't dealing with a specifically "Rails" problem.

Moving on to the current discussion, it seems the root issue is still database contention, possibly in conjunction with a client polling architecture that may be reaching the limit of scale. See http://www.highscalability.com/scaling-twitter-making-twitter-10000-percent-faster and http://www.readwriteweb.com/archives/xmpp_web.php for decent overviews.

Having used Rails, Ruby and PHP (though not having scaled anything), I can say that as far as I see there is nothing inherent in either that solves or significantly exacerbates either of these issues. On the database side, applications built using either language/framework will generate SQL statements that query the database. In pure PHP, you usually write the SQL yourself. Rails does a lot of the SQL work for you, but it is certainly possible to override that assist and do it yourself if you've developed a situation where further optimization is required.

In short, all factors seem to point to inherent architectural issues that Twitter is struggling with. There are always ways to approach these issues, but it's far more complex than switching from one language or framework to another.

Twitter is a big ol' international hodgepodge of communities, conversations, and observations to the tune of 700,000+ active users.  Being international, it also tends to be multi-lingual.  Herein lies a problem: I can't understand, say, Arabic.  Or French or Japanese for that matter.  In our brave new world of computerized , automated everything, there's no technical reason these conversations can't be translated on the fly, if only we had a little metadata.

Metadata, in this case, would be data about the language someone is tweeting in.  You see, from a consumer perspective the real problem of automating translation isn't the translation part.  Google and Yahoo! have that pretty well nailed on the level of 140 character tweets that often appear to have been run through a translation program anyway.  The problem is figuring out, automatically, the language of the post and the language it should be translated into.

Enter Twitter nanoformats.  Nanoformats are closely related to microformats.  So closely related that they even live on the Microformats wiki.  Anyway, nanoformats are little pieces of text you can insert into tweets and other things to provide some sort of metadata, like location, tags, or (surprise) language.  The language nanoformat consists of the string "lang:" followed by the iso 639-1 code for whatever language the Tweet is tweeted in.

In order to enable auto-translation, a Twitter user would have to do two things.  First, our user would insert a lang nanoformat in their own bio to indicate the language they normally tweet in.  This bio nanoformat will also be used to figure out the language our user would like to receive other tweets in, so choose wisely.  Second, our user would insert a lang nanoformat in tweets they make that are not in the language their bio indicates.

So, for example, I normally twitter in English, so my profile indicates that with the "lang:en" nanoformat.  But if I tweet something in Spanish, I would add the "lang:es" nanoformat to that tweet.  (Don't worry, I won't subject you to any Spanish tweets, notwithstanding the example below.)

Okay, here's how it works

Now that we've done all that groundwork, the magic can begin.

Warning, the following paragraphs contain forward-looking statements.  No client actually does this stuff yet, though it's conceivable that I am under-informed on that point.

When I fire up my Twitter client and log in, the client checks out my bio and sees that I want my posts in "en", which means "English" in ISO-ese.  We'll call this imaginary client Litter, but we could just as well call it Twhirl, Snitter, Tweetr, or Twitterific.

Now, whenever Litter sees a tweet, it checks the tweet for the lang nanoformat and if it finds one, and it isn't "lang:en" it makes a quick call to http://translate.google.com to translate the tweet, sans nanoformat.  If the client doesn't find a lang nanoformat in the tweet, it should check the bio of the person the tweet belongs to for a lang nanoformat and perform the same call.

So the tweet "Hablo español lang:es" would result in the following call in my Litter client:

http://translate.google.com/translate_t?text=Hablo español&langpair=es|en

Click that and see what comes out.

What happens next involves some screen scraping of dubious legality, but maybe some agreement can be reached.  Anyway, Litter then pulls the result of this query out of the HTML mishmash it receives back.  It provides that translated result as the content of the tweet.

A little discussion goes a long way

There are a lot of reasons this won't work.

First of all, there is a 70 request per hour limit on the Twitter API.  Litter would exhaust that limit in about 5 minutes if it were trying to translate every tweet in the stream I receive.  As a result, I suggest that the translation function be implemented with a manual trigger, at least initially.  The imaginary Litter allows a user to right-click on a tweet and choose "Translate" from the context menu.

It also appears Google's terms of use don't allow this sort of use of the Google Translate service.  Similarly Yahoo! doesn't appear to allow this use of Babelfish.  However, there are a lot of dubious uses of these sorts of tools, so maybe we can come to some sort of agreement.  In the meantime, a working proof on concept would be pretty neat.

Lastly, there should always be some way to easily get at the original tweet within the client, just in case something goes horribly wrong in the translation.  This is technically an opt-in system, since it won't work on a person's tweets if they don't use the nanoformat, but we should still be considerate.

Despite these hurdles I'm certainly looking forward to this type of functionality becoming common, and not just on Twitter.  I know others are as well.  What do you think?  How can we make this happen?