esjewett.com

I've been complaining for months that there is apparently no way to get a Yahoo! Pipes -> Tarpipe connection. What I'm really looking to do is drive a Tarpipe workflow using an RSS feed, so when a new item shows up in the RSS or Atom feed it will kick off a Tarpipe workflow for that item.

Unfortunately Tarpipe doesn't natively support feeds to kick off a workflow. It allows for the use of email or a specifically formatted request to one of its two APIs. Yahoo! Pipes meanwhile can consume RSS feeds second to none, but it is limited in the after-effects that it can trigger. Pipes does provide a web service operator that can send a request to an arbitrary URL, but the person building the pipe has very little control over the format of the request, so this is not a satisfactory way to interact with arbitrary APIs, and it doesn't work with Tarpipe's API.

For a couple of months I've been wondering if the relatively new Yahoo! Query Language (YQL) might be the missing link. In perusing the YQL documentation yesterday I noticed that YQL now supports INSERT, UPDATE, and DELETE operations in addition to its SELECT operation. Given the built in ability to make state-changing requests to APIs specified in YQL's Open Data Table (ODT) format, I decided to give it a shot.

I'll be darned, but it works.

I threw up an Open Data Table definition for the Tarpipe 1.0 API on Github, based on the provided YQL examples and the very understandable YQL documentation. Open Data Table definitions are XML files that specify the possible operations on a "table" as well as optional Javascript to be executed when the operation is called.

The complete Open Data Table for the Tarpipe 1.0 API (with the exception of the "image" field) is now available on my Github fork of spullara's fork of the yql-tables repository. Whew. Gotta love Github for creating a simple interface to a tangle of forks.

The stable raw XML file is here.

This definition specifies an ODT that responds to a query of the format

INSERT into tarpipe.process_1_0 (workflow_key,title,body) VALUES ("workflow_key_here","This is my title","This is the body")


Given a valid Tarpipe workflow key, which is found in the definition of a Tarpipe workflow based on a REST receptor, this call will kick off the workflow with the provided title and body parameters.

Since this ODT is not currently included in the standard environment, we also have to specify the ODT definition to use, so the complete query we need to send to YQL looks like

USE "http://www.esjewett.com/files/tarpipe.process_1_0.xml"; INSERT into tarpipe.process_1_0 (workflow_key,title,body) VALUES ("workflow_key_here","This is my title","This is the body")


And that's it. I (or rather Github) have already provided the hosted ODT definition, so all you need to do is run the YQL query above in order to kick off a Tarpipe workflow. The query can be run in any of the manners allowed by the YQL system, including the console available here. Simply replace the text 'workflow_key_here' (keep the double-quotes) with your Tarpipe workflow key and you should be good to go.

Next up, I'm hoping we'll be able to go through how to use this techinque to wire Yahoo! Pipes directly into Tarpipe workflows. I'm not quite there yet as it's tough to get Pipes to ignore old entries, but I think we can figure something out...

UPDATE (7 Aug, 2009) - Due to an obscure incompatibility between Ruby on Rails versions higher than 2.1 and the Yahoo! infrastructure, the github.com versions of the open data table XML files no longer work. I am now hosting the XML file necessary for this blog at http://www.esjewett.com/files/tarpipe.process_1_0.xml. I've changed the code and links above to point to this file.