Oct 10, 2011

PubSubHubBub and ColdFusion

I came across a publisher\subscriber protocol called PubSubHubBub. It is a server-to-server web-hook-based pubsub (publish/subscribe) protocol - an extension to Atom and RSS. Here the parties (servers) get an instant notification when a feed URL, that  they are interested in is updated.

Traditionally a subscriber would subscribe to a feed and poll for it at regular intervals, to see if there is an updated feed available. In this protocol rather than polling for a feed, the content is pushed out from the publisher. The theory here is that the subscriber can subscribe to a feed via a 'Hub', which then would inform the subscribers when the feed is updated.

How PubSubHubBub works?
  • A Publisher instead of sending an update to every subscriber, it includes a in its feed URL and sends an update to the Hub.
  • A Subscriber sends a subscription request to the Hub with the feed URL that it is interested in. The request also contains a callback URL to which the Hub should send an update.
  • To verify the subscription request, the Hub sends a GET request to subscribers' callback URL. The
    Subscriber then verifies itself by responding to the request.
  • When the Publisher posts new content, it notifies the Hub of the updates by sending a ping notification (POST request).
  • Hub on receiving a notification from the Publisher, fetches the new content and then POSTs an update to the Subscribers callback URL.
If the feed has multiple subscribers, then the Hub would send an update to each of these subscribers. The Subscription flow posted on the PubSubHubBub site is here:

Where is the Hub?

Anybody can run a hub, it is not owned by any company. The protocol is decentralized and free. A couple of implementations include pubsubhubbub.appspot.com and superfeedr.com


Whenever a publisher adds new content (a new blog post), the subscribers are to be notified of the updated feed. A notification (POST request) is sent to the Hub by the Publisher. The Content-Type header must be set to application/x-www-form-urlencoded and the request should contain the parameters 'hub.mode' and 'hub.url' in the request body. The parameter hub.mode is set to 'publish' and the hub.url is set to topic URL that is updated.

The Hub then accepts the POST request and fetches the new content by sending a GET request to the topic URL. After fetching the content the Hub determines whether the feed has changed. The Hub would then send this information about the changes to each subscriber.


The Subscriber initiates its subscription by sending a POST request to the Hub URL. The Content-Type header must be set to application/x-www-form-urlencoded and it should contain the following parameters in the request body:
  • hub.mode - set to 'subscribe'
  • hub.callback - subscribers callback URL where the notifications should be delivered by the Hub.
  • hub.topic - the topic URL to which the subscriber wishes to subscribe
  • hub.verify - it can be either sync/async. 
    • sync when the verification request must occur before the subscription request's HTTP response is returned.
    • async when the verification request may occur at a later point after the subscription request has returned.

Verifying the subscription:

The Hub on receiving a subscription request verifies the same by sending a GET request to the subscriber's callback URL. The request contains the following query strings:
  • hub.mode - subscribe or unsubscribe depending on what was set in the original request.
  • hub.topic - the topic URL given in the subscription request.
  • hub.challenge - a random string.  
The subscriber's callback URL should respond to this request by echoing the hub.challenge string in response.


Registering the Publisher with the Hub:

The Publisher and Subscriber can be registered at pubsubhubbub.appspot.com and it is a one time activity. However, one can do the same programmatically:

<cfset machineIP = CreateObject("java", "java.net.InetAddress").getLocalHost().getHostAddress()> <cfhttp method="POST" url="http://pubsubhubbub.appspot.com" result="pub_result> <cfhttpparam type="header" name="Content-Type" value="application/x-www-form-urlencoded"> <cfhttpparam type="url" name="hub.mode" value="publish> <cfhttpparam type="url" name="hub.url" value="http://#machineIP#/ZeusTutorial/pubsubhubbub/example/Publisher.cfc?method=getFeed"> </cfhttp> <cfif #pub_result.Responseheader.Status_Code# eq 204> Publisher registered successfully <cfelse> Publisher failed to register returned #pub_result.Responseheader.Status_Code# </cfif>

The Hub responds with a '204 No content' if the publishers feed is successfully registered.

Publishing content:

A form accepting the post title and description:
<cfparam name="FORM.feedTitle" default="" > <cfparam name="FORM.feedDescription" default="" > <cfif FORM.feedTitle NEQ "" AND FORM.feedDescription NEQ ""> <cfset publisherObject = createObject("component","Publisher") > <cfset publisherObject.publish_update(FORM.feedTitle,FORM.feedDescription) > </cfif> <form action="publisher_form.cfm" method="post"> Add a new post here:
Title: <input name="feedTitle" type="text" style="width:300px">
<textarea name="feedDescription" rows="5" cols="50" ></textarea>
<input type="submit" value="Submit"> </form>

A ping notification is sent to the Hub, to notify of the new content. The function publish_update does that:
<cffunction name="publish_update" access="remote" returntype="String"> <cfargument name="feedTitle" required="true"/> <cfargument name="feedDescription" required="true"/> <cfscript> if(!isDefined("application.feedQuery")) { application.feedQuery = queryNew("title,description,pubdate"); } queryAddRow(application.feedQuery, 1); querySetCell(application.feedQuery, "title", '#arguments.feedTitle#'); querySetCell(application.feedQuery, "description", '#arguments.feedDescription#'); querySetCell(application.feedQuery, "pubdate", '#DATEFORMAT(now(), "mm/dd/yyyy")#'); </cfscript> <cfhttp method="POST" url="http://pubsubhubbub.appspot.com" result="new_content"> <cfhttpparam type="header" name="Content-Type" value="application/x-www-form-urlencoded"> <cfhttpparam type="url" name="hub.mode" value="publish"> <cfhttpparam type="url" name="hub.url" value="http://#machineIP#/ZeusTutorial/pubsubhubbub/example/Publisher.cfc?method=getFeed"> </cfhttp>

The Hub then fetches the feed from the mentioned URL (hub.url). Here it is the getFeed method in Publisher.cfc:

<cffunction name="getFeed" access="remote"> <cfscript> /* Feed metadata structure */ feedMetaData = structNew(); feedMetaData.author = arrayNew(1); //arrayAppend(feedMetaData.author,{name:'Sagar Ganatra',email:'sagar@sagarganatra.com'}); feedMetaData.encoding = "UTF-8"; feedMetaData.link = arrayNew(1); //arrayAppend(feedMetaData.link,{href:'http://pubsubhubbub.appspot.com',rel:'rel'}); feedMetaData.title = structNew(); feedMetaData.title.type = "text"; feedMetaData.title.value = "Sagar's Kitchen"; feedMetaData.updated = "#now()#"; </cfscript> <cfset colMap = {publishedDate="pubdate", title="title", content="description"}> <cffeed action="create" query="#application.feedQuery#" properties="#feedMetaData#" columnmap="#colMap#" xmlvar="xmlResult"> <cfcontent type="text/xml" reset="true"> <cfoutput>#xmlResult#</cfoutput> </cffunction>

Registering a Subscriber with the Hub:

The Subscriber can subscribe to the feed by specifying the topic URL in the request body:

<cfset machineIP = CreateObject("java", "java.net.InetAddress").getLocalHost().getHostAddress()> <cfhttp method="POST" url="http://pubsubhubbub.appspot.com" result="sub_result"> <cfhttpparam type="header" name="Content-Type" value="application/x-www-form-urlencoded"> <cfhttpparam type="url" name="hub.mode" value="subscribe"> <cfhttpparam type="url" name="hub.verify" value="sync"> <cfhttpparam type="url" name="hub.topic" value="http://#machineIP#/ZeusTutorial/pubsubhubbub/example/Publisher.cfc?method=getFeed"> <cfhttpparam type="url" name="hub.callback" value="http://#machineIP#/ZeusTutorial/pubsubhubbub/example/subscriber.cfc?method=receiveFeed> </cfhttp> <cfif #sub_result.Responseheader.Status_Code# eq 204> Subscriber registered successfully <cfelse> Subscriber failed to register. Returned <cfoutput>#sub_result.Responseheader.Status_Code#</cfoutput> </cfif>


As mentioned earlier, Hub checks the Subscribers' intent by sending a GET request to the callback URL. The Subscriber is then required to respond, by echoing back the 'hub.challenge' parameter present in the URL:
<cffunction name="receiveFeed" access="remote" returnformat="plain" returntype="string" output="false"> <cfheader name="Content-Type" value="text/plain"> <cfif structKeyExists(URL, "hub.challenge")> <cflog text="#url['hub.challenge']#"> <cfreturn #url['hub.challenge']> <cfelse> <cfset feeddata="getHTTPRequestData().content"> <cfset feeddata="feeddata" feedsource.xml="feedsource.xml" filewrite="filewrite" ram:="ram:" trim="trim"> <cffeed action="read" query="feedReceived" source="ram:///feedSource.xml"> <cfdump output="console" var="#feedReceived#"> <cfheader statuscode="200"> <cfreturn> </cfif> </cffunction>
As seen in the above code the feed is read in else part. One can store the read content in a database for later retrieval or send the same over a websocket to the clients so that it can be read live.

Who is using PubSubHubBub?

Google products - Google Alerts, FeedBurner, Blogger and Google Reader are using PubSubHubBub.


Project Home - http://code.google.com/p/pubsubhubbub/
Protocol Basics - http://pubsubhubbub.googlecode.com/svn/trunk/pubsubhubbub-core-0.3.html

Oct 6, 2011

Steve Jobs - An insanely brilliant man passes away

Today when I woke up and switched on the Television to see what's making the news; I heard about Steve Jobs death. For the first time in my life, I'm feeling very sorry for a tech giant who passed away. Steve was an extraordinary human being. He kissed success not just once but many times. He changed the way we experience technology. His charisma, passion and more importantly desperation to do something great is matchless.

Oct 3, 2011

Building resposive Web applications with HTML5 Web Workers

One of the key aspects of building web applications that deliver great user experience is to build applications that are highly responsive. Browser vendors are trying to improve the speed of their JavaScript engines and are enabling the web applications to perform well. Since JavaScript was introduced, there has been no way to execute the code outside of the browser UI thread i.e. it has remained single threaded. The Web Workers API introduced in HTML5 enables web applications to run scripts in the background, independent of the UI thread.

The performance of a web application can be greatly improved by using Web Workers since each worker would spawn its own thread. These threads can be used to perform computationally intensive tasks in the background without affecting the performance of the entire application.