Skip to main content

PubSubHubBub and ColdFusion

I came across a publisher\subscriber protocol called PubSubHubBub. It is a server-to-server web-hook-based pubsub (publish/subscribe) protocol - an extension to Atom and RSS. Here the parties (servers) get an instant notification when a feed URL, that  they are interested in is updated.

Traditionally a subscriber would subscribe to a feed and poll for it at regular intervals, to see if there is an updated feed available. In this protocol rather than polling for a feed, the content is pushed out from the publisher. The theory here is that the subscriber can subscribe to a feed via a 'Hub', which then would inform the subscribers when the feed is updated.

How PubSubHubBub works?
  • A Publisher instead of sending an update to every subscriber, it includes a in its feed URL and sends an update to the Hub.
  • A Subscriber sends a subscription request to the Hub with the feed URL that it is interested in. The request also contains a callback URL to which the Hub should send an update.
  • To verify the subscription request, the Hub sends a GET request to subscribers' callback URL. The
    Subscriber then verifies itself by responding to the request.
  • When the Publisher posts new content, it notifies the Hub of the updates by sending a ping notification (POST request).
  • Hub on receiving a notification from the Publisher, fetches the new content and then POSTs an update to the Subscribers callback URL.
If the feed has multiple subscribers, then the Hub would send an update to each of these subscribers. The Subscription flow posted on the PubSubHubBub site is here:



Where is the Hub?

Anybody can run a hub, it is not owned by any company. The protocol is decentralized and free. A couple of implementations include pubsubhubbub.appspot.com and superfeedr.com

Publishing

Whenever a publisher adds new content (a new blog post), the subscribers are to be notified of the updated feed. A notification (POST request) is sent to the Hub by the Publisher. The Content-Type header must be set to application/x-www-form-urlencoded and the request should contain the parameters 'hub.mode' and 'hub.url' in the request body. The parameter hub.mode is set to 'publish' and the hub.url is set to topic URL that is updated.

The Hub then accepts the POST request and fetches the new content by sending a GET request to the topic URL. After fetching the content the Hub determines whether the feed has changed. The Hub would then send this information about the changes to each subscriber.

Subscribing

The Subscriber initiates its subscription by sending a POST request to the Hub URL. The Content-Type header must be set to application/x-www-form-urlencoded and it should contain the following parameters in the request body:
  • hub.mode - set to 'subscribe'
  • hub.callback - subscribers callback URL where the notifications should be delivered by the Hub.
  • hub.topic - the topic URL to which the subscriber wishes to subscribe
  • hub.verify - it can be either sync/async. 
    • sync when the verification request must occur before the subscription request's HTTP response is returned.
    • async when the verification request may occur at a later point after the subscription request has returned.

Verifying the subscription:

The Hub on receiving a subscription request verifies the same by sending a GET request to the subscriber's callback URL. The request contains the following query strings:
  • hub.mode - subscribe or unsubscribe depending on what was set in the original request.
  • hub.topic - the topic URL given in the subscription request.
  • hub.challenge - a random string.  
The subscriber's callback URL should respond to this request by echoing the hub.challenge string in response.

Code:

Registering the Publisher with the Hub:

The Publisher and Subscriber can be registered at pubsubhubbub.appspot.com and it is a one time activity. However, one can do the same programmatically:

<cfset machineIP = CreateObject("java", "java.net.InetAddress").getLocalHost().getHostAddress()> <cfhttp method="POST" url="http://pubsubhubbub.appspot.com" result="pub_result> <cfhttpparam type="header" name="Content-Type" value="application/x-www-form-urlencoded"> <cfhttpparam type="url" name="hub.mode" value="publish> <cfhttpparam type="url" name="hub.url" value="http://#machineIP#/ZeusTutorial/pubsubhubbub/example/Publisher.cfc?method=getFeed"> </cfhttp> <cfif #pub_result.Responseheader.Status_Code# eq 204> Publisher registered successfully <cfelse> Publisher failed to register returned #pub_result.Responseheader.Status_Code# </cfif>

The Hub responds with a '204 No content' if the publishers feed is successfully registered.

Publishing content:

A form accepting the post title and description:
publisher_form.cfm:
<cfparam name="FORM.feedTitle" default="" > <cfparam name="FORM.feedDescription" default="" > <cfif FORM.feedTitle NEQ "" AND FORM.feedDescription NEQ ""> <cfset publisherObject = createObject("component","Publisher") > <cfset publisherObject.publish_update(FORM.feedTitle,FORM.feedDescription) > </cfif> <form action="publisher_form.cfm" method="post"> Add a new post here:
Title: <input name="feedTitle" type="text" style="width:300px">
Description:
<textarea name="feedDescription" rows="5" cols="50" ></textarea>
<input type="submit" value="Submit"> </form>

A ping notification is sent to the Hub, to notify of the new content. The function publish_update does that:
<cffunction name="publish_update" access="remote" returntype="String"> <cfargument name="feedTitle" required="true"/> <cfargument name="feedDescription" required="true"/> <cfscript> if(!isDefined("application.feedQuery")) { application.feedQuery = queryNew("title,description,pubdate"); } queryAddRow(application.feedQuery, 1); querySetCell(application.feedQuery, "title", '#arguments.feedTitle#'); querySetCell(application.feedQuery, "description", '#arguments.feedDescription#'); querySetCell(application.feedQuery, "pubdate", '#DATEFORMAT(now(), "mm/dd/yyyy")#'); </cfscript> <cfhttp method="POST" url="http://pubsubhubbub.appspot.com" result="new_content"> <cfhttpparam type="header" name="Content-Type" value="application/x-www-form-urlencoded"> <cfhttpparam type="url" name="hub.mode" value="publish"> <cfhttpparam type="url" name="hub.url" value="http://#machineIP#/ZeusTutorial/pubsubhubbub/example/Publisher.cfc?method=getFeed"> </cfhttp>

The Hub then fetches the feed from the mentioned URL (hub.url). Here it is the getFeed method in Publisher.cfc:

<cffunction name="getFeed" access="remote"> <cfscript> /* Feed metadata structure */ feedMetaData = structNew(); feedMetaData.author = arrayNew(1); //arrayAppend(feedMetaData.author,{name:'Sagar Ganatra',email:'sagar@sagarganatra.com'}); feedMetaData.encoding = "UTF-8"; feedMetaData.link = arrayNew(1); //arrayAppend(feedMetaData.link,{href:'http://pubsubhubbub.appspot.com',rel:'rel'}); feedMetaData.title = structNew(); feedMetaData.title.type = "text"; feedMetaData.title.value = "Sagar's Kitchen"; feedMetaData.updated = "#now()#"; </cfscript> <cfset colMap = {publishedDate="pubdate", title="title", content="description"}> <cffeed action="create" query="#application.feedQuery#" properties="#feedMetaData#" columnmap="#colMap#" xmlvar="xmlResult"> <cfcontent type="text/xml" reset="true"> <cfoutput>#xmlResult#</cfoutput> </cffunction>

Registering a Subscriber with the Hub:

The Subscriber can subscribe to the feed by specifying the topic URL in the request body:

<cfset machineIP = CreateObject("java", "java.net.InetAddress").getLocalHost().getHostAddress()> <cfhttp method="POST" url="http://pubsubhubbub.appspot.com" result="sub_result"> <cfhttpparam type="header" name="Content-Type" value="application/x-www-form-urlencoded"> <cfhttpparam type="url" name="hub.mode" value="subscribe"> <cfhttpparam type="url" name="hub.verify" value="sync"> <cfhttpparam type="url" name="hub.topic" value="http://#machineIP#/ZeusTutorial/pubsubhubbub/example/Publisher.cfc?method=getFeed"> <cfhttpparam type="url" name="hub.callback" value="http://#machineIP#/ZeusTutorial/pubsubhubbub/example/subscriber.cfc?method=receiveFeed> </cfhttp> <cfif #sub_result.Responseheader.Status_Code# eq 204> Subscriber registered successfully <cfelse> Subscriber failed to register. Returned <cfoutput>#sub_result.Responseheader.Status_Code#</cfoutput> </cfif>

Subscribing

As mentioned earlier, Hub checks the Subscribers' intent by sending a GET request to the callback URL. The Subscriber is then required to respond, by echoing back the 'hub.challenge' parameter present in the URL:
<cffunction name="receiveFeed" access="remote" returnformat="plain" returntype="string" output="false"> <cfheader name="Content-Type" value="text/plain"> <cfif structKeyExists(URL, "hub.challenge")> <cflog text="#url['hub.challenge']#"> <cfreturn #url['hub.challenge']> <cfelse> <cfset feeddata="getHTTPRequestData().content"> <cfset feeddata="feeddata" feedsource.xml="feedsource.xml" filewrite="filewrite" ram:="ram:" trim="trim"> <cffeed action="read" query="feedReceived" source="ram:///feedSource.xml"> <cfdump output="console" var="#feedReceived#"> <cfheader statuscode="200"> <cfreturn> </cfif> </cffunction>
As seen in the above code the feed is read in else part. One can store the read content in a database for later retrieval or send the same over a websocket to the clients so that it can be read live.

Who is using PubSubHubBub?

Google products - Google Alerts, FeedBurner, Blogger and Google Reader are using PubSubHubBub.

Reference:

Project Home - http://code.google.com/p/pubsubhubbub/
Protocol Basics - http://pubsubhubbub.googlecode.com/svn/trunk/pubsubhubbub-core-0.3.html

Comments

  1. Thanks for this post. It's been helpful to me in learning how to implement the PubSubHubbub protocol in ColdFusion.

    FYI: I think there is an error in line 11 of your getFeed() method. The link to the hub server should have rel="hub" but your code uses rel="rel".

    ReplyDelete
  2. Thanks Richard,

    I've updated the blog post now.

    ReplyDelete
  3. Thanks a lot for the useful and detailed post, I am currently looking for a publisher for myself and consider sticking to this one. Seems to be a little complicated at first sight, but I am sure I will fiagure all the features out with such helpful tips.

     

    ReplyDelete

Post a Comment

Popular posts from this blog

Adding beforeRender and afterRender functions to a Backbone View

I was working on a Backbone application that updated the DOM when a response was received from the server. In a Backbone View, the initialize method would perform some operations and then call the render method to update the view. This worked fine, however there was scenario where in I wanted to perform some tasks before and after rendering the view. This can be considered as firing an event before and after the function had completed its execution. I found a very simple way to do this with Underscore's wrap method.

De-obfuscating javascript code in Chrome Developer Tools

I had blogged about JavaScript debugging with Chrome Developer Tools  some time back, wherein I have explained how these developer tools can help in debugging javascript code. Today Google Chrome 12 was released and my Chrome browser was updated to this version. As with every release, there have been some improvements made on performance, usability etc,. One feature that stood out for me is the ability to De-obfuscate the javascript code. What is Minification? Minification is the process of removing unnecessary characters such as white spaces, comments, new lines from the source code. These otherwise would be added to make the code more readable. Minifying the source code helps in reducing the file size and thereby reducing the time taken to download the file. This is the reason why most of the popular javascript libraries such as jQuery are minified. A minified jQuery file is of 31 KB in size where as an uncompressed one is about 229 KB. Unfortunately, debugging minified javascript f

On GraphQL and building an application using React Apollo

When I visualize building an application, I would think of using React and Redux on the front-end which talks to a set of RESTful services built with Node and Hapi (or Express). However, over a period of time, I've realized that this approach does not scale well when you add new features to the front-end. For example, consider a page that displays user information along with courses that a user has enrolled in. At a later point, you decide to add a section that displays popular book titles that one can view and purchase. If every entity is considered as a microservice then to get data from three different microservices would require three http  requests to be sent by the front-end app. The performance of the app would degrade with the increase in the number of http requests. I read about GraphQL and knew that it is an ideal way of building an app and I need not look forward to anything else. The GraphQL layer can be viewed as a facade which sits on top of your RESTful services o