Adaptation: How We Add New Concepts to BlueOrganizer

We love talking about internals of the BlueOrganizer on this blog. Recently we wrote about BlueOrganizer metadata format. In this post we will take this conversation to a whole new level and talk about how easy it is to extend the BlueOrganizer and add support for a new concept. Those of you who have the BlueOrganizer installed and looked inside MySites options know that we currently supports over 20 concepts including Books, Music Albums, Movies, Restaurants and Wine.

But you probably did not realize that these concepts are not hardwired into the BlueOrganizer. Instead, the BlueOrganizer dynamically picks up these concepts from adaptiveblue.com when it starts up. This makes BlueOrganizer different from the traditional desktop software, because it is actually adaptive.

To define a concept in a BlueOrganizer takes three things:

• Model - describes the attributes of the concept
• Shortcuts - describe the shortcuts for the new concept
• Adapters - describe how to recognize these new things

So all it takes to introduce a new concept, is to create XML files for a model, shortcuts and adapters and push them to adaptiveblue.com.
The users would get access to this new concept next time when restart Firefox. Cool, heh? Read on to see the details of how this all works.

Creating the New Model

The first step in adding a new concept is to create a model file. This file describes a few things about this new concept, most importantly its attributes. For example, a Book model defines title and author as attributes, the movie model defines title, director and actors, etc. Here we will show you an example of a simplified model file for a new concept that we are rolling out soon - stocks:

The actual model file is a bit more complex, but you get the whats going on - we are describing that the concept of stock has attributes such as symbol and title, and that symbol is the key that identifies it uniquely. Those of you familiar with relational databases can think of the model as simply definition of a table. So the model specifies what attributes of stock to look for in web pages and store. Thats all.

Creating the Shortcuts

Next we need to define shortcuts or searches that fit this new concept. What sort of things might we want to do for stocks? Here are some ideas:

• Checkout the Profile
• Get Quote
• Look at the Chart
• Review Historical Prices
• Add to your Portfolio
• Get news / Read Blogs

Once we figure out what sort of things make sense, we research and pick a few top sites for each of these shortcuts, so that you can have a choice. Okay, lets now look at a specific example. Here is how we define a shortcut that lets you lookup a profile of a stock on Yahoo!

You see how simple this is? All shortcut is a query base and then a set of parameters. Note that %symbol% is a variable that gets plugged with the value of this attribute depending what page you are on. If you are familiar with templates programming technique, thats exactly what shortcuts are - a parameterized queries or templates. Each of them is really easy to create. We assemble them into sets and thats how you get SmartMenu on each page and every BlueMark.

Creating the Page Adapters

In the perfect world each web site would be a web service. Alas, we are yet to get there. In the mean time, BlueOrganizer uses adapters to get information about a concept from a web page. For example, for the upcoming stock collection we created adapters that recognize stocks on Yahoo! and Google finance pages. Each adapter consist of a collection of little agents. Each agent is concerned with figuring out how to get one attribute (as defined in the model) from the page.

Technically speaking, the agents are similar to regular expressions. However, we split each of them into two parts - a matcher and a grabber. The matcher is concerned with identifying a place in the page where the piece of information is, while grabber is concerned with getting this piece of information out. Here is a simple example:

The site is Yahoo! finance and we are looking at Google stock. The objective is to get the symbol, which is GOOG. We outlined in the red box that symbol is actually part of the URL. Here is the agent that figures out how to get this attribute:

The code above says that resulting attribute is symbol and it relies on the link attribute being already filled in. The grabber is configured to look in between s= and & or end of the URL. The process is repeated for other required attributes as described in the model. All little agents are grouped together into a page adapter. When you navigate to another stock page on Yahoo! finance, the agents run and grab the corresponding information, leveraging the fact that all of these pages have the same structure.

Page adapters are certainly the most complex piece of rolling out the new concept. We will continue to do all this heavy lifting here for you. But really, anyone familiar with basic programming is able to quickly master creating page adapters (we had a high school intern doing this last summer). So in the long run we are planning to open up page adapters too and tap into the power of community. We are also looking into companies like Dapper to make the process accessible to less technical users, but this is a subject of another post.

Scaling the BlueOrganizer

Jeff and I spent maybe 8 hours total working on adding support for stocks to the BlueOrganizer. That is very little time by any measure. We have capability to add a lot of new concepts pretty quickly. So why do not we do that? We feel like it is important to keep the set of concepts small, focused on everyday things. Our objective here is not to have support for all natural language concepts - thats is neither feasible nor necessary. Instead, we think we can add a lot of value by delivering shortcuts and personalization for a few dozens of concepts.

But another important point is BlueOrganizer is extensible. As we get requests from users to add support for new concept, we can do it quickly. We think that this is what adaptive software should be like for now. In the absence of ways to truly teach computers to learn like we do, we can instead create architectures that are extensible and adaptable to the new situations. It does not matter that we programmed BlueOrganizer to “understand” stocks. Your user experience is that software was able to handle brand new concept. And thats really all that matters.

Preview of the Stock Collection

As I mentioned above, the Stock collection is complete and now is going through internal testing. Look for it sometime next week inside your BlueOrganizer. In the mean time, here is a preview:

The SmartLinks for Stock should make a debut on Howard’s blog soon. Since he came up with the idea and persuaded us to do it sooner rather than later, we let him introduce them.