The Micro Web (The µ Web)

The Micro Web is a way of using Web technology (Javascript, DOM, URIs, HTTP) and good Web architecture, including event-driven servers, to deliver all the essential Web 2.0 functionality (Microformats, tags, POSH, OpenID, user presence, DHTML, Ajax, Comet, REST, Atom, APIs, pub-sub, JSON, widgets, mashups) with a uniform, simple model.

The Micro Web's uniform, simple model is highly:

• Productive - an end to Javascript coding and a big reduction in server-side coding;
• Mashable - an end to writing all that hand-crafted glue code to wire everything up;
• Scalable - an end to scalability worries and bolted-on memcacheds.

The simple recipe

1. A Web of linked-up JSON 'micro resources'
2. Two-way, interactive HTTP: micro resources animated by events
3. User micro resources animated by Javascript

Step 1: A Web of linked-up JSON 'micro resources'

Micro resources link up in a Web. See the links in the picture (coloured green) between micro resources (labelled 'µ'). Of course, links to and from the Mother Web are also welcome.

Here is a simple JSON micro resource:

micro(
  { "head": { 
      "uid":  "-521c3340-fe3328-",
      "content-type": "text/json",
      "perm": "*",
    },
    "body": { 
      "name": "Big List",
      "list": [ "-8de20e-993b-" ] 
    }
  }
)
  

UIDs and URLs within a micro resource are surrounded by hyphens to distinguish them. Inter-micro resource links within the micro resource are also indicated by surrounding hyphens. Ordinary Web links can also appear within these hyphens. Links between micro resources are transparent - if you JSONPath into a micro resource, it transparently jumps over inter-micro resource links, and then jumps over the "body" tag into the actual body content. Hence, you may split and combine micro resources, to keep them slim and well-balanced, without having to change how they're mined.

The "content-type" is that of the "body". Alternatives are "text/plain" and binary types such as "image/png", which must then be given a 'content-transfer-encoding' such as "base64". Being JSON, we assume UTF-8. Of course, binary resources will generally be linked to directly, rather than encoded in micro resources. The "perm" value is an Access Control List: indicating who or what can read this micro resource.

Here are some Microformat-based micro resources - an hAtom feed, an hAtom entry and an hCard:

{ "head": { 
    "uid":  "-92ac0311-353f20-",
    "content-type": "text/json",
    "perm": "*",
  },
  "body": { 
    "hfeed": [ "-9938b-de20e-", "-30ed9c-223a-" ] 
  }
}
  

{ "head": { 
    "uid":  "-9938b-de20e-",
    "content-type": "text/json",
    "perm": "*",
  },
  "body": { 
    "hentry": {
      "title":  "BT CIO retires",
      "published": "2007-08-22T12:34+00:00",
      "updated":  "2007-05-02T23:25:59Z",
      "author":  "-9938b-de20e-",
      "summary":  [ "Today, the", {"person": "CIO" }, "of", {"company": "BT"}, "retires"],
      "content": { "dom": { "p": [ [ "Today, the", {"person": "CIO" }, "of", {"company": "BT"}, "retires"],
                                     "He had a good innings" ],
                 }        },
      "category": [{"term": "BT",        "label": "British Telecom", "link": "-29e-9f9-" },
                   {"term": "companies", "label": "Companies",       "link": "-acc-301-" }],
      "bookmark": [ "Link to this entry", "-http://blogblog.org/#9938b-de20e-" ]
      "related": [ "Related entries", [ "-http://blogblog.org/#30ed9c-223a-" ] ]
    }
  }
}
  

{ "head": { 
    "uid":  "-9938b-de20e-",
    "content-type": "text/json",
    "perm": "*",
  },
  "body": { 
    "vcard": { 
      "fn": "-51-",
      "n": { "-id-":"-51-", 
        "given-name":      "Duncan",
        "additional-name": "Beaumont",
        "family-name":     "Cragg"
      },
      "url":   "http:\/\/duncan-cragg.org\/",
      "email": "xxx@xxx",
      "adr": {
        "country-name": "England"
      },
      "bday": "1964-01-01",
      "org": "-4ca-761-",
      "photo": [ "Photo of Duncan Cragg", "-http://duncan-cragg.org/duncan-cragg-fonecam-small.jpg-" ],
      "nickname": "duncan_b_cragg"
    }
  }
}
  

The cumbersome name 'micro resource' is shortened to just 'micro' in the following.

Step 2: Two-way, interactive HTTP: micro resources animated by events

In the Micro Web, things are much more balanced. Micro resources themselves do GETs and POSTs on each other, peer-to-peer style. The reason for this is that micros are seen as self-animated entities whose state depends on that of their peer micros. To discover that state, they GET it themselves. Or, better still, have it POSTed to them...

Normally, HTTP is one-shot: a GET ends with the response. In the Micro Web, a micro may not only GET, but optionally subscribe to another micro. A simple header flag in the GET allows a micro to add its own URI to a list of subscribers held by the target micro, to be notified when that target changes. Micros also store another list: their own outgoing subscriptions ("pubs" and "subs", resp., in the JSON micro's "head").

POST is given a single, specific purpose, now - to notify peer resources of a new state. Since micros are so small, this will probably be a POST of the whole new state itself, although it could be just an empty change notification, allowing the subscriber to fetch the state if and when needed (filling caches in the process).

Now, if POST-back is unavailable due to the asymmetry of the Web's infrastructure, a second alternative is available: the long poll. Here, after the subscription GET, a single monitoring GET is left open, waiting for changes on one or more micros. This long poll would be handled on behalf of a number of subscribers by their container, which would turn incoming updates into POSTs for the target subscribers. In some cases, simple, traditional short polling of the state of several micros may be used to detect changes.

Note that there is nothing inherently unscalable in either the POST notification or long poll GET approaches - as long as the server is asynchronous and event-driven (and uses epoll!). There is a slight overhead in the POST response (which can be empty), but its arrival can be used as an acknowledgement or its content have the target's latest state piggy-backed onto it.

These self-animated micro resources are 'micro programmed' - the behaviour of the whole system emerges as the sum of many tiny decisions made 'inside' each micro resource. Those decisions are made according to rules applied to the current state of a micro resource and its peers. The Observer Pattern could be used to implement some of this, or the 'Distributed Observer Pattern' as its Micro Web extension.

Actually, some micros are not animated by micro programming, but by a mapping from some external, changing state, like a legacy database table, the temperature outside, or the user...

Step 3: User micro resources animated by Javascript

Here you see two user micros (in light brown) and three other micros (in light blue). The top user can see the middle JSON micro, and then see both of its sub-micros.

In fact, the top user has decided that the top micro is interested in its own state, and is POSTing itself to it, perhaps as a result of a button push. The bottom user can just see the second JSON micro. Indeed, it's watching carefully, with a long-poll subscription delivered by Comet.

Clearly, these user micros are created and animated inside the browser by Javascript. Actually, by a single Javascript that effectively extends the browser to a "Micro Browser". The script 'rehydrates' or reanimates a user micro pulled in from a home server. This user micro may include data such as hCard information, OpenID and chat/Twitter-style content such as 'saying', 'viewing', or 'doing'.

After fetching the user, this script then pulls in a bunch of micros starting from the top one the user is looking at, and assembles them in the browser's DOM - a kind of 'Client-Side Include'. Micros may be pulled from one or many servers or domains (using XHR or script elements - modulo security checks).

The script doesn't just render the visible micros blindly - it can detect certain types of micro and decorate them accordingly. For example, an editable date triggers a calendar widget, a geo location embeds a Google map, etc. An hAtom feed micro may allow a certain user to open up a form to create a new blog entry, or an hAtom entry micro allow anyone to comment.

This way, a little declarative markup can trigger a lot of interactive functionality in the browser. Any JSON micro clearly has to be converted in its entirety to HTML or DOM elements. There will tend to be JSON forms of many XHTML POSH structures and Microformats. There will be support for 'GUI' building.

If the content isn't recognised, perhaps because it is more 'pure logical data' than interface or presentation, it is possible to make a best-guess rendering of it (with an editable form that allows strings to be entered in text boxes, etc..). Micro resource creators should, however, be encouraged to reuse existing content types and schemas wherever appropriate.

It's not just the server that can run inter-micro rules: the client-side Javascript can get into the transformation game, too. Notice how CSS, XSL and templates are all different ways of doing the same thing: decorating and transforming data. It is possible for some pure logical data to be transformed and decorated all the way into the DOM through the application of a series of in-browser rules, identical to those run server-side. And the state of the user micro itself can be added to the dependencies guiding any rule.

An action by the user on one of the visible micros then causes a POST back to that affected micro in the form of a representation of the User micro doing that action. In fact, all actions by the user are translated into expressions of what the user micro thinks the whole new state of the action target should be. Here is an example user micro attempting to set the state of a visible peer:

{ "head": {
    "uid":  "-2432-42f3ac-",
    "content-type": "text/json",
    "perm": "*",
    "subs": [ "-521c3340-fe3328-" ], 
    "pubs": [ "-521c3340-fe3328-" ] 
  },
  "body": {
    "user": { 
      "name": "Joe Bloggs",
      "openid": "...",
      "hcard": "...",
      "saying": "Hello...",
      "viewing": [ "-521c3340-fe3328-" ],
      "applying": [ "-521c3340-fe3328-", 
                { "name": "Big List", 
                  "list": [ "-8de20e-993b-", "-2432-42f3ac-" ]
                }
      ]
    }
  }
}
  

Hence, notice that we've had to repeat the name "Big List". Since micro resources are generally small, the overhead of drawing out the entire new state like this is outweighed by the benefit of not having to get into delta, patch or partial-setting notations, or large numbers of different action types (push-button, set-text, select-checkbox, add-to-list, delete, reorder, etc, etc).

It's now up to the list whether or not it sets itself to this suggested state.

Now, if anyone else is subscribed to this user, they can see them take this action, or say things, or change their point of view, etc. Micros keep a permissions list ("perm" above) and needn't publish their state to just anyone. The user could even just POST itself to a single target if it wants to.

Note that, since there's a path down to the bottom user from the top user (perhaps that lower JSON micro is a chat room), the top user can watch the bottom user. Of course, since the browsers can't see each other, it can only do this if the bottom user POSTs itself to a cache that the other user is watching.

Accessibility of the Micro Web, and visibility to non-Micro Web crawlers, is ensured through a server-side module that duplicates the functions of this single script, mapping to and from traditional full Web pages.

Cheaper, Wider, Faster

Cheaper

There's no Javascript (DHTML, Ajax, Comet) to write - the browser, with its 'Micro Web extension' script, is now smart enough to 'talk Web 2.0' (via Microformats and other standardised JSON structures). You don't have to set up functions on the server to be called from Javascript.

The Micro Web should be at least as productive as Rails. It shares some characteristics with the Rails approach - and with 'Naked Objects'. After writing some micros and some CSS, you may have 90% of the interactive site you want. A final snippet of reactive rules inside one or two micros, or an adaptor from a legacy system into micro form, and you're done. You don't write URI mappers, O-R mappers or templates (unless you really want or need to).

Wider

Your previously silo'd user now has a cross-site experience, able to jump seamlessly from one site to another with a transferrable identity and a calling-hCard. They can take their stuff with them from place to place. In some cases, it may start to look a bit like a 2D version of Second Life, or a 'shared WebOS'.

Faster

There are several, compounding reasons for the scalability of the Micro Web, beginning with the fact that there is only one Javascript shared amongst all Micro Web sites, saving on download times.

Next, you have the fine-grained tunability of micro resource cacheability, a REST advantage magnified by the smaller size and more compact JSON of micro resources. Personalised and highly changeable resources, often a headache in REST, are narrowed down to much smaller chunks.

Only changed data ever needs to be transferred in the Micro Web, either at the time it changes or just when it is needed. You can choose eager or lazy state transfer, push or pull, to optimise performance. Fine-grained, often lazy loading of micros smooths the traffic to exactly what is needed at the time it's needed. Alternatively, caches can be filled pro-actively by POST.

Additionally, micros are a natural for in-process memory cacheing and cross-process cache sharing. In fact, the Micro Web architecture has a lot in common with memcached, meaning you'll never need it.

Micro Web architecture also has a lot in common with the database-unconstrained or database-free approaches that are seeing much blog traffic of late. Micro resources may be stored in files. The basic level of 'crash-only' termination to prevent corruption is all that's needed. Other ACID features, conflict resolution functions and replication are pushed up to the micros themselves that care about it. This frees up the application from the locking, bottlenecks and single points of failure characteristic of the pervasive reach of an architecture underpinned by traditional databases.

Finally, another benefit of its REST pedigree, the Micro Web is inherently parallelisable and partitionable. There needs be no single, central, silo database application. Instead, the Micro Web means a distributed mesh of data, animated locally.

This partitioning and pushing of responsibility onto individual micro resources opens up applications to the benefits of parallel processing. Similarly to Erlang, each micro resource using the Distributed Observer Pattern can do its localised processing independently of its peers, allowing great optimisation of CPU allocation. Micro Web servers are best written thread-free and event-driven. Event-driven, asynchronous servers have excellent scalability characteristics. Indeed, systems encoded in local rules have excellent scalability characteristics.

Conclusion

If you'd like to help create the Micro Web, whether you're a Web standards person or an Ajaxian, I'd be delighted to hear from you. Here's a link to my blog, which has an email link on the left bar if you'd like to chat more about helping build the Micro Web:

What Not How - Duncan Cragg on Declarative Architectures (http://duncan-cragg.org/blog/)

The Micro Web main site (where this document is located):

The µ Web (http://the-u-web.org/)