Discovering XQuery

Countdown to the MarkLogic User Conference

Just hours away from the first events of the User Conference and XQuery geeks all over are sharpening their MacBooks, testing out iPad integrations and, if you are like me, polishing up some XQuery code to show everyone that gotta have it feature.

Highlights from the show include great customer talks about XQuery powered applications from Royal Society of Chemistry, JetBlue, Yell, Bowker and more.  

And there is plenty of XQuery (and XSLT) geeking out with:

• Darin McBeath's (Elsevier) session on building all XQuery applications
• How XML and XQuery can do Semantic and Linked Data queries with MarkLogic's Jason Monberg and Michael Blakeley
• The 'super' session (is it a Bird etc) on the MarkLogic App server by Justin Makeig of MarkLogic and Mindie Sorenson and Ryan Semerau, LDS

There will also be a reprise of my favorite customer session ever, the McGraw-Hill Create's Nimble Elephant session.  This open discussion about creating and executing an innovative project wowed 'em at the Digital Publishing Summit in December and Shannon and crew are back again for the user conference.

Besides the sessions there are good times to be had, including a conference wide party at the California Academy of Sciences . . . and of course the Demo Jam tonight - sponsored by Avalon Consulting (thank you Avalon!)

 

This brings me to my XQuery I'm working on.  Demo Jam is an event that I started to let developers show what they've been working on.  The format is simple:  you have 5 minutes to show your demo.  At the end of the demos, each person's name and demo title is read and we take a decibel meter reading to select the winner (that would be the one with loudest crowd noise ).

To make things fair I've tried to keep the order picking random, using names in a hat or some quick XQuery hacking like this:

              for $pres in ("presenter1", "presenter2", "etc")
              order by xdmp:random()
              return
              $pres

But since this Demo Jam is for our customers and partners and has some real XML and XQuery gurus (see the full list here), I thought it would be fun to pick the order AND open it up in PowerPoint . . . all with XQuery.

First I loaded a blank Demo Jam contestant list into the MarkLogic Server and let it do the Office OpenXML conversion.

Then I run this XQuery to edit the slide, putting my new order into it:

    declare namespace a="http://schemas.openxmlformats.org/drawingml/2006/main";
    declare namespace p="http://schemas.openxmlformats.org/presentationml/2006/main";

    let $presenters := ("presenter1","presenter2", "etc")

    let $node := doc("/demojam/demo-list_pptx_parts/ppt/slides/slide1.xml")//p:txBody
    let $new-node :=  let $line-up := let $order := for $pres in $presenters
                  order by xdmp:random()
                  return
                  $pres

    for $presenter at $number in $order
    return

    (: Make the Bullet for each presenter :)

    <a:p><a:pPr marL="457200" indent="-457200">
    <a:lnSpc><a:spcPct val="170000"/></a:lnSpc>
    <a:buAutoNum type="arabicPeriod"/></a:pPr>
    <a:r><a:rPr lang="en-US" sz="2000" dirty="0" err="1" smtClean="0">
    <a:latin typeface="Lucida Sans"/><a:cs typeface="Lucida Sans"/>
    </a:rPr><a:t>{$presenter}</a:t>
    </a:r><a:endParaRPr lang="en-US" sz="2000" dirty="0" smtClean="0"><a:latin typeface="Lucida Sans"/><a:cs typeface="Lucida Sans"/>
    </a:endParaRPr>
    </a:p>

    return

    (: write out the header as well :)

    <p:txBody><a:bodyPr/><a:lstStyle/>
    <a:p><a:pPr><a:lnSpc>
    <a:spcPct val="170000"/></a:lnSpc>
    <a:buFont typeface="Arial" pitchFamily="-108" charset="0"/><a:buNone/></a:pPr>
    <a:r><a:rPr lang="en-US" dirty="0" smtClean="0">
    <a:latin typeface="Lucida Sans"/><a:cs typeface="Lucida Sans"/>
    </a:rPr>
    <a:t>2010 User Conference Demo Jam</a:t></a:r>
    </a:p><a:p><a:pPr>
    <a:lnSpc><a:spcPct val="170000"/>
    </a:lnSpc><a:buNone/>
    </a:pPr><a:r><a:rPr lang="en-US" sz="2000" dirty="0" smtClean="0">
    <a:latin typeface="Lucida Sans"/><a:cs typeface="Lucida Sans"/>
    </a:rPr>
    <a:t>Line Up</a:t></a:r></a:p>
    {$line-up}
    </p:txBody>

    return

    (: put all that back into slide1.xml :)

    xdmp:node-replace($node, $new-node)

Then I use the handy tools from the MarkLogic Toolkit for PowerPoint to put it all together and return open it up in Excel:

    xquery version "1.0-ml";
    import module namespace ppt=  "http://marklogic.com/openxml/powerpoint" at "/MarkLogic/openxml/presentation-ml-support.xqy";

    xdmp:set-response-content-type("application/powerpoint"),

    let $presentation-dir:="/demojam/demo-list_pptx_parts/"
    let $map := ppt:package-map($presentation-dir)
    return
    ppt:package-map-zip($map)

If this doesn't all work live tonight at Demo Jam, maybe I'll check out the Microsoft Office session Pete Aven, MarkLogic and Jeffery Brown, LDS are putting on for some pointers.

See you there (in just a couple of hours!),

Matt

Making Pizza Right Part 2: The XQuery Application

[This is part 2 of a series on the creation of a Facebook application to let fans of New Haven pizza order a virtual pie.  Part 1 is here.]

Now that I've created our XML data model for the Pizza places, I can move on to developing the XQuery code to power the application.   Creating applications using XQuery is something I do a lot at MarkLogic and seems to be getting some mainstream attention lately.   IBM has been developing resources such as this article on using XQuery to develop applications, this tutorial XQuery idioms and this guide to using XQuery to create dashboards.  An open source group called 28msec recently published this paper which takes the even better approach of describing an end-to-end XQuery development paradigm.

MarkLogic's own Norm Walsh has also been leading the way with an all XQuery AtomPub server and a recent project that was all XML + XQuery.  And the results of using XQuery from top to bottom can be seen in such great applications as the  MarkMail email archive and the Digital Library from the Princeton Theological Seminary Library.  The Princeton app opens up the archives of the world's 3rd largest seminary library to the public with rich search and actual images of the original books all with just the XML about the books and XQuery.

For our application, we'll be using XQuery to create an application to let New Haven pizza lovers create a pie from their favorite pizza place that is accurate in the toppings you can select and also only available when the actual places are open in New Haven (a big part of New Haven pizza is waiting for it).

Our XML data for the application looks like this:

<new-haven-pizza>
    <restaurant>
        <name>Sally's</name>
        <hours>
            <set>
                <day>Tuesday</day>
                ...

                <open>17:00:00</open>
                <close>22:30:00</close>
            </set>
        </hours>
        <apizza>
            <sizes>
                <size>Small</size>
                ...

            </sizes>
            <toppings>
                <topping>Mozzarella</topping>
                <topping>Anchovies</topping>

                <special>...</special>
        apizza>
    </restaurant>
</new-haven-pizza>

There are some variations that we explored in Part 1 and we'll account for these in our code.

I plan to have the application be a Facebook app that allows users to select a restaurant and then order a pie.  To do this we need a set of services to return values for each step in the process.  I'll collect these services in a single XQuery module and then call them from the main code of the Facebook interface (that I'll put together in Part 3).

The first service is a list of restaurants.   Since I want to be true to the restaurants hours, I need to use some of the extensive XQuery date functions to see if things are open as well as use the day of week name function from FunctX, the XQuery function library.

I'll start with this query that returns the list of restaurants:

    for $restaurant in /new-haven-pizza/restaurant/name return $restaurant

I need to add to this basic function to check the day and time and only show what's open.  I also need to have an error message and, since this is a service that will be feeding a web application directly, I'll go ahead and make the everything return the appropriate XHTML.  So <select> elements from forms and <p>s for error messages etc.

(: module declaration and namespace :)
module namespace nh="http://xquery.typepad.com/nh-apizza";

(: import the functx module with the  day of week name function :)
import module namespace functx="http://www.functx.com" at "functx.xqy";

(: get restaurant function :)
declare function nh:get-restaurants() {

(: get the current date and time :)
    let $current-daytime := fn:current-dateTime()

 (: use the day of week name function to see what day it is :)
    let $day := functx:day-of-week-name($current-daytime)
 (: get the time :)
    let $time := xs:time($current-daytime)
 (: use the day and time to get the restaurants that are open :)
    let $restaurants :=  /new-haven-pizza/restaurant[./hours/set/day eq $day]
                            (: [xs:time(./hours/set/open) < $time]
                            [xs:time(./hours/set/close) > $time] :)     
return
(: return the restaurants Or an error :)
    if ($restaurants) then
        <select name="restaurant">
            {for $restaurant in $restaurants
            return
(: lets use the fullname if its available for the label :)
                <option value="{fn:string($restaurant/name)}">{fn:string(if ($restaurant/full-name) then $restaurant/full-name else $restaurant/name)}</option>
        }</select>
    else
        <p>Nothing is open right now!  It is {$day} {$time} Eastern Time.</p>
};

The next functions pick up from here by listing the options for making a pie once you have selected a restaurant.  These are the sizes, the toppings (with the specials as toppings if the restaurant allows it) and the specials.

For each function we do a simple value test on the name element and then go get what we need:

declare function nh:get-sizes($restaurant) {
    <select name="size">
    {for $size in /new-haven-pizza/restaurant[./name eq $restaurant]/(apizza | tomato-pie)/sizes/size
    return
        <option value="{fn:string($size)}">{fn:string($size)}</option>
    }</select>
};

To get the full topping list and include the specials that could be used as toppings (an oddity of Modern) we need to do some additional checking on the <special> element and also make use of the flexible XPath model (element1 | element2) to return topping *and* special elements in the same statement:

declare function nh:get-toppings($restaurant) {
    for $topping in /new-haven-pizza/restaurant[./name eq $restaurant]/(apizza | tomato-pie)/toppings/(topping | special[../../combine-specials eq "yes"])
    return
      <input type="checkbox" name="topping" value="{fn:string($topping)}">{fn:string($topping)}</input>
};

And then the pretty simple list of specials:

declare function nh:get-specials($restaurant) {
    for $topping in /new-haven-pizza/restaurant[./name eq $restaurant]/(apizza | tomato-pie)/toppings/special
    return
        <input type="radio" name="special" value="{fn:string($topping)}">{fn:string($topping)}</input>
};

All of these functions provide the information needed to create the pie selection forms.   Once we gather that data back, I need a pie maker function to present the finished pie:

declare function nh:make-a-pie($size, $toppings, $special, $restaurant) {
    let $pie := if ($special) then
                    $special
                else
                    fn:string-join($toppings, ', ')
    return
        <p>Here is your {$size} {$pie} from {$restaurant}.  Enjoy and thanks for using the New Haven Pizza Maker!</p>
 };

And thats really it!  Its under 60 lines of code to make a complete application.  This simplicity has a lot to do with the fact that I'm never leaving the world of XML.  There is no impedance mismatch (that Dave Kellogg blogged about in this post) and XQuery has everything you need to execute a content application, all in one language whose output *is* XML.

This is why XQuery applications are a growing trend - people are discovering what Joris Petrus Maria Graaumans proved in his Doctoral Thesis three years ago - XQuery is just plain better!  My entry on his paper is  here and the whole thing is here.

In the next part of this series I'll put it all together, build the Facebook interface and launch the New Haven Pizza maker.

See you next time,

Matt

Making Pizza Right Part 1: An XML Model for a Pizza Place

I've had an idea for a while now to make a Facebook application that will let lovers of New Haven Pizza sample a pie (no slices here) when they are not able to get to their shrines of Pepe's, Sally's and Modern.

For those of you NOT familiar with New Haven Pizza, it, like all great things, inspires a great deal of loyalty, religious devotion and controversy.  The Facebook group 'I live for New Haven Pizza' has 6000+ members and is a 24/7 debate room full of arguments over which is the best, when to go to avoid the lines and rude dismissals of other inferior sources of tomato pie.

As an act of community service to the lovers of my hometown's pizza, I thought i would create an application that would let users create pies that faithfully represented the topping combinations from the three best spots while respecting their quirky hours (only Pepe's is open Mondays, only Modern has late hours and Sally's never opens before 5pm).

It would also be a good way to explore some of the themes I've been working on including modeling content in XML, using XQuery to build applications and delivering content to new  distribution channels.

Since there is a lot to cover and to help get this started, I've broken this up into a series:

• In this first segment I will look at the XML model behind the application.
• The next segment will cover using XQuery to power the application and how MarkLogic Server can be the single resource you need for to power an information product (even if its just about fake tomato pie).
• In the last segment I'll talk about delivery and how XQuery is perfect for integrating with the new web application platforms like Facebook.

Part 1: An XML model for a Pizza place

Using the right data model for an application is something that has been coming up a lot lately and in particular for applications that want to use XQuery but may not have XML.   XQuery has come a long way and I think that the arguments I made in this blog entry a couple years ago really ring true:  modeling data as XML can be much more flexible than relational and lets you use great tools like MarkLogic Server.  The proof is in great applications such as MarkMail or AuthorMapper all with only XML data and powered by MarkLogic.

For our example we'll take a Pizza place.  The first thing you will notice is that I didn't say 'lets take Pizza'.   This is because the application is about these places as much as it is about the food (see above re: the fanatical user base).  So I want to capture whatever details I can and use them in the app if possible . . . and I want to be able to add more later.

Instead of breaking up the information into little bits and making tables, I simply made one document with an XML schema that looks like this:

<new-haven-pizza>
    <restaurant>
        <name>Sally's</name>
        <location>Wooster Street</location>
        <payment>cash only</payment>
        <est>1938</est>
        <website>sallysapizza.net</website>
        <notes>Sally's was established in 1938 by Salvatore Consiglio. Renown for thin crust pizza ...</notes>
        <hours>
            <set>
                <day>Tuesday</day>
                <day>Wednesday</day>
                ...

                <open>17:00</open>
                <close>22:30</close>
            </set>
            </hours>
        <apizza>
            <sizes>
                <size>Small</size>
                <size>Medium</size>
                <size>Large</size>
            </sizes>
            <toppings>
                <topping>Mozzarella</topping>
                <topping>Anchovies</topping>
                <topping>Bacon</topping>
            </toppings>
            <notes>At Sally's mozzarella is considered a topping, so if you want it on your pie let us know!.                        </notes>
        </apizza>
    </restaurant>
</new-haven-pizza>

This schema lets me capture everything I need to run my application and also capture many of the differences between the places. 

For instance Modern and Sally's call it APizza, but at Pepe's your order a Tomato Pie.  So the entry for Pepe's has a <tomato-pie> element instead of the <apizza> one.  And even though Sally's still has just a photo of their menu, they are the only ones with some text useful to include.  So the others don't, for the moment, have notes.

The toppings and kinds of pie are also pretty different.  While Sally's just has toppings, Pepe's has a mix of toppings and specials.  This is the topping section for Pepe's:

            <tomato-pie>
                ...
                <toppings>

                <topping>Mozzarella</topping>
                <topping>Sausage</topping>
                <topping>Mushroom</topping>
                ...
                <special>Clam white or red sauce only</special>
                <special>Clam with mozzarella</special>
                <special>Chicken</special>
                ...

                </toppings>
            </tomato-pie>

Modern also has specials, but its a mix of things that you can add to regular pies and real specials like the famous 'Italian Bomb'.  The data is faithful to the Modern menu but has a <combine-specials> element to record what the waiter would explain if you asked.

The item ordering of the menus is also preserved.  They each have different, unique, non-alphabetic order.  Since this was easy to capture just by putting them in document order I thought it best to record it and stay on the good side of the fanatics out there.

Finally, the hours structure is modeled on the way the way restaurants describe hours - Monday's we're closed, Tue-Fri we are open until 10 etc etc. and is easy to extend with new sets of rules such as holidays and summer vacations (which I didn't put in but are indeed part of the picture).

Approaching even this simple example as a relational model gets really complex, really fast.  A rough estimate for a practical (not fully normalized) schema would be at least 5 or 6 main tables and 8 to 10 relationship tables.  This would take at least a couple of hours to model and then create in a database.

But to top it off (sorry), it would be pretty hard to model for the differences.  In fact, the only way to do it is to add more complexity to the table structure and even more relationships to note that one place has specials and another lets you mix specials and toppings etc etc.  And when you want to add more information to power new features of our application, you would go back to the drawing board, make some new tables and have to redo the whole thing again.

By contrast, the XML I am using is a simple document I made in 15 minutes and, when I load it into MarkLogic lets me do queries like this to see where I can get a Bacon APizza:

/new-haven-pizza/restaurant[./apizza/toppings/topping eq "Bacon"]/name
->
<name>Modern</name>
<name>Sally's</name>            

Or a Clam Tomato Pie:

/new-haven-pizza/restaurant[./tomato-pie/toppings/topping eq "Clam"]/name -> <name>Pepe's</name>

Having created and loaded my data, I'm now ready to start the next tasks of building an application that will let users query the XML to find out where they can get a pie and create a virtual pizza and then work on wrapping it up so it can be used within Facebook.

Right now I'm getting pretty hungry and sadly am nowhere near New Haven!

Thanks,

Matt

XQuery: The Real X in AJAX

Like the real Napster in the movie The Italian Job (the remake), XQuery might have a bit of a chip on its shoulder about the X in AJAX.

Sure, it stands for XML since the idea is that return an XML fragment to the browser to update content in a div, fill in form fields or even create drop down menu options on the fly.

But how do you create that XML?  Using static XML files works, but the whole idea is to dynamically respond to user actions and give them information without reloading the whole page.

And what's the best way to dynamically create XML?  XQuery of course!

To prove it, lets do a simple example to create a drop-down form field for Shakespeare's characters (using the Shakespeare XML we loaded in the tutorial).  This field will auto-complete using AJAX and XQuery and give the user the characters found in the XML that begin with the letters entered into the field.

To get a head start, we'll use the popular AJAX tool Scriptaculous that takes care of all of the hard javascript stuff and lets us just work on creating the backend to deliver the content.

We'll also make use of MarkLogic Server's app server built-ins and its ability to run an HTTP server to make a complete application that presents the HTML, including Scriptaculous.  To do this, we'll start with the /modules directory we created and accessed with WebDav in the tutorial (where the CQ application was placed).

Assuming you have that set up, here are the steps to get the client side set up:

1. create a js/ directory and install all of the .js scriptaculous libraries that came with the distribution (located here)
2. Make a lookup.css in /modules with the sample styles from this page
3. create lookup.xqy under the /modules directory with the following HTML in it:

xdmp:set-response-content-type("text/html; charset=utf-8")
(:  sets the mime type :)
,
<html>
    <head>
        <title>Shakespeare Lookup</title>
        (: reference the stylesheet :)

       

<link rel="stylesheet" type="text/css" href="lookup.css" media="screen"/>

         (: get the scriptaculous scripts loaded - the " " is to prevent them being optimized into an
        empty XML node like <script...  /> which some browsers don't like :)

        <script src="js/prototype.js">{" "}</script>
        <script src="js/scriptaculous.js">{" "}</script>

    </head>
<body>
    <h1>Shakespeare Character Lookup</h1>

    <div>
        <form>
        (: create the placeholder for the autocomplete field :)
        <input type="text" id="autocomplete" name="autocomplete_parameter"/>
        <input type="submit" value="select character"/>
        </form>
        <div id="autocomplete_choices" class="autocomplete"></div>

        (: run the scriptaculous autocomplete :)
        <script type="text/javascript">
            new Ajax.Autocompleter("autocomplete", "autocomplete_choices",                
            "request.xqy", {{}});
        </script>
    </div>
</body>
</html>

If you are like me and learned HTML and javascript way way back, it may take you a moment to realize that the <input> element named 'autocomplete' is NOT what actually shows up in the browser.  The script Ajax.Autocompleter replaces that element with the fully decked out, onClick enabled <input> element that does all of the auto-lookuping including making the calls to your request.xqy.  You can create all this yourself . . . but Scriptaculous does it for you, so go ahead and enjoy it!

This should all result in a simple web page with a text field on it that you can get to at http://localhost:8002/lookup.xqy (or wherever your MarkLogic Server is installed).  But it won't do anything until we create the backend XQuery.

In the Ajax.autocompleter code, we gave request.xqy as our source for the lookups.  We need to create this in the /modules directory and it's contents can be something as simple as this:

(: get the value of the field - sent to us as a POST from the Scriptaculous autocompleter :)

let $query-base := substring-after(xdmp:get-request-body(),"autocomplete_parameter=")

(: add an '*' to it to create a wildcard search :)

let $query := fn:concat($query-base, "*")

return
    <ul>{

        (: use the MarkLogic built-in cts:element-value-match()* to search all of the values in the PERSONA element in the loaded XML plays :)

        for $item in cts:element-value-match(xs:QName("PERSONA"),$query)
        return

        (: return the <li> elements scriptaculous expects for its list :)

        <li>{fn:string($item)}</li>
        }
    </ul>

Yup - thats all there is to it:  9 lines of code to query some XML and return XML.

When all this is hooked up and running, you should have a mini-application that looks something like this:

It's now up to you how you will use the power of XQuery to create dynamic content elements for AJAX.    Will you populate complex taxonomies and even bring back the content from leaf nodes?  Will you create search interfaces that give you the answers in the form?

How about an amazing interface for searching XML content?  Check out markmail.org, which is all XQuery and AJAX, for some inspiration.

So XQuery really must be the real X in AJAX, right?

Well it turns out there is a bit of room under the X in AJAX these days.  JSON is a popular alternative to XML (and XQuery's got that covered - check out this library Jason Hunter wrote to generate JSON from XQuery) and there are lots of ways to generate both XML and JSON.

But for those of us in the know, XQuery is the only way to go.  And with the growing number of XQuery powered content applications, no one can shut down the real X!

Matt

*NOTE: cts:element-value-match() is a MarkLogic built-in that requires an Element Range Index be configured for the element in question.  This is pretty straight forward: select your database under the Databases tab in the MarkLogic admin interface (also covered in the tutorial). Under Element Range Index, select Add.  For the scalar type select string, namespace can be blank and enter PERSONA for localname.  This will create an index for ordering that can also be used to perform the character lookup.

*ALSO NOTE:  there are plenty of other ways to get a list of PERSONA values based on user's input such as:

//PERSONA[cts:contains(., "p*")] (: still uses MarkLogic search built-ins :)
//PERSONA[fn:starts-with(fn:lower-case(string(.)), "p")] (: standard XQuery :)

But like scriptaculous, MarkLogic's search built-ins do the work for you (and also do it much more efficiently) so let's just enjoy using them too!

Santa's Really Big XQuery List

Let's say you are the head of an international syndicate headquartered up North that gets real busy this time of year getting a gift for every kid in the world (!!).

And let's say you really need to be efficient about it - Wired estimates it would take $27 billion just for the U.S. alone!

Maybe you could use some XQuery to move the process along and make that really big delivery list for all the kids?

If you had a gygnormous database of kids like this:

<kid>
    <name>Jane</name>
    <naughty/>
</kid>
<kid>
    <name>Josh</name>
    <nice/>
</kid>
<kid>
    <name>Michael</name>
    <nice/>
</kid>
<kid>
    <name>Lila</name>
    <nice/>
</kid>

And an equally huge list of toys that looks like this:

<toys>
    <toy>Jakks EyeClops Bionic Eye</toy>
    <toy>IlluStory Make Your Own Story Kit</toy>
    <toy>Blokus Strategy Board Game</toy>
    <toy>LeapFrog ClickStart My First Computer</toy>
    ...
</toys>

(For this example I've used Mark Logic's capabilities to scrape content from the web and grabbed Amazon's top gifts for kids - we all know Santa has his own sources).

You could use a bit of XQuery to bring them together and make sure we get one toy per kid:

(: start by looping through only the kids that get toys.  So limit it to the ones with <nice> elements and that DON'T yet have a <toy>
This method of selecting entries comes in handy for large content sets as we'll see below :)

for $kid in /kid[./nice][not(./toy)]

(: get a quick count of the toys for our random feature :)

let $toys := count(/toys/toy)
return

    (: get a toy based on a random number using MarkLogic's random function :)

    let $toy :=
        let $random := xdmp:random($toys)
        return
            (/toys/toy)[$random]

            return

                    (: add the toy node to the kid :)

                xdmp:node-insert-child($kid, $toy)

Run this, and all our kids now have toys:

<kid><name>Jane</name><naughty/></kid>
<kid><name>Josh</name><nice/><toy>Hasbro Playskool Step Start Walk 'n Ride</toy></kid>
<kid><name>Michael</name><nice/><toy>Scrabble</toy></kid>
<kid><name>Lila</name><nice/><toy>Fisher-Price Little Superstar Sing-Along Stage</toy></kid>

. . . except for naughty Jane of course.

But those of you with an eye for big things might be thinking, this is well and good for 4 kids . .. and maybe 400, but what about a couple billion!

Well that first test is the key.  It lets you grab a set of records to process without having to iterate through records and do commits after x number (like you would with a relational system).

Instead, you can have this code work on a small batch at a time but adding a predicate to limit the number returned by the query:

for $kid in /kid[./nice][not(./toy)][1 to 1000]

Then all you need to do is run it over and over - each run always only picking up the kids that need toys.  When everyone has a toy, it's done.

MarkLogic Server provides some ways of automating this with a function called xdmp:spawn that lets you put code onto a task server.  So to get through the billions of kids, you would take the above code (with the predicate for 1000), save it as a module and then run that module over and over until it there were no more kids who needed toys:

if (/kid[./nice][not(./toy)]) then
    xdmp:spawn("match-toy.xqy")
else "All Done - ready for Christmas Deliver!!"

Just like the big guy up North, XQuery keeps going until every kid has a toy.

Happy Holidays Everyone!

Matt

XQuery and Microsoft Office (2007) XML

Last week at the XML conference I gave a talk entitled First Encounters with Office Open XML. 

I learned a ton about this new format and had a lot of fun taking it apart and using XQuery to manipulate content and create new Office Open XML documents that Microsoft Word (2007) can open.

The basics are that the format is XML contained in a zip file.  So you can, with desktop zip tools, open up any Office 2007 file and work with the XML outside of the MS Office 2007 tools.

Combining this with a content server like MarkLogic Server, which has included built-ins to do the unzipping and zipping, you can load Office Open XML into an XML repository, query across multiple documents with XQuery and use XQuery to create new Office Open XML documents.

As I said in my talk, Office Open XML *is* XML and you can open it up and mess with it!

MarkLogic's Pete Aven is starting a series of posts called MarkLogic Server and Office XML.

The first post shows how to open up, query and repackage Office Open XML focusing on the Microsoft Word format.  It is here under Office Logic.

The second post explores the details around Excel and is called Excel-ing with XQuery.

My slides from the presentation are here:

| View | Upload your own

They are also posted to the XML conference site: http://2007.xmlconference.org/public/schedule/detail/362.

In my presentation I had a lot of fun going between the Office Open XML format XML and the tools inside Microsoft Word to view and manage content.  And, as Pete's posts show, it's pretty easy.

One thing I did was to generate Office Open XML format from Shakespeare content I used in the original tutorial.

To do this, I used the same transformation I used in the XQuery Transformers tutorial.  This starts with a recursive function and a typeswitch for each of the elements:

define function recursion($x, $options){
    for $z in $x/node() return mapping($z, $options)
}

define function mapping($x, $options)
{
  typeswitch ($x)
      case element(PLAY) return play($x, $options) 
      case element(TITLE) return title($x, $options) 
      case element(ACT) return section($x, $options)
...
}

But instead of returning an HTML element for each mapping, I created an Office Open element, in this example, the Title element.

First we wrap it in a <customXml> element to add the structure to the Word file.  Then we make a <w:p> element which is the basic building block of a word document.

Because this is a heading, we want some style - so depending on where we encounter this element, we are create style inside of the <w:pStyle> element using the standard word styles "Heading1", "Heading2" etc..

Finally we create a 'run' to hold the text with a <w:r> and put the text inside of the <w:t> element:

define function title($x as element(), $params as node()) as element()
{
    <w:customXml w:element="TITLE">
    <w:p w:rsidR="00592BC3" w:rsidRDefault="009E30CC" w:rsidP="009E30CC">
          <w:pPr>
          <w:pStyle w:val="{
              if ($x/parent::PLAY) then
                   "Heading1"
              else if ($x/parent::ACT) then
                   "Heading2"
              else if ($x/parent::PERSONAE or $x/parent::SCENE) then
                    "Heading3"
              else
                    "Heading4"
           }"/>
           </w:pPr>
           <w:r>
               <w:t>{passthru($x, $params)}</w:t>
           </w:r>
    </w:p>
    </customXml>
}

 

The result is some XML that we can zip up in an Office Open Package (see Pete's posts or the slides on how to do this) and open inside of word:

Pretty neat!

Office Open XML really opens things up in the new Office 2007 suite letting you access and query the documents created in the suite, manipulate that content or create new Office content from other source.

Its XML after all, have fun messing with it!

Matt

Code with the XQuery Experts: 11/29, London U.K.

Right after the (American) Thanksgiving holiday and just before London Online, the world's best XQuery coders, Jason Hunter and Ryan Grimm, will be over in Royal London hosting an XQuery day.  The event details are:

Code with the XQuery Experts
Friday, November 30, 2007
8:30 am PT - 5:00 pm
Olympia Grand Hall
London, England

Sign up for it here.

Jason and Ryan have been using XQuery (and MarkLogic Server) from the very start and it should be an excellent event to see firsthand why XQuery is THE application language for content applications.

What's more, you can bring your own XQuery chops and win an 8GB iPhone for the Best XQuery App at the event.  There are some helpful tools in the Mark Logic code workshop to get you started and I expect full credit if something in my tutorial helps you win first place!

But you might want to 'enhance' this one:  I asked Jason and Ryan for something neat from the amazing XQuery powered email discovery application they've built called MarkMail and they send me this very elegant FAQ generator.

The first cool thing is that it's a complete FAQ in a single complete XQuery - starting with the content and then the code to present it:

(: content as XHTML in a div - edited by MT to be a sample :)

let $content :=
<div id="content">
<a name="general"/>
<h1>GENERAL FAQ</h1>

<a name="quick"/>
<h2>Given 15 seconds, what should I know?</h2>
<ul><li>MarkMail lets you search 4,000,000+ emails across 500+ Apache mailing lists</li>
...</ul>
<a name="whatisit"/>
<h2>What is MarkMail?</h2>
<p>
MarkMail is a community-focused searchable message archive, accessible at <a
href="http://markmail.org">http://markmail.org</a>, developed and hosted by <a
href="http://www.marklogic.com">Mark Logic Corporation</a>.
...
</p>
...
<a name="techie"/>

<h1>TECHIE FAQ</h1>
<a name="whatshard"/>
<h2>What's hard about searching email?</h2>
<p>
Email doesn't work well in a relational model because there's too much free
text.  It doesn't work well in a search engine either because there's too much ad hoc structure and hierarchy ... We've found email works naturally as XML.
...
</p>
<a name="store"/>
<h2>How do you store the emails?</h2>
<p>
Each email is stored an XML document inside MarkLogic Server.
...
</p>
...
</div>

(: from this XHTML node we can generate the table of contents including a split between regular and techie FAQ
:)

let $toc :=
    <div class="toc">
        <h1>Table of Contents</h1>
        <ul>
        {
            for $head in $content/h2[. << $content/h1[. = "TECHIE FAQ"]]
            let $name := $head/preceding::*[1][name(.) = "a"]/@name
            return <li><a href="#{ $name }">{ string($head) }</a></li>
        }
        </ul>
        <h3>Techie FAQ</h3>
        <ul>
        {
            for $head in $content/h2[. >> $content/h1[. = "TECHIE FAQ"]]
            let $name := $head/preceding::*[1][name(.) = "a"]/@name
            return <li><a href="#{ $name }">{ string($head) }</a></li>
        }
        </ul>
    </div>

(: then we put it all together :)

let $body := (
    <div id="docs">
        { $toc }
        { $content }
    </div>,
    <div style="clear: both"/>
    )

return

(: and output it :)

$body

I like that XQuery gives you a compete tool kit for content:   even if you just have simple HTML you can do things like use the << and >> order comparison operators to pull out all of the <h2> elements that come before the Techie FAQ H1 and grab the <a> element right before the <h2> using the preceding axis.

And you're creating the output as you go - with an XHTML FAQ generated in less than 30 lines.  To see the 'live' FAQ click here.

 

I hope you can make it out to the event at Olympia and can see first hand the many cool things you can do with XQuery, the right tool for the content application job.

Matt

XQuery at Work

I got some good feedback on my post about enrichment (thanks!) and I thought I would expand a bit on the first step of getting the list of items to power the enrichment.

I like this example because it just seems to be something that comes up over and over again.  I've been working with web technologies for  10+ years now and this was one of the first things I learned how to do . . . and its something I am still doing day to day.  Its a truism of working with the web: at some point you will need to reach out and grab something off another website. 

XQuery is great at this because no matter what the context - from SOA to complex sounding (but pretty simple) federated search to just needing to get a list of weapons to enrich Shakespeare - the basics are make an HTTP connect and request and parse the usually XML response.

And there is nothing better for parsing and processing XML than XQuery.

So in this example:

fn:string-join(
for $weapon in xdmp:tidy(xdmp:http-get( "http://en.wikipedia.org/wiki/List_of_medieval_weapons")[2])[2]//*:li/*:a
return
fn:string($weapon), '","')

the idea is to get the web page and make it into a sequence of strings.  In case you are wondering, every XQuery returns a sequence - even if its sequence of 1.  A sequence can have any number of items of any type.  Pretty useful as we will see.

The first step is to get the page with xdmp:http-get() - a MarkLogic XQuery extension.  This returns a sequence of two nodes.  The first is the response node with the header info.  The second is the actual page/image/whatever that you requested:

xdmp:http-get( "http://en.wikipedia.org/wiki/List_of_medieval_weapons")

returns

(<response xmlns="xdmp:http">
<code>200</code>
...
<headers>
<date>Wed, 03 Oct 2007 00:44:16 GMT</date>
<server>Apache</server>
. . .
</headers>
</response>
,
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en" dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
...
</head>
<body>
...
</body>
</html>)

So while it looks like XML, that second item it really text.  We need to turn it into XML with tidy:

xdmp:tidy(xdmp:http-get( "http://en.wikipedia.org/wiki/List_of_medieval_weapons")[2])

The [2] gives us the second node of the response sequence and tidy returns:

(<status xmlns="xdmp:tidy">
<message>Info: . . .
</message>
</status>
,
<html xml:lang="en" lang="en" dir="ltr" version="-//W3C//DTD XHTML 1.1//EN" xmlns="http://www.w3.org/1999/xhtml">
<head>
...
</head>
<body>
</body>
</html>
)

Yup, another sequence . .. but now that second node is XML as produced by tidy (with all of the errors etc noted in the <status>).

From here on, we can use XQuery to process XML into that sequence of we need to do matches inside the text of the Shakespeare plays.

First we need to get the list out of the page.  It turns out that within that Wikipedia page, all of the items are listed within <li> elements AND are with link anchors - <a>.

Our first step is to use XPath to get just the sequence of <a> elements - and to do this I'm using * as the namespace - it's likely the XHTML namespace, but this way I don't have to even check:

xdmp:tidy(xdmp:http-get( "http://en.wikipedia.org/wiki/List_of_medieval_weapons")[2])[2]//*:li/*:a

This gives us another sequence - this time of <a> elements:

(<a href="#Axes" xmlns="http://www.w3.org/1999/xhtml"><span class="tocnumber">1</span> <span class="toctext">Axes</span></a>
,<a href="#Daggers_and_knives" xmlns="http://www.w3.org/1999/xhtml"><span class="tocnumber">2</span> <span class="toctext">Daggers and knives</span></a>
,<a href="#Swords" xmlns="http://www.w3.org/1999/xhtml"><span class="tocnumber">3</span> <span class="toctext">Swords</span></a>
,...
)

We can now use the FLOWR structure of XQuery to process this sequence one <a> at a time.  FLOWR stands for For Let Order by Where Return.  For assigns each item of a sequence to a variable, let (not used in our example) can hold additional values related to that item, order by allows you sort (default is document order) where allows you to filter and return is the output for each item in the sequence.

In our example we assign each <a> element to a variable, then use the fn:string() function to get the string value.  Running this over the entire sequence of <a> elements creates a sequence of strings:

for $weapon in
xdmp:tidy(xdmp:http-get( "http://en.wikipedia.org/wiki/List_of_medieval_weapons")[2])[2]//*:li/*:a
return
fn:string($weapon)

This returns:

("1 Axes","2 Daggers and knives", "3 Swords", .... "Broadsword", "Claymore", "Cutlass", "Falchion", ...)

We're getting the header values, but thats OK - they just won't match anything.

And we now have a sequence of strings.

Except that I wanted to save this as a file so I could run it over and over again while I got the enrichment right.

So while I've written it out as a sequence of strings (and this is the accurate state within the server) if I output this, I actually get this:

1 Axes
2 Daggers and knives
3 Swords
...
Broadsword
Claymore
Cutlass
Falchion
...

This is also correct . . . when output, the strings are represented as, well, strings.  And certainly not as the  comma delimited list of quoted items that is really a string sequence constructor . . . and what I need.  So I need to make a single string that *is* comma delimited and has values in quotes so I can then stick that inside my code and it will become a sequence of strings.

To do this I just use my favorite XQuery function, fn:string-join().  This takes ANY sequence and makes it into a nice single string with whatever delimiter you select between each element:

fn:string-join(
for $weapon in
xdmp:tidy(xdmp:http-get( "http://en.wikipedia.org/wiki/List_of_medieval_weapons")[2])[2]//*:li/*:a
return
fn:string($weapon)
, '","')

The delimiter here is ",".  Like many content friendly languages, XQuery lets you use single and double quotes so you can return these special characters.

And unlike many other languages, fn:string-join correctly puts the delimiters *between* every item in the list . .. and doesn't also put it on the last item which you then have to correct when you write this function yourself in java, perl . . . just about anything else.

The result is *almost* a single string that represents a sequence of string that can be put into a file or another XQuery:

1 Axes","2 Daggers and knives","3 Swords","4 Blunt weapons" ...

I didn't do the extra step of adding the first '("' and the last '")' . . . we can just concat these on:

concat(
'("',
fn:string-join(
for $weapon
...
,'")'
)

And now we have out string for enrichment:

for $doc in xdmp:directory("/content/bill/")
let $weapons := ("axe", "sword","dagger", "falchion" , "etc., etc.")
return
    ....

To read the rest of the story about what to do with a list of weapons and the works of Shakespeare, Click here.

For my part, I use this pattern and process all the time.  Just last week someone suggested that it would be really cool to use Flickr to augment information on cameras by showing actual photos taken with that camera.

What a good idea!  And, once you know all these tricks in XQuery, its a one-liner, thanks to flickr's nicely structured pages (where sample images are in a "DetailPic" class):

xdmp:tidy(xdmp:http-get("http://www.flickr.com/search/?q=photo&cm=nikon%2Fd200")[2])[2]//*:td[@class="DetailPic"]//*:img

I'm asking for my camera (Nikon D200) and a generic term 'photo' and I get these nice pictures:

All ready to be inserted into a rich, XQuery powered, content application.

Matt

XQuery and Lazy Enrichment: Keeping me Busy

I was searching for a blog topic and my son Josh suggested that I write a blog about how busy I've been in the last couple of weeks.

Excellent idea Josh!  Because it's XQuery and the cool things you can do with it that have been keeping me busy.

On top of helping people explore new ways to get the most of their XML content from with XQuery powered projects in Digital Asset Delivery (that take advantage of the benefits of the XML meta-data model I wrote about a while ago) and working on XML powered content production workflows (similar to the ground breaking XQuery powered SafariU from O'Reilly), I've been also been exploring using XQuery to enhance XML with something called lazy enrichment (a term coined by Mark Logic CEO Dave Kellogg).

The idea goes something like this:  if you have some content (say the complete works of Shakespeare) and they are loaded into an XML content server (say MarkLogic Server as in this tutorial), then wouldn't be interesting to cross cut the content with a topic like medieval weapons and be able to explore the Shakespeare texts in this new context?

This is text analytics: process text to get match topics and categories of items and give that content new meaning.  There are great engines out there that do this.  They analyze text, sentence structure and patterns and can automatically create categories, lists of people, specific places and even market specific items like company names or medical terms.

But using XQuery and some XML you can do it yourself with lazy enrichment and end up with something much better than lists of items.

The first step is to get some information on your chosen topic.  While the engines that you pay lots of money for have detailed databases on specific topics, they all start with a taxonomy or controlled vocabulary of the topic.  But if you know what your topic is, you can make your own list (and chances are you already have one).

For our example, lets go get a list of medieval weapons from wikipedia:

fn:string-join(
for $weapon in xdmp:tidy(xdmp:http-get( "http://en.wikipedia.org/wiki/List_of_medieval_weapons")[2])[2]//*:li/*:a
return
fn:string($weapon), '","')

This uses the MarkLogic Server HTTP built-ins to get the page and then does some processing to get just the list of weapons.  It's not called the best scraper on the web for nothing.  While we're at it, lets make a nice comma delimited list with quotes since we will want a text sequence for our enrichment process.

Now that we our list of weapons, lets process the text:

for $doc in xdmp:directory("/content/bill/")
let $weapons := ("axe", "sword","dagger", "falchion" , "etc., etc.")
return
    xdmp:document-insert(xdmp:node-uri($doc)),
        cts:highlight($doc,
            cts:or-query(
            for $test in $weapons
            return
            cts:word-query(fn:lower-case(fn:string($test)))
            )
            ,<weapon>{$cts:text}</weapon>
        )
    )

This takes every play I've loaded and, using MarkLogic Server's search features, finds the words in the play that match the list and creates new markup around that word to enrich the content with <weapon> elements.

This is lazy enrichment - taking our own knowledge of a topic, creating some rules around the matching (that can go well beyond simple search string matching) and then enriching the content in place.  We're not creating any separate lists or extracting this to a database - the content now has the topic embedded in it.

What can we do with this?  Well how about some really cool queries:

The first one is to actually create those topic lists - they are after all very useful.  So what are all the weapons that were in the plays?

fn:string-join(cts:element-values(xs:QName("weapon")), ",")

Using MarkLogic Server's element-value indexes, this gives us a report on the weapons found.

But how about something a bit more interesting:  which characters talk about weapons (so we can stay away from them)?

for $speaker in distinct-values(//SPEECH[./LINE/weapon]/SPEAKER)
return
<violent>{fn:string($speaker)}</violent>

Or maybe we'd like to know more about a specific weapon - in the list created in the first query in my database something called a 'falchion' showed up.  What the heck is a 'falchion'?  Lets make a report:

for $weapon in //SPEECH/LINE/weapon[.="falchion"]
return
<weapon>
    <name>{fn:string($weapon)}</name>
    <play>{fn:string($weapon/ancestor::PLAY/TITLE)}</play>
    <character>{fn:string($weapon/ancestor::SPEECH/SPEAKER)}</character>
    <speech>
        {$weapon/ancestor::SPEECH/LINE}
    </speech>
</weapon>

This returns us a nice report:

<weapon>
    <name>falchion</name>
    <play>The Tragedy of King Lear</play>
    <character>KING LEAR</character>
    <speech><LINE>Did I not, fellow?</LINE>
                <LINE>I have seen the day, with my good biting          
                <weapon>falchion</weapon></LINE>
                <LINE>I would have made them skip: I am old now,</LINE>
                <LINE>And these same crosses spoil me. Who are you?</LINE>
                <LINE>Mine eyes are not o' the best: I'll tell you straight.</LINE>
    </speech>
</weapon>

Because the weapon is marked up in the actual content, I can leverage the structure, the content around it and XQuery to give me as complex a report and analysis as I can possibly want.

But I still don't really know what a falchion is - besides the fact that its 'biting' and that Lear was fierce with it when he was young.

So for some extra credit, lets reach out to another source and, in place, augment our reading of Shakespeare to give us some new understanding.  In our transformation of the plays for display (using the XQuery transformers), let's go ask google what our weapon is:

define function weapon($x as element(), $params as node())
{
      <span>
             <span onclick="toggleDisplay(document.getElementById('pop-up'))">
             { passthru($x, $params) }</span>
             <span id="pop-up" style="position: absolute; display:none; border-style: outset; border-width: 2; background: #ffffff; font-family: Arial; font-size: 11px">
             <div style="background-color: #FFC770;">
              <h3>Google Search Results</h3></div>
             <div><ul class="basicMenu">
             {
             for $a in
             (xdmp:tidy(xdmp:http-get(fn:concat("http://www.google.com/images?q=", fn:string($x)))[2])[2]//*:div//*:td)[1 to 2]
             return
             <p>{$a}</p>
             }
             {for $a in
             (xdmp:tidy(xdmp:http-get(fn:concat("http://www.google.com/search?q=", fn:string($x) , "&submit=Google+Search"))[2])[2]//*:div[.//*:table]/*:a)[1 to 3]
             return
             <p>{$a}</p>
             }            
             </ul></div></span>
      </span>
}

This gives us a nice reading of the plays that looks like this:

And we can see that a Falchion is a mean looking, huge sword.  Watch out for Lear!

Lazy enrichment turns out to be pretty powerful stuff!  You can annotate and augment texts with your own concepts and build rich displays to get new meaning out of even tried and true Shakespeare.

Thanks for the idea for the blog Josh - it turns out that even a 'lazy' idea in XQuery is exciting stuff that certainly does keep you busy!

(XQuery) Transformers! More than meets the eye!

Hoping to capitalize on marketing for the upcoming Transformer movie, I thought I would take a look at what makes XQuery the ultimate XML transformer.  (What?  You haven't heard?  It's actually not bad looking looking.)

Just like the nutty car and truck toys, XML has always been made to be transformed into something else.  And while the end formats of HTML, mobile devices and print ready PDFs aren't as much fun as 5 story tall robots, the idea is pretty much the same: your content is in a presentation free format (just like Bumblebee's Camaro) and can instantaneously be transformed into something else (lets just say . . . a huge yellow robot).  Everything needed to do the transformation is actually in the content . . . you just need an engine and some rules to make it happen.

A couple of weeks back I posted about making PDFs from the Shakespeare XML in the tutorial that looked a bit like this:

   for $speech in //SPEECH
    return
      <formatting-for-speech>
            {fn:string($speech)}
        </formatting-for-speech>

While this worked, it's more of a report type of XQuery.  What we want to do is feed in some XML and get out the new format.

To do this, we'll use a design pattern that is very similar to what an XSLT engine does behind the scenes:  we'll recurse through the XML and then perform formatting as we hit certain elements.

The first part is to set up the recursion:

define function recursion($x, $options) {
    for $z in $x/node() return mapping($z, $options) 
}

Then we'll make its counterpart - the mapping:

define function mapping($x, $options)
{
  typeswitch ($x)
      case element(PLAY) return play($x, $options) 
      case element(TITLE) return title($x, $options) 
      case element(ACT) return section($x, $options) 
      case element(SPEECH) return p($x, $options) 
      case element(LINE) return p($x, $options) 
      case element(PERSONAE) return section($x,  $options) 
      case element(PERSONA) return p($x, $options) 
      case element(STAGEDIR) return i($x, $options) 
      case text() return output-text($x, $options)
      case element(FM) return ()
      default return p($x, $options)
}

While it's similar to an XSL transformation, the important thing is that we control the recursion.  If we wanted to add some (admittedly silly) logic to recursion() we can just go ahead and do it:

define function recursion($x, $options) {
    if (fn:contains($x, "blood")) then
        <p>warning: blood coming up: {for $z in $x/node() return mapping($z, $options)}</p>
    else
        for $z in $x/node() return mapping($z, $options)
}

We now have the list of parts (just like the Transformers wheels, door handles, etc) and we now need to make our 'template' functions to make them into something new.  Instead of a fancy robot, we'll just make some HTML:

define function p($x, $options) {
        <p>{recursion($x, $options)}</p>
}
define function i($x, $options) {
        <i>{recursion($x, $options)}</i>
}
define function output-text($x, $options) {
  if (fn:empty($x)) then () else text {$x}
}

These basic transformations work with the mapping --> if we have a <LINE> it is now a <p> because it went through the p() function.

These are, after all, XQuery functions so we can also do some tests and complex formatting as well:

define function title($x, $options) {
    if ($x/parent::PLAY) then
        <h1>{recursion($x, $options)}</h1>
    else
        <h3>{recursion($x, $options)}</h3>
}
define function section($x, $options){
        <div style="margin-top: 25px;">{recursion($x, $options)}<hr/></div>
}

The super cool thing is that $x node we are passing around carries it's context.  Here we are doing a test of it's parent element.  We could reach up to the root node and get the play title or anything else . . .  the possibilities are endless and since transformations always have a catch, this flexibility comes in very handy.  Also, I'm pretty sure this is how the Transformers can do such cool tricks :).

Finally, we'll do the equivalent of a Transformer Powerlink (sure to be the climax of the movie).  This is where two Transformers bend and twist and make a new, bigger, more powerful robot.

To make our more powerful XML transformation, we'll use that $options node that we've been passing around:

define function play($x, $options) {
<html><head>
<title>{fn:string($options/desc)}: {fn:string($x/TITLE)}</title>
<meta name="keywords" content="{$options/meta-info}"/>
</head><body>
{recursion($x, $options)}
</body></html>
}

(: main function :)
let $meta-info :=
    let $personae :=  //PERSONA/text()
    return
        fn:string-join($personae, ",")
let $options := <options><desc>XQuery
                        Transformers!</desc>
                        <meta-info>{$meta-info}</meta-info></options>
for $play in (/PLAY)[1]
return
    mapping($play, $options)

We are pulling all the Shakespeare characters for extra large meta-data and passing along my new tag line: XQuery Transformers!  In the transformation for the PLAY element we stick this in the head of the document (a common PowerLink move) to make an HTML wrapper that packs some extra power.

Putting this all together in a single stored XQuery module will give you the building blocks for powerful , fully programmable transformations. 

XQuery Transformers: more than meets the eye!