Discovering XQuery

I'm Back ... Anything Change?

Wow - its been a while! In fact, it's all the way back to 2010 since I last posted here. Remember 2010?

I started this blog to document my own journey of working with native XML tools, specifically XQuery and MarkLogic's database platform. Along the way I had a lot of fun and wrote about:

• How to not end up wiping your hands on your pants after going to the bathroom (XQuery just works)
• Why its better to drive a Ford GT instead of an AMC (even if it is the AMC Bizzarini)
• How Transformers are more than just huge robots
• What a Falchion is (and also what lazy enrichment is)
• And how Santa really makes his really big list (note - some foreshadowing here...)

So, what's been going on out there in the meantime? Anything happening with data, information and content? I can tell you one thing, I never finished that New Haven Pizza XQuery app ... but I sure have had a lot of great pizza in the last 8 years. And I've also done a lot of work with XML ... just maybe not had the time to write it up.

Well that might be changing. It turns out that XML hasn't gone anywhere. In fact, with the rise of NoSQL and the acknowledgment that data from many sources needs to be integrated to get the complete picture of your business, it turns out a LOT of that data is either in XML or is exchanged between systems in XML formats.

There is also a lot of JSON, growing investment in ontologies and semantic data, and good old relational data. But sometimes you just need to describe some really complex data and make it available to machines and people. Self-describing, standardized, and widely adopted, XML is the right tool in the toolkit.

Today, this data is so much more than content. Things like manifests, bills of materials, customers, parts and component data, assets and maintenance records are all expressed as XML and, increasingly, are being modeled and managed as XML inside large enterprises.

Is XML making a comeback? I don't think so ... because I don't think it ever went away. In fact, my bet is that it's in even wider use than you might think. First all all, every type of data continues to exponentially grow. But, more to the point, XML is an important part of the data toolkit for anyone working to get value from their data.

So, will there be more Discovering XQuery posts? Stay tuned ... I am, after all, writing this re-introduction entry on purpose :)

Still XML hacking away,

Matt

 

Countdown to the MarkLogic User Conference

Just hours away from the first events of the User Conference and XQuery geeks all over are sharpening their MacBooks, testing out iPad integrations and, if you are like me, polishing up some XQuery code to show everyone that gotta have it feature.

Highlights from the show include great customer talks about XQuery powered applications from Royal Society of Chemistry, JetBlue, Yell, Bowker and more.  

And there is plenty of XQuery (and XSLT) geeking out with:

• Darin McBeath's (Elsevier) session on building all XQuery applications
• How XML and XQuery can do Semantic and Linked Data queries with MarkLogic's Jason Monberg and Michael Blakeley
• The 'super' session (is it a Bird etc) on the MarkLogic App server by Justin Makeig of MarkLogic and Mindie Sorenson and Ryan Semerau, LDS

There will also be a reprise of my favorite customer session ever, the McGraw-Hill Create's Nimble Elephant session.  This open discussion about creating and executing an innovative project wowed 'em at the Digital Publishing Summit in December and Shannon and crew are back again for the user conference.

Besides the sessions there are good times to be had, including a conference wide party at the California Academy of Sciences . . . and of course the Demo Jam tonight - sponsored by Avalon Consulting (thank you Avalon!)

 

This brings me to my XQuery I'm working on.  Demo Jam is an event that I started to let developers show what they've been working on.  The format is simple:  you have 5 minutes to show your demo.  At the end of the demos, each person's name and demo title is read and we take a decibel meter reading to select the winner (that would be the one with loudest crowd noise ).

To make things fair I've tried to keep the order picking random, using names in a hat or some quick XQuery hacking like this:

              for $pres in ("presenter1", "presenter2", "etc")
              order by xdmp:random()
              return
              $pres

But since this Demo Jam is for our customers and partners and has some real XML and XQuery gurus (see the full list here), I thought it would be fun to pick the order AND open it up in PowerPoint . . . all with XQuery.

First I loaded a blank Demo Jam contestant list into the MarkLogic Server and let it do the Office OpenXML conversion.

Then I run this XQuery to edit the slide, putting my new order into it:

    declare namespace a="http://schemas.openxmlformats.org/drawingml/2006/main";
    declare namespace p="http://schemas.openxmlformats.org/presentationml/2006/main";

    let $presenters := ("presenter1","presenter2", "etc")

    let $node := doc("/demojam/demo-list_pptx_parts/ppt/slides/slide1.xml")//p:txBody
    let $new-node :=  let $line-up := let $order := for $pres in $presenters
                  order by xdmp:random()
                  return
                  $pres

    for $presenter at $number in $order
    return

    (: Make the Bullet for each presenter :)

    <a:p><a:pPr marL="457200" indent="-457200">
    <a:lnSpc><a:spcPct val="170000"/></a:lnSpc>
    <a:buAutoNum type="arabicPeriod"/></a:pPr>
    <a:r><a:rPr lang="en-US" sz="2000" dirty="0" err="1" smtClean="0">
    <a:latin typeface="Lucida Sans"/><a:cs typeface="Lucida Sans"/>
    </a:rPr><a:t>{$presenter}</a:t>
    </a:r><a:endParaRPr lang="en-US" sz="2000" dirty="0" smtClean="0"><a:latin typeface="Lucida Sans"/><a:cs typeface="Lucida Sans"/>
    </a:endParaRPr>
    </a:p>

    return

    (: write out the header as well :)

    <p:txBody><a:bodyPr/><a:lstStyle/>
    <a:p><a:pPr><a:lnSpc>
    <a:spcPct val="170000"/></a:lnSpc>
    <a:buFont typeface="Arial" pitchFamily="-108" charset="0"/><a:buNone/></a:pPr>
    <a:r><a:rPr lang="en-US" dirty="0" smtClean="0">
    <a:latin typeface="Lucida Sans"/><a:cs typeface="Lucida Sans"/>
    </a:rPr>
    <a:t>2010 User Conference Demo Jam</a:t></a:r>
    </a:p><a:p><a:pPr>
    <a:lnSpc><a:spcPct val="170000"/>
    </a:lnSpc><a:buNone/>
    </a:pPr><a:r><a:rPr lang="en-US" sz="2000" dirty="0" smtClean="0">
    <a:latin typeface="Lucida Sans"/><a:cs typeface="Lucida Sans"/>
    </a:rPr>
    <a:t>Line Up</a:t></a:r></a:p>
    {$line-up}
    </p:txBody>

    return

    (: put all that back into slide1.xml :)

    xdmp:node-replace($node, $new-node)

Then I use the handy tools from the MarkLogic Toolkit for PowerPoint to put it all together and return open it up in Excel:

    xquery version "1.0-ml";
    import module namespace ppt=  "http://marklogic.com/openxml/powerpoint" at "/MarkLogic/openxml/presentation-ml-support.xqy";

    xdmp:set-response-content-type("application/powerpoint"),

    let $presentation-dir:="/demojam/demo-list_pptx_parts/"
    let $map := ppt:package-map($presentation-dir)
    return
    ppt:package-map-zip($map)

If this doesn't all work live tonight at Demo Jam, maybe I'll check out the Microsoft Office session Pete Aven, MarkLogic and Jeffery Brown, LDS are putting on for some pointers.

See you there (in just a couple of hours!),

Matt

Getting Ready: T-minus 3 Days to the Mark Logic User Conference

The run up to the Mark Logic User Conference (The World's Largest XQuery Gathering) is usually hectic and this year is no different.  I've been busy preparing for the 1-day orientation class that I will be teaching and desparately trying to squeeze in bike rides so I don't embarrass myself on the 'unofficial' conference ride I'm putting together.  Getting ready for the class is somewhat easier as it mostly involves brushing up on Shakespeare.  We use the complete Shakespeare XML as a basis for the examples and its always fun to do some exploring of the text with the class.  I am hoping to expand my set of queries for cool things like King Lear's Falchion.

Other folks are also plenty busy.  Scott Able, AKA The Content Wrangler, will be our official Blogger at the conference and his interview (located here) with MarkLogic's Norm Walsh is a great warm up for the conference.  It has excellent examples of XML and XQuery in action, some sound reasoning on the correctness of using XML for content and some thoughts on its widespread (but still unsung?) use in almost everything.  I particularly like his retelling of how Dave Kellogg steered him in the right direction in a response to his newbie question on a MarkLogic internal list.  Norm, the rest of us just thought you were crazy :)

Dave's attention to detail does pay off - this week he was busy collecting awards:  MarkLogic Server just won the SIIA Codie Award for Best Databse Management System and Dave's blog (where among other things he related that story about Norm and talked about the bike ride) won for Best Corporate Blog.  Way to go Dave . . . way to go MarkLogic Server team!

Over at O'Reilly, Kurt Cagle has been pushing the boundaries of MarkLogic Server and our implementation of XQuery for his presentation at the conference.  This post looks at some of the neat features that Mark Logic has added to XQuery to make programming easier.  Function mapping will automatically execute a function for every item in a sequence and maps let you work with hash maps within XQuery.  I particularly like Kurt's post because he gives good XQuery code samples, something I am always trying to create more of.

I'll have to make sure I add Kurt's session (Tuesday at 2:45) to my conference plan.  It's getting pretty full with so many great customer sessions (gotta make BusinessWeek and Wiley) and can't miss technical presentations (AtomPub from Norm, Kurt's session and the MarkLogic feature sessions) that I'm afraid I'll miss out on all the good side conversations about how XQuery is changing the face of content applications. 

If I can just make it through getting ready, it is going to be really fun!

Matt

World's Largest XQuery Gathering Only Two Weeks Away!

The Mark Logic User Conference is just two weeks away!   Once again, XQuery enthusiasts will flock to San Francisco for a great mix of hands on technical sessions, best practices and, my favorite, customer application profiles.  Look for lots of live demos, interactive and frank discussions about using XQuery and some very eye opening applications. 

Publishing customers like BusinessWeek and Wiley will be highlighting their new content applications, companies like JetBlue talking about how XQuery driven applications have impacted their enterprise, government integrators Boeing and Booz Allen will be talking about how MarkLogic Server helps them get the job done and organizations like the The Church of Jesus Christ of Latter-day Saints who will showcase their use of MarkLogic Server to power all kinds of applications.

More than few key members of the XML and XQuery community will also take part with Kurt Cagel talking about ReST services and Norm Walsh highlighting AtomPub powered by XQuery.

What all these sessions share is a strong belief that the XML model and XQuery programming really work, get the job done faster and help create wonderful new applications that just aren't possible using the traditional (old?) tools like databases and search engines.

And with over 400 people attending, it is indeed the World's Largest XQuery event!  So be there or be ... relational or something.

Matt

Mark Logic User Conference 2009 Breakout Session Guide

Good News (for XQuery at least)

Lots of good things are happening with XQuery which is a nice change from the regular news.

First and foremost, MarkLogic Server 4.0 is out!  With this release, MarkLogic Server is updated to XQuery 1.0 and has added some very cool features to extend the capabilities of XQuery.

The highlights are:

• Geospatial support.  Lets you add location as a dimension to queries and find where your content is at.  Like this search result from some content about earthquakes where the search is for the word 'new' AND anything within a 500 mile radius of Utica, NY:

 

• Entity Enrichment.  Puts some smarts right into your content by marking up the people, places and things.  This is coupled with an enrichment framework to plug your own text mining tool into the process.

• Alerting.  Lets you notify people about new content they may be interested in.  A key part of this is something called reverse-query which returns the queries that match incoming content.  Very cool.

As usual theres a lot more in MarkLogic Server 4.0 and here are links to an overview and the developer site.

But it's not just Mark Logic that has been busy.  At Elsevier the Article 2.0 contest is off and running.  It's a chance to win $4000 by coming up with new ways to present scholarly articles.  Who couldn't use a little bit of extra $$$ in these times?  All you have to do is create the best new way to display the source XML Elsevier is providing.  Here are some hints on how to use XQuery to give you an edge.

And overall XQuery is getting more and more attention.  When I first started writing this blog (back in April of 2006  - wow thats more than 2 years ago!)  XQuery was that strange language that used squiggly brackets and smiley faces and there just wasn't a lot of dicsussion about it.

In just the last couple of weeks I've seen:

• A video of an XQuery lecture from the University of Irvine
• Posts on XRX (XForms, REST, XQuery) outlining an new application stack where adopting the XML model and using XQuery greatly simplifies processing (here are some similar thoughts from me, me again and Dave Kellogg)
• The continuation of the wonderful XQuery, Multi-platform future meme/spam.  As I discussed in this post a year ago, this very lightweight article was replicated throughout the internet as a simple traffic driving trick.  XQuery had finally made enough of an impact to merit someone actually spending time spreading it around . . . and he's still doing it - just last week I got this article in another google alert.

So it's nice to see XQuery bring us some good news - sure beats the other kind.

Matt

Its (Really) About XQuery

I finally got on the Wordle bandwandwagon and ran the Discovering XQuery Blog through their clever wordcloud generator thingy.  Actually, I ran it more than a couple of times since it's a lot of fun to let it create random patterns (warning: time suck!).

Here is what the most recent posts (it uses the RSS feed) produced:

(click here to see in on Wordle)

What a shocker!  Yup, XQuery is front and center here at Discovering XQuery and with XML to base it on you have all you need to create and work with content (of course using MarkLogic Server).  I also like the accidental sentence 'XQuery can script'.  Youbetcha!

A good example of this is this paper from the upcoming International Conference on Music Information Retrieval (ISMIR2008).  In between the sessions on music recognition and automated transcription (my favorite: Multiple-Feature Fusion Based Onset Detection for Solo Singing Voice) is this session on working with MusicML: Using XQuery on MusicXML Databases for Musicological Analysis.  The authors got some MusicXML from wikifonia (a free site that collects MusicXML and provides a nice rendering of the music) and used XQuery like a buzzsaw to slice up the content sorting out duplicate titles, listing key signatures and time signatures and even trying to find motives (sequences of chords or rhythms) in the corpus.  Pretty cool stuff!

I thought I'd give it a try myself and add maybe add to their collection of queries.

The first step is to get some content.  Lloading XML from an external source looks like this in MarkLogic:

for $i in (1 to 200)
let $song-source := concat("http://static.wikifonia.org/" ,$i , "/musicxml.xml")
let $database-uri := concat("/songs/" ,$i , "/music.xml")
return
xdmp:document-load($song-source,

xmlns="xdmp:document-load">

<uri>{$database-uri}uri>

xml

)

This grabs the first 200 songs and inserts them into a database.

The XML is pretty neat and, like all good XML, nicely self-describing:

partwise version="2.0">

    <movement-title>What's New</movement-title>
    <identification>
        <creator type="composer">Bob Haggard</creator>
   ...

    </identification>
    <part-list>
        <score-part id="P1">
            <part-name>Voice</part-name>
            <score-instrument id="P1-I1">
                <instrument-name>Voice</instrument-name>

<!-- Then the actual music in measures -->

   <measure number="1">
        <note>
            <pitch>
               <step>D</step>
               <octave>4</octave>
            </pitch>
            <duration>384</duration>
            <voice>1</voice>
            <type>eighth</type>
            <lyric number="1" name="verse">
                <syllabic>end</syllabic>
                <text>You</text>
            </lyric>
        </note>

        ....
    </measure>
....
</score>

Having loaded some MusicXML, it's time to run use some XQuery to explore the music.  You can run the ones in the paper like this one that lists the titles that have no rests:

for $i in //score-partwise
return
$i[count($i//rest) eq 0]//movement-title

This gives us a sequence of titles:

Zachtjes gaan de paardenvoetjes

Sinterklaas is jarig

Or you can do your own thing - like listing the instruments that have parts written for them:

fn:distinct-values(//score-partwise//score-instrument/instrument-name)

And then using one of the values in the list to get all the notes (that aren't rests) for a given instrument:

let $notes := //score-partwise[.//score-instrument/instrument-name eq "Grand Piano"]//note
let $rest-notes := //score-partwise[.//score-instrument/instrument-name eq "Grand Piano"]//note[./rest]
return
$notes except $rest-notes

And we get our notes for the piano:

<note>
<pitch>
<step>B</step>
<octave>4</octave>
</pitch>
<duration>2</duration>
<type>eighth</type>
...
</note>
<note>
...

This makes use of the cool 'except' operator to give you only the <note> elements in $notes that are NOT in the sequence of <note> elements that have <rest> children in $rest-notes.  It turns out that these are always <note> elements with <pitch> children, but this is a very handy tool when you are only sure what you *don't* want.

The abstract of the paper ends with these thoughts:

This shows the feasibility of automated musicological
analysis on digital score libraries using the latest software
tools. Bottom line: it’s easy.

I couldn't agree more - and I have the Wordle to prove it:  if you have some XML, just put some XQuery on top and off you go!

Matt

Mark Logic User Conference 2008: Generate Some XQuery Buzz

The Mark Logic User Conference, the world's largest gathering of XQuery users, experts and fans, is only two weeks away!

The jam packed agenda of sessions on ground breaking and diverse XQuery applications is a feast for XQuery fans.  You'll get to see real-world XQuery applications in action from content syndication at Simon & Schuster to the Electronic Flight Bag at United Airlines to the Army's Knowledge Management Systems.  There are also best practice sessions and sessions on the latest XQuery tools from Mark Logic.

So sign up here to and come on out to San Francisco to see how XQuery is generating some buzz across a wide range of industries and uses.

And really why would we expect anything less?  We've seen that XQuery (with some help from the easy to use MarkLogic Server) can quickly get you up and querying XML in minutes, scrape the web with ease, create new XML, enrich your content and power AJAX.   All from just plain old XML (which is the right model for content of course).

And as for generating buzz . . . well how about generating XQuery?  Like all good languages XQuery can be used in generative (or automatic) programming.  Thankfully with AJAX you don't need to generate Javascript so much any more (phew!) . . . but this approach still has plenty of uses.  For instance, I recently had to generate some XQuery for the very handy performancemeters to run on some sample content.  To generate the script, I used XQuery (of course):

<h:script xmlns:h="http://marklogic.com/xdmp/harness"> {
for $count in (1 to 100)  (: get 100 calls :)
let $selected := xdmp:random(3001) (: get a random number to choose from the 3000 documents :)
let $uri := let $uri := fn:base-uri(doc()[$selected]) (: get the uri of the selected document :)
return
   <h:test>
     <h:name>sample test</h:name>
     <h:comment-expected-result>transform {$uri}</h:comment-expected-result>
     <h:set-up/>
     <h:query>
     (: generate the XQuery to run for the test :)
      import module namespace tx="http://www.marklogic.com/test/transformtest" at  
         "transform-test.xqy"
    
         tx:transform("{$uri}")
     </h:query>
     <h:tear-down/>
   </h:test>
}</h:script>

In this simple XQuery I'm collecting some random URIs and and creating the calls to the transform function (which as a simple transformation ala XQuery Transformers) but you can see that this would be very handy to use the values in the XML to write custom XQuery for just a test . . . or even as part of your XQuery content application.

With all that XQuery can do its no wonder its good at generating buzz!

See you at the user conference,

Matt

XML lists on MarkMail and XQuery at XML 2007

Just in time for the upcoming XML conference the MarkMail team (none other than Mark Logic's XQuery gurus Jason Hunter and Ryan Grimm) have added the xml-dev, xquery-talk and xsl-list mailing lists to the already impressive collection of developer mailing lists in MarkMail.

MarkMail is an XQuery powered next generation email search application that not only lets you easily find a specific message in a big stack of email (over 4,000,000 messages) but also presents analytics to let you see the patterns and trends across topics.

So far I've been looking at the big picture with the XML lists and its interesting to see the activity around XQL in the late 90s  when it was picking up steam . . .

(click here to see it live)

the way Quilt (the real ancestor to XQuery) sort of picked up the ball at first but quickly faded . . .

(click here to see it live)

and how XQuery really took over as it became a the standard (and got its own list so then things tapered off).

(click here to see it live)

I also like how Jonathan Robie is the number 1 poster for all of these searches - thank you Jonathon!!

But most of all I like that I was able to find all this great info with just a few searches and some drilling down into content.

MarkMail is a great example of a content application:  its data is email modeled as XML and its application layer is XQuery to take full advantage of the structure in the content and the powerful set of application buidling features XQuery provides.

And since its powered entirely from the XML content, you can read every post in the same interface.  So for the Quilt search I was able to scan the messages and get a feel that most people were actually just referring to Quilt when actually talking about other things (like XQuery):

(to see this click any message in this result)

So whether you are looking for trends - such as the decline of DSSSL or your favorite entry in a perma-thread
I think you will find MarkMail a valuable resource for all things on the XML lists.

And the timing couldn't be better.  Next week at the XML conference Jason will be giving the closing keynote "You're Darn Right XML has a Future on the Web" and the power of applications like MarkMail certainly underscore just what XML and XQuery can do.

And I will also be speaking at the conference subbing for Kelly Stirman in a session called "First Encounters with Open Office XML".  I'll be looking that this format then doing some live demos using XQuery to query it, take it apart and put it back together.

Hope to see you there and enjoy the new XML savvy MarkMail.

Matt

Code with the XQuery Experts . . . in San Carlos, CA NOT London

Sorry to all the folks over in the U.K. but it turns out the Code with the XQuery Experts event is NOT in London, but in Mark Logic's offices in San Carlos, CA.

New event details are:

Code with the XQuery Experts
Friday, November 30, 2007
8:30 am PT - 5:00 pm PT
Mark Logic Offices
San Carlos, CA

Registration is here

What better way to work off the soporific Thanksgiving turkey than some vigorous XQuery coding!  Plus the Best XQuery App contest lets you bring your XQuery chops and maybe win an iPhone.

For those of you in and around London, Mark Logic is hosting a cocktail party Wednesday November 5th during London Online.  Feel free to come by.  You can hand out with some other Mark Logic XQuery gurus and get a free drink to lessen your disappointment.  Registration for this event is here.

Sorry for the confusion - it is, however, nice to have so much activity XQuery to be confused about.

Matt

Agile Publishing and XQuery

This week, Mark Logic is sponsoring a breakfast seminar on agile publishing. 

The speakers are great.  I've seen Howard Ratner, the CTO of Nature Publishing, speak a couple of times and he is entertaining and informative.  I've also met with David Worlock who is both an amazing font of knowledge and an engaging speaker.

And the topic is spot on:  Looking at product development through the agile lens yields many new possibilities for how you can both build and deliver new content products.

For me, it has always been about ways to speed up and simplify the product development process.  As the Technical Director at PC World Online I was on the receiving end of hundreds of small, 'can't you just' requests.  I hated saying no - these were the good ideas that made us innovate and if I said 'this isn't in the schedule' or 'maybe next year' then we'd get nowhere.  And in 1997 at the start of web publishing we had a LOT of ground to cover!

So we did a lot of small projects, we launched things in days rather than months and we made the most of our tools, pushing XML into databases the best we could and using Tcl (!!) and Vignette as a basic framework for rapidly developing many, simultaneous projects.  It was certainly agile (if with a little 'a') and that term applied to not just the tech team, but everyone working together to create new products.

These same trends are still around and are maybe even more important as publishers today need to invent, create new products and break the mold of the traditional products (which we were very busy inventing 10 years ago!).

And the tech teams are still getting those 'can't you just' questions . .  . but now you can use XQuery instead of all those clunky database/webcms/app layer toolsets.  XQuery is *the* native programming language for XML and XML is *the* model for content.  So with XQuery you don't spend a lot of time translating your content between tables, objects and outputs.  Instead, you just get right to work on those 'can't you just' questions.

And when you use an XQuery engine like MarkLogic Server,  things go even faster because you can load any content without up-front configuration and can perform any query on any part of the XML.  This turbo-charges the process but letting you get a-hold of some content and right away start writing your application (like we did in the first tutorial).

I often say that we wish we had XQuery back then.  Well, you do have XQuery now and what a difference it makes!

So come on out this week and explore the world of agile content products - here are the details:

The Agile Publishing Imperative:
Accelerate the Creation of Information Products

Thursday, November 8
8:00 am - 11:00 am
Four Seasons Hotel
Cost: Complimentary

Registration

Hope to see you there,

Matt