This paper attempts to give an overview of the current understanding of what XML can be used for by presenting six use cases, some real and some imaginary. The paper argues that the main benefits of XML over SGML are the widespread support for XML and that it will be (and partly is already) supported in web browsers.
This paper was presented at the SGML/XML Finland '98 conference in Jyväskylä, on the 8th of October 1998 and later at an SGML Users Group meeting in Oslo on the 14th of January 1999.
XML is really SGML, SGML without all the parts that are hard to implement in software products. This is the only technical difference between SGML and XML, but there is one other important difference as well: XML is designed to be used on the web and supported by web tools such as browsers. This means that XML will see widespread use on the web, something SGML never achieved.
This deployment on the web opens up many new opportunities for using XML that were never available with SGML, and these are still being imagined and understood. This paper attempts to give an overview of the current understanding of the new kinds of uses that XML can lend itself to. The reader should bear in mind that this understanding is something that is still developing.
The use cases in this paper are chosen in order to demonstrate the advantages XML has over SGML in some kinds of applications. This does not mean that XML is better than SGML, or even that use of XML excludes use of SGML. In some of these use cases SGML could be used as the source format and XML just as the web publishing format. There are also cases where XML is insufficient, and full SGML is required, but these do not belong in this paper, and so will not be covered here.
XSA (XML Software Autoupdate) is a very simple XML application designed to solve a simple problem: Most authors of freeware that is distributed over the net have their programs registered in many different software indexes. When the number of products in each index numbers in the hundreds and thousands, keeping the indexes up to date by checking all products in it for new releases is simply too much work for most index maintainers.
The program authors/vendors are usually unaware of many of the places in which their product is listed, and the result of this is that most software indices are out of date, since new releases of the listed products have appeared after the products were listed and described.
What is needed is an automatic way to alert index maintainers to new releases, and the irony of the situation is that all the information required to keep track of new releases is in the product home pages already, but in such a form that it cannot be reliably extracted from them. This is what the XSA system attempts to change, by encoding the information in XML.
Note: XSA is still under development at the time of writing (19980902), and so may have changed before this paper is published.
An XSA document contains vendor information (name, email and home page) as well as information about the vendors products (name, version, home page and an optional text field giving the changes since the last release). Products in the document have unique and stable IDs, which enables software to tell the products apart.
In the simplest XSA model product vendors and authors simply write an XSA document (slightly helped by a web form) and put it on the net somewhere. Index maintainers are then informed of the location of the XSA document, either by discovering a link to it from the product home page or the document can be registered via a web form.
The product maintainer will then update the document whenever new releases (or address/name changes) occur. These changes can then be discovered by index maintainers when they run their XSA monitoring software. Very likely this will be done at regular intervals, such as once a week. (This is what is called polling, where clients check for new information at intervals. The alternative is push, where the index maintainers run servers that are informed whenever new information is available. XSA can be used in both models.)
A simple Java SDK (Software Development Kit) has been developed that index maintainers can use to monitor XSA documents. This SDK comes with a simple client that just prints out a log of changes since the last time it was run, but there is also an API (Application Programming Interface) that can be used to develop other clients and integrate the SDK with existing systems.
A different XSA monitoring system might be a service to the public that allows users to register their interest in a set of products and then checks these, emailing the user whenever a change considered interesting by the user occurs. It is also conceivable that XSA could be used by auto-update software that could monitor XSA documents for new versions and automatically download and install the new version. (Such systems already exist for the Linux installation manager RPM (RedHat Package Manager), but they do not use XSA.)
However, the XSA model described here requires product vendors to maintain their product information in two places, that is, on the product home page and in the XSA document. It would be better if the information could somehow be extracted from the product home page directly. The difficulty with this is that the home page might be stored in HTML (which does not have elements for this purpose) or in some unknown XML DTD.
An XML architecture might solve this problem, by mapping the product home page to an XSA-like DTD. This would allow the XSA SDK to extract the information from product home pages regardless of the DTD used, and remove the need for separate XSA documents. (Products that can do architectural forms processing in XML already exist.)
XSA is one of a kind of system that I think will become common in the future: a system that extracts information from web documents and uses it intelligently. In these cases the same information will often be used in different ways and in many cases architectural forms are likely to be used to allow information suppliers to use whichever DTD they prefer. (Namespaces could also be used for this purpose, but is in my opinion an inferior alternative.)
What is new about this kind of system is that it is web-based, which was not really an alternative for SGML systems before the arrival of XML. Stand-alone documents might be encoded in SGML, but web pages would in reality not be, since web tools in general do not support SGML.
Also, an SDK like the XSA one could not really be developed in a platform-independent and easily deployable manner, since SGML parsers are not available in cross-platform languages like Perl, Python or Java, but would have to be installed separately.
Every year all Norwegians have to submit a tax form stating their income and holdings for the past year. Keeping track of this mountain of paper forms is a formidable task, and the Norwegian tax department is looking at ways of solving this electronically. What they currently have in mind is a model where forms can be submitted by email, via a web interface, on paper, via X.25 and possibly also by other means. (The tax department is planning to use EDIFACT, for historical reasons. Attempts are being made to make them use XML instead.)
Submitted forms will have to be validated to check that they are filled in correctly and then passed on to several different applications inside the tax department. XML is a natural choice as a data format for the forms, since a well-documented DTD can accurately express the syntax and semantics of the format.
However, this is only the most obvious way in which XML can be used in this project. Tax laws change very frequently, which means that the forms themselves are also subject to frequent change. This means that the user interface code for the form would have to change, as would the validating application. However, changing application code is complex, time-consuming and expensive.
A better way to handle the problem might be to express the form itself as an XML document that described all the fields, the text in the form and the relationships between the fields. The user interface code for web submission could then use this information in a Java applet to set up the user interface correctly, and the validation application could use it to validate received information. Changes to the forms could then be done by just modifying the XML document, without changing any of the application code.
Some of the constraints that might conceivably be expressed in an XML document are:
that field X is the sum of fields W, Y and Z
that field X should contain Y percent of the amount in field Z
that the value of field X should be between Y and Z
that fields X and Y should contain the same value
that if the value in field X is Y, then fields W-Z should not be filled in
These should all be easily expressible in XML, and the resulting documents should be simple enough that non-programmers can modify them when needed.
The really interesting thing about this application is the use of XML to express the relationships between forms and the use of this information to control both the user interface and a validating application.
SGML might have been used for this purpose, but the validating software will use CORBA (Common Object Request Broker Architecture) and would probably be written in Java. Current SGML parsers are not easily available from Java and are not CORBA-aware, which means that using SGML would have been much more difficult.
For the graphical interface SGML would have been out of the question for use in the applet, both because an SGML parser would be much too large and because no SGML parsers exist in Java, while XML parsers exist in Java that only take 15k in compressed form.
The main benefit XML brings to this application over that of SGML is the availability of tools and the ease with which XML tools can be used by new applications. This will, I think, lead to widespread use of XML as a format for many kinds of input to applications, something that was not feasible with SGML, because of the complexity and scarcity of SGML parsers.
RML (Recipe Markup Language) is a fictive language for marking up ordinary food recipes that allows for the description of ingredients (kind, amount and alternatives), the preparation steps (with times, temperatures and alternatives) and metadata (descriptive text, country of origin, preparation time, difficulty etc).
A web site offering a collection of recipes could then mark these up with RML, something that would open for entirely new possibilites unavailable with recipes stored in HTML. The most obvious would be a search facility where users could search for, say, simple Italian soups that can be prepared in less than an hour.
This, however, could also be done just as well with a simple database solution. The advantage over a database solution is that since the recipe is in XML the details about the ingredients are available and could be put to use. This would allow users of the service to enter the ingredients available to them, and subsequent searches would then only return recipes that could be prepared with the ingredients available to them.
And there's no need to stop there. Add databases of information about the nutritional values of different kinds of ingredients and maybe even the prices of ingredients in different countries and the service could suddenly be even more useful. Now the user could also search for recipes with less than 5000 calories and a price below 50 USD.
Smart site developers could take this even further and personalize the service. Users could then register with the service to have it set up monthly menues according to their preferences (nutritional contents, price, availability, time to prepare etc), and shopping lists could also be generated. (Very likely one could ask rather high prices for tailored advertisements in the shopping list. There are also some ethical issues here.)
When all these intelligent data are available on the server it would be a shame not to allow the client to use them as well. With a Java applet this suddenly becomes possible: The browser could display the recipe using a style sheet and the user could then tell the applet that 'I have a microwave oven', 'I will use this ingredient alternative instead of that' and so on. The applet could then change the display of the recipe to not show the alternatives in the ingredient list and preparation steps that the user had excluded.
The applet could also also suggest spices, calculate the nutritional content of the combination of ingredients and the time to prepare the dish with those ingredients, updating the calculations as the user changes the choice of ingredients.
And since the applet knows all the steps in the preparation of the recipe it could (if it were kept informed about progress) help the user keep track of the different steps to be performed, warning the user about ovens to be pre-heated, food to be removed etc.
If we try to look a little into the future it becomes likely that most househould appliances such as ovens and refrigerators will have their own IP addresses and be available on the house intranet. There are efforts underway to enable these appliances to run Java software. This might enable the applet to actually know the contents of the refrigerator, relieving the user of the burden of having to type it in.
During preparation the applet could also use the intranet to actually control ovens and other cooking appliances to help the user. It would probably be a very good idea to consult the user before actually taking action, though, in order to avoid unforeseen problems.
The interesting thing about RML is that it is an example of what kind of new web sites become possible once XML technology is widely deployed on the web. Not only can web sites have more advanced functionality on the server side, but since the XML can be downloaded to the client with Java applets the web sites can also extend the functionality on the client side far beyond what is feasible today.
Similar sites can be developed for other kinds of structured contents, such as all kinds of electronic components, laws, radio and TV programs etc. The list is potentially endless, and nobody really knows what kinds of innovative applications will spring up in the future.
Genealogical research is today a popular hobby, and many people have made sizeable web pages about their research and exchange data with fellow enthusiasts over the net. GedML is a an XML DTD for marking up genealogical data, proposed by Michael Kay. It is mainly intended for exchange between applications and users.
A binary exchange format called GEDCOM already exists, so why make an XML DTD for this? There are a couple of obvious advantages: a well-documented DTD serves as a readable and formal machine-useable syntax description. This means that there is unlikely to be many differences in interpretation of the format between GedML implementors. Furthermore, GedML is easily parsed, since XML parsers are available for most languages and platforms. These things combine to make GedML very well suited as an exchange format and gives it several advantages over GEDCOM.
However, the really major benefit to using XML as a data format is that the companion standards of XML can also be used on GedML with only little extra effort. In other words, GedML documents can have linking, be linked into, be automatically transformed/displayed with XSL (eXtensible Style Language), manipulated through the DOM (Document Object Model) etc
Once data are stored in GedML they can be used in many different ways on the web. A package of software could be developed that could be used to add features to any web site that published GedML pages, such as searching for a specific person or family or browsing a family tree with a Java applet. One could also use style sheets for browsing, but with large amounts of data a navigational Java applet combined with the style sheet would probably be much more useful.
Another function that could be added is one that would allow users to import a branch of a GedML family tree into their own GedML base and so extend it.
If linking features were added to GedML, using XLink (XML Linking Language) and XPointer (XML Pointer Language), users could collaborate on maintaining their genealogical data, each concentrating on a separate branch of an overall family tree. Each could then have a separate GedML document, but link into the documents of the others. (Linking has been mentioned as a possible GedML feature, but is not currently implemented.)
Browsing and searching software could then follow these links and make the separate GedML files on different sites appear as one vast web of genealogical data. It is conceivable that the eventual "GedML web" could become quite large.
As this example should show, XML is excellently suited for exchange formats, because it provides a high-level syntax description and because parsers are widely available. XML is also very well suited for data formats, since XML's companion standards can easily be used on XML data formats. The last benefit brought by XML is easy integration with web browsers.
SGML has many of the same advantages over the usual binary formats, but suffers from a lack of parsers, the size of parsers and from not being available in web tools.
For years scientists all over the world have struggled to present mathematical equations online, and the current best solution is to use a TeX-to-HTML converter that stores the equations as images. This is of course resource-intensive and not fully satisfactory.
Partly to solve this problem the W3C (World Wide Web Consortium) developed MathML (Mathematical Markup Language), a markup language for mathematical equations and formulae. MathML is a completed specification, and is already supported by many software products.
Since MathML allows for detailed description of mathematical formulae this enables XSL style sheets to display MathML formulae in browsers, which is much faster (and adapts better to different viewing situations) than having to download one image for each formula.
Editors (Amaya, EzMath and MathType) are already available for editing MathML in a graphical interface. The Maple computer algebra system can import, export and evaluate MathML, later versions will also be able to display MathML. Several math publishing systems already support MathML and more plan to do so in the future. The IBM techexplorer technical document viewer can already display MathML.
WebEQ is a collection of Java applets (and an API) that can display equations both in standard mathematical notation and as graphs. By using the API developers can develop new MathML applets and applications for their own purposes.
What all these developments mean is that you can write MathML equations in a graphical editor, include them in your web pages, use them in your technical documents, import them into your computer algebra system, publish them and even develop your own applets and applications that work with MathML.
This last possibility opens for new uses in fields like education, where online courses can include mathematical equations and show them as graphs and allow the readers to tweak the display of the graphs as they wish. Applets can also accept numerical values from the user and compute results on the fly.
In fact, a simpler version of the MathType equation editor mentioned above is used as the equation editor in many different software packages such as Microsoft Office, Corel WordPerfect, ClarisWorks and Nisus Writer. If this simpler version also gets MathML support users will be able to create MathML equations in their usual desktop applications.
MathML is yet another example of XML being used where SGML cannot be used because it requires too large, scarce and complex tools for it to be feasible. It is also a good example of the new possibilities that emerge when structured information becomes available on the client side.
RDF (Resource Description Framework) is a W3C Proposed Recommendation describing a standard for representing information about web resources, based on XML. RDF is a general framework that can be adapted to many purposes, but only one possible use will be outlined here.
One interesting purpose it could be used for is site maps, that is, making a single document for a site that describes the different resources that make up the site, their relationships and their types (tutorial, presentation, link page, home page etc). A site map would then be a list of documents and their titles and types and with relationships stating that document A is a tutorial, document B is the table of contents of the same tutorial, while document C is section 1 etc
This would enable searches for "XML tutorial" and would enable one to find tutorials that mentioned XML in some way, instead of a random collection of documents that mentioned the words XML and tutorial.
Today searches often return hits such as slides 31, 40 and 67 of a presentation, or chapters 3 and 5 of a tutorial etc. Using a site map the search engine could know that these belonged to a greater whole and group them as one hit, referring the user to the top page of the presentation or tutorial. (And also informing the user that the hit is a presentation or tutorial.)
Todays browsers have rather primitive navigational capabilities, mostly because they know so little about the web they browse. The current browsing model is one where the current document has a number of links leading out of it, and is one in a list of the documents the user has seen. The user can either choose a link out of the current document, or move backwards or forwards in the list of seen documents.
With site maps the browser would know much more about its surroundings because the it would be able to distinguish between site structure links, cross-links and links out of the site. This would enable more useful navigational models, such as a view of the current site as a tree structure, with brief (and configurable) descriptions of each resource. The user would then be able to see the current position in the site and to jump to other nodes in the site directly.
This would enable a user jumping into a site from a search engine to an article in an online magazine to see that the article was one of 15 articles in the February edition. The user would then be able to jump to other articles in the same edition, another edition, the search page or the magazine home page.
This browsing model would probably change the way web pages are laid out and maintained, since many web sites today have a primitive version of this site map on each page.
Site maps are yet another example of what happens when structured data are no longer confined to the server side, but can be downloaded by user agents (browsers, robots etc) and interpreted on the client side. The result is that several different ways of using the information suddenly present themselves, some of which are quite revolutionary.
As I said in the introduction, XML is a subset of SGML, with fewer features than SGML, but paradoxically this is what enables it to do things SGML never could. Section 2.1 of Lisp guru Richard P. Gabriels article Lisp: Good News, Bad News, How to Win Big, has become famous for its analysis of this paradox, which he calls "The Rise of Worse is Better".
His argument is essentially that simple systems require less effort to implement, less resources than complex, but functionally superior systems and that this will lead to support for the simple systems being much more widespread than that for the complex systems. He writes:
First, the right thing needs to be designed. Then its implementation needs to be designed. Finally it is implemented. Because it is the right thing, it has nearly 100% of desired functionality, and implementation simplicity was never a concern so it takes a long time to implement. It is large and complex. It requires complex tools to use properly. The last 20% takes 80% of the effort, and so the right thing takes a long time to get out, and it only runs satisfactorily on the most sophisticated hardware.
He was talking about Common Lisp, but he might as well have described SGML (except for the hardware problem). It has everything you want, but the problem is that it also has a host of things you probably don't need and that require a lot of effort to implement. This is why there are only a handful SGML parsers twelve years after the introduction of the standard (only one of which implements the entire standard).
In contrast, XML is simple, provides 80% of what is wanted, and unsurprisingly more than 20 parsers are available six months after the release of the XML recommendation. This is exactly as Gabriel predicted for what he calls the "worse is better"-philosophy that XML is an example of.
In other words: XML and SGML are sibling standards that serve different user communities and usage contexts. XML is suitable for online use and for low-end systems, while SGML is suitable for local high-end systems. Since neither can cover the same needs as the other, it is likely that both will survive, and indeed thrive on the successes of its sibling.
Until the arrival of XML, structured data that could be processed and used (as opposed to just displayed) were only available on the server side of the web. This meant that all intelligent applications had to be on the server, while the client was just a display engine.
All the applications that let you search a set of data or register information into a set of data, both of which are very widespread on the web today, are examples of this. The data are on the server, and the client has to send a request to the server whenever the view of the data is to change. Java applets are mostly decorative, or allow the user to work with information supplied by herself.
This is what XML changes: suddenly structured information can be downloaded into the client and applets can work with it in an intelligent fashion in the client (browser). This enables entirely new ways to use the information, since user interfaces can be made much more advanced. (RML, MathML, GedML and site maps are examples of this.)
The last implication of this is that other web clients than browsers can act on this information as well, allowing for systems that extract information from web sites and act intelligently on it, such as XSA and the use of site maps in searching.
Since XML is so simple and is likely to be supported more or less everywhere a de facto standard for exchange formats for structured information and storage formats for simple data. This will enable the development of many new exchange formats, of which GedML is probably just the first example. In fact, as I wrote this article today XBEL (XML Bookmark Exchange Language) was conceived, developed and already supports the MSIE 5.0 bookmark format. Support for Opera and Netscape is expected over the weekend.
And not only is XML an excellent choice for exchange formats, most applications also store configuration files and simple files with small collections of data. Since XML support will be (is) so widespread, it is likely that many of these formats will also use XML in the future. The tax forms example is probably just the first in a long line of such formats.
Thanks to Chris Maden for providing the DSSSL style sheet used for converting this document to HTML, Steve Pepper for giving me the opportunity to speak at the conference and Silvester Pozarnik for useful criticism.
Lars Marius Garshol is an employee of STEP Infotek where he has been working on (among other things) the SigmaLink document database server and the Reference Works Module specification.
He is also working on a MSc thesis on publishing systems at the University of Oslo, and is the author of a validating XML parser, the translation of the SAX interface to Python and an XPointer implementation. He is a regular contributor to the xml-dev mailing list.