Mozilla RDF / Z39.50 Integration Project

This is a project to investigate the integration of Z39.50 search capabilities with Mozilla's RDF-based information management environment.

Aims

• Make Z39.50 data sources accessible for searching from within Mozilla
• Find an RDF representation of Z39.50 attribute sets

There are thousands of networked Z39.50 servers in existence already. It should be possible to identify some mechanism whereby the Mozilla user interface allows people to send queries to these servers and have the resulting records appear within the standard Mozilla bookmarks/sitemaps interface.

Z39.50 Background

ANSI/NISO Z39.50 is a communications protocol for information retrieval in a client/server environment. Widely used in bibliographic and digital library applications, Z39.50 represents a significant body of experience relating to bibliographic data retrieval and the specification of interoperable semantics for community-specific metadata attribute sets. The Z39.50 Integration Project outlined here is intended to identify appropriate mechanisms that will allow us to exploit Z39.50 data sources and metadata attribute sets from within Mozilla's RDF-based 'Aurora' environment.

Getting Started

This page is intended to provide a useful set of resources from which contributors from the Z39.50, RDF and Mozilla communities can make a start on addressing common problems. Z39.50 can be rather complex for those unfamiliar with the protocol, but there are many developers out there who have considerable experience in this area. What we really need is a good introductory document aimed at outside contributors wanting to hack new functionality into Mozilla in the most sensible way possible. Some of the docs linked from the Mozilla RDF pages make a start in this direction. A good way to get started is simply to download one of the early-release copies of Mozilla and take a look at the way it works.

For Z39.50 developers unfamiliar with Mozilla...

If you understand Z39.50, you'll probably find this easy... The entire Mozilla UI is built from "XUL" files, Javascript snippets and calls to XP-COM interfaces for services such as RDF query and aggregation. Read up on the Cross Platform front end, which describes how the user interface is specified in XUL. XUL is an XML application that allows interfaces to be built using XML, HTML, Javascript and RDF. The XUL Template Reference describes in some detail how XUL widgets (trees, menus) are populated by querying RDF data graphs. You should read up on the formal RDF specs at the W3C RDF home page and explore the various RDF-in-Mozilla links on this site. In Mozilla, most interesting data sources manifest themselves to the user interface through the RDF graph abstraction. There are a number of options to explore regarding how we might get data from a Z39.50 server into such a graph and from there into XUL-based user interface elements. See the Datasource HOWTO for a step by step guide for implementors of new RDF datasources.

The basic recipe is as follows: we compile our component (on as many platforms as possible), implementing the RDF DataSource APIs and any additional XP-COM APIs we care to define (perhaps digging out some of the IDL for Z39.50 references floating around). We drop the files into the 'components/' directory of Mozilla and when the browser starts up, it'll inspect the new component, which will dutifully describe itself so that it can show up in the Javascript, RDF and XUL world from which Mozilla is built (see also the Aug'99 article in WebReview.com for a high level overview of the Mozilla architecture).

Here's the basic approach to calling using components from Javascript. They each have a name (eg. 'component://z3950service') and satisfy one or more scriptable interfaces. The fictitious snippet below shows the initialisation of a Z39.50 object that can be treated as if it were an RDF graph, while also offering a more Z39.50 native interface.

 var z3950 = Components.classes["component://z3950service"].createInstance();
 z3950 = z3950.QueryInterface(Components.interfaces.nsIRDFDataSource);
 zapi = z3950.QueryInterface(Components.interfaces.IZ3950Service);

What happens next? Firstly, it is important that people don't go away and expend loads of effort working on something that may not fit in with the bigger picture sketched in the Mozilla development roadmap and on the Mozilla RDF pages. The best way to ensure this doesn't happen is to float ideas on the mozilla-rdf forum (see the Community section for details on accessing this as newsgroup or mailing list).

Prototypes

This should probably be what happens next! It should be relatively simple to put something together (eg. CGI script Z39.50 client outputting an XML/RDF file) which shows what a Mozilla Z39.50 client might look like. Once this is done, it'll be easier to appreciate the issues involved and figure out what we'd want from a full Z39.50 datasource (see below). For example: do we want to be able to store query results as bookmarks? Can we simply use the Dublin Core RDF vocabulary to represent these data structures?

Notealso that the M9 Mozilla milestone includes some non-Z39.50 internet search functionality (based on 'screenscraping' results from HTML pages returned by search engine queries).

Roadmap for a Z39.50 RDF datasource

If native client-side support for Z39.50 is to be integrated into Mozilla, we'll need to find code that implements a Z39.50 client and can be wrappped up as an RDF data source in Mozilla. It looks as is we can use the Yaz toolkit from IndexData, which will give us a pretty solid base for the Z39.50 client code (Yaz ships under a liberal license whose main constraints concern attribution statements and 'no warranty implied'). The main substantive piece of work is likely to be gluing Yaz to the RDF APIs and thinking through the different ways in which the RDF and Z39.50 data models might be related.

Open Issues

This should evolve into a more structured issues list on a separate page; for now, a freeform brainstorm seems more appropriate.

What is Mozilla RDF's notion of a 'searchable thing'? What other networked search protocols will also be supported in Mozilla? Is it feasible to consider adding full Z39.50 support into the client (eg. through contrib'd library?) or will tunnelling over HTTP (via XML/RDF or otherwise) prove to be a more realistic option? What constraints are there on code contributed by 3rd parties? How exactly do we set about hooking up externally authored libraries (eg. Yaz) as XP-COM plugin datasources?

Discussion

See the main Mozilla RDF pages for links to relevant lists/newsgroups.

Resources

Z39.50 reading

• Z39.50 Maintainance Agency at the Library of Congress
• Z39.50 Core Services - a nice overview of the protocol
• The Z39.50 Information Retrieval Standard (D-Lib article by Clifford Lynch)
• The Z39.50 specification (the spec itself)
• Example datasources - GILS Locator Records describing in XML a large number of Z39.50 servers
• a draft proposal from Eliot Christian that provides an RDF Schema for the GILS locator records (see also an opinion piece on Z39.50 and the RDF Semantic Web...)
• The Yaz Z39.50 programmer's toolkit from IndexData (candidate code for use with Mozilla)
• Profile for Z39.50 over HTTP (draft)
• Z39.50 Attribute Architecture
• The XER encoding rules (proposal to replace the traditional BER encoding of ASN.1 data structures with an XML representation)
• More Z39.50 related resources for background reading.
• See also Query Language Issues in a Distributed Indexing Environment (discussion paper from QL'98 workshop)

Mozilla reading

• Mozilla development roadmap
• Getting involved...
• RDF Datasource HOWTO
• XP-COM home page
• Roadmap for XPCOM, XP-Connect, XPTCall and XPIDL
• it might be worth having a look at the directory / LDAP code in Mozilla, since the LDAP libraries provide tools to handle BER encoded messages.

Disclaimer

This is an exploratory project, and an experiment to see whether we can make it easy for the Z39.50 community to contribute their expertise to the search environment under construction for Mozilla. It does not constitute any commitment to ship Z code with the '5.0' release, nor a promise that a great deal of engineering resources can be poured into this problem. This probably doesn't need stating, since Mozilla isn't NSCP/AOL. Basically, it's likely that some XML/RDF-based client/server metadata search functionality will appear in the Mozilla browser. Hopefully, this will be informed by and compatible with existing Z39.50 implementations. But we need your help...