A Multi-Platform Application Programming Interface for the World Wide Web

Name P.M.Hounslow
Address Department of Computer Science, The University of Reading
Whiteknights, Reading, Berkshire, England, RG6 6AY
Affiliation The University of Reading
EMail Paul@triagonl.demon.co.uk

Abstract

This paper describes a design of a programming interface between application programs and a library of modules. These modules in turn provide an interface to distributed systems, such as the World Wide Web. Third party application developers will be able to use this library on a wide variety of computer platforms, including text only displays and keyboardless systems such as Kiosks.

Introduction

The design of the interface to the modules is based on an informal communication from Dave Raggett. As outlined here it allows access at the ‘C’ and ‘C++’ level, although support for other languages, such as Visual Basic, is to follow. It is the intention that each module is platform independent. It is the aim of this design that applications can incorporate the modules by different methods including; static linked libraries, dynamic linked libraries, controls and objects.

Previous work with similar aims and functionality includes:

The CERN WWW Library:
Although the CERN library [1] is large and monolithic, it nevertheless provides a good, if complex, starting point for developers wishing to provide access to the World Wide Web in their applications.

The design outlined in this is paper is based on a similar concept, with the aim of providing a more simple and modular system. The intention is to provide quick and easy initial access to distributed documents, with the opportunity of later expansion, to an application developer.

Java:
Sun Microsystems Java [2], an excellent programming language and environment for distributed systems. The World Wide Web application of which, HotJava [3] has amply proven the concept.

This API outlined here is designed to allow a programmer to include access to distributed information, using a language and tools that the programmer already has and knows well, on a system compiled for the target computer, not a virtual machine [4]. Thus it will work well and be, relatively, simple to integrate into existing applications, as well as new ones. For this reason it has been kept simple and flexible, consisting of a hierarchy of discrete modules performing distinct operations:

Protocol
The method used to obtain a ‘document’.
Tokenising
The action of producing the symbolic representation a document.
Translation
Conversion of the parsed document into the target format.
The primary level modules provide a consistent interface to these operations. The second level modules provide each of the primary modules a variety of protocols and documents. Thus, by adding further second level modules, support for new protocols (such as HTTP-NG [5]) and document formats (such as HTML-3 [6], or new graphic standards) is provided. In addition, revision updates allow easy modification to existing modules. This is particularly true with systems that support dynamic linking, as no recompilation need take place. It also simplifies testing, and allows earlier release dates of both the libraries and the applications dependent upon them. Furthermore, applications wishing to provide a sub-set of available protocols and document formats need not include redundant code. As some of these secondary modules will, by their nature, use similar or identical routines, this design incorporates a support module layer for common functions (such as network access). A diagram of the hierarchy of an application using this library is shown below.

[Diagram of module hierarchy]

Each module is designed to be used on its own, or in conjunction with one or more of the others. This system will allow a developer to quickly add WWW facilities to their application, yet still maintain control over its operation. This facilitates the development of additional or replacement modules for specific operations. It is envisaged that the development life cycle of an application using these modules would consist of the following stages:

Primary Modules

There are three top-level modules:

  1. Protocol support; to get a document from a given URI. [7]
  2. Tokenising operations; to produce a symbolic representation of a document.
  3. Translation routines; to convert the symbolic representation of a document into the required format (eg for display).

The first module provides a common and consistent interface for protocol support. It registers a list of protocols, and their associated call-back functions (supplied by the application programmer), against a list of the protocols it can support (from the protocol modules it has). It operates by taking the requested document URL [8] or URI [7] and passing it to the appropriate protocol module together with the destination call-back function. As the protocol module receives the document it is passed to the call-back function.

The second module provides a method of tokenising a given document into a form suitable for display formatting. Secondary modules provide parsing for different document types. Thus there may be GIF and JPG modules as well as text and HTML [9].

The last primary module takes the tokenised document and formats it as required by the application. For this project, this is a form suitable for presentation. However, it is expected that this may be replaced within an application by other modules for converting the document to some internal format (eg, for a Word Processor, to its document format). This is achieved by taking a tokenised document as its input (such as that produced by the previous primary module). For the presentation version it also requires current information about the display and will produce a formatted ‘Meta-File’ suitable for the display routines of the target architecture.

Secondary Modules

There are two varieties of Secondary Modules, those providing protocol operation and those providing document parsing. While a number of modules (for protocol and tokenising operations) are required, only the HTTP [10] protocol and the text and HTML [9] tokenising modules will be provided initially. The interface to these modules is beyond the scope of this paper.

Support Modules

To provide support for the above modules, and to prevent needless repetition of identical or similar sections of code, a number of support modules will be required. This is beyond the scope of this paper.

Implementation Strategy

The first implementation of this design is for Microsoft Windows, with the initial aim to produce a set of Dynamically Linked Libraries (DLLs) analogous to the Design Modules. As proof of concept, a simple Web Browser will also be produced. To support this, the first Protocol Module to be developed is an HTTP Module. To complement the HTTP Protocol Module, an HTML Tokenising Module is required. A single Support Module is to be produced to format fetched documents for display.

Primary Modules

Protocol

This module gets a requested ‘document’ (text, image or other) from a specified URI.

The module is initialised by using the function OpenProtocolLib to ‘register’ a list of supported protocols and their associated ‘call-back’ functions, one of which must be an error handler. The Call-back function is the same as the ParseDocument function of the Tokenising Module. However, by registering a ‘call-back’ function for each protocol, it is intended that all but the most simple applications provide some processing prior to calling this. Thus allowing the application to provide facilities not included in these libraries; such as; displaying the number of bytes received, providing time-out facilities, and so on.

A document is requested by passing its URI to the GetDocument function. This is then passed, to the appropriate Secondary module, or if there is no secondary module for that protocol, then an error is returned. The Document identified in the URI is sent to the appropriate call-back function as it is received. Any further errors are sent to the registered error handler function.

To stop a specified document fetch (GetDocument operation) the AbortDocument function is provided. This calls the registered error handler function with the results of the operation.

The function CloseProtocolLib is used when no further protocol operations will be required. It provides any cleaning required, aborts any current document fetches, releases any memory allocations and terminates the Protocol module.

The output of a this module is sent to the appropriate ‘call-back’ function as a ‘raw’ document (Doc_RawDocument). This holds; an octet (byte) stream containing the latest section of the document received, the document type, and a unique document identifier.

OpenProtocolLib

Parameters:
A Proto_List data structure of supported Protocols and their call-back functions, this must include an error handler. The call-back functions used by Proto_List require a Doc_RawDocument data structure, and a handle for identification.
Return Value:
A Boolean, TRUE if the protocols are supported, otherwise FALSE. FALSE will be returned if no error handler is identified.

The Protocols (in Proto_List) are identified by a text string, eg "HTTP", "FTP", or "ERROR". The latter identifying the error handling routine.

typedef char* Proto_Type;        // Protocol to use
typedef char* Doc_Type;          // MIME type document identification
typedef unsigned int Doc_Handle; // Application document reference

struct Doc_RawDocument {
   BYTE        *Data;            // Document data
   Doc_Type    Type;             // MIME type document identification
   Doc_Handle  Handle;           // Document Id
}

typedef BOOL (*Proto_CallBack)(Doc_RawDocument*, Doc_Handle);

struct Proto_List {
   Proto_List     *Next;         // Pointer to next protocol
   Proto_Type     Type;          // Requested Protocol
   Proto_CallBack CallBack;      // Document processing function
}

BOOL OpenProtocolLib(Proto_List ProtoList);

For example, an MS Windows program might use the following code fragment:

ProtoList = new Proto_List;    // Create list.
PL = Proto_List;

while(GetNextProtocol(PL->Type, PL->CallBack)) {
   PL->Next = new Proto_List;
   PL = PL->Next;
}
PL->Type = "Error";            // Error type
PL->CallBack = &ErrorHandler;  // Error Call-back.
PL->Next = NULL;
if(!OpenProtocolLib(ProtoList)) {
   MessageBox(hWnd, "Unable to Open Protocol Library", NULL, 
              MB_OK | MB_ICONEXCLAMATION);
   return FALSE;
}

GetDocument

Parameters:
The URI of the document to fetch.

The ‘Handle’ of an ‘owner’ document. If there is no ‘owner’ document, then this is set to 0.

Return Value:
An integer, a unique identifier (Doc_Handle) for the document being acquired, otherwise NULL if the protocol is not supported.
UINT GetDocument(char *URI, Doc_Handle hDoc);

For example, an MS Windows program might use the following code fragment:

hDoc = GetDocument(szURI, hOwner);
if(!hDoc) {
   char szMsg[MESSAGE_LEN+1];
   wsprintf(szMsg, “Unsupported protocol: %s”, szURI);
   MessageBox(hWnd, szMsg, NULL, MB_OK | MB_ICONEXCLAMATION);
   return FALSE;
}

AbortDocument

Parameters:
The ‘Handle’, a unique identifier (Doc_Handle) for the document, currently being acquired, for which all protocol operations are to cease.
Return Value:
None.
void AbortDocument(Doc_Handle);

CloseProtocolLib

Parameters:
None.
Return Value:
None.
void CloseProtocolLib(void);

Tokenising

This module provides routines for processing a document. For an HTML document this will be an HTML parser, for JPG it will be a conversion routine (to a native graphics format).

The module is initialised by using the function OpenTokenLib to ‘register’ a list of supported documents (identified by MIME [11, 12], or similar type) and their associated ‘call-back’ functions, one of which must be an error handler. The Call-back function is the same as the FormatDocument function of the Translation Module. It also requires a hook for requesting further documents. Thus enabling the acquisition of compound documents (eg Web pages with in-line graphics). This may be the GetDocument function of the Protocol Module. As a ‘call-back’ function is registered for each document type, the application may provide facilities not included in these libraries. Thus all but the most simple can do some processing prior to Translation.

A document is tokenised by passing it in its raw state to the ParseDocument function. This is then passed, to the appropriate Secondary module, or if there is no secondary module for that document type, then an error is returned. The token stream for that document is sent to the appropriate call-back function after translation. Any other documents required by that the document being tokenised (such as in-line graphics in HTML) are fetched using the supplied ‘call-back’ function. Any further errors are sent to the registered error handler function.

The function CloseTokenLib is used when no further document tokenising operations will be required. This operation provides any cleaning required, releases any memory allocations and terminates the Tokenising module.

The output of this module is a tokenised representation of the document (Token_Document). This is a tree like structure based around two entities, a Tag and a Container. The Tag structure represents an element of the document. Each element holds the data associated with it in a Container structure. A Tag structure can only hold one Container. However, a Container may hold a list of Tag’s.

The basis of all the document elements is the TAG_Unknown tag. This holds a Container, and a pointer to the next Tag in the list. The basis of all the Container’s is the Container_Empty. This contains no data.

[Diagram of Base Token Elements]

The Tag classes are based upon a super-set of the elements of an HTML document as enumerated in Token_TAG. The Container classes are based upon the data types enumerated in Token_ENTITY.

An example of the Tag and Container classes are given below:

typedef enum {UNKNOWN, HTML, HEAD, BODY, ...} Token_TAG;
typedef enum {EMPTY, TAG, TEXT, IMAGE, ...} Token_ENTITY;

struct TAG_Unknown {
   Token_TAG       Tag;   // Document element type.
   Unknown         *Next; // Next element in the list.
   Container_Empty *Data; // The data for this element.
   TAG_Unknown() : Next(NULL), Data(NULL) { Tag = UNKNOWN);};
};

struct Container_Empty {
   Token_ENTITY Entity;   // This contains data of type.
   Container_Empty() {Entity = EMPTY};
};

struct Token_Document {
   TAG_Unknown *Document; // Tokenised document tree
   Doc_Handle    Handle;  // Document identifier.
   Token_Document(TAG_Unknown *Doc, Doc_Handle h) 
      : Document(Doc), Handle(h) {};
};

struct Container_TAG : public Container_Empty {
   TAG_Unknown *Token;    // List of document elements
   Container_TAG(TAG_Unknown *T) : TAG_Unknown(T) {};
}
struct TAG_HTML : public TAG_Unknown {
   TAG_HTML(Container_TAG *D) : Tag(HTML), Next(NULL), Data(D) {};
}

An example of a tokenised HTML document is shown below.

[Diagram of Example Tokanised HTML Document]

OpenTokenLib

Parameters:
A Doc_List data structure of supported Documents and their call-back functions. This list must include an error handler. The Documents (in Doc_List) are identified by a, MIME [11, 12] type text string.

A pointer to a function (of type Token_CallBack) for requesting further documents.
Return Value:
A Boolean, TRUE if the documents types are supported, otherwise FALSE. FALSE will be returned if no error handler is identified.
typedef BOOL (*Token_CallBack)(Token_Document*);

struct Doc_List {
   Doc_List       *Next;
   Doc_Type       Type;
   Token_CallBack CallBack;
}

BOOL OpenTokenLib(Doc_List DocList, 
                  UINT (*GetDocument)(char *URI, Doc_Handle hDoc));

For example, an MS Windows program might use the following code fragment:

DocList = new Doc_List;    // Create list.
DL = DocList;

while(GetNextProtocol(DL->Type, DL->CallBack)) {
   DL->Next = new Doc_List;
   Dl = DL->Next;
}
DL->Type = "Error";            // Error type
DL->CallBack = &ErrorHandler;  // Error Call-back.
DL->Next = NULL;

if(!OpenTokenLib(DocList, &GetDocument)) {
   MessageBox(hWnd, “Unable to Open Tokenising Library”, NULL, 
              MB_OK | MB_ICONEXCLAMATION);
   return FALSE;
}

ParseDocument

Parameters:
A raw document (Doc_RawDocument).
Return Value:
A Boolean, TRUE if the document type is supported, otherwise FALSE.
BOOL ParseDocument(Doc_RawDocument*);

CloseTokenLib

Parameters:
None.
Return Value:
None.
void CloseTokenLib(void);

Translation

This set of operations is responsible for converting the results of the Tokenising modules into a form required by the application. The initial version of this module will display the results on the screen or in a window. As versions of this module may have to provide very different translation operations, it is beyond the scope of this paper to fully specify the interface. However, the following functions must exist: OpenTranslationLib, ProcessDocument, and CloseTranslationLib.

OpenTranslationLib

Parameters:
These are beyond the scope of this paper.
Return Value:
A Boolean, TRUE if the module was initialised correctly, otherwise FALSE.
BOOL OpenTranslationLib(...);

ProcessDocument

Parameters:
A tokenised document (Token_Document) for translation.
Return Value:
A Boolean, TRUE if the document was processed correctly, otherwise FALSE.
BOOL ProcessDocument(Token_Document*);

CloseTranslationLib

Parameters:
None.
Return Value:
None.
void CloseTokenLib(void);

Conclusion

The design for a modular programming interface to the World Wide Web has been analysed. The objective is to develop a library of interface code that gives programs not already associated with networks a simple means to access distributed systems.

Due the nature of this design and the World Wide Web itself, it is expected that some of the above modules may themselves be documents on the World Wide Web. This allows for many possibilities from easy provision of support for new documents and protocols, to very complex dynamic applications and composite documents.

References

[1] H. Frystyk, H. W. Lie, Towards a Uniform Library of Common Code: A Presentation of the CERN World-Wide Web Library, Presented at The Second Annual World Wide Web Conference: October 1994, Chicago USA, Available from: <URL:http://www.w3.org/hypertext/WWW/Library/User/Paper/LibraryPaper.html>
[2] Sun Microsystems The Java Language: A White Paper, Available from: <URL:http://java.sun.com/1.0alpha2/doc/overview/java/index.html>
[3] Sun Microsystems The HotJava Browser: A White Paper, Available from: <URL:http://java.sun.com/1.0alpha2/doc/overview/hotjava/index.html>
[4] Sun Microsystems, The Java Virtual Machine Specification, Available from: <URL:http://java.sun.com/1.0alpha2/doc/vmspec/vmspec_1.html>
[5] S. Spero, Progress on HTTP-NG, UNC Sunsite/EIT, Avalable from: <URL:http://www.w3.org/hypertext/WWW/Protocols/HTTP-NG/http-ng-status.html>
[6] D. Raggett, HyperText Markup Language Specification Version 3.0, Work in Progress (draft-ietf-html-specv3-00.txt), March 1995.
[7] T. Berners-Lee. Universal Resource Identifiers in WWW: A Unifying Syntax for the Expression of Names and Addresses of Objects on the Network as used in the World-Wide Web. RFC 1630, CERN, June 1994.
[8] T. Berners-Lee, L. Masinter, and M. McCahill. Uniform Resource Locators (URL) RFC 1738, CERN, Xerox PARC, University of Minnesota, October 1994.
[9] T. Berners-Lee and D. Connolly. HyperText Markup Language Specification - 2.0, Work in Progress (draft-ietf-html-spec-01.txt), CERN, HaL Computer Systems, February 1995.
[10] T. Berners-Lee, Hypertext Transfer Protocol (HTTP/1.0), Work in Progress (draft-ietf-http-v10-spec-00.txt), CERN, March 1995.
[11] N. Borenstein and N. Freed. MIME (Multipurpose Internet Mail Extensions) Part One: Mechanisms for Specifying and Describing the Format of Internet Message Bodies RFC 1521, Bellcore, Innosoft, September 1993.
[12] K. Moore. MIME (Multipurpose Internet Mail Extensions) Part Two: Message Header Extensions for Non-ASCII Text RFC 1522, University of Tennessee, September 1993.

Paul Hounslow 3 August 1997

Paul@triagonl.demon.co.uk