METS documents which conform to this profile may use all of the same controlled vocabularies as the "ECHO Dep Generic METS Profile for Preservation and Digital Repository Interoperability" parent profile.

Used with the TYPE attribute of div elements for web captures.

Used with the USE attribute of file elements for web captures.

METS documents which conform to this profile must follow all of the structural requirements for the METS Root Element as the "ECHO Dep Generic METS Profile for Preservation and Digital Repository Interoperability" parent profile.

METS documents which conform to this profile must follow all of the structural requirements for the METS Header as the "ECHO Dep Generic METS Profile for Preservation and Digital Repository Interoperability" parent profile.

METS documents which conform to this profile must follow all of the structural requirements for the Description Metadata Section as the "ECHO Dep Generic METS Profile for Preservation and Digital Repository Interoperability" parent profile.

METS documents which conform to this profile must follow all of the structural requirements for the Administrative Metadata Section as the "ECHO Dep Generic METS Profile for Preservation and Digital Repository Interoperability" parent profile.

METS documents which conform to this profile must follow all of the structural requirements for the Content File Section as the "ECHO Dep Generic METS Profile for Preservation and Digital Repository Interoperability" parent profile, except as indicated below.

Unlike the ECHO Dep Generic METS Profile for Preservation and Digital Repository Interoperability" parent profile, this profile supports two different file packaging mechanisms. 1) All the files which comprise the package must exists in the same location as the METS document itself or in subdirectories under the location where the METS document resides (the same as the parent profile). 2) All the files which comprise the package must be contained in one or more ARC (http://www.archive.org/web/researcher/ArcFileFormat.php) files which are in the same directory or subdirectories below where the METS document resides. In both cases these files are referenced via the Flocat element.

Files contained in an ARC file (see below), are not required to be extracted from the ARC file before they are referenced in documents that conform to this profile.

For the MIMETYPE attribute, if the file is an ARC file the value of "application/octet-stream" must be used. If an official MIME type is ever registered for ARC files we will use that value instead.

The OWNERID may be used to identify components of an ARC file (see below).

Multiple ARC files may be referenced.

The file elements representing the ARC files must contain an FLocat element which points to the ARC file. The FLocat must have a LOCTYPE attribute with a value of 'URL'. It must also have an xlink:href attribute which contains the URL which points to the file. The URL should be a relative URI which is relative to the location of the METS document itself.

Each file element representing an ARC file must have USE attribute with a value of 'ARC'.

Each file element representing a ARC file must still conform to the previous requirements for all file elements, including attributes such as SIZE, MIMETYPE, etc.

Technical metadata pertaining to the web crawl or other process that generated the ARC, for example, the logs associated with a run of Heretrix, should be associated with the ARC file via an ADMID attribute attached to the file element which represent the whole ARC file.

Nested under each of the file elements which represents an ARC should be subordinate file elements which represent the individual documents contained in the ARC file which are relevant to the package. For submission information packages (SIPs) where every document contained in the ARC file is relevant to the intellectual entity represented by the METS package, the subordinate file list is optional but recommended. The preservation system to which the package is being submitted should decompose the ARC file and add file elements for all of the documents contained in the ARC.

Not every document contained in the ARC file must be listed, but the documents which are considered to be significant to the representation of the intellectual entity must be listed. One reason that some documents may be omitted is that the ARC file may contain insignificant documents which are not considered part of the representation of the entity, such as HTTP 404 Not Found responses; those documents should not be listed here unless they are intended to be part of the intellectual entity represented by the package.

Another case where documents may be omitted is if the ARC file or files contains documents which cross multiple intellectual entities. In this case only the documents which are part of the intellectual entity represented by the current METS file should be listed. The same ARC file may be referenced in different METS packages with different documents. In general the relationship between the documents contained in an ARC file and the files which comprise an intellectual entity is many-to-many: the files comprising a single intellectual entity may be spread across multiple ARC files, and a single ARC file may contain documents relevant to multiple intellectual entities. Although this type of situation is not recommended, it can be accommodated by this profile. However, in complex cases like this, the relevant subordinate files must always be listed nested beneath the ARC file element. In general, if no subordinate files are listed below the file representing the ARC file the assumption is that all documents contained in the ARC file are required for the representation of the intellectual entity. Likewise, if subordinate files are listed beneath the ARC file element, the assumption is that only those files listed are relevant to the representation of the intellectual entity, even if there are additional documents contained in the ARC.

The reason that the nested file elements are required by this profile, especially when the METS file is an AIP, is that if there are descriptive, technical, or provenance metadata which are tied to specific documents contained in the ARC file, the nested file elements are what link back to the dmdSec, techMD, or digiprovMD elements via the nested file elements' DMDID and ADMID attributes. Because of this it is recommended that a preservation system to which ARC files are being submitted decompose the ARC file into its constituent files and manage these files as individual objects.

The nested file elements must have a USE attribute with a value of 'ARC-URL-RECORD'. They must also have an OWNERID attribute which contains the exact URL-record line corresponding to the document as it occurs in the ARC file, such as:

    http://www.dryswamp.edu:80/index.html 127.10.100.2 19961104142103 text/html 202

    http://www.dryswamp.edu:80/index.html 127.10.100.2 19961104142103 text/html 200 fac069150613fe55599cc7fa88aa089d - 209 IA-001102.arc 202

The OWNERID values should be sufficient to identify and extract any document from the ARC.

Nested file elements which represent documents contained in an ARC file must also still conform to the previous requirements for all file elements, including attributes such as SIZE, MIMETYPE, etc.

METS documents which conform to this profile must follow all of the structural requirements for the Structural Map section as the "ECHO Dep Generic METS Profile for Preservation and Digital Repository Interoperability" parent profile, except as indicated below.

Documents which are contained in an ARC file must be referenced from the structMap via the area element beneath an fptr element. The area element must have a FILEID attribute that references the ARC file containing the document. It must also have a BEGIN attribute which contains the offset to the beginning of the document in bytes. This is the same as the offset value in the URL-record line in the ARC file. The area element must also have an EXTENT attribute with contains the length of the document in the ARC file. This is the same as the length value in the URL-record line in the ARC file. The BETYPE and EXTTYPE attributes must both have a value of 'BYTE'.

We will describe a preferred structural map for web captures. What we mean by a web capture is a collection of resources which have been harvested using the HTTP protocol usually by a single run of a web harvesting system, such as Heritrix. The web capture must have a specific starting and ending time. In this profile, different captures of the same web resources which occur at different times are treated as separate intellectual entities each having its own METS document. However, subsequent web captures should be related to previous versions via the relatedItem type='preceding' element in the primary MODS dmdSec. The metadata contained in the relatedItem metadata should contain enough information to allow a known-item search of the previous web capture. This will usually be an identifier element referring the OBJID of the previous METS document. In this profile the web capture is the primary object being represented by the METS file. Therefore, there must be a unifying theme associated with the web capture which can be described via the primary MODS metadata associated with the web capture object. This could be a single web site or multiple web sites which are related by a common subject.

The root div element immediately below the structMap element represents the whole web capture. It must have a TYPE attribute of "WEB_CAPTURE". The LABEL attribute may optionally contain the title of the web site being captured.

Each resource retrieved as part of the web capture (the body of each HTTP GET or POST response) should be represented by a second-level div below the root div. This should include everything that was retrieved as part of the capture, such as HTML web pages, images, PDF documents, MS Word documents, CSS and XSLT style sheets, Javascript files, Java Applets, etc. Responses which represent HTTP errors, such as 404 Not Found, may be omitted. Second-level div elements must have a TYPE attribute of "WEB_RESOURCE". If a structLink section is used to represent links between these second-level div elements then an xlink:label attribute must be present so that it can be used in the xlink:from and xlink:to attributes of the smLink element. Even though the xlink:label attribute is not an ID-type, it must still follow some of the rules for attributes with a type of ID, namely each xlink:label value must be unique within the METS document.

Dependent resources, such as images embedded in an HTML web page, may be represented by one or more third-level div elements nested below the second-level div. There must be one third-level div for each dependent resource. For HTML web pages typical dependent objects include images embedded via an IMG or INPUT element or the BODY element's background attribute. It also includes FRAME and IFRAME elements for embedded web pages. It includes SCRIPT referenced via the src attribute and style sheets referenced via the LINK element's href attribute. Dependent resources could also include long descriptions for images or frames referenced via the longdesc attribute. They may also include OBJECT and APPLET elements. In general these should be dependent resources that are required for an accurate representation of the primary resource. Resources which are linked to or from this resource, but which are not required in order to represent the resource, should not be listed in these third-level div elements, but instead may be represented in the structLink section. These third-level div elements must have a TYPE attribute of "DEPENDENT_WEB_RESOURCE".

Dependent resources may in turn have their own dependencies. This is particularly true for frames which may be arbitrarily nested. If this is the case, additional nested div elements may be used to any level required. These nested div elements must follow the same rules as already outlined for dependent resources, and must have a TYPE attribute with a value of "DEPENDENT_WEB_RESOURCE". Note, that it is likely that resources which are listed as "DEPENDENT_WEB_RESOURCE" will also occur as "WEB_RESOURCE" elsewhere in the same structMap.

In case there are web resources or dependent web resources which have their own descriptive metadata, the DMDID attribute of the div element which represents that resource may point to the corresponding relatedItem type="constituent" elements in the primary MODS record. Thus, descriptive metadata such as title is included in the primary MODS dmdSec, linked from the structMap to the particular relatedItem type="constituent" using the DMDID and ID attributes. Optionally, web resources which have a title may also have that title represented by the LABEL attribute of the div element.

Example .

METS documents which conform to this profile must follow all of the structural requirements for the Structural Map linking section as the "ECHO Dep Generic METS Profile for Preservation and Digital Repository Interoperability" parent profile.

METS documents which conform to this profile must follow all of the structural requirements for the Behavior Section as the "ECHO Dep Generic METS Profile for Preservation and Digital Repository Interoperability" parent profile.

METS documents which conform to this profile must follow all of the structural requirements for the Multiple Sections as the "ECHO Dep Generic METS Profile for Preservation and Digital Repository Interoperability" parent profile, except as defined below:

     file[@USE='ARC']/@ADMID ->> techMD[*** Heritrix Config and Log Files *** ]/@ID

METS documents which conform to this profile must follow all of the technical requirements for Content Files as the "ECHO Dep Generic METS Profile for Preservation and Digital Repository Interoperability" parent profile, except as defined below:

When documents that conform to this profile are used for submission or dissemination packages, one or more ARC files may be used to contain the files or bitstreams without first extracting the files from the ARC. For details refer to the Structural Requirements given above. This differs from the technical requirements given in the "ECHO Dep Generic METS Profile for Preservation and Digital Repository Interoperability" parent profile which requires that files contained in other archive files, like Zip or TAR, must be extracted before being referenced in the METS document.

However, this profile does strongly recommend that if the METS document is being used as an archival information package (AIP) that the files be extracted from the ARC and referenced directly.

METS documents which conform to this profile must follow all of the technical requirements for Behavior Files as the "ECHO Dep Generic METS Profile for Preservation and Digital Repository Interoperability" parent profile.

METS documents which conform to this profile must follow all of the technical requirements for Metadata Files as the "ECHO Dep Generic METS Profile for Preservation and Digital Repository Interoperability" parent profile.