Preservation planning¶

Formats¶

In the FPR, a “format” is a record representing one or more related format versions, which are records representing a specific file format. For example, the format record for “Graphics Interchange Format” (GIF) is comprised of format versions for both GIF 1987a and 1989a.

When creating a new format version, the following fields are available:

• Description (required) - Text describing the format. This will be saved in METS files.
• Version (required) - The version number for this specific format version (not the FPR record). For example, for Adobe Illustrator 14 .ai files, you might choose “14”.
• PRONOM ID - The specific format version’s unique identifier in PRONOM, the UK National Archives’s format registry. This is optional, but highly recommended. Many tools, including FIDO which is available as an identification tool option in the format identification micro-service in transfer and ingest, use PRONOM ID’s to recognize formats.
• Access format and Preservation format - Indicates whether this format is suitable as an access format for end users, and for preservation.

Groups¶

A format group is a convenient grouping of related file formats which share common properties. For instance, the FPR includes an “Image (raster)” group which contains format records for GIF, JPEG, and PNG. Each format can belong to one (and only one) format group.

The following format groups come pre-populated in the FPR:

• Audio
• Binary (Data)
• Binary (Executable)
• CAD
• Database
• Desktop Publishing
• Disk Image
• Email
• Font
• GIS
• Image (Raster)
• Image (Vector)
• Package
• Portable Document Format
• Presentation
• Spreadsheet
• Statistics
• Text (Markup)
• Text (Plain)
• Text (Source Code)
• Test (Structured)
• Unknown
• Video
• Word Processing

The groups as determined by Artefactual should be perceived as arbritrary and are simply meant to make the Format Policy Registry easier to read and navigate. If an institution so desired, they could change the names and population of the groups in their local Preservation planning tab.

Format policy rules¶

Throughout the FPR, Format policy rules allow commands to be associated with specific file types. For example, in the case of Normalization rules, this allows you to configure the command that uses ImageMagick to create thumbnails to be run on .gif and .jpeg files, while selecting a different command to be run on .png files.

When creating a format policy rule, the following mandatory fields must be filled out:

• Purpose - Allows Archivematica to distinguish rules that should be used to normalize for preservation, normalize for access, to extract information, etc.
• Format - The file format the associated command should be selected for.
• Command - The specific command to call when this rule is used.

Instructions on writing commands can be found in the FPR Administrators manual.

Identification¶

Tools

The identification tool properties in Archivematica control the ways in which Archivematica identifies files and associates them with the FPR’s version records. The current version of the FPR server contains three tools: a script based on the Open Planets Foundation’s FIDO tool, which identifies based on the IDs in PRONOM; a simple script which identifies files by their file extension; and Siegfried which like FIDO, is based on PRONOM ID and provides detailed information on the basis for format matches in its output. You can use the identification tools portion of FPR to customize the behaviour of the existing tools, or to write your own.

Rules

Identification rules allow you to define the relationship between the output created by an identification tool, and one of the formats which exists in the FPR. This must be done for the format to be tracked internally by Archivematica, and for it to be used by normalization later on. For instance, if you created a FIDO configuration which returns MIME types, you could create a rule which associates the output “image/jpeg” with the “Generic JPEG” format in the FPR.

Identification rules are necessary only when a tool is configured to return file extensions or MIME types. Because PUIDs are universal, Archivematica will always look these up for you without requiring any rules to be created, regardless of what tool is being used.

Instructions on writing an identification rule can be found in the FPR Administrators manual.

Commands

Identification commands contain the actual code that a tool will run when identifying a file. This command will be run on every file in a transfer.

Instructions on writing an identification command can be found in the FPR Administrators manual.

Format policy registry tools¶

Format policy tools control how Archivematica processes files during ingest. The most common kind of these tools are normalization tools, which produce preservation and access copies from ingested files. Archivematica comes configured with a number of commands and scripts to normalize several file formats, and you can use this section of the FPR to customize them or to create your own. These are organized similarly to the Identification Tools documented above.

Archivematica uses the following kinds of format policy rules:

• Characterization
• Extraction
• Normalization - Access, preservation and thumbnails
• Event detail - Extracts information about a given tool in order to be inserted into a generated METS file.
• Transcription
• Verification - Validates a file produced by another command. For instance, a tool could use Exiftool or JHOVE to determine whether a thumbnail produced by a normalization command was valid and well-formed.

Characterization¶

Characterization is the process of producing technical metadata for an object. Archivematica’s characterization aims both to document the object’s significant properties and to extract technical metadata contained within the object.

Prior to Archivematica 1.2, the characterization micro-service always ran the FITS tool. As of Archivematica 1.2, characterization is fully customizable by the Archivematica administrator.

Tools

Archivematica has four default characterization tools upon installation. Which tool will run on a given file depends on the type of file, as determined by the selected identification tool.

Default

The default characterization tool is FITS; it will be used if no specific characterization rule exists for the file being scanned.

It is possible to create new default characterization commands, which can either replace FITS or run alongside it on every file.

Multimedia

Archivematica 1.2 introduced three new multimedia characterization tools. These tools were selected for their rich metadata extraction, as well as for their speed. Depending on the type of the file being scanned, one or more of these tools may be called instead of FITS.

• FFprobe, a characterization tool built on top of the

  same core as FFmpeg, the normalization software used by Archivematica

• MediaInfo, a characterization tool

  oriented towards audio and video data

• ExifTool, a

  characterization tool oriented towards still image data and extraction of embedded metadata

Commands

Information on writing new characterization commands can be found in the FPR administrator’s manual.

A characterization command is designed to run a tool and produce output to standard out. Output from characterization commands is expected to be valid XML, and will be included in the AIP’s METS document within the file’s <objectCharacteristicsExtension> element.

Event Detail¶

A command to provide information about the software running a command. This will be written to the METS file as the “event detail” property. For example, the normalization commands which use ffmpeg use an event detail command to extract ffmpeg’s version number.

For more information, see the FPR Administrators manual.

Extraction¶

Archivematica supports extracting contents from files during the transfer phase.

Many transfers contain files which are packages of other files; examples of these include compressed archives, such as ZIP files, or disk images. Archivematica provides a transcription microservice which comes with several predefined rules to extract packages, and which is fully customizeable by Archivematica administrators. Administrators can write new commands, and assign existing formats to run for other file formats.

Commands

An extraction command is passed two arguments: the file to extract, and the path to which the package should be extracted. Similar to normalization commands, these arguments will be interpolated directly into “bashScript” and “command” scripts, and passed as positional arguments to “pythonScript” and “asIs” scripts.

Information on writing a new Extraction command can be found in the FPR Administrators manual.

Normalization¶

Normalization rules determine how Archivematica will handle the creation of access and preservation copies during the Ingest of packages. Normalization commands take those rules and express them as actions that Archivematica will perform with one of its integrated tools.

Information on writing a new Normalization command can be found in the FPR Administrators manual.

Transcription¶

Archivematica 1.2 introduces a new transcription microservice. This microservice provides tools to transcribe the contents of media objects. In Archivematica 1.2 it is used to perform OCR on images of textual material, but it can also be used to create commands which perform other kinds of transcription.

Commands

Writing a transcription command is very similar to writing an identification command or a normalization command.

Transcription commands are expected to write their data to disk inside the SIP. For commands which perform OCR, metadata can be placed inside the metadata/OCRfiles" directory inside the SIP; other kinds of transcription should produce files within``metadata``.

Information on writing a new Transcription command can be found in the FPR Administrators manual.

Validation¶

Archivematica runs commands to validate files on transfer using JHOVE. As of version 1.2, rules for valiadation are defined in the Format Policy Registry.

Verification¶

Verification is run after normalization commands. Archivematica will run two commands: one checks if a file exists, and the other checks if the file exists and is greater than 0 bytes in size.

Verification is run on the output of normalization, not on the original file.

Identification commands¶

Identification commands are very simple to write, though they require some familiarity with Unix scripting.

An identification command run once for every file in a transfer. It will be passed a single argument (the path to the file to identify), and no switches.

On success, a command should:

• Print the identifier to stdout
• Exit 0

On failure, a command should:

• Print nothing to stdout
• Exit non-zero (Archivematica does not assign special significance to non-zero exit codes)

A command can print anything to stderr on success or error, but this is purely informational - Archivematica won’t do anything special with it. Anything printed to stderr by the command will be shown to the user in the Archivematica dashboard’s detailed tool output page. You should print any useful error output to stderr if identification fails, but you can also print any useful extra information to stderr if identification succeeds.

Here’s a very simple Python script that identifies files by their file extension:

import os.path, sys
(_, extension) = os.path.splitext(sys.argv[1])
if len(extension) == 0:
        exit(1)
else:
        print extension.lower()

Here’s a more complex Python example, which uses Exiftool’s XML output to return the MIME type of a file:

#!/usr/bin/env python

from lxml import etree
import subprocess
import sys

try:
    xml = subprocess.check_output(['exiftool', '-X', sys.argv[1]])
    doc = etree.fromstring(xml)
    print doc.find('.//{http://ns.exiftool.ca/File/1.0/}MIMEType').text
except Exception as e:
    print >> sys.stderr, e
    exit(1)

Once you’ve written an identification command, you can register it in the FPR using the following steps:

1. Navigate to the “Preservation Planning” tab in the Archivematica dashboard.
2. Navigate to the “Identification Tools” page, and click “Create New Tool”.
3. Fill out the name of the tool and the version number of the tool in use. In our example, this would be “exiftool” and “9.37”.
4. Click “Create”.

Next, create a record for the command itself:

1. Click “Create New Command”.
2. Select your tool from the “Tool” dropdown box.
3. Fill out the Identifier with text to describe to a user what this tool does. For instance, we might choose “Identify MIME-type using Exiftool”.
4. Select the appropriate script type - in this case, “Python Script”.
5. Enter the source code for your script in the “Command” box.
6. Click “Create Command”.

Finally, you must create rules which associate the possible outputs of your tool with the FPR’s format records. This needs to be done once for every supported format; we’ll show it with MP3, as an example.

1. Navigate to the “Identification Rules” page, and click “Create New Rule”.
2. Choose the appropriate foramt from the Format dropdown - in our case, “Audio: MPEG Audio: MPEG 1/2 Audio Layer 3”.
3. Choose your command from the Command dropdown.
4. Enter the text your command will output when it identifies this format. For example, when our Exiftool command identifies an MP3 file, it will output “audio/mpeg”.
5. Click “Create”.

Once this is complete, any new transfers you create will be able to use your new tool in the identification step.

Normalization Commands¶

The goal of a normalization command is to take an input file and transform it into a new format. For instance, Archivematica provides commands to transform video content into FFV1 for preservation, and into H.264 for access.

Archivematica provides several parameters specifying input and output filenames and other useful information. Several of the most common are shown in the examples below; a more complete list is in a later section of the documentation: Normalization command variables and arguments.

When writing a bash script or a command line, you can reference the variables directly in your code, like this:

inkscape -z "%fileFullName%" --export-pdf="%outputDirectory%%prefix%%fileName%%postfix%.pdf"

When writing a script in Python or other languages, the values will be passed to your script as commandline options, which you will need to parse. The following script provides an example using the argparse module that comes with Python:

import argparse
import subprocess

parser = argparse.ArgumentParser()

parser.add_argument('--file-full-name', dest='filename')
parser.add_argument('--output-file-name', dest='output')
parsed, _ = parser.parse_known_args()
args = [
    'ffmpeg', '-vsync', 'passthrough',
    '-i', parsed.filename,
    '-map', '0:v', '-map', '0:a',
    '-vcodec', 'ffv1', '-g', '1',
    '-acodec', 'pcm_s16le',
    parsed.output+'.mkv'
]

subprocess.call(args)

Once you’ve created a command, the process of registering it is similar to creating a new identification tool. The folling examples will use the Python normalization script above.

First, create a new tool record:

1. Navigate to the “Preservation Planning” tab in the Archivematica dashboard.
2. Navigate to the “Identification Tools” page, and click “Create New Tool”.
3. Fill out the name of the tool and the version number of the tool in use. In our example, this would be “exiftool” and “9.37”.
4. Click “Create”.

Next, create a record for your new command:

1. Click “Create New Tool Command”.
2. Fill out the Description with text to describe to a user what this tool does. For instance, we might choose “Normalize to mkv using ffmpeg”.
3. Enter the source for your command in the Command textbox.
4. Select the appropriate script type - in this case, “Python Script”.
5. Select the appropriate output format from the dropdown. This indicates to Archivematica what kind of file this command will produce. In this case, choose “Video: Matroska: Generic MKV”.
6. Enter the location the video will be saved to, using the script variables. You can usually use the %outputFileName% variable, and add the file extension - in this case %outputFileName%.mkv
7. Select a verification command. Archivematica will try to use this tool to ensure that the file your command created works. Archivematica ships with two simple tools, which test whether the file exists and whether it’s larger than 0 bytes, but you can create new commands that perform more complicated verifications.
8. Finally, choose a command to produce the “Event detail” text that will be written in the section of the METS file covering the normalization event. Archivematica already includes a suitable command for ffmpeg, but you can also create a custom command.
9. Click “Create command”.

Finally, you must create rules which will associate your command with the formats it should run on.

Normalization command variables and arguments¶

The following variables and arguments control the behaviour of format policy command scripts.

Extraction commands¶

An extraction command is passed two arguments: the file to extract, and the path to which the package should be extracted. Similar to normalization commands, these arguments will be interpolated directly into bashScript and command scripts, and passed as positional arguments to pythonScript and asIs scripts.

Here’s a simple example of how to call an existing tool (7-zip) without any extra logic:

7z x -bd -o"%outputDirectory%" "%inputFile%"

This Python script example is more complex, and attempts to determine whether any files were extracted in order to determine whether to exit 0 or 1 (and report success or failure):

from __future__ import print_function
import re
import subprocess
import sys

def extract(package, outdir):
    # -a extracts only allocated files; we're not capturing unallocated files
    try:
        process = subprocess.Popen(['tsk_recover', package, '-a', outdir],
            stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
        stdout, stderr = process.communicate()

        match = re.match(r'Files Recovered: (\d+)', stdout.splitlines()[0])
        if match:
            if match.groups()[0] == '0':
                raise Exception('tsk_recover failed to extract any files with the message: {}'.format(stdout))
            else:
                print(stdout)
    except Exception as e:
        return e

    return 0

def main(package, outdir):
    return extract(package, outdir)

if __name__ == '__main__':
    package = sys.argv[1]
    outdir = sys.argv[2]
    sys.exit(main(package, outdir))

Transcription commands¶

Transcription commands are expected to write their data to disk inside the SIP. For commands which perform OCR, metadata can be placed inside the “metadata/OCRfiles” directory inside the SIP; other kinds of transcription should produce files within “metadata”.

For example, the following bash script is used by Archivematica to transcribe images using the Tesseract software:

ocrfiles="%SIPObjectsDirectory%metadata/OCRfiles"
test -d "$ocrfiles" || mkdir -p "$ocrfiles"

tesseract %fileFullName% "$ocrfiles/%fileName%"

>>>>>>> 0e96a35… Add admonition re: FPR server to 1.6 docs

Back to the top