Transfer¶

Permissions¶

In order for Archivematica to be able to carry out preservation tasks, the Archivematica user needs to have execute privileges for directories and read permissions for files in the transfer source location.

If the Archivematica user does not have sufficient permissions, you may see that a directory or compressed file is not available to add to your transfer (the object will be greyed out and the name will have a strikethrough.) Depending on the exact permissions setup, you may be able to add the directory to your transfer, but the transfer will never start.

Transfer types¶

Some materials that are transferred into Archivematica require special processing. To kick off these specialized workflows, you can select a specific transfer type. There are seven different transfer types available.

Standard: Standard transfers are the default in Archivematica. All materials can be transferred using the standard transfer type. No special processing tasks are carried out. If you’re unsure which transfer type to choose, start with a standard transfer. For more information on how standard transfers can be structured, see basic transfers and subsequent sections below.

Zipped directory: Zipped directory transfers are a variant of the default transfer type in Archivematica. All materials can be transferred using the Zipped directory transfer type; the material just happens to be in a .zip, .tgz, or .tar.gz file at the outset. When a Zipped directory transfer starts, the zip will be decompressed and the material then processed using the Standard transfer workflow documented above. The internal structure of a zipped directory transfer matches that of the standard transfer and so can either be a loose collection of files, or more structured, for example, including an objects and metadata directory including Archivematica’s special files such as metadata.csv, or identifiers.json.

Unzipped bags: Archivematica can recognize and make use of materials that have been packaged according to the BagIt File Packaging Format, colloquially known as bags. If you are in the practice of making bags before transferring material into Archivematica, you can continue to do so. Archivematica will verify the bag early on in the transfer process, looking at manifest information created during the bagging process such as checksums and the payload oxum. The Unzipped bags transfer type can be used for bags that are not saved in a compressed format. For more information about how Archivematica implements bags, see Bags.

This screenshot shows the layout of a bagged transfer. The bagging process moves the digital objects to be preserved to a data directory and creates various metadata files required by the BagIt specification (i.e. bag-info.txt, manifest-md5.txt).

Zipped bags: Similar to Unzipped bags, the Zipped bags transfer type can br used for materials that have been packaged according to the BagIt File Packaging Format. The Zipped bags transfer type should be used for bags that have been saved in a compressed (or zipped) format. Archivematica accepts .zip, .tgz, or .tar.gz formats. When you switch the transfer type to Zipped bags, only files saved as .zip, .tgz, and .tar.gz will be available to be selected in the transfer browser. For more information about how Archivematica implements bags, see Bags.

DSpace: When you export materials from a DSpace repository, the export is packaged with a METS file for the export as a whole as well as each individual item. Archivematica can reuse some of the data in these METS files. For more information, see DSpace.

Disk image: Selecting the disk image transfer type is not required to preserve disk images (you can use the standard transfer type or the bag transfer types, if your disk image is bagged); however, it does give you an extra disk image-specific metadata form where you can record information about the imaging process. For more information, see Forensic disk images.

Dataverse: Similar to the DSpace transfer type, materials exported from a Dataverse repository contain metadata that Archivematica can reuse. For more information, see Dataverse.

Basic transfers¶

This section describes the basic setup for a standard transfer. If you are planning to use any other transfer type, please see the Transfer types section above for more information about your transfer type.

With the exception of zipped bag transfers, Archivematica requires that all of the materials in the transfer are contained within a top-level directory. The directory structure of the transfer can be simple (i.e. all files located in the same directory) or it can be nested and hierarchical.

The following screenshot shows a basic transfer called basicTransfer. Four digital objects sit within the top-level directory, while two more objects are nested within a subdirectory.

The transfer can have as many subdirectories as you wish, though this may impact processing time. The digital objects can be in any format, though what exactly Archivematica can do with a given file format varies widely. For more information on how Archivematica carries out file format preservation, see Preservation planning.

This is the basic structure of a transfer in Archivematica; however, there are many variations. The following sections build on the basic standard transfer to achieve specific goals relating to the preservation of the digital objects.

Transfers with descriptive, rights, and/or event metadata¶

To include descriptive, rights, and/or event metadata with your transfer, you should add a subdirectory called metadata to the top level of the transfer. The metadata directory name is a reserved name in Archivematica; it must not be used for anything else. Including the metadata directory ensures that your metadata files are appropriately marked as such in the METS file.

For more information about importing metadata into Archivematica, including how to structure the metadata and rights CSV files or event XML file so that the contents can be transposed into the AIP METS, see Import metadata.

Transfers with submission documentation¶

Submission documentation is a concept in Archivematica that accounts for materials that are related to the digital objects being preserved, but aren’t strictly part of the collection - for example, donor agreements, correspondence about the materials, conservation reports, etc. If Archivematica sees that a transfer includes submission documentation, it can include descriptions of this material in the AIP METS file.

Submission documentation can be added to standard, unzipped, zipped, and disk image transfer types.

To create a transfer that includes submission documentation, your top-level directory must contain a metadata directory. Inside the metadata directory, a nested directory called submissionDocumentation contains the submission documentation files.

Create a transfer with existing checksums¶

Archivematica can verify MD5, SHA1, SHA256, and SHA512 checksums that were created outside of the system. Creating checksums outside of Archivematica is a good idea if you are concerned about loss of data integrity during the move into Archivematica. Checksums are checked during the Verify transfer checksums microservice on the Transfer tab.

Checksums can be added to standard, DSpace, and disk image transfer types. Note that checksums are added to both unzipped and zipped transfer types as part of the bag requirements.

Checksum files are placed in the metadata directory. Note that you can also place descriptive and rights metadata CSVs in this directory, as per the Transfers with descriptive and/or rights metadata section above.

Checksum files should be named as follows:

• checksum.md5,
• checksum.sha1,
• checksum.sha256, or
• checksum.sha512.

The checksum file itself should contain one line for each checksum, beginning with the checksum, followed by two spaces, followed by the file path:

2121dca88ad7f701d3f3e2d041004a56  beihai.tif
7f42199657dea535b6ad1963a6c7a2ac  bird.mp3
6dc1519418859ea5c20fd708e89d7254  ocr-image.png
4737e4dacfc9510915ea58cf12e51712  View_from_lookout_over_Queenstown_towards_the_Remarkables_in_spring.jpg
75388a532283b988f79206d63f65e9a2  subdirectory/piiTestDataCreditCardNumbers.txt
1d7193ea3b2193c79f55ea7e645503a9  subdirectory/piiTestDataSocialSecurityNumbers.txt

If your checksum check fails, the Verify transfer checksums microservice will show an error and the transfer will fail. Expanding the microservice will show that the job Verify metadata directory checksums is red. To review the error, click on the gear icon for the job.

Transfers with existing persistent identifiers¶

Archivematica can import persistent identifiers that were created outside of Archivematica, such as DOI identifiers or identifiers created by your cataloging system. You can bind these to the PREMIS objects created by Archivematica for your files by including an identifiers.json file in the transfer’s metadata directory.

[
    {
        "file": "objects/bird.mp3",
        "identifiers": [
            {
                "identifier": "01D9T8NGAQ2DBN8M4D23HM5FYN",
                "identifierType": "ULID"
            }
        ]
    },
    {
        "file": "objects/beihai.tif",
        "identifiers": [
            {
                "identifier": "https://example.com/file/id/2367652602",
                "identifierType": "EXID"
            }
        ]
    }
]

The identifiers.json file is parsed during the Process metadata directory microservice on the Ingest tab.

Once the AIP has been created, the identifier will be linked as a premis:objectIdentifier within the techMD for each object within the AIP METS file.

<mets:techMD ID="techMD_4">
  <mets:mdWrap MDTYPE="PREMIS:OBJECT">
    <mets:xmlData>
      <premis:object xmlns:premis="http://www.loc.gov/premis/v3" xsi:type="premis:file" xsi:schemaLocation="http://www.loc.gov/premis/v3 http://www.loc.gov/standards/premis/v3/premis.xsd" version="3.0">
        <premis:objectIdentifier>
          <premis:objectIdentifierType>UUID</premis:objectIdentifierType>
          <premis:objectIdentifierValue>2251bc66-85c1-4d64-952e-b0d7ea2db385</premis:objectIdentifierValue>
        </premis:objectIdentifier>
        <premis:objectIdentifier>
          <premis:objectIdentifierType>EXID</premis:objectIdentifierType>
          <premis:objectIdentifierValue>https://example.com/file/id/2367652602</premis:objectIdentifierValue>
        </premis:objectIdentifier>
      </premis:object>
    </mets:xmlData>
  </mets:mdWrap>
</mets:techMD>
<mets:techMD ID="techMD_5">
  <mets:mdWrap MDTYPE="PREMIS:OBJECT">
    <mets:xmlData>
      <premis:object xmlns:premis="http://www.loc.gov/premis/v3" xsi:type="premis:file" xsi:schemaLocation="http://www.loc.gov/premis/v3 http://www.loc.gov/standards/premis/v3/premis.xsd" version="3.0">
        <premis:objectIdentifier>
          <premis:objectIdentifierType>UUID</premis:objectIdentifierType>
          <premis:objectIdentifierValue>480d31e0-f57f-4254-a5bf-c813ed5734d5</premis:objectIdentifierValue>
        </premis:objectIdentifier>
        <premis:objectIdentifier>
          <premis:objectIdentifierType>ULID</premis:objectIdentifierType>
          <premis:objectIdentifierValue>01D9T8NGAQ2DBN8M4D23HM5FYN</premis:objectIdentifierValue>
        </premis:objectIdentifier>
      </premis:object>
    </mets:xmlData>
  </mets:mdWrap>
</mets:techMD>

For more information on how metadata elements are encoded used in Archivematica, please see METS in Archivematica and PREMIS metadata in Archivematica.

Transferring material with preservation or access derivatives (manual normalization)¶

Archivematica’s main strategy for preserving files is to normalize them according to the normalization rules listed in the Preservation Planning tab. However, you may want to create preservation or access copies of your digital objects outside of Archivematica (i.e. during a digitization project). Archivematica can recognize manual normalization work and use the preservation and action copies instead of creating new derivatives.

For more information about structuring a transfer with manually normalized preservation and access derivatives, see Manual normalization.

Transferring material with access derivatives only¶

If you are only transferring originals plus their access derivatives, there is a slightly more streamlined workflow than the manual normalization option described above.

Access derivatives can be added to standard transfer types.

Inside your top-level directory, create an access subdirectory and place the access copies of your originals inside this file.

In the example shown above, the original files are PNGs (i.e. beihai.png). For each file, there is an access copy in the form of a JPG (beihai.jpg). Note that the base filenames (before the extension) must match in order for Archivematica to recognize the links between the original file and access copy.

When you process this transfer, Archivematica will automatically recognize the existence of the access copies. At the normalization microservice on the Ingest tab, fewer normalization options will appear because Archivematica will always make a Dissemination Information Package (DIP) from the provided access copies.

Transferring material with service (mezzanine) files¶

Somewhat similar to the manual normalization and manual normalization options described above, Archivematica can also recognize the presence of service (or mezzanine) files. Service files are high-quality derivatives created from the original file, which are then used to create every other derivative. For example, during a digitization project you may scan an image as a very high-quality TIFF, then generate a high-quality JP2000 from the TIFF. Instead of accessing the TIFF every time a new access derivative is required, you would then use the JP2000 to make new copies of the file.

Service (or mezzanine) files can be added to standard transfer types.

Inside your top-level directory, create a service subdirectory and place the service copies of your originals inside this file.

In the example shown above, the original files are TIFFs (i.e. beihai.tif). For each file, there is a service copy in the form of a JP2000 (beihai.jp2). Note that the base filenames (before the extension) must match in order for Archivematica to recognize the links between the original file and the service copy.

When you process this transfer, Archivematica will automatically recognize the existence of the service copies. At the normalization microservice on the Ingest tab, you will be given the option to use the service copies to generate access derivatives, rather than the original files.

Start a transfer¶

1. Sign in to Archivematica and navigate to the Transfer tab by clicking on Transfer or the Archivematica logo.

2. Select your transfer type from the Transfer types dropdown menu.

3. Name your transfer. The transfer name will become the name of your AIP, so make sure that the name is meaningful.

4. If needed, also add an accession number and/or access system ID, and check or uncheck the Approve transfer checkbox.

5. Click Browse to open the transfer browser, where you can select a the digital object(s) you want to preserve. You can toggle between transfer source locations by clicking on the top bar of the transfer browser (called archivematica-sampledata in the screenshot below).

6. Click on the yellow folder icon to explore nested directories. To select the material that you want to preserve, click on the name of the folder (or .zip file, if you are transferring a Zipped Bag) so that it is highlighted and click Add. To close the browser, click on Browse again.

   

Choose a processing configuration¶

1. Once you’ve clicked the Add button the Start transfer button becomes active.

2. On the far right of the green Start Transfer button is a downward arrow. When clicked, it presents a drop-down menu from which you can choose which processing configuration profile will be used for the transfer.

   These profiles are defined on Archivematica’s Administration tab. They allow users to configure the job decision points presented by Archivematica during Transfer and Ingest (e.g. whether to generate thumbnails, which AIP storage location to use, etc.). See Processing configuration in the Administrator’s manual for more information.

3. If you define a new processing configuration profile in the Administration tab, it will be presented as an option in the Start Transfer dropdown (e.g. “my_custom_config”).

   

4. If you would like to use this processing configuration profile for your transfer, select it from the drop-down menu and press the Start Transfer button.

5. You will be notified by Archivematica that your transfer has started and the processing configuration profile that it is going to use for the decision points encountered during Transfer and Ingest.

   

6. If you ignore the drop-down menu and just press the Start Transfer menu then the “default” processing configuration will be used.

Transfer tab microservices¶

Once you click on Start transfer, the transfer should appear below the transfer preparation area. This should only take a few seconds, though there might be a longer delay if the transfer is very large. If the transfer never appears, do not try to start another transfer - instead, contact your systems administrator as it is possible that you have run out of processing space and starting another new transfer will only compound the issue.

Each microservice in the Transfer tab is carrying out some step towards preparing your material to become a Submission Information Package (SIP) and then an Archival Information Package (AIP). For more information about Archivematica’s microservice architecture, please see microservices.

Some microservices occur with no human intervention, while others will prompt the user for a decision. An early example is the Identify file format microservice, which prompts the user to decide if they would like to run a file format identification tool on the files in the transfer.

Note that it is possible to automate all decision points in Archivematica. For more information, see Processing configuration.

The microservices that run on the Transfer tab include:

• Verify transfer compliance: verifies that the transfer is properly structured according to the requirements of the transfer type.
• Rename with transfer UUID: assigns a unique universal identifier for the transfer as a whole.
• Assign file UUIDs and checksums to objects: assigns a unique universal identifier and checksums to each digital object.
• Verify transfer checksums: verifies any checksums included with the transfer.
• Generate METS.xml document: creates a METS file capturing the original order of the transfer. This METS file is added to any SIPs generated from this transfer.
• Scan for viruses: scans for viruses and malware. For more information, see Scan for viruses.
• Generate transfer structure report: generates a directory tree of the original transfer and places as a text file in the AIP.
• Change transfer filenames: removes prohibited characters from folder and filenames, such as ampersands. For more information, see Change filenames.
• Identify file format: allows the user to choose whether or not to identify file formats. Selecting “Yes” will prompt the enabled file identification command to run. See Identification for more information.
• Extract packages: extracts contents from zipped or otherwise packaged files. See Extraction for more information.
• Characterize and extract metadata: extracts technical metadata embedded in the files. See Characterization for more information.
• Validation: validates file formats against the format’s specification. See Validation for more information.
• Examine contents: runs Bulk Extractor.
• Create SIP from transfer: gives users the chance to send the transfer to the Backlog tab, where it can be stored for processing later. Using the backlog also gives users a chance to carry out appraisal tasks. Alternately, the user can turn the transferred material into a SIP and send it along to the Ingest tab. The transfer can also be rejected at this point.

A transfer that is in the middle of processing will show which microservices have been completed (green) and which are in progress (orange). When a microservice fails or encounters an error, the microservice background turns from green to pink and a “failed” icon appears next to the transfer or SIP name. See Error handling for more information about how to handle an error.

Remove a single transfer¶

1. Ensure that the transfer you want to remove doesn’t require any user input. You must complete all user inputs and either complete the transfer (i.e. send to backlog or create a SIP) or reject the transfer before it can be removed from the dashboard.
2. When you are ready to remove a transfer from the dashboard, click the red circle icon to the right of the transfer name.
3. Click the Confirm button to remove the transfer from the dashboard.

Remove all completed transfers¶

1. Ensure that the transfers you want to remove are complete (i.e. sent to backlog or ingest). Note that this feature only works on completed transfers; rejected transfers will have to be removed one at a time.
2. When you are ready to remove all completed transfers, click the red circle icon in the table header of the list of transfers.
3. Click the Confirm button to remove all completed transfers from the dashboard.

Back to the top