Administering the Storage Service¶

Arkivum¶

Archivematica can use Arkivum’s A-Stor as an access protocol in version 0.7 and higher. A-Stor can expose a CIFS share to the Storage Service so that the Storage Service can copy files to an A-Stor datapool for AIP storage, for example.

Add an entry to /etc/fstab on the Storage Service, then mount the A-Stor CIFS share.

Example:

//ARK00092/astor /mnt/astor cifs
defaults,guest,file_mode=0666,dir_mode=0777,uid=archivematica,gid
=archivematica,forcegid,forceuid,rw 0 1

In this example, ARK00092 is the name of the appliance and should be resolvable through DNS or be set as an entry in /etc/hosts.

Then, choosing Arkivum as the access protocol, create a new space in the Storage Service.

Fields:

• Size: the maximum size allowed for this space. Leave blank for unlimited. This field is optional.
• Path: the local path on the Storage Service machine to the CIFS share, e.g. /mnt/astor
• Staging Path: the absolute path to a staging area. Must be UNIX filesystem compatible, preferably on the same filesystem as the path, e.g. /mnt/astor/archivematica1/tmp
• Host: the hostname of the Arkivum web instance or IP address with port, e.g. arkivum.example.com:8443
• Remote user: the username on the remote machine accessible via passwordless ssh. This field is optional.
• Remote name: the name or IP of the remote machine. This field is optional.

Dataverse¶

Dataverse Integration is supported with Archivematica v1.8 (and higher) and Storage Service v0.13 (and higher).

Fields:

• Size: the maximum size allowed for this space. Leave blank for unlimited. This field is optional.
• Path: the absolute path to the Space on the local filesystem.
• Staging path: the absolute path to a staging area. Must be UNIX filesystem compatible and preferably will be located on the same filesystem as the path.
• Host: Hostname or IP address of the Dataverse instance, e.g. test.dataverse.org
• API key: the key generated by Dataverse for a specific user account
• Agent name: a string that will be used in Archivematica METS file to identify this Dataverse instance as a PREMIS agent, e.g. My Institution's Dataverse Instance.
• Agent type: a string that will be used in Archivematica METS file to identify the type of PREMIS agent named above, e.g. organization.
• Agent identifier: a string that will be used as an identifier in the Archivematica METS file to uniquely identify the PREMIS agent named above.

Dataverse spaces support Transfer Source Locations (Locations for other purposes are not currently supported). At least one location should be created as a Transfer Source.

Within this location, the relative path can be used to set two of the parameters available in the Dataverse Search API. The q (or “Query”) parameter is a general search parameter. The ’subtree’ parameter can be used to indicate a sub-dataverse. For example, the following entry in ‘Relative Path’:

Query:*
Subtree:Archivematica

will return all datasets within the Archivematica dataverse. The other API parameters are set using the fields described above for the space, or are set with fixed values. The parameters used are:

URL: https://<Host field set in Space configuration>/api/search/

{
  'q': '<Relative Path field set in Location configuration>',
  'sort': 'name',
  'key': u '<API Key field set in Space configuration>',
  'start': 50,
  'per_page': 50,
  'show_entity_ids': True,
  'type': 'dataset',
  'subtree': '<Relative Path field set in Location configuration>',
  'order': 'asc'
}

Search results are currently limited to 50 datasets. For repositories with more than 50 datasets we recommend creating multiple Locations with more specific search criteria. For further details of the API parameters, see the Dataverse api guide.

DuraCloud¶

Archivematica can use DuraCloud as an access protocol for the Storage Service in version 0.5 and higher. A Storage Service space has a one-to-one relationship to a space within DuraCloud.

To set up your Archivematica instance with DuraCloud, please see Using DuraCloud with Archivematica.

Fields:

• Access protocol: DuraCloud
• Size: the maximum size allowed for this space. Leave blank for unlimited. This field is optional.
• Path: Leave this field blank.
• Staging path: A location on your local disk where Archivematica can place files for staging purposes, for example var/archivematica/storage_service/duracloud_staging
• Host: Hostname of the DuraCloud instance, e.g. site.duracloud.org
• Username: The username for the Archivematica user that you created in DuraCloud.
• Password: The password for the Archivematica user that you created in DuraCloud.
• Duraspace: The name of the space in DuraCloud you are creating this Storage Service space for (e.g. transfer-source, aip-store, etc).

DSpace via REST API¶

DSpace via REST API locations are supported for both AIP and DIP Storage locations. Because DSpace is typically used as a public-facing system, the behaviour is different than when using other access protocols for AIP Storage. Upon deposit in DSpace, the AIP will be deposited as a single compressed objects file (bitstream), which contains all original and normalized objects as well as the metadata pertaining to them. The DIP will be deposited as uncompressed files with the folder structure flattened.

Presently, the Storage Service and Dashboard are not capable of downloading/reconstituting the AIP - this must be done manually from the DSpace interface.

The DSpace via REST API space offers a novel way to facilitate increased automation. It is possible to supply metadata with the transfer as either a CSV or a JSON file. In the entry for the root folder objects it is possible to define any number of Dublin Core Metadata Elements Set properties which will map on to the corresponding DSpace record created for the AIP/DIP. In addition there are three optional custom properties which can be defined:

• dspace_dip_collection: the UUID of the DSpace collection into which the DIP will be deposited. E.g. a12d749c-7727-4121-b6be-478cacde658f
• dspace_aip_collection: the UUID of the DSpace collection into which the AIP will be deposited. E.g. 80c3519d-7a07-4830-beae-a868c149ecbe
• archivesspace_dip_collection: the identifier of the archival object for which a child digital object will be created which will link to the DSpace DIP record. E.g. 135569

Fields:

• Size: the maximum size allowed for this space. Leave blank for unlimited. This field is optional.
• Path: the absolute path to the Space on the local filesystem.
• Staging path: the absolute path to a staging area. Must be UNIX filesystem compatible and preferably will be located on the same filesystem as the path.
• REST URL: URL to the “REST” webapp. E.g. http://localhost:8080/rest/; for production systems, this address will be slightly different, such as: https://demo.dspace.org/rest/.
• User: a username for the DSpace instance with sufficient permissions to permit authentication.
• Password: the password for the username above.
• Default DSpace DIP collection id: the UUID of the collection into which the DIP will be deposited barring it being designated in a transfer metadata file.
• Default DSpace AIP collection id: the UUID of the collection into which the AIP will be deposited barring it being designated in a transfer metadata file.

• ArchivesSpace URL: URL to the ArchiveSpace server. E.g.: http://sandbox.archivesspace.org/.
• ArchivesSpace user: ArchivesSpace username to authenticate as
• ArchivesSpace password: ArchivesSpace password to authenticate with
• Default ArchivesSpace repository: Identifier of the default ArchivesSpace repository
• Default ArchivesSpace archival object: Identifier of the default ArchivesSpace archival object barring it being designated in a transfer metadata file
• Send AIP to Tivoli Storage Manager: this is a feature specific to the requirements of Edinburgh University which sponsored the development of this space. Essentially it executes a bash command using a binary called dsmc.
• Verify SSL certificates: Requests verifies SSL certificates for HTTPS requests, just like a web browser. By default, SSL verification is enabled, and Requests will throw a SSLError if it’s unable to verify the certificate:

DSpace via SWORD2 API¶

DSpace via SWORD2 locations are currently supported only for AIP Storage locations. Because DSpace is typically used as a public-facing system, the behaviour is different than when using other access protocols for AIP Storage: upon deposit in DSpace, the AIP will be split into two parts:

• a compressed objects file (bitstream), which contains all original and normalized objects
• as well as a metadata file (bitstream), which contains all of the bag artifacts, metadata and logs

The metadata bitstream can optionally be restricted; see below. Presently, the Storage Service and Dashboard are not capable of downloading/reconstituting the AIP - this must be done manually from the DSpace interface.

If using DSpace as the AIP location in conjunction with the ArchivesSpace workflow in the Appraisal tab, a post Store AIP hook will send the DSpace handle to the ArchivesSpace digital object record upon AIP storage. The ArchivesSpace configuration is set up in the Dashboard administration tab.

Fields:

• Size: the maximum size allowed for this space. Leave blank for unlimited. This field is optional.
• Path: the absolute path to the Space on the local filesystem.
• Staging path: the absolute path to a staging area. Must be UNIX filesystem compatible and preferably will be located on the same filesystem as the path.
• Service Document IRI: URL of the service document. E.g. http://demo.dspace.org/swordv2/servicedocument, where servicedocument is the handle to the community or collection being used for deposit.
• User: a username for the DSpace instance with sufficient permissions to permit authentication.
• REST URL: URL to the “REST” webapp. E.g. http://localhost:8080/rest/; for production systems, this address will be slightly different, such as: https://demo.dspace.org/rest/.
• Password: the password for the username above.
• Restricted metadata policy: Use to restrict access to the metadata bitstream. Must be specified as a list of objects in JSON, e.g. [{"action":"READ","groupId":"5","rpType":"TYPE_CUSTOM"}]. This will override existing policies.

Fedora via SWORD2¶

Fedora via SWORD2 is currently supported in the Storage Service as an Access Protocol to facilitate use of the Archidora plugin, which allows ingest of material from Islandora to Archivematica. This workflow is in beta testing as of Storage Service 0.9/Archivematica 1.5/Islandora 7.x-1.6.

Fields:

• Size: the maximum size allowed for this space. Leave blank for unlimited. This field is optional.
• Path: the absolute path to the Space on the local filesystem.
• Staging path: the absolute path to a staging area. Must be UNIX filesystem compatible and preferably will be located on the same filesystem as the path.
• Fedora user: Fedora user name (for SWORD functionality).
• Fedora password: Fedora password (for SWORD functionality).
• Fedora name: Name or IP of the remote Fedora machine.

GPG encryption on local file system¶

Creating a GPG encryption space will allow users to create encrypted AIPs and transfers. Only AIP storage, Transfer backlog and Replicator locations can be created in a GPG encryption space.

Encrypted AIPs and transfers can be downloaded unencrypted via the Storage Service and Archivematica dashboard.

Before creating a GPG encryption space ensure that you have created or imported a GPG key on the Administration page.

Fields:

• Size: the maximum size allowed for this space. Leave blank for unlimited. This field is optional.
• Path: the absolute path to the Space on the local filesystem.
• Staging path: the absolute path to a staging area. Must be UNIX filesystem compatible and preferably will be located on the same filesystem as the path.
• Key: the encryption key to be used for the space.

Local Filesystem¶

Local Filesystem spaces handle storage that is available locally on the machine running the Storage Service. Typically this is the hard drive, SSD or raid array attached to the machine, but it could also encompass remote storage that has already been mounted. For remote storage that has been locally mounted, we recommend using a more specific Space if one is available.

Fields:

• Size: the maximum size allowed for this space. Leave blank for unlimited. This field is optional.
• Path: the absolute path to the Space on the local filesystem.
• Staging path: the absolute path to a staging area. Must be UNIX filesystem compatible and preferably will be located on the same filesystem as the path.

LOCKSS¶

Archivematica can store AIPs in a LOCKSS network via LOCKSS-O-Matic, which uses SWORD to communicate between the Storage Service and a Private LOCKSS Network (PLN).

Fields:

• Size: the maximum size allowed for this space. Leave blank for unlimited. This field is optional.
• Path: the absolute path to the Space on the local filesystem.
• Staging path: the absolute path to a staging area. Must be UNIX filesystem compatible and preferably will be located on the same filesystem as the path.
• Service document IRI: the URL of the LOCKSS-o-matic service document IRI, e.g. http://lockssomatic.example.org/api/sword/2.0/sd-iri.
• Content Provider ID: the On-Behalf-Of value when communicating with LOCKSS-o-matic.
• Externally available domain: the base URL for this server that LOCKSS will be able to access. Generally this is the URL for the home page of the Storage Service.
• Keep local copy?: check this box if you wish to store a local copy of the AIPs even after they are stored in LOCKSS.

NFS¶

NFS spaces are for NFS exports mounted on the Storage Service server and the Archivematica pipeline.

Fields:

• Size: the maximum size allowed for this space. Leave blank for unlimited. This field is optional.
• Path: the absolute path to where the space is mounted on the filesystem local to the Storage Service.
• Staging path: the absolute path to a staging area. Must be UNIX filesystem compatible and preferably will be located on the same filesystem as the path.
• Remote name: the hostname or IP address of the remote computer exporting the NFS mount.
• Remote path: the export path on the NFS server
• Version: the version of the filesystem, e.g. nfs or nfs4,as would be passed to the mount command.
• Manually mounted: This is a placeholder for a feature that is not yet available.

Pipeline Local Filesystem¶

Pipeline Local Filesystems refer to the storage that is local to the Archivematica pipeline, but remote to the Storage Service. For this Space to work properly, passwordless SSH must be set up between the Storage Service host and the Archivematica host.

For example, the Storage Service is hosted on storage_service_host and Archivematica is running on archivematica1. The transfer sources for Archivematica are stored locally on archivematica1, but the Storage Service needs access to them. The Space for that transfer source would be a Pipeline Local Filesystem.

Fields:

• Size: the maximum size allowed for this space. Leave blank for unlimited. This field is optional.
• Path: the absolute path to where the space is mounted on the filesystem local to the Storage Service.
• Staging path: the absolute path to a staging area. Must be UNIX filesystem compatible and preferably will be located on the same filesystem as the path.
• Remote user: the username on the remote host.
• Remote name: the hostname or IP address of the computer running Archivematica. This should be SSH accessible from the Storage Service computer.
• Assume remote host serving files with rsync daemon: if checked, the Storage Service will use rsync daemon-style commands instead of the default rsync with remote shell.
• Rsync password: the password for the rsync daemon

RClone¶

rclone is a command-line program to manage files in cloud storage, and is available as an access protocol in Storage Service 0.20 and higher.

The RClone space allows for use of over 40 cloud providers with Archivematica as Transfer Source, AIP Store, DIP Store, and Replicator locations. Configuration of details such as access keys can be done with a configuration file or via environment variables (the recommended method). See the rclone documentation on configuration via environment variables.

Fields:

• Access protocol: RClone
• Size: the maximum size allowed for this space. Leave blank for unlimited. This field is optional.
• Path: Leave this field blank.
• Staging path: A location on your local disk where Archivematica can place files for staging purposes, for example var/archivematica/storage_service/rclone_staging.
• Remote name: Remote name for the rclone configuration to use with this Space. Must match value in environment variables, case-insensitive.
• Container/Bucket name: Container or bucket name to use in configured remote (optional, depending on service being used via rclone).

Swift¶

OpenStack’s Swift is available as an access protocol in Storage Service 0.7 and higher.

Fields:

• Size: the maximum size allowed for this space. Leave blank for unlimited. This field is optional.
• Path: the absolute path to the Space on the local filesystem.
• Staging path: the absolute path to a staging area. Must be UNIX filesystem compatible and preferably will be located on the same filesystem as the path.
• Auth URL: the URL to authenticate against.
• Auth version: the OpenStack authentication version.
• Username: the Swift username that will be used for authentication.
• Password: the password for the above username.
• Container: the name of the Swift container. To list available containers in your Swift installation, run swift list from the command line.
• Tenant: the tenant/account name, required when connecting to an auth 2.0 system.
• Region: the region in Swift. This field is optional.

S3 (Amazon)¶

Amazon S3 is available as an access protocol as of Storage Service version 0.12. Locations within S3 can be used as AIP Storage, DIP Storage, Replicator and Transfer Source locations.

Fields:

• Size: the maximum size allowed for this space. Leave blank for unlimited. This field is optional.
• Path: Not required: see below.
• Staging path: the absolute path to a staging area on the same server as the Storage Service.
• S3 Endpoint URL: the URL of the S3 endpoint, e.g. https://s3.amazonaws.com.
• Access Key ID to authenticate: the public key generated by S3. Not required: see below
• Secret Access Key to authenticate with: the secret key generated by S3. Not required: see below
• Region: the region that the S3 instance uses, e.g. us-east-2.
• S3 Bucket: the name of the designated S3 Bucket. This field is optional.

s3
└── d6d618dd-2b7f-4177-8b59-10e242066cb7
    └── storage-space-path
        └── storage-location-relative-path
            └── cd20
                └── f886
                    │
                    └── ...additional pair-tree folders...
                        │
                        └── f1f1
                            └── cd20f886-0c40-4202-af80-399b6ca9f1f1.7z

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied
ERROR     2024-03-04 12:56:10  django.request:log:log_response:241:  Internal Server Error: /api/v2/location/34664de6-025f-40a2-87f5-8720ce51169d/browse

botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
ERROR     2024-03-04 12:59:36  django.request:log:log_response:241:  Internal Server Error: /api/v2/file/bf96a036-2631-4fb6-bcdb-781d7690163e/download/

botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
ERROR     2024-03-04 12:59:19  django.request:log:log_response:241:  Internal Server Error: /api/v2/file/bf96a036-2631-4fb6-bcdb-781d7690163e/extract_file/

locations.models.StorageException: An error occurred (AccessDenied) when calling the GetBucketLocation operation: Access Denied
ERROR     2024-03-04 13:03:04  django.request:log:log_response:241:  Internal Server Error: /api/v2/file/

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
ERROR     2024-03-04 13:06:26  django.request:log:log_response:241:  Internal Server Error: /api/v2/file/

ERROR     2024-03-05 10:20:41  django.request:log:log_response:241:  Internal Server Error: /packages/package_delete_request/
Traceback (most recent call last):
...
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the DeleteObject operation: Access Denied

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the CreateBucket operation: Access Denied
ERROR     2024-03-05 08:11:34  django.request:log:log_response:241:  Internal Server Error: /api/v2/file/

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:DeleteObject",
        "s3:GetBucketLocation"
      ],
      "Resource": [
        "arn:aws:s3:::mybucket/*",
        "arn:aws:s3:::mybucket"
      ]
    }
  ]
}

"boto3": {"level": "INFO"},
"botocore": {"level": "INFO"}

Write-Only Replica Staging on local filesystem¶

Write-Only Replica Staging spaces allow users to stage AIP replicas on a local filesystem for delivery to offline storage systems such as tape robots. Only Replicator locations can be created in a Write-Only Replica Staging space.

Fields:

• Size: the maximum size allowed for this space. Leave blank for unlimited. This field is optional.
• Path: the absolute path to the Space on the local filesystem.
• Staging path: the absolute path to a staging area. Must be UNIX filesystem compatible and preferably will be located on the same filesystem as the path.

AIP Recovery¶

AIP Recovery locations are where the AIP recovery feature looks for an AIP to recover. No more than one AIP recovery location should be associated with a given pipeline. The default value is /var/archivematica/storage_service/recover in a Local Filesystem. This is only required if AIP recovery is used.

AIP Storage¶

AIP Storage locations are where the completed AIPs are placed for long-term storage. For pipelines using the default locations, the default path is `/var/archivematica/sharedDirectory/www/AIPsStore in a Local Filesystem. An AIP storage location is required to store and retrieve AIPs.

Currently Processing¶

Archivematica uses the Currently Processing location associated with that pipeline to store materials during active processing. Exactly one currently processing location should be associated with a given pipeline. For pipelines using the default locations, the default path is /var/archivematica/sharedDirectory in a Local Filesystem. A currently processing location is required for Archivematica to run.

DIP Storage¶

The DIP Storage location is used for storing DIPs until such a time that they can be uploaded to an access system. For pipelines using the default locations, the default path is /var/archivematica/sharedDirectory/www/DIPsStore in a Local Filesystem. A DIP storage location is required to store and retrieve DIPs, but it is not required to upload DIPs to access systems.

FEDORA Deposit¶

A FEDORA Deposit location is used with the Archidora plugin to ingest material from Islandora. This is only available to the FEDORA Space, and is only required for that space.

Storage Service Internal Processing¶

There should only be one Storage Service Internal Processing location for a Storage Service installation. For pipelines using the default locations, the default path is /var/archivematica/storage_service in a Local Filesystem. This is required for the Storage Service to run, and must be locally available to the Storage Service. It should not be associated with any pipelines.

Transfer Backlog¶

Transfer Backlog locations store transfers until the user continues processing them. For pipelines using the default locations, the default path is /var/archivematica/sharedDirectory/www/AIPsStore/transferBacklog in a Local Filesystem. This is required to store and retrieve transfers in backlog.

Transfer Source¶

A list of Transfer source locations are displayed in the transfer source dropdown on the Archivematica pipeline’s Transfer tab. Any folder in a transfer source can be selected to become a transfer. The default value is /home in a local filesystem.

Replicator¶

Replicator locations can be configured to replicate the AIPs in one or more AIP storage locations. Replicators can be configured for each of the following protocols supported by the Storage Service:

• Duracloud.
• Local filesystem, including GPG encrypted and write-only replica staging.
• S3.

Replicators are associated with an AIP storage location via the storage location’s configuration page. As such Replicators must be configured before they can be selected for use alongside an AIP store. There is additional information about this under configure location. If you would like your replicated AIPs to be encrypted, create the location in an encrypted space.

Recovery request¶

These fields allow you to set up notifications for events related to the AIP deletion workflow - for example, deletion approvals or rejections.

Fields:

• Recovery request: URL to notify: the server that should receive the JSON-encoded event message from the Storage Service.
• Recovery request notification: Username (optional): A username for basic access authentication.
• Recovery request notification: Password (optional) A password for basic access authentication.

Default locations¶

The default location settings allow you to define default locations for any new pipeline that is registered with the Storage Service. You can define default locations for the following:

• Transfer source
• AIP storage
• DIP storage
• Transfer backlog
• AIP recovery

Multiple transfer source or AIP storage locations can be configured by holding down Ctrl when selecting them.

A Currently Processing location is also created for every new pipeline, since one is required.