Scaling Archivematica¶

Limit task threads in MCPServer¶

The MCPServer can accept instructions to process work from multiple sources (e.g. the Dashboard, API, a watched directory) and will create new threads as new instructions are received, up to a defined limit. The limit helps to mitigate the risk of concurrency errors (conflicts or contention when the machine is trying to do too many things at once).

The ARCHIVEMATICA_MCPSERVER_PROTOCOL_LIMITTASKTHREADS parameter defines the maximum number of threads that MCPServer will run simultaneously. The default value is 75. This number could be increased on machines with a large number of CPUs or decreased on machines with very few (2) CPUs. See MCPServer for details.

Deploy multiple MCPClients¶

The MCPClient component is responsible for the most significant sources of load in an Archivematica deployment. In Archivematica 1.8.x and higher, it can in some cases make sense to deploy more than one MCPClient component on the same machine, due to the introduction of batching.

The MCPClient accepts batches of files to process at once. Batches are processed using a single process by default, but some client scripts can run multi-threaded or fork processes where possible (see MCPClient concurrency for more details). This approach is a trade-off that delivers significant improvements in most circumstances.

However, there will be times when certain tasks are not run concurrently, and CPU becomes underutilized. Deploying multiple instances of the MCPClient component may improve overall performance, particularly where a machine has many CPUs.

To deploy another instance of MCPClient on the same machine we recommend creating an additional systemd unit. A second unit can be created by creating another unit file for the MCPClient service. For example, in CentOS the existing MCPClient unit file is located in /etc/systemd/system/archivematica-mcp-client.service. To add a unit, create a copy of that unit file with a different name, such as/etc/systemd/system/archivematica-mcp-client-two.service and ask systemd to reload all unit files using $ systemctl daemon-reload.

Optimize batch size for large files¶

The ARCHIVEMATICA_MCPSERVER_MCPSERVER_BATCH_SIZE parameter defines the maximum number of files that will be included in any one batch. The default value is 128, which is the threshold at which increasing batch size does not deliver further performance gains.

When processing large files, smaller batch sizes may improve performance when there is more than one instance of MCPClient available. For example, processing large audio-visual files can take significant resources and time per file. A transfer with 100 files in it will all get included in one batch assuming the default batch size of 128. Reducing the batch size to 25 would result in 4 batches, which would all run concurrently if there were 4 instances of the MCPClient available.

Currently we do not have detailed recommendations on for optimizations of this type. We encourage testing and evaluation to find the best settings for your circumstances.

To set the batch size parameter, see MCPServer Configuration.

Distributing components on multiple machines¶

Optimizing settings across machines¶

Optimising what and how preservation actions get executed¶

The final strategy for improving the performance and capacity of your Archivematica deployment is to ensure that Archivematica is only carrying out the work you deem important and necessary. Archivematica provides a wide range of preservation actions and the default settings tend to make use of the majority of them. There are several techniques for limiting which actions are taken, that can have a significant impact on the overall time and compute required to process a particular Transfer or SIP.

Environment configuration options¶

Processing configuration options¶

The Processing configuration screen provides many options for controlling what actions Archivematica performs. The following settings can improve performance.

Select file format identification command (Transfer): Using Siegfried for file identification has been shown to be faster than Fido in this benchmarking study.

Select file format identification command (Ingest): If you are using the Archivematica backlog and have accumulated items in the backlog for a long period of time, e.g. months or even years, then you might want Archivematica to refresh its file format identification at the Ingest stage of the workflow. If Ingest is happening shortly after Transfer, selecting Use existing data should be adequate and will save processing time.

Generate thumbnails: If you are generating thumbnails, selecting yes without defaults will only generate thumbnail images for formats that have a specific thumbnail rule defined. The default thumbnail rule only produces a grey icon and for many formats has little value, but in transfers with many files, can take significant processing time. It is also possible to disable generation of thumbnails entirely by selecting no.

Select compression algorithm: AIP compression causes an AIP to be put into a container (e.g. a 7Zip container). Using containers makes AIP storage and transfer easier because the AIP is easier to move around as a single file. The AIP file size also has the potential to be reduced, which saves storage space and speeds up transfers to external AIP stores. The disadvantage is that compression can take significant processing time and resources. AIP compression introduces three extra steps in the workflow: compression to create the container, then decompression to allow for a final checksum validation step. In transfers with very large numbers of files (thousands) we have seen significant performance improvements by not compressing the AIP.

Select compression level: Selecting a higher compression level means that the resulting AIP is smaller, but compression also takes longer. Lower compression levels mean quicker compression, but a larger AIP.

Preservation Action Rules¶

Some of the default preservation action rules can take considerable processing time and resources. We have found the following rules useful to change in some cases.

Turn off default characterization rule: FITS is used to characterise files that don’t have a recognised file format. Executing this rule takes processing time and adds raw output to the METS file that can be low value for some formats. For example, in scientific datasets with large numbers of generic text files, or binary files created by instruments in scientific experiments, the output can be verbose without being useful.

Reduce number of image characterization rules: Archivematica has rules defined for all image and audio-visual formats to use ExifTool, Mediainfo and ffprobe for characterisation. Using multiple tools ensures as much characterization output as possible, but also introduces some level of duplication. Only using one of the three tools for certain formats may provide an adequate level of characterization with the benefit of reducing processing time and the size of the final AIP.

See Characterization and Altering commands and rules for more details.

General Configuration Settings¶

Checksum Algorithm: In the General settings screen you can select which checksum algorithm Archivematica will use during the Assign UUIDs and checksums microservice. For the purposes of fixity checking, the MD5 algorithm may be adequate and takes less processing time to create (and check) than the alternatives (e.g. SHA-256).

Back to the top