Scaling Archivematica¶

Optimising the number of packages processed at a time¶

MCPServer allows processing of up to a set number of packages concurrently. The ARCHIVEMATICA_MCPSERVER_MCPSERVER_CONCURRENT_PACKAGES parameter defines the maximum number of packages that MCPServer will process simultaneously. The default value is 0.5 * <cpu count>, rounded up. This number could be increased if you are constantly processing transfers, but not seeing full CPU utilization. See MCPServer for more details.

Optimising the number of MCPServer threads¶

MCPServer has two settings for modifying the number of threads run:

• ARCHIVEMATICA_MCPSERVER_MCPSERVER_RPC_THREADS sets the number of “RPC Server” workers available to respond to status requests from other components (such as the Dashboard). The default value is 4. If you are seeing slow status updates in the Dashboard, increasing this value _may_ help.
• ARCHIVEMATICA_MCPSERVER_MCPSERVER_WORKER_THREADS sets the number of threads available to handle MCPServer processing tasks, such as submitting Gearman jobs for processing by MCPClient. The default value is <cpu count> + 1. Increasing this value _may_ help increase CPU utilization.

Deploy multiple MCPClients¶

The MCPClient component is responsible for the most significant sources of load in an Archivematica deployment. In Archivematica 1.8.x and higher, it can in some cases make sense to deploy more than one MCPClient component on the same machine, due to the introduction of batching.

The MCPClient accepts batches of files to process at once. Batches are processed using a single process by default, but some client scripts can run multi-threaded or fork processes where possible (see MCPClient concurrency for more details). This approach is a trade-off that delivers significant improvements in most circumstances.

However, there will be times when certain tasks are not run concurrently, and CPU becomes underutilized. Deploying multiple instances of the MCPClient component may improve overall performance, particularly where a machine has many CPUs.

To deploy another instance of MCPClient on the same machine we recommend creating an additional systemd unit. A second unit can be created by creating another unit file for the MCPClient service. For example, in CentOS the existing MCPClient unit file is located in /etc/systemd/system/archivematica-mcp-client.service. To add a unit, create a copy of that unit file with a different name, such as/etc/systemd/system/archivematica-mcp-client-two.service and ask systemd to reload all unit files using $ systemctl daemon-reload.

Optimise batch size for large files¶

The ARCHIVEMATICA_MCPSERVER_MCPSERVER_BATCH_SIZE parameter defines the maximum number of files that will be included in any one batch. The default value is 128, which is the threshold at which increasing batch size does not deliver further performance gains.

When processing large files, smaller batch sizes may improve performance when there is more than one instance of MCPClient available. For example, processing large audio-visual files can take significant resources and time per file. A transfer with 100 files in it will all get included in one batch assuming the default batch size of 128. Reducing the batch size to 25 would result in 4 batches, which would all run concurrently if there were 4 instances of the MCPClient available.

Currently we do not have detailed recommendations on for optimizations of this type. We encourage testing and evaluation to find the best settings for your circumstances.

To set the batch size parameter, see MCPServer Configuration.

Optimising the number of Dashboard workers¶

The Dashboard process can run multiple worker processes via Gunicorn. The AM_GUNICORN_WORKERS value (default 1) sets the number of workers used. Increasing this value _may_ help improve Dashboard responsiveness. See Dashboard for more details.

Optimising the number of Storage Service workers¶

The Storage Service process can run multiple worker processes via Gunicorn. The SS_GUNICORN_WORKERS value (default 1) sets the number of workers used. Increasing this value _may_ help improve Storage Service responsiveness.

Distributing components on multiple machines¶

Optimising settings across machines¶

Optimising what and how preservation actions get executed¶

The final strategy for improving the performance and capacity of your Archivematica deployment is to ensure that Archivematica is only carrying out the work you deem important and necessary. Archivematica provides a wide range of preservation actions and the default settings tend to make use of the majority of them. There are several techniques for limiting which actions are taken, that can have a significant impact on the overall time and compute required to process a particular Transfer or SIP.

Environment configuration options¶

Processing configuration options¶

The Processing configuration screen provides many options for controlling what actions Archivematica performs. The following settings can improve performance.

Select file format identification command (Transfer): Using Siegfried for file identification has been shown to be faster than Fido in this benchmarking study. See Identification for more information.

Select file format identification command (Ingest): If you are using the Archivematica backlog and have accumulated items in the backlog for a long period of time, e.g. months or even years, then you might want Archivematica to refresh its file format identification at the Ingest stage of the workflow. If Ingest is happening shortly after Transfer, selecting Use existing data should be adequate and will save processing time.

Generate thumbnails: If you are generating thumbnails, selecting yes without defaults will only generate thumbnail images for formats that have a specific thumbnail rule defined. The default thumbnail rule only produces a grey icon and for many formats has little value, but in transfers with many files, can take significant processing time. It is also possible to disable generation of thumbnails entirely by selecting no.

Select compression algorithm: AIP compression causes an AIP to be put into a container (e.g. a 7Zip container). Using containers makes AIP storage and transfer easier because the AIP is easier to move around as a single file. The AIP file size also has the potential to be reduced, which saves storage space and speeds up transfers to external AIP stores. The disadvantage is that compression can take significant processing time and resources. AIP compression introduces three extra steps in the workflow: compression to create the container, then decompression to allow for a final checksum validation step. In transfers with very large numbers of files (thousands) we have seen significant performance improvements by not compressing the AIP.

Select compression level: Selecting a higher compression level means that the resulting AIP is smaller, but compression also takes longer. Lower compression levels mean quicker compression, but a larger AIP.

Preservation action rules¶

Some of the default preservation action rules can take considerable processing time and resources. We have found the following rules useful to change in some cases.

Turn off default characterization rule: FITS is used to characterise files that don’t have a recognised file format. Executing this rule takes processing time and adds raw output to the METS file that can be low value for some formats. For example, in scientific datasets with large numbers of generic text files, or binary files created by instruments in scientific experiments, the output can be verbose without being useful.

Reduce number of image characterization rules: Archivematica has rules defined for all image and audio-visual formats to use ExifTool, Mediainfo and ffprobe for characterisation. Using multiple tools ensures as much characterization output as possible, but also introduces some level of duplication. Only using one of the three tools for certain formats may provide an adequate level of characterization with the benefit of reducing processing time and the size of the final AIP.

See Characterization and Altering commands and rules for more details.

General Configuration Settings¶

Checksum Algorithm: In the General settings screen you can select which checksum algorithm Archivematica will use during the Assign UUIDs and checksums microservice. For the purposes of fixity checking, the MD5 algorithm may be adequate and takes less processing time to create (and check) than the alternatives (e.g. SHA-256).

Back to the top