The California Digital Library supports the Merritt digital preservation system which manages 170 TB of unique content (55 million unique digital files) utilizing 3 cloud storage providers.
The Merritt Digital Preservation system comprises a dozen microservices and supporting services. Our production and stage environments consist of dozens of server instances.
Some microservices are modified in nearly every sprint cycle while other microservices are modified annually. Our team found that it was not cost effective to maintain and patch another fleet of servers for our development environment.
This created 2 challenges for the team:
- How do we develop and test code without a complete development environment?
- How do we safely on board new team members giving them confidence to modify code?
Our solution was to replace our development server environment with a stack of dockerized services. Once the stack was containerized, we discovered that we were able to create 3 variants of our development stack with different persistence strategies for database content and cloud storage.
- Run the docker stack on a single development server instance with semi-persistent content. (The environment is refreshed 2-3 times per year.) This environment is useful for performing some types of capacity testing and query optimization.
- Run the docker stack on a single development server with transient persistence using docker volumes. This is particularly useful for unit testing.
- Run the docker stack on a developer’s desktop with minimal persistence using docker volumes.
The demonstration portion of the presentation will illustrate the operation of each of these stack configurations and the types of testing that we are able to perform against each configuration.