How do you select your storage solution?

Choosing the storage solution that best matches your needs is key to performance efficiency. The optimal storage solution for a given system will be determined by the following factors: the kind of access method (block, file, or object), patterns of access (random or sequential), required throughput, frequency of access (online, offline, archival) frequency of update (WORM, dynamic), and availability and durability constraints. According to the AWS Best Practices, Well-Architected Systems use a variety of storage solutions with different features meant to increase the efficiency with which they use their resources. In this article, we will discuss the main aspects to take into account when it comes to choosing the best storage solution for your architecture.



Understanding storage characteristics and requirements


When it comes to choosing among object storage, block storage, file storage or instance storage, you first need to understand a set characteristics that will determine your choice, for example, file size, cache size, access patterns, latency, throughput, and persistence of data. To achieve this understanding, you might begin by defining storage performance requirements. As part of a data driven approach, you should implement improvements using benchmarking or load testing. The resulting data will help you identify whether your storage solution is constrained and examine any available configuration options that could potentially improve your solution. In addition, you should be working to determine the expected growth rate of your workload. This information will contribute to choosing a storage solution that will meet those rates in the future. Amazon S3 and Amazon Elastic File System, for example, enable unlimited storage.



Evaluating available configuration options


Selecting the best storage solution is not just a matter of picking the best alternative for you. You also have to take into account available configuration options after you’ve made your choice. You should take your time in order to take into account various characteristics and configuration options and how they, in turn, relate to storage. You should be paying close attention to how to use provisioned IOPS, SSDs. magnetic storage, archival storage, or ephemeral storage to optimize the storage space and performance of your workload in the AWS cloud.



Making decisions based on access patterns and metrics


Storage systems should be selected based on your specific needs. You should be looking at your workload’s access patterns and configure your storage solution based on how your workload accesses data. Storage efficiency can be increased by choosing object storage over block storage. Your data access patterns will inform the configuration options you choose to match your needs.

To optimize your storage usage and access patterns, you should determine what is the best place to store data that will enable your team to meet your own unique requirements while keeping overhead as low as possible. When configuring and interacting with data, you should be using performance optimizations based on the characteristics of the storage solution you selected.

You must also ensure that you’re selecting the relevant storage metrics for your workload. Different storage options offer different metrics to track your workload performance over time. For storage systems that are fixed sized, such as Amazon Elastic Block Store or Amazon FSx, ensure that you are monitoring the amount of storage used versus the overall storage size. Whenever possible, you should be creating automations to increase the storage size when reaching a threshold.


Useful resources:

Cloud Storage with AWS

Deep dive on Amazon EBS (STG303-R1)

Optimize your storage performance with Amazon S3 (STG343)

Amazon EC2 Storage

Amazon EBS Volume Types

EBS I/O Characteristics

Amazon S3: Request Rate and Performance Considerations

Amazon Glacier: Amazon Glacier Documentation

Amazon EFS: Amazon EFS Performance

Amazon FSx for Lustre Performance

Amazon FSx for Windows File Server Performance