Storage Data Flow Management based on Streaming Machine Learning
Hybrid data storage systems, if utilized correctly, can be instrumental in meeting the increasing data storage and I/O demands of modern large-scale data analytics and HPC workloads. However, the complexity of data movement across the storage tiers and caches increases significantly, making it harder for applications to take advantage of the higher I/O performance offered by the system. The general objective of this project is to automate data flow management for caching and storage tiering in hybrid data storage solutions, while relying on newly-developed artificial intelligent algorithms, in order to achieve optimal performance to cost ratio under different capacity storage media. The proposed methodology involves combining mathematical modeling with streaming machine learning for the first time, for guiding the decisions of data storage systems. The expected outcome of the project will be instrumental in meeting the increasing data storage and I/O demands of modern large-scale data analytics and high performance computing workloads, as it could offer sustainable high performance with lower cost.
At the same time, DITIS, a new simulator for distributed multi-tiered data storage systems is developed that simulates the end-to-end execution of file system requests through the different layers and storage nodes of the system using numerous pluggable policies that control every aspect of the execution. DITIS can be configured with different numbers of storage tiers, caches, nodes, and media devices (e.g., HDD, SSD, NVRAM, DRAM), where each media can have its own performance characteristics guiding fine-grained performance cost models. DITIS' architecture is based on the Actor Model, where each key component can exchange synchronous or asynchronous messages with each other, much like a real distributed multi-threaded system. As a result, developers can use DITIS to narrow down the design spaces, evaluate design trade-offs, develop, test, and evaluate different setups and policies, and reduce prototyping efforts, while end users can use it to better understand the system's behavior and identify the system configuration that best satisfies their requirements.
