Simplify Large-scale Data Management with a Unified Data Mover

Abstract

Data management is the practice of collecting, moving, keeping, and using data in a secure, efficient, and cost-effective manner. Note that all such activities involve data movement. When the scale is large, more challenges appear. The DOE Technical Report “Data Movement Categories” classifies the commonly seen data movement needs, as observed in data-intensive institutions (both commercial and non-profit), into four categories. We believe that the fourth category, “Strictly interactive data movements and end user oriented” can be satisfactorily solved using freeware on the server side, and freeware and/or low-cost applications for end users. Nevertheless, our view is that the other three categories should be tackled using a unified data mover. This view is motivated by the observation that the various large-scale and complex data management tasks involving data movement at a distributed data-intensive organization can be much simplified with such a data mover – it offers simpler usage, more consistent operation, and enhanced efficiency. Zettar zx has been designed and implemented with such a vision in mind. The motivation for each category, a brief history of the software’s evolution, and its current capabilities are explained and illustrated using past trials and live demos. Comparison with other common data movers, whenever results are available, are provided. Possible future evolution is also described.

Bio

Chin Fang is the founder & CEO of Zettar Inc. (Zettar). He holds a master’s and a doctoral engineering degree from Stanford University. Zettar’s mission is to create a simple, scale-out capable, and efficient unified data mover software for moving data at scale and speed. Targeted users include modern distributed data-intensive enterprises and large science projects such as the Linac Coherent Light Source II (LCLS-II). In March 2019, Zettar was awarded the only Overall Winner of the Supercomputing Asia Data Mover Challenge, a grueling two-month long international competition at the highest level. Dr. Fang led the Zettar team, which beat out six other elite national teams from the U.S. and Japan by a wide margin. Since 2015, Zettar has been engaged to support the ambitious data movement requirements of LCLS-II, a premier U.S. DOE Exascale Computing preparation project. Zettar is supported by its revenue and has been awarded highly competitive grants by two U.S. federal agencies, the National Science Foundation and the Department of Energy.