Quote/Unix has a uniform interface
Jump to navigation
Jump to search
In the Unix world, the unifrom interface that allows one program to be composed with another is files and pipes; in MapReduce, that interface is a distributed filesystem. We saw that dataflow engines add their own pipe-like data transport mechanisms to avoid materializing intermediate state to the distributed filesystem, but the initial input and final output of a job is still usually HDFS.
References
- ↑ Kleppmann, Martin (Mar 1, 2017). Designing Data-Intensive Applications:The Big Ideas Behind Reliable, Scalable, and Maintainable Systems. local page: O'reily Press. , Chapter 10: Batch Processing, Page 429
Related Pages
Authored by:Martin Kleppmann