Difference between revisions of "Quote/Unix has a uniform interface"
Jump to navigation
Jump to search
(Created page with "{{Blockquote |text=In the Unix world, the unifrom interface that allows one program to be composed with another is files and pipes; in MapReduce, that interface is a distribut...") |
|||
| Line 1: | Line 1: | ||
{{Blockquote | {{Blockquote | ||
|text=In the Unix world, the unifrom interface that allows one program to be composed with another is files and pipes; in MapReduce, that interface is a distributed filesystem. We saw that dataflow engines add their own pipe-like data transport mechanisms to avoid materializing intermediate state to the distributed filesystem, but the initial input and final output of a job is still usually HDFS. | |text=In the Unix world, the unifrom interface that allows one program to be composed with another is files and pipes; in MapReduce, that interface is a distributed filesystem. We saw that dataflow engines add their own pipe-like data transport mechanisms to avoid materializing intermediate state to the distributed filesystem, but the initial input and final output of a job is still usually HDFS. | ||
|sign=[[Martin Kleppmann]]ref>{{:Book/Designing Data-Intensive Applications}}, Chapter 10: Batch Processing, Page 429</ref> | |sign=[[Martin Kleppmann]]<ref>{{:Book/Designing Data-Intensive Applications}}, Chapter 10: Batch Processing, Page 429</ref> | ||
}} | }} | ||
<noinclude> | <noinclude> | ||
Latest revision as of 10:38, 16 July 2022
In the Unix world, the unifrom interface that allows one program to be composed with another is files and pipes; in MapReduce, that interface is a distributed filesystem. We saw that dataflow engines add their own pipe-like data transport mechanisms to avoid materializing intermediate state to the distributed filesystem, but the initial input and final output of a job is still usually HDFS.
References
- ↑ Kleppmann, Martin (Mar 1, 2017). Designing Data-Intensive Applications:The Big Ideas Behind Reliable, Scalable, and Maintainable Systems. local page: O'reily Press. , Chapter 10: Batch Processing, Page 429
Related Pages
Authored by:Martin Kleppmann