| WordCount, PageRank, and more... |
| Convenient user interface for writing Big Data algorithms as dataflow graphs with imperative actions. Contains the Context and DIA classes.|
See List of DIA Operation for a comprehensive overview.
| Distributed data structures and algorithms used to build API: Shuffle/Reduce Table, StageBuilder. |
| Manages transfer of large amounts of data between workers. Contains Serialization, File, BlockWriter, Channel, and Multiplexer. || Controls connections between compute nodes, contains collective communication primitives like Broadcast, AllReduce for simple datatypes.|
Backends: net::mock, net::tcp, net::mpi.
| asynchronous I/O, direct file access impl, and disk allocation. (mostly borrowed from STXXL) || for transparent access to POSIX, S3, and in future HDFS. |
| Independent common tools such as Logger, ThreadPool, Delegates, ConcurrentQueue, CmdlineParser, etc. || Keeps track of memory consumption of all stakeholders in system. Extra memory pool for I/O data structures. |