Thrill  0.1
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
Thrill Documentation Overview

Getting Started


WordCount, PageRank, and more...
Convenient user interface for writing Big Data algorithms as dataflow graphs with imperative actions. Contains the Context and DIA classes.
See List of DIA Operation for a comprehensive overview.
Distributed data structures and algorithms used to build API: Shuffle/Reduce Table, StageBuilder.
Manages transfer of large amounts of data between workers. Contains Serialization, File, BlockWriter, Channel, and Multiplexer. Controls connections between compute nodes, contains collective communication primitives like Broadcast, AllReduce for simple datatypes.
Backends: net::mock, net::tcp, net::mpi.
asynchronous I/O, direct file access impl, and disk allocation. (mostly borrowed from STXXL) for transparent access to POSIX, S3, and in future HDFS.
Independent common tools such as Logger, ThreadPool, Delegates, ConcurrentQueue, CmdlineParser, etc. Keeps track of memory consumption of all stakeholders in system. Extra memory pool for I/O data structures.