Thrill  0.1
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
thrill::data Namespace Reference

Namespaces

 detail
 
 serialization_cereal
 

Classes

class  Block
 Block combines a reference to a read-only ByteBlock and book-keeping information. More...
 
class  BlockPool
 Pool to allocate, keep, swap out/in, and free all ByteBlocks on the host. More...
 
class  BlockPoolMemoryHolder
 RAII class for allocating memory from a BlockPool. More...
 
class  BlockQueue
 A BlockQueue is a thread-safe queue used to hand-over Block objects between threads. More...
 
class  BlockQueueSink
 BlockSink which interfaces to a File. More...
 
class  BlockReader
 BlockReader takes Block objects from BlockSource and allows reading of a) serializable Items or b) arbitray data from the Block sequence. More...
 
class  BlockSink
 Pure virtual base class for all things that can receive Blocks from a BlockWriter. More...
 
class  BlockWriter
 BlockWriter contains a temporary Block object into which a) any serializable item can be stored or b) any arbitrary integral data can be appended. More...
 
class  BoundedBlockSink
 Derivative BlockSink which counts and limits how many bytes it has delivered as ByteBlocks for writing. More...
 
class  ByteBlock
 A ByteBlock is the basic storage units of containers like File, BlockQueue, etc. More...
 
class  CacheBlockQueueSource
 A BlockSource to read Blocks from a BlockQueue using a BlockReader, and at the same time CACHE all items received. More...
 
class  CatBlockSource
 CatBlockSource is a BlockSource which concatenates all Blocks available from a vector of BlockSources. More...
 
class  CatStream
 Ownership handle onto a CatStreamData. More...
 
class  CatStreamData
 A Stream is a virtual set of connections to all other worker instances, hence a "Stream" bundles them to a logical communication context. More...
 
class  ConsumeBlockQueueSource
 A BlockSource to read Block from a BlockQueue using a BlockReader. More...
 
class  ConsumeFileBlockSource
 A BlockSource to read and simultaneously consume Blocks from a File. More...
 
class  DiscardSink
 DiscardSink is an BlockSink that discards all Blocks delivered to it. More...
 
class  DynBlockSource
 This is the actual BlockSource used to instantiate BlockReader. More...
 
class  DynBlockSourceAdapter
 Adapter class to wrap any existing BlockSource concept class into a DynBlockSourceInterface. More...
 
class  DynBlockSourceInterface
 This is a pure virtual base which will be used to fetch Blocks for the BlockReader from different sources. More...
 
class  File
 A File is an ordered sequence of Block objects for storing items. More...
 
class  FileBlockSink
 BlockSink which interfaces to a File. More...
 
class  FullException
 An Exception is thrown by BlockWriter when the underlying sink does not allow allocation of a new block, which is needed to serialize the item. More...
 
class  KeepFileBlockSource
 A BlockSource to read Blocks from a File. More...
 
class  MixBlockQueue
 Implements reading an unordered sequence of items from multiple workers, which sends Blocks. More...
 
class  MixBlockQueueReader
 Reader to retrieve items in unordered sequence from a MixBlockQueue. More...
 
class  MixStream
 Ownership handle onto a MixStream. More...
 
class  MixStreamData
 A Stream is a virtual set of connections to all other worker instances, hence a "Stream" bundles them to a logical communication context. More...
 
class  Multiplexer
 Multiplexes virtual Connections on Dispatcher. More...
 
class  MultiplexerHeader
 
class  PartitionMultiplexerHeader
 
class  PinnedBlock
 A pinned / pin-counted derivative of a Block. More...
 
class  PinnedByteBlockPtr
 A pinned / pin-counted pointer to a ByteBlock. More...
 
class  PinRequest
 
struct  Serialization
 
struct  Serialization< Archive, core::HyperLogLogRegisters< p > >
 
struct  Serialization< Archive, std::array< T, N >, typename std::enable_if< !std::is_pod< std::array< T, N > >::value >::type >
 
struct  Serialization< Archive, std::pair< U, V > >
 
struct  Serialization< Archive, std::string >
 
struct  Serialization< Archive, std::tuple< Args...> >
 
struct  Serialization< Archive, std::vector< T > >
 
struct  Serialization< Archive, T, typename std::enable_if< cereal::traits::is_input_serializable< T, Archive >::value &&!std::is_pod< T >::value &&!std::is_same< T, std::string >::value &&!tlx::is_std_pair< T >::value &&!tlx::is_std_array< T >::value &&!tlx::is_std_vector< T >::value &&!tlx::is_std_tuple< T >::value >::type >
 
struct  Serialization< Archive, T, typename std::enable_if< has_member_thrill_is_fixed_size< T >::value >::type >
 
struct  Serialization< Archive, T, typename std::enable_if< std::is_pod< T >::value &&!std::is_pointer< T >::value >::type >
 
class  Stream
 Stream - base class for CatStream and MixStream. More...
 
class  StreamData
 Base class for common structures for ConcatStream and MixedStream. More...
 
class  StreamMultiplexerHeader
 Block header is sent before a sequence of blocks it indicates the number of elements and their boundaries. More...
 
class  StreamSet
 Simple structure that holds a all stream instances for the workers on the local host for a given stream id. More...
 
class  StreamSetBase
 Base class for StreamSet. More...
 
class  StreamSink
 StreamSink is an BlockSink that sends data via a network socket to the StreamData object on a different worker. More...
 

Typedefs

using Byte = uint8_t
 type of underlying memory area More...
 
using ByteBlockPtr = ByteBlock::ByteBlockPtr
 
using CatStreamDataPtr = tlx::CountingPtr< CatStreamData >
 
using CatStreamPtr = tlx::CountingPtr< CatStream >
 
using CatStreamSet = StreamSet< CatStreamData >
 
using CatStreamSetPtr = tlx::CountingPtr< CatStreamSet >
 
using DiscardWriter = BlockWriter< DiscardSink >
 
using DynBlockReader = BlockReader< DynBlockSource >
 Instantiation of BlockReader for reading from the polymorphic source. More...
 
using FilePtr = tlx::CountingPtr< File >
 
using MixStreamDataPtr = tlx::CountingPtr< MixStreamData >
 
using MixStreamPtr = tlx::CountingPtr< MixStream >
 
using MixStreamSet = StreamSet< MixStreamData >
 
using MixStreamSetPtr = tlx::CountingPtr< MixStreamSet >
 
using PinRequestPtr = tlx::CountingPtr< PinRequest, mem::GPoolDeleter< PinRequest > >
 
using ReadingMap = std::unordered_map< ByteBlock *, PinRequestPtr, std::hash< ByteBlock * >, std::equal_to<>, mem::GPoolAllocator< std::pair< ByteBlock *const, PinRequestPtr > > >
 type of set of ByteBlocks currently begin read from EM. More...
 
using StreamDataPtr = tlx::CountingPtr< StreamData >
 
using StreamId = size_t
 
using WritingMap = std::unordered_map< ByteBlock *, io::RequestPtr, std::hash< ByteBlock * >, std::equal_to<>, mem::GPoolAllocator< std::pair< ByteBlock *const, io::RequestPtr > > >
 type of set of ByteBlocks currently begin written to EM. More...
 

Enumerations

enum  MagicByte : uint8_t { Invalid, CatStreamBlock, MixStreamBlock, PartitionBlock }
 

Functions

template<typename BlockSource , typename... Params>
DynBlockReader ConstructDynBlockReader (Params &&...params)
 Method to construct a DynBlockReader from a non-polymorphic BlockSource. More...
 
template<typename BlockSource , typename... Params>
DynBlockSource ConstructDynBlockSource (Params &&...params)
 Method to construct a DynBlockSource from a non-polymorphic BlockSource. More...
 
std::ostream & operator<< (std::ostream &os, const Block &b)
 
std::ostream & operator<< (std::ostream &os, const PinnedBlock &b)
 
std::ostream & operator<< (std::ostream &os, const ByteBlock &b)
 
std::ostream & operator<< (std::ostream &os, const File &f)
 
std::ostream & operator<< (std::ostream &os, const BlockPool::PinCount &p)
 
static void OurNewHandler ()
 
template<typename Reader >
void StartPrefetch (std::vector< Reader > &readers, size_t prefetch)
 Take a vector of Readers and prefetch equally from them. More...
 
 TLX_MAKE_HAS_MEMBER (thrill_is_fixed_size)
 

Variables

static constexpr bool debug_alloc = false
 debug block memory alloc and dealloc More...
 
static constexpr bool debug_blc = false
 debug block life cycle output: create, destroy More...
 
static constexpr bool debug_em = false
 debug block eviction: evict, write complete, read complete More...
 
static constexpr bool debug_mem = false
 debug memory requests More...
 
static constexpr bool debug_pin = false
 debug block pinning: More...
 
size_t default_block_size = 2 * 1024 * 1024
 default size of blocks in File, Channel, BlockQueue, etc. More...
 
static std::atomic< bool > in_new_handler
 
static std::vector< BlockPool * > s_blockpools
 
static std::recursive_mutex s_new_mutex
 
size_t start_block_size = 4 * 1024
 starting size of blocks in BlockWriter. More...
 
class
thrill::data::MultiplexerHeader 
TLX_ATTRIBUTE_PACKED
 

Typedef Documentation

using ReadingMap = std::unordered_map< ByteBlock*, PinRequestPtr, std::hash<ByteBlock*>, std::equal_to<>, mem::GPoolAllocator< std::pair<ByteBlock* const, PinRequestPtr> > >

type of set of ByteBlocks currently begin read from EM.

Definition at line 240 of file block_pool.cpp.

using WritingMap = std::unordered_map< ByteBlock*, io::RequestPtr, std::hash<ByteBlock*>, std::equal_to<>, mem::GPoolAllocator<std::pair<ByteBlock* const, io::RequestPtr> > >

type of set of ByteBlocks currently begin written to EM.

Definition at line 233 of file block_pool.cpp.

Function Documentation

std::ostream& thrill::data::operator<< ( std::ostream &  os,
const Block &  b 
)
std::ostream& thrill::data::operator<< ( std::ostream &  os,
const PinnedBlock &  b 
)

Definition at line 53 of file block.cpp.

References Block::byte_block_.

std::ostream& thrill::data::operator<< ( std::ostream &  os,
const ByteBlock &  b 
)
std::ostream& thrill::data::operator<< ( std::ostream &  os,
const File &  f 
)

Definition at line 94 of file file.cpp.

References File::blocks_, and ReferenceCounter::reference_count().

std::ostream& thrill::data::operator<< ( std::ostream &  os,
const BlockPool::PinCount &  p 
)

Definition at line 216 of file block_pool.cpp.

References tlx::join().

static void thrill::data::OurNewHandler ( )
static

Variable Documentation

constexpr bool debug_alloc = false
static

debug block memory alloc and dealloc

Definition at line 42 of file block_pool.cpp.

Referenced by BlockPool::AllocateByteBlock(), BlockPool::DestroyBlock(), BlockPool::OnReadComplete(), and BlockPool::OnWriteComplete().

constexpr bool debug_blc = false
static

debug block life cycle output: create, destroy

Definition at line 39 of file block_pool.cpp.

Referenced by BlockPool::AllocateByteBlock(), BlockPool::DestroyBlock(), and BlockPool::MapExternalBlock().

constexpr bool debug_em = false
static

debug block eviction: evict, write complete, read complete

Definition at line 51 of file block_pool.cpp.

Referenced by BlockPool::DestroyBlock(), BlockPool::OnReadComplete(), BlockPool::OnWriteComplete(), BlockPool::PinBlock(), and BlockPool::~BlockPool().

constexpr bool debug_mem = false
static

debug memory requests

Definition at line 48 of file block_pool.cpp.

Referenced by BlockPool::AdviseFree().

constexpr bool debug_pin = false
static
std::atomic<bool> in_new_handler
static
Initial value:
{
false
}

Definition at line 59 of file block_pool.cpp.

Referenced by OurNewHandler().

std::vector<BlockPool*> s_blockpools
static

Definition at line 57 of file block_pool.cpp.

Referenced by BlockPool::BlockPool(), OurNewHandler(), and BlockPool::~BlockPool().

std::recursive_mutex s_new_mutex
static

Definition at line 56 of file block_pool.cpp.

Referenced by BlockPool::BlockPool(), OurNewHandler(), and BlockPool::~BlockPool().