Namespaces
	detail

	serialization_cereal

Classes
class	Block
	Block combines a reference to a read-only ByteBlock and book-keeping information. More...

class	BlockPool
	Pool to allocate, keep, swap out/in, and free all ByteBlocks on the host. More...

class	BlockPoolMemoryHolder
	RAII class for allocating memory from a BlockPool. More...

class	BlockQueue
	A BlockQueue is a thread-safe queue used to hand-over Block objects between threads. More...

class	BlockQueueSink
	BlockSink which interfaces to a File. More...

class	BlockReader
	BlockReader takes Block objects from BlockSource and allows reading of a) serializable Items or b) arbitray data from the Block sequence. More...

class	BlockSink
	Pure virtual base class for all things that can receive Blocks from a BlockWriter. More...

class	BlockWriter
	BlockWriter contains a temporary Block object into which a) any serializable item can be stored or b) any arbitrary integral data can be appended. More...

class	BoundedBlockSink
	Derivative BlockSink which counts and limits how many bytes it has delivered as ByteBlocks for writing. More...

class	ByteBlock
	A ByteBlock is the basic storage units of containers like File, BlockQueue, etc. More...

class	ByteBlockPtr
	A non-pinned counting pointer to a ByteBlock. More...

class	CacheBlockQueueSource
	A BlockSource to read Blocks from a BlockQueue using a BlockReader, and at the same time CACHE all items received. More...

class	CatBlockSource
	CatBlockSource is a BlockSource which concatenates all Blocks available from a vector of BlockSources. More...

class	CatStream
	Ownership handle onto a CatStreamData. More...

class	CatStreamData
	A Stream is a virtual set of connections to all other worker instances, hence a "Stream" bundles them to a logical communication context. More...

class	ConsumeBlockQueueSource
	A BlockSource to read Block from a BlockQueue using a BlockReader. More...

class	ConsumeFileBlockSource
	A BlockSource to read and simultaneously consume Blocks from a File. More...

class	DiscardSink
	DiscardSink is an BlockSink that discards all Blocks delivered to it. More...

class	DynBlockSource
	This is the actual BlockSource used to instantiate BlockReader. More...

class	DynBlockSourceAdapter
	Adapter class to wrap any existing BlockSource concept class into a DynBlockSourceInterface. More...

class	DynBlockSourceInterface
	This is a pure virtual base which will be used to fetch Blocks for the BlockReader from different sources. More...

class	File
	A File is an ordered sequence of Block objects for storing items. More...

class	FileBlockSink
	BlockSink which interfaces to a File. More...

class	FullException
	An Exception is thrown by BlockWriter when the underlying sink does not allow allocation of a new block, which is needed to serialize the item. More...

class	KeepFileBlockSource
	A BlockSource to read Blocks from a File. More...

class	MixBlockQueue
	Implements reading an unordered sequence of items from multiple workers, which sends Blocks. More...

class	MixBlockQueueReader
	Reader to retrieve items in unordered sequence from a MixBlockQueue. More...

class	MixStream
	Ownership handle onto a MixStream. More...

class	MixStreamData
	A Stream is a virtual set of connections to all other worker instances, hence a "Stream" bundles them to a logical communication context. More...

class	Multiplexer
	Multiplexes virtual Connections on Dispatcher. More...

class	MultiplexerHeader

class	PartitionMultiplexerHeader

class	PinnedBlock
	A pinned / pin-counted derivative of a Block. More...

class	PinnedByteBlockPtr
	A pinned / pin-counted pointer to a ByteBlock. More...

class	PinRequest

struct	Serialization

struct	Serialization< Archive, core::HyperLogLogRegisters< p > >

struct	Serialization< Archive, std::array< T, N >, typename std::enable_if< !std::is_pod< std::array< T, N > >::value >::type >

struct	Serialization< Archive, std::pair< U, V > >

struct	Serialization< Archive, std::string >

struct	Serialization< Archive, std::tuple< Args... > >

struct	Serialization< Archive, std::vector< T > >

struct	Serialization< Archive, T, typename std::enable_if< cereal::traits::is_input_serializable< T, Archive >::value &&!std::is_pod< T >::value &&!std::is_same< T, std::string >::value &&!tlx::is_std_pair< T >::value &&!tlx::is_std_array< T >::value &&!tlx::is_std_vector< T >::value &&!tlx::is_std_tuple< T >::value >::type >

struct	Serialization< Archive, T, typename std::enable_if< has_member_thrill_is_fixed_size< T >::value >::type >

struct	Serialization< Archive, T, typename std::enable_if< std::is_pod< T >::value &&!std::is_pointer< T >::value >::type >

class	Stream
	Stream - base class for CatStream and MixStream. More...

class	StreamData
	Base class for common structures for ConcatStream and MixedStream. More...

class	StreamMultiplexerHeader
	Block header is sent before a sequence of blocks it indicates the number of elements and their boundaries. More...

class	StreamSet
	Simple structure that holds a all stream instances for the workers on the local host for a given stream id. More...

class	StreamSetBase
	Base class for StreamSet. More...

class	StreamSink
	StreamSink is an BlockSink that sends data via a network socket to the StreamData object on a different worker. More...

Typedefs
using	Byte = uint8_t
	type of underlying memory area More...

using	CatStreamDataPtr = tlx::CountingPtr< CatStreamData >

using	CatStreamPtr = tlx::CountingPtr< CatStream >

using	CatStreamSet = StreamSet< CatStreamData >

using	CatStreamSetPtr = tlx::CountingPtr< CatStreamSet >

using	DiscardWriter = BlockWriter< DiscardSink >

using	DynBlockReader = BlockReader< DynBlockSource >
	Instantiation of BlockReader for reading from the polymorphic source. More...

using	FilePtr = tlx::CountingPtr< File >

using	MixStreamDataPtr = tlx::CountingPtr< MixStreamData >

using	MixStreamPtr = tlx::CountingPtr< MixStream >

using	MixStreamSet = StreamSet< MixStreamData >

using	MixStreamSetPtr = tlx::CountingPtr< MixStreamSet >

using	PinRequestPtr = tlx::CountingPtr< PinRequest, mem::GPoolDeleter< PinRequest > >

using	ReadingMap = std::unordered_map< ByteBlock , PinRequestPtr, std::hash< ByteBlock >, std::equal_to<>, mem::GPoolAllocator< std::pair< ByteBlock *const, PinRequestPtr > > >
	type of set of ByteBlocks currently begin read from EM. More...

using	StreamDataPtr = tlx::CountingPtr< StreamData >

using	StreamId = size_t

using	WritingMap = std::unordered_map< ByteBlock , foxxll::request_ptr, std::hash< ByteBlock >, std::equal_to<>, mem::GPoolAllocator< std::pair< ByteBlock *const, foxxll::request_ptr > > >
	type of set of ByteBlocks currently begin written to EM. More...

Enumerations
enum	MagicByte : uint8_t { Invalid, CatStreamBlock, MixStreamBlock, PartitionBlock }

Functions
template<typename BlockSource , typename... Params>
DynBlockReader	ConstructDynBlockReader (Params &&... params)
	Method to construct a DynBlockReader from a non-polymorphic BlockSource. More...

template<typename BlockSource , typename... Params>
DynBlockSource	ConstructDynBlockSource (Params &&... params)
	Method to construct a DynBlockSource from a non-polymorphic BlockSource. More...

std::ostream &	operator<< (std::ostream &os, const Block &b)

std::ostream &	operator<< (std::ostream &os, const PinnedBlock &b)

std::ostream &	operator<< (std::ostream &os, const ByteBlock &b)

std::ostream &	operator<< (std::ostream &os, const File &f)

std::ostream &	operator<< (std::ostream &os, const BlockPool::PinCount &p)

static void	OurNewHandler ()

template<typename Reader >
void	StartPrefetch (std::vector< Reader > &readers, size_t prefetch_size)
	Take a vector of Readers and prefetch equally from them. More...

	TLX_MAKE_HAS_MEMBER (thrill_is_fixed_size)

Variables
static constexpr bool	debug_alloc = false
	debug block memory alloc and dealloc More...

static constexpr bool	debug_blc = false
	debug block life cycle output: create, destroy More...

static constexpr bool	debug_em = false
	debug block eviction: evict, write complete, read complete More...

static constexpr bool	debug_mem = false
	debug memory requests More...

static constexpr bool	debug_pin = false
	debug block pinning: More...

size_t	default_block_size = 2 * 1024 * 1024
	default size of blocks in File, Channel, BlockQueue, etc. More...

static std::atomic< bool >	in_new_handler

static std::vector< BlockPool * >	s_blockpools

static std::recursive_mutex	s_new_mutex

size_t	start_block_size = 4 * 1024
	starting size of blocks in BlockWriter. More...

class thrill::data::MultiplexerHeader	TLX_ATTRIBUTE_PACKED

Typedef Documentation

◆ ReadingMap

using ReadingMap = std::unordered_map< ByteBlock*, PinRequestPtr, std::hash<ByteBlock*>, std::equal_to<>, mem::GPoolAllocator< std::pair<ByteBlock* const, PinRequestPtr> > >

type of set of ByteBlocks currently begin read from EM.

Definition at line 240 of file block_pool.cpp.

◆ WritingMap

using WritingMap = std::unordered_map< ByteBlock*, foxxll::request_ptr, std::hash<ByteBlock*>, std::equal_to<>, mem::GPoolAllocator<std::pair<ByteBlock* const, foxxll::request_ptr> > >

type of set of ByteBlocks currently begin written to EM.

Definition at line 233 of file block_pool.cpp.

Function Documentation

◆ operator<<() [1/5]

std::ostream& thrill::data::operator<<	(	std::ostream &	os,
		const Block &	b
	)

Definition at line 22 of file block.cpp.

References Block::begin_, Block::byte_block_, Block::end_, Block::first_item_, CountingPtr< Type, Deleter >::get(), Block::IsValid(), and Block::num_items_.

◆ operator<<() [2/5]

std::ostream& thrill::data::operator<<	(	std::ostream &	os,
		const PinnedBlock &	b
	)

Definition at line 53 of file block.cpp.

References Block::byte_block_.

◆ operator<<() [3/5]

std::ostream& thrill::data::operator<<	(	std::ostream &	os,
		const ByteBlock &	b
	)

Definition at line 79 of file byte_block.cpp.

References ByteBlock::block_pool_, ByteBlock::data_, ByteBlock::ext_file_, ByteBlock::size_, and ByteBlock::total_pins_.

◆ operator<<() [4/5]

std::ostream& thrill::data::operator<<	(	std::ostream &	os,
		const File &	f
	)

Definition at line 94 of file file.cpp.

References File::blocks_, and ReferenceCounter::reference_count().

◆ operator<<() [5/5]

std::ostream& thrill::data::operator<<	(	std::ostream &	os,
		const BlockPool::PinCount &	p
	)

Definition at line 216 of file block_pool.cpp.

References tlx::join().

◆ OurNewHandler()

static void thrill::data::OurNewHandler ( )

static

Definition at line 63 of file block_pool.cpp.

References die_unless, in_new_handler, LOG1, thrill::mem::malloc_tracker_print_status(), max(), operator+=(), gen_data::value, and BlockPool::workers_per_host().

Referenced by BlockPool::BlockPool().

Variable Documentation

◆ debug_alloc

constexpr bool debug_alloc = false

static

debug block memory alloc and dealloc

Definition at line 42 of file block_pool.cpp.

◆ debug_blc

constexpr bool debug_blc = false

static

debug block life cycle output: create, destroy

Definition at line 39 of file block_pool.cpp.

◆ debug_em

constexpr bool debug_em = false

static

debug block eviction: evict, write complete, read complete

Definition at line 51 of file block_pool.cpp.

◆ debug_mem

constexpr bool debug_mem = false

static

debug memory requests

Definition at line 48 of file block_pool.cpp.

◆ debug_pin

constexpr bool debug_pin = false

static

debug block pinning:

Definition at line 45 of file block_pool.cpp.

◆ in_new_handler

std::atomic<bool> in_new_handler

static

Initial value:

{
    false
}

Definition at line 59 of file block_pool.cpp.

Referenced by OurNewHandler().

◆ s_blockpools

std::vector<BlockPool*> s_blockpools

static

Definition at line 57 of file block_pool.cpp.

◆ s_new_mutex

std::recursive_mutex s_new_mutex

static

Definition at line 56 of file block_pool.cpp.

Namespaces

Classes

Typedefs

Enumerations

Functions

Variables

Typedef Documentation

◆ ReadingMap

◆ WritingMap

Function Documentation

◆ operator<<() [1/5]

◆ operator<<() [2/5]

◆ operator<<() [3/5]

◆ operator<<() [4/5]

◆ operator<<() [5/5]

◆ OurNewHandler()

Variable Documentation

◆ debug_alloc

◆ debug_blc

◆ debug_em

◆ debug_mem

◆ debug_pin

◆ in_new_handler

◆ s_blockpools

◆ s_new_mutex