Modules
	Data Internals

Classes
class	Block
	Block combines a reference to a read-only ByteBlock and book-keeping information. More...

class	BlockPool
	Pool to allocate, keep, swap out/in, and free all ByteBlocks on the host. More...

class	BlockPoolMemoryHolder
	RAII class for allocating memory from a BlockPool. More...

class	BlockQueue
	A BlockQueue is a thread-safe queue used to hand-over Block objects between threads. More...

class	BlockQueueSink
	BlockSink which interfaces to a File. More...

class	BlockReader< BlockSource >
	BlockReader takes Block objects from BlockSource and allows reading of a) serializable Items or b) arbitray data from the Block sequence. More...

class	BlockSink
	Pure virtual base class for all things that can receive Blocks from a BlockWriter. More...

class	BlockWriter< BlockSink >
	BlockWriter contains a temporary Block object into which a) any serializable item can be stored or b) any arbitrary integral data can be appended. More...

class	BoundedBlockSink
	Derivative BlockSink which counts and limits how many bytes it has delivered as ByteBlocks for writing. More...

class	ByteBlock
	A ByteBlock is the basic storage units of containers like File, BlockQueue, etc. More...

class	ByteBlockPtr
	A non-pinned counting pointer to a ByteBlock. More...

class	CacheBlockQueueSource
	A BlockSource to read Blocks from a BlockQueue using a BlockReader, and at the same time CACHE all items received. More...

class	CatBlockSource< BlockSource >
	CatBlockSource is a BlockSource which concatenates all Blocks available from a vector of BlockSources. More...

class	CatStream
	Ownership handle onto a CatStreamData. More...

class	CatStreamData
	A Stream is a virtual set of connections to all other worker instances, hence a "Stream" bundles them to a logical communication context. More...

class	ConsumeBlockQueueSource
	A BlockSource to read Block from a BlockQueue using a BlockReader. More...

class	ConsumeFileBlockSource
	A BlockSource to read and simultaneously consume Blocks from a File. More...

class	DiscardSink
	DiscardSink is an BlockSink that discards all Blocks delivered to it. More...

class	DynBlockSource
	This is the actual BlockSource used to instantiate BlockReader. More...

class	DynBlockSourceAdapter< BlockSource >
	Adapter class to wrap any existing BlockSource concept class into a DynBlockSourceInterface. More...

class	DynBlockSourceInterface
	This is a pure virtual base which will be used to fetch Blocks for the BlockReader from different sources. More...

class	File
	A File is an ordered sequence of Block objects for storing items. More...

class	FileBlockSink
	BlockSink which interfaces to a File. More...

class	FullException
	An Exception is thrown by BlockWriter when the underlying sink does not allow allocation of a new block, which is needed to serialize the item. More...

class	KeepFileBlockSource
	A BlockSource to read Blocks from a File. More...

class	MixBlockQueue
	Implements reading an unordered sequence of items from multiple workers, which sends Blocks. More...

class	MixBlockQueueReader
	Reader to retrieve items in unordered sequence from a MixBlockQueue. More...

class	MixStream
	Ownership handle onto a MixStream. More...

class	MixStreamData
	A Stream is a virtual set of connections to all other worker instances, hence a "Stream" bundles them to a logical communication context. More...

class	Multiplexer
	Multiplexes virtual Connections on Dispatcher. More...

class	MultiplexerHeader

class	PartitionMultiplexerHeader

class	PinnedBlock
	A pinned / pin-counted derivative of a Block. More...

class	PinnedByteBlockPtr
	A pinned / pin-counted pointer to a ByteBlock. More...

class	PinRequest

struct	Serialization< Archive, T, Enable >

struct	Serialization< Archive, std::array< T, N >, typename std::enable_if< !std::is_pod< std::array< T, N > >::value >::type >

struct	Serialization< Archive, std::pair< U, V > >

struct	Serialization< Archive, std::string >

struct	Serialization< Archive, std::tuple< Args... > >

struct	Serialization< Archive, std::vector< T > >

struct	Serialization< Archive, T, typename std::enable_if< cereal::traits::is_input_serializable< T, Archive >::value &&!std::is_pod< T >::value &&!std::is_same< T, std::string >::value &&!tlx::is_std_pair< T >::value &&!tlx::is_std_array< T >::value &&!tlx::is_std_vector< T >::value &&!tlx::is_std_tuple< T >::value >::type >

struct	Serialization< Archive, T, typename std::enable_if< has_member_thrill_is_fixed_size< T >::value >::type >

struct	Serialization< Archive, T, typename std::enable_if< std::is_pod< T >::value &&!std::is_pointer< T >::value >::type >

class	Stream
	Stream - base class for CatStream and MixStream. More...

class	StreamData
	Base class for common structures for ConcatStream and MixedStream. More...

class	StreamMultiplexerHeader
	Block header is sent before a sequence of blocks it indicates the number of elements and their boundaries. More...

class	StreamSet< StreamData >
	Simple structure that holds a all stream instances for the workers on the local host for a given stream id. More...

class	StreamSetBase
	Base class for StreamSet. More...

class	StreamSink
	StreamSink is an BlockSink that sends data via a network socket to the StreamData object on a different worker. More...

Typedefs
using	Byte = uint8_t
	type of underlying memory area More...

using	CatStreamDataPtr = tlx::CountingPtr< CatStreamData >

using	CatStreamPtr = tlx::CountingPtr< CatStream >

using	CatStreamSet = StreamSet< CatStreamData >

using	CatStreamSetPtr = tlx::CountingPtr< CatStreamSet >

using	DiscardWriter = BlockWriter< DiscardSink >

using	DynBlockReader = BlockReader< DynBlockSource >
	Instantiation of BlockReader for reading from the polymorphic source. More...

using	FilePtr = tlx::CountingPtr< File >

using	MixStreamDataPtr = tlx::CountingPtr< MixStreamData >

using	MixStreamPtr = tlx::CountingPtr< MixStream >

using	MixStreamSet = StreamSet< MixStreamData >

using	MixStreamSetPtr = tlx::CountingPtr< MixStreamSet >

using	PinRequestPtr = tlx::CountingPtr< PinRequest, mem::GPoolDeleter< PinRequest > >

using	StreamDataPtr = tlx::CountingPtr< StreamData >

using	StreamId = size_t

Enumerations
enum	MagicByte : uint8_t { Invalid, CatStreamBlock, MixStreamBlock, PartitionBlock }

Functions
	ByteBlockPtr (const PinnedByteBlockPtr &p)
	(protected) ctor from PinnedByteBlockPtr More...

	ByteBlockPtr (PinnedByteBlockPtr &&p)
	(protected) ctor from PinnedByteBlockPtr More...

template<typename BlockSource , typename... Params>
DynBlockReader	ConstructDynBlockReader (Params &&... params)
	Method to construct a DynBlockReader from a non-polymorphic BlockSource. More...

template<typename BlockSource , typename... Params>
DynBlockSource	ConstructDynBlockSource (Params &&... params)
	Method to construct a DynBlockSource from a non-polymorphic BlockSource. More...

template<typename ItemType , typename CompareFunction = std::less<ItemType>>
size_t	GetIndexOf (const ItemType &item, size_t tie, size_t left, size_t right, const CompareFunction &func=CompareFunction()) const
	Get index of the given item, or the next greater item, in this file. More...

template<typename ItemType >
ItemType	GetItemAt (size_t index) const

template<typename ItemType >
std::vector< Block >	GetItemRange (size_t begin, size_t end) const

template<typename Reader >
void	StartPrefetch (std::vector< Reader > &readers, size_t prefetch_size)
	Take a vector of Readers and prefetch equally from them. More...

	TLX_MAKE_HAS_MEMBER (thrill_is_fixed_size)

Variables
size_t	default_block_size = 2 * 1024 * 1024
	default size of blocks in File, Channel, BlockQueue, etc. More...

size_t	start_block_size = 4 * 1024
	starting size of blocks in BlockWriter. More...

class thrill::data::MultiplexerHeader	TLX_ATTRIBUTE_PACKED

Writers and Readers
template<typename ItemType >
KeepReader	GetReaderAt (size_t index, size_t prefetch=default_prefetch_size_) const
	Get BlockReader seeked to the corresponding item index. More...

Detailed Description

Typedef Documentation

◆ Byte

using Byte = uint8_t

type of underlying memory area

Definition at line 37 of file byte_block.hpp.

◆ CatStreamDataPtr

typedef tlx::CountingPtr< CatStreamData > CatStreamDataPtr

Definition at line 149 of file cat_stream.hpp.

◆ CatStreamPtr

typedef tlx::CountingPtr< CatStream > CatStreamPtr

Definition at line 198 of file cat_stream.hpp.

◆ CatStreamSet

typedef StreamSet< CatStreamData > CatStreamSet

Definition at line 151 of file cat_stream.hpp.

◆ CatStreamSetPtr

using CatStreamSetPtr = tlx::CountingPtr<CatStreamSet>

Definition at line 152 of file cat_stream.hpp.

◆ DiscardWriter

using DiscardWriter = BlockWriter<DiscardSink>

Definition at line 61 of file discard_sink.hpp.

◆ DynBlockReader

using DynBlockReader = BlockReader<DynBlockSource>

Instantiation of BlockReader for reading from the polymorphic source.

Definition at line 71 of file dyn_block_reader.hpp.

◆ FilePtr

using FilePtr = tlx::CountingPtr<File>

Definition at line 285 of file file.hpp.

◆ MixStreamDataPtr

typedef tlx::CountingPtr< MixStreamData > MixStreamDataPtr

Definition at line 36 of file mix_block_queue.hpp.

◆ MixStreamPtr

typedef tlx::CountingPtr< MixStream > MixStreamPtr

Definition at line 163 of file mix_stream.hpp.

◆ MixStreamSet

typedef StreamSet< MixStreamData > MixStreamSet

Definition at line 122 of file mix_stream.hpp.

◆ MixStreamSetPtr

using MixStreamSetPtr = tlx::CountingPtr<MixStreamSet>

Definition at line 123 of file mix_stream.hpp.

◆ PinRequestPtr

using PinRequestPtr = tlx::CountingPtr<PinRequest, mem::GPoolDeleter<PinRequest> >

Definition at line 34 of file block.hpp.

◆ StreamDataPtr

typedef tlx::CountingPtr< StreamData > StreamDataPtr

Definition at line 196 of file stream_data.hpp.

◆ StreamId

using StreamId = size_t

Definition at line 32 of file stream_data.hpp.

Enumeration Type Documentation

◆ MagicByte

enum MagicByte : uint8_t

strong

Enumerator
Invalid
CatStreamBlock
MixStreamBlock
PartitionBlock

Definition at line 34 of file stream_data.hpp.

Function Documentation

◆ ByteBlockPtr() [1/2]

ByteBlockPtr ( const PinnedByteBlockPtr & p )

inlineprotected

(protected) ctor from PinnedByteBlockPtr

Definition at line 282 of file byte_block.hpp.

◆ ByteBlockPtr() [2/2]

ByteBlockPtr ( PinnedByteBlockPtr && p )

inlineprotected

(protected) ctor from PinnedByteBlockPtr

Definition at line 285 of file byte_block.hpp.

◆ ConstructDynBlockReader()

DynBlockReader thrill::data::ConstructDynBlockReader ( Params &&... params )

Method to construct a DynBlockReader from a non-polymorphic BlockSource.

The variadic parameters are passed to the constructor of the existing BlockSource.

Definition at line 123 of file dyn_block_reader.hpp.

References gen_data::params.

◆ ConstructDynBlockSource()

DynBlockSource thrill::data::ConstructDynBlockSource ( Params &&... params )

Method to construct a DynBlockSource from a non-polymorphic BlockSource.

The variadic parameters are passed to the constructor of the existing BlockSource.

Definition at line 111 of file dyn_block_reader.hpp.

References tlx::make_counting(), and gen_data::params.

◆ GetIndexOf()

size_t GetIndexOf	(	const ItemType &	item,
		size_t	tie,
		size_t	left,
		size_t	right,
		const CompareFunction &	func = `CompareFunction()`
	)		const

Get index of the given item, or the next greater item, in this file.

The file has to be ordered according to the given compare function. The tie value can be used to make a decision in case of many successive equal elements. The tie is compared with the local rank of the element.

WARNING: This method uses GetItemAt combined with a binary search and is therefore not efficient. The method should be reimplemented in near future.

Definition at line 534 of file file.hpp.

References debug, LOG, File::num_items(), and gen_data::value.

Referenced by File::GetIndexOf(), and File::ItemsStartIn().

◆ GetItemAt()

ItemType GetItemAt ( size_t index ) const

Get item at the corresponding position. Do not use this method for reading multiple successive items.

Definition at line 528 of file file.hpp.

References BlockReader< BlockSource >::Next().

Referenced by File::ItemsStartIn(), and SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >::OnPreOpFile().

◆ GetItemRange()

std::vector< Block > GetItemRange	(	size_t	begin,
		size_t	end
	)		const

Seek in File: return a Block range containing items begin, end of given type.

Definition at line 576 of file file.hpp.

Referenced by File::GetIndexOf().

◆ GetReaderAt()

File::KeepReader GetReaderAt	(	size_t	index,
		size_t	prefetch = `default_prefetch_size_`
	)		const

Get BlockReader seeked to the corresponding item index.

Definition at line 467 of file file.hpp.

References File::blocks_, BlockReader< BlockSource >::CopyBlock(), debug, die, BlockReader< BlockSource >::HasNext(), File::KeepFileBlockSource, BlockSink::local_worker_id_, File::num_items_sum_, BlockReader< BlockSource >::Skip(), sLOG, and BlockReader< BlockSource >::typecode_verify().

◆ StartPrefetch()

void thrill::data::StartPrefetch	(	std::vector< Reader > &	readers,
		size_t	prefetch_size
	)

Take a vector of Readers and prefetch equally from them.

Definition at line 585 of file file.hpp.

References thrill::data::default_block_size.

Referenced by JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::MakePuller(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::MergeFiles(), SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >::PartialMultiwayMerge(), GroupByNode< ValueType, KeyExtractor, GroupFunction, HashFunction, UseLocationDetection >::PushData(), and SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >::PushData().

◆ TLX_MAKE_HAS_MEMBER()

thrill::data::TLX_MAKE_HAS_MEMBER ( thrill_is_fixed_size )

Variable Documentation

◆ default_block_size

size_t default_block_size = 2 * 1024 * 1024

default size of blocks in File, Channel, BlockQueue, etc.

Definition at line 25 of file byte_block.cpp.

Referenced by MixStreamData::GetWriters(), CatStreamData::GetWriters(), BlockPool::MaxMergeDegreePrefetch(), Multiplexer::Multiplexer(), WriteBinaryNode< ValueType >::PreOpMemUse(), ReadLinesNode::PushDataMemUse(), ReadBinaryNode< ValueType >::ReadBinaryNode(), thrill::api::SetupBlockSize(), thrill::data::StartPrefetch(), BlockQueue::timespan(), WriteBinaryNode< ValueType >::WriteBinaryNode(), and WriteLinesNode< ValueType >::WriteLinesNode().

◆ start_block_size

size_t start_block_size = 4 * 1024

starting size of blocks in BlockWriter.

Definition at line 24 of file byte_block.cpp.

Referenced by thrill::api::SetupBlockSize().

◆ TLX_ATTRIBUTE_PACKED

thrill::data::PartitionMultiplexerHeader TLX_ATTRIBUTE_PACKED

Referenced by StreamMultiplexerHeader::CalcHostRank(), and PartitionMultiplexerHeader::IsEnd().

Modules

Classes

Typedefs

Enumerations

Functions

Variables

Writers and Readers