Thrill  0.1
Data Subsystem Layer

Modules

 Data Internals
 

Classes

class  Block
 Block combines a reference to a read-only ByteBlock and book-keeping information. More...
 
class  BlockPool
 Pool to allocate, keep, swap out/in, and free all ByteBlocks on the host. More...
 
class  BlockPoolMemoryHolder
 RAII class for allocating memory from a BlockPool. More...
 
class  BlockQueue
 A BlockQueue is a thread-safe queue used to hand-over Block objects between threads. More...
 
class  BlockQueueSink
 BlockSink which interfaces to a File. More...
 
class  BlockReader< BlockSource >
 BlockReader takes Block objects from BlockSource and allows reading of a) serializable Items or b) arbitray data from the Block sequence. More...
 
class  BlockSink
 Pure virtual base class for all things that can receive Blocks from a BlockWriter. More...
 
class  BlockWriter< BlockSink >
 BlockWriter contains a temporary Block object into which a) any serializable item can be stored or b) any arbitrary integral data can be appended. More...
 
class  BoundedBlockSink
 Derivative BlockSink which counts and limits how many bytes it has delivered as ByteBlocks for writing. More...
 
class  ByteBlock
 A ByteBlock is the basic storage units of containers like File, BlockQueue, etc. More...
 
class  ByteBlockPtr
 A non-pinned counting pointer to a ByteBlock. More...
 
class  CacheBlockQueueSource
 A BlockSource to read Blocks from a BlockQueue using a BlockReader, and at the same time CACHE all items received. More...
 
class  CatBlockSource< BlockSource >
 CatBlockSource is a BlockSource which concatenates all Blocks available from a vector of BlockSources. More...
 
class  CatStream
 Ownership handle onto a CatStreamData. More...
 
class  CatStreamData
 A Stream is a virtual set of connections to all other worker instances, hence a "Stream" bundles them to a logical communication context. More...
 
class  ConsumeBlockQueueSource
 A BlockSource to read Block from a BlockQueue using a BlockReader. More...
 
class  ConsumeFileBlockSource
 A BlockSource to read and simultaneously consume Blocks from a File. More...
 
class  DiscardSink
 DiscardSink is an BlockSink that discards all Blocks delivered to it. More...
 
class  DynBlockSource
 This is the actual BlockSource used to instantiate BlockReader. More...
 
class  DynBlockSourceAdapter< BlockSource >
 Adapter class to wrap any existing BlockSource concept class into a DynBlockSourceInterface. More...
 
class  DynBlockSourceInterface
 This is a pure virtual base which will be used to fetch Blocks for the BlockReader from different sources. More...
 
class  File
 A File is an ordered sequence of Block objects for storing items. More...
 
class  FileBlockSink
 BlockSink which interfaces to a File. More...
 
class  FullException
 An Exception is thrown by BlockWriter when the underlying sink does not allow allocation of a new block, which is needed to serialize the item. More...
 
class  KeepFileBlockSource
 A BlockSource to read Blocks from a File. More...
 
class  MixBlockQueue
 Implements reading an unordered sequence of items from multiple workers, which sends Blocks. More...
 
class  MixBlockQueueReader
 Reader to retrieve items in unordered sequence from a MixBlockQueue. More...
 
class  MixStream
 Ownership handle onto a MixStream. More...
 
class  MixStreamData
 A Stream is a virtual set of connections to all other worker instances, hence a "Stream" bundles them to a logical communication context. More...
 
class  Multiplexer
 Multiplexes virtual Connections on Dispatcher. More...
 
class  MultiplexerHeader
 
class  PartitionMultiplexerHeader
 
class  PinnedBlock
 A pinned / pin-counted derivative of a Block. More...
 
class  PinnedByteBlockPtr
 A pinned / pin-counted pointer to a ByteBlock. More...
 
class  PinRequest
 
struct  Serialization< Archive, T, Enable >
 
struct  Serialization< Archive, std::array< T, N >, typename std::enable_if< !std::is_pod< std::array< T, N > >::value >::type >
 
struct  Serialization< Archive, std::pair< U, V > >
 
struct  Serialization< Archive, std::string >
 
struct  Serialization< Archive, std::tuple< Args... > >
 
struct  Serialization< Archive, std::vector< T > >
 
struct  Serialization< Archive, T, typename std::enable_if< cereal::traits::is_input_serializable< T, Archive >::value &&!std::is_pod< T >::value &&!std::is_same< T, std::string >::value &&!tlx::is_std_pair< T >::value &&!tlx::is_std_array< T >::value &&!tlx::is_std_vector< T >::value &&!tlx::is_std_tuple< T >::value >::type >
 
struct  Serialization< Archive, T, typename std::enable_if< has_member_thrill_is_fixed_size< T >::value >::type >
 
struct  Serialization< Archive, T, typename std::enable_if< std::is_pod< T >::value &&!std::is_pointer< T >::value >::type >
 
class  Stream
 Stream - base class for CatStream and MixStream. More...
 
class  StreamData
 Base class for common structures for ConcatStream and MixedStream. More...
 
class  StreamMultiplexerHeader
 Block header is sent before a sequence of blocks it indicates the number of elements and their boundaries. More...
 
class  StreamSet< StreamData >
 Simple structure that holds a all stream instances for the workers on the local host for a given stream id. More...
 
class  StreamSetBase
 Base class for StreamSet. More...
 
class  StreamSink
 StreamSink is an BlockSink that sends data via a network socket to the StreamData object on a different worker. More...
 

Typedefs

using Byte = uint8_t
 type of underlying memory area More...
 
using CatStreamDataPtr = tlx::CountingPtr< CatStreamData >
 
using CatStreamPtr = tlx::CountingPtr< CatStream >
 
using CatStreamSet = StreamSet< CatStreamData >
 
using CatStreamSetPtr = tlx::CountingPtr< CatStreamSet >
 
using DiscardWriter = BlockWriter< DiscardSink >
 
using DynBlockReader = BlockReader< DynBlockSource >
 Instantiation of BlockReader for reading from the polymorphic source. More...
 
using FilePtr = tlx::CountingPtr< File >
 
using MixStreamDataPtr = tlx::CountingPtr< MixStreamData >
 
using MixStreamPtr = tlx::CountingPtr< MixStream >
 
using MixStreamSet = StreamSet< MixStreamData >
 
using MixStreamSetPtr = tlx::CountingPtr< MixStreamSet >
 
using PinRequestPtr = tlx::CountingPtr< PinRequest, mem::GPoolDeleter< PinRequest > >
 
using StreamDataPtr = tlx::CountingPtr< StreamData >
 
using StreamId = size_t
 

Enumerations

enum  MagicByte : uint8_t { Invalid, CatStreamBlock, MixStreamBlock, PartitionBlock }
 

Functions

 ByteBlockPtr (const PinnedByteBlockPtr &p)
 (protected) ctor from PinnedByteBlockPtr More...
 
 ByteBlockPtr (PinnedByteBlockPtr &&p)
 (protected) ctor from PinnedByteBlockPtr More...
 
template<typename BlockSource , typename... Params>
DynBlockReader ConstructDynBlockReader (Params &&... params)
 Method to construct a DynBlockReader from a non-polymorphic BlockSource. More...
 
template<typename BlockSource , typename... Params>
DynBlockSource ConstructDynBlockSource (Params &&... params)
 Method to construct a DynBlockSource from a non-polymorphic BlockSource. More...
 
template<typename ItemType , typename CompareFunction = std::less<ItemType>>
size_t GetIndexOf (const ItemType &item, size_t tie, size_t left, size_t right, const CompareFunction &func=CompareFunction()) const
 Get index of the given item, or the next greater item, in this file. More...
 
template<typename ItemType >
ItemType GetItemAt (size_t index) const
 
template<typename ItemType >
std::vector< BlockGetItemRange (size_t begin, size_t end) const
 
template<typename Reader >
void StartPrefetch (std::vector< Reader > &readers, size_t prefetch_size)
 Take a vector of Readers and prefetch equally from them. More...
 
 TLX_MAKE_HAS_MEMBER (thrill_is_fixed_size)
 

Variables

size_t default_block_size = 2 * 1024 * 1024
 default size of blocks in File, Channel, BlockQueue, etc. More...
 
size_t start_block_size = 4 * 1024
 starting size of blocks in BlockWriter. More...
 
class thrill::data::MultiplexerHeader TLX_ATTRIBUTE_PACKED
 

Writers and Readers

template<typename ItemType >
KeepReader GetReaderAt (size_t index, size_t prefetch=default_prefetch_size_) const
 Get BlockReader seeked to the corresponding item index. More...
 

Detailed Description

Typedef Documentation

◆ Byte

using Byte = uint8_t

type of underlying memory area

Definition at line 37 of file byte_block.hpp.

◆ CatStreamDataPtr

Definition at line 149 of file cat_stream.hpp.

◆ CatStreamPtr

Definition at line 198 of file cat_stream.hpp.

◆ CatStreamSet

Definition at line 151 of file cat_stream.hpp.

◆ CatStreamSetPtr

Definition at line 152 of file cat_stream.hpp.

◆ DiscardWriter

Definition at line 61 of file discard_sink.hpp.

◆ DynBlockReader

Instantiation of BlockReader for reading from the polymorphic source.

Definition at line 71 of file dyn_block_reader.hpp.

◆ FilePtr

Definition at line 285 of file file.hpp.

◆ MixStreamDataPtr

Definition at line 36 of file mix_block_queue.hpp.

◆ MixStreamPtr

Definition at line 163 of file mix_stream.hpp.

◆ MixStreamSet

Definition at line 122 of file mix_stream.hpp.

◆ MixStreamSetPtr

Definition at line 123 of file mix_stream.hpp.

◆ PinRequestPtr

◆ StreamDataPtr

Definition at line 196 of file stream_data.hpp.

◆ StreamId

using StreamId = size_t

Definition at line 32 of file stream_data.hpp.

Enumeration Type Documentation

◆ MagicByte

enum MagicByte : uint8_t
strong
Enumerator
Invalid 
CatStreamBlock 
MixStreamBlock 
PartitionBlock 

Definition at line 34 of file stream_data.hpp.

Function Documentation

◆ ByteBlockPtr() [1/2]

ByteBlockPtr ( const PinnedByteBlockPtr p)
inlineprotected

(protected) ctor from PinnedByteBlockPtr

Definition at line 282 of file byte_block.hpp.

◆ ByteBlockPtr() [2/2]

ByteBlockPtr ( PinnedByteBlockPtr &&  p)
inlineprotected

(protected) ctor from PinnedByteBlockPtr

Definition at line 285 of file byte_block.hpp.

◆ ConstructDynBlockReader()

DynBlockReader thrill::data::ConstructDynBlockReader ( Params &&...  params)

Method to construct a DynBlockReader from a non-polymorphic BlockSource.

The variadic parameters are passed to the constructor of the existing BlockSource.

Definition at line 123 of file dyn_block_reader.hpp.

References gen_data::params.

◆ ConstructDynBlockSource()

DynBlockSource thrill::data::ConstructDynBlockSource ( Params &&...  params)

Method to construct a DynBlockSource from a non-polymorphic BlockSource.

The variadic parameters are passed to the constructor of the existing BlockSource.

Definition at line 111 of file dyn_block_reader.hpp.

References tlx::make_counting(), and gen_data::params.

◆ GetIndexOf()

size_t GetIndexOf ( const ItemType &  item,
size_t  tie,
size_t  left,
size_t  right,
const CompareFunction &  func = CompareFunction() 
) const

Get index of the given item, or the next greater item, in this file.

The file has to be ordered according to the given compare function. The tie value can be used to make a decision in case of many successive equal elements. The tie is compared with the local rank of the element.

WARNING: This method uses GetItemAt combined with a binary search and is therefore not efficient. The method should be reimplemented in near future.

Definition at line 534 of file file.hpp.

References debug, LOG, File::num_items(), and gen_data::value.

Referenced by File::GetIndexOf(), and File::ItemsStartIn().

◆ GetItemAt()

ItemType GetItemAt ( size_t  index) const

Get item at the corresponding position. Do not use this method for reading multiple successive items.

Definition at line 528 of file file.hpp.

References BlockReader< BlockSource >::Next().

Referenced by File::ItemsStartIn(), and SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >::OnPreOpFile().

◆ GetItemRange()

std::vector< Block > GetItemRange ( size_t  begin,
size_t  end 
) const

Seek in File: return a Block range containing items begin, end of given type.

Definition at line 576 of file file.hpp.

Referenced by File::GetIndexOf().

◆ GetReaderAt()

◆ StartPrefetch()

◆ TLX_MAKE_HAS_MEMBER()

thrill::data::TLX_MAKE_HAS_MEMBER ( thrill_is_fixed_size  )

Variable Documentation

◆ default_block_size

◆ start_block_size

size_t start_block_size = 4 * 1024

starting size of blocks in BlockWriter.

Definition at line 24 of file byte_block.cpp.

Referenced by thrill::api::SetupBlockSize().

◆ TLX_ATTRIBUTE_PACKED