Detailed Description

template<typename BlockSink>
class thrill::data::BlockWriter< BlockSink >

BlockWriter contains a temporary Block object into which a) any serializable item can be stored or b) any arbitrary integral data can be appended.

It counts how many serializable items are stored and the offset of the first new item. When a Block is full it is emitted to an attached BlockSink, like a File, a ChannelSink, etc. for further delivery. The BlockWriter takes care of segmenting items when a Block is full.

Definition at line 53 of file block_writer.hpp.

Inheritance diagram for BlockWriter< BlockSink >:

Collaboration diagram for BlockWriter< BlockSink >:

#include <block_writer.hpp>

Public Member Functions
	BlockWriter (BlockSink &&sink, size_t max_block_size=default_block_size)
	Start build (appending blocks) to a File. More...

	BlockWriter ()=default
	default constructor More...

	BlockWriter (const BlockWriter &)=delete
	non-copyable: delete copy-constructor More...

	BlockWriter (BlockWriter &&bw) noexcept
	move-constructor More...

	~BlockWriter ()
	On destruction, the last partial block is flushed. More...

void	AppendBlocks (const std::vector< Block > &blocks)

void	AppendBlocks (const std::deque< Block > &blocks)

size_t	block_size () const
	Returns block_size_. More...

void	Close ()
	Explicitly close the writer. More...

void	Flush ()
	Flush the current block (only really meaningful for a network sink). More...

bool	HasBufferData () const
	Return true if any data is buffered. More...

bool	IsValid () const
	Return whether an actual BlockSink is attached. More...

BlockWriter &	operator= (const BlockWriter &)=delete
	non-copyable: delete assignment operator More...

BlockWriter &	operator= (BlockWriter &&bw) noexcept
	move-assignment More...

BlockSink &	sink ()
	Returns sink_. More...

Appending (Generic) Serializable Items
TLX_ATTRIBUTE_ALWAYS_INLINE BlockWriter &	MarkItem ()
	Mark beginning of an item. More...

template<typename T >
TLX_ATTRIBUTE_ALWAYS_INLINE BlockWriter &	Put (const T &x)
	Put appends a complete item, or fails with a FullException. More...

template<typename T >
TLX_ATTRIBUTE_ALWAYS_INLINE BlockWriter &	PutNoSelfVerify (const T &x)

template<typename T , bool NoSelfVerify = false>
TLX_ATTRIBUTE_ALWAYS_INLINE BlockWriter &	PutSafe (const T &x)
	appends a complete item, or fails safely with a FullException. More...

template<typename T , bool NoSelfVerify = false>
TLX_ATTRIBUTE_ALWAYS_INLINE BlockWriter &	PutUnsafe (const T &x)
	appends a complete item, or aborts with a FullException. More...

Appending Write Functions
BlockWriter &	Append (const void *data, size_t size)
	Append a memory range to the block. More...

BlockWriter &	PutByte (Byte data)
	Append a single byte to the block. More...

BlockWriter &	Append (const std::string &str)

template<typename Type >
TLX_ATTRIBUTE_ALWAYS_INLINE BlockWriter &	PutRaw (const Type &item)

Public Member Functions inherited from ItemWriterToolsBase< BlockWriter< BlockSink > >
BlockWriter< BlockSink > &	PutString (const char *data, size_t len)
	Put a string by saving it's length followed by the data itself. More...

BlockWriter< BlockSink > &	PutString (const uint8_t *data, size_t len)
	Put a string by saving it's length followed by the data itself. More...

BlockWriter< BlockSink > &	PutString (const std::string &str)
	Put a string by saving it's length followed by the data itself. More...

BlockWriter< BlockSink > &	PutVarint (uint64_t v)
	Append a varint to the writer. More...

BlockWriter< BlockSink > &	PutVarint32 (uint32_t v)
	Append a varint to the writer. More...

Static Public Attributes
static constexpr bool	debug = false

static constexpr bool	self_verify = common::g_self_verify

Private Member Functions
void	AllocateBlock ()
	Allocate a new block (overwriting the existing one). More...

Private Attributes
size_t	block_size_
	size of data blocks to construct More...

PinnedByteBlockPtr	bytes_

bool	closed_ = false
	Flag if Close was called explicitly. More...

Byte *	current_ = nullptr
	current write pointer into block. More...

bool	do_queue_ = false
	boolean whether to queue blocks More...

Byte *	end_ = nullptr

size_t	first_offset_ = 0
	offset of first item More...

size_t	max_block_size_
	size of data blocks to construct More...

size_t	nitems_ = 0
	number of items in current block More...

BlockSink	sink_
	file or stream sink to output blocks to. More...

std::deque< PinnedBlock >	sink_queue_
	queue of blocks to flush when the current item has fully been serialized More...

Constructor & Destructor Documentation

◆ BlockWriter() [1/4]

BlockWriter	(	BlockSink &&	sink,
		size_t	max_block_size = `default_block_size`
	)

inlineexplicit

Start build (appending blocks) to a File.

Definition at line 62 of file block_writer.hpp.

◆ BlockWriter() [2/4]

BlockWriter ( )

default

default constructor

◆ BlockWriter() [3/4]

BlockWriter ( const BlockWriter< BlockSink > & )

delete

non-copyable: delete copy-constructor

◆ BlockWriter() [4/4]

BlockWriter ( BlockWriter< BlockSink > && bw )

inlinenoexcept

move-constructor

Definition at line 79 of file block_writer.hpp.

◆ ~BlockWriter()

~BlockWriter ( )

inline

On destruction, the last partial block is flushed.

Definition at line 116 of file block_writer.hpp.

Member Function Documentation

◆ AllocateBlock()

void AllocateBlock ( )

inlineprivate

Allocate a new block (overwriting the existing one).

Definition at line 409 of file block_writer.hpp.

References FullException::FullException(), and sLOG.

◆ Append() [1/2]

BlockWriter& Append	(	const void *	data,
		size_t	size
	)

inline

Append a memory range to the block.

Definition at line 343 of file block_writer.hpp.

References TLX_UNLIKELY.

◆ Append() [2/2]

BlockWriter& Append ( const std::string & str )

inline

Append to contents of a std::string, excluding the null (which isn't contained in the string size anyway).

Definition at line 380 of file block_writer.hpp.

References TLX_ATTRIBUTE_ALWAYS_INLINE.

◆ AppendBlocks() [1/2]

void AppendBlocks ( const std::vector< Block > & blocks )

inline

Directly write Blocks to the underlying BlockSink (after flushing the current one if need be).

Definition at line 172 of file block_writer.hpp.

◆ AppendBlocks() [2/2]

void AppendBlocks ( const std::deque< Block > & blocks )

inline

Directly write Blocks to the underlying BlockSink (after flushing the current one if need be).

Definition at line 182 of file block_writer.hpp.

References TLX_ATTRIBUTE_ALWAYS_INLINE.

◆ block_size()

size_t block_size ( ) const

inline

Returns block_size_.

Definition at line 140 of file block_writer.hpp.

◆ Close()

void Close ( )

inline

Explicitly close the writer.

Definition at line 121 of file block_writer.hpp.

Referenced by GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::FlushVectorToFile(), GroupByNode< ValueType, KeyExtractor, GroupFunction, HashFunction, UseLocationDetection >::FlushVectorToFile(), ReduceByIndexPostPhase< TableItem, Key, ValueType, KeyExtractor, ReduceFunction, thrill::api::ReduceToIndexNode::Emitter, VolatileKey, ReduceConfig >::PushData(), RebalanceNode< ValueType >::StopPreOp(), WriteLinesOneNode< ValueType >::StopPreOp(), CacheNode< ValueType >::StopPreOp(), PrefixSumNode< ValueType, SumFunction, Inclusive >::StopPreOp(), ZipWithIndexNode< ValueType, ZipFunction >::StopPreOp(), BaseWindowNode< ValueType, Input, WindowFunction, PartialWindowFunction >::StopPreOp(), ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::StopPreOp(), MergeNode< ValueType, Comparator, kNumInputs >::StopPreOp(), GroupByNode< ValueType, KeyExtractor, GroupFunction, HashFunction, UseLocationDetection >::StopPreOp(), SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >::StopPreOp(), and JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::StopPreOp().

◆ Flush()

void Flush ( )

inline

Flush the current block (only really meaningful for a network sink).

Definition at line 143 of file block_writer.hpp.

References sLOG.

◆ HasBufferData()

bool HasBufferData ( ) const

inline

Return true if any data is buffered.

Definition at line 132 of file block_writer.hpp.

◆ IsValid()

bool IsValid ( ) const

inline

Return whether an actual BlockSink is attached.

Definition at line 129 of file block_writer.hpp.

◆ MarkItem()

TLX_ATTRIBUTE_ALWAYS_INLINE BlockWriter& MarkItem ( )

inline

Mark beginning of an item.

Definition at line 195 of file block_writer.hpp.

References TLX_ATTRIBUTE_ALWAYS_INLINE.

◆ operator=() [1/2]

BlockWriter& operator= ( const BlockWriter< BlockSink > & )

delete

non-copyable: delete assignment operator

◆ operator=() [2/2]

BlockWriter& operator= ( BlockWriter< BlockSink > && bw )

inlinenoexcept

move-assignment

Definition at line 96 of file block_writer.hpp.

◆ Put()

TLX_ATTRIBUTE_ALWAYS_INLINE BlockWriter& Put ( const T & x )

inline

Put appends a complete item, or fails with a FullException.

Definition at line 210 of file block_writer.hpp.

References BlockSink::allocate_can_fail_, TLX_ATTRIBUTE_ALWAYS_INLINE, and gen_data::x.

Referenced by JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::AddEqualKeysToFile(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::AddEqualKeysToVec(), CacheNode< ValueType >::CacheNode(), ConcatNode< ValueType >::ConcatNode(), GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::FlushVectorToFile(), GroupByNode< ValueType, KeyExtractor, GroupFunction, HashFunction, UseLocationDetection >::FlushVectorToFile(), ReduceByIndexPostPhase< TableItem, Key, ValueType, KeyExtractor, ReduceFunction, thrill::api::ReduceToIndexNode::Emitter, VolatileKey, ReduceConfig >::Insert(), MergeNode< ValueType, Comparator, kNumInputs >::RegisterParent::operator()(), ConcatNode< ValueType >::RegisterParent::operator()(), ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::RegisterParent::operator()(), ZipWindowNode< ValueType, ZipFunction_, Pad_, UnequalCheck, UseStdArray, kNumInputs_ >::RegisterParent::operator()(), PrefixSumNode< ValueType, SumFunction, Inclusive >::PreOp(), WriteLinesOneNode< ValueType >::PreOp(), BaseWindowNode< ValueType, Input, WindowFunction, PartialWindowFunction >::PreOp(), SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >::PreOp(), GroupByNode< ValueType, KeyExtractor, GroupFunction, HashFunction, UseLocationDetection >::PreOp(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::PreOp1(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::PreOp2(), RebalanceNode< ValueType >::RebalanceNode(), ReduceOldProbingHashTable< TableItem, Key, Value, KeyExtractor, ReduceFunction, Emitter, VolatileKey, ReduceConfig_, IndexFunction, KeyEqualFunction >::SpillPartition(), ReduceBucketHashTable< TableItem, Key, Value, KeyExtractor, ReduceFunction, Emitter, VolatileKey, ReduceConfig, IndexFunction, KeyEqualFunction >::SpillPartition(), ReduceProbingHashTable< TableItem, Key, Value, KeyExtractor, ReduceFunction, Emitter, VolatileKey, ReduceConfig_, IndexFunction, KeyEqualFunction >::SpillPartition(), and ZipWithIndexNode< ValueType, ZipFunction >::ZipWithIndexNode().

◆ PutByte()

BlockWriter& PutByte ( Byte data )

inline

Append a single byte to the block.

Definition at line 368 of file block_writer.hpp.

References TLX_UNLIKELY.

◆ PutNoSelfVerify()

TLX_ATTRIBUTE_ALWAYS_INLINE BlockWriter& PutNoSelfVerify ( const T & x )

inline

PutNoSelfVerify appends a complete item without any self verification information, or fails with a FullException.

Definition at line 223 of file block_writer.hpp.

References BlockSink::allocate_can_fail_, TLX_ATTRIBUTE_ALWAYS_INLINE, and gen_data::x.

◆ PutRaw()

TLX_ATTRIBUTE_ALWAYS_INLINE BlockWriter& PutRaw ( const Type & item )

inline

Put (append) a single item of the template type T to the buffer. Be careful with implicit type conversions!

Definition at line 388 of file block_writer.hpp.

References TLX_LIKELY, and gen_data::value.

◆ PutSafe()

TLX_ATTRIBUTE_ALWAYS_INLINE BlockWriter& PutSafe ( const T & x )

inline

appends a complete item, or fails safely with a FullException.

Definition at line 235 of file block_writer.hpp.

References FullException::FullException(), sLOG, PinnedBlock::StealPinnedByteBlock(), TLX_ATTRIBUTE_ALWAYS_INLINE, and TLX_UNLIKELY.

◆ PutUnsafe()

TLX_ATTRIBUTE_ALWAYS_INLINE BlockWriter& PutUnsafe ( const T & x )

inline

appends a complete item, or aborts with a FullException.

Definition at line 311 of file block_writer.hpp.

References TLX_UNLIKELY.

◆ sink()

BlockSink& sink ( )

inline

Returns sink_.

Definition at line 137 of file block_writer.hpp.

Member Data Documentation

◆ block_size_

size_t block_size_

private

size of data blocks to construct

Definition at line 453 of file block_writer.hpp.

◆ bytes_

PinnedByteBlockPtr bytes_

private

current block, already allocated as shared ptr, since we want to use make_shared.

Definition at line 428 of file block_writer.hpp.

◆ closed_

bool closed_ = false

private

Flag if Close was called explicitly.

Definition at line 459 of file block_writer.hpp.

◆ current_

Byte* current_ = nullptr

private

current write pointer into block.

Definition at line 431 of file block_writer.hpp.

◆ debug

constexpr bool debug = false

static

Definition at line 57 of file block_writer.hpp.

◆ do_queue_

bool do_queue_ = false

private

boolean whether to queue blocks

Definition at line 447 of file block_writer.hpp.

◆ end_

Byte* end_ = nullptr

private

current end of block pointer. this is == bytes_.end(), just one indirection less.

Definition at line 435 of file block_writer.hpp.

◆ first_offset_

size_t first_offset_ = 0

private

offset of first item

Definition at line 441 of file block_writer.hpp.

◆ max_block_size_

size_t max_block_size_

private

size of data blocks to construct

Definition at line 456 of file block_writer.hpp.

◆ nitems_

size_t nitems_ = 0

private

number of items in current block

Definition at line 438 of file block_writer.hpp.

◆ self_verify

constexpr bool self_verify = common::g_self_verify

static

Definition at line 59 of file block_writer.hpp.

◆ sink_

BlockSink sink_

private

file or stream sink to output blocks to.

Definition at line 444 of file block_writer.hpp.

◆ sink_queue_

std::deque<PinnedBlock> sink_queue_

private

queue of blocks to flush when the current item has fully been serialized

Definition at line 450 of file block_writer.hpp.

The documentation for this class was generated from the following file:

thrill/data/block_writer.hpp

Detailed Description

template<typename BlockSink> class thrill::data::BlockWriter< BlockSink >

Public Member Functions

Static Public Attributes

Private Member Functions

Private Attributes

Constructor & Destructor Documentation

◆ BlockWriter() [1/4]

◆ BlockWriter() [2/4]

◆ BlockWriter() [3/4]

◆ BlockWriter() [4/4]

◆ ~BlockWriter()

Member Function Documentation

◆ AllocateBlock()

◆ Append() [1/2]

◆ Append() [2/2]

◆ AppendBlocks() [1/2]

◆ AppendBlocks() [2/2]

◆ block_size()

◆ Close()

◆ Flush()

◆ HasBufferData()

◆ IsValid()

◆ MarkItem()

◆ operator=() [1/2]

◆ operator=() [2/2]

◆ Put()

◆ PutByte()

◆ PutNoSelfVerify()

◆ PutRaw()

◆ PutSafe()

◆ PutUnsafe()

◆ sink()

Member Data Documentation

◆ block_size_

◆ bytes_

◆ closed_

◆ current_

◆ debug

◆ do_queue_

◆ end_

◆ first_offset_

◆ max_block_size_

◆ nitems_

◆ self_verify

◆ sink_

◆ sink_queue_

template<typename BlockSink>
class thrill::data::BlockWriter< BlockSink >