Detailed Description

template<typename BlockSource>
class thrill::data::BlockReader< BlockSource >

BlockReader takes Block objects from BlockSource and allows reading of a) serializable Items or b) arbitray data from the Block sequence.

It takes care of fetching the next Block when the previous one underruns and also of data items split between two Blocks.

Definition at line 42 of file block_reader.hpp.

Inheritance diagram for BlockReader< BlockSource >:

Collaboration diagram for BlockReader< BlockSource >:

#include <block_reader.hpp>

Public Member Functions
	BlockReader (BlockSource &&source)
	Start reading a BlockSource. More...

	BlockReader ()=default
	default constructor More...

	BlockReader (const BlockReader &)=delete
	non-copyable: delete copy-constructor More...

	BlockReader (BlockReader &&)=default
	move-constructor: default More...

ByteBlockPtr	byte_block () const
	return current ByteBlock More...

PinnedBlock	CopyBlock () const
	return current block for debugging More...

BlockReader &	operator= (const BlockReader &)=delete
	non-copyable: delete assignment operator More...

BlockReader &	operator= (BlockReader &&)=default
	move-assignment operator: default More...

BlockSource &	source ()
	Return reference to enclosed BlockSource. More...

size_t	typecode_verify () const
	Returns typecode_verify_. More...

Reading (Generic) Items
template<typename T >
TLX_ATTRIBUTE_ALWAYS_INLINE T	Next ()
	Next() reads a complete item T. More...

template<typename T >
TLX_ATTRIBUTE_ALWAYS_INLINE T	NextNoSelfVerify ()

TLX_ATTRIBUTE_ALWAYS_INLINE bool	HasNext ()
	HasNext() returns true if at least one more item is available. More...

template<typename ItemType >
std::vector< ItemType >	ReadComplete ()

template<typename ItemType >
std::vector< Block >	GetItemBatch (size_t n)

Cursor Reading Methods
BlockReader &	Read (void *outdata, size_t size)

std::string	Read (size_t datalen)

BlockReader &	Skip (size_t items, size_t bytes)
	Advance the cursor given number of bytes without reading them. More...

Byte	GetByte ()
	Fetch a single byte from the current block, advancing the cursor. More...

template<typename Type >
TLX_ATTRIBUTE_ALWAYS_INLINE Type	GetRaw ()

Public Member Functions inherited from ItemReaderToolsBase< BlockReader< BlockSource > >
std::string	GetString ()
	Fetch a string which was Put via Put_string(). More...

uint64_t	GetVarint ()
	Fetch a 64-bit varint from the reader at the cursor. More...

uint32_t	GetVarint32 ()
	Fetch a varint with up to 32-bit from the reader at the cursor. More...

uint32_t	PeekVarint32 () const
	Fetch a varint with up to 32-bit from the reader at the cursor. More...

Static Public Attributes
static constexpr bool	self_verify = common::g_self_verify

Private Member Functions
void	LoadBlock (PinnedBlock &&block)
	Load Block into Reader. More...

bool	NextBlock ()
	Call source_.NextBlock with appropriate parameters. More...

Private Attributes
PinnedBlock	block_
	The current block being read, this holds a shared pointer reference. More...

std::vector< PinnedBlock > *	block_collect_ = nullptr
	pointer to vector to collect blocks in GetItemRange. More...

const Byte *	current_ = nullptr
	current read pointer into current block of file. More...

const Byte *	end_ = nullptr
	pointer to end of current block. More...

size_t	num_items_ = 0
	remaining number of items starting in this block More...

BlockSource	source_

bool	typecode_verify_

Constructor & Destructor Documentation

◆ BlockReader() [1/4]

BlockReader ( BlockSource && source )

inlineexplicit

Start reading a BlockSource.

Definition at line 49 of file block_reader.hpp.

◆ BlockReader() [2/4]

BlockReader ( )

default

default constructor

Referenced by BlockReader< CatBlockSource >::BlockReader(), and BlockReader< CatBlockSource >::source().

◆ BlockReader() [3/4]

BlockReader ( const BlockReader< BlockSource > & )

delete

non-copyable: delete copy-constructor

◆ BlockReader() [4/4]

BlockReader ( BlockReader< BlockSource > && )

default

move-constructor: default

Member Function Documentation

◆ byte_block()

ByteBlockPtr byte_block ( ) const

inline

return current ByteBlock

Definition at line 78 of file block_reader.hpp.

Referenced by BlockReader< CatBlockSource >::GetItemBatch().

◆ CopyBlock()

PinnedBlock CopyBlock ( ) const

inline

return current block for debugging

Definition at line 69 of file block_reader.hpp.

Referenced by File::GetReaderAt().

◆ GetByte()

Byte GetByte ( )

inline

Fetch a single byte from the current block, advancing the cursor.

Definition at line 313 of file block_reader.hpp.

◆ GetItemBatch()

std::vector<Block> GetItemBatch ( size_t n )

inline

Read n items, however, do not deserialize them but deliver them as a vector of (unpinned) Block objects. This is used to take out a range of items, the internal item cursor is advanced by n.

Definition at line 140 of file block_reader.hpp.

◆ GetRaw()

TLX_ATTRIBUTE_ALWAYS_INLINE Type GetRaw ( )

inline

Fetch a single item of the template type Type from the buffer, advancing the cursor. Be careful with implicit type conversions!

Definition at line 326 of file block_reader.hpp.

◆ HasNext()

TLX_ATTRIBUTE_ALWAYS_INLINE bool HasNext ( )

inline

HasNext() returns true if at least one more item is available.

Definition at line 118 of file block_reader.hpp.

Referenced by ReduceByHashPostPhase< TableItem, Key, ValueType, KeyExtractor, ReduceFunction, thrill::api::ReduceNode::Emitter, VolatileKey, ReduceConfig, thrill::core::ReduceByHash, KeyEqualFunction >::Flush(), LocationDetection< thrill::api::JoinNode::HashCount >::Flush(), ReducePrePhase< TableItem, Key, Value, KeyExtractor, ReduceFunction, VolatileKey, BlockWriter, ReduceConfig, IndexFunction, EqualToFunction, HashFunction, true >::FlushPartition(), BlockReader< CatBlockSource >::GetItemBatch(), File::GetReaderAt(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::JoinAllElements(), BlockReader< CatBlockSource >::Next(), BlockReader< CatBlockSource >::NextNoSelfVerify(), DistributeNode< ValueType >::PushData(), ZipWithIndexNode< ValueType, ZipFunction >::PushData(), ReduceByIndexPostPhase< TableItem, Key, ValueType, KeyExtractor, ReduceFunction, thrill::api::ReduceToIndexNode::Emitter, VolatileKey, ReduceConfig >::PushData(), ReduceByHashPostPhase< TableItem, Key, ValueType, KeyExtractor, ReduceFunction, thrill::api::ReduceNode::Emitter, VolatileKey, ReduceConfig, thrill::core::ReduceByHash, KeyEqualFunction >::PushData(), ReadBinaryNode< ValueType >::PushData(), ConcatNode< ValueType >::PushData(), DIANode< StackInput >::PushFile(), BlockReader< CatBlockSource >::ReadComplete(), and Stream::ScatterKeep().

◆ LoadBlock()

void LoadBlock ( PinnedBlock && block )

inlineprivate

Load Block into Reader.

Definition at line 391 of file block_reader.hpp.

Referenced by BlockReader< CatBlockSource >::GetItemBatch().

◆ Next()

TLX_ATTRIBUTE_ALWAYS_INLINE T Next ( )

inline

Next() reads a complete item T.

Definition at line 89 of file block_reader.hpp.

Referenced by WriteLinesOneNode< ValueType >::Execute(), ReduceByHashPostPhase< TableItem, Key, ValueType, KeyExtractor, ReduceFunction, thrill::api::ReduceNode::Emitter, VolatileKey, ReduceConfig, thrill::core::ReduceByHash, KeyEqualFunction >::Flush(), LocationDetection< thrill::api::JoinNode::HashCount >::Flush(), ReducePrePhase< TableItem, Key, Value, KeyExtractor, ReduceFunction, VolatileKey, BlockWriter, ReduceConfig, IndexFunction, EqualToFunction, HashFunction, true >::FlushPartition(), File::GetItemAt(), DistributeNode< ValueType >::PushData(), PrefixSumNode< ValueType, SumFunction, Inclusive >::PushData(), ReduceByIndexPostPhase< TableItem, Key, ValueType, KeyExtractor, ReduceFunction, thrill::api::ReduceToIndexNode::Emitter, VolatileKey, ReduceConfig >::PushData(), OverlapWindowNode< ValueType, Input, WindowFunction, PartialWindowFunction >::PushData(), ReduceByHashPostPhase< TableItem, Key, ValueType, KeyExtractor, ReduceFunction, thrill::api::ReduceNode::Emitter, VolatileKey, ReduceConfig, thrill::core::ReduceByHash, KeyEqualFunction >::PushData(), ConcatNode< ValueType >::PushData(), DisjointWindowNode< ValueType, Input, WindowFunction, PartialWindowFunction >::PushData(), DIANode< StackInput >::PushFile(), and SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >::TransmitItems().

◆ NextBlock()

bool NextBlock ( )

inlineprivate

Call source_.NextBlock with appropriate parameters.

Definition at line 371 of file block_reader.hpp.

Referenced by BlockReader< CatBlockSource >::GetByte(), BlockReader< CatBlockSource >::HasNext(), BlockReader< CatBlockSource >::Read(), and BlockReader< CatBlockSource >::Skip().

◆ NextNoSelfVerify()

TLX_ATTRIBUTE_ALWAYS_INLINE T NextNoSelfVerify ( )

inline

Next() reads a complete item T, without item counter or self verification

Definition at line 111 of file block_reader.hpp.

◆ operator=() [1/2]

BlockReader& operator= ( const BlockReader< BlockSource > & )

delete

non-copyable: delete assignment operator

Referenced by BlockReader< CatBlockSource >::source().

◆ operator=() [2/2]

BlockReader& operator= ( BlockReader< BlockSource > && )

default

move-assignment operator: default

◆ Read() [1/2]

BlockReader& Read	(	void *	outdata,
		size_t	size
	)

inline

Fetch a number of unstructured bytes from the current block, advancing the cursor.

Definition at line 265 of file block_reader.hpp.

Referenced by BlockReader< CatBlockSource >::GetRaw(), and BlockReader< CatBlockSource >::Read().

◆ Read() [2/2]

std::string Read ( size_t datalen )

inline

Fetch a number of unstructured bytes from the buffer as std::string, advancing the cursor.

Definition at line 290 of file block_reader.hpp.

◆ ReadComplete()

std::vector<ItemType> ReadComplete ( )

inline

Return complete contents until empty as a std::vector<T>. Use this only if you are sure that it will fit into memory, -> only use it for tests.

Definition at line 130 of file block_reader.hpp.

◆ Skip()

BlockReader& Skip	(	size_t	items,
		size_t	bytes
	)

inline

Advance the cursor given number of bytes without reading them.

Definition at line 297 of file block_reader.hpp.

Referenced by BlockReader< CatBlockSource >::GetItemBatch(), and File::GetReaderAt().

◆ source()

BlockSource& source ( )

inline

Return reference to enclosed BlockSource.

Definition at line 56 of file block_reader.hpp.

◆ typecode_verify()

size_t typecode_verify ( ) const

inline

Returns typecode_verify_.

Definition at line 81 of file block_reader.hpp.

Referenced by File::GetReaderAt().

Member Data Documentation

◆ block_

PinnedBlock block_

private

The current block being read, this holds a shared pointer reference.

Definition at line 352 of file block_reader.hpp.

Referenced by BlockReader< CatBlockSource >::byte_block(), BlockReader< CatBlockSource >::CopyBlock(), BlockReader< CatBlockSource >::GetItemBatch(), and BlockReader< CatBlockSource >::NextBlock().

◆ block_collect_

std::vector<PinnedBlock>* block_collect_ = nullptr

private

pointer to vector to collect blocks in GetItemRange.

Definition at line 364 of file block_reader.hpp.

Referenced by BlockReader< CatBlockSource >::GetItemBatch().

◆ current_

const Byte* current_ = nullptr

private

current read pointer into current block of file.

Definition at line 355 of file block_reader.hpp.

Referenced by BlockReader< CatBlockSource >::CopyBlock(), BlockReader< CatBlockSource >::GetByte(), BlockReader< CatBlockSource >::GetItemBatch(), BlockReader< CatBlockSource >::GetRaw(), BlockReader< CatBlockSource >::HasNext(), BlockReader< CatBlockSource >::Read(), and BlockReader< CatBlockSource >::Skip().

◆ end_

const Byte* end_ = nullptr

private

pointer to end of current block.

Definition at line 358 of file block_reader.hpp.

Referenced by BlockReader< CatBlockSource >::CopyBlock(), BlockReader< CatBlockSource >::GetByte(), BlockReader< CatBlockSource >::GetItemBatch(), BlockReader< CatBlockSource >::GetRaw(), BlockReader< CatBlockSource >::HasNext(), BlockReader< CatBlockSource >::Read(), and BlockReader< CatBlockSource >::Skip().

◆ num_items_

size_t num_items_ = 0

private

remaining number of items starting in this block

Definition at line 361 of file block_reader.hpp.

Referenced by BlockReader< CatBlockSource >::CopyBlock(), BlockReader< CatBlockSource >::GetItemBatch(), BlockReader< CatBlockSource >::Next(), and BlockReader< CatBlockSource >::Skip().

◆ self_verify

constexpr bool self_verify = common::g_self_verify

static

Definition at line 46 of file block_reader.hpp.

◆ source_

BlockSource source_

private

Instance of BlockSource. This is NOT a reference, as to enable embedding of FileBlockSource to compose classes into File::Reader.

Definition at line 349 of file block_reader.hpp.

Referenced by BlockReader< CatBlockSource >::GetItemBatch(), and BlockReader< CatBlockSource >::source().

◆ typecode_verify_

bool typecode_verify_

private

flag whether the underlying data contains self verify type codes from BlockReader, this is false to needed to read external files.

Definition at line 368 of file block_reader.hpp.

Referenced by BlockReader< CatBlockSource >::CopyBlock(), BlockReader< CatBlockSource >::GetItemBatch(), BlockReader< CatBlockSource >::Next(), and BlockReader< CatBlockSource >::typecode_verify().

The documentation for this class was generated from the following file:

thrill/data/block_reader.hpp

Detailed Description

template<typename BlockSource> class thrill::data::BlockReader< BlockSource >

Public Member Functions

Static Public Attributes

Private Member Functions

Private Attributes

Constructor & Destructor Documentation

◆ BlockReader() [1/4]

◆ BlockReader() [2/4]

◆ BlockReader() [3/4]

◆ BlockReader() [4/4]

Member Function Documentation

◆ byte_block()

◆ CopyBlock()

◆ GetByte()

◆ GetItemBatch()

◆ GetRaw()

◆ HasNext()

◆ LoadBlock()

◆ Next()

◆ NextBlock()

◆ NextNoSelfVerify()

◆ operator=() [1/2]

◆ operator=() [2/2]

◆ Read() [1/2]

◆ Read() [2/2]

◆ ReadComplete()

◆ Skip()

◆ source()

◆ typecode_verify()

Member Data Documentation

◆ block_

◆ block_collect_

◆ current_

◆ end_

◆ num_items_

◆ self_verify

◆ source_

◆ typecode_verify_

template<typename BlockSource>
class thrill::data::BlockReader< BlockSource >