Detailed Description

template<typename ValueType>
class thrill::api::ReadBinaryNode< ValueType >

A DIANode which performs a line-based Read operation.

Read reads a file from the file system and emits it as a DIA.

Definition at line 45 of file read_binary.hpp.

Inheritance diagram for ReadBinaryNode< ValueType >:

Collaboration diagram for ReadBinaryNode< ValueType >:

#include <read_binary.hpp>

Classes
struct	FileInfo
	structure to store info on what to read from files More...

class	VfsFileBlockSource

Public Types
using	Super = SourceNode< ValueType >

using	VfsFileBlockReader = data::BlockReader< VfsFileBlockSource >

Public Types inherited from SourceNode< ValueType >
using	Super = DIANode< ValueType >

Public Types inherited from DIANode< ValueType >
using	Callback = tlx::delegate< void(const ValueType &)>

Public Types inherited from DIABase
using	DIABasePtr = tlx::CountingPtr< DIABase >

Public Member Functions
	ReadBinaryNode (Context &ctx, const std::vector< std::string > &globlist, uint64_t size_limit, bool local_storage)

	ReadBinaryNode (Context &ctx, const std::string &glob, uint64_t size_limit, bool local_storage)

void	Dispose () final
	Virtual clear method. Triggers actual disposing in sub-classes. More...

void	PushData (bool consume) final
	Virtual method for pushing data. Triggers actual pushing in sub-classes. More...

Public Member Functions inherited from SourceNode< ValueType >
	SourceNode (Context &ctx, const char *label)

void	Execute () override
	SourceNodes generally do not Execute, they only PushData. More...

Public Member Functions inherited from DIANode< ValueType >
	DIANode (Context &ctx, const char *label, const std::initializer_list< size_t > &parent_ids, const std::initializer_list< DIABasePtr > &parents)
	Constructor for a DIANode, which sets references to the parent nodes. More...

	DIANode (Context &ctx, const char *label, std::vector< size_t > &&parent_ids, std::vector< DIABasePtr > &&parents)
	Constructor for a DIANode, which sets references to the parent nodes. More...

virtual void	AddChild (DIABase *node, const Callback &callback=Callback(), size_t parent_index=0)
	Enables children to push their "folded" function chains to their parent. More...

std::vector< DIABase * >	children () const override
	Returns the children of this DIABase. More...

void	PushFile (data::File &file, bool consume) const

void	PushItem (const ValueType &item) const
	Method for derived classes to Push a single item to all children. More...

void	RemoveAllChildren () override

void	RemoveChild (DIABase *node) override

void	RunPushData () override

Public Member Functions inherited from DIABase
	DIABase (Context &ctx, const char *label, const std::initializer_list< size_t > &parent_ids, const std::initializer_list< DIABasePtr > &parents)
	The constructor for a DIABase. More...

	DIABase (Context &ctx, const char *label, std::vector< size_t > &&parent_ids, std::vector< DIABasePtr > &&parents)
	The constructor for a DIABase. More...

	DIABase (const DIABase &)=delete
	non-copyable: delete copy-constructor More...

	DIABase (DIABase &&)=default
	move-constructor: default More...

virtual	~DIABase ()
	Virtual destructor for a DIABase. More...

virtual size_t	consume_counter () const
	Returns consume_counter_. More...

Context &	context ()
	Returns the api::Context of this DIABase. More...

virtual void	DecConsumeCounter (size_t counter)

const size_t &	dia_id () const
	return unique id of DIANode subclass as stored by StatsNode More...

virtual bool	ForwardDataOnly () const

virtual void	IncConsumeCounter (size_t counter)

const char *	label () const
	return label() of DIANode subclass as stored by StatsNode More...

mem::Manager &	mem_manager ()
	Return the Context's memory manager. More...

DIABase &	operator= (const DIABase &)=delete
	non-copyable: delete assignment operator More...

DIABase &	operator= (DIABase &&)=default
	move-assignment operator: default More...

std::vector< size_t >	parent_ids () const
	Returns the parents of this DIABase. More...

const std::vector< DIABasePtr > &	parents () const
	Returns the parents of this DIABase. More...

void	RemoveParent (DIABase *p)
	Remove a parent. More...

virtual bool	RequireParentPushData (size_t) const

void	RunScope ()

void	set_mem_limit (const DIAMemUse &mem_limit)

void	set_state (const DIAState &state)

virtual void	SetConsumeCounter (size_t counter)

DIAState	state () const

virtual DIAMemUse	PreOpMemUse ()
	Amount of RAM used by PreOp after StartPreOp() More...

virtual void	StartPreOp (size_t)
	Virtual method for preparing start of PushData. More...

virtual bool	OnPreOpFile (const data::File &, size_t)

virtual void	StopPreOp (size_t)
	Virtual method for preparing end of PushData. More...

virtual DIAMemUse	ExecuteMemUse ()
	Amount of RAM used by Execute() More...

virtual DIAMemUse	PushDataMemUse ()
	Amount of RAM used by PushData() More...

Public Member Functions inherited from ReferenceCounter
	ReferenceCounter () noexcept
	new objects have zero reference count More...

	ReferenceCounter (const ReferenceCounter &) noexcept
	coping still creates a new object with zero reference count More...

	~ReferenceCounter ()

bool	dec_reference () const noexcept
	Call whenever resetting (i.e. More...

void	inc_reference () const noexcept
	Call whenever setting a pointer to the object. More...

ReferenceCounter &	operator= (const ReferenceCounter &) noexcept
	assignment operator, leaves pointers unchanged More...

size_t	reference_count () const noexcept
	Return the number of references to this object (for debugging) More...

bool	unique () const noexcept
	Test if the ReferenceCounter is referenced by only one CountingPtr. More...

Static Public Attributes
static constexpr size_t	fixed_size_
	fixed size of ValueType or zero. More...

static constexpr bool	is_fixed_size_
	flag whether ValueType is fixed size More...

static constexpr uint64_t	no_size_limit_
	sentinel to disable size limit More...

Static Public Attributes inherited from DIABase
static constexpr size_t	kNeverConsume = static_cast<size_t>(-1)
	Never full consume. More...

Private Attributes
data::File	ext_file_ { context_.GetFile(this) }

std::vector< FileInfo >	my_files_
	list of files for non-mapped File push More...

size_t	stats_total_bytes = 0

size_t	stats_total_reads = 0

bool	use_ext_file_ = false
	File containing Blocks mapped directly to a io fileimpl. More...

Static Private Attributes
static constexpr bool	debug = false

static constexpr bool	debug_no_extfile = false
	for testing old method of pushing items instead of PushFile(). More...

Additional Inherited Members
Public Attributes inherited from DIABase
common::JsonLogger	logger_

Protected Attributes inherited from DIANode< ValueType >
std::vector< Child >	children_
	Callback functions from the child nodes. More...

Protected Attributes inherited from DIABase
Context &	context_
	associated Context More...

const size_t	dia_id_
	DIA serial id. More...

const char *const	label_
	DOp node static label. More...

DIAState	state_ = DIAState::NEW
	State of the DIANode. State is NEW on creation. More...

std::vector< DIABasePtr >	parents_
	Parents of this DIABase. More...

DIAMemUse	mem_limit_ = 0

size_t	consume_counter_ = 1

Member Typedef Documentation

◆ Super

using Super = SourceNode<ValueType>

Definition at line 56 of file read_binary.hpp.

◆ VfsFileBlockReader

using VfsFileBlockReader = data::BlockReader<VfsFileBlockSource>

Definition at line 59 of file read_binary.hpp.

Constructor & Destructor Documentation

◆ ReadBinaryNode() [1/2]

ReadBinaryNode	(	Context &	ctx,
		const std::vector< std::string > &	globlist,
		uint64_t	size_limit,
		bool	local_storage
	)

inline

Definition at line 82 of file read_binary.hpp.

References File::AppendBlock(), Range::begin, Context::block_pool(), Context::CalculateLocalRange(), Context::CalculateLocalRangeOnHost(), FileList::contains_compressed, FileList::contains_remote_uri, DIABase::context_, thrill::data::default_block_size, die, Range::end, ReadBinaryNode< ValueType >::ext_file_, thrill::vfs::File, ReadBinaryNode< ValueType >::fixed_size_, thrill::vfs::Glob(), ReadBinaryNode< ValueType >::FileInfo::is_compressed, tlx::join(), LOG, BlockPool::MapExternalBlock(), max(), min(), ReadBinaryNode< ValueType >::my_files_, file::NO_LOCK, Context::num_workers(), ReadBinaryNode< ValueType >::FileInfo::path, ReadBinaryNode< ValueType >::FileInfo::range, file::RDONLY, FileList::size_ex_psum(), FileList::size_inc_psum(), sLOG, FileList::total_size, and ReadBinaryNode< ValueType >::use_ext_file_.

◆ ReadBinaryNode() [2/2]

ReadBinaryNode	(	Context &	ctx,
		const std::string &	glob,
		uint64_t	size_limit,
		bool	local_storage
	)

inline

Definition at line 224 of file read_binary.hpp.

Member Function Documentation

◆ Dispose()

void Dispose ( )

inlinefinalvirtual

Virtual clear method. Triggers actual disposing in sub-classes.

Reimplemented from DIABase.

Definition at line 257 of file read_binary.hpp.

References File::Clear(), ReadBinaryNode< ValueType >::ext_file_, ReadBinaryNode< ValueType >::my_files_, and tlx::vector_free().

◆ PushData()

void PushData ( bool consume )

inlinefinalvirtual

Virtual method for pushing data. Triggers actual pushing in sub-classes.

Implements DIABase.

Definition at line 229 of file read_binary.hpp.

References DIABase::context_, ReadBinaryNode< ValueType >::ext_file_, BlockReader< BlockSource >::HasNext(), LOG, DIABase::logger_, ReadBinaryNode< ValueType >::my_files_, DIANode< ValueType >::PushFile(), DIANode< ValueType >::PushItem(), ReadBinaryNode< ValueType >::stats_total_bytes, ReadBinaryNode< ValueType >::stats_total_reads, and ReadBinaryNode< ValueType >::use_ext_file_.

Member Data Documentation

◆ debug

constexpr bool debug = false

staticprivate

Definition at line 47 of file read_binary.hpp.

◆ debug_no_extfile

constexpr bool debug_no_extfile = false

staticprivate

for testing old method of pushing items instead of PushFile().

Definition at line 50 of file read_binary.hpp.

◆ ext_file_

data::File ext_file_ { context_.GetFile(this) }

private

Definition at line 268 of file read_binary.hpp.

Referenced by ReadBinaryNode< ValueType >::Dispose(), ReadBinaryNode< ValueType >::PushData(), and ReadBinaryNode< ValueType >::ReadBinaryNode().

◆ fixed_size_

constexpr size_t fixed_size_

static

Initial value:

=

data::Serialization<VfsFileBlockReader, ValueType>::fixed_size

fixed size of ValueType or zero.

Definition at line 66 of file read_binary.hpp.

Referenced by ReadBinaryNode< ValueType >::ReadBinaryNode().

◆ is_fixed_size_

constexpr bool is_fixed_size_

static

Initial value:

=

data::Serialization<VfsFileBlockReader, ValueType>::is_fixed_size

flag whether ValueType is fixed size

Definition at line 62 of file read_binary.hpp.

◆ my_files_

std::vector<FileInfo> my_files_

private

list of files for non-mapped File push

Definition at line 264 of file read_binary.hpp.

Referenced by ReadBinaryNode< ValueType >::Dispose(), ReadBinaryNode< ValueType >::PushData(), and ReadBinaryNode< ValueType >::ReadBinaryNode().

◆ no_size_limit_

constexpr uint64_t no_size_limit_

static

Initial value:

=

std::numeric_limits<uint64_t>::max()

sentinel to disable size limit

Definition at line 79 of file read_binary.hpp.

◆ stats_total_bytes

size_t stats_total_bytes = 0

private

Definition at line 270 of file read_binary.hpp.

Referenced by ReadBinaryNode< ValueType >::PushData().

◆ stats_total_reads

size_t stats_total_reads = 0

private

Definition at line 271 of file read_binary.hpp.

Referenced by ReadBinaryNode< ValueType >::PushData().

◆ use_ext_file_

bool use_ext_file_ = false

private

File containing Blocks mapped directly to a io fileimpl.

Definition at line 267 of file read_binary.hpp.

Referenced by ReadBinaryNode< ValueType >::PushData(), and ReadBinaryNode< ValueType >::ReadBinaryNode().

The documentation for this class was generated from the following file:

thrill/api/read_binary.hpp

Detailed Description

template<typename ValueType> class thrill::api::ReadBinaryNode< ValueType >

Classes

Public Types

Public Member Functions

Static Public Attributes

Private Attributes

Static Private Attributes

Additional Inherited Members

Member Typedef Documentation

◆ Super

◆ VfsFileBlockReader

Constructor & Destructor Documentation

◆ ReadBinaryNode() [1/2]

◆ ReadBinaryNode() [2/2]

Member Function Documentation

◆ Dispose()

◆ PushData()

Member Data Documentation

◆ debug

◆ debug_no_extfile

◆ ext_file_

◆ fixed_size_

◆ is_fixed_size_

◆ my_files_

◆ no_size_limit_

◆ stats_total_bytes

◆ stats_total_reads

◆ use_ext_file_

template<typename ValueType>
class thrill::api::ReadBinaryNode< ValueType >