Detailed Description

template<typename ValueType>
class thrill::api::WriteLinesNode< ValueType >

Definition at line 33 of file write_lines.hpp.

Inheritance diagram for WriteLinesNode< ValueType >:

Collaboration diagram for WriteLinesNode< ValueType >:

#include <write_lines.hpp>

Public Types
using	Super = ActionNode

using	ValueType_ = ValueType
	input type is the parent's output value type. More...

Public Types inherited from DIABase
using	DIABasePtr = tlx::CountingPtr< DIABase >

Public Member Functions
template<typename ParentDIA >
	WriteLinesNode (const ParentDIA &parent, const std::string &path_out, size_t target_file_size)

void	Execute () final
	Virtual execution method. Triggers actual computation in sub-classes. More...

void	PreOp (const std::string &input)

DIAMemUse	PreOpMemUse () final
	Amount of RAM used by PreOp after StartPreOp() More...

void	StartPreOp (size_t) final
	Virtual method for preparing start of PushData. More...

void	StopPreOp (size_t) final
	Closes the output file, write last buffer. More...

Public Member Functions inherited from ActionNode
	ActionNode (Context &ctx, const char *label, const std::initializer_list< size_t > &parent_ids, const std::initializer_list< DIABasePtr > &parents)

std::vector< DIABase * >	children () const final
	ActionNodes do not have children. More...

void	DecConsumeCounter (size_t) final

void	IncConsumeCounter (size_t) final

void	PushData (bool) final
	ActionNodes do not push data, they only Execute. More...

void	RemoveAllChildren () final
	ActionNodes do not have children. More...

void	RemoveChild (DIABase *) final
	ActionNodes do not have children. More...

void	RunPushData () final
	ActionNodes do not push data, they only Execute. More...

void	SetConsumeCounter (size_t) final

Public Member Functions inherited from DIABase
	DIABase (Context &ctx, const char *label, const std::initializer_list< size_t > &parent_ids, const std::initializer_list< DIABasePtr > &parents)
	The constructor for a DIABase. More...

	DIABase (Context &ctx, const char *label, std::vector< size_t > &&parent_ids, std::vector< DIABasePtr > &&parents)
	The constructor for a DIABase. More...

	DIABase (const DIABase &)=delete
	non-copyable: delete copy-constructor More...

	DIABase (DIABase &&)=default
	move-constructor: default More...

virtual	~DIABase ()
	Virtual destructor for a DIABase. More...

virtual size_t	consume_counter () const
	Returns consume_counter_. More...

Context &	context ()
	Returns the api::Context of this DIABase. More...

const size_t &	dia_id () const
	return unique id of DIANode subclass as stored by StatsNode More...

virtual bool	ForwardDataOnly () const

const char *	label () const
	return label() of DIANode subclass as stored by StatsNode More...

mem::Manager &	mem_manager ()
	Return the Context's memory manager. More...

DIABase &	operator= (const DIABase &)=delete
	non-copyable: delete assignment operator More...

DIABase &	operator= (DIABase &&)=default
	move-assignment operator: default More...

std::vector< size_t >	parent_ids () const
	Returns the parents of this DIABase. More...

const std::vector< DIABasePtr > &	parents () const
	Returns the parents of this DIABase. More...

void	RemoveParent (DIABase *p)
	Remove a parent. More...

virtual bool	RequireParentPushData (size_t) const

void	RunScope ()

void	set_mem_limit (const DIAMemUse &mem_limit)

void	set_state (const DIAState &state)

DIAState	state () const

virtual bool	OnPreOpFile (const data::File &, size_t)

virtual DIAMemUse	ExecuteMemUse ()
	Amount of RAM used by Execute() More...

virtual DIAMemUse	PushDataMemUse ()
	Amount of RAM used by PushData() More...

virtual void	Dispose ()
	Virtual clear method. Triggers actual disposing in sub-classes. More...

Public Member Functions inherited from ReferenceCounter
	ReferenceCounter () noexcept
	new objects have zero reference count More...

	ReferenceCounter (const ReferenceCounter &) noexcept
	coping still creates a new object with zero reference count More...

	~ReferenceCounter ()

bool	dec_reference () const noexcept
	Call whenever resetting (i.e. More...

void	inc_reference () const noexcept
	Call whenever setting a pointer to the object. More...

ReferenceCounter &	operator= (const ReferenceCounter &) noexcept
	assignment operator, leaves pointers unchanged More...

size_t	reference_count () const noexcept
	Return the number of references to this object (for debugging) More...

bool	unique () const noexcept
	Test if the ReferenceCounter is referenced by only one CountingPtr. More...

Private Attributes
size_t	current_buffer_size_ = 0
	Current buffer size. More...

size_t	current_file_size_ = 0
	Current file size in bytes. More...

size_t	max_buffer_size_
	Maximum buffer size. More...

std::string	out_pathbase_
	Base path of the output file. More...

size_t	out_serial_ = 1
	File serial number for this worker. More...

size_t	stats_total_bytes_ = 0

size_t	stats_total_elements_ = 0

size_t	stats_total_writes_ = 0

vfs::WriteStreamPtr	stream_
	File to wrtie to. More...

size_t	target_file_size_
	Targetl file size in bytes. More...

common::StatsTimerStopped	timer

net::BufferBuilder	write_buffer_
	Write buffer. More...

Static Private Attributes
static constexpr bool	debug = false

Additional Inherited Members
Public Attributes inherited from DIABase
common::JsonLogger	logger_

Static Public Attributes inherited from DIABase
static constexpr size_t	kNeverConsume = static_cast<size_t>(-1)
	Never full consume. More...

Protected Attributes inherited from DIABase
Context &	context_
	associated Context More...

const size_t	dia_id_
	DIA serial id. More...

const char *const	label_
	DOp node static label. More...

DIAState	state_ = DIAState::NEW
	State of the DIANode. State is NEW on creation. More...

std::vector< DIABasePtr >	parents_
	Parents of this DIABase. More...

DIAMemUse	mem_limit_ = 0

size_t	consume_counter_ = 1

Member Typedef Documentation

◆ Super

using Super = ActionNode

Definition at line 38 of file write_lines.hpp.

◆ ValueType_

using ValueType_ = ValueType

input type is the parent's output value type.

Definition at line 42 of file write_lines.hpp.

Constructor & Destructor Documentation

◆ WriteLinesNode()

WriteLinesNode	(	const ParentDIA &	parent,
		const std::string &	path_out,
		size_t	target_file_size
	)

inline

Definition at line 45 of file write_lines.hpp.

References DIABase::context_, thrill::data::default_block_size, thrill::vfs::FillFilePattern(), WriteLinesNode< ValueType >::max_buffer_size_, min(), Context::my_rank(), thrill::vfs::OpenWriteStream(), WriteLinesNode< ValueType >::out_pathbase_, WriteLinesNode< ValueType >::PreOp(), tlx::round_up_to_power_of_two(), sLOG, WriteLinesNode< ValueType >::stream_, and WriteLinesNode< ValueType >::target_file_size_.

Member Function Documentation

◆ Execute()

void Execute ( )

inlinefinalvirtual

Virtual execution method. Triggers actual computation in sub-classes.

Implements DIABase.

Definition at line 141 of file write_lines.hpp.

◆ PreOp()

void PreOp ( const std::string & input )

inline

Definition at line 79 of file write_lines.hpp.

Referenced by WriteLinesNode< ValueType >::WriteLinesNode().

◆ PreOpMemUse()

DIAMemUse PreOpMemUse ( )

inlinefinalvirtual

Amount of RAM used by PreOp after StartPreOp()

Reimplemented from DIABase.

Definition at line 71 of file write_lines.hpp.

References WriteLinesNode< ValueType >::max_buffer_size_.

◆ StartPreOp()

void StartPreOp ( size_t )

inlinefinalvirtual

Virtual method for preparing start of PushData.

Reimplemented from DIABase.

Definition at line 75 of file write_lines.hpp.

References WriteLinesNode< ValueType >::max_buffer_size_, BufferBuilder::Reserve(), and WriteLinesNode< ValueType >::write_buffer_.

◆ StopPreOp()

void StopPreOp ( size_t )

inlinefinalvirtual

Closes the output file, write last buffer.

Reimplemented from DIABase.

Definition at line 122 of file write_lines.hpp.

References WriteLinesNode< ValueType >::current_buffer_size_, BufferBuilder::data(), DIABase::logger_, WriteLinesNode< ValueType >::out_serial_, sLOG, WriteLinesNode< ValueType >::stats_total_bytes_, WriteLinesNode< ValueType >::stats_total_elements_, WriteLinesNode< ValueType >::stats_total_writes_, WriteLinesNode< ValueType >::stream_, WriteLinesNode< ValueType >::timer, and WriteLinesNode< ValueType >::write_buffer_.

Member Data Documentation

◆ current_buffer_size_

size_t current_buffer_size_ = 0

private

Current buffer size.

Definition at line 163 of file write_lines.hpp.

Referenced by WriteLinesNode< ValueType >::PreOp(), and WriteLinesNode< ValueType >::StopPreOp().

◆ current_file_size_

size_t current_file_size_ = 0

private

Current file size in bytes.

Definition at line 148 of file write_lines.hpp.

Referenced by WriteLinesNode< ValueType >::PreOp().

◆ debug

constexpr bool debug = false

staticprivate

Definition at line 35 of file write_lines.hpp.

◆ max_buffer_size_

size_t max_buffer_size_

private

Maximum buffer size.

Definition at line 160 of file write_lines.hpp.

Referenced by WriteLinesNode< ValueType >::PreOp(), WriteLinesNode< ValueType >::PreOpMemUse(), WriteLinesNode< ValueType >::StartPreOp(), and WriteLinesNode< ValueType >::WriteLinesNode().

◆ out_pathbase_

std::string out_pathbase_

private

Base path of the output file.

Definition at line 145 of file write_lines.hpp.

Referenced by WriteLinesNode< ValueType >::PreOp(), and WriteLinesNode< ValueType >::WriteLinesNode().

◆ out_serial_

size_t out_serial_ = 1

private

File serial number for this worker.

Definition at line 151 of file write_lines.hpp.

Referenced by WriteLinesNode< ValueType >::PreOp(), and WriteLinesNode< ValueType >::StopPreOp().

◆ stats_total_bytes_

size_t stats_total_bytes_ = 0

private

Definition at line 170 of file write_lines.hpp.

Referenced by WriteLinesNode< ValueType >::PreOp(), and WriteLinesNode< ValueType >::StopPreOp().

◆ stats_total_elements_

size_t stats_total_elements_ = 0

private

Definition at line 171 of file write_lines.hpp.

Referenced by WriteLinesNode< ValueType >::PreOp(), and WriteLinesNode< ValueType >::StopPreOp().

◆ stats_total_writes_

size_t stats_total_writes_ = 0

private

Definition at line 172 of file write_lines.hpp.

Referenced by WriteLinesNode< ValueType >::PreOp(), and WriteLinesNode< ValueType >::StopPreOp().

◆ stream_

vfs::WriteStreamPtr stream_

private

File to wrtie to.

Definition at line 154 of file write_lines.hpp.

Referenced by WriteLinesNode< ValueType >::PreOp(), WriteLinesNode< ValueType >::StopPreOp(), and WriteLinesNode< ValueType >::WriteLinesNode().

◆ target_file_size_

size_t target_file_size_

private

Targetl file size in bytes.

Definition at line 166 of file write_lines.hpp.

Referenced by WriteLinesNode< ValueType >::PreOp(), and WriteLinesNode< ValueType >::WriteLinesNode().

◆ timer

common::StatsTimerStopped timer

private

Definition at line 168 of file write_lines.hpp.

Referenced by WriteLinesNode< ValueType >::PreOp(), and WriteLinesNode< ValueType >::StopPreOp().

◆ write_buffer_

net::BufferBuilder write_buffer_

private

Write buffer.

Definition at line 157 of file write_lines.hpp.

Referenced by WriteLinesNode< ValueType >::PreOp(), WriteLinesNode< ValueType >::StartPreOp(), and WriteLinesNode< ValueType >::StopPreOp().

The documentation for this class was generated from the following file:

thrill/api/write_lines.hpp

Detailed Description

template<typename ValueType> class thrill::api::WriteLinesNode< ValueType >

Public Types

Public Member Functions

Private Attributes

Static Private Attributes

Additional Inherited Members

Member Typedef Documentation

◆ Super

◆ ValueType_

Constructor & Destructor Documentation

◆ WriteLinesNode()

Member Function Documentation

◆ Execute()

◆ PreOp()

◆ PreOpMemUse()

◆ StartPreOp()

◆ StopPreOp()

Member Data Documentation

◆ current_buffer_size_

◆ current_file_size_

◆ debug

◆ max_buffer_size_

◆ out_pathbase_

◆ out_serial_

◆ stats_total_bytes_

◆ stats_total_elements_

◆ stats_total_writes_

◆ stream_

◆ target_file_size_

◆ timer

◆ write_buffer_

template<typename ValueType>
class thrill::api::WriteLinesNode< ValueType >