Detailed Description

template<typename ValueType, typename ZipFunction, bool Pad, bool UnequalCheck, bool NoRebalance, size_t kNumInputs>
class thrill::api::ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >

A DIANode which performs a Zip operation.

Zip combines two DIAs element-by-element. The ZipNode stores the zip_function UDF. The chainable LOps are stored in the Stack.

               ParentStack0 ParentStack1
                +--------+   +--------+
                |        |   |        |  A ParentStackX is called with
                |        |   |        |  ParentInputX, and must deliver
                |        |   |        |  a ZipArgX item.
              +-+--------+---+--------+-+
              | | PreOp0 |   | PreOp1 | |
              | +--------+   +--------+ |
   DIA<T> --> |           Zip           |
              |        +-------+        |
              |        |PostOp |        |
              +--------+-------+--------+
                       |       | New DIA<T>::stack_ is started
                       |       | with PostOp to chain next nodes.
                       +-------+

Template Parameters

ValueType	Output type of the Zip operation.
ParentDIA0	Function stack, which contains the chained lambdas between the last and this DIANode for first input DIA.
ParentDIA1	Function stack, which contains the chained lambdas between the last and this DIANode for second input DIA.
ZipFunction	Type of the ZipFunction.

Definition at line 77 of file zip.hpp.

Inheritance diagram for ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >:

Collaboration diagram for ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >:

#include <zip.hpp>

Classes
class	ReaderNext
	Access CatReaders for different different parents. More...

class	RegisterParent
	Register Parent PreOp Hooks, instantiated and called for each Zip parent. More...

Public Member Functions
template<typename ParentDIA0 , typename... ParentDIAs>
	ZipNode (const ZipFunction &zip_function, const ZipArgsTuple &padding, const ParentDIA0 &parent0, const ParentDIAs &... parents)
	Constructor for a ZipNode. More...

void	Dispose () final
	Virtual clear method. Triggers actual disposing in sub-classes. More...

void	Execute () final
	Virtual execution method. Triggers actual computation in sub-classes. More...

bool	OnPreOpFile (const data::File &file, size_t parent_index) final
	Receive a whole data::File of ValueType, but only if our stack is empty. More...

void	PushData (bool consume) final
	Virtual method for pushing data. Triggers actual pushing in sub-classes. More...

void	StartPreOp (size_t parent_index) final
	Virtual method for preparing start of PushData. More...

void	StopPreOp (size_t parent_index) final
	Virtual method for preparing end of PushData. More...

Public Member Functions inherited from DOpNode< ValueType >
	DOpNode (Context &ctx, const char *label, const std::initializer_list< size_t > &parent_ids, const std::initializer_list< DIABasePtr > &parents)
	Constructor for a DOpNode, which sets references to the parent nodes. More...

	DOpNode (Context &ctx, const char *label, std::vector< size_t > &&parent_ids, std::vector< DIABasePtr > &&parents)
	Constructor for a DOpNode, which sets references to the parent nodes. More...

Public Member Functions inherited from DIANode< ValueType >
	DIANode (Context &ctx, const char *label, const std::initializer_list< size_t > &parent_ids, const std::initializer_list< DIABasePtr > &parents)
	Constructor for a DIANode, which sets references to the parent nodes. More...

	DIANode (Context &ctx, const char *label, std::vector< size_t > &&parent_ids, std::vector< DIABasePtr > &&parents)
	Constructor for a DIANode, which sets references to the parent nodes. More...

virtual void	AddChild (DIABase *node, const Callback &callback=Callback(), size_t parent_index=0)
	Enables children to push their "folded" function chains to their parent. More...

std::vector< DIABase * >	children () const override
	Returns the children of this DIABase. More...

void	PushFile (data::File &file, bool consume) const

void	PushItem (const ValueType &item) const
	Method for derived classes to Push a single item to all children. More...

void	RemoveAllChildren () override

void	RemoveChild (DIABase *node) override

void	RunPushData () override

Public Member Functions inherited from DIABase
	DIABase (Context &ctx, const char *label, const std::initializer_list< size_t > &parent_ids, const std::initializer_list< DIABasePtr > &parents)
	The constructor for a DIABase. More...

	DIABase (Context &ctx, const char *label, std::vector< size_t > &&parent_ids, std::vector< DIABasePtr > &&parents)
	The constructor for a DIABase. More...

	DIABase (const DIABase &)=delete
	non-copyable: delete copy-constructor More...

	DIABase (DIABase &&)=default
	move-constructor: default More...

virtual	~DIABase ()
	Virtual destructor for a DIABase. More...

virtual size_t	consume_counter () const
	Returns consume_counter_. More...

Context &	context ()
	Returns the api::Context of this DIABase. More...

virtual void	DecConsumeCounter (size_t counter)

const size_t &	dia_id () const
	return unique id of DIANode subclass as stored by StatsNode More...

virtual bool	ForwardDataOnly () const

virtual void	IncConsumeCounter (size_t counter)

const char *	label () const
	return label() of DIANode subclass as stored by StatsNode More...

mem::Manager &	mem_manager ()
	Return the Context's memory manager. More...

DIABase &	operator= (const DIABase &)=delete
	non-copyable: delete assignment operator More...

DIABase &	operator= (DIABase &&)=default
	move-assignment operator: default More...

std::vector< size_t >	parent_ids () const
	Returns the parents of this DIABase. More...

const std::vector< DIABasePtr > &	parents () const
	Returns the parents of this DIABase. More...

void	RemoveParent (DIABase *p)
	Remove a parent. More...

virtual bool	RequireParentPushData (size_t) const

void	RunScope ()

void	set_mem_limit (const DIAMemUse &mem_limit)

void	set_state (const DIAState &state)

virtual void	SetConsumeCounter (size_t counter)

DIAState	state () const

virtual DIAMemUse	PreOpMemUse ()
	Amount of RAM used by PreOp after StartPreOp() More...

virtual DIAMemUse	ExecuteMemUse ()
	Amount of RAM used by Execute() More...

virtual DIAMemUse	PushDataMemUse ()
	Amount of RAM used by PushData() More...

Public Member Functions inherited from ReferenceCounter
	ReferenceCounter () noexcept
	new objects have zero reference count More...

	ReferenceCounter (const ReferenceCounter &) noexcept
	coping still creates a new object with zero reference count More...

	~ReferenceCounter ()

bool	dec_reference () const noexcept
	Call whenever resetting (i.e. More...

void	inc_reference () const noexcept
	Call whenever setting a pointer to the object. More...

ReferenceCounter &	operator= (const ReferenceCounter &) noexcept
	assignment operator, leaves pointers unchanged More...

size_t	reference_count () const noexcept
	Return the number of references to this object (for debugging) More...

bool	unique () const noexcept
	Test if the ReferenceCounter is referenced by only one CountingPtr. More...

Private Types
using	Super = DOpNode< ValueType >

template<size_t Index>
using	ZipArgN = typename common::FunctionTraits< ZipFunction >::template arg_plain< Index >

using	ZipArgsTuple = typename common::FunctionTraits< ZipFunction >::args_tuple_plain

Private Member Functions
template<size_t Index>
void	DoScatter ()
	Scatter items from DIA "Index" to other workers if necessary. More...

void	MainOp ()
	Receive elements from other workers. More...

Private Attributes
std::vector< data::File >	files_
	Files for intermediate storage. More...

const ZipArgsTuple	padding_
	padding for shorter DIAs More...

const std::array< bool, kNumInputs >	parent_stack_empty_
	Whether the parent stack is empty. More...

data::CatStreamPtr	streams_ [kNumInputs]
	Array of inbound CatStreams. More...

data::File::Writer	writers_ [kNumInputs]
	Writers to intermediate files. More...

ZipFunction	zip_function_
	Zip function. More...

Variables for Calculating Exchange
std::array< size_t, kNumInputs >	size_prefixsum_
	exclusive prefix sum over the number of items in workers More...

size_t	result_size_
	shortest size of Zipped inputs More...

Static Private Attributes
static constexpr bool	debug = false

static constexpr bool	stats_enabled = false
	Set this variable to true to enable generation and output of stats. More...

Additional Inherited Members
Public Types inherited from DOpNode< ValueType >
using	Super = DIANode< ValueType >

Public Types inherited from DIANode< ValueType >
using	Callback = tlx::delegate< void(const ValueType &)>

Public Types inherited from DIABase
using	DIABasePtr = tlx::CountingPtr< DIABase >

Public Attributes inherited from DIABase
common::JsonLogger	logger_

Static Public Attributes inherited from DIABase
static constexpr size_t	kNeverConsume = static_cast<size_t>(-1)
	Never full consume. More...

Protected Attributes inherited from DIANode< ValueType >
std::vector< Child >	children_
	Callback functions from the child nodes. More...

Protected Attributes inherited from DIABase
Context &	context_
	associated Context More...

const size_t	dia_id_
	DIA serial id. More...

const char *const	label_
	DOp node static label. More...

DIAState	state_ = DIAState::NEW
	State of the DIANode. State is NEW on creation. More...

std::vector< DIABasePtr >	parents_
	Parents of this DIABase. More...

DIAMemUse	mem_limit_ = 0

size_t	consume_counter_ = 1

Member Typedef Documentation

◆ Super

using Super = DOpNode<ValueType>

private

Definition at line 84 of file zip.hpp.

◆ ZipArgN

using ZipArgN = typename common::FunctionTraits<ZipFunction>::template arg_plain<Index>

private

Definition at line 89 of file zip.hpp.

◆ ZipArgsTuple

using ZipArgsTuple = typename common::FunctionTraits<ZipFunction>::args_tuple_plain

private

Definition at line 91 of file zip.hpp.

Constructor & Destructor Documentation

◆ ZipNode()

ZipNode	(	const ZipFunction &	zip_function,
		const ZipArgsTuple &	padding,
		const ParentDIA0 &	parent0,
		const ParentDIAs &...	parents
	)

inline

Constructor for a ZipNode.

Definition at line 98 of file zip.hpp.

References tlx::call_foreach_with_index(), DIABase::context_, ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::files_, Context::GetFile(), thrill::mem::padding, ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::padding_, ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::parent_stack_empty_, DIABase::parents(), and ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::zip_function_.

Member Function Documentation

◆ Dispose()

void Dispose ( )

inlinefinalvirtual

Virtual clear method. Triggers actual disposing in sub-classes.

Reimplemented from DIABase.

Definition at line 189 of file zip.hpp.

References ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::files_.

◆ DoScatter()

void DoScatter ( )

inlineprivate

Scatter items from DIA "Index" to other workers if necessary.

Definition at line 259 of file zip.hpp.

References DIABase::context_, Context::GetNewCatStream(), LOG, min(), Context::num_workers(), and ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::result_size_.

◆ Execute()

void Execute ( )

inlinefinalvirtual

Virtual execution method. Triggers actual computation in sub-classes.

Implements DIABase.

Definition at line 145 of file zip.hpp.

References ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::MainOp().

◆ MainOp()

void MainOp ( )

inlineprivate

Receive elements from other workers.

Definition at line 293 of file zip.hpp.

References DIABase::context_, die, FlowControlChannel::ExPrefixSumTotal(), Context::net, Context::PrintCollectiveMeanStdev(), sLOG, and thrill::common::VecToStr().

Referenced by ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::Execute().

◆ OnPreOpFile()

bool OnPreOpFile	(	const data::File &	file,
		size_t	parent_index
	)

inlinefinalvirtual

Receive a whole data::File of ValueType, but only if our stack is empty.

Reimplemented from DIABase.

Definition at line 125 of file zip.hpp.

References ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::files_, thrill::common::g_debug_push_file, LOGC, and ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::parent_stack_empty_.

◆ PushData()

void PushData ( bool consume )

inlinefinalvirtual

Virtual method for pushing data. Triggers actual pushing in sub-classes.

Implements DIABase.

Definition at line 149 of file zip.hpp.

References tlx::apply_tuple(), DIABase::context_, ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::files_, ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::ReaderNext< Reader >::HasNext(), Context::PrintCollectiveMeanStdev(), DIANode< ValueType >::PushItem(), ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::result_size_, ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::streams_, and ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::zip_function_.

◆ StartPreOp()

void StartPreOp ( size_t )

inlinefinalvirtual

Virtual method for preparing start of PushData.

Reimplemented from DIABase.

Definition at line 120 of file zip.hpp.

References ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::files_, and ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::writers_.

◆ StopPreOp()

void StopPreOp ( size_t )

inlinefinalvirtual

Virtual method for preparing end of PushData.

Reimplemented from DIABase.

Definition at line 140 of file zip.hpp.

References BlockWriter< BlockSink >::Close(), LOG, and ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::writers_.

Member Data Documentation

◆ debug

constexpr bool debug = false

staticprivate

Definition at line 79 of file zip.hpp.

◆ files_

std::vector<data::File> files_

private

Files for intermediate storage.

Definition at line 204 of file zip.hpp.

Referenced by ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::Dispose(), ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::OnPreOpFile(), ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::PushData(), ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::StartPreOp(), and ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::ZipNode().

◆ padding_

const ZipArgsTuple padding_

private

padding for shorter DIAs

Definition at line 198 of file zip.hpp.

Referenced by ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::ZipNode().

◆ parent_stack_empty_

const std::array<bool, kNumInputs> parent_stack_empty_

private

Whether the parent stack is empty.

Definition at line 201 of file zip.hpp.

Referenced by ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::OnPreOpFile(), and ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::ZipNode().

◆ result_size_

size_t result_size_

private

shortest size of Zipped inputs

Definition at line 219 of file zip.hpp.

Referenced by ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::DoScatter(), and ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::PushData().

◆ size_prefixsum_

std::array<size_t, kNumInputs> size_prefixsum_

private

exclusive prefix sum over the number of items in workers

Definition at line 216 of file zip.hpp.

◆ stats_enabled

constexpr bool stats_enabled = false

staticprivate

Set this variable to true to enable generation and output of stats.

Definition at line 82 of file zip.hpp.

◆ streams_

data::CatStreamPtr streams_[kNumInputs]

private

Array of inbound CatStreams.

Definition at line 210 of file zip.hpp.

Referenced by ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::PushData().

◆ writers_

data::File::Writer writers_[kNumInputs]

private

Writers to intermediate files.

Definition at line 207 of file zip.hpp.

Referenced by ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::RegisterParent::operator()(), ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::StartPreOp(), and ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::StopPreOp().

◆ zip_function_

ZipFunction zip_function_

private

Zip function.

Definition at line 195 of file zip.hpp.

Referenced by ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::PushData(), and ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::ZipNode().

The documentation for this class was generated from the following file:

thrill/api/zip.hpp

Detailed Description

template<typename ValueType, typename ZipFunction, bool Pad, bool UnequalCheck, bool NoRebalance, size_t kNumInputs> class thrill::api::ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >

Classes

Public Member Functions

Private Types

Private Member Functions

Private Attributes

Static Private Attributes

Additional Inherited Members

Member Typedef Documentation

◆ Super

◆ ZipArgN

◆ ZipArgsTuple

Constructor & Destructor Documentation

◆ ZipNode()

Member Function Documentation

◆ Dispose()

◆ DoScatter()

◆ Execute()

◆ MainOp()

◆ OnPreOpFile()

◆ PushData()

◆ StartPreOp()

◆ StopPreOp()

Member Data Documentation

◆ debug

◆ files_

◆ padding_

◆ parent_stack_empty_

◆ result_size_

◆ size_prefixsum_

◆ stats_enabled

◆ streams_

◆ writers_

◆ zip_function_

template<typename ValueType, typename ZipFunction, bool Pad, bool UnequalCheck, bool NoRebalance, size_t kNumInputs>
class thrill::api::ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >