Detailed Description

template<typename ValueType>
class thrill::api::UnionNode< ValueType >

Implements a Union operation by hooking any number of input DIANodes and forwarding the output immediately to all children.

The problem with Union is that children can be added to the node after creation (as with all other nodes). This however requires the UnionNode to remember which of its children has already got which input's items. This is recorded in each UnionChild's pushed_inputs array.

For example this occurs in the following DIA graph:

[Gen.1] ---—v [Union.3] –> [Sort.4] ----—> [Size.6] +—^ +----------------------------—> [Size.7] [Gen.2] –+ +---------------------—> [Size.5]

Size.5 triggers Execute and PushData such that Sort.4's PreOp gets all data from Gen.2, but not from Gen.1. Then, running Size.6 requires Union.3 to get data from Gen.1 and NOT run Gen.2 again.

Another situation then occur, when [Size.7] is added later.

Definition at line 53 of file union.hpp.

Inheritance diagram for UnionNode< ValueType >:

Collaboration diagram for UnionNode< ValueType >:

#include <union.hpp>

Classes
class	RegisterParent

struct	UnionChild

Public Types
using	Callback = typename Super::Callback

enum	ChildStatus { NEW, PUSHING, DONE }

using	Super = DIANode< ValueType >

Public Types inherited from DIANode< ValueType >
using	Callback = tlx::delegate< void(const ValueType &)>

Public Types inherited from DIABase
using	DIABasePtr = tlx::CountingPtr< DIABase >

Public Member Functions
template<typename ParentDIA0 , typename... ParentDIAs>
	UnionNode (const ParentDIA0 &parent0, const ParentDIAs &... parents)

template<typename ParentDIA >
	UnionNode (const std::vector< ParentDIA > &parents)

template<typename ParentDIA >
	UnionNode (const std::initializer_list< ParentDIA > &parents)

void	AddChild (DIABase *node, const Callback &callback, size_t parent_index=0) final
	Enables children to push their "folded" function chains to their parent. More...

std::vector< DIABase * >	children () const final
	Returns the children of this DIABase. More...

size_t	consume_counter () const final
	Returns consume_counter_. More...

void	DecConsumeCounter (size_t consume) final

void	Execute () final
	Virtual execution method. Triggers actual computation in sub-classes. More...

bool	ForwardDataOnly () const final
	A UnionNode cannot be executed, it never contains any data. More...

void	IncConsumeCounter (size_t consume) final

void	PushData (bool) final
	Virtual method for pushing data. Triggers actual pushing in sub-classes. More...

void	PushItem (const ValueType &item, size_t parent_index) const
	Method for derived classes to Push a single item to all children. More...

void	RemoveAllChildren () final

void	RemoveChild (DIABase *node) final

bool	RequireParentPushData (size_t parent_index) const final
	Check whether we need PushData() from the specific parent. More...

void	RunPushData () final

void	SetConsumeCounter (size_t consume) final

void	StartPreOp (size_t parent_index) final
	Virtual method for preparing start of PushData. More...

void	StopPreOp (size_t parent_index) final
	Virtual method for preparing end of PushData. More...

Public Member Functions inherited from DIANode< ValueType >
	DIANode (Context &ctx, const char *label, const std::initializer_list< size_t > &parent_ids, const std::initializer_list< DIABasePtr > &parents)
	Constructor for a DIANode, which sets references to the parent nodes. More...

	DIANode (Context &ctx, const char *label, std::vector< size_t > &&parent_ids, std::vector< DIABasePtr > &&parents)
	Constructor for a DIANode, which sets references to the parent nodes. More...

void	PushFile (data::File &file, bool consume) const

void	PushItem (const ValueType &item) const
	Method for derived classes to Push a single item to all children. More...

Public Member Functions inherited from DIABase
	DIABase (Context &ctx, const char *label, const std::initializer_list< size_t > &parent_ids, const std::initializer_list< DIABasePtr > &parents)
	The constructor for a DIABase. More...

	DIABase (Context &ctx, const char *label, std::vector< size_t > &&parent_ids, std::vector< DIABasePtr > &&parents)
	The constructor for a DIABase. More...

	DIABase (const DIABase &)=delete
	non-copyable: delete copy-constructor More...

	DIABase (DIABase &&)=default
	move-constructor: default More...

virtual	~DIABase ()
	Virtual destructor for a DIABase. More...

Context &	context ()
	Returns the api::Context of this DIABase. More...

const size_t &	dia_id () const
	return unique id of DIANode subclass as stored by StatsNode More...

const char *	label () const
	return label() of DIANode subclass as stored by StatsNode More...

mem::Manager &	mem_manager ()
	Return the Context's memory manager. More...

DIABase &	operator= (const DIABase &)=delete
	non-copyable: delete assignment operator More...

DIABase &	operator= (DIABase &&)=default
	move-assignment operator: default More...

std::vector< size_t >	parent_ids () const
	Returns the parents of this DIABase. More...

const std::vector< DIABasePtr > &	parents () const
	Returns the parents of this DIABase. More...

void	RemoveParent (DIABase *p)
	Remove a parent. More...

void	RunScope ()

void	set_mem_limit (const DIAMemUse &mem_limit)

void	set_state (const DIAState &state)

DIAState	state () const

virtual DIAMemUse	PreOpMemUse ()
	Amount of RAM used by PreOp after StartPreOp() More...

virtual bool	OnPreOpFile (const data::File &, size_t)

virtual DIAMemUse	ExecuteMemUse ()
	Amount of RAM used by Execute() More...

virtual DIAMemUse	PushDataMemUse ()
	Amount of RAM used by PushData() More...

virtual void	Dispose ()
	Virtual clear method. Triggers actual disposing in sub-classes. More...

Public Member Functions inherited from ReferenceCounter
	ReferenceCounter () noexcept
	new objects have zero reference count More...

	ReferenceCounter (const ReferenceCounter &) noexcept
	coping still creates a new object with zero reference count More...

	~ReferenceCounter ()

bool	dec_reference () const noexcept
	Call whenever resetting (i.e. More...

void	inc_reference () const noexcept
	Call whenever setting a pointer to the object. More...

ReferenceCounter &	operator= (const ReferenceCounter &) noexcept
	assignment operator, leaves pointers unchanged More...

size_t	reference_count () const noexcept
	Return the number of references to this object (for debugging) More...

bool	unique () const noexcept
	Test if the ReferenceCounter is referenced by only one CountingPtr. More...

Private Attributes
std::vector< UnionChild >	children_
	Callback functions from the child nodes. More...

size_t	num_inputs_

Static Private Attributes
static constexpr bool	debug = false

Additional Inherited Members
Public Attributes inherited from DIABase
common::JsonLogger	logger_

Static Public Attributes inherited from DIABase
static constexpr size_t	kNeverConsume = static_cast<size_t>(-1)
	Never full consume. More...

Protected Attributes inherited from DIANode< ValueType >
std::vector< Child >	children_
	Callback functions from the child nodes. More...

Protected Attributes inherited from DIABase
Context &	context_
	associated Context More...

const size_t	dia_id_
	DIA serial id. More...

const char *const	label_
	DOp node static label. More...

DIAState	state_ = DIAState::NEW
	State of the DIANode. State is NEW on creation. More...

std::vector< DIABasePtr >	parents_
	Parents of this DIABase. More...

DIAMemUse	mem_limit_ = 0

size_t	consume_counter_ = 1

Member Typedef Documentation

◆ Callback

using Callback = typename Super::Callback

Definition at line 60 of file union.hpp.

◆ Super

using Super = DIANode<ValueType>

Definition at line 58 of file union.hpp.

Member Enumeration Documentation

◆ ChildStatus

enum ChildStatus

strong

Enumerator
NEW
PUSHING
DONE

Definition at line 62 of file union.hpp.

Constructor & Destructor Documentation

◆ UnionNode() [1/3]

UnionNode	(	const ParentDIA0 &	parent0,
		const ParentDIAs &...	parents
	)

inlineexplicit

Constructor for variant with variadic parent parameter pack, which each parent may have a different FunctionStack.

Definition at line 89 of file union.hpp.

References tlx::call_foreach_with_index(), UnionNode< ValueType >::num_inputs_, and DIABase::parents().

◆ UnionNode() [2/3]

UnionNode ( const std::vector< ParentDIA > & parents )

inlineexplicit

Constructor for variant with a std::vector of parents all with the same (usually empty) FunctionStack.

Definition at line 102 of file union.hpp.

References thrill::common::MapVector(), UnionNode< ValueType >::num_inputs_, DIABase::parents(), and UnionNode< ValueType >::PushItem().

◆ UnionNode() [3/3]

UnionNode ( const std::initializer_list< ParentDIA > & parents )

inlineexplicit

Constructor for variant with a std::initializer_list of parents all with the same (usually empty) FunctionStack.

Definition at line 128 of file union.hpp.

Member Function Documentation

◆ AddChild()

void AddChild	(	DIABase *	node,
		const Callback &	callback,
		size_t	parent_index = `0`
	)

inlinefinalvirtual

Enables children to push their "folded" function chains to their parent.

This way the parent can push all its result elements to each of the children. This procedure enables the minimization of IO-accesses.

Reimplemented from DIANode< ValueType >.

Definition at line 162 of file union.hpp.

References UnionNode< ValueType >::children_, UnionNode< ValueType >::NEW, and UnionNode< ValueType >::num_inputs_.

◆ children()

std::vector<DIABase*> children ( ) const

inlinefinalvirtual

Returns the children of this DIABase.

Reimplemented from DIANode< ValueType >.

Definition at line 206 of file union.hpp.

References UnionNode< ValueType >::children_.

◆ consume_counter()

size_t consume_counter ( ) const

inlinefinalvirtual

Returns consume_counter_.

Reimplemented from DIABase.

Definition at line 268 of file union.hpp.

References DIABase::kNeverConsume, min(), and DIABase::parents_.

◆ DecConsumeCounter()

void DecConsumeCounter ( size_t counter )

inlinefinalvirtual

Virtual SetConsume flag which is called by the user via .Keep() or .Consume() to set consumption.

Reimplemented from DIABase.

Definition at line 284 of file union.hpp.

References DIABase::parents_.

◆ Execute()

void Execute ( )

inlinefinalvirtual

Virtual execution method. Triggers actual computation in sub-classes.

Implements DIABase.

Definition at line 225 of file union.hpp.

◆ ForwardDataOnly()

bool ForwardDataOnly ( ) const

inlinefinalvirtual

A UnionNode cannot be executed, it never contains any data.

Reimplemented from DIABase.

Definition at line 215 of file union.hpp.

◆ IncConsumeCounter()

void IncConsumeCounter ( size_t counter )

inlinefinalvirtual

Virtual SetConsume flag which is called by the user via .Keep() or .Consume() to set consumption.

Reimplemented from DIABase.

Definition at line 277 of file union.hpp.

References DIABase::parents_.

◆ PushData()

void PushData ( bool consume )

inlinefinalvirtual

Virtual method for pushing data. Triggers actual pushing in sub-classes.

Implements DIABase.

Definition at line 266 of file union.hpp.

◆ PushItem()

void PushItem	(	const ValueType &	item,
		size_t	parent_index
	)		const

inline

Method for derived classes to Push a single item to all children.

Definition at line 241 of file union.hpp.

References UnionNode< ValueType >::children_.

Referenced by UnionNode< ValueType >::RegisterParent::operator()(), and UnionNode< ValueType >::UnionNode().

◆ RemoveAllChildren()

void RemoveAllChildren ( )

inlinefinalvirtual

Virtual method for removing all childs. Triggers actual removing in sub-classes.

Reimplemented from DIANode< ValueType >.

Definition at line 182 of file union.hpp.

References UnionNode< ValueType >::children_, and UnionNode< ValueType >::DONE.

◆ RemoveChild()

void RemoveChild ( DIABase * node )

inlinefinalvirtual

Remove a child from the vector of children. This method is called by the destructor of children.

Reimplemented from DIANode< ValueType >.

Definition at line 172 of file union.hpp.

References UnionNode< ValueType >::children_.

◆ RequireParentPushData()

bool RequireParentPushData ( size_t parent_index ) const

inlinefinalvirtual

Check whether we need PushData() from the specific parent.

Reimplemented from DIABase.

Definition at line 218 of file union.hpp.

References UnionNode< ValueType >::children_.

◆ RunPushData()

void RunPushData ( )

inlinefinalvirtual

Performing push operation. Notifies children and calls actual push method. Then cleans up the DIA graph by freeing parent references of children.

Reimplemented from DIANode< ValueType >.

Definition at line 264 of file union.hpp.

◆ SetConsumeCounter()

void SetConsumeCounter ( size_t counter )

inlinefinalvirtual

Virtual SetConsume flag which is called by the user via .Keep() or .Consume() to set consumption.

Reimplemented from DIABase.

Definition at line 291 of file union.hpp.

References DIABase::parents_.

◆ StartPreOp()

void StartPreOp ( size_t )

inlinefinalvirtual

Virtual method for preparing start of PushData.

Reimplemented from DIABase.

Definition at line 227 of file union.hpp.

References UnionNode< ValueType >::children_, LOG, LOG0, UnionNode< ValueType >::NEW, and UnionNode< ValueType >::PUSHING.

◆ StopPreOp()

void StopPreOp ( size_t )

inlinefinalvirtual

Virtual method for preparing end of PushData.

Reimplemented from DIABase.

Definition at line 248 of file union.hpp.

References UnionNode< ValueType >::children_, UnionNode< ValueType >::DONE, LOG, LOG0, and UnionNode< ValueType >::PUSHING.

Member Data Documentation

◆ children_

std::vector<UnionChild> children_

private

Callback functions from the child nodes.

Definition at line 302 of file union.hpp.

Referenced by UnionNode< ValueType >::AddChild(), UnionNode< ValueType >::children(), UnionNode< ValueType >::PushItem(), UnionNode< ValueType >::RemoveAllChildren(), UnionNode< ValueType >::RemoveChild(), UnionNode< ValueType >::RequireParentPushData(), UnionNode< ValueType >::StartPreOp(), and UnionNode< ValueType >::StopPreOp().

◆ debug

constexpr bool debug = false

staticprivate

Definition at line 55 of file union.hpp.

◆ num_inputs_

size_t num_inputs_

private

Definition at line 299 of file union.hpp.

Referenced by UnionNode< ValueType >::AddChild(), and UnionNode< ValueType >::UnionNode().

The documentation for this class was generated from the following file:

thrill/api/union.hpp

Detailed Description

template<typename ValueType> class thrill::api::UnionNode< ValueType >

Classes

Public Types

Public Member Functions

Private Attributes

Static Private Attributes

Additional Inherited Members

Member Typedef Documentation

◆ Callback

◆ Super

Member Enumeration Documentation

◆ ChildStatus

Constructor & Destructor Documentation

◆ UnionNode() [1/3]

◆ UnionNode() [2/3]

◆ UnionNode() [3/3]

Member Function Documentation

◆ AddChild()

◆ children()

◆ consume_counter()

◆ DecConsumeCounter()

◆ Execute()

◆ ForwardDataOnly()

◆ IncConsumeCounter()

◆ PushData()

◆ PushItem()

◆ RemoveAllChildren()

◆ RemoveChild()

◆ RequireParentPushData()

◆ RunPushData()

◆ SetConsumeCounter()

◆ StartPreOp()

◆ StopPreOp()

Member Data Documentation

◆ children_

◆ debug

◆ num_inputs_

template<typename ValueType>
class thrill::api::UnionNode< ValueType >