Detailed Description

template<typename ValueType, typename KeyExtractor, typename ReduceFunction, typename ReduceConfig, bool VolatileKey, bool SkipPreReducePhase>
class thrill::api::ReduceToIndexNode< ValueType, KeyExtractor, ReduceFunction, ReduceConfig, VolatileKey, SkipPreReducePhase >

A DIANode which performs a ReduceToIndex operation.

ReduceToIndex groups the elements in a DIA by their key and reduces every key bucket to a single element each. The ReduceToIndexNode stores the key_extractor and the reduce_function UDFs. The chainable LOps ahead of the Reduce operation are stored in the Stack. The ReduceToIndexNode has the type ValueType, which is the result type of the reduce_function. The key type is an unsigned integer and the output DIA will have element with key K at index K.

Template Parameters

ParentType	Input type of the Reduce operation
ValueType	Output type of the Reduce operation
ParentStack	Function stack, which contains the chained lambdas between the last and this DIANode.
KeyExtractor	Type of the key_extractor function.
ReduceFunction	Type of the reduce_function

Definition at line 60 of file reduce_to_index.hpp.

Inheritance diagram for ReduceToIndexNode< ValueType, KeyExtractor, ReduceFunction, ReduceConfig, VolatileKey, SkipPreReducePhase >:

Collaboration diagram for ReduceToIndexNode< ValueType, KeyExtractor, ReduceFunction, ReduceConfig, VolatileKey, SkipPreReducePhase >:

#include <reduce_to_index.hpp>

Classes
class	Emitter
	Emitter for PostPhase to push elements to next DIA object. More...

Public Member Functions
template<typename ParentDIA >
	ReduceToIndexNode (const ParentDIA &parent, const char *label, const KeyExtractor &key_extractor, const ReduceFunction &reduce_function, size_t result_size, const ValueType &neutral_element, const ReduceConfig &config)
	Constructor for a ReduceToIndexNode. More...

void	Dispose () final
	Virtual clear method. Triggers actual disposing in sub-classes. More...

void	Execute () final
	Virtual execution method. Triggers actual computation in sub-classes. More...

DIAMemUse	PreOpMemUse () final
	Amount of RAM used by PreOp after StartPreOp() More...

void	ProcessChannel ()
	process the inbound data in the post reduce phase More...

void	PushData (bool consume) final
	Virtual method for pushing data. Triggers actual pushing in sub-classes. More...

DIAMemUse	PushDataMemUse () final
	Amount of RAM used by PushData() More...

void	StartPreOp (size_t) final
	Virtual method for preparing start of PushData. More...

void	StopPreOp (size_t) final
	Virtual method for preparing end of PushData. More...

Public Member Functions inherited from DOpNode< ValueType >
	DOpNode (Context &ctx, const char *label, const std::initializer_list< size_t > &parent_ids, const std::initializer_list< DIABasePtr > &parents)
	Constructor for a DOpNode, which sets references to the parent nodes. More...

	DOpNode (Context &ctx, const char *label, std::vector< size_t > &&parent_ids, std::vector< DIABasePtr > &&parents)
	Constructor for a DOpNode, which sets references to the parent nodes. More...

Public Member Functions inherited from DIANode< ValueType >
	DIANode (Context &ctx, const char *label, const std::initializer_list< size_t > &parent_ids, const std::initializer_list< DIABasePtr > &parents)
	Constructor for a DIANode, which sets references to the parent nodes. More...

	DIANode (Context &ctx, const char *label, std::vector< size_t > &&parent_ids, std::vector< DIABasePtr > &&parents)
	Constructor for a DIANode, which sets references to the parent nodes. More...

virtual void	AddChild (DIABase *node, const Callback &callback=Callback(), size_t parent_index=0)
	Enables children to push their "folded" function chains to their parent. More...

std::vector< DIABase * >	children () const override
	Returns the children of this DIABase. More...

void	PushFile (data::File &file, bool consume) const

void	PushItem (const ValueType &item) const
	Method for derived classes to Push a single item to all children. More...

void	RemoveAllChildren () override

void	RemoveChild (DIABase *node) override

void	RunPushData () override

Public Member Functions inherited from DIABase
	DIABase (Context &ctx, const char *label, const std::initializer_list< size_t > &parent_ids, const std::initializer_list< DIABasePtr > &parents)
	The constructor for a DIABase. More...

	DIABase (Context &ctx, const char *label, std::vector< size_t > &&parent_ids, std::vector< DIABasePtr > &&parents)
	The constructor for a DIABase. More...

	DIABase (const DIABase &)=delete
	non-copyable: delete copy-constructor More...

	DIABase (DIABase &&)=default
	move-constructor: default More...

virtual	~DIABase ()
	Virtual destructor for a DIABase. More...

virtual size_t	consume_counter () const
	Returns consume_counter_. More...

Context &	context ()
	Returns the api::Context of this DIABase. More...

virtual void	DecConsumeCounter (size_t counter)

const size_t &	dia_id () const
	return unique id of DIANode subclass as stored by StatsNode More...

virtual bool	ForwardDataOnly () const

virtual void	IncConsumeCounter (size_t counter)

const char *	label () const
	return label() of DIANode subclass as stored by StatsNode More...

mem::Manager &	mem_manager ()
	Return the Context's memory manager. More...

DIABase &	operator= (const DIABase &)=delete
	non-copyable: delete assignment operator More...

DIABase &	operator= (DIABase &&)=default
	move-assignment operator: default More...

std::vector< size_t >	parent_ids () const
	Returns the parents of this DIABase. More...

const std::vector< DIABasePtr > &	parents () const
	Returns the parents of this DIABase. More...

void	RemoveParent (DIABase *p)
	Remove a parent. More...

virtual bool	RequireParentPushData (size_t) const

void	RunScope ()

void	set_mem_limit (const DIAMemUse &mem_limit)

void	set_state (const DIAState &state)

virtual void	SetConsumeCounter (size_t counter)

DIAState	state () const

virtual bool	OnPreOpFile (const data::File &, size_t)

virtual DIAMemUse	ExecuteMemUse ()
	Amount of RAM used by Execute() More...

Public Member Functions inherited from ReferenceCounter
	ReferenceCounter () noexcept
	new objects have zero reference count More...

	ReferenceCounter (const ReferenceCounter &) noexcept
	coping still creates a new object with zero reference count More...

	~ReferenceCounter ()

bool	dec_reference () const noexcept
	Call whenever resetting (i.e. More...

void	inc_reference () const noexcept
	Call whenever setting a pointer to the object. More...

ReferenceCounter &	operator= (const ReferenceCounter &) noexcept
	assignment operator, leaves pointers unchanged More...

size_t	reference_count () const noexcept
	Return the number of references to this object (for debugging) More...

bool	unique () const noexcept
	Test if the ReferenceCounter is referenced by only one CountingPtr. More...

Private Types
using	Key = typename common::FunctionTraits< KeyExtractor >::result_type

using	Super = DOpNode< ValueType >

using	TableItem = typename std::conditional< VolatileKey, std::pair< Key, ValueType >, ValueType >::type

Private Attributes
data::CatStreamPtr	cat_stream_

data::Stream::Writers	emitters_

data::MixStreamPtr	mix_stream_

core::ReduceByIndexPostPhase< TableItem, Key, ValueType, KeyExtractor, ReduceFunction, Emitter, VolatileKey, ReduceConfig >	post_phase_

core::ReducePrePhase< TableItem, Key, ValueType, KeyExtractor, ReduceFunction, VolatileKey, data::Stream::Writer, ReduceConfig, core::ReduceByIndex< Key > >	pre_phase_

bool	reduced_ = false

size_t	result_size_

std::thread	thread_
	handle to additional thread for post phase More...

Static Private Attributes
static constexpr bool	debug = false

static constexpr bool	use_mix_stream_ = ReduceConfig::use_mix_stream_

static constexpr bool	use_post_thread_ = ReduceConfig::use_post_thread_

Additional Inherited Members
Public Types inherited from DOpNode< ValueType >
using	Super = DIANode< ValueType >

Public Types inherited from DIANode< ValueType >
using	Callback = tlx::delegate< void(const ValueType &)>

Public Types inherited from DIABase
using	DIABasePtr = tlx::CountingPtr< DIABase >

Public Attributes inherited from DIABase
common::JsonLogger	logger_

Static Public Attributes inherited from DIABase
static constexpr size_t	kNeverConsume = static_cast<size_t>(-1)
	Never full consume. More...

Protected Attributes inherited from DIANode< ValueType >
std::vector< Child >	children_
	Callback functions from the child nodes. More...

Protected Attributes inherited from DIABase
Context &	context_
	associated Context More...

const size_t	dia_id_
	DIA serial id. More...

const char *const	label_
	DOp node static label. More...

DIAState	state_ = DIAState::NEW
	State of the DIANode. State is NEW on creation. More...

std::vector< DIABasePtr >	parents_
	Parents of this DIABase. More...

DIAMemUse	mem_limit_ = 0

size_t	consume_counter_ = 1

Member Typedef Documentation

◆ Key

using Key = typename common::FunctionTraits<KeyExtractor>::result_type

private

Definition at line 67 of file reduce_to_index.hpp.

◆ Super

using Super = DOpNode<ValueType>

private

Definition at line 64 of file reduce_to_index.hpp.

◆ TableItem

using TableItem = typename std::conditional< VolatileKey, std::pair<Key, ValueType>, ValueType>::type

private

Definition at line 71 of file reduce_to_index.hpp.

Constructor & Destructor Documentation

◆ ReduceToIndexNode()

ReduceToIndexNode	(	const ParentDIA &	parent,
		const char *	label,
		const KeyExtractor &	key_extractor,
		const ReduceFunction &	reduce_function,
		size_t	result_size,
		const ValueType &	neutral_element,
		const ReduceConfig &	config
	)

inline

Constructor for a ReduceToIndexNode.

Sets the parent, stack, key_extractor and reduce_function.

Definition at line 98 of file reduce_to_index.hpp.

References DIANode< ValueType >::AddChild().

Member Function Documentation

◆ Dispose()

void Dispose ( )

inlinefinalvirtual

Virtual clear method. Triggers actual disposing in sub-classes.

Reimplemented from DIABase.

Definition at line 228 of file reduce_to_index.hpp.

◆ Execute()

void Execute ( )

inlinefinalvirtual

Virtual execution method. Triggers actual computation in sub-classes.

Implements DIABase.

Definition at line 185 of file reduce_to_index.hpp.

◆ PreOpMemUse()

DIAMemUse PreOpMemUse ( )

inlinefinalvirtual

Amount of RAM used by PreOp after StartPreOp()

Reimplemented from DIABase.

Definition at line 137 of file reduce_to_index.hpp.

References DIAMemUse::Max().

◆ ProcessChannel()

void ProcessChannel ( )

inline

process the inbound data in the post reduce phase

Definition at line 207 of file reduce_to_index.hpp.

References sLOG.

◆ PushData()

void PushData ( bool consume )

inlinefinalvirtual

Virtual method for pushing data. Triggers actual pushing in sub-classes.

Implements DIABase.

Definition at line 191 of file reduce_to_index.hpp.

References DIABase::mem_limit_.

◆ PushDataMemUse()

DIAMemUse PushDataMemUse ( )

inlinefinalvirtual

Amount of RAM used by PushData()

Reimplemented from DIABase.

Definition at line 187 of file reduce_to_index.hpp.

References DIAMemUse::Max().

◆ StartPreOp()

void StartPreOp ( size_t )

inlinefinalvirtual

Virtual method for preparing start of PushData.

Reimplemented from DIABase.

Definition at line 143 of file reduce_to_index.hpp.

References thrill::common::CreateThread(), and DIABase::mem_limit_.

◆ StopPreOp()

void StopPreOp ( size_t )

inlinefinalvirtual

Virtual method for preparing end of PushData.

Reimplemented from DIABase.

Definition at line 171 of file reduce_to_index.hpp.

References LOG.

Member Data Documentation

◆ cat_stream_

data::CatStreamPtr cat_stream_

private

Definition at line 236 of file reduce_to_index.hpp.

◆ debug

constexpr bool debug = false

staticprivate

Definition at line 62 of file reduce_to_index.hpp.

◆ emitters_

data::Stream::Writers emitters_

private

Definition at line 238 of file reduce_to_index.hpp.

◆ mix_stream_

data::MixStreamPtr mix_stream_

private

Definition at line 235 of file reduce_to_index.hpp.

◆ post_phase_

core::ReduceByIndexPostPhase< TableItem, Key, ValueType, KeyExtractor, ReduceFunction, Emitter, VolatileKey, ReduceConfig> post_phase_

private

Definition at line 252 of file reduce_to_index.hpp.

◆ pre_phase_

core::ReducePrePhase< TableItem, Key, ValueType, KeyExtractor, ReduceFunction, VolatileKey, data::Stream::Writer, ReduceConfig, core::ReduceByIndex<Key> > pre_phase_

private

Definition at line 248 of file reduce_to_index.hpp.

◆ reduced_

bool reduced_ = false

private

Definition at line 254 of file reduce_to_index.hpp.

◆ result_size_

size_t result_size_

private

Definition at line 240 of file reduce_to_index.hpp.

◆ thread_

std::thread thread_

private

handle to additional thread for post phase

Definition at line 243 of file reduce_to_index.hpp.

◆ use_mix_stream_

constexpr bool use_mix_stream_ = ReduceConfig::use_mix_stream_

staticprivate

Definition at line 76 of file reduce_to_index.hpp.

◆ use_post_thread_

constexpr bool use_post_thread_ = ReduceConfig::use_post_thread_

staticprivate

Definition at line 77 of file reduce_to_index.hpp.

The documentation for this class was generated from the following file:

thrill/api/reduce_to_index.hpp

Detailed Description

template<typename ValueType, typename KeyExtractor, typename ReduceFunction, typename ReduceConfig, bool VolatileKey, bool SkipPreReducePhase> class thrill::api::ReduceToIndexNode< ValueType, KeyExtractor, ReduceFunction, ReduceConfig, VolatileKey, SkipPreReducePhase >

Classes

Public Member Functions

Private Types

Private Attributes

Static Private Attributes

Additional Inherited Members

Member Typedef Documentation

◆ Key

◆ Super

◆ TableItem

Constructor & Destructor Documentation

◆ ReduceToIndexNode()

Member Function Documentation

◆ Dispose()

◆ Execute()

◆ PreOpMemUse()

◆ ProcessChannel()

◆ PushData()

◆ PushDataMemUse()

◆ StartPreOp()

◆ StopPreOp()

Member Data Documentation

◆ cat_stream_

◆ debug

◆ emitters_

◆ mix_stream_

◆ post_phase_

◆ pre_phase_

◆ reduced_

◆ result_size_

◆ thread_

◆ use_mix_stream_

◆ use_post_thread_

template<typename ValueType, typename KeyExtractor, typename ReduceFunction, typename ReduceConfig, bool VolatileKey, bool SkipPreReducePhase>
class thrill::api::ReduceToIndexNode< ValueType, KeyExtractor, ReduceFunction, ReduceConfig, VolatileKey, SkipPreReducePhase >