Detailed Description

template<typename ValueType, typename KeyExtractor, typename GroupFunction>
class thrill::api::GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >

Definition at line 43 of file group_by_iterator.hpp.

Inheritance diagram for GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >:

Collaboration diagram for GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >:

#include <group_by_iterator.hpp>

Classes
class	ValueComparator

Public Member Functions
template<typename ParentDIA >
	GroupToIndexNode (const ParentDIA &parent, const KeyExtractor &key_extractor, const GroupFunction &groupby_function, size_t result_size, const ValueOut &neutral_element)
	Constructor for a GroupToIndexNode. More...

void	Dispose () override
	Virtual clear method. Triggers actual disposing in sub-classes. More...

void	Execute () override
	Virtual execution method. Triggers actual computation in sub-classes. More...

void	PreOp (const ValueIn &v)
	Send all elements to their designated PEs. More...

void	PushData (bool consume) final
	Virtual method for pushing data. Triggers actual pushing in sub-classes. More...

void	StopPreOp (size_t) final
	Virtual method for preparing end of PushData. More...

Public Member Functions inherited from DOpNode< ValueType >
	DOpNode (Context &ctx, const char *label, const std::initializer_list< size_t > &parent_ids, const std::initializer_list< DIABasePtr > &parents)
	Constructor for a DOpNode, which sets references to the parent nodes. More...

	DOpNode (Context &ctx, const char *label, std::vector< size_t > &&parent_ids, std::vector< DIABasePtr > &&parents)
	Constructor for a DOpNode, which sets references to the parent nodes. More...

Public Member Functions inherited from DIANode< ValueType >
	DIANode (Context &ctx, const char *label, const std::initializer_list< size_t > &parent_ids, const std::initializer_list< DIABasePtr > &parents)
	Constructor for a DIANode, which sets references to the parent nodes. More...

	DIANode (Context &ctx, const char *label, std::vector< size_t > &&parent_ids, std::vector< DIABasePtr > &&parents)
	Constructor for a DIANode, which sets references to the parent nodes. More...

virtual void	AddChild (DIABase *node, const Callback &callback=Callback(), size_t parent_index=0)
	Enables children to push their "folded" function chains to their parent. More...

std::vector< DIABase * >	children () const override
	Returns the children of this DIABase. More...

void	PushFile (data::File &file, bool consume) const

void	PushItem (const ValueType &item) const
	Method for derived classes to Push a single item to all children. More...

void	RemoveAllChildren () override

void	RemoveChild (DIABase *node) override

void	RunPushData () override

Public Member Functions inherited from DIABase
	DIABase (Context &ctx, const char *label, const std::initializer_list< size_t > &parent_ids, const std::initializer_list< DIABasePtr > &parents)
	The constructor for a DIABase. More...

	DIABase (Context &ctx, const char *label, std::vector< size_t > &&parent_ids, std::vector< DIABasePtr > &&parents)
	The constructor for a DIABase. More...

	DIABase (const DIABase &)=delete
	non-copyable: delete copy-constructor More...

	DIABase (DIABase &&)=default
	move-constructor: default More...

virtual	~DIABase ()
	Virtual destructor for a DIABase. More...

virtual size_t	consume_counter () const
	Returns consume_counter_. More...

Context &	context ()
	Returns the api::Context of this DIABase. More...

virtual void	DecConsumeCounter (size_t counter)

const size_t &	dia_id () const
	return unique id of DIANode subclass as stored by StatsNode More...

virtual bool	ForwardDataOnly () const

virtual void	IncConsumeCounter (size_t counter)

const char *	label () const
	return label() of DIANode subclass as stored by StatsNode More...

mem::Manager &	mem_manager ()
	Return the Context's memory manager. More...

DIABase &	operator= (const DIABase &)=delete
	non-copyable: delete assignment operator More...

DIABase &	operator= (DIABase &&)=default
	move-assignment operator: default More...

std::vector< size_t >	parent_ids () const
	Returns the parents of this DIABase. More...

const std::vector< DIABasePtr > &	parents () const
	Returns the parents of this DIABase. More...

void	RemoveParent (DIABase *p)
	Remove a parent. More...

virtual bool	RequireParentPushData (size_t) const

void	RunScope ()

void	set_mem_limit (const DIAMemUse &mem_limit)

void	set_state (const DIAState &state)

virtual void	SetConsumeCounter (size_t counter)

DIAState	state () const

virtual DIAMemUse	PreOpMemUse ()
	Amount of RAM used by PreOp after StartPreOp() More...

virtual void	StartPreOp (size_t)
	Virtual method for preparing start of PushData. More...

virtual bool	OnPreOpFile (const data::File &, size_t)

virtual DIAMemUse	ExecuteMemUse ()
	Amount of RAM used by Execute() More...

virtual DIAMemUse	PushDataMemUse ()
	Amount of RAM used by PushData() More...

Public Member Functions inherited from ReferenceCounter
	ReferenceCounter () noexcept
	new objects have zero reference count More...

	ReferenceCounter (const ReferenceCounter &) noexcept
	coping still creates a new object with zero reference count More...

	~ReferenceCounter ()

bool	dec_reference () const noexcept
	Call whenever resetting (i.e. More...

void	inc_reference () const noexcept
	Call whenever setting a pointer to the object. More...

ReferenceCounter &	operator= (const ReferenceCounter &) noexcept
	assignment operator, leaves pointers unchanged More...

size_t	reference_count () const noexcept
	Return the number of references to this object (for debugging) More...

bool	unique () const noexcept
	Test if the ReferenceCounter is referenced by only one CountingPtr. More...

Private Types
using	Key = typename common::FunctionTraits< KeyExtractor >::result_type

using	Super = DOpNode< ValueType >

using	ValueIn = typename common::FunctionTraits< KeyExtractor >::template arg_plain< 0 >

using	ValueOut = ValueType

Private Member Functions
void	FlushVectorToFile (std::vector< ValueIn > &v)
	Sort and store elements in a file. More...

void	MainOp ()
	Receive elements from other workers. More...

void	RunUserFunc (data::File &f, bool consume)

Private Attributes
data::CatStream::Writers	emitters_ { stream_->GetWriters() }

std::vector< data::File >	files_

GroupFunction	groupby_function_

KeyExtractor	key_extractor_

const common::Range	key_range_

ValueOut	neutral_element_

const size_t	result_size_

data::CatStreamPtr	stream_ { context_.GetNewCatStream(this) }

size_t	totalsize_ = 0

Static Private Attributes
static constexpr bool	debug = false

Additional Inherited Members
Public Types inherited from DOpNode< ValueType >
using	Super = DIANode< ValueType >

Public Types inherited from DIANode< ValueType >
using	Callback = tlx::delegate< void(const ValueType &)>

Public Types inherited from DIABase
using	DIABasePtr = tlx::CountingPtr< DIABase >

Public Attributes inherited from DIABase
common::JsonLogger	logger_

Static Public Attributes inherited from DIABase
static constexpr size_t	kNeverConsume = static_cast<size_t>(-1)
	Never full consume. More...

Protected Attributes inherited from DIANode< ValueType >
std::vector< Child >	children_
	Callback functions from the child nodes. More...

Protected Attributes inherited from DIABase
Context &	context_
	associated Context More...

const size_t	dia_id_
	DIA serial id. More...

const char *const	label_
	DOp node static label. More...

DIAState	state_ = DIAState::NEW
	State of the DIANode. State is NEW on creation. More...

std::vector< DIABasePtr >	parents_
	Parents of this DIABase. More...

DIAMemUse	mem_limit_ = 0

size_t	consume_counter_ = 1

Member Typedef Documentation

◆ Key

using Key = typename common::FunctionTraits<KeyExtractor>::result_type

private

Definition at line 49 of file group_to_index.hpp.

◆ Super

using Super = DOpNode<ValueType>

private

Definition at line 46 of file group_to_index.hpp.

◆ ValueIn

using ValueIn = typename common::FunctionTraits<KeyExtractor>::template arg_plain<0>

private

Definition at line 52 of file group_to_index.hpp.

◆ ValueOut

using ValueOut = ValueType

private

Definition at line 50 of file group_to_index.hpp.

Constructor & Destructor Documentation

◆ GroupToIndexNode()

GroupToIndexNode	(	const ParentDIA &	parent,
		const KeyExtractor &	key_extractor,
		const GroupFunction &	groupby_function,
		size_t	result_size,
		const ValueOut &	neutral_element
	)

inline

Constructor for a GroupToIndexNode.

Sets the DataManager, parent, stack, key_extractor and reduce_function.

Definition at line 73 of file group_to_index.hpp.

References thrill::common::CalculateLocalRange(), DIABase::context_, GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::groupby_function_, GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::key_extractor_, GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::key_range_, Context::my_rank(), GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::neutral_element_, Context::num_workers(), GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::PreOp(), and GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::result_size_.

Member Function Documentation

◆ Dispose()

void Dispose ( )

inlineoverridevirtual

Virtual clear method. Triggers actual disposing in sub-classes.

Reimplemented from DIABase.

Definition at line 172 of file group_to_index.hpp.

◆ Execute()

void Execute ( )

inlineoverridevirtual

Virtual execution method. Triggers actual computation in sub-classes.

Implements DIABase.

Definition at line 112 of file group_to_index.hpp.

References GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::MainOp().

◆ FlushVectorToFile()

void FlushVectorToFile ( std::vector< ValueIn > & v )

inlineprivate

Sort and store elements in a file.

Definition at line 218 of file group_to_index.hpp.

References BlockWriter< BlockSink >::Close(), DIABase::context_, Context::GetFile(), File::GetWriter(), BlockWriter< BlockSink >::Put(), and GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::ValueComparator::ValueComparator().

Referenced by GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::MainOp().

◆ MainOp()

void MainOp ( )

inlineprivate

Receive elements from other workers.

Definition at line 234 of file group_to_index.hpp.

References GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::FlushVectorToFile(), LOG, thrill::mem::memory_exceeded, CountingPtr< Type, Deleter >::reset(), GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::stream_, and tlx::vector_free().

Referenced by GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::Execute().

◆ PreOp()

void PreOp ( const ValueIn & v )

inline

Send all elements to their designated PEs.

Definition at line 98 of file group_to_index.hpp.

References thrill::common::CalculatePartition(), DIABase::context_, GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::emitters_, GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::key_extractor_, Context::num_workers(), and GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::result_size_.

Referenced by GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::GroupToIndexNode().

◆ PushData()

void PushData ( bool consume )

inlinefinalvirtual

Virtual method for pushing data. Triggers actual pushing in sub-classes.

Implements DIABase.

Definition at line 116 of file group_to_index.hpp.

References Range::begin, Range::end, GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::files_, GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::groupby_function_, GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::key_extractor_, GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::key_range_, GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::neutral_element_, DIANode< ValueType >::PushItem(), GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::RunUserFunc(), sLOG, and GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::ValueComparator::ValueComparator().

◆ RunUserFunc()

void RunUserFunc	(	data::File &	f,
		bool	consume
	)

inlineprivate

Definition at line 186 of file group_to_index.hpp.

References Range::begin, Range::end, File::GetReader(), GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::groupby_function_, GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::key_extractor_, and DIANode< ValueType >::PushItem().

Referenced by GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::PushData().

◆ StopPreOp()

void StopPreOp ( size_t )

inlinefinalvirtual

Virtual method for preparing end of PushData.

Reimplemented from DIABase.

Definition at line 107 of file group_to_index.hpp.

References StreamData::Writers::Close(), and GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::emitters_.

Member Data Documentation

◆ debug

constexpr bool debug = false

staticprivate

Definition at line 44 of file group_to_index.hpp.

◆ emitters_

data::CatStream::Writers emitters_ { stream_->GetWriters() }

private

Definition at line 183 of file group_to_index.hpp.

Referenced by GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::PreOp(), and GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::StopPreOp().

◆ files_

std::vector<data::File> files_

private

Definition at line 184 of file group_to_index.hpp.

Referenced by GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::PushData().

◆ groupby_function_

GroupFunction groupby_function_

private

Definition at line 176 of file group_to_index.hpp.

Referenced by GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::GroupToIndexNode(), GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::PushData(), and GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::RunUserFunc().

◆ key_extractor_

KeyExtractor key_extractor_

private

Definition at line 175 of file group_to_index.hpp.

Referenced by GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::GroupToIndexNode(), GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::ValueComparator::operator()(), GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::PreOp(), GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::PushData(), and GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::RunUserFunc().

◆ key_range_

const common::Range key_range_

private

Definition at line 178 of file group_to_index.hpp.

Referenced by GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::GroupToIndexNode(), and GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::PushData().

◆ neutral_element_

ValueOut neutral_element_

private

Definition at line 179 of file group_to_index.hpp.

Referenced by GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::GroupToIndexNode(), and GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::PushData().

◆ result_size_

const size_t result_size_

private

Definition at line 177 of file group_to_index.hpp.

Referenced by GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::GroupToIndexNode(), and GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::PreOp().

◆ stream_

data::CatStreamPtr stream_ { context_.GetNewCatStream(this) }

private

Definition at line 182 of file group_to_index.hpp.

Referenced by GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::MainOp().

◆ totalsize_

size_t totalsize_ = 0

private

Definition at line 180 of file group_to_index.hpp.

The documentation for this class was generated from the following files:

thrill/api/group_by_iterator.hpp
thrill/api/group_to_index.hpp

Detailed Description

template<typename ValueType, typename KeyExtractor, typename GroupFunction> class thrill::api::GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >

Classes

Public Member Functions

Private Types

Private Member Functions

Private Attributes

Static Private Attributes

Additional Inherited Members

Member Typedef Documentation

◆ Key

◆ Super

◆ ValueIn

◆ ValueOut

Constructor & Destructor Documentation

◆ GroupToIndexNode()

Member Function Documentation

◆ Dispose()

◆ Execute()

◆ FlushVectorToFile()

◆ MainOp()

◆ PreOp()

◆ PushData()

◆ RunUserFunc()

◆ StopPreOp()

Member Data Documentation

◆ debug

◆ emitters_

◆ files_

◆ groupby_function_

◆ key_extractor_

◆ key_range_

◆ neutral_element_

◆ result_size_

◆ stream_

◆ totalsize_

template<typename ValueType, typename KeyExtractor, typename GroupFunction>
class thrill::api::GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >