Thrill  0.1
GroupByNode< ValueType, KeyExtractor, GroupFunction, HashFunction, UseLocationDetection > Class Template Reference

Detailed Description

template<typename ValueType, typename KeyExtractor, typename GroupFunction, typename HashFunction, bool UseLocationDetection>
class thrill::api::GroupByNode< ValueType, KeyExtractor, GroupFunction, HashFunction, UseLocationDetection >

Definition at line 39 of file group_by_iterator.hpp.

+ Inheritance diagram for GroupByNode< ValueType, KeyExtractor, GroupFunction, HashFunction, UseLocationDetection >:
+ Collaboration diagram for GroupByNode< ValueType, KeyExtractor, GroupFunction, HashFunction, UseLocationDetection >:

#include <group_by_iterator.hpp>

Classes

class  HashCount
 
struct  ValueComparator
 

Public Member Functions

template<typename ParentDIA >
 GroupByNode (const ParentDIA &parent, const KeyExtractor &key_extractor, const GroupFunction &groupby_function, const HashFunction &hash_function=HashFunction())
 Constructor for a GroupByNode. More...
 
void Dispose () override
 Virtual clear method. Triggers actual disposing in sub-classes. More...
 
void Execute () override
 Virtual execution method. Triggers actual computation in sub-classes. More...
 
DIAMemUse ExecuteMemUse () final
 Amount of RAM used by Execute() More...
 
void PreOp (const ValueIn &v)
 Send all elements to their designated PEs. More...
 
DIAMemUse PreOpMemUse () final
 Amount of RAM used by PreOp after StartPreOp() More...
 
void PushData (bool consume) final
 Virtual method for pushing data. Triggers actual pushing in sub-classes. More...
 
DIAMemUse PushDataMemUse () final
 Amount of RAM used by PushData() More...
 
void StartPreOp (size_t) final
 Virtual method for preparing start of PushData. More...
 
void StopPreOp (size_t) final
 Virtual method for preparing end of PushData. More...
 
- Public Member Functions inherited from DOpNode< ValueType >
 DOpNode (Context &ctx, const char *label, const std::initializer_list< size_t > &parent_ids, const std::initializer_list< DIABasePtr > &parents)
 Constructor for a DOpNode, which sets references to the parent nodes. More...
 
 DOpNode (Context &ctx, const char *label, std::vector< size_t > &&parent_ids, std::vector< DIABasePtr > &&parents)
 Constructor for a DOpNode, which sets references to the parent nodes. More...
 
- Public Member Functions inherited from DIANode< ValueType >
 DIANode (Context &ctx, const char *label, const std::initializer_list< size_t > &parent_ids, const std::initializer_list< DIABasePtr > &parents)
 Constructor for a DIANode, which sets references to the parent nodes. More...
 
 DIANode (Context &ctx, const char *label, std::vector< size_t > &&parent_ids, std::vector< DIABasePtr > &&parents)
 Constructor for a DIANode, which sets references to the parent nodes. More...
 
virtual void AddChild (DIABase *node, const Callback &callback=Callback(), size_t parent_index=0)
 Enables children to push their "folded" function chains to their parent. More...
 
std::vector< DIABase * > children () const override
 Returns the children of this DIABase. More...
 
void PushFile (data::File &file, bool consume) const
 
void PushItem (const ValueType &item) const
 Method for derived classes to Push a single item to all children. More...
 
void RemoveAllChildren () override
 
void RemoveChild (DIABase *node) override
 
void RunPushData () override
 
- Public Member Functions inherited from DIABase
 DIABase (Context &ctx, const char *label, const std::initializer_list< size_t > &parent_ids, const std::initializer_list< DIABasePtr > &parents)
 The constructor for a DIABase. More...
 
 DIABase (Context &ctx, const char *label, std::vector< size_t > &&parent_ids, std::vector< DIABasePtr > &&parents)
 The constructor for a DIABase. More...
 
 DIABase (const DIABase &)=delete
 non-copyable: delete copy-constructor More...
 
 DIABase (DIABase &&)=default
 move-constructor: default More...
 
virtual ~DIABase ()
 Virtual destructor for a DIABase. More...
 
virtual size_t consume_counter () const
 Returns consume_counter_. More...
 
Contextcontext ()
 Returns the api::Context of this DIABase. More...
 
virtual void DecConsumeCounter (size_t counter)
 
const size_t & dia_id () const
 return unique id of DIANode subclass as stored by StatsNode More...
 
virtual bool ForwardDataOnly () const
 
virtual void IncConsumeCounter (size_t counter)
 
const char * label () const
 return label() of DIANode subclass as stored by StatsNode More...
 
mem::Managermem_manager ()
 Return the Context's memory manager. More...
 
DIABaseoperator= (const DIABase &)=delete
 non-copyable: delete assignment operator More...
 
DIABaseoperator= (DIABase &&)=default
 move-assignment operator: default More...
 
std::vector< size_t > parent_ids () const
 Returns the parents of this DIABase. More...
 
const std::vector< DIABasePtr > & parents () const
 Returns the parents of this DIABase. More...
 
void RemoveParent (DIABase *p)
 Remove a parent. More...
 
virtual bool RequireParentPushData (size_t) const
 
void RunScope ()
 
void set_mem_limit (const DIAMemUse &mem_limit)
 
void set_state (const DIAState &state)
 
virtual void SetConsumeCounter (size_t counter)
 
DIAState state () const
 
virtual bool OnPreOpFile (const data::File &, size_t)
 
- Public Member Functions inherited from ReferenceCounter
 ReferenceCounter () noexcept
 new objects have zero reference count More...
 
 ReferenceCounter (const ReferenceCounter &) noexcept
 coping still creates a new object with zero reference count More...
 
 ~ReferenceCounter ()
 
bool dec_reference () const noexcept
 Call whenever resetting (i.e. More...
 
void inc_reference () const noexcept
 Call whenever setting a pointer to the object. More...
 
ReferenceCounteroperator= (const ReferenceCounter &) noexcept
 assignment operator, leaves pointers unchanged More...
 
size_t reference_count () const noexcept
 Return the number of references to this object (for debugging) More...
 
bool unique () const noexcept
 Test if the ReferenceCounter is referenced by only one CountingPtr. More...
 

Private Types

using Key = typename common::FunctionTraits< KeyExtractor >::result_type
 
using Super = DOpNode< ValueType >
 
using ValueIn = typename common::FunctionTraits< KeyExtractor >::template arg_plain< 0 >
 
using ValueOut = ValueType
 

Private Member Functions

void FlushVectorToFile (std::vector< ValueIn > &v)
 Sort and store elements in a file. More...
 
void MainOp ()
 Receive elements from other workers. More...
 
void RunUserFunc (data::File &f, bool consume)
 

Private Attributes

data::CatStream::Writers emitters_
 
std::deque< data::Filefiles_
 
GroupFunction groupby_function_
 
HashFunction hash_function_
 
KeyExtractor key_extractor_
 
core::LocationDetection< HashCountlocation_detection_
 
data::File pre_file_
 location detection and associated files More...
 
data::File::Writer pre_writer_
 
data::File sorted_elems_ { context_.GetFile(this) }
 
data::CatStreamPtr stream_ { context_.GetNewCatStream(this) }
 
size_t totalsize_ = 0
 

Static Private Attributes

static constexpr bool debug = false
 

Additional Inherited Members

- Public Types inherited from DOpNode< ValueType >
using Super = DIANode< ValueType >
 
- Public Types inherited from DIANode< ValueType >
using Callback = tlx::delegate< void(const ValueType &)>
 
- Public Types inherited from DIABase
using DIABasePtr = tlx::CountingPtr< DIABase >
 
- Public Attributes inherited from DIABase
common::JsonLogger logger_
 
- Static Public Attributes inherited from DIABase
static constexpr size_t kNeverConsume = static_cast<size_t>(-1)
 Never full consume. More...
 
- Protected Attributes inherited from DIANode< ValueType >
std::vector< Childchildren_
 Callback functions from the child nodes. More...
 
- Protected Attributes inherited from DIABase
Contextcontext_
 associated Context More...
 
const size_t dia_id_
 DIA serial id. More...
 
const char *const label_
 DOp node static label. More...
 
DIAState state_ = DIAState::NEW
 State of the DIANode. State is NEW on creation. More...
 
std::vector< DIABasePtrparents_
 Parents of this DIABase. More...
 
DIAMemUse mem_limit_ = 0
 
size_t consume_counter_ = 1
 

Member Typedef Documentation

◆ Key

using Key = typename common::FunctionTraits<KeyExtractor>::result_type
private

Definition at line 55 of file group_by_key.hpp.

◆ Super

using Super = DOpNode<ValueType>
private

Definition at line 52 of file group_by_key.hpp.

◆ ValueIn

using ValueIn = typename common::FunctionTraits<KeyExtractor>::template arg_plain<0>
private

Definition at line 58 of file group_by_key.hpp.

◆ ValueOut

using ValueOut = ValueType
private

Definition at line 56 of file group_by_key.hpp.

Constructor & Destructor Documentation

◆ GroupByNode()

Member Function Documentation

◆ Dispose()

void Dispose ( )
inlineoverridevirtual

Virtual clear method. Triggers actual disposing in sub-classes.

Reimplemented from DIABase.

Definition at line 294 of file group_by_key.hpp.

◆ Execute()

◆ ExecuteMemUse()

DIAMemUse ExecuteMemUse ( )
inlinefinalvirtual

Amount of RAM used by Execute()

Reimplemented from DIABase.

Definition at line 169 of file group_by_key.hpp.

References DIAMemUse::Max().

◆ FlushVectorToFile()

◆ MainOp()

◆ PreOp()

◆ PreOpMemUse()

DIAMemUse PreOpMemUse ( )
inlinefinalvirtual

Amount of RAM used by PreOp after StartPreOp()

Reimplemented from DIABase.

Definition at line 165 of file group_by_key.hpp.

References DIAMemUse::Max().

◆ PushData()

◆ PushDataMemUse()

DIAMemUse PushDataMemUse ( )
inlinefinalvirtual

◆ RunUserFunc()

◆ StartPreOp()

◆ StopPreOp()

void StopPreOp ( size_t  )
inlinefinalvirtual

Virtual method for preparing end of PushData.

Reimplemented from DIABase.

Definition at line 161 of file group_by_key.hpp.

References BlockWriter< BlockSink >::Close(), and GroupByNode< ValueType, KeyExtractor, GroupFunction, HashFunction, UseLocationDetection >::pre_writer_.

Member Data Documentation

◆ debug

constexpr bool debug = false
staticprivate

Definition at line 50 of file group_by_key.hpp.

◆ emitters_

◆ files_

◆ groupby_function_

◆ hash_function_

◆ key_extractor_

◆ location_detection_

◆ pre_file_

◆ pre_writer_

◆ sorted_elems_

data::File sorted_elems_ { context_.GetFile(this) }
private

Definition at line 307 of file group_by_key.hpp.

◆ stream_

◆ totalsize_

size_t totalsize_ = 0
private

Definition at line 308 of file group_by_key.hpp.


The documentation for this class was generated from the following files: