Thrill  0.1
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
DIABase Class Referenceabstract

Detailed Description

The DIABase is the untyped super class of DIANode.

DIABases are used to build the execution graph, which is used to execute the computation.

Each DIABase knows it's parents. Parents are node which have to computed previously. Not all DIABases have children (ActionNodes do not), hence, children are first introduced in DIANode.

Definition at line 87 of file dia_base.hpp.

+ Inheritance diagram for DIABase:
+ Collaboration diagram for DIABase:

#include <dia_base.hpp>

Public Types

using DIABasePtr = tlx::CountingPtr< DIABase >
 

Public Member Functions

 DIABase (Context &ctx, const char *label, const std::initializer_list< size_t > &parent_ids, const std::initializer_list< DIABasePtr > &parents)
 The constructor for a DIABase. More...
 
 DIABase (Context &ctx, const char *label, std::vector< size_t > &&parent_ids, std::vector< DIABasePtr > &&parents)
 The constructor for a DIABase. More...
 
 DIABase (const DIABase &)=delete
 non-copyable: delete copy-constructor More...
 
 DIABase (DIABase &&)=default
 move-constructor: default More...
 
virtual ~DIABase ()
 Virtual destructor for a DIABase. More...
 
virtual std::vector< DIABase * > children () const =0
 Returns the children of this DIABase. More...
 
virtual size_t consume_counter () const
 Returns consume_counter_. More...
 
Contextcontext ()
 Returns the api::Context of this DIABase. More...
 
virtual void DecConsumeCounter (size_t counter)
 
virtual bool ForwardDataOnly () const
 
const size_t & id () const
 return unique id() of DIANode subclass as stored by StatsNode More...
 
virtual void IncConsumeCounter (size_t counter)
 
const char * label () const
 return label() of DIANode subclass as stored by StatsNode More...
 
mem::Managermem_manager ()
 Return the Context's memory manager. More...
 
DIABaseoperator= (const DIABase &)=delete
 non-copyable: delete assignment operator More...
 
DIABaseoperator= (DIABase &&)=default
 move-assignment operator: default More...
 
std::vector< size_t > parent_ids () const
 Returns the parents of this DIABase. More...
 
const std::vector< DIABasePtr > & parents () const
 Returns the parents of this DIABase. More...
 
virtual void RemoveAllChildren ()=0
 
virtual void RemoveChild (DIABase *node)=0
 Virtual method for removing a child. More...
 
void RemoveParent (DIABase *p)
 Remove a parent. More...
 
virtual bool RequireParentPushData (size_t) const
 
virtual void RunPushData ()=0
 
void RunScope ()
 
void set_mem_limit (const DIAMemUse &mem_limit)
 
void set_state (const DIAState &state)
 
virtual void SetConsumeCounter (size_t counter)
 
DIAState state () const
 
Pure Virtual Methods called by StageBuilder
virtual DIAMemUse PreOpMemUse ()
 Amount of RAM used by PreOp after StartPreOp() More...
 
virtual void StartPreOp (size_t)
 Virtual method for preparing start of PushData. More...
 
virtual bool OnPreOpFile (const data::File &, size_t)
 
virtual void StopPreOp (size_t)
 Virtual method for preparing end of PushData. More...
 
virtual DIAMemUse ExecuteMemUse ()
 Amount of RAM used by Execute() More...
 
virtual void Execute ()=0
 Virtual execution method. Triggers actual computation in sub-classes. More...
 
virtual DIAMemUse PushDataMemUse ()
 Amount of RAM used by PushData() More...
 
virtual void PushData (bool consume)=0
 Virtual method for pushing data. Triggers actual pushing in sub-classes. More...
 
virtual void Dispose ()
 Virtual clear method. Triggers actual disposing in sub-classes. More...
 
- Public Member Functions inherited from ReferenceCounter
 ReferenceCounter () noexcept
 new objects have zero reference count More...
 
 ReferenceCounter (const ReferenceCounter &) noexcept
 coping still creates a new object with zero reference count More...
 
 ~ReferenceCounter ()
 
bool dec_reference () const noexcept
 Call whenever resetting (i.e. More...
 
void inc_reference () const noexcept
 Call whenever setting a pointer to the object. More...
 
ReferenceCounteroperator= (const ReferenceCounter &) noexcept
 assignment operator, leaves pointers unchanged More...
 
size_t reference_count () const noexcept
 Return the number of references to this object (for debugging) More...
 
bool unique () const noexcept
 Test if the ReferenceCounter is referenced by only one CountingPtr. More...
 

Public Attributes

common::JsonLogger logger_
 

Static Public Attributes

static constexpr size_t kNeverConsume = static_cast<size_t>(-1)
 Never full consume. More...
 

Protected Attributes

Fixed DIA Information
Contextcontext_
 associated Context More...
 
const size_t id_
 DIA serial id. More...
 
const char *const label_
 DOp node static label. More...
 
Runtime Operational Variables
DIAState state_ = DIAState::NEW
 State of the DIANode. State is NEW on creation. More...
 
std::vector< DIABasePtrparents_
 Parents of this DIABase. More...
 
DIAMemUse mem_limit_ = 0
 
size_t consume_counter_ = 1
 

Friends

std::ostream & operator<< (std::ostream &os, const DIABase &d)
 make ostream-able. More...
 

Member Typedef Documentation

Definition at line 90 of file dia_base.hpp.

Constructor & Destructor Documentation

DIABase ( Context ctx,
const char *  label,
const std::initializer_list< size_t > &  parent_ids,
const std::initializer_list< DIABasePtr > &  parents 
)
inline

The constructor for a DIABase.

Sets the parents for this node, but does not register it has a child, since this must be done with a callback.

Definition at line 96 of file dia_base.hpp.

References DIABase::logger_, and DIABase::parent_ids().

DIABase ( Context ctx,
const char *  label,
std::vector< size_t > &&  parent_ids,
std::vector< DIABasePtr > &&  parents 
)
inline

The constructor for a DIABase.

Sets the parents for this node, but does not register it has a child, since this must be done with a callback.

Definition at line 111 of file dia_base.hpp.

References DIABase::logger_, and DIABase::parent_ids().

DIABase ( const DIABase )
delete

non-copyable: delete copy-constructor

DIABase ( DIABase &&  )
default

move-constructor: default

virtual ~DIABase ( )
inlinevirtual

Virtual destructor for a DIABase.

Definition at line 132 of file dia_base.hpp.

References DIABase::logger_, DIABase::parent_ids(), and DIABase::parents_.

Member Function Documentation

virtual std::vector<DIABase*> children ( ) const
pure virtual
virtual size_t consume_counter ( ) const
inlinevirtual

Returns consume_counter_.

Reimplemented in UnionNode< ValueType >.

Definition at line 226 of file dia_base.hpp.

References DIABase::consume_counter_.

Referenced by DIA< ValueType_, Stack_ >::Keep(), and DIANode< StackInput >::RunPushData().

virtual void DecConsumeCounter ( size_t  counter)
inlinevirtual

Virtual SetConsume flag which is called by the user via .Keep() or .Consume() to set consumption.

Reimplemented in UnionNode< ValueType >, and ActionNode.

Definition at line 237 of file dia_base.hpp.

References DIABase::consume_counter_.

Referenced by DIANode< StackInput >::RunPushData().

virtual DIAMemUse ExecuteMemUse ( )
inlinevirtual
virtual bool ForwardDataOnly ( ) const
inlinevirtual

Virtual method to determine whether a node contains data or not, and hence if it can be Executed() and PushData() or whether it is only a forwarding node. This is currently true only for Collapse() and Union().

Reimplemented in UnionNode< ValueType >.

Definition at line 148 of file dia_base.hpp.

Referenced by thrill::api::FindStages(), and DIABase::RunScope().

virtual void IncConsumeCounter ( size_t  counter)
inlinevirtual

Virtual SetConsume flag which is called by the user via .Keep() or .Consume() to set consumption.

Reimplemented in UnionNode< ValueType >, and ActionNode.

Definition at line 230 of file dia_base.hpp.

References DIABase::consume_counter_, and DIABase::kNeverConsume.

Referenced by DIA< ValueType_, Stack_ >::Keep().

const char* label ( ) const
inline

return label() of DIANode subclass as stored by StatsNode

Definition at line 218 of file dia_base.hpp.

References DIABase::label_.

Referenced by thrill::api::operator<<().

mem::Manager& mem_manager ( )
inline

Return the Context's memory manager.

Definition at line 278 of file dia_base.hpp.

References DIABase::context_, and Context::mem_manager().

Referenced by thrill::api::FindStages(), and DIABase::RunScope().

virtual bool OnPreOpFile ( const data::File ,
size_t   
)
inlinevirtual

Virtual method for receiving a whole data::File of ValueType from parent. Returns true if the file was accepted (requires that the child's function stack is empty and that it can accept whole data::Files).

Reimplemented in ConcatNode< ValueType >, and MergeNode< ValueType, Comparator, kNumInputs >.

Definition at line 168 of file dia_base.hpp.

DIABase& operator= ( const DIABase )
delete

non-copyable: delete assignment operator

DIABase& operator= ( DIABase &&  )
default

move-assignment operator: default

std::vector<size_t> parent_ids ( ) const
inline
const std::vector<DIABasePtr>& parents ( ) const
inline

Returns the parents of this DIABase.

Definition at line 253 of file dia_base.hpp.

References DIABase::parents_.

Referenced by thrill::api::FindStages().

virtual DIAMemUse PreOpMemUse ( )
inlinevirtual
virtual DIAMemUse PushDataMemUse ( )
inlinevirtual
virtual void RemoveAllChildren ( )
pure virtual

Virtual method for removing all childs. Triggers actual removing in sub-classes.

Implemented in UnionNode< ValueType >, DIANode< ValueType >, DIANode< std::string >, DIANode< StackInput >, and ActionNode.

virtual void RemoveChild ( DIABase node)
pure virtual

Virtual method for removing a child.

Implemented in UnionNode< ValueType >, DIANode< ValueType >, DIANode< std::string >, DIANode< StackInput >, and ActionNode.

void RemoveParent ( DIABase p)
inline

Remove a parent.

Definition at line 265 of file dia_base.hpp.

References DIABase::parents_.

virtual bool RequireParentPushData ( size_t  ) const
inlinevirtual

Virtual method used by StageBuilder to request information whether it must call PushData on the parent of a CollapseNode or UnionNode to correctly deliver data.

Reimplemented in UnionNode< ValueType >.

Definition at line 153 of file dia_base.hpp.

Referenced by thrill::api::FindStages().

virtual void RunPushData ( )
pure virtual

Performing push operation. Notifies children and calls actual push method. Then cleans up the DIA graph by freeing parent references of children.

Implemented in UnionNode< ValueType >, DIANode< ValueType >, DIANode< std::string >, DIANode< StackInput >, and ActionNode.

void set_mem_limit ( const DIAMemUse mem_limit)
inline

Definition at line 286 of file dia_base.hpp.

References DIABase::mem_limit_.

void set_state ( const DIAState state)
inline

Definition at line 284 of file dia_base.hpp.

References DIABase::state(), and DIABase::state_.

virtual void SetConsumeCounter ( size_t  counter)
inlinevirtual

Virtual SetConsume flag which is called by the user via .Keep() or .Consume() to set consumption.

Reimplemented in UnionNode< ValueType >, and ActionNode.

Definition at line 248 of file dia_base.hpp.

References DIABase::consume_counter_.

Referenced by DIA< ValueType_, Stack_ >::KeepForever().

virtual void StartPreOp ( size_t  )
inlinevirtual
virtual void StopPreOp ( size_t  )
inlinevirtual

Friends And Related Function Documentation

std ::ostream& operator<< ( std::ostream &  os,
const DIABase d 
)
friend

make ostream-able.

Definition at line 449 of file dia_base.cpp.

Member Data Documentation

size_t consume_counter_ = 1
protected

Consumption counter: when it reaches zero, PushData() is called with consume = true

Definition at line 318 of file dia_base.hpp.

Referenced by DIABase::consume_counter(), DIABase::DecConsumeCounter(), DIABase::IncConsumeCounter(), and DIABase::SetConsumeCounter().

Context& context_
protected

associated Context

Definition at line 293 of file dia_base.hpp.

Referenced by JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::AddEqualKeysToVec(), DIABase::context(), ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::DoScatter(), ZipWindowNode< ValueType, ZipFunction_, Pad_, UnequalCheck, UseStdArray, kNumInputs_ >::DoScatter(), DistributeNode< ValueType >::Execute(), OverlapWindowNode< ValueType, Input, WindowFunction, PartialWindowFunction >::Execute(), ConcatNode< ValueType >::Execute(), DisjointWindowNode< ValueType, Input, WindowFunction, PartialWindowFunction >::Execute(), SortNode< ValueType, CompareFunction, SortAlgorithm >::FindAndSendSplitters(), GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::FlushVectorToFile(), GroupByNode< ValueType, KeyExtractor, GroupFunction, HashFunction, UseLocationDetection >::FlushVectorToFile(), MergeNode< ValueType, Comparator, kNumInputs >::GetGlobalRanks(), ReadLinesNode::InputLineIteratorCompressed::InputLineIteratorCompressed(), ReadLinesNode::InputLineIteratorUncompressed::InputLineIteratorUncompressed(), ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::MainOp(), ZipWindowNode< ValueType, ZipFunction_, Pad_, UnequalCheck, UseStdArray, kNumInputs_ >::MainOp(), MergeNode< ValueType, Comparator, kNumInputs >::MainOp(), SortNode< ValueType, CompareFunction, SortAlgorithm >::MainOp(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::MakePuller(), DIABase::mem_manager(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::MergeFiles(), WriteBinaryNode< ValueType >::OpenNextFile(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::PreOp1(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::PreOp2(), EqualToDIANode< ValueType >::PushData(), GenerateNode< ValueType, GenerateFunction >::PushData(), MergeNode< ValueType, Comparator, kNumInputs >::PushData(), OverlapWindowNode< ValueType, Input, WindowFunction, PartialWindowFunction >::PushData(), DisjointWindowNode< ValueType, Input, WindowFunction, PartialWindowFunction >::PushData(), ReadBinaryNode< ValueType >::ReadBinaryNode(), SortNode< ValueType, CompareFunction, SortAlgorithm >::ReceiveItems(), DIABase::RunScope(), MergeNode< ValueType, Comparator, kNumInputs >::SelectPivots(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::SortAndWriteToFile(), and SortNode< ValueType, CompareFunction, SortAlgorithm >::SortAndWriteToFile().

const size_t id_
protected

DIA serial id.

Definition at line 296 of file dia_base.hpp.

Referenced by DIABase::id().

constexpr size_t kNeverConsume = static_cast<size_t>(-1)
static
const char* const label_
protected

DOp node static label.

Definition at line 299 of file dia_base.hpp.

Referenced by DIABase::label().

DIAState state_ = DIAState::NEW
protected

State of the DIANode. State is NEW on creation.

Definition at line 307 of file dia_base.hpp.

Referenced by DIABase::RunScope(), DIABase::set_state(), and DIABase::state().


The documentation for this class was generated from the following files: