Thrill  0.1
Public Types | Public Member Functions | Public Attributes | Static Public Attributes | Friends | List of all members
DIABase Class Referenceabstract

Detailed Description

The DIABase is the untyped super class of DIANode.

DIABases are used to build the execution graph, which is used to execute the computation.

Each DIABase knows it's parents. Parents are node which have to computed previously. Not all DIABases have children (ActionNodes do not), hence, children are first introduced in DIANode.

Definition at line 87 of file dia_base.hpp.

+ Inheritance diagram for DIABase:
+ Collaboration diagram for DIABase:

#include <dia_base.hpp>

Public Types

using DIABasePtr = tlx::CountingPtr< DIABase >
 

Public Member Functions

 DIABase (Context &ctx, const char *label, const std::initializer_list< size_t > &parent_ids, const std::initializer_list< DIABasePtr > &parents)
 The constructor for a DIABase. More...
 
 DIABase (Context &ctx, const char *label, std::vector< size_t > &&parent_ids, std::vector< DIABasePtr > &&parents)
 The constructor for a DIABase. More...
 
 DIABase (const DIABase &)=delete
 non-copyable: delete copy-constructor More...
 
 DIABase (DIABase &&)=default
 move-constructor: default More...
 
virtual ~DIABase ()
 Virtual destructor for a DIABase. More...
 
virtual std::vector< DIABase * > children () const =0
 Returns the children of this DIABase. More...
 
virtual size_t consume_counter () const
 Returns consume_counter_. More...
 
Context & context ()
 Returns the api::Context of this DIABase. More...
 
virtual void DecConsumeCounter (size_t counter)
 
const size_t & dia_id () const
 return unique id of DIANode subclass as stored by StatsNode More...
 
virtual bool ForwardDataOnly () const
 
virtual void IncConsumeCounter (size_t counter)
 
const char * label () const
 return label() of DIANode subclass as stored by StatsNode More...
 
mem::Manager & mem_manager ()
 Return the Context's memory manager. More...
 
DIABase & operator= (const DIABase &)=delete
 non-copyable: delete assignment operator More...
 
DIABase & operator= (DIABase &&)=default
 move-assignment operator: default More...
 
std::vector< size_t > parent_ids () const
 Returns the parents of this DIABase. More...
 
const std::vector< DIABasePtr > & parents () const
 Returns the parents of this DIABase. More...
 
virtual void RemoveAllChildren ()=0
 
virtual void RemoveChild (DIABase *node)=0
 Virtual method for removing a child. More...
 
void RemoveParent (DIABase *p)
 Remove a parent. More...
 
virtual bool RequireParentPushData (size_t) const
 
virtual void RunPushData ()=0
 
void RunScope ()
 
void set_mem_limit (const DIAMemUse &mem_limit)
 
void set_state (const DIAState &state)
 
virtual void SetConsumeCounter (size_t counter)
 
DIAState state () const
 
Pure Virtual Methods called by StageBuilder
virtual DIAMemUse PreOpMemUse ()
 Amount of RAM used by PreOp after StartPreOp() More...
 
virtual void StartPreOp (size_t)
 Virtual method for preparing start of PushData. More...
 
virtual bool OnPreOpFile (const data::File &, size_t)
 
virtual void StopPreOp (size_t)
 Virtual method for preparing end of PushData. More...
 
virtual DIAMemUse ExecuteMemUse ()
 Amount of RAM used by Execute() More...
 
virtual void Execute ()=0
 Virtual execution method. Triggers actual computation in sub-classes. More...
 
virtual DIAMemUse PushDataMemUse ()
 Amount of RAM used by PushData() More...
 
virtual void PushData (bool consume)=0
 Virtual method for pushing data. Triggers actual pushing in sub-classes. More...
 
virtual void Dispose ()
 Virtual clear method. Triggers actual disposing in sub-classes. More...
 
- Public Member Functions inherited from ReferenceCounter
 ReferenceCounter () noexcept
 new objects have zero reference count More...
 
 ReferenceCounter (const ReferenceCounter &) noexcept
 coping still creates a new object with zero reference count More...
 
 ~ReferenceCounter ()
 
bool dec_reference () const noexcept
 Call whenever resetting (i.e. More...
 
void inc_reference () const noexcept
 Call whenever setting a pointer to the object. More...
 
ReferenceCounter & operator= (const ReferenceCounter &) noexcept
 assignment operator, leaves pointers unchanged More...
 
size_t reference_count () const noexcept
 Return the number of references to this object (for debugging) More...
 
bool unique () const noexcept
 Test if the ReferenceCounter is referenced by only one CountingPtr. More...
 

Public Attributes

common::JsonLogger logger_
 

Static Public Attributes

static constexpr size_t kNeverConsume = static_cast<size_t>(-1)
 Never full consume. More...
 

Protected Attributes

Fixed DIA Information
Context & context_
 associated Context More...
 
const size_t dia_id_
 DIA serial id. More...
 
const char *const label_
 DOp node static label. More...
 
Runtime Operational Variables
DIAState state_ = DIAState::NEW
 State of the DIANode. State is NEW on creation. More...
 
std::vector< DIABasePtr > parents_
 Parents of this DIABase. More...
 
DIAMemUse mem_limit_ = 0
 
size_t consume_counter_ = 1
 

Friends

std::ostream & operator<< (std::ostream &os, const DIABase &d)
 make ostream-able. More...
 

Member Typedef Documentation

◆ DIABasePtr

using DIABasePtr = tlx::CountingPtr<DIABase>

Definition at line 90 of file dia_base.hpp.

Constructor & Destructor Documentation

◆ DIABase() [1/4]

DIABase ( Context &  ctx,
const char *  label,
const std::initializer_list< size_t > &  parent_ids,
const std::initializer_list< DIABasePtr > &  parents 
)
inline

The constructor for a DIABase.

Sets the parents for this node, but does not register it has a child, since this must be done with a callback.

Definition at line 96 of file dia_base.hpp.

◆ DIABase() [2/4]

DIABase ( Context &  ctx,
const char *  label,
std::vector< size_t > &&  parent_ids,
std::vector< DIABasePtr > &&  parents 
)
inline

The constructor for a DIABase.

Sets the parents for this node, but does not register it has a child, since this must be done with a callback.

Definition at line 111 of file dia_base.hpp.

◆ DIABase() [3/4]

DIABase ( const DIABase &  )
delete

non-copyable: delete copy-constructor

◆ DIABase() [4/4]

DIABase ( DIABase &&  )
default

move-constructor: default

◆ ~DIABase()

virtual ~DIABase ( )
inlinevirtual

Virtual destructor for a DIABase.

Definition at line 132 of file dia_base.hpp.

Member Function Documentation

◆ children()

virtual std::vector<DIABase*> children ( ) const
pure virtual

Returns the children of this DIABase.

Implemented in UnionNode< ValueType >, DIANode< ValueType >, DIANode< std::string >, DIANode< StackInput >, and ActionNode.

◆ consume_counter()

virtual size_t consume_counter ( ) const
inlinevirtual

Returns consume_counter_.

Reimplemented in UnionNode< ValueType >, and CollapseNode< ValueType >.

Definition at line 226 of file dia_base.hpp.

Referenced by DIANode< StackInput >::RunPushData().

◆ context()

Context& context ( )
inline

Returns the api::Context of this DIABase.

Definition at line 208 of file dia_base.hpp.

Referenced by DIANode< StackInput >::RunPushData().

◆ DecConsumeCounter()

virtual void DecConsumeCounter ( size_t  counter)
inlinevirtual

Virtual SetConsume flag which is called by the user via .Keep() or .Consume() to set consumption.

Reimplemented in UnionNode< ValueType >, CollapseNode< ValueType >, and ActionNode.

Definition at line 237 of file dia_base.hpp.

Referenced by DIANode< StackInput >::RunPushData().

◆ dia_id()

const size_t& dia_id ( ) const
inline

return unique id of DIANode subclass as stored by StatsNode

Definition at line 213 of file dia_base.hpp.

Referenced by Context::GetFile(), Context::GetFilePtr(), Context::GetNewCatStream(), Context::GetNewMixStream(), GroupByNode< ValueType, KeyExtractor, GroupFunction, HashFunction, UseLocationDetection >::GroupByNode(), SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >::MainOp(), SizeNode< ValueType >::OnPreOpFile(), and thrill::api::operator<<().

◆ Dispose()

virtual void Dispose ( )
inlinevirtual

Virtual clear method. Triggers actual disposing in sub-classes.

Reimplemented in ConcatNode< ValueType >, GroupByNode< ValueType, KeyExtractor, GroupFunction, HashFunction, UseLocationDetection >, JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >, ZipWindowNode< ValueType, ZipFunction_, Pad_, UnequalCheck, UseStdArray, kNumInputs_ >, SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >, ReadBinaryNode< ValueType >, ReduceToIndexNode< ValueType, KeyExtractor, ReduceFunction, ReduceConfig, VolatileKey, SkipPreReducePhase >, ReduceNode< ValueType, KeyExtractor, ReduceFunction, ReduceConfig, KeyHashFunction, KeyEqualFunction, VolatileKey, UseDuplicateDetection >, ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >, MergeNode< ValueType, Comparator, kNumInputs >, GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >, SampleNode< ValueType >, PrefixSumNode< ValueType, SumFunction, Inclusive >, ZipWithIndexNode< ValueType, ZipFunction >, BaseWindowNode< ValueType, Input, WindowFunction, PartialWindowFunction >, RebalanceNode< ValueType >, DistributeNode< ValueType >, CacheNode< ValueType >, EqualToDIANode< ValueType >, and ConcatToDIANode< ValueType >.

Definition at line 188 of file dia_base.hpp.

Referenced by DIANode< StackInput >::RunPushData().

◆ Execute()

virtual void Execute ( )
pure virtual

Virtual execution method. Triggers actual computation in sub-classes.

Implemented in DisjointWindowNode< ValueType, Input, WindowFunction, PartialWindowFunction >, ZipWindowNode< ValueType, ZipFunction_, Pad_, UnequalCheck, UseStdArray, kNumInputs_ >, UnionNode< ValueType >, SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >, ConcatNode< ValueType >, ReduceToIndexNode< ValueType, KeyExtractor, ReduceFunction, ReduceConfig, VolatileKey, SkipPreReducePhase >, GroupByNode< ValueType, KeyExtractor, GroupFunction, HashFunction, UseLocationDetection >, ReduceNode< ValueType, KeyExtractor, ReduceFunction, ReduceConfig, KeyHashFunction, KeyEqualFunction, VolatileKey, UseDuplicateDetection >, OverlapWindowNode< ValueType, Input, WindowFunction, PartialWindowFunction >, JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >, MergeNode< ValueType, Comparator, kNumInputs >, ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >, WriteLinesNode< ValueType >, GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >, WriteBinaryNode< ValueType >, AllGatherNode< ValueType >, ZipWithIndexNode< ValueType, ZipFunction >, PrefixSumNode< ValueType, SumFunction, Inclusive >, SampleNode< ValueType >, CacheNode< ValueType >, GatherNode< ValueType >, RebalanceNode< ValueType >, WriteLinesOneNode< ValueType >, AllReduceNode< ValueType, ReduceFunction >, SizeNode< ValueType >, DistributeNode< ValueType >, CollapseNode< ValueType >, HyperLogLogNode< p, ValueType >, SourceNode< ValueType >, and SourceNode< std::string >.

◆ ExecuteMemUse()

virtual DIAMemUse ExecuteMemUse ( )
inlinevirtual

Amount of RAM used by Execute()

Reimplemented in JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >, SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >, and GroupByNode< ValueType, KeyExtractor, GroupFunction, HashFunction, UseLocationDetection >.

Definition at line 176 of file dia_base.hpp.

◆ ForwardDataOnly()

virtual bool ForwardDataOnly ( ) const
inlinevirtual

Virtual method to determine whether a node contains data or not, and hence if it can be Executed() and PushData() or whether it is only a forwarding node. This is currently true only for Collapse() and Union().

Reimplemented in UnionNode< ValueType >, and CollapseNode< ValueType >.

Definition at line 148 of file dia_base.hpp.

◆ IncConsumeCounter()

virtual void IncConsumeCounter ( size_t  counter)
inlinevirtual

Virtual SetConsume flag which is called by the user via .Keep() or .Consume() to set consumption.

Reimplemented in UnionNode< ValueType >, CollapseNode< ValueType >, and ActionNode.

Definition at line 230 of file dia_base.hpp.

◆ label()

const char* label ( ) const
inline

return label() of DIANode subclass as stored by StatsNode

Definition at line 218 of file dia_base.hpp.

References thrill::api::operator<<().

Referenced by thrill::api::operator<<().

◆ mem_manager()

mem::Manager& mem_manager ( )
inline

Return the Context's memory manager.

Definition at line 278 of file dia_base.hpp.

◆ OnPreOpFile()

virtual bool OnPreOpFile ( const data::File &  ,
size_t   
)
inlinevirtual

Virtual method for receiving a whole data::File of ValueType from parent. Returns true if the file was accepted (requires that the child's function stack is empty and that it can accept whole data::Files).

Reimplemented in ZipWindowNode< ValueType, ZipFunction_, Pad_, UnequalCheck, UseStdArray, kNumInputs_ >, ConcatNode< ValueType >, SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >, MergeNode< ValueType, Comparator, kNumInputs >, ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >, BaseWindowNode< ValueType, Input, WindowFunction, PartialWindowFunction >, ZipWithIndexNode< ValueType, ZipFunction >, AllGatherNode< ValueType >, CollapseNode< ValueType >, PrefixSumNode< ValueType, SumFunction, Inclusive >, CacheNode< ValueType >, RebalanceNode< ValueType >, and SizeNode< ValueType >.

Definition at line 168 of file dia_base.hpp.

◆ operator=() [1/2]

DIABase& operator= ( const DIABase &  )
delete

non-copyable: delete assignment operator

◆ operator=() [2/2]

DIABase& operator= ( DIABase &&  )
default

move-assignment operator: default

◆ parent_ids()

std::vector<size_t> parent_ids ( ) const
inline

Returns the parents of this DIABase.

Definition at line 258 of file dia_base.hpp.

Referenced by JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::StartPreOp().

◆ parents()

const std::vector<DIABasePtr>& parents ( ) const
inline

Returns the parents of this DIABase.

Definition at line 253 of file dia_base.hpp.

Referenced by ConcatNode< ValueType >::ConcatNode(), MergeNode< ValueType, Comparator, kNumInputs >::MergeNode(), UnionNode< ValueType >::UnionNode(), and ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::ZipNode().

◆ PreOpMemUse()

virtual DIAMemUse PreOpMemUse ( )
inlinevirtual

Amount of RAM used by PreOp after StartPreOp()

Reimplemented in JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >, GroupByNode< ValueType, KeyExtractor, GroupFunction, HashFunction, UseLocationDetection >, ReduceToIndexNode< ValueType, KeyExtractor, ReduceFunction, ReduceConfig, VolatileKey, SkipPreReducePhase >, ReduceNode< ValueType, KeyExtractor, ReduceFunction, ReduceConfig, KeyHashFunction, KeyEqualFunction, VolatileKey, UseDuplicateDetection >, WriteLinesNode< ValueType >, WriteBinaryNode< ValueType >, and BaseWindowNode< ValueType, Input, WindowFunction, PartialWindowFunction >.

Definition at line 160 of file dia_base.hpp.

◆ PushData()

virtual void PushData ( bool  consume)
pure virtual

Virtual method for pushing data. Triggers actual pushing in sub-classes.

Implemented in DisjointWindowNode< ValueType, Input, WindowFunction, PartialWindowFunction >, ConcatNode< ValueType >, UnionNode< ValueType >, ZipWindowNode< ValueType, ZipFunction_, Pad_, UnequalCheck, UseStdArray, kNumInputs_ >, ReadBinaryNode< ValueType >, SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >, JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >, GroupByNode< ValueType, KeyExtractor, GroupFunction, HashFunction, UseLocationDetection >, ReduceToIndexNode< ValueType, KeyExtractor, ReduceFunction, ReduceConfig, VolatileKey, SkipPreReducePhase >, OverlapWindowNode< ValueType, Input, WindowFunction, PartialWindowFunction >, ReduceNode< ValueType, KeyExtractor, ReduceFunction, ReduceConfig, KeyHashFunction, KeyEqualFunction, VolatileKey, UseDuplicateDetection >, MergeNode< ValueType, Comparator, kNumInputs >, ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >, SampleNode< ValueType >, GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >, RebalanceNode< ValueType >, ZipWithIndexNode< ValueType, ZipFunction >, PrefixSumNode< ValueType, SumFunction, Inclusive >, CollapseNode< ValueType >, DistributeNode< ValueType >, ReadLinesNode, CacheNode< ValueType >, GenerateNode< ValueType, GenerateFunction >, ConcatToDIANode< ValueType >, EqualToDIANode< ValueType >, and ActionNode.

Referenced by DIANode< StackInput >::RunPushData().

◆ PushDataMemUse()

virtual DIAMemUse PushDataMemUse ( )
inlinevirtual

Amount of RAM used by PushData()

Reimplemented in JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >, SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >, ReduceToIndexNode< ValueType, KeyExtractor, ReduceFunction, ReduceConfig, VolatileKey, SkipPreReducePhase >, GroupByNode< ValueType, KeyExtractor, GroupFunction, HashFunction, UseLocationDetection >, ReduceNode< ValueType, KeyExtractor, ReduceFunction, ReduceConfig, KeyHashFunction, KeyEqualFunction, VolatileKey, UseDuplicateDetection >, BaseWindowNode< ValueType, Input, WindowFunction, PartialWindowFunction >, and ReadLinesNode.

Definition at line 182 of file dia_base.hpp.

◆ RemoveAllChildren()

virtual void RemoveAllChildren ( )
pure virtual

Virtual method for removing all childs. Triggers actual removing in sub-classes.

Implemented in UnionNode< ValueType >, DIANode< ValueType >, DIANode< std::string >, DIANode< StackInput >, and ActionNode.

◆ RemoveChild()

virtual void RemoveChild ( DIABase *  node)
pure virtual

Virtual method for removing a child.

Implemented in UnionNode< ValueType >, DIANode< ValueType >, DIANode< std::string >, DIANode< StackInput >, and ActionNode.

◆ RemoveParent()

void RemoveParent ( DIABase *  p)
inline

Remove a parent.

Definition at line 265 of file dia_base.hpp.

◆ RequireParentPushData()

virtual bool RequireParentPushData ( size_t  ) const
inlinevirtual

Virtual method used by StageBuilder to request information whether it must call PushData on the parent of a CollapseNode or UnionNode to correctly deliver data.

Reimplemented in UnionNode< ValueType >, and CollapseNode< ValueType >.

Definition at line 153 of file dia_base.hpp.

◆ RunPushData()

virtual void RunPushData ( )
pure virtual

Performing push operation. Notifies children and calls actual push method. Then cleans up the DIA graph by freeing parent references of children.

Implemented in UnionNode< ValueType >, DIANode< ValueType >, DIANode< std::string >, DIANode< StackInput >, and ActionNode.

◆ RunScope()

void RunScope ( )

Run Scope and parents such that this node (usually an ActionNode) is EXECUTED.

Definition at line 381 of file dia_base.cpp.

References debug, thrill::api::EXECUTED, thrill::api::FindStages(), LOG, thrill::mem::malloc_tracker_print_status(), thrill::api::NEW, and thrill::api::TopoSortStages().

Referenced by DIA< BfsNode >::AllGather(), DIA< BfsNode >::Gather(), DIA< BfsNode >::Max(), DIA< BfsNode >::Min(), DIA< BfsNode >::Print(), DIA< BfsNode >::Size(), DIA< BfsNode >::WriteBinary(), DIA< BfsNode >::WriteLines(), and DIA< BfsNode >::WriteLinesOne().

◆ set_mem_limit()

void set_mem_limit ( const DIAMemUse &  mem_limit)
inline

Definition at line 286 of file dia_base.hpp.

◆ set_state()

void set_state ( const DIAState &  state)
inline

Definition at line 284 of file dia_base.hpp.

◆ SetConsumeCounter()

virtual void SetConsumeCounter ( size_t  counter)
inlinevirtual

Virtual SetConsume flag which is called by the user via .Keep() or .Consume() to set consumption.

Reimplemented in UnionNode< ValueType >, CollapseNode< ValueType >, and ActionNode.

Definition at line 248 of file dia_base.hpp.

◆ StartPreOp()

virtual void StartPreOp ( size_t  )
inlinevirtual

Virtual method for preparing start of PushData.

Reimplemented in JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >, UnionNode< ValueType >, ZipWindowNode< ValueType, ZipFunction_, Pad_, UnequalCheck, UseStdArray, kNumInputs_ >, ReduceToIndexNode< ValueType, KeyExtractor, ReduceFunction, ReduceConfig, VolatileKey, SkipPreReducePhase >, ReduceNode< ValueType, KeyExtractor, ReduceFunction, ReduceConfig, KeyHashFunction, KeyEqualFunction, VolatileKey, UseDuplicateDetection >, GroupByNode< ValueType, KeyExtractor, GroupFunction, HashFunction, UseLocationDetection >, SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >, ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >, WriteLinesNode< ValueType >, BaseWindowNode< ValueType, Input, WindowFunction, PartialWindowFunction >, AllGatherNode< ValueType >, CollapseNode< ValueType >, and GatherNode< ValueType >.

Definition at line 163 of file dia_base.hpp.

◆ state()

DIAState state ( ) const
inline

Definition at line 282 of file dia_base.hpp.

◆ StopPreOp()

virtual void StopPreOp ( size_t  )
inlinevirtual

Virtual method for preparing end of PushData.

Reimplemented in JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >, UnionNode< ValueType >, ZipWindowNode< ValueType, ZipFunction_, Pad_, UnequalCheck, UseStdArray, kNumInputs_ >, ConcatNode< ValueType >, SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >, ReduceToIndexNode< ValueType, KeyExtractor, ReduceFunction, ReduceConfig, VolatileKey, SkipPreReducePhase >, GroupByNode< ValueType, KeyExtractor, GroupFunction, HashFunction, UseLocationDetection >, ReduceNode< ValueType, KeyExtractor, ReduceFunction, ReduceConfig, KeyHashFunction, KeyEqualFunction, VolatileKey, UseDuplicateDetection >, MergeNode< ValueType, Comparator, kNumInputs >, ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >, WriteLinesNode< ValueType >, GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >, BaseWindowNode< ValueType, Input, WindowFunction, PartialWindowFunction >, WriteBinaryNode< ValueType >, ZipWithIndexNode< ValueType, ZipFunction >, AllGatherNode< ValueType >, PrefixSumNode< ValueType, SumFunction, Inclusive >, CollapseNode< ValueType >, CacheNode< ValueType >, GatherNode< ValueType >, WriteLinesOneNode< ValueType >, and RebalanceNode< ValueType >.

Definition at line 173 of file dia_base.hpp.

Friends And Related Function Documentation

◆ operator<<

std::ostream& operator<< ( std::ostream &  os,
const DIABase &  d 
)
friend

make ostream-able.

Definition at line 449 of file dia_base.cpp.

Member Data Documentation

◆ consume_counter_

size_t consume_counter_ = 1
protected

Consumption counter: when it reaches zero, PushData() is called with consume = true

Definition at line 318 of file dia_base.hpp.

◆ context_

Context& context_
protected

associated Context

Definition at line 293 of file dia_base.hpp.

Referenced by JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::AddEqualKeysToVec(), ConcatNode< ValueType >::ConcatNode(), ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::DoScatter(), HyperLogLogNode< p, ValueType >::Execute(), DistributeNode< ValueType >::Execute(), SizeNode< ValueType >::Execute(), AllReduceNode< ValueType, ReduceFunction >::Execute(), RebalanceNode< ValueType >::Execute(), WriteLinesOneNode< ValueType >::Execute(), SampleNode< ValueType >::Execute(), PrefixSumNode< ValueType, SumFunction, Inclusive >::Execute(), ZipWithIndexNode< ValueType, ZipFunction >::Execute(), OverlapWindowNode< ValueType, Input, WindowFunction, PartialWindowFunction >::Execute(), ConcatNode< ValueType >::Execute(), SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >::Execute(), DisjointWindowNode< ValueType, Input, WindowFunction, PartialWindowFunction >::Execute(), SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >::FindAndSendSplitters(), GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::FlushVectorToFile(), GroupByNode< ValueType, KeyExtractor, GroupFunction, HashFunction, UseLocationDetection >::FlushVectorToFile(), GatherNode< ValueType >::GatherNode(), MergeNode< ValueType, Comparator, kNumInputs >::GetGlobalRanks(), GroupByNode< ValueType, KeyExtractor, GroupFunction, HashFunction, UseLocationDetection >::GroupByNode(), GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::GroupToIndexNode(), ReadLinesNode::InputLineIteratorCompressed::InputLineIteratorCompressed(), ReadLinesNode::InputLineIteratorUncompressed::InputLineIteratorUncompressed(), ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::MainOp(), MergeNode< ValueType, Comparator, kNumInputs >::MainOp(), SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >::MainOp(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::MakePuller(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::MergeFiles(), MergeNode< ValueType, Comparator, kNumInputs >::MergeNode(), ReadBinaryNode< ValueType >::VfsFileBlockSource::NextBlock(), CollapseNode< ValueType >::OnPreOpFile(), SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >::OnPreOpFile(), WriteBinaryNode< ValueType >::OpenNextFile(), SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >::PartialMultiwayMerge(), WriteLinesNode< ValueType >::PreOp(), GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::PreOp(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::PreOp1(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::PreOp2(), EqualToDIANode< ValueType >::PushData(), GenerateNode< ValueType, GenerateFunction >::PushData(), ZipWithIndexNode< ValueType, ZipFunction >::PushData(), ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::PushData(), MergeNode< ValueType, Comparator, kNumInputs >::PushData(), OverlapWindowNode< ValueType, Input, WindowFunction, PartialWindowFunction >::PushData(), GroupByNode< ValueType, KeyExtractor, GroupFunction, HashFunction, UseLocationDetection >::PushData(), SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >::PushData(), ReadBinaryNode< ValueType >::PushData(), DisjointWindowNode< ValueType, Input, WindowFunction, PartialWindowFunction >::PushData(), ReadBinaryNode< ValueType >::ReadBinaryNode(), SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >::ReceiveItems(), MergeNode< ValueType, Comparator, kNumInputs >::SelectPivots(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::SortAndWriteToFile(), SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >::SortAndWriteToFile(), SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >::StopPreOp(), WriteLinesNode< ValueType >::WriteLinesNode(), and ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::ZipNode().

◆ dia_id_

const size_t dia_id_
protected

DIA serial id.

Definition at line 296 of file dia_base.hpp.

◆ kNeverConsume

constexpr size_t kNeverConsume = static_cast<size_t>(-1)
static

Never full consume.

Definition at line 324 of file dia_base.hpp.

Referenced by CollapseNode< ValueType >::consume_counter(), UnionNode< ValueType >::consume_counter(), DIA< BfsNode >::KeepForever(), and DIANode< StackInput >::RunPushData().

◆ label_

const char* const label_
protected

DOp node static label.

Definition at line 299 of file dia_base.hpp.

◆ logger_

common::JsonLogger logger_
Initial value:
{
&context_.logger_, "dia_id", dia_id(), "label", label()
}

Definition at line 329 of file dia_base.hpp.

Referenced by WriteLinesOneNode< ValueType >::Execute(), SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >::MainOp(), ReadBinaryNode< ValueType >::PushData(), SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >::SortAndWriteToFile(), WriteBinaryNode< ValueType >::StopPreOp(), WriteLinesNode< ValueType >::StopPreOp(), and ReadLinesNode::InputLineIterator::~InputLineIterator().

◆ mem_limit_

DIAMemUse mem_limit_ = 0
protected

Amount of memory the current execution stage of the DIA implementation is allowed to use.

Definition at line 314 of file dia_base.hpp.

Referenced by JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::JoinCapacity(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::MainOp(), ReduceNode< ValueType, KeyExtractor, ReduceFunction, ReduceConfig, KeyHashFunction, KeyEqualFunction, VolatileKey, UseDuplicateDetection >::PushData(), ReduceToIndexNode< ValueType, KeyExtractor, ReduceFunction, ReduceConfig, VolatileKey, SkipPreReducePhase >::PushData(), SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >::ReceiveItems(), GroupByNode< ValueType, KeyExtractor, GroupFunction, HashFunction, UseLocationDetection >::StartPreOp(), ReduceNode< ValueType, KeyExtractor, ReduceFunction, ReduceConfig, KeyHashFunction, KeyEqualFunction, VolatileKey, UseDuplicateDetection >::StartPreOp(), ReduceToIndexNode< ValueType, KeyExtractor, ReduceFunction, ReduceConfig, VolatileKey, SkipPreReducePhase >::StartPreOp(), and JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::StartPreOp().

◆ parents_

std::vector<DIABasePtr> parents_
protected

Parents of this DIABase.

Definition at line 310 of file dia_base.hpp.

Referenced by CollapseNode< ValueType >::consume_counter(), UnionNode< ValueType >::consume_counter(), CollapseNode< ValueType >::DecConsumeCounter(), UnionNode< ValueType >::DecConsumeCounter(), CollapseNode< ValueType >::IncConsumeCounter(), UnionNode< ValueType >::IncConsumeCounter(), CollapseNode< ValueType >::SetConsumeCounter(), and UnionNode< ValueType >::SetConsumeCounter().

◆ state_

DIAState state_ = DIAState::NEW
protected

State of the DIANode. State is NEW on creation.

Definition at line 307 of file dia_base.hpp.

The documentation for this class was generated from the following files: