Thrill  0.1
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs > Class Template Referencefinal

Detailed Description

template<typename ValueType, typename ZipFunction, bool Pad, bool UnequalCheck, bool NoRebalance, size_t kNumInputs>
class thrill::api::ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >

A DIANode which performs a Zip operation.

Zip combines two DIAs element-by-element. The ZipNode stores the zip_function UDF. The chainable LOps are stored in the Stack.

               ParentStack0 ParentStack1
                +--------+   +--------+
                |        |   |        |  A ParentStackX is called with
                |        |   |        |  ParentInputX, and must deliver
                |        |   |        |  a ZipArgX item.
              +-+--------+---+--------+-+
              | | PreOp0 |   | PreOp1 | |
              | +--------+   +--------+ |
   DIA<T> --> |           Zip           |
              |        +-------+        |
              |        |PostOp |        |
              +--------+-------+--------+
                       |       | New DIA<T>::stack_ is started
                       |       | with PostOp to chain next nodes.
                       +-------+
Template Parameters
ValueTypeOutput type of the Zip operation.
ParentDIA0Function stack, which contains the chained lambdas between the last and this DIANode for first input DIA.
ParentDIA1Function stack, which contains the chained lambdas between the last and this DIANode for second input DIA.
ZipFunctionType of the ZipFunction.

Definition at line 77 of file zip.hpp.

+ Inheritance diagram for ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >:
+ Collaboration diagram for ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >:

#include <zip.hpp>

Classes

class  ReaderNext
 Access CatReaders for different different parents. More...
 
class  RegisterParent
 Register Parent PreOp Hooks, instantiated and called for each Zip parent. More...
 

Public Member Functions

template<typename ParentDIA0 , typename... ParentDIAs>
 ZipNode (const ZipFunction &zip_function, const ZipArgsTuple &padding, const ParentDIA0 &parent0, const ParentDIAs &...parents)
 Constructor for a ZipNode. More...
 
template<size_t Index>
void DoScatter ()
 Scatter items from DIA "Index" to other workers if necessary. More...
 
void MainOp ()
 Receive elements from other workers. More...
 
- Public Member Functions inherited from DOpNode< ValueType >
 DOpNode (Context &ctx, const char *label, const std::initializer_list< size_t > &parent_ids, const std::initializer_list< DIABasePtr > &parents)
 Constructor for a DOpNode, which sets references to the parent nodes. More...
 
 DOpNode (Context &ctx, const char *label, std::vector< size_t > &&parent_ids, std::vector< DIABasePtr > &&parents)
 Constructor for a DOpNode, which sets references to the parent nodes. More...
 
- Public Member Functions inherited from DIANode< ValueType >
 DIANode (Context &ctx, const char *label, const std::initializer_list< size_t > &parent_ids, const std::initializer_list< DIABasePtr > &parents)
 Constructor for a DIANode, which sets references to the parent nodes. More...
 
 DIANode (Context &ctx, const char *label, std::vector< size_t > &&parent_ids, std::vector< DIABasePtr > &&parents)
 Constructor for a DIANode, which sets references to the parent nodes. More...
 
virtual void AddChild (DIABase *node, const Callback &callback=Callback(), size_t parent_index=0)
 Enables children to push their "folded" function chains to their parent. More...
 
std::vector< DIABase * > children () const override
 Returns the children of this DIABase. More...
 
void PushFile (data::File &file, bool consume) const
 
void PushItem (const ValueType &item) const
 Method for derived classes to Push a single item to all children. More...
 
void RemoveAllChildren () override
 
void RemoveChild (DIABase *node) override
 
void RunPushData () override
 
- Public Member Functions inherited from DIABase
 DIABase (Context &ctx, const char *label, const std::initializer_list< size_t > &parent_ids, const std::initializer_list< DIABasePtr > &parents)
 The constructor for a DIABase. More...
 
 DIABase (Context &ctx, const char *label, std::vector< size_t > &&parent_ids, std::vector< DIABasePtr > &&parents)
 The constructor for a DIABase. More...
 
 DIABase (const DIABase &)=delete
 non-copyable: delete copy-constructor More...
 
 DIABase (DIABase &&)=default
 move-constructor: default More...
 
virtual ~DIABase ()
 Virtual destructor for a DIABase. More...
 
virtual size_t consume_counter () const
 Returns consume_counter_. More...
 
Contextcontext ()
 Returns the api::Context of this DIABase. More...
 
virtual void DecConsumeCounter (size_t counter)
 
virtual bool ForwardDataOnly () const
 
const size_t & id () const
 return unique id() of DIANode subclass as stored by StatsNode More...
 
virtual void IncConsumeCounter (size_t counter)
 
const char * label () const
 return label() of DIANode subclass as stored by StatsNode More...
 
mem::Managermem_manager ()
 Return the Context's memory manager. More...
 
DIABaseoperator= (const DIABase &)=delete
 non-copyable: delete assignment operator More...
 
DIABaseoperator= (DIABase &&)=default
 move-assignment operator: default More...
 
std::vector< size_t > parent_ids () const
 Returns the parents of this DIABase. More...
 
const std::vector< DIABasePtr > & parents () const
 Returns the parents of this DIABase. More...
 
void RemoveParent (DIABase *p)
 Remove a parent. More...
 
virtual bool RequireParentPushData (size_t) const
 
void RunScope ()
 
void set_mem_limit (const DIAMemUse &mem_limit)
 
void set_state (const DIAState &state)
 
virtual void SetConsumeCounter (size_t counter)
 
DIAState state () const
 
virtual DIAMemUse PreOpMemUse ()
 Amount of RAM used by PreOp after StartPreOp() More...
 
virtual void StartPreOp (size_t)
 Virtual method for preparing start of PushData. More...
 
virtual bool OnPreOpFile (const data::File &, size_t)
 
virtual void StopPreOp (size_t)
 Virtual method for preparing end of PushData. More...
 
virtual DIAMemUse ExecuteMemUse ()
 Amount of RAM used by Execute() More...
 
virtual void Execute ()=0
 Virtual execution method. Triggers actual computation in sub-classes. More...
 
virtual DIAMemUse PushDataMemUse ()
 Amount of RAM used by PushData() More...
 
virtual void PushData (bool consume)=0
 Virtual method for pushing data. Triggers actual pushing in sub-classes. More...
 
virtual void Dispose ()
 Virtual clear method. Triggers actual disposing in sub-classes. More...
 
- Public Member Functions inherited from ReferenceCounter
 ReferenceCounter () noexcept
 new objects have zero reference count More...
 
 ReferenceCounter (const ReferenceCounter &) noexcept
 coping still creates a new object with zero reference count More...
 
 ~ReferenceCounter ()
 
bool dec_reference () const noexcept
 Call whenever resetting (i.e. More...
 
void inc_reference () const noexcept
 Call whenever setting a pointer to the object. More...
 
ReferenceCounteroperator= (const ReferenceCounter &) noexcept
 assignment operator, leaves pointers unchanged More...
 
size_t reference_count () const noexcept
 Return the number of references to this object (for debugging) More...
 
bool unique () const noexcept
 Test if the ReferenceCounter is referenced by only one CountingPtr. More...
 

Public Attributes

std::vector< data::Filefiles_
 Files for intermediate storage. More...
 
const ZipArgsTuple padding_
 padding for shorter DIAs More...
 
const std::array< bool,
kNumInputs > 
parent_stack_empty_
 Whether the parent stack is empty. More...
 
data::CatStreamPtr streams_ [kNumInputs]
 Array of inbound CatStreams. More...
 
data::File::Writer writers_ [kNumInputs]
 Writers to intermediate files. More...
 
Variables for Calculating Exchange
std::array< size_t, kNumInputs > size_prefixsum_
 exclusive prefix sum over the number of items in workers More...
 
size_t result_size_
 shortest size of Zipped inputs More...
 
- Public Attributes inherited from DIABase
common::JsonLogger logger_
 

Private Types

using Super = DOpNode< ValueType >
 
template<size_t Index>
using ZipArgN = typename common::FunctionTraits< ZipFunction >::template arg_plain< Index >
 
using ZipArgsTuple = typename common::FunctionTraits< ZipFunction >::args_tuple_plain
 

Static Private Attributes

static constexpr bool debug = false
 
static constexpr bool stats_enabled = false
 Set this variable to true to enable generation and output of stats. More...
 

Additional Inherited Members

- Public Types inherited from DOpNode< ValueType >
using Super = DIANode< ValueType >
 
- Public Types inherited from DIANode< ValueType >
using Callback = tlx::delegate< void(const ValueType &)>
 
- Public Types inherited from DIABase
using DIABasePtr = tlx::CountingPtr< DIABase >
 
- Static Public Attributes inherited from DIABase
static constexpr size_t kNeverConsume = static_cast<size_t>(-1)
 Never full consume. More...
 
- Protected Attributes inherited from DIANode< ValueType >
std::vector< Childchildren_
 Callback functions from the child nodes. More...
 
- Protected Attributes inherited from DIABase
Contextcontext_
 associated Context More...
 
const size_t id_
 DIA serial id. More...
 
const char *const label_
 DOp node static label. More...
 
DIAState state_ = DIAState::NEW
 State of the DIANode. State is NEW on creation. More...
 
std::vector< DIABasePtrparents_
 Parents of this DIABase. More...
 
DIAMemUse mem_limit_ = 0
 
size_t consume_counter_ = 1
 

Member Typedef Documentation

using Super = DOpNode<ValueType>
private

Definition at line 84 of file zip.hpp.

using ZipArgN = typename common::FunctionTraits<ZipFunction>::template arg_plain<Index>
private

Definition at line 89 of file zip.hpp.

using ZipArgsTuple = typename common::FunctionTraits<ZipFunction>::args_tuple_plain
private

Definition at line 91 of file zip.hpp.

Constructor & Destructor Documentation

ZipNode ( const ZipFunction &  zip_function,
const ZipArgsTuple padding,
const ParentDIA0 &  parent0,
const ParentDIAs &...  parents 
)
inline

Constructor for a ZipNode.

Definition at line 98 of file zip.hpp.

Member Function Documentation

void DoScatter ( )
inline

Member Data Documentation

constexpr bool debug = false
staticprivate

Definition at line 79 of file zip.hpp.

std::vector<data::File> files_

Files for intermediate storage.

Definition at line 205 of file zip.hpp.

const std::array<bool, kNumInputs> parent_stack_empty_

Whether the parent stack is empty.

Definition at line 202 of file zip.hpp.

size_t result_size_

shortest size of Zipped inputs

Definition at line 220 of file zip.hpp.

Referenced by ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::DoScatter().

std::array<size_t, kNumInputs> size_prefixsum_

exclusive prefix sum over the number of items in workers

Definition at line 217 of file zip.hpp.

constexpr bool stats_enabled = false
staticprivate

Set this variable to true to enable generation and output of stats.

Definition at line 82 of file zip.hpp.

Referenced by ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::MainOp().

data::CatStreamPtr streams_[kNumInputs]

Array of inbound CatStreams.

Definition at line 211 of file zip.hpp.

data::File::Writer writers_[kNumInputs]

Writers to intermediate files.

Definition at line 208 of file zip.hpp.

Referenced by ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::RegisterParent::operator()().


The documentation for this class was generated from the following file: