Thrill  0.1
DIA< ValueType_, Stack_ > Class Template Reference

Detailed Description

template<typename ValueType_, typename Stack_ = tlx::FunctionStack<ValueType_>>
class thrill::api::DIA< ValueType_, Stack_ >

DIA is the interface between the user and the Thrill framework.

A DIA can be imagined as an immutable array, even though the data does not need to be which represents the state after the previous DOp or Action. Additionally, a DIA stores the local lambda function chain of type Stack, which can transform elements of the DIANode to elements of this DIA. DOps/Actions create a DIA and a new DIANode, to which the DIA links to. LOps only create a new DIA, which link to the previous DIANode.

Template Parameters
ValueTypeType of elements currently in this DIA.
StackType of the function chain.
Examples:
examples/tutorial/k-means_step1.cpp, examples/tutorial/k-means_step4.cpp, examples/tutorial/k-means_step5.cpp, and examples/tutorial/k-means_step6.cpp.

Definition at line 141 of file dia.hpp.

+ Inheritance diagram for DIA< ValueType_, Stack_ >:
+ Collaboration diagram for DIA< ValueType_, Stack_ >:

#include <dia.hpp>

Public Types

using DIANodePtr = tlx::CountingPtr< DIANode< StackInput > >
 
using Stack = Stack_
 Type of this function stack. More...
 
using StackInput = typename Stack::Input
 
using ValueType = ValueType_
 

Public Member Functions

 DIA ()=default
 default-constructor: invalid DIA More...
 
 DIA (const DIANodePtr &node, const Stack &stack, size_t dia_id, const char *label)
 Constructor of a new DIA with a pointer to a DIANode and a function chain from the DIANode to this DIA. More...
 
 DIA (DIANodePtr &&node, const Stack &stack, size_t dia_id, const char *label)
 Constructor of a new DIA supporting move semantics of nodes. More...
 
 DIA (DIANodePtr &&node)
 Constructor of a new DIA with a real backing DIABase. More...
 
template<typename AnyStack >
 DIA (const DIA< ValueType, AnyStack > &rhs)
 Copy-Constructor of a DIA with empty function chain from a DIA with a non-empty chain. More...
 
void AssertValid () const
 Assert that the DIA is valid. More...
 
const DIADispose () const
 Dispose of the DIANode's data. More...
 
const DIAExecute () const
 Execute DIA's scope and parents such that this (Action)Node is Executed. More...
 
template<typename ValueOut , bool LocationDetectionValue, typename KeyExtractor , typename GroupFunction , typename HashFunction >
auto GroupByKey (const LocationDetectionFlag< LocationDetectionValue > &, const KeyExtractor &key_extractor, const GroupFunction &groupby_function, const HashFunction &hash_function) const
 
template<typename ValueOut , typename KeyExtractor, typename GroupFunction , typename HashFunction >
auto GroupByKey (const KeyExtractor &key_extractor, const GroupFunction &groupby_function, const HashFunction &hash_function) const
 
template<typename ValueOut , typename KeyExtractor, typename GroupFunction >
auto GroupByKey (const KeyExtractor &key_extractor, const GroupFunction &groupby_function) const
 
template<typename ValueOut , typename KeyExtractor , typename GroupFunction >
auto GroupToIndex (const KeyExtractor &key_extractor, const GroupFunction &groupby_function, const size_t result_size, const ValueOut &neutral_element) const
 
bool IsValid () const
 Return whether the DIA is valid. More...
 
const DIAKeep (size_t increase=1) const
 Mark the referenced DIANode for keeping, which makes children not consume the data when executing. More...
 
const DIAKeepForever () const
 Mark the referenced DIANode for keeping forever, which makes children not consume the data when executing. More...
 
Const Accessors
const DIANodePtrnode () const
 Returns a pointer to the according DIANode. More...
 
size_t node_refcount () const
 Returns the number of references to the according DIANode. More...
 
const Stackstack () const
 Returns the stored function chain. More...
 
Contextcontext () const
 Return context_ of DIANode, e.g. for creating new LOps and DOps. More...
 
Contextctx () const
 Return context_ of DIANode, e.g. for creating new LOps and DOps. More...
 
size_t id () const
 Returns id_. More...
 
const char * label () const
 Returns label_. More...
 
Local Operations (LOps)
template<typename MapFunction >
auto Map (const MapFunction &map_function) const
 Map applies map_function : \( A \to B \) to each item of a DIA and delivers a new DIA contains the returned values, which may be of a different type. More...
 
template<typename FilterFunction >
auto Filter (const FilterFunction &filter_function) const
 Each item of a DIA is tested using filter_function : \( A \to \textrm{bool} \) to determine whether it is copied into the output DIA or excluded. More...
 
template<typename ResultType = ValueType, typename FlatmapFunction >
auto FlatMap (const FlatmapFunction &flatmap_function) const
 Each item of a DIA is expanded by the flatmap_function : \( A \to \textrm{array}(B) \) to zero or more items of different type, which are concatenated in the resulting DIA. More...
 
auto BernoulliSample (double p) const
 Each item of a DIA is copied into the output DIA with success probability p (an independent Bernoulli trial). More...
 
template<typename SecondDIA >
auto Union (const SecondDIA &second_dia) const
 Union is a LOp, which creates the union of all items from any number of DIAs as a single DIA, where the items are in an arbitrary order. More...
 
Actions
size_t Size () const
 Computes the total size of all elements across all workers. More...
 
Future< size_t > SizeFuture () const
 Lazily computes the total size of all elements across all workers. More...
 
std::vector< ValueTypeAllGather () const
 Returns the whole DIA in an std::vector on each worker. More...
 
void AllGather (std::vector< ValueType > *out_vector) const
 AllGather is an Action, which returns the whole DIA in an std::vector on each worker. This is only for testing purposes and should not be used on large datasets. More...
 
Future< std::vector< ValueType > > AllGatherFuture () const
 Returns the whole DIA in an std::vector on each worker. More...
 
void Print (const std::string &name=std::string()) const
 Print is an Action, which collects all data of the DIA at the worker 0 and prints using ostream serialization. More...
 
void Print (const std::string &name, std::ostream &out) const
 Print is an Action, which collects all data of the DIA at the worker 0 and prints using ostream serialization. More...
 
std::vector< ValueTypeGather (size_t target_id=0) const
 Gather is an Action, which collects all data of the DIA into a vector at the given worker. More...
 
void Gather (size_t target_id, std::vector< ValueType > *out_vector) const
 Gather is an Action, which collects all data of the DIA into a vector at the given worker. More...
 
auto Sample (size_t sample_size) const
 Select up to sample_size items uniformly at random and return a new DIA<T>. More...
 
template<typename ReduceFunction >
ValueType AllReduce (const ReduceFunction &reduce_function) const
 AllReduce is an Action, which computes the reduction sum of all elements globally and delivers the same value on all workers. More...
 
template<typename ReduceFunction >
ValueType AllReduce (const ReduceFunction &reduce_function, const ValueType &initial_value) const
 AllReduce is an Action, which computes the reduction sum of all elements globally and delivers the same value on all workers. More...
 
template<typename ReduceFunction >
Future< ValueTypeAllReduceFuture (const ReduceFunction &reduce_function) const
 AllReduce is an ActionFuture, which computes the reduction sum of all elements globally and delivers the same value on all workers. More...
 
template<typename ReduceFunction >
Future< ValueTypeAllReduceFuture (const ReduceFunction &reduce_function, const ValueType &initial_value) const
 AllReduce is an ActionFuture, which computes the reduction sum of all elements globally and delivers the same value on all workers. More...
 
template<typename SumFunction = std::plus<ValueType>>
ValueType Sum (const SumFunction &sum_function=SumFunction()) const
 Sum is an Action, which computes the sum of all elements globally. More...
 
template<typename SumFunction = std::plus<ValueType>>
ValueType Sum (const SumFunction &sum_function, const ValueType &initial_value) const
 Sum is an Action, which computes the sum of all elements globally. More...
 
template<typename SumFunction = std::plus<ValueType>>
Future< ValueTypeSumFuture (const SumFunction &sum_function=SumFunction(), const ValueType &initial_value=ValueType()) const
 Sum is an ActionFuture, which computes the sum of all elements globally. More...
 
ValueType Min () const
 Min is an Action, which computes the minimum of all elements globally. More...
 
ValueType Min (const ValueType &initial_value) const
 Min is an Action, which computes the minimum of all elements globally. More...
 
Future< ValueTypeMinFuture () const
 Min is an ActionFuture, which computes the minimum of all elements globally. More...
 
Future< ValueTypeMinFuture (const ValueType &initial_value) const
 Min is an ActionFuture, which computes the minimum of all elements globally. More...
 
ValueType Max () const
 Max is an Action, which computes the maximum of all elements globally. More...
 
ValueType Max (const ValueType &initial_value) const
 Max is an Action, which computes the maximum of all elements globally. More...
 
Future< ValueTypeMaxFuture () const
 Max is an ActionFuture, which computes the maximum of all elements globally. More...
 
Future< ValueTypeMaxFuture (const ValueType &initial_value) const
 Max is an ActionFuture, which computes the maximum of all elements globally. More...
 
template<size_t p>
double HyperLogLog () const
 Compute the approximate number of distinct elements in the DIA. More...
 
void WriteLinesOne (const std::string &filepath) const
 WriteLinesOne is an Action, which writes std::strings to a single output file. More...
 
Future< void > WriteLinesOneFuture (const std::string &filepath) const
 WriteLinesOne is an ActionFuture, which writes std::strings to a single output file. More...
 
void WriteLines (const std::string &filepath, size_t target_file_size=128 *1024 *1024) const
 WriteLines is an Action, which writes std::strings to multiple output files. More...
 
Future< void > WriteLinesFuture (const std::string &filepath, size_t target_file_size=128 *1024 *1024) const
 WriteLines is an ActionFuture, which writes std::strings to multiple output files. More...
 
void WriteBinary (const std::string &filepath, size_t max_file_size=128 *1024 *1024) const
 WriteBinary is a function, which writes a DIA to many files per worker. More...
 
Future< void > WriteBinaryFuture (const std::string &filepath, size_t max_file_size=128 *1024 *1024) const
 WriteBinary is a function, which writes a DIA to many files per worker. More...
 
Distributed Operations (DOps)

This list of DOps are methods of the main DIA class and called as A.Method(params). Methods combining two or more DIAs are available as free functions.

template<typename KeyExtractor , typename ReduceFunction , typename ReduceConfig = class DefaultReduceConfig>
auto ReduceByKey (const KeyExtractor &key_extractor, const ReduceFunction &reduce_function, const ReduceConfig &reduce_config=ReduceConfig()) const
 ReduceByKey is a DOp, which groups elements of the DIA with the key_extractor and reduces each key-bucket to a single element using the associative reduce_function. More...
 
template<typename KeyExtractor , typename ReduceFunction , typename ReduceConfig , typename KeyHashFunction >
auto ReduceByKey (const KeyExtractor &key_extractor, const ReduceFunction &reduce_function, const ReduceConfig &reduce_config, const KeyHashFunction &key_hash_function) const
 ReduceByKey is a DOp, which groups elements of the DIA with the key_extractor and reduces each key-bucket to a single element using the associative reduce_function. More...
 
template<typename KeyExtractor , typename ReduceFunction , typename ReduceConfig , typename KeyHashFunction , typename KeyEqualFunction >
auto ReduceByKey (const KeyExtractor &key_extractor, const ReduceFunction &reduce_function, const ReduceConfig &reduce_config, const KeyHashFunction &key_hash_function, const KeyEqualFunction &key_equal_function) const
 ReduceByKey is a DOp, which groups elements of the DIA with the key_extractor and reduces each key-bucket to a single element using the associative reduce_function. More...
 
template<bool VolatileKeyValue, typename KeyExtractor , typename ReduceFunction , typename ReduceConfig = class DefaultReduceConfig, typename KeyHashFunction = std::hash<typename FunctionTraits<KeyExtractor>::result_type>, typename KeyEqualFunction = std::equal_to<typename FunctionTraits<KeyExtractor>::result_type>>
auto ReduceByKey (const VolatileKeyFlag< VolatileKeyValue > &, const KeyExtractor &key_extractor, const ReduceFunction &reduce_function, const ReduceConfig &reduce_config=ReduceConfig(), const KeyHashFunction &key_hash_function=KeyHashFunction(), const KeyEqualFunction &key_equal_function=KeyEqualFunction()) const
 ReduceByKey is a DOp, which groups elements of the DIA with the key_extractor and reduces each key-bucket to a single element using the associative reduce_function. More...
 
template<bool DuplicateDetectionValue, typename KeyExtractor , typename ReduceFunction , typename ReduceConfig = class DefaultReduceConfig, typename KeyHashFunction = std::hash<typename FunctionTraits<KeyExtractor>::result_type>, typename KeyEqualFunction = std::equal_to<typename FunctionTraits<KeyExtractor>::result_type>>
auto ReduceByKey (const DuplicateDetectionFlag< DuplicateDetectionValue > &, const KeyExtractor &key_extractor, const ReduceFunction &reduce_function, const ReduceConfig &reduce_config=ReduceConfig(), const KeyHashFunction &key_hash_function=KeyHashFunction(), const KeyEqualFunction &key_equal_function=KeyEqualFunction()) const
 ReduceByKey is a DOp, which groups elements of the DIA with the key_extractor and reduces each key-bucket to a single element using the associative reduce_function. More...
 
template<bool VolatileKeyValue, bool DuplicateDetectionValue, typename KeyExtractor , typename ReduceFunction , typename ReduceConfig = class DefaultReduceConfig, typename KeyHashFunction = std::hash<typename FunctionTraits<KeyExtractor>::result_type>, typename KeyEqualFunction = std::equal_to<typename FunctionTraits<KeyExtractor>::result_type>>
auto ReduceByKey (const VolatileKeyFlag< VolatileKeyValue > &, const DuplicateDetectionFlag< DuplicateDetectionValue > &, const KeyExtractor &key_extractor, const ReduceFunction &reduce_function, const ReduceConfig &reduce_config=ReduceConfig(), const KeyHashFunction &key_hash_function=KeyHashFunction(), const KeyEqualFunction &key_equal_function=KeyEqualFunction()) const
 ReduceByKey is a DOp, which groups elements of the DIA with the key_extractor and reduces each key-bucket to a single element using the associative reduce_function. More...
 
template<typename ReduceFunction , typename ReduceConfig = class DefaultReduceConfig>
auto ReducePair (const ReduceFunction &reduce_function, const ReduceConfig &reduce_config=ReduceConfig()) const
 ReducePair is a DOp, which groups key-value-pairs in the input DIA by their key and reduces each key-bucket to a single element using the associative reduce_function. More...
 
template<typename ReduceFunction , typename ReduceConfig , typename KeyHashFunction >
auto ReducePair (const ReduceFunction &reduce_function, const ReduceConfig &reduce_config, const KeyHashFunction &key_hash_function) const
 ReducePair is a DOp, which groups key-value-pairs in the input DIA by their key and reduces each key-bucket to a single element using the associative reduce_function. More...
 
template<typename ReduceFunction , typename ReduceConfig , typename KeyHashFunction , typename KeyEqualFunction >
auto ReducePair (const ReduceFunction &reduce_function, const ReduceConfig &reduce_config, const KeyHashFunction &key_hash_function, const KeyEqualFunction &key_equal_function) const
 ReducePair is a DOp, which groups key-value-pairs in the input DIA by their key and reduces each key-bucket to a single element using the associative reduce_function. More...
 
template<bool DuplicateDetectionValue, typename ReduceFunction , typename ReduceConfig = class DefaultReduceConfig, typename KeyHashFunction , typename KeyEqualFunction >
auto ReducePair (const DuplicateDetectionFlag< DuplicateDetectionValue > &, const ReduceFunction &reduce_function, const ReduceConfig &reduce_config=ReduceConfig(), const KeyHashFunction &key_hash_function=KeyHashFunction(), const KeyEqualFunction &key_equal_function=KeyEqualFunction()) const
 ReducePair is a DOp, which groups key-value-pairs in the input DIA by their key and reduces each key-bucket to a single element using the associative reduce_function. More...
 
template<typename KeyExtractor , typename ReduceFunction , typename ReduceConfig = class DefaultReduceToIndexConfig>
auto ReduceToIndex (const KeyExtractor &key_extractor, const ReduceFunction &reduce_function, size_t size, const ValueType &neutral_element=ValueType(), const ReduceConfig &reduce_config=ReduceConfig()) const
 ReduceToIndex is a DOp, which groups elements of the DIA with the key_extractor returning an unsigned integers and reduces each key-bucket to a single element using the associative reduce_function. More...
 
template<bool VolatileKeyValue, typename KeyExtractor , typename ReduceFunction , typename ReduceConfig = class DefaultReduceToIndexConfig>
auto ReduceToIndex (const VolatileKeyFlag< VolatileKeyValue > &, const KeyExtractor &key_extractor, const ReduceFunction &reduce_function, size_t size, const ValueType &neutral_element=ValueType(), const ReduceConfig &reduce_config=ReduceConfig()) const
 ReduceToIndex is a DOp, which groups elements of the DIA with the key_extractor returning an unsigned integers and reduces each key-bucket to a single element using the associative reduce_function. More...
 
template<typename KeyExtractor , typename ReduceFunction , typename ReduceConfig = class DefaultReduceToIndexConfig>
auto ReduceToIndex (const struct SkipPreReducePhaseTag &, const KeyExtractor &key_extractor, const ReduceFunction &reduce_function, size_t size, const ValueType &neutral_element=ValueType(), const ReduceConfig &reduce_config=ReduceConfig()) const
 ReduceToIndex is a DOp, which groups elements of the DIA with the key_extractor returning an unsigned integers and reduces each key-bucket to a single element using the associative reduce_function. More...
 
template<typename ValueOut , typename KeyExtractor , typename GroupByFunction >
auto GroupByKey (const KeyExtractor &key_extractor, const GroupByFunction &groupby_function) const
 GroupByKey is a DOp, which groups elements of the DIA by its key. More...
 
template<typename ValueOut , typename KeyExtractor , typename GroupByFunction , typename HashFunction >
auto GroupByKey (const KeyExtractor &key_extractor, const GroupByFunction &groupby_function, const HashFunction &hash_function) const
 GroupByKey is a DOp, which groups elements of the DIA by its key. More...
 
template<typename ValueOut , bool LocationDetectionTagValue, typename KeyExtractor , typename GroupByFunction , typename HashFunction = std::hash<typename FunctionTraits<KeyExtractor>::result_type>>
auto GroupByKey (const LocationDetectionFlag< LocationDetectionTagValue > &, const KeyExtractor &key_extractor, const GroupByFunction &groupby_function, const HashFunction &hash_function=HashFunction()) const
 GroupByKey is a DOp, which groups elements of the DIA by its key. More...
 
template<typename ValueOut , typename KeyExtractor , typename GroupByFunction >
auto GroupToIndex (const KeyExtractor &key_extractor, const GroupByFunction &groupby_function, const size_t size, const ValueOut &neutral_element=ValueOut()) const
 GroupBy is a DOp, which groups elements of the DIA by its key. More...
 
template<typename ZipFunction , typename SecondDIA >
auto Zip (const SecondDIA &second_dia, const ZipFunction &zip_function) const
 Zips two DIAs of equal size in style of functional programming by applying zip_function to the i-th elements of both input DIAs to form the i-th element of the output DIA. More...
 
template<typename ZipFunction , typename SecondDIA >
auto Zip (struct CutTag const &, const SecondDIA &second_dia, const ZipFunction &zip_function) const
 Zips two DIAs in style of functional programming by applying zip_function to the i-th elements of both input DIAs to form the i-th element of the output DIA. More...
 
template<typename ZipFunction , typename SecondDIA >
auto Zip (struct PadTag const &, const SecondDIA &second_dia, const ZipFunction &zip_function) const
 Zips two DIAs in style of functional programming by applying zip_function to the i-th elements of both input DIAs to form the i-th element of the output DIA. More...
 
template<typename ZipFunction , typename SecondDIA >
auto Zip (struct NoRebalanceTag const &, const SecondDIA &second_dia, const ZipFunction &zip_function) const
 Zips two DIAs in style of functional programming by applying zip_function to the i-th elements of both input DIAs to form the i-th element of the output DIA. More...
 
template<typename ZipFunction >
auto ZipWithIndex (const ZipFunction &zip_function) const
 Zips each item of a DIA with its zero-based array index. More...
 
template<typename CompareFunction = std::less<ValueType>>
auto Sort (const CompareFunction &compare_function=CompareFunction()) const
 Sort is a DOp, which sorts a given DIA according to the given compare_function. More...
 
template<typename CompareFunction , typename SortAlgorithm >
auto Sort (const CompareFunction &compare_function, const SortAlgorithm &sort_algorithm) const
 Sort is a DOp, which sorts a given DIA according to the given compare_function. More...
 
template<typename CompareFunction = std::less<ValueType>>
auto SortStable (const CompareFunction &compare_function=CompareFunction()) const
 SortStable is a DOp, which sorts a given DIA stably according to the given compare_function. More...
 
template<typename CompareFunction , typename SortAlgorithm >
auto SortStable (const CompareFunction &compare_function, const SortAlgorithm &sort_algorithm) const
 SortStable is a DOp, which sorts a given DIA stably according to the given compare_function. More...
 
template<typename Comparator = std::less<ValueType>, typename SecondDIA >
auto Merge (const SecondDIA &second_dia, const Comparator &comparator=Comparator()) const
 Merge is a DOp, which merges two sorted DIAs to a single sorted DIA. More...
 
template<typename SumFunction = std::plus<ValueType>>
auto PrefixSum (const SumFunction &sum_function=SumFunction(), const ValueType &initial_element=ValueType()) const
 PrefixSum is a DOp, which computes the (inclusive) prefix sum of all elements. More...
 
template<typename SumFunction = std::plus<ValueType>>
auto ExPrefixSum (const SumFunction &sum_function=SumFunction(), const ValueType &initial_element=ValueType()) const
 ExPrefixSum is a DOp, which computes the exclusive prefix sum of all elements. More...
 
template<typename WindowFunction >
auto Window (size_t window_size, const WindowFunction &window_function=WindowFunction()) const
 Window is a DOp, which applies a window function to every k consecutive items in a DIA. More...
 
template<typename WindowFunction , typename PartialWindowFunction >
auto Window (size_t window_size, const WindowFunction &window_function, const PartialWindowFunction &partial_window_function) const
 Window is a DOp, which applies a window function to every k consecutive items in a DIA. More...
 
template<typename WindowFunction >
auto Window (struct DisjointTag const &, size_t window_size, const WindowFunction &window_function) const
 Window is a DOp, which applies a window function to every k consecutive items in a DIA. More...
 
template<typename ValueOut , typename WindowFunction >
auto FlatWindow (size_t window_size, const WindowFunction &window_function=WindowFunction()) const
 FlatWindow is a DOp, which applies a window function to every k consecutive items in a DIA. More...
 
template<typename ValueOut , typename WindowFunction , typename PartialWindowFunction >
auto FlatWindow (size_t window_size, const WindowFunction &window_function, const PartialWindowFunction &partial_window_function) const
 FlatWindow is a DOp, which applies a window function to every k consecutive items in a DIA. More...
 
template<typename ValueOut , typename WindowFunction >
auto FlatWindow (struct DisjointTag const &, size_t window_size, const WindowFunction &window_function) const
 FlatWindow is a DOp, which applies a window function to every k consecutive items in a DIA. More...
 
template<typename SecondDIA >
auto Concat (const SecondDIA &second_dia) const
 Concat is a DOp, which concatenates any number of DIAs to a single DIA. More...
 
auto Rebalance () const
 Rebalance is a DOp, which rebalances a single DIA among all workers; in general, this operation is needed only if previous steps are known to create heavy imbalance (e.g. More...
 
DIA< ValueTypeCollapse () const
 Create a CollapseNode which is mainly used to collapse the LOp chain into a DIA<T> with an empty stack. More...
 
DIA< ValueTypeCache () const
 Create a CacheNode which contains all items of a DIA in calculated plain format. More...
 

Static Public Attributes

static constexpr bool stack_empty = Stack::empty
 boolean indication whether this FunctionStack is empty More...
 

Private Types

template<typename Function >
using FunctionTraits = common::FunctionTraits< Function >
 alias for convenience. More...
 

Private Member Functions

size_t next_dia_id ()
 deliver next DIA serial id More...
 

Private Attributes

size_t dia_id_ = 0
 DIA serial id for logging, matches DIANode::id_ for DOps. More...
 
const char * label_ = nullptr
 static DIA (LOp or DOp) node label string, may match DIANode::label_. More...
 
DIANodePtr node_
 
Stack stack_
 

Member Typedef Documentation

◆ DIANodePtr

type of pointer to the real node object implementation. This object has base item type StackInput which is transformed by the function stack lambdas further. But even pushing more lambdas does not change the stack input type.

Definition at line 169 of file dia.hpp.

◆ FunctionTraits

using FunctionTraits = common::FunctionTraits<Function>
private

alias for convenience.

Definition at line 147 of file dia.hpp.

◆ Stack

using Stack = Stack_

Type of this function stack.

Definition at line 155 of file dia.hpp.

◆ StackInput

using StackInput = typename Stack::Input

type of the items delivered by the DOp, and pushed down the function stack towards the next nodes. If the function stack contains LOps nodes, these may transform the type.

Definition at line 160 of file dia.hpp.

◆ ValueType

using ValueType = ValueType_

type of the items virtually in the DIA, which is the type emitted by the current LOp stack.

Definition at line 152 of file dia.hpp.

Constructor & Destructor Documentation

◆ DIA() [1/5]

DIA ( )
default

default-constructor: invalid DIA

◆ DIA() [2/5]

DIA ( const DIANodePtr node,
const Stack stack,
size_t  dia_id,
const char *  label 
)
inline

Constructor of a new DIA with a pointer to a DIANode and a function chain from the DIANode to this DIA.

Parameters
nodePointer to the last DIANode, DOps and Actions create a new DIANode, LOps link to the DIANode of the previous DIA.
stackFunction stack consisting of functions between last DIANode and this DIA.
dia_idSerial id of DIA, which includes LOps
labelstatic string label of DIA.

Definition at line 194 of file dia.hpp.

◆ DIA() [3/5]

DIA ( DIANodePtr &&  node,
const Stack stack,
size_t  dia_id,
const char *  label 
)
inline

Constructor of a new DIA supporting move semantics of nodes.

Parameters
nodePointer to the last DIANode, DOps and Actions create a new DIANode, LOps link to the DIANode of the previous DIA.
stackFunction stack consisting of functions between last DIANode and this DIA.
dia_idSerial id of DIA, which includes LOps
labelstatic string label of DIA.

Definition at line 212 of file dia.hpp.

◆ DIA() [4/5]

DIA ( DIANodePtr &&  node)
inlineexplicit

Constructor of a new DIA with a real backing DIABase.

Parameters
nodePointer to the last DIANode, DOps and Actions create a new DIANode, LOps link to the DIANode of the previous DIA.

Definition at line 223 of file dia.hpp.

◆ DIA() [5/5]

DIA ( const DIA< ValueType, AnyStack > &  rhs)

Copy-Constructor of a DIA with empty function chain from a DIA with a non-empty chain.

The functionality of the chain is stored in a newly created LOpNode. The current DIA than points to this LOpNode. This is needed to support assignment operations between DIA's.

Member Function Documentation

◆ AssertValid()

void AssertValid ( ) const
inline

Assert that the DIA is valid.

Definition at line 178 of file dia.hpp.

Referenced by thrill::api::Zip(), and thrill::api::ZipWindow().

◆ context()

Context& context ( ) const
inline

Return context_ of DIANode, e.g. for creating new LOps and DOps.

Definition at line 273 of file dia.hpp.

Referenced by examples::page_rank::PageRank(), examples::page_rank::PageRankJoin(), examples::select::PickPivots(), and examples::select::Select().

◆ ctx()

Context& ctx ( ) const
inline

Return context_ of DIANode, e.g. for creating new LOps and DOps.

Definition at line 279 of file dia.hpp.

◆ Dispose()

const DIA& Dispose ( ) const
inline

Dispose of the DIANode's data.

Definition at line 295 of file dia.hpp.

◆ Execute()

const DIA& Execute ( ) const
inline

Execute DIA's scope and parents such that this (Action)Node is Executed.

This does not create a new DIA, but returns the existing one.

Definition at line 335 of file dia.hpp.

◆ GroupByKey() [1/3]

auto GroupByKey ( const LocationDetectionFlag< LocationDetectionValue > &  ,
const KeyExtractor &  key_extractor,
const GroupFunction &  groupby_function,
const HashFunction &  hash_function 
) const

Definition at line 384 of file group_by_key.hpp.

◆ GroupByKey() [2/3]

auto GroupByKey ( const KeyExtractor &  key_extractor,
const GroupFunction &  groupby_function,
const HashFunction &  hash_function 
) const

Definition at line 410 of file group_by_key.hpp.

◆ GroupByKey() [3/3]

auto GroupByKey ( const KeyExtractor &  key_extractor,
const GroupFunction &  groupby_function 
) const

Definition at line 421 of file group_by_key.hpp.

◆ GroupToIndex()

auto GroupToIndex ( const KeyExtractor &  key_extractor,
const GroupFunction &  groupby_function,
const size_t  result_size,
const ValueOut &  neutral_element 
) const

Definition at line 261 of file group_to_index.hpp.

◆ id()

size_t id ( ) const
inline

Returns id_.

Definition at line 285 of file dia.hpp.

◆ IsValid()

bool IsValid ( ) const
inline

Return whether the DIA is valid.

Definition at line 175 of file dia.hpp.

Referenced by CollapseSwitch< ValueType, Stack >::MakeCollapse().

◆ Keep()

◆ KeepForever()

const DIA& KeepForever ( ) const
inline

Mark the referenced DIANode for keeping forever, which makes children not consume the data when executing.

This does not create a new DIA, but returns the existing one.

Definition at line 325 of file dia.hpp.

◆ label()

const char* label ( ) const
inline

Returns label_.

Definition at line 288 of file dia.hpp.

◆ next_dia_id()

size_t next_dia_id ( )
inlineprivate

deliver next DIA serial id

Definition at line 2041 of file dia.hpp.

◆ node()

const DIANodePtr& node ( ) const
inline

Returns a pointer to the according DIANode.

Definition at line 255 of file dia.hpp.

◆ node_refcount()

size_t node_refcount ( ) const
inline

Returns the number of references to the according DIANode.

Definition at line 261 of file dia.hpp.

◆ stack()

const Stack& stack ( ) const
inline

Returns the stored function chain.

Definition at line 267 of file dia.hpp.

Member Data Documentation

◆ dia_id_

size_t dia_id_ = 0
private

DIA serial id for logging, matches DIANode::id_ for DOps.

Definition at line 2035 of file dia.hpp.

◆ label_

const char* label_ = nullptr
private

static DIA (LOp or DOp) node label string, may match DIANode::label_.

Definition at line 2038 of file dia.hpp.

◆ node_

DIANodePtr node_
private

The DIANode which DIA points to. The node represents the latest DOp or Action performed previously.

Definition at line 2028 of file dia.hpp.

◆ stack_

Stack stack_
private

The local function chain, which stores the chained lambda function from the last DIANode to this DIA.

Definition at line 2032 of file dia.hpp.

◆ stack_empty

constexpr bool stack_empty = Stack::empty
static

boolean indication whether this FunctionStack is empty

Definition at line 163 of file dia.hpp.


The documentation for this class was generated from the following files: