Thrill
0.1
|
Modules | |
Free Operation Functions | |
Distributed Operations (DOps) | |
This list of DOps are methods of the main DIA class and called as | |
template<typename KeyExtractor , typename ReduceFunction , typename ReduceConfig = class DefaultReduceConfig> | |
auto | ReduceByKey (const KeyExtractor &key_extractor, const ReduceFunction &reduce_function, const ReduceConfig &reduce_config=ReduceConfig()) const |
ReduceByKey is a DOp, which groups elements of the DIA with the key_extractor and reduces each key-bucket to a single element using the associative reduce_function. More... | |
template<typename KeyExtractor , typename ReduceFunction , typename ReduceConfig , typename KeyHashFunction > | |
auto | ReduceByKey (const KeyExtractor &key_extractor, const ReduceFunction &reduce_function, const ReduceConfig &reduce_config, const KeyHashFunction &key_hash_function) const |
ReduceByKey is a DOp, which groups elements of the DIA with the key_extractor and reduces each key-bucket to a single element using the associative reduce_function. More... | |
template<typename KeyExtractor , typename ReduceFunction , typename ReduceConfig , typename KeyHashFunction , typename KeyEqualFunction > | |
auto | ReduceByKey (const KeyExtractor &key_extractor, const ReduceFunction &reduce_function, const ReduceConfig &reduce_config, const KeyHashFunction &key_hash_function, const KeyEqualFunction &key_equal_function) const |
ReduceByKey is a DOp, which groups elements of the DIA with the key_extractor and reduces each key-bucket to a single element using the associative reduce_function. More... | |
template<bool VolatileKeyValue, typename KeyExtractor , typename ReduceFunction , typename ReduceConfig = class DefaultReduceConfig, typename KeyHashFunction = std::hash<typename FunctionTraits<KeyExtractor>::result_type>, typename KeyEqualFunction = std::equal_to<typename FunctionTraits<KeyExtractor>::result_type>> | |
auto | ReduceByKey (const VolatileKeyFlag< VolatileKeyValue > &, const KeyExtractor &key_extractor, const ReduceFunction &reduce_function, const ReduceConfig &reduce_config=ReduceConfig(), const KeyHashFunction &key_hash_function=KeyHashFunction(), const KeyEqualFunction &key_equal_function=KeyEqualFunction()) const |
ReduceByKey is a DOp, which groups elements of the DIA with the key_extractor and reduces each key-bucket to a single element using the associative reduce_function. More... | |
template<bool DuplicateDetectionValue, typename KeyExtractor , typename ReduceFunction , typename ReduceConfig = class DefaultReduceConfig, typename KeyHashFunction = std::hash<typename FunctionTraits<KeyExtractor>::result_type>, typename KeyEqualFunction = std::equal_to<typename FunctionTraits<KeyExtractor>::result_type>> | |
auto | ReduceByKey (const DuplicateDetectionFlag< DuplicateDetectionValue > &, const KeyExtractor &key_extractor, const ReduceFunction &reduce_function, const ReduceConfig &reduce_config=ReduceConfig(), const KeyHashFunction &key_hash_function=KeyHashFunction(), const KeyEqualFunction &key_equal_function=KeyEqualFunction()) const |
ReduceByKey is a DOp, which groups elements of the DIA with the key_extractor and reduces each key-bucket to a single element using the associative reduce_function. More... | |
template<bool VolatileKeyValue, bool DuplicateDetectionValue, typename KeyExtractor , typename ReduceFunction , typename ReduceConfig = class DefaultReduceConfig, typename KeyHashFunction = std::hash<typename FunctionTraits<KeyExtractor>::result_type>, typename KeyEqualFunction = std::equal_to<typename FunctionTraits<KeyExtractor>::result_type>> | |
auto | ReduceByKey (const VolatileKeyFlag< VolatileKeyValue > &, const DuplicateDetectionFlag< DuplicateDetectionValue > &, const KeyExtractor &key_extractor, const ReduceFunction &reduce_function, const ReduceConfig &reduce_config=ReduceConfig(), const KeyHashFunction &key_hash_function=KeyHashFunction(), const KeyEqualFunction &key_equal_function=KeyEqualFunction()) const |
ReduceByKey is a DOp, which groups elements of the DIA with the key_extractor and reduces each key-bucket to a single element using the associative reduce_function. More... | |
template<typename ReduceFunction , typename ReduceConfig = class DefaultReduceConfig> | |
auto | ReducePair (const ReduceFunction &reduce_function, const ReduceConfig &reduce_config=ReduceConfig()) const |
ReducePair is a DOp, which groups key-value-pairs in the input DIA by their key and reduces each key-bucket to a single element using the associative reduce_function. More... | |
template<typename ReduceFunction , typename ReduceConfig , typename KeyHashFunction > | |
auto | ReducePair (const ReduceFunction &reduce_function, const ReduceConfig &reduce_config, const KeyHashFunction &key_hash_function) const |
ReducePair is a DOp, which groups key-value-pairs in the input DIA by their key and reduces each key-bucket to a single element using the associative reduce_function. More... | |
template<typename ReduceFunction , typename ReduceConfig , typename KeyHashFunction , typename KeyEqualFunction > | |
auto | ReducePair (const ReduceFunction &reduce_function, const ReduceConfig &reduce_config, const KeyHashFunction &key_hash_function, const KeyEqualFunction &key_equal_function) const |
ReducePair is a DOp, which groups key-value-pairs in the input DIA by their key and reduces each key-bucket to a single element using the associative reduce_function. More... | |
template<bool DuplicateDetectionValue, typename ReduceFunction , typename ReduceConfig = class DefaultReduceConfig, typename KeyHashFunction , typename KeyEqualFunction > | |
auto | ReducePair (const DuplicateDetectionFlag< DuplicateDetectionValue > &, const ReduceFunction &reduce_function, const ReduceConfig &reduce_config=ReduceConfig(), const KeyHashFunction &key_hash_function=KeyHashFunction(), const KeyEqualFunction &key_equal_function=KeyEqualFunction()) const |
ReducePair is a DOp, which groups key-value-pairs in the input DIA by their key and reduces each key-bucket to a single element using the associative reduce_function. More... | |
template<typename KeyExtractor , typename ReduceFunction , typename ReduceConfig = class DefaultReduceToIndexConfig> | |
auto | ReduceToIndex (const KeyExtractor &key_extractor, const ReduceFunction &reduce_function, size_t size, const ValueType &neutral_element=ValueType(), const ReduceConfig &reduce_config=ReduceConfig()) const |
ReduceToIndex is a DOp, which groups elements of the DIA with the key_extractor returning an unsigned integers and reduces each key-bucket to a single element using the associative reduce_function. More... | |
template<bool VolatileKeyValue, typename KeyExtractor , typename ReduceFunction , typename ReduceConfig = class DefaultReduceToIndexConfig> | |
auto | ReduceToIndex (const VolatileKeyFlag< VolatileKeyValue > &, const KeyExtractor &key_extractor, const ReduceFunction &reduce_function, size_t size, const ValueType &neutral_element=ValueType(), const ReduceConfig &reduce_config=ReduceConfig()) const |
ReduceToIndex is a DOp, which groups elements of the DIA with the key_extractor returning an unsigned integers and reduces each key-bucket to a single element using the associative reduce_function. More... | |
template<typename KeyExtractor , typename ReduceFunction , typename ReduceConfig = class DefaultReduceToIndexConfig> | |
auto | ReduceToIndex (const struct SkipPreReducePhaseTag &, const KeyExtractor &key_extractor, const ReduceFunction &reduce_function, size_t size, const ValueType &neutral_element=ValueType(), const ReduceConfig &reduce_config=ReduceConfig()) const |
ReduceToIndex is a DOp, which groups elements of the DIA with the key_extractor returning an unsigned integers and reduces each key-bucket to a single element using the associative reduce_function. More... | |
template<typename ValueOut , typename KeyExtractor , typename GroupByFunction > | |
auto | GroupByKey (const KeyExtractor &key_extractor, const GroupByFunction &groupby_function) const |
GroupByKey is a DOp, which groups elements of the DIA by its key. More... | |
template<typename ValueOut , typename KeyExtractor , typename GroupByFunction , typename HashFunction > | |
auto | GroupByKey (const KeyExtractor &key_extractor, const GroupByFunction &groupby_function, const HashFunction &hash_function) const |
GroupByKey is a DOp, which groups elements of the DIA by its key. More... | |
template<typename ValueOut , bool LocationDetectionTagValue, typename KeyExtractor , typename GroupByFunction , typename HashFunction = std::hash<typename FunctionTraits<KeyExtractor>::result_type>> | |
auto | GroupByKey (const LocationDetectionFlag< LocationDetectionTagValue > &, const KeyExtractor &key_extractor, const GroupByFunction &groupby_function, const HashFunction &hash_function=HashFunction()) const |
GroupByKey is a DOp, which groups elements of the DIA by its key. More... | |
template<typename ValueOut , typename KeyExtractor , typename GroupByFunction > | |
auto | GroupToIndex (const KeyExtractor &key_extractor, const GroupByFunction &groupby_function, const size_t size, const ValueOut &neutral_element=ValueOut()) const |
GroupBy is a DOp, which groups elements of the DIA by its key. More... | |
template<typename ZipFunction , typename SecondDIA > | |
auto | Zip (const SecondDIA &second_dia, const ZipFunction &zip_function) const |
Zips two DIAs of equal size in style of functional programming by applying zip_function to the i-th elements of both input DIAs to form the i-th element of the output DIA. More... | |
template<typename ZipFunction , typename SecondDIA > | |
auto | Zip (struct CutTag const &, const SecondDIA &second_dia, const ZipFunction &zip_function) const |
Zips two DIAs in style of functional programming by applying zip_function to the i-th elements of both input DIAs to form the i-th element of the output DIA. More... | |
template<typename ZipFunction , typename SecondDIA > | |
auto | Zip (struct PadTag const &, const SecondDIA &second_dia, const ZipFunction &zip_function) const |
Zips two DIAs in style of functional programming by applying zip_function to the i-th elements of both input DIAs to form the i-th element of the output DIA. More... | |
template<typename ZipFunction , typename SecondDIA > | |
auto | Zip (struct NoRebalanceTag const &, const SecondDIA &second_dia, const ZipFunction &zip_function) const |
Zips two DIAs in style of functional programming by applying zip_function to the i-th elements of both input DIAs to form the i-th element of the output DIA. More... | |
template<typename ZipFunction > | |
auto | ZipWithIndex (const ZipFunction &zip_function) const |
Zips each item of a DIA with its zero-based array index. More... | |
template<typename CompareFunction = std::less<ValueType>> | |
auto | Sort (const CompareFunction &compare_function=CompareFunction()) const |
Sort is a DOp, which sorts a given DIA according to the given compare_function. More... | |
template<typename CompareFunction , typename SortAlgorithm > | |
auto | Sort (const CompareFunction &compare_function, const SortAlgorithm &sort_algorithm) const |
Sort is a DOp, which sorts a given DIA according to the given compare_function. More... | |
template<typename CompareFunction = std::less<ValueType>> | |
auto | SortStable (const CompareFunction &compare_function=CompareFunction()) const |
SortStable is a DOp, which sorts a given DIA stably according to the given compare_function. More... | |
template<typename CompareFunction , typename SortAlgorithm > | |
auto | SortStable (const CompareFunction &compare_function, const SortAlgorithm &sort_algorithm) const |
SortStable is a DOp, which sorts a given DIA stably according to the given compare_function. More... | |
template<typename Comparator = std::less<ValueType>, typename SecondDIA > | |
auto | Merge (const SecondDIA &second_dia, const Comparator &comparator=Comparator()) const |
Merge is a DOp, which merges two sorted DIAs to a single sorted DIA. More... | |
template<typename SumFunction = std::plus<ValueType>> | |
auto | PrefixSum (const SumFunction &sum_function=SumFunction(), const ValueType &initial_element=ValueType()) const |
PrefixSum is a DOp, which computes the (inclusive) prefix sum of all elements. More... | |
template<typename SumFunction = std::plus<ValueType>> | |
auto | ExPrefixSum (const SumFunction &sum_function=SumFunction(), const ValueType &initial_element=ValueType()) const |
ExPrefixSum is a DOp, which computes the exclusive prefix sum of all elements. More... | |
template<typename WindowFunction > | |
auto | Window (size_t window_size, const WindowFunction &window_function=WindowFunction()) const |
Window is a DOp, which applies a window function to every k consecutive items in a DIA. More... | |
template<typename WindowFunction , typename PartialWindowFunction > | |
auto | Window (size_t window_size, const WindowFunction &window_function, const PartialWindowFunction &partial_window_function) const |
Window is a DOp, which applies a window function to every k consecutive items in a DIA. More... | |
template<typename WindowFunction > | |
auto | Window (struct DisjointTag const &, size_t window_size, const WindowFunction &window_function) const |
Window is a DOp, which applies a window function to every k consecutive items in a DIA. More... | |
template<typename ValueOut , typename WindowFunction > | |
auto | FlatWindow (size_t window_size, const WindowFunction &window_function=WindowFunction()) const |
FlatWindow is a DOp, which applies a window function to every k consecutive items in a DIA. More... | |
template<typename ValueOut , typename WindowFunction , typename PartialWindowFunction > | |
auto | FlatWindow (size_t window_size, const WindowFunction &window_function, const PartialWindowFunction &partial_window_function) const |
FlatWindow is a DOp, which applies a window function to every k consecutive items in a DIA. More... | |
template<typename ValueOut , typename WindowFunction > | |
auto | FlatWindow (struct DisjointTag const &, size_t window_size, const WindowFunction &window_function) const |
FlatWindow is a DOp, which applies a window function to every k consecutive items in a DIA. More... | |
template<typename SecondDIA > | |
auto | Concat (const SecondDIA &second_dia) const |
Concat is a DOp, which concatenates any number of DIAs to a single DIA. More... | |
auto | Rebalance () const |
Rebalance is a DOp, which rebalances a single DIA among all workers; in general, this operation is needed only if previous steps are known to create heavy imbalance (e.g. More... | |
DIA< ValueType > | Collapse () const |
Create a CollapseNode which is mainly used to collapse the LOp chain into a DIA<T> with an empty stack. More... | |
DIA< ValueType > | Cache () const |
Create a CacheNode which contains all items of a DIA in calculated plain format. More... | |
Create a CollapseNode which is mainly used to collapse the LOp chain into a DIA<T> with an empty stack.
This is most often necessary for iterative algorithms, where a DIA<T> reference variable is updated in each iteration.
Definition at line 159 of file collapse.hpp.
Referenced by examples::suffix_sorting::ConstructWaveletTree(), and Process().
auto Concat | ( | const SecondDIA< ValueType_, Stack_ > & | second_dia | ) | const |
Concat is a DOp, which concatenates any number of DIAs to a single DIA.
All input DIAs must contain the same type, which is also the output DIA's type.
The concat operation balances all input data, so that each worker will have an equal number of elements when the concat completes.
Definition at line 388 of file concat.hpp.
auto ExPrefixSum | ( | const SumFunction & | sum_function = SumFunction() , |
const ValueType & | initial_element = ValueType() |
||
) | const |
ExPrefixSum is a DOp, which computes the exclusive prefix sum of all elements.
The sum function defines how two elements are combined to a single element.
sum_function | Sum function (any associative function). |
initial_element | Initial element of the sum function. |
Definition at line 24 of file ex_prefix_sum.hpp.
auto FlatWindow | ( | size_t | window_size, |
const WindowFunction & | window_function = WindowFunction() |
||
) | const |
FlatWindow is a DOp, which applies a window function to every k consecutive items in a DIA.
The window function is also given the index of the first item, and can output zero or more items via an Emitter.
window_size | the size of the delivered window. Signature: TODO(tb). |
window_function | Window function applied to each k item. |
Definition at line 271 of file window.hpp.
auto FlatWindow | ( | size_t | window_size, |
const WindowFunction & | window_function, | ||
const PartialWindowFunction & | partial_window_function | ||
) | const |
FlatWindow is a DOp, which applies a window function to every k consecutive items in a DIA.
The window function is also given the index of the first item, and can output zero or more items via an Emitter.
window_size | the size of the delivered window. Signature: TODO(tb). |
window_function | Window function applied to each k item. |
partial_window_function | Window function applied to less than k items. |
Definition at line 251 of file window.hpp.
auto FlatWindow | ( | struct DisjointTag const & | , |
size_t | window_size, | ||
const WindowFunction & | window_function | ||
) | const |
FlatWindow is a DOp, which applies a window function to every k consecutive items in a DIA.
The window function is also given the index of the first item, and can output zero or more items via an Emitter.
window_size | the size of the delivered window. Signature: TODO(tb). |
window_function | Window function applied to each k item. |
Definition at line 507 of file window.hpp.
auto GroupByKey | ( | const KeyExtractor & | key_extractor, |
const GroupByFunction & | groupby_function | ||
) | const |
GroupByKey is a DOp, which groups elements of the DIA by its key.
After having grouped all elements of one key, all elements of one key will be processed according to the GroupByFunction and returns an output Contrary to Reduce, GroupBy allows usage of functions that require all elements of one key at once as GroupByFunction will be applied after all elements with the same key have been grouped. However because of this reason, the communication overhead is also higher. If possible, usage of Reduce is therefore recommended.
As GroupBy is a DOp, it creates a new DIANode. The DIA returned by Reduce links to this newly created DIANode. The stack_ of the returned DIA consists of the PostOp of Reduce, as a reduced element can directly be chained to the following LOps.
KeyExtractor | Type of the key_extractor function. The key_extractor function is equal to a map function. |
key_extractor | Key extractor function, which maps each element to a key of possibly different type. |
GroupByFunction | Type of the groupby_function. This is a function taking an iterator for all elements of the same key as input. |
groupby_function | Reduce function, which defines how the key buckets are grouped and processed. input param: api::GroupByReader with functions HasNext() and Next() |
auto GroupByKey | ( | const KeyExtractor & | key_extractor, |
const GroupByFunction & | groupby_function, | ||
const HashFunction & | hash_function | ||
) | const |
GroupByKey is a DOp, which groups elements of the DIA by its key.
After having grouped all elements of one key, all elements of one key will be processed according to the GroupByFunction and returns an output Contrary to Reduce, GroupBy allows usage of functions that require all elements of one key at once as GroupByFunction will be applied after all elements with the same key have been grouped. However because of this reason, the communication overhead is also higher. If possible, usage of Reduce is therefore recommended.
As GroupBy is a DOp, it creates a new DIANode. The DIA returned by Reduce links to this newly created DIANode. The stack_ of the returned DIA consists of the PostOp of Reduce, as a reduced element can directly be chained to the following LOps.
KeyExtractor | Type of the key_extractor function. The key_extractor function is equal to a map function. |
key_extractor | Key extractor function, which maps each element to a key of possibly different type. |
GroupByFunction | Type of the groupby_function. This is a function taking an iterator for all elements of the same key as input. |
groupby_function | Reduce function, which defines how the key buckets are grouped and processed. input param: api::GroupByReader with functions HasNext() and Next() |
hash_function | Hash method for Keys |
auto GroupByKey | ( | const LocationDetectionFlag< LocationDetectionTagValue > & | , |
const KeyExtractor & | key_extractor, | ||
const GroupByFunction & | groupby_function, | ||
const HashFunction & | hash_function = HashFunction() |
||
) | const |
GroupByKey is a DOp, which groups elements of the DIA by its key.
After having grouped all elements of one key, all elements of one key will be processed according to the GroupByFunction and returns an output Contrary to Reduce, GroupBy allows usage of functions that require all elements of one key at once as GroupByFunction will be applied after all elements with the same key have been grouped. However because of this reason, the communication overhead is also higher. If possible, usage of Reduce is therefore recommended.
As GroupBy is a DOp, it creates a new DIANode. The DIA returned by Reduce links to this newly created DIANode. The stack_ of the returned DIA consists of the PostOp of Reduce, as a reduced element can directly be chained to the following LOps.
KeyExtractor | Type of the key_extractor function. The key_extractor function is equal to a map function. |
key_extractor | Key extractor function, which maps each element to a key of possibly different type. |
GroupByFunction | Type of the groupby_function. This is a function taking an iterator for all elements of the same key as input. |
groupby_function | Reduce function, which defines how the key buckets are grouped and processed. input param: api::GroupByReader with functions HasNext() and Next() |
hash_function | Hash method for Keys |
auto GroupToIndex | ( | const KeyExtractor & | key_extractor, |
const GroupByFunction & | groupby_function, | ||
const size_t | size, | ||
const ValueOut & | neutral_element = ValueOut() |
||
) | const |
GroupBy is a DOp, which groups elements of the DIA by its key.
After having grouped all elements of one key, all elements of one key will be processed according to the GroupByFunction and returns an output Contrary to Reduce, GroupBy allows usage of functions that require all elements of one key at once as GroupByFunction will be applied after all elements with the same key have been grouped. However because of this reason, the communication overhead is also higher. If possible, usage of Reduce is therefore recommended.
In contrast to GroupBy, GroupToIndex returns a DIA in a defined order, which has the reduced element with key i in position i. As GroupBy is a DOp, it creates a new DIANode. The DIA returned by Reduce links to this newly created DIANode. The stack_ of the returned DIA consists of the PostOp of Reduce, as a reduced element can directly be chained to the following LOps.
KeyExtractor | Type of the key_extractor function. The key_extractor function is equal to a map function. |
key_extractor | Key extractor function, which maps each element to a key of possibly different type. |
GroupByFunction | Type of the groupby_function. This is a function taking an iterator for all elements of the same key as input. |
groupby_function | Reduce function, which defines how the key buckets are grouped and processed. input param: api::GroupByReader with functions HasNext() and Next() |
size | Resulting DIA size. Consequently, the key_extractor function but always return < size for any element in the input DIA. |
neutral_element | Item value with which to start the reduction in each array cell. |
auto Merge | ( | const SecondDIA< ValueType_, Stack_ > & | second_dia, |
const Comparator & | comparator = Comparator() |
||
) | const |
Merge is a DOp, which merges two sorted DIAs to a single sorted DIA.
Both input DIAs must be used sorted conforming to the given comparator. The type of the output DIA will be the type of this DIA.
The merge operation balances all input data, so that each worker will have an equal number of elements when the merge completes.
auto PrefixSum | ( | const SumFunction & | sum_function = SumFunction() , |
const ValueType & | initial_element = ValueType() |
||
) | const |
PrefixSum is a DOp, which computes the (inclusive) prefix sum of all elements.
The sum function defines how two elements are combined to a single element.
sum_function | Sum function (any associative function). |
initial_element | Initial element of the sum function. |
Definition at line 134 of file prefix_sum.hpp.
Referenced by examples::suffix_sorting::PrefixDoublingDiscarding(), examples::suffix_sorting::PrefixDoublingSorting(), examples::suffix_sorting::PrefixDoublingWindow(), and examples::suffix_sorting::PrefixQuadrupling().
auto Rebalance | ( | ) | const |
Rebalance is a DOp, which rebalances a single DIA among all workers; in general, this operation is needed only if previous steps are known to create heavy imbalance (e.g.
like Filter()s which cut DIAs to ranges).
Definition at line 122 of file rebalance.hpp.
auto ReduceByKey | ( | const KeyExtractor & | key_extractor, |
const ReduceFunction & | reduce_function, | ||
const ReduceConfig & | reduce_config = ReduceConfig() |
||
) | const |
ReduceByKey is a DOp, which groups elements of the DIA with the key_extractor and reduces each key-bucket to a single element using the associative reduce_function.
The reduce_function defines how two elements can be reduced to a single element of equal type.
The key of the reduced element has to be equal to the keys of the input elements. Since ReduceBy is a DOp, it creates a new DIANode. The DIA returned by Reduce links to this newly created DIANode. The stack_ of the returned DIA consists of the PostOp of Reduce, as a reduced element can directly be chained to the following LOps.
key_extractor | Key extractor function, which maps each element to a key of possibly different type. |
ReduceFunction | Type of the reduce_function. This is a function reducing two elements of L's result type to a single element of equal type. |
reduce_function | Reduce function, which defines how the key buckets are reduced to a single element. This function is applied associative but not necessarily commutative. |
reduce_config | Reduce configuration. |
Definition at line 243 of file reduce_by_key.hpp.
Referenced by examples::word_count::WordCount().
auto ReduceByKey | ( | const KeyExtractor & | key_extractor, |
const ReduceFunction & | reduce_function, | ||
const ReduceConfig & | reduce_config, | ||
const KeyHashFunction & | key_hash_function | ||
) | const |
ReduceByKey is a DOp, which groups elements of the DIA with the key_extractor and reduces each key-bucket to a single element using the associative reduce_function.
The reduce_function defines how two elements can be reduced to a single element of equal type.
The key of the reduced element has to be equal to the keys of the input elements. Since ReduceBy is a DOp, it creates a new DIANode. The DIA returned by Reduce links to this newly created DIANode. The stack_ of the returned DIA consists of the PostOp of Reduce, as a reduced element can directly be chained to the following LOps.
key_extractor | Key extractor function, which maps each element to a key of possibly different type. |
ReduceFunction | Type of the reduce_function. This is a function reducing two elements of L's result type to a single element of equal type. |
reduce_function | Reduce function, which defines how the key buckets are reduced to a single element. This function is applied associative but not necessarily commutative. |
reduce_config | Reduce configuration. |
key_hash_function | Function to hash keys extracted by KeyExtractor. |
Definition at line 258 of file reduce_by_key.hpp.
auto ReduceByKey | ( | const KeyExtractor & | key_extractor, |
const ReduceFunction & | reduce_function, | ||
const ReduceConfig & | reduce_config, | ||
const KeyHashFunction & | key_hash_function, | ||
const KeyEqualFunction & | key_equal_function | ||
) | const |
ReduceByKey is a DOp, which groups elements of the DIA with the key_extractor and reduces each key-bucket to a single element using the associative reduce_function.
The reduce_function defines how two elements can be reduced to a single element of equal type.
The key of the reduced element has to be equal to the keys of the input elements. Since ReduceBy is a DOp, it creates a new DIANode. The DIA returned by Reduce links to this newly created DIANode. The stack_ of the returned DIA consists of the PostOp of Reduce, as a reduced element can directly be chained to the following LOps.
key_extractor | Key extractor function, which maps each element to a key of possibly different type. |
ReduceFunction | Type of the reduce_function. This is a function reducing two elements of L's result type to a single element of equal type. |
reduce_function | Reduce function, which defines how the key buckets are reduced to a single element. This function is applied associative but not necessarily commutative. |
reduce_config | Reduce configuration. |
key_hash_function | Function to hash keys extracted by KeyExtractor. |
key_equal_function | Function to compare keys in reduce hash tables. |
auto ReduceByKey | ( | const VolatileKeyFlag< VolatileKeyValue > & | volatile_key_flag, |
const KeyExtractor & | key_extractor, | ||
const ReduceFunction & | reduce_function, | ||
const ReduceConfig & | reduce_config = ReduceConfig() , |
||
const KeyHashFunction & | key_hash_function = KeyHashFunction() , |
||
const KeyEqualFunction & | key_equal_function = KeyEqualFunction() |
||
) | const |
ReduceByKey is a DOp, which groups elements of the DIA with the key_extractor and reduces each key-bucket to a single element using the associative reduce_function.
The reduce_function defines how two elements can be reduced to a single element of equal type.
In contrast to ReduceBy, the reduce_function is allowed to change the key (Example: Integers with modulo function as key_extractor). Creates overhead as both key and value have to be sent in shuffle step. Since ReduceByKey is a DOp, it creates a new DIANode. The DIA returned by Reduce links to this newly created DIANode. The stack_ of the returned DIA consists of the PostOp of Reduce, as a reduced element can directly be chained to the following LOps.
volatile_key_flag | tag |
key_extractor | Key extractor function, which maps each element to a key of possibly different type. |
ReduceFunction | Type of the reduce_function. This is a function reducing two elements of L's result type to a single element of equal type. |
reduce_function | Reduce function, which defines how the key buckets are reduced to a single element. This function is applied associative but not necessarily commutative. |
reduce_config | Reduce configuration. |
key_hash_function | Function to hash keys extracted by KeyExtractor. |
key_equal_function | Function to compare keys in reduce hash tables. |
Definition at line 275 of file reduce_by_key.hpp.
auto ReduceByKey | ( | const DuplicateDetectionFlag< DuplicateDetectionValue > & | duplicate_detection_flag, |
const KeyExtractor & | key_extractor, | ||
const ReduceFunction & | reduce_function, | ||
const ReduceConfig & | reduce_config = ReduceConfig() , |
||
const KeyHashFunction & | key_hash_function = KeyHashFunction() , |
||
const KeyEqualFunction & | key_equal_function = KeyEqualFunction() |
||
) | const |
ReduceByKey is a DOp, which groups elements of the DIA with the key_extractor and reduces each key-bucket to a single element using the associative reduce_function.
The reduce_function defines how two elements can be reduced to a single element of equal type.
In contrast to ReduceBy, the reduce_function is allowed to change the key (Example: Integers with modulo function as key_extractor). Creates overhead as both key and value have to be sent in shuffle step. Since ReduceByKey is a DOp, it creates a new DIANode. The DIA returned by Reduce links to this newly created DIANode. The stack_ of the returned DIA consists of the PostOp of Reduce, as a reduced element can directly be chained to the following LOps.
duplicate_detection_flag | tag |
key_extractor | Key extractor function, which maps each element to a key of possibly different type. |
ReduceFunction | Type of the reduce_function. This is a function reducing two elements of L's result type to a single element of equal type. |
reduce_function | Reduce function, which defines how the key buckets are reduced to a single element. This function is applied associative but not necessarily commutative. |
reduce_config | Reduce configuration. |
key_hash_function | Function to hash keys extracted by KeyExtractor. |
key_equal_function | Function to compare keys in reduce hash tables. |
Definition at line 293 of file reduce_by_key.hpp.
auto ReduceByKey | ( | const VolatileKeyFlag< VolatileKeyValue > & | , |
const DuplicateDetectionFlag< DuplicateDetectionValue > & | , | ||
const KeyExtractor & | key_extractor, | ||
const ReduceFunction & | reduce_function, | ||
const ReduceConfig & | reduce_config = ReduceConfig() , |
||
const KeyHashFunction & | key_hash_function = KeyHashFunction() , |
||
const KeyEqualFunction & | key_equal_function = KeyEqualFunction() |
||
) | const |
ReduceByKey is a DOp, which groups elements of the DIA with the key_extractor and reduces each key-bucket to a single element using the associative reduce_function.
The reduce_function defines how two elements can be reduced to a single element of equal type.
In contrast to ReduceBy, the reduce_function is allowed to change the key (Example: Integers with modulo function as key_extractor). Creates overhead as both key and value have to be sent in shuffle step. Since ReduceByKey is a DOp, it creates a new DIANode. The DIA returned by Reduce links to this newly created DIANode. The stack_ of the returned DIA consists of the PostOp of Reduce, as a reduced element can directly be chained to the following LOps.
key_extractor | Key extractor function, which maps each element to a key of possibly different type. |
ReduceFunction | Type of the reduce_function. This is a function reducing two elements of L's result type to a single element of equal type. |
reduce_function | Reduce function, which defines how the key buckets are reduced to a single element. This function is applied associative but not necessarily commutative. |
reduce_config | Reduce configuration. |
key_hash_function | Function to hash keys extracted by KeyExtractor. |
key_equal_function | Function to compare keys in reduce hash tables. |
Definition at line 312 of file reduce_by_key.hpp.
auto ReducePair | ( | const ReduceFunction & | reduce_function, |
const ReduceConfig & | reduce_config = ReduceConfig() |
||
) | const |
ReducePair is a DOp, which groups key-value-pairs in the input DIA by their key and reduces each key-bucket to a single element using the associative reduce_function.
The reduce_function defines how two elements can be reduced to a single element of equal type. The reduce_function is allowed to change the key. Since ReducePair is a DOp, it creates a new DIANode. The DIA returned by Reduce links to this newly created DIANode. The stack_ of the returned DIA consists of the PostOp of Reduce, as a reduced element can directly be chained to the following LOps.
ReduceFunction | Type of the reduce_function. This is a function reducing two elements of L's result type to a single element of equal type. |
reduce_function | Reduce function, which defines how the key buckets are reduced to a single element. This function is applied associative but not necessarily commutative. |
reduce_config | Reduce configuration. |
Definition at line 370 of file reduce_by_key.hpp.
Referenced by examples::page_rank::PageRankJoin().
auto ReducePair | ( | const ReduceFunction & | reduce_function, |
const ReduceConfig & | reduce_config, | ||
const KeyHashFunction & | key_hash_function | ||
) | const |
ReducePair is a DOp, which groups key-value-pairs in the input DIA by their key and reduces each key-bucket to a single element using the associative reduce_function.
The reduce_function defines how two elements can be reduced to a single element of equal type. The reduce_function is allowed to change the key. Since ReducePair is a DOp, it creates a new DIANode. The DIA returned by Reduce links to this newly created DIANode. The stack_ of the returned DIA consists of the PostOp of Reduce, as a reduced element can directly be chained to the following LOps.
ReduceFunction | Type of the reduce_function. This is a function reducing two elements of L's result type to a single element of equal type. |
reduce_function | Reduce function, which defines how the key buckets are reduced to a single element. This function is applied associative but not necessarily commutative. |
reduce_config | Reduce configuration. |
key_hash_function | Function to hash keys extracted by KeyExtractor. |
Definition at line 382 of file reduce_by_key.hpp.
auto ReducePair | ( | const ReduceFunction & | reduce_function, |
const ReduceConfig & | reduce_config, | ||
const KeyHashFunction & | key_hash_function, | ||
const KeyEqualFunction & | key_equal_function | ||
) | const |
ReducePair is a DOp, which groups key-value-pairs in the input DIA by their key and reduces each key-bucket to a single element using the associative reduce_function.
The reduce_function defines how two elements can be reduced to a single element of equal type. The reduce_function is allowed to change the key. Since ReducePair is a DOp, it creates a new DIANode. The DIA returned by Reduce links to this newly created DIANode. The stack_ of the returned DIA consists of the PostOp of Reduce, as a reduced element can directly be chained to the following LOps.
ReduceFunction | Type of the reduce_function. This is a function reducing two elements of L's result type to a single element of equal type. |
reduce_function | Reduce function, which defines how the key buckets are reduced to a single element. This function is applied associative but not necessarily commutative. |
reduce_config | Reduce configuration. |
key_hash_function | Function to hash keys extracted by KeyExtractor. |
key_equal_function | Function to compare keys in reduce hash tables. |
Definition at line 395 of file reduce_by_key.hpp.
auto ReducePair | ( | const DuplicateDetectionFlag< DuplicateDetectionValue > & | , |
const ReduceFunction & | reduce_function, | ||
const ReduceConfig & | reduce_config = ReduceConfig() , |
||
const KeyHashFunction & | key_hash_function = KeyHashFunction() , |
||
const KeyEqualFunction & | key_equal_function = KeyEqualFunction() |
||
) | const |
ReducePair is a DOp, which groups key-value-pairs in the input DIA by their key and reduces each key-bucket to a single element using the associative reduce_function.
The reduce_function defines how two elements can be reduced to a single element of equal type. The reduce_function is allowed to change the key. Since ReducePair is a DOp, it creates a new DIANode. The DIA returned by Reduce links to this newly created DIANode. The stack_ of the returned DIA consists of the PostOp of Reduce, as a reduced element can directly be chained to the following LOps.
ReduceFunction | Type of the reduce_function. This is a function reducing two elements of L's result type to a single element of equal type. |
reduce_function | Reduce function, which defines how the key buckets are reduced to a single element. This function is applied associative but not necessarily commutative. |
reduce_config | Reduce configuration. |
key_hash_function | Function to hash keys extracted by KeyExtractor. |
key_equal_function | Function to compare keys in reduce hash tables. |
Definition at line 410 of file reduce_by_key.hpp.
auto ReduceToIndex | ( | const KeyExtractor & | key_extractor, |
const ReduceFunction & | reduce_function, | ||
size_t | size, | ||
const ValueType & | neutral_element = ValueType() , |
||
const ReduceConfig & | reduce_config = ReduceConfig() |
||
) | const |
ReduceToIndex is a DOp, which groups elements of the DIA with the key_extractor returning an unsigned integers and reduces each key-bucket to a single element using the associative reduce_function.
In contrast to ReduceBy, ReduceToIndex returns a DIA in a defined order, which has the reduced element with key i in position i.
The reduce_function defines how two elements can be reduced to a single element of equal type. The key of the reduced element has to be equal to the keys of the input elements. Since ReduceToIndex is a DOp, it creates a new DIANode. The DIA returned by ReduceToIndex links to this newly created DIANode. The stack_ of the returned DIA consists of the PostOp of ReduceToIndex, as a reduced element can directly be chained to the following LOps.
key_extractor | Key extractor function, which maps each element to a key of possibly different type. |
ReduceFunction | Type of the reduce_function. This is a function reducing two elements of L's result type to a single element of equal type. |
reduce_function | Reduce function, which defines how the key buckets are reduced to a single element. This function is applied associative but not necessarily commutative. |
size | Resulting DIA size. Consequently, the key_extractor function but always return < size for any element in the input DIA. |
neutral_element | Item value with which to start the reduction in each array cell. |
reduce_config | Reduce configuration. |
Definition at line 259 of file reduce_to_index.hpp.
Referenced by examples::page_rank::PageRank(), and examples::suffix_sorting::PrefixDoublingWindow().
auto ReduceToIndex | ( | const VolatileKeyFlag< VolatileKeyValue > & | , |
const KeyExtractor & | key_extractor, | ||
const ReduceFunction & | reduce_function, | ||
size_t | size, | ||
const ValueType & | neutral_element = ValueType() , |
||
const ReduceConfig & | reduce_config = ReduceConfig() |
||
) | const |
ReduceToIndex is a DOp, which groups elements of the DIA with the key_extractor returning an unsigned integers and reduces each key-bucket to a single element using the associative reduce_function.
In contrast to ReduceByKey, ReduceToIndex returns a DIA in a defined order, which has the reduced element with key i in position i. The reduce_function defines how two elements can be reduced to a single element of equal type.
ReduceToIndex is the equivalent to ReduceByKey, as the reduce_function is allowed to change the key. Since ReduceToIndex is a DOp, it creates a new DIANode. The DIA returned by ReduceToIndex links to this newly created DIANode. The stack_ of the returned DIA consists of the PostOp of ReduceToIndex, as a reduced element can directly be chained to the following LOps.
key_extractor | Key extractor function, which maps each element to a key of possibly different type. |
ReduceFunction | Type of the reduce_function. This is a function reducing two elements of L's result type to a single element of equal type. |
reduce_function | Reduce function, which defines how the key buckets are reduced to a single element. This function is applied associative but not necessarily commutative. |
size | Resulting DIA size. Consequently, the key_extractor function but always return < size for any element in the input DIA. |
neutral_element | Item value with which to start the reduction in each array cell. |
reduce_config | Reduce configuration. |
Definition at line 274 of file reduce_to_index.hpp.
auto ReduceToIndex | ( | const struct SkipPreReducePhaseTag & | , |
const KeyExtractor & | key_extractor, | ||
const ReduceFunction & | reduce_function, | ||
size_t | size, | ||
const ValueType & | neutral_element = ValueType() , |
||
const ReduceConfig & | reduce_config = ReduceConfig() |
||
) | const |
ReduceToIndex is a DOp, which groups elements of the DIA with the key_extractor returning an unsigned integers and reduces each key-bucket to a single element using the associative reduce_function.
In contrast to ReduceByKey, ReduceToIndex returns a DIA in a defined order, which has the reduced element with key i in position i. The reduce_function defines how two elements can be reduced to a single element of equal type.
ReduceToIndex is the equivalent to ReduceByKey, as the reduce_function is allowed to change the key. Since ReduceToIndex is a DOp, it creates a new DIANode. The DIA returned by ReduceToIndex links to this newly created DIANode. The stack_ of the returned DIA consists of the PostOp of ReduceToIndex, as a reduced element can directly be chained to the following LOps.
key_extractor | Key extractor function, which maps each element to a key of possibly different type. |
ReduceFunction | Type of the reduce_function. This is a function reducing two elements of L's result type to a single element of equal type. |
reduce_function | Reduce function, which defines how the key buckets are reduced to a single element. This function is applied associative but not necessarily commutative. |
size | Resulting DIA size. Consequently, the key_extractor function but always return < size for any element in the input DIA. |
neutral_element | Item value with which to start the reduction in each array cell. |
reduce_config | Reduce configuration. |
Definition at line 332 of file reduce_to_index.hpp.
auto Sort | ( | const CompareFunction & | compare_function = CompareFunction() | ) | const |
Sort is a DOp, which sorts a given DIA according to the given compare_function.
CompareFunction | Type of the compare_function. Should be (ValueType,ValueType)->bool |
compare_function | Function, which compares two elements. Returns true, if first element is smaller than second. False otherwise. |
Definition at line 800 of file sort.hpp.
Referenced by examples::suffix_sorting::ConstructWaveletTree(), examples::suffix_sorting::DC7Recursive(), examples::suffix_sorting::PrefixDoublingSorting(), and examples::suffix_sorting::PrefixQuadrupling().
auto Sort | ( | const CompareFunction & | compare_function, |
const SortAlgorithm & | sort_algorithm | ||
) | const |
Sort is a DOp, which sorts a given DIA according to the given compare_function.
CompareFunction | Type of the compare_function. Should be (ValueType,ValueType)->bool |
compare_function | Function, which compares two elements. Returns true, if first element is smaller than second. False otherwise. |
sort_algorithm | Algorithm class used to sort items. Merging is always done using a tournament tree with compare_function. |
auto SortStable | ( | const CompareFunction & | compare_function = CompareFunction() | ) | const |
SortStable is a DOp, which sorts a given DIA stably according to the given compare_function.
CompareFunction | Type of the compare_function. Should be (ValueType,ValueType)->bool |
compare_function | Function, which compares two elements. Returns true, if first element is smaller than second. False otherwise. |
auto SortStable | ( | const CompareFunction & | compare_function, |
const SortAlgorithm & | sort_algorithm | ||
) | const |
SortStable is a DOp, which sorts a given DIA stably according to the given compare_function.
CompareFunction | Type of the compare_function. Should be (ValueType,ValueType)->bool |
compare_function | Function, which compares two elements. Returns true, if first element is smaller than second. False otherwise. |
sort_algorithm | Algorithm class used to stably sort items. Merging is always done using a tournament tree with compare_function. In order for the sorting to be stable, this must be a stable sorting algorithm. |
auto Window | ( | size_t | window_size, |
const WindowFunction & | window_function = WindowFunction() |
||
) | const |
Window is a DOp, which applies a window function to every k consecutive items in a DIA.
The window function is also given the index of the first item, and can output zero or more items via an Emitter.
window_size | the size of the delivered window. Signature: TODO(tb). |
window_function | Window function applied to each k item. |
Definition at line 286 of file window.hpp.
Referenced by examples::suffix_sorting::ConstructWaveletTree().
auto Window | ( | size_t | window_size, |
const WindowFunction & | window_function, | ||
const PartialWindowFunction & | partial_window_function | ||
) | const |
Window is a DOp, which applies a window function to every k consecutive items in a DIA.
The window function is also given the index of the first item, and can output zero or more items via an Emitter.
window_size | the size of the delivered window. Signature: TODO(tb). |
window_function | Window function applied to each k item. |
partial_window_function | Window function applied to less than k items. |
Definition at line 333 of file window.hpp.
auto Window | ( | struct DisjointTag const & | , |
size_t | window_size, | ||
const WindowFunction & | window_function | ||
) | const |
Window is a DOp, which applies a window function to every k consecutive items in a DIA.
The window function is also given the index of the first item, and can output zero or more items via an Emitter.
window_size | the size of the delivered window. |
window_function | Window function applied to each k item. |
Definition at line 526 of file window.hpp.
auto Zip | ( | const SecondDIA< ValueType_, Stack_ > & | second_dia, |
const ZipFunction & | zip_function | ||
) | const |
Zips two DIAs of equal size in style of functional programming by applying zip_function to the i-th elements of both input DIAs to form the i-th element of the output DIA.
The type of the output DIA can be inferred from the zip_function.
The two input DIAs are required to be of equal size, otherwise use the CutTag variant.
ZipFunction | Type of the zip_function. This is a function with two input elements, both of the local type, and one output element, which is the type of the Zip node. |
zip_function | Zip function, which zips two elements together |
second_dia | DIA, which is zipped together with the original DIA. |
Definition at line 686 of file zip.hpp.
Referenced by examples::suffix_sorting::DC3Recursive(), examples::suffix_sorting::DC7Recursive(), and examples::page_rank::PageRank().
auto Zip | ( | struct CutTag const & | , |
const SecondDIA< ValueType_, Stack_ > & | second_dia, | ||
const ZipFunction & | zip_function | ||
) | const |
Zips two DIAs in style of functional programming by applying zip_function to the i-th elements of both input DIAs to form the i-th element of the output DIA.
The type of the output DIA can be inferred from the zip_function.
If the two input DIAs are of unequal size, the result is the shorter of both. Otherwise use PadTag().
ZipFunction | Type of the zip_function. This is a function with two input elements, both of the local type, and one output element, which is the type of the Zip node. |
auto Zip | ( | struct PadTag const & | , |
const SecondDIA< ValueType_, Stack_ > & | second_dia, | ||
const ZipFunction & | zip_function | ||
) | const |
Zips two DIAs in style of functional programming by applying zip_function to the i-th elements of both input DIAs to form the i-th element of the output DIA.
The type of the output DIA can be inferred from the zip_function.
The output DIA's length is the maximum of all input DIAs, shorter DIAs are padded with default-constructed items.
ZipFunction | Type of the zip_function. This is a function with two input elements, both of the local type, and one output element, which is the type of the Zip node. |
auto Zip | ( | struct NoRebalanceTag const & | , |
const SecondDIA< ValueType_, Stack_ > & | second_dia, | ||
const ZipFunction & | zip_function | ||
) | const |
Zips two DIAs in style of functional programming by applying zip_function to the i-th elements of both input DIAs to form the i-th element of the output DIA.
The type of the output DIA can be inferred from the zip_function.
In this variant, the DIA partitions on all PEs must have matching length. No rebalancing is performed, and the program will die if any partition mismatches. This enables Zip to proceed without any communication.
ZipFunction | Type of the zip_function. This is a function with two input elements, both of the local type, and one output element, which is the type of the Zip node. |
auto ZipWithIndex | ( | const ZipFunction & | zip_function | ) | const |
Zips each item of a DIA with its zero-based array index.
This requires a full data store/retrieve cycle because the input DIA's size is generally unknown.
zip_function | Zip function, which gets each element together with its array index. |
Definition at line 140 of file zip_with_index.hpp.
Referenced by examples::suffix_sorting::DC3Recursive(), and examples::suffix_sorting::DC7Recursive().