Detailed Description

template<typename ValueType_, typename Stack_ = tlx::FunctionStack<ValueType_>>
class thrill::api::DIA< ValueType_, Stack_ >

DIA is the interface between the user and the Thrill framework.

A DIA can be imagined as an immutable array, even though the data does not need to be which represents the state after the previous DOp or Action. Additionally, a DIA stores the local lambda function chain of type Stack, which can transform elements of the DIANode to elements of this DIA. DOps/Actions create a DIA and a new DIANode, to which the DIA links to. LOps only create a new DIA, which link to the previous DIANode.

Template Parameters

ValueType	Type of elements currently in this DIA.
Stack	Type of the function chain.

Examples:: examples/tutorial/k-means_step1.cpp, examples/tutorial/k-means_step4.cpp, examples/tutorial/k-means_step5.cpp, and examples/tutorial/k-means_step6.cpp.

Definition at line 141 of file dia.hpp.

Inheritance diagram for DIA< ValueType_, Stack_ >:

Collaboration diagram for DIA< ValueType_, Stack_ >:

#include <dia.hpp>

Public Types
using	DIANodePtr = tlx::CountingPtr< DIANode< StackInput > >

using	Stack = Stack_
	Type of this function stack. More...

using	StackInput = typename Stack::Input

using	ValueType = ValueType_

Public Member Functions
	DIA ()=default
	default-constructor: invalid DIA More...

	DIA (const DIANodePtr &node, const Stack &stack, size_t dia_id, const char *label)
	Constructor of a new DIA with a pointer to a DIANode and a function chain from the DIANode to this DIA. More...

	DIA (DIANodePtr &&node, const Stack &stack, size_t dia_id, const char *label)
	Constructor of a new DIA supporting move semantics of nodes. More...

	DIA (DIANodePtr &&node)
	Constructor of a new DIA with a real backing DIABase. More...

template<typename AnyStack >
	DIA (const DIA< ValueType, AnyStack > &rhs)
	Copy-Constructor of a DIA with empty function chain from a DIA with a non-empty chain. More...

void	AssertValid () const
	Assert that the DIA is valid. More...

const DIA &	Dispose () const
	Dispose of the DIANode's data. More...

const DIA &	Execute () const
	Execute DIA's scope and parents such that this (Action)Node is Executed. More...

template<typename ValueOut , bool LocationDetectionValue, typename KeyExtractor , typename GroupFunction , typename HashFunction >
auto	GroupByKey (const LocationDetectionFlag< LocationDetectionValue > &, const KeyExtractor &key_extractor, const GroupFunction &groupby_function, const HashFunction &hash_function) const

template<typename ValueOut , typename KeyExtractor, typename GroupFunction , typename HashFunction >
auto	GroupByKey (const KeyExtractor &key_extractor, const GroupFunction &groupby_function, const HashFunction &hash_function) const

template<typename ValueOut , typename KeyExtractor, typename GroupFunction >
auto	GroupByKey (const KeyExtractor &key_extractor, const GroupFunction &groupby_function) const

template<typename ValueOut , typename KeyExtractor , typename GroupFunction >
auto	GroupToIndex (const KeyExtractor &key_extractor, const GroupFunction &groupby_function, const size_t result_size, const ValueOut &neutral_element) const

bool	IsValid () const
	Return whether the DIA is valid. More...

const DIA &	Keep (size_t increase=1) const
	Mark the referenced DIANode for keeping, which makes children not consume the data when executing. More...

const DIA &	KeepForever () const
	Mark the referenced DIANode for keeping forever, which makes children not consume the data when executing. More...

Const Accessors
const DIANodePtr &	node () const
	Returns a pointer to the according DIANode. More...

size_t	node_refcount () const
	Returns the number of references to the according DIANode. More...

const Stack &	stack () const
	Returns the stored function chain. More...

Context &	context () const
	Return context_ of DIANode, e.g. for creating new LOps and DOps. More...

Context &	ctx () const
	Return context_ of DIANode, e.g. for creating new LOps and DOps. More...

size_t	id () const
	Returns id_. More...

const char *	label () const
	Returns label_. More...

Local Operations (LOps)
template<typename MapFunction >
auto	Map (const MapFunction &map_function) const
	Map applies `map_function` : \( A \to B \) to each item of a DIA and delivers a new DIA contains the returned values, which may be of a different type. More...

template<typename FilterFunction >
auto	Filter (const FilterFunction &filter_function) const
	Each item of a DIA is tested using `filter_function` : \( A \to \textrm{bool} \) to determine whether it is copied into the output DIA or excluded. More...

template<typename ResultType = ValueType, typename FlatmapFunction >
auto	FlatMap (const FlatmapFunction &flatmap_function) const
	Each item of a DIA is expanded by the `flatmap_function` : \( A \to \textrm{array}(B) \) to zero or more items of different type, which are concatenated in the resulting DIA. More...

auto	BernoulliSample (double p) const
	Each item of a DIA is copied into the output DIA with success probability p (an independent Bernoulli trial). More...

template<typename SecondDIA >
auto	Union (const SecondDIA &second_dia) const
	Union is a LOp, which creates the union of all items from any number of DIAs as a single DIA, where the items are in an arbitrary order. More...

Actions
size_t	Size () const
	Computes the total size of all elements across all workers. More...

Future< size_t >	SizeFuture () const
	Lazily computes the total size of all elements across all workers. More...

std::vector< ValueType >	AllGather () const
	Returns the whole DIA in an std::vector on each worker. More...

void	AllGather (std::vector< ValueType > *out_vector) const
	AllGather is an Action, which returns the whole DIA in an std::vector on each worker. This is only for testing purposes and should not be used on large datasets. More...

Future< std::vector< ValueType > >	AllGatherFuture () const
	Returns the whole DIA in an std::vector on each worker. More...

void	Print (const std::string &name=std::string()) const
	Print is an Action, which collects all data of the DIA at the worker 0 and prints using ostream serialization. More...

void	Print (const std::string &name, std::ostream &out) const
	Print is an Action, which collects all data of the DIA at the worker 0 and prints using ostream serialization. More...

std::vector< ValueType >	Gather (size_t target_id=0) const
	Gather is an Action, which collects all data of the DIA into a vector at the given worker. More...

void	Gather (size_t target_id, std::vector< ValueType > *out_vector) const
	Gather is an Action, which collects all data of the DIA into a vector at the given worker. More...

auto	Sample (size_t sample_size) const
	Select up to sample_size items uniformly at random and return a new DIA<T>. More...

template<typename ReduceFunction >
ValueType	AllReduce (const ReduceFunction &reduce_function) const
	AllReduce is an Action, which computes the reduction sum of all elements globally and delivers the same value on all workers. More...

template<typename ReduceFunction >
ValueType	AllReduce (const ReduceFunction &reduce_function, const ValueType &initial_value) const
	AllReduce is an Action, which computes the reduction sum of all elements globally and delivers the same value on all workers. More...

template<typename ReduceFunction >
Future< ValueType >	AllReduceFuture (const ReduceFunction &reduce_function) const
	AllReduce is an ActionFuture, which computes the reduction sum of all elements globally and delivers the same value on all workers. More...

template<typename ReduceFunction >
Future< ValueType >	AllReduceFuture (const ReduceFunction &reduce_function, const ValueType &initial_value) const
	AllReduce is an ActionFuture, which computes the reduction sum of all elements globally and delivers the same value on all workers. More...

template<typename SumFunction = std::plus<ValueType>>
ValueType	Sum (const SumFunction &sum_function=SumFunction()) const
	Sum is an Action, which computes the sum of all elements globally. More...

template<typename SumFunction = std::plus<ValueType>>
ValueType	Sum (const SumFunction &sum_function, const ValueType &initial_value) const
	Sum is an Action, which computes the sum of all elements globally. More...

template<typename SumFunction = std::plus<ValueType>>
Future< ValueType >	SumFuture (const SumFunction &sum_function=SumFunction(), const ValueType &initial_value=ValueType()) const
	Sum is an ActionFuture, which computes the sum of all elements globally. More...

ValueType	Min () const
	Min is an Action, which computes the minimum of all elements globally. More...

ValueType	Min (const ValueType &initial_value) const
	Min is an Action, which computes the minimum of all elements globally. More...

Future< ValueType >	MinFuture () const
	Min is an ActionFuture, which computes the minimum of all elements globally. More...

Future< ValueType >	MinFuture (const ValueType &initial_value) const
	Min is an ActionFuture, which computes the minimum of all elements globally. More...

ValueType	Max () const
	Max is an Action, which computes the maximum of all elements globally. More...

ValueType	Max (const ValueType &initial_value) const
	Max is an Action, which computes the maximum of all elements globally. More...

Future< ValueType >	MaxFuture () const
	Max is an ActionFuture, which computes the maximum of all elements globally. More...

Future< ValueType >	MaxFuture (const ValueType &initial_value) const
	Max is an ActionFuture, which computes the maximum of all elements globally. More...

template<size_t p>
double	HyperLogLog () const
	Compute the approximate number of distinct elements in the DIA. More...

void	WriteLinesOne (const std::string &filepath) const
	WriteLinesOne is an Action, which writes std::strings to a single output file. More...

Future< void >	WriteLinesOneFuture (const std::string &filepath) const
	WriteLinesOne is an ActionFuture, which writes std::strings to a single output file. More...

void	WriteLines (const std::string &filepath, size_t target_file_size=128 1024 1024) const
	WriteLines is an Action, which writes std::strings to multiple output files. More...

Future< void >	WriteLinesFuture (const std::string &filepath, size_t target_file_size=128 1024 1024) const
	WriteLines is an ActionFuture, which writes std::strings to multiple output files. More...

void	WriteBinary (const std::string &filepath, size_t max_file_size=128 1024 1024) const
	WriteBinary is a function, which writes a DIA to many files per worker. More...

Future< void >	WriteBinaryFuture (const std::string &filepath, size_t max_file_size=128 1024 1024) const
	WriteBinary is a function, which writes a DIA to many files per worker. More...

Distributed Operations (DOps)
This list of DOps are methods of the main DIA class and called as `A.Method(params)`. Methods combining two or more DIAs are available as free functions.
template<typename KeyExtractor , typename ReduceFunction , typename ReduceConfig = class DefaultReduceConfig>
auto	ReduceByKey (const KeyExtractor &key_extractor, const ReduceFunction &reduce_function, const ReduceConfig &reduce_config=ReduceConfig()) const
	ReduceByKey is a DOp, which groups elements of the DIA with the key_extractor and reduces each key-bucket to a single element using the associative reduce_function. More...

template<typename KeyExtractor , typename ReduceFunction , typename ReduceConfig , typename KeyHashFunction >
auto	ReduceByKey (const KeyExtractor &key_extractor, const ReduceFunction &reduce_function, const ReduceConfig &reduce_config, const KeyHashFunction &key_hash_function) const
	ReduceByKey is a DOp, which groups elements of the DIA with the key_extractor and reduces each key-bucket to a single element using the associative reduce_function. More...

template<typename KeyExtractor , typename ReduceFunction , typename ReduceConfig , typename KeyHashFunction , typename KeyEqualFunction >
auto	ReduceByKey (const KeyExtractor &key_extractor, const ReduceFunction &reduce_function, const ReduceConfig &reduce_config, const KeyHashFunction &key_hash_function, const KeyEqualFunction &key_equal_function) const
	ReduceByKey is a DOp, which groups elements of the DIA with the key_extractor and reduces each key-bucket to a single element using the associative reduce_function. More...

template<bool VolatileKeyValue, typename KeyExtractor , typename ReduceFunction , typename ReduceConfig = class DefaultReduceConfig, typename KeyHashFunction = std::hash<typename FunctionTraits<KeyExtractor>::result_type>, typename KeyEqualFunction = std::equal_to<typename FunctionTraits<KeyExtractor>::result_type>>
auto	ReduceByKey (const VolatileKeyFlag< VolatileKeyValue > &, const KeyExtractor &key_extractor, const ReduceFunction &reduce_function, const ReduceConfig &reduce_config=ReduceConfig(), const KeyHashFunction &key_hash_function=KeyHashFunction(), const KeyEqualFunction &key_equal_function=KeyEqualFunction()) const
	ReduceByKey is a DOp, which groups elements of the DIA with the key_extractor and reduces each key-bucket to a single element using the associative reduce_function. More...

template<bool DuplicateDetectionValue, typename KeyExtractor , typename ReduceFunction , typename ReduceConfig = class DefaultReduceConfig, typename KeyHashFunction = std::hash<typename FunctionTraits<KeyExtractor>::result_type>, typename KeyEqualFunction = std::equal_to<typename FunctionTraits<KeyExtractor>::result_type>>
auto	ReduceByKey (const DuplicateDetectionFlag< DuplicateDetectionValue > &, const KeyExtractor &key_extractor, const ReduceFunction &reduce_function, const ReduceConfig &reduce_config=ReduceConfig(), const KeyHashFunction &key_hash_function=KeyHashFunction(), const KeyEqualFunction &key_equal_function=KeyEqualFunction()) const
	ReduceByKey is a DOp, which groups elements of the DIA with the key_extractor and reduces each key-bucket to a single element using the associative reduce_function. More...

template<bool VolatileKeyValue, bool DuplicateDetectionValue, typename KeyExtractor , typename ReduceFunction , typename ReduceConfig = class DefaultReduceConfig, typename KeyHashFunction = std::hash<typename FunctionTraits<KeyExtractor>::result_type>, typename KeyEqualFunction = std::equal_to<typename FunctionTraits<KeyExtractor>::result_type>>
auto	ReduceByKey (const VolatileKeyFlag< VolatileKeyValue > &, const DuplicateDetectionFlag< DuplicateDetectionValue > &, const KeyExtractor &key_extractor, const ReduceFunction &reduce_function, const ReduceConfig &reduce_config=ReduceConfig(), const KeyHashFunction &key_hash_function=KeyHashFunction(), const KeyEqualFunction &key_equal_function=KeyEqualFunction()) const
	ReduceByKey is a DOp, which groups elements of the DIA with the key_extractor and reduces each key-bucket to a single element using the associative reduce_function. More...

template<typename ReduceFunction , typename ReduceConfig = class DefaultReduceConfig>
auto	ReducePair (const ReduceFunction &reduce_function, const ReduceConfig &reduce_config=ReduceConfig()) const
	ReducePair is a DOp, which groups key-value-pairs in the input DIA by their key and reduces each key-bucket to a single element using the associative reduce_function. More...

template<typename ReduceFunction , typename ReduceConfig , typename KeyHashFunction >
auto	ReducePair (const ReduceFunction &reduce_function, const ReduceConfig &reduce_config, const KeyHashFunction &key_hash_function) const
	ReducePair is a DOp, which groups key-value-pairs in the input DIA by their key and reduces each key-bucket to a single element using the associative reduce_function. More...

template<typename ReduceFunction , typename ReduceConfig , typename KeyHashFunction , typename KeyEqualFunction >
auto	ReducePair (const ReduceFunction &reduce_function, const ReduceConfig &reduce_config, const KeyHashFunction &key_hash_function, const KeyEqualFunction &key_equal_function) const
	ReducePair is a DOp, which groups key-value-pairs in the input DIA by their key and reduces each key-bucket to a single element using the associative reduce_function. More...

template<bool DuplicateDetectionValue, typename ReduceFunction , typename ReduceConfig = class DefaultReduceConfig, typename KeyHashFunction , typename KeyEqualFunction >
auto	ReducePair (const DuplicateDetectionFlag< DuplicateDetectionValue > &, const ReduceFunction &reduce_function, const ReduceConfig &reduce_config=ReduceConfig(), const KeyHashFunction &key_hash_function=KeyHashFunction(), const KeyEqualFunction &key_equal_function=KeyEqualFunction()) const
	ReducePair is a DOp, which groups key-value-pairs in the input DIA by their key and reduces each key-bucket to a single element using the associative reduce_function. More...

template<typename KeyExtractor , typename ReduceFunction , typename ReduceConfig = class DefaultReduceToIndexConfig>
auto	ReduceToIndex (const KeyExtractor &key_extractor, const ReduceFunction &reduce_function, size_t size, const ValueType &neutral_element=ValueType(), const ReduceConfig &reduce_config=ReduceConfig()) const
	ReduceToIndex is a DOp, which groups elements of the DIA with the key_extractor returning an unsigned integers and reduces each key-bucket to a single element using the associative reduce_function. More...

template<bool VolatileKeyValue, typename KeyExtractor , typename ReduceFunction , typename ReduceConfig = class DefaultReduceToIndexConfig>
auto	ReduceToIndex (const VolatileKeyFlag< VolatileKeyValue > &, const KeyExtractor &key_extractor, const ReduceFunction &reduce_function, size_t size, const ValueType &neutral_element=ValueType(), const ReduceConfig &reduce_config=ReduceConfig()) const
	ReduceToIndex is a DOp, which groups elements of the DIA with the key_extractor returning an unsigned integers and reduces each key-bucket to a single element using the associative reduce_function. More...

template<typename KeyExtractor , typename ReduceFunction , typename ReduceConfig = class DefaultReduceToIndexConfig>
auto	ReduceToIndex (const struct SkipPreReducePhaseTag &, const KeyExtractor &key_extractor, const ReduceFunction &reduce_function, size_t size, const ValueType &neutral_element=ValueType(), const ReduceConfig &reduce_config=ReduceConfig()) const
	ReduceToIndex is a DOp, which groups elements of the DIA with the key_extractor returning an unsigned integers and reduces each key-bucket to a single element using the associative reduce_function. More...

template<typename ValueOut , typename KeyExtractor , typename GroupByFunction >
auto	GroupByKey (const KeyExtractor &key_extractor, const GroupByFunction &groupby_function) const
	GroupByKey is a DOp, which groups elements of the DIA by its key. More...

template<typename ValueOut , typename KeyExtractor , typename GroupByFunction , typename HashFunction >
auto	GroupByKey (const KeyExtractor &key_extractor, const GroupByFunction &groupby_function, const HashFunction &hash_function) const
	GroupByKey is a DOp, which groups elements of the DIA by its key. More...

template<typename ValueOut , bool LocationDetectionTagValue, typename KeyExtractor , typename GroupByFunction , typename HashFunction = std::hash<typename FunctionTraits<KeyExtractor>::result_type>>
auto	GroupByKey (const LocationDetectionFlag< LocationDetectionTagValue > &, const KeyExtractor &key_extractor, const GroupByFunction &groupby_function, const HashFunction &hash_function=HashFunction()) const
	GroupByKey is a DOp, which groups elements of the DIA by its key. More...

template<typename ValueOut , typename KeyExtractor , typename GroupByFunction >
auto	GroupToIndex (const KeyExtractor &key_extractor, const GroupByFunction &groupby_function, const size_t size, const ValueOut &neutral_element=ValueOut()) const
	GroupBy is a DOp, which groups elements of the DIA by its key. More...

template<typename ZipFunction , typename SecondDIA >
auto	Zip (const SecondDIA &second_dia, const ZipFunction &zip_function) const
	Zips two DIAs of equal size in style of functional programming by applying zip_function to the i-th elements of both input DIAs to form the i-th element of the output DIA. More...

template<typename ZipFunction , typename SecondDIA >
auto	Zip (struct CutTag const &, const SecondDIA &second_dia, const ZipFunction &zip_function) const
	Zips two DIAs in style of functional programming by applying zip_function to the i-th elements of both input DIAs to form the i-th element of the output DIA. More...

template<typename ZipFunction , typename SecondDIA >
auto	Zip (struct PadTag const &, const SecondDIA &second_dia, const ZipFunction &zip_function) const
	Zips two DIAs in style of functional programming by applying zip_function to the i-th elements of both input DIAs to form the i-th element of the output DIA. More...

template<typename ZipFunction , typename SecondDIA >
auto	Zip (struct NoRebalanceTag const &, const SecondDIA &second_dia, const ZipFunction &zip_function) const
	Zips two DIAs in style of functional programming by applying zip_function to the i-th elements of both input DIAs to form the i-th element of the output DIA. More...

template<typename ZipFunction >
auto	ZipWithIndex (const ZipFunction &zip_function) const
	Zips each item of a DIA with its zero-based array index. More...

template<typename CompareFunction = std::less<ValueType>>
auto	Sort (const CompareFunction &compare_function=CompareFunction()) const
	Sort is a DOp, which sorts a given DIA according to the given compare_function. More...

template<typename CompareFunction , typename SortAlgorithm >
auto	Sort (const CompareFunction &compare_function, const SortAlgorithm &sort_algorithm) const
	Sort is a DOp, which sorts a given DIA according to the given compare_function. More...

template<typename CompareFunction = std::less<ValueType>>
auto	SortStable (const CompareFunction &compare_function=CompareFunction()) const
	SortStable is a DOp, which sorts a given DIA stably according to the given compare_function. More...

template<typename CompareFunction , typename SortAlgorithm >
auto	SortStable (const CompareFunction &compare_function, const SortAlgorithm &sort_algorithm) const
	SortStable is a DOp, which sorts a given DIA stably according to the given compare_function. More...

template<typename Comparator = std::less<ValueType>, typename SecondDIA >
auto	Merge (const SecondDIA &second_dia, const Comparator &comparator=Comparator()) const
	Merge is a DOp, which merges two sorted DIAs to a single sorted DIA. More...

template<typename SumFunction = std::plus<ValueType>>
auto	PrefixSum (const SumFunction &sum_function=SumFunction(), const ValueType &initial_element=ValueType()) const
	PrefixSum is a DOp, which computes the (inclusive) prefix sum of all elements. More...

template<typename SumFunction = std::plus<ValueType>>
auto	ExPrefixSum (const SumFunction &sum_function=SumFunction(), const ValueType &initial_element=ValueType()) const
	ExPrefixSum is a DOp, which computes the exclusive prefix sum of all elements. More...

template<typename WindowFunction >
auto	Window (size_t window_size, const WindowFunction &window_function=WindowFunction()) const
	Window is a DOp, which applies a window function to every k consecutive items in a DIA. More...

template<typename WindowFunction , typename PartialWindowFunction >
auto	Window (size_t window_size, const WindowFunction &window_function, const PartialWindowFunction &partial_window_function) const
	Window is a DOp, which applies a window function to every k consecutive items in a DIA. More...

template<typename WindowFunction >
auto	Window (struct DisjointTag const &, size_t window_size, const WindowFunction &window_function) const
	Window is a DOp, which applies a window function to every k consecutive items in a DIA. More...

template<typename ValueOut , typename WindowFunction >
auto	FlatWindow (size_t window_size, const WindowFunction &window_function=WindowFunction()) const
	FlatWindow is a DOp, which applies a window function to every k consecutive items in a DIA. More...

template<typename ValueOut , typename WindowFunction , typename PartialWindowFunction >
auto	FlatWindow (size_t window_size, const WindowFunction &window_function, const PartialWindowFunction &partial_window_function) const
	FlatWindow is a DOp, which applies a window function to every k consecutive items in a DIA. More...

template<typename ValueOut , typename WindowFunction >
auto	FlatWindow (struct DisjointTag const &, size_t window_size, const WindowFunction &window_function) const
	FlatWindow is a DOp, which applies a window function to every k consecutive items in a DIA. More...

template<typename SecondDIA >
auto	Concat (const SecondDIA &second_dia) const
	Concat is a DOp, which concatenates any number of DIAs to a single DIA. More...

auto	Rebalance () const
	Rebalance is a DOp, which rebalances a single DIA among all workers; in general, this operation is needed only if previous steps are known to create heavy imbalance (e.g. More...

DIA< ValueType >	Collapse () const
	Create a CollapseNode which is mainly used to collapse the LOp chain into a DIA<T> with an empty stack. More...

DIA< ValueType >	Cache () const
	Create a CacheNode which contains all items of a DIA in calculated plain format. More...

Static Public Attributes
static constexpr bool	stack_empty = Stack::empty
	boolean indication whether this FunctionStack is empty More...

Private Types
template<typename Function >
using	FunctionTraits = common::FunctionTraits< Function >
	alias for convenience. More...

Private Member Functions
size_t	next_dia_id ()
	deliver next DIA serial id More...

Private Attributes
size_t	dia_id_ = 0
	DIA serial id for logging, matches DIANode::id_ for DOps. More...

const char *	label_ = nullptr
	static DIA (LOp or DOp) node label string, may match DIANode::label_. More...

DIANodePtr	node_

Stack	stack_

Member Typedef Documentation

◆ DIANodePtr

using DIANodePtr = tlx::CountingPtr<DIANode<StackInput> >

type of pointer to the real node object implementation. This object has base item type StackInput which is transformed by the function stack lambdas further. But even pushing more lambdas does not change the stack input type.

Definition at line 169 of file dia.hpp.

◆ FunctionTraits

using FunctionTraits = common::FunctionTraits<Function>

private

alias for convenience.

Definition at line 147 of file dia.hpp.

◆ Stack

using Stack = Stack_

Type of this function stack.

Definition at line 155 of file dia.hpp.

◆ StackInput

using StackInput = typename Stack::Input

type of the items delivered by the DOp, and pushed down the function stack towards the next nodes. If the function stack contains LOps nodes, these may transform the type.

Definition at line 160 of file dia.hpp.

◆ ValueType

using ValueType = ValueType_

type of the items virtually in the DIA, which is the type emitted by the current LOp stack.

Definition at line 152 of file dia.hpp.

Constructor & Destructor Documentation

◆ DIA() [1/5]

DIA ( )

default

default-constructor: invalid DIA

◆ DIA() [2/5]

DIA	(	const DIANodePtr &	node,
		const Stack &	stack,
		size_t	dia_id,
		const char *	label
	)

inline

Constructor of a new DIA with a pointer to a DIANode and a function chain from the DIANode to this DIA.

Parameters

node	Pointer to the last DIANode, DOps and Actions create a new DIANode, LOps link to the DIANode of the previous DIA.
stack	Function stack consisting of functions between last DIANode and this DIA.
dia_id	Serial id of DIA, which includes LOps
label	static string label of DIA.

Definition at line 194 of file dia.hpp.

◆ DIA() [3/5]

DIA	(	DIANodePtr &&	node,
		const Stack &	stack,
		size_t	dia_id,
		const char *	label
	)

inline

Constructor of a new DIA supporting move semantics of nodes.

Parameters

node	Pointer to the last DIANode, DOps and Actions create a new DIANode, LOps link to the DIANode of the previous DIA.
stack	Function stack consisting of functions between last DIANode and this DIA.
dia_id	Serial id of DIA, which includes LOps
label	static string label of DIA.

Definition at line 212 of file dia.hpp.

◆ DIA() [4/5]

DIA ( DIANodePtr && node )

inlineexplicit

Constructor of a new DIA with a real backing DIABase.

Parameters

node	Pointer to the last DIANode, DOps and Actions create a new DIANode, LOps link to the DIANode of the previous DIA.

Definition at line 223 of file dia.hpp.

◆ DIA() [5/5]

DIA ( const DIA< ValueType, AnyStack > & rhs )

Copy-Constructor of a DIA with empty function chain from a DIA with a non-empty chain.

The functionality of the chain is stored in a newly created LOpNode. The current DIA than points to this LOpNode. This is needed to support assignment operations between DIA's.

Member Function Documentation

◆ AssertValid()

void AssertValid ( ) const

inline

Assert that the DIA is valid.

Definition at line 178 of file dia.hpp.

Referenced by thrill::api::Zip(), and thrill::api::ZipWindow().

◆ context()

Context& context ( ) const

inline

Return context_ of DIANode, e.g. for creating new LOps and DOps.

Definition at line 273 of file dia.hpp.

Referenced by examples::page_rank::PageRank(), examples::page_rank::PageRankJoin(), examples::select::PickPivots(), and examples::select::Select().

◆ ctx()

Context& ctx ( ) const

inline

Return context_ of DIANode, e.g. for creating new LOps and DOps.

Definition at line 279 of file dia.hpp.

◆ Dispose()

const DIA& Dispose ( ) const

inline

Dispose of the DIANode's data.

Definition at line 295 of file dia.hpp.

◆ Execute()

const DIA& Execute ( ) const

inline

Execute DIA's scope and parents such that this (Action)Node is Executed.

This does not create a new DIA, but returns the existing one.

Definition at line 335 of file dia.hpp.

◆ GroupByKey() [1/3]

auto GroupByKey	(	const LocationDetectionFlag< LocationDetectionValue > &	,
		const KeyExtractor &	key_extractor,
		const GroupFunction &	groupby_function,
		const HashFunction &	hash_function
	)		const

Definition at line 384 of file group_by_key.hpp.

◆ GroupByKey() [2/3]

auto GroupByKey	(	const KeyExtractor &	key_extractor,
		const GroupFunction &	groupby_function,
		const HashFunction &	hash_function
	)		const

Definition at line 410 of file group_by_key.hpp.

◆ GroupByKey() [3/3]

auto GroupByKey	(	const KeyExtractor &	key_extractor,
		const GroupFunction &	groupby_function
	)		const

Definition at line 421 of file group_by_key.hpp.

◆ GroupToIndex()

auto GroupToIndex	(	const KeyExtractor &	key_extractor,
		const GroupFunction &	groupby_function,
		const size_t	result_size,
		const ValueOut &	neutral_element
	)		const

Definition at line 261 of file group_to_index.hpp.

◆ id()

size_t id ( ) const

inline

Returns id_.

Definition at line 285 of file dia.hpp.

◆ IsValid()

bool IsValid ( ) const

inline

Return whether the DIA is valid.

Definition at line 175 of file dia.hpp.

Referenced by CollapseSwitch< ValueType, Stack >::MakeCollapse().

◆ Keep()

const DIA& Keep ( size_t increase = 1 ) const

inline

Mark the referenced DIANode for keeping, which makes children not consume the data when executing.

This does not create a new DIA, but returns the existing one.

Definition at line 310 of file dia.hpp.

Referenced by examples::suffix_sorting::ConstructWaveletTree(), examples::suffix_sorting::DC3Recursive(), examples::suffix_sorting::DC7Recursive(), examples::select::PickPivots(), examples::suffix_sorting::PrefixDoublingSorting(), examples::suffix_sorting::PrefixDoublingWindow(), examples::suffix_sorting::PrefixQuadrupling(), examples::suffix_sorting::PrefixQuadruplingDiscarding(), and examples::select::Select().

◆ KeepForever()

const DIA& KeepForever ( ) const

inline

Mark the referenced DIANode for keeping forever, which makes children not consume the data when executing.

This does not create a new DIA, but returns the existing one.

Definition at line 325 of file dia.hpp.

◆ label()

const char* label ( ) const

inline

Returns label_.

Definition at line 288 of file dia.hpp.

◆ next_dia_id()

size_t next_dia_id ( )

inlineprivate

deliver next DIA serial id

Definition at line 2041 of file dia.hpp.

◆ node()

const DIANodePtr& node ( ) const

inline

Returns a pointer to the according DIANode.

Definition at line 255 of file dia.hpp.

◆ node_refcount()

size_t node_refcount ( ) const

inline

Returns the number of references to the according DIANode.

Definition at line 261 of file dia.hpp.

◆ stack()

const Stack& stack ( ) const

inline

Returns the stored function chain.

Definition at line 267 of file dia.hpp.

Member Data Documentation

◆ dia_id_

size_t dia_id_ = 0

private

DIA serial id for logging, matches DIANode::id_ for DOps.

Definition at line 2035 of file dia.hpp.

◆ label_

const char* label_ = nullptr

private

static DIA (LOp or DOp) node label string, may match DIANode::label_.

Definition at line 2038 of file dia.hpp.

◆ node_

DIANodePtr node_

private

The DIANode which DIA points to. The node represents the latest DOp or Action performed previously.

Definition at line 2028 of file dia.hpp.

◆ stack_

Stack stack_

private

The local function chain, which stores the chained lambda function from the last DIANode to this DIA.

Definition at line 2032 of file dia.hpp.

◆ stack_empty

constexpr bool stack_empty = Stack::empty

static

boolean indication whether this FunctionStack is empty

Definition at line 163 of file dia.hpp.

The documentation for this class was generated from the following files:

thrill/api/dia.hpp
thrill/api/all_gather.hpp
thrill/api/all_reduce.hpp
thrill/api/bernoulli_sample.hpp
thrill/api/cache.hpp
thrill/api/collapse.hpp
thrill/api/concat.hpp
thrill/api/ex_prefix_sum.hpp
thrill/api/gather.hpp
thrill/api/group_by_key.hpp
thrill/api/group_to_index.hpp
thrill/api/hyperloglog.hpp
thrill/api/max.hpp
thrill/api/merge.hpp
thrill/api/min.hpp
thrill/api/prefix_sum.hpp
thrill/api/print.hpp
thrill/api/rebalance.hpp
thrill/api/reduce_by_key.hpp
thrill/api/reduce_to_index.hpp
thrill/api/sample.hpp
thrill/api/size.hpp
thrill/api/sort.hpp
thrill/api/sum.hpp
thrill/api/union.hpp
thrill/api/window.hpp
thrill/api/write_binary.hpp
thrill/api/write_lines.hpp
thrill/api/write_lines_one.hpp
thrill/api/zip.hpp
thrill/api/zip_with_index.hpp

Detailed Description

template<typename ValueType_, typename Stack_ = tlx::FunctionStack<ValueType_>> class thrill::api::DIA< ValueType_, Stack_ >

Public Types

Public Member Functions

Static Public Attributes

Private Types

Private Member Functions

Private Attributes

Member Typedef Documentation

◆ DIANodePtr

◆ FunctionTraits

◆ Stack

◆ StackInput

◆ ValueType

Constructor & Destructor Documentation

◆ DIA() [1/5]

◆ DIA() [2/5]

◆ DIA() [3/5]

◆ DIA() [4/5]

◆ DIA() [5/5]

Member Function Documentation

◆ AssertValid()

◆ context()

◆ ctx()

◆ Dispose()

◆ Execute()

◆ GroupByKey() [1/3]

◆ GroupByKey() [2/3]

◆ GroupByKey() [3/3]

◆ GroupToIndex()

◆ id()

◆ IsValid()

◆ Keep()

◆ KeepForever()

◆ label()

◆ next_dia_id()

◆ node()

◆ node_refcount()

◆ stack()

Member Data Documentation

◆ dia_id_

◆ label_

◆ node_

◆ stack_

◆ stack_empty

template<typename ValueType_, typename Stack_ = tlx::FunctionStack<ValueType_>>
class thrill::api::DIA< ValueType_, Stack_ >