Free Operation Functions

Functions
template<typename FirstDIA , typename... DIAs>
auto	Concat (const FirstDIA &first_dia, const DIAs &... dias)
	Concat is a DOp, which concatenates any number of DIAs to a single DIA. More...
template<typename ValueType >
auto	Concat (const std::initializer_list< DIA< ValueType > > &dias)
	Concat is a DOp, which concatenates any number of DIAs to a single DIA. More...
template<typename ValueType >
auto	Concat (const std::vector< DIA< ValueType > > &dias)
	Concat is a DOp, which concatenates any number of DIAs to a single DIA. More...
template<bool LocationDetectionValue, typename FirstDIA , typename SecondDIA , typename KeyExtractor1 , typename KeyExtractor2 , typename JoinFunction , typename HashFunction = std::hash<typename common::FunctionTraits<KeyExtractor1>::result_type>>
auto	InnerJoin (const LocationDetectionFlag< LocationDetectionValue > &, const FirstDIA &first_dia, const SecondDIA &second_dia, const KeyExtractor1 &key_extractor1, const KeyExtractor2 &key_extractor2, const JoinFunction &join_function, const HashFunction &hash_function=HashFunction())
	Performs an inner join between this DIA and the DIA given in the first parameter. More...
template<typename FirstDIA , typename SecondDIA , typename KeyExtractor1 , typename KeyExtractor2 , typename JoinFunction , typename HashFunction = std::hash<typename common::FunctionTraits<KeyExtractor1>::result_type>>
auto	InnerJoin (const FirstDIA &first_dia, const SecondDIA &second_dia, const KeyExtractor1 &key_extractor1, const KeyExtractor2 &key_extractor2, const JoinFunction &join_function, const HashFunction &hash_function=HashFunction())
	Performs an inner join between this DIA and the DIA given in the first parameter. More...
template<typename Comparator , typename FirstDIA , typename... DIAs>
auto	Merge (const Comparator &comparator, const FirstDIA &first_dia, const DIAs &... dias)
	Merge is a DOp, which merges any number of sorted DIAs to a single sorted DIA. More...
template<typename ZipFunction , typename FirstDIAType , typename FirstDIAStack , typename... DIAs>
auto	Zip (const ZipFunction &zip_function, const DIA< FirstDIAType, FirstDIAStack > &first_dia, const DIAs &... dias)
	Zips two DIAs of equal size in style of functional programming by applying zip_function to the i-th elements of both input DIAs to form the i-th element of the output DIA. More...
template<typename ZipFunction , typename FirstDIAType , typename FirstDIAStack , typename... DIAs>
auto	Zip (struct CutTag, const ZipFunction &zip_function, const DIA< FirstDIAType, FirstDIAStack > &first_dia, const DIAs &... dias)
	Zips any number of DIAs of equal size in style of functional programming by applying zip_function to the i-th elements of both input DIAs to form the i-th element of the output DIA. More...
template<typename ZipFunction , typename FirstDIAType , typename FirstDIAStack , typename... DIAs>
auto	Zip (struct PadTag, const ZipFunction &zip_function, const DIA< FirstDIAType, FirstDIAStack > &first_dia, const DIAs &... dias)
	Zips any number of DIAs in style of functional programming by applying zip_function to the i-th elements of both input DIAs to form the i-th element of the output DIA. More...
template<typename ZipFunction , typename FirstDIAType , typename FirstDIAStack , typename... DIAs>
auto	Zip (struct PadTag, const ZipFunction &zip_function, const typename common::FunctionTraits< ZipFunction >::args_tuple_plain &padding, const DIA< FirstDIAType, FirstDIAStack > &first_dia, const DIAs &... dias)
	Zips any number of DIAs in style of functional programming by applying zip_function to the i-th elements of both input DIAs to form the i-th element of the output DIA. More...
template<typename ZipFunction , typename FirstDIAType , typename FirstDIAStack , typename... DIAs>
auto	Zip (struct NoRebalanceTag, const ZipFunction &zip_function, const DIA< FirstDIAType, FirstDIAStack > &first_dia, const DIAs &... dias)
	Zips any number of DIAs in style of functional programming by applying zip_function to the i-th elements of both input DIAs to form the i-th element of the output DIA. More...
template<typename ZipFunction , typename FirstDIAType , typename FirstDIAStack , typename... DIAs>
auto	ZipWindow (const std::array< size_t, 1+sizeof ...(DIAs)> &window_size, const ZipFunction &zip_function, const DIA< FirstDIAType, FirstDIAStack > &first_dia, const DIAs &... dias)
	Zips two DIAs of equal size in style of functional programming by applying zip_function to the i-th fixed-sized windows of both input DIAs to form the i-th element of the output DIA. More...
template<typename ZipFunction , typename FirstDIAType , typename FirstDIAStack , typename... DIAs>
auto	ZipWindow (struct CutTag, const std::array< size_t, 1+sizeof ...(DIAs)> &window_size, const ZipFunction &zip_function, const DIA< FirstDIAType, FirstDIAStack > &first_dia, const DIAs &... dias)
	Zips two DIAs of equal size in style of functional programming by applying zip_function to the i-th fixed-sized windows of both input DIAs to form the i-th element of the output DIA. More...
template<typename ZipFunction , typename FirstDIAType , typename FirstDIAStack , typename... DIAs>
auto	ZipWindow (struct PadTag, const std::array< size_t, 1+sizeof ...(DIAs)> &window_size, const ZipFunction &zip_function, const typename ZipWindowTraits< ZipFunction >::value_type_tuple_plain &padding, const DIA< FirstDIAType, FirstDIAStack > &first_dia, const DIAs &... dias)
	Zips two DIAs of equal size in style of functional programming by applying zip_function to the i-th fixed-sized windows of both input DIAs to form the i-th element of the output DIA. More...
template<typename ZipFunction , typename FirstDIAType , typename FirstDIAStack , typename... DIAs>
auto	ZipWindow (struct PadTag, const std::array< size_t, 1+sizeof ...(DIAs)> &window_size, const ZipFunction &zip_function, const DIA< FirstDIAType, FirstDIAStack > &first_dia, const DIAs &... dias)
	Zips two DIAs of equal size in style of functional programming by applying zip_function to the i-th fixed-sized windows of both input DIAs to form the i-th element of the output DIA. More...
template<typename ZipFunction , typename FirstDIAType , typename FirstDIAStack , typename... DIAs>
auto	ZipWindow (struct ArrayTag, struct PadTag, const std::array< size_t, 1+sizeof ...(DIAs)> &window_size, const ZipFunction &zip_function, const typename ZipWindowTraits< ZipFunction >::value_type_tuple_plain &padding, const DIA< FirstDIAType, FirstDIAStack > &first_dia, const DIAs &... dias)
	Zips two DIAs of equal size in style of functional programming by applying zip_function to the i-th fixed-sized windows of both input DIAs to form the i-th element of the output DIA. More...
template<typename ZipFunction , typename FirstDIAType , typename FirstDIAStack , typename... DIAs>
auto	ZipWindow (struct ArrayTag, struct PadTag, const std::array< size_t, 1+sizeof ...(DIAs)> &window_size, const ZipFunction &zip_function, const DIA< FirstDIAType, FirstDIAStack > &first_dia, const DIAs &... dias)
	Zips two DIAs of equal size in style of functional programming by applying zip_function to the i-th fixed-sized windows of both input DIAs to form the i-th element of the output DIA. More...

Detailed Description

Function Documentation

Concat() [1/3]

auto thrill::api::Concat	(	const FirstDIA &	first_dia,
		const DIAs &...	dias
	)

Concat is a DOp, which concatenates any number of DIAs to a single DIA.

All input DIAs must contain the same type, which is also the output DIA's type.

The concat operation balances all input data, so that each worker will have an equal number of elements when the concat completes.

Parameters

first_dia	first DIA
dias	DIAs, which are concatd with the first DIA.

Definition at line 331 of file concat.hpp.

References tlx::vexpand().

Referenced by DIA< BfsNode >::Concat(), and DIA< BfsNode >::FlatMap().

Concat() [2/3]

auto thrill::api::Concat

(

const std::initializer_list< DIA< ValueType > > &

dias

)

Concat is a DOp, which concatenates any number of DIAs to a single DIA.

All input DIAs must contain the same type, which is also the output DIA's type.

The concat operation balances all input data, so that each worker will have an equal number of elements when the concat completes.

Parameters

dias	DIAs, which is concatenated.

Definition at line 354 of file concat.hpp.

Concat() [3/3]

auto thrill::api::Concat

(

const std::vector< DIA< ValueType > > &

dias

)

Concat is a DOp, which concatenates any number of DIAs to a single DIA.

All input DIAs must contain the same type, which is also the output DIA's type.

The concat operation balances all input data, so that each worker will have an equal number of elements when the concat completes.

Parameters

dias	DIAs, which is concatenated.

Definition at line 376 of file concat.hpp.

InnerJoin() [1/2]

auto thrill::api::InnerJoin	(	const LocationDetectionFlag< LocationDetectionValue > &	,
		const FirstDIA &	first_dia,
		const SecondDIA &	second_dia,
		const KeyExtractor1 &	key_extractor1,
		const KeyExtractor2 &	key_extractor2,
		const JoinFunction &	join_function,
		const HashFunction &	hash_function = HashFunction()
	)

Performs an inner join between this DIA and the DIA given in the first parameter.

The key from each DIA element is hereby extracted with a key extractor function. All pairs of elements with equal keys from both DIAs are then joined with the join function.

Template Parameters

KeyExtractor1	Type of the key_extractor1 function. This is a function from FirstDIA::ValueType to the key type.
KeyExtractor2	Type of the key_extractor2 function. This is a function from SecondDIA::ValueType to the key type.
JoinFunction	Type of the join_function. This is a function from ValueType and SecondDIA::ValueType to the type of the output DIA.

Parameters

first_dia	First DIA to join.
second_dia	Second DIA to join.
key_extractor1	Key extractor for this DIA
key_extractor2	Key extractor for second DIA
join_function	Join function applied to all equal key pairs
hash_function	If necessary a hash funtion for Key

Definition at line 710 of file inner_join.hpp.

References gen_data::value.

Referenced by examples::triangles::CountTriangles(), thrill::api::InnerJoin(), JoinTPCH4(), and examples::page_rank::PageRankJoin().

InnerJoin() [2/2]

auto thrill::api::InnerJoin	(	const FirstDIA &	first_dia,
		const SecondDIA &	second_dia,
		const KeyExtractor1 &	key_extractor1,
		const KeyExtractor2 &	key_extractor2,
		const JoinFunction &	join_function,
		const HashFunction &	hash_function = HashFunction()
	)

Performs an inner join between this DIA and the DIA given in the first parameter.

The key from each DIA element is hereby extracted with a key extractor function. All pairs of elements with equal keys from both DIAs are then joined with the join function.

Template Parameters

KeyExtractor1	Type of the key_extractor1 function. This is a function from FirstDIA::ValueType to the key type.
KeyExtractor2	Type of the key_extractor2 function. This is a function from SecondDIA::ValueType to the key type.
JoinFunction	Type of the join_function. This is a function from ValueType and SecondDIA::ValueType to the type of the output DIA.

Parameters

first_dia	First DIA to join.
second_dia	Second DIA to join.
key_extractor1	Key extractor for this DIA
key_extractor2	Key extractor for second DIA
join_function	Join function applied to all equal key pairs
hash_function	If necessary a hash funtion for Key

Definition at line 806 of file inner_join.hpp.

References thrill::api::InnerJoin(), and thrill::api::LocationDetectionTag.

Merge()

auto thrill::api::Merge	(	const Comparator &	comparator,
		const FirstDIA &	first_dia,
		const DIAs &...	dias
	)

Merge is a DOp, which merges any number of sorted DIAs to a single sorted DIA.

All input DIAs must be sorted conforming to the given comparator. The type of the output DIA will be the type of this DIA.

The merge operation balances all input data, so that each worker will have an equal number of elements when the merge completes.

Template Parameters

Comparator

Comparator to specify the order of input and output.

Parameters

comparator	Comparator to specify the order of input and output.
first_dia	first DIA
dias	DIAs, which is merged with this DIA.

Definition at line 674 of file merge.hpp.

References gen_data::value, and tlx::vexpand().

Referenced by DIA< BfsNode >::FlatMap(), and DIA< BfsNode >::Merge().

Zip() [1/5]

auto thrill::api::Zip	(	const ZipFunction &	zip_function,
		const DIA< FirstDIAType, FirstDIAStack > &	first_dia,
		const DIAs &...	dias
	)

Zips two DIAs of equal size in style of functional programming by applying zip_function to the i-th elements of both input DIAs to form the i-th element of the output DIA.

The type of the output DIA can be inferred from the zip_function.

The two input DIAs are required to be of equal size, otherwise use the CutTag variant.

Template Parameters

ZipFunction

Type of the zip_function. This is a function with two input elements, both of the local type, and one output element, which is the type of the Zip node.

Parameters

zip_function	Zip function, which zips two elements together
first_dia	the initial DIA.
dias	DIAs, which is zipped together with the original DIA.

Definition at line 426 of file zip.hpp.

References DIA< ValueType_, Stack_ >::AssertValid(), gen_data::value, and tlx::vexpand().

Referenced by BFSNextLevel(), examples::suffix_sorting::CheckSA(), examples::suffix_sorting::ConstructBWT(), DIA< BfsNode >::FlatMap(), and DIA< BfsNode >::Zip().

Zip() [2/5]

auto thrill::api::Zip	(	struct CutTag	,
		const ZipFunction &	zip_function,
		const DIA< FirstDIAType, FirstDIAStack > &	first_dia,
		const DIAs &...	dias
	)

Zips any number of DIAs of equal size in style of functional programming by applying zip_function to the i-th elements of both input DIAs to form the i-th element of the output DIA.

The type of the output DIA can be inferred from the zip_function.

If the two input DIAs are of unequal size, the result is the shorter of both. Otherwise use the PadTag variant.

Template Parameters

ZipFunction

Type of the zip_function. This is a function with two input elements, both of the local type, and one output element, which is the type of the Zip node.

Parameters

zip_function	Zip function, which zips two elements together
first_dia	the initial DIA.
dias	DIAs, which is zipped together with the original DIA.

Definition at line 481 of file zip.hpp.

References DIA< ValueType_, Stack_ >::AssertValid(), gen_data::value, and tlx::vexpand().

Zip() [3/5]

auto thrill::api::Zip	(	struct PadTag	,
		const ZipFunction &	zip_function,
		const DIA< FirstDIAType, FirstDIAStack > &	first_dia,
		const DIAs &...	dias
	)

Zips any number of DIAs in style of functional programming by applying zip_function to the i-th elements of both input DIAs to form the i-th element of the output DIA.

The type of the output DIA can be inferred from the zip_function.

The output DIA's length is the maximum of all input DIAs, shorter DIAs are padded with default-constructed items.

Template Parameters

ZipFunction

Type of the zip_function. This is a function with two input elements, both of the local type, and one output element, which is the type of the Zip node.

Parameters

zip_function	Zip function, which zips two elements together
first_dia	the initial DIA.
dias	DIAs, which is zipped together with the first DIA.

Definition at line 537 of file zip.hpp.

References DIA< ValueType_, Stack_ >::AssertValid(), gen_data::value, and tlx::vexpand().

Zip() [4/5]

auto thrill::api::Zip	(	struct PadTag	,
		const ZipFunction &	zip_function,
		const typename common::FunctionTraits< ZipFunction >::args_tuple_plain &	padding,
		const DIA< FirstDIAType, FirstDIAStack > &	first_dia,
		const DIAs &...	dias
	)

Zips any number of DIAs in style of functional programming by applying zip_function to the i-th elements of both input DIAs to form the i-th element of the output DIA.

The type of the output DIA can be inferred from the zip_function.

The output DIA's length is the maximum of all input DIAs, shorter DIAs are padded with items given by the padding parameter.

Template Parameters

ZipFunction

Type of the zip_function. This is a function with two input elements, both of the local type, and one output element, which is the type of the Zip node.

Parameters

zip_function	Zip function, which zips two elements together
padding	std::tuple<args> of padding sentinels delivered to ZipFunction if an input dia is too short.
first_dia	the initial DIA.
dias	DIAs, which is zipped together with the original DIA.

Definition at line 596 of file zip.hpp.

References DIA< ValueType_, Stack_ >::AssertValid(), thrill::mem::padding, gen_data::value, and tlx::vexpand().

Zip() [5/5]

auto thrill::api::Zip	(	struct NoRebalanceTag	,
		const ZipFunction &	zip_function,
		const DIA< FirstDIAType, FirstDIAStack > &	first_dia,
		const DIAs &...	dias
	)

Zips any number of DIAs in style of functional programming by applying zip_function to the i-th elements of both input DIAs to form the i-th element of the output DIA.

The type of the output DIA can be inferred from the zip_function.

In this variant, the DIA partitions on all PEs must have matching length. No rebalancing is performed, and the program will die if any partition mismatches. This enables Zip to proceed without any communication.

Template Parameters

ZipFunction

Type of the zip_function. This is a function with two input elements, both of the local type, and one output element, which is the type of the Zip node.

Parameters

zip_function	Zip function, which zips two elements together
first_dia	the initial DIA.
dias	DIAs, which is zipped together with the original DIA.

Definition at line 652 of file zip.hpp.

References DIA< ValueType_, Stack_ >::AssertValid(), gen_data::value, and tlx::vexpand().

ZipWindow() [1/6]

auto thrill::api::ZipWindow	(	const std::array< size_t, 1+sizeof ...(DIAs)> &	window_size,
		const ZipFunction &	zip_function,
		const DIA< FirstDIAType, FirstDIAStack > &	first_dia,
		const DIAs &...	dias
	)

Zips two DIAs of equal size in style of functional programming by applying zip_function to the i-th fixed-sized windows of both input DIAs to form the i-th element of the output DIA.

The input DIAs length must be multiples of the corresponding window size. The type of the output DIA can be inferred from the zip_function.

The two input DIAs are required to be of equal window multiples, otherwise use the CutTag variant.

Definition at line 621 of file zip_window.hpp.

References DIA< ValueType_, Stack_ >::AssertValid(), gen_data::value, and tlx::vexpand().

Referenced by examples::suffix_sorting::DC3Recursive(), examples::suffix_sorting::DC7Recursive(), and thrill::api::ZipWindow().

ZipWindow() [2/6]

auto thrill::api::ZipWindow	(	struct CutTag	,
		const std::array< size_t, 1+sizeof ...(DIAs)> &	window_size,
		const ZipFunction &	zip_function,
		const DIA< FirstDIAType, FirstDIAStack > &	first_dia,
		const DIAs &...	dias
	)

Zips two DIAs of equal size in style of functional programming by applying zip_function to the i-th fixed-sized windows of both input DIAs to form the i-th element of the output DIA.

The input DIAs length must be multiples of the corresponding window size. The type of the output DIA can be inferred from the zip_function.

If the two input DIAs are of unequal size, the result is the shorter of both. Otherwise use the PadTag variant.

Definition at line 667 of file zip_window.hpp.

References DIA< ValueType_, Stack_ >::AssertValid(), gen_data::value, and tlx::vexpand().

ZipWindow() [3/6]

auto thrill::api::ZipWindow	(	struct PadTag	,
		const std::array< size_t, 1+sizeof ...(DIAs)> &	window_size,
		const ZipFunction &	zip_function,
		const typename ZipWindowTraits< ZipFunction >::value_type_tuple_plain &	padding,
		const DIA< FirstDIAType, FirstDIAStack > &	first_dia,
		const DIAs &...	dias
	)

Zips two DIAs of equal size in style of functional programming by applying zip_function to the i-th fixed-sized windows of both input DIAs to form the i-th element of the output DIA.

The input DIAs length must be multiples of the corresponding window size. The type of the output DIA can be inferred from the zip_function.

The output DIA's length is the maximum of all input DIAs, shorter DIAs are padded with items given by the padding parameter.

Definition at line 714 of file zip_window.hpp.

References DIA< ValueType_, Stack_ >::AssertValid(), thrill::mem::padding, gen_data::value, and tlx::vexpand().

ZipWindow() [4/6]

auto thrill::api::ZipWindow	(	struct PadTag	,
		const std::array< size_t, 1+sizeof ...(DIAs)> &	window_size,
		const ZipFunction &	zip_function,
		const DIA< FirstDIAType, FirstDIAStack > &	first_dia,
		const DIAs &...	dias
	)

Zips two DIAs of equal size in style of functional programming by applying zip_function to the i-th fixed-sized windows of both input DIAs to form the i-th element of the output DIA.

The input DIAs length must be multiples of the corresponding window size. The type of the output DIA can be inferred from the zip_function.

The output DIA's length is the maximum of all input DIAs, shorter DIAs are padded with default-constructed items.

Definition at line 760 of file zip_window.hpp.

References thrill::api::ZipWindow().

ZipWindow() [5/6]

auto thrill::api::ZipWindow	(	struct ArrayTag	,
		struct PadTag	,
		const std::array< size_t, 1+sizeof ...(DIAs)> &	window_size,
		const ZipFunction &	zip_function,
		const typename ZipWindowTraits< ZipFunction >::value_type_tuple_plain &	padding,
		const DIA< FirstDIAType, FirstDIAStack > &	first_dia,
		const DIAs &...	dias
	)

Zips two DIAs of equal size in style of functional programming by applying zip_function to the i-th fixed-sized windows of both input DIAs to form the i-th element of the output DIA.

The input DIAs length must be multiples of the corresponding window size. The type of the output DIA can be inferred from the zip_function.

The output DIA's length is the maximum of all input DIAs, shorter DIAs are padded with items given by the padding parameter.

Definition at line 789 of file zip_window.hpp.

References DIA< ValueType_, Stack_ >::AssertValid(), thrill::mem::padding, and tlx::vexpand().

ZipWindow() [6/6]

auto thrill::api::ZipWindow	(	struct ArrayTag	,
		struct PadTag	,
		const std::array< size_t, 1+sizeof ...(DIAs)> &	window_size,
		const ZipFunction &	zip_function,
		const DIA< FirstDIAType, FirstDIAStack > &	first_dia,
		const DIAs &...	dias
	)

Zips two DIAs of equal size in style of functional programming by applying zip_function to the i-th fixed-sized windows of both input DIAs to form the i-th element of the output DIA.

The input DIAs length must be multiples of the corresponding window size. The type of the output DIA can be inferred from the zip_function.

The output DIA's length is the maximum of all input DIAs, shorter DIAs are padded with default constructed items.

Definition at line 836 of file zip_window.hpp.

References thrill::api::ArrayTag, and thrill::api::ZipWindow().