Actions
size_t	Size () const
	Computes the total size of all elements across all workers. More...

Future< size_t >	SizeFuture () const
	Lazily computes the total size of all elements across all workers. More...

std::vector< ValueType >	AllGather () const
	Returns the whole DIA in an std::vector on each worker. More...

void	AllGather (std::vector< ValueType > *out_vector) const
	AllGather is an Action, which returns the whole DIA in an std::vector on each worker. This is only for testing purposes and should not be used on large datasets. More...

Future< std::vector< ValueType > >	AllGatherFuture () const
	Returns the whole DIA in an std::vector on each worker. More...

void	Print (const std::string &name=std::string()) const
	Print is an Action, which collects all data of the DIA at the worker 0 and prints using ostream serialization. More...

void	Print (const std::string &name, std::ostream &out) const
	Print is an Action, which collects all data of the DIA at the worker 0 and prints using ostream serialization. More...

std::vector< ValueType >	Gather (size_t target_id=0) const
	Gather is an Action, which collects all data of the DIA into a vector at the given worker. More...

void	Gather (size_t target_id, std::vector< ValueType > *out_vector) const
	Gather is an Action, which collects all data of the DIA into a vector at the given worker. More...

template<typename ReduceFunction >
ValueType	AllReduce (const ReduceFunction &reduce_function) const
	AllReduce is an Action, which computes the reduction sum of all elements globally and delivers the same value on all workers. More...

template<typename ReduceFunction >
ValueType	AllReduce (const ReduceFunction &reduce_function, const ValueType &initial_value) const
	AllReduce is an Action, which computes the reduction sum of all elements globally and delivers the same value on all workers. More...

template<typename ReduceFunction >
Future< ValueType >	AllReduceFuture (const ReduceFunction &reduce_function) const
	AllReduce is an ActionFuture, which computes the reduction sum of all elements globally and delivers the same value on all workers. More...

template<typename ReduceFunction >
Future< ValueType >	AllReduceFuture (const ReduceFunction &reduce_function, const ValueType &initial_value) const
	AllReduce is an ActionFuture, which computes the reduction sum of all elements globally and delivers the same value on all workers. More...

template<typename SumFunction = std::plus<ValueType>>
ValueType	Sum (const SumFunction &sum_function=SumFunction()) const
	Sum is an Action, which computes the sum of all elements globally. More...

template<typename SumFunction = std::plus<ValueType>>
ValueType	Sum (const SumFunction &sum_function, const ValueType &initial_value) const
	Sum is an Action, which computes the sum of all elements globally. More...

template<typename SumFunction = std::plus<ValueType>>
Future< ValueType >	SumFuture (const SumFunction &sum_function=SumFunction(), const ValueType &initial_value=ValueType()) const
	Sum is an ActionFuture, which computes the sum of all elements globally. More...

ValueType	Min () const
	Min is an Action, which computes the minimum of all elements globally. More...

ValueType	Min (const ValueType &initial_value) const
	Min is an Action, which computes the minimum of all elements globally. More...

Future< ValueType >	MinFuture () const
	Min is an ActionFuture, which computes the minimum of all elements globally. More...

Future< ValueType >	MinFuture (const ValueType &initial_value) const
	Min is an ActionFuture, which computes the minimum of all elements globally. More...

ValueType	Max () const
	Max is an Action, which computes the maximum of all elements globally. More...

ValueType	Max (const ValueType &initial_value) const
	Max is an Action, which computes the maximum of all elements globally. More...

Future< ValueType >	MaxFuture () const
	Max is an ActionFuture, which computes the maximum of all elements globally. More...

Future< ValueType >	MaxFuture (const ValueType &initial_value) const
	Max is an ActionFuture, which computes the maximum of all elements globally. More...

template<size_t p>
double	HyperLogLog () const
	Compute the approximate number of distinct elements in the DIA. More...

void	WriteLinesOne (const std::string &filepath) const
	WriteLinesOne is an Action, which writes std::strings to a single output file. More...

Future< void >	WriteLinesOneFuture (const std::string &filepath) const
	WriteLinesOne is an ActionFuture, which writes std::strings to a single output file. More...

void	WriteLines (const std::string &filepath, size_t target_file_size=128 1024 1024) const
	WriteLines is an Action, which writes std::strings to multiple output files. More...

Future< void >	WriteLinesFuture (const std::string &filepath, size_t target_file_size=128 1024 1024) const
	WriteLines is an ActionFuture, which writes std::strings to multiple output files. More...

void	WriteBinary (const std::string &filepath, size_t max_file_size=128 1024 1024) const
	WriteBinary is a function, which writes a DIA to many files per worker. More...

Future< void >	WriteBinaryFuture (const std::string &filepath, size_t max_file_size=128 1024 1024) const
	WriteBinary is a function, which writes a DIA to many files per worker. More...

auto	Sample (size_t sample_size) const
	Select up to sample_size items uniformly at random and return a new DIA<T>. More...

Detailed Description

Function Documentation

◆ AllGather() [1/2]

std::vector< ValueType > AllGather ( ) const

Returns the whole DIA in an std::vector on each worker.

This is only for testing purposes and should not be used on large datasets.

Definition at line 114 of file all_gather.hpp.

Referenced by Process().

◆ AllGather() [2/2]

void AllGather ( std::vector< ValueType > * out_vector ) const

AllGather is an Action, which returns the whole DIA in an std::vector on each worker. This is only for testing purposes and should not be used on large datasets.

Definition at line 130 of file all_gather.hpp.

◆ AllGatherFuture()

Future< std::vector< ValueType > > AllGatherFuture ( ) const

Returns the whole DIA in an std::vector on each worker.

This is only for testing purposes and should not be used on large datasets.

Definition at line 143 of file all_gather.hpp.

◆ AllReduce() [1/2]

ValueType AllReduce ( const ReduceFunction & reduce_function ) const

AllReduce is an Action, which computes the reduction sum of all elements globally and delivers the same value on all workers.

Parameters

reduce_function Reduce function.

Definition at line 89 of file all_reduce.hpp.

◆ AllReduce() [2/2]

ValueType AllReduce	(	const ReduceFunction &	reduce_function,
		const ValueType &	initial_value
	)		const

AllReduce is an Action, which computes the reduction sum of all elements globally and delivers the same value on all workers.

Parameters

reduce_function	Reduce function.
initial_value	Initial value of the reduction.

Definition at line 124 of file all_reduce.hpp.

◆ AllReduceFuture() [1/2]

Future< ValueType > AllReduceFuture ( const ReduceFunction & reduce_function ) const

AllReduce is an ActionFuture, which computes the reduction sum of all elements globally and delivers the same value on all workers.

Parameters

reduce_function Reduce function.

Definition at line 159 of file all_reduce.hpp.

◆ AllReduceFuture() [2/2]

Future< ValueType > AllReduceFuture	(	const ReduceFunction &	reduce_function,
		const ValueType &	initial_value
	)		const

AllReduce is an ActionFuture, which computes the reduction sum of all elements globally and delivers the same value on all workers.

Parameters

reduce_function	Reduce function.
initial_value	Initial value of the reduction.

Definition at line 192 of file all_reduce.hpp.

◆ Gather() [1/2]

std::vector< ValueType > Gather ( size_t target_id = 0 ) const

Gather is an Action, which collects all data of the DIA into a vector at the given worker.

This should only be done if the received data can fit into RAM of the one worker.

Definition at line 93 of file gather.hpp.

Referenced by examples::select::Select().

◆ Gather() [2/2]

void Gather	(	size_t	target_id,
		std::vector< ValueType > *	out_vector
	)		const

Gather is an Action, which collects all data of the DIA into a vector at the given worker.

This should only be done if the received data can fit into RAM of the one worker.

Definition at line 109 of file gather.hpp.

◆ HyperLogLog()

double HyperLogLog ( ) const

Compute the approximate number of distinct elements in the DIA.

Template Parameters

p	Number of bits to use for index. Should be between 4 and 16.

Definition at line 64 of file hyperloglog.hpp.

◆ Max() [1/2]

ValueType Max ( ) const

Max is an Action, which computes the maximum of all elements globally.

Definition at line 22 of file max.hpp.

◆ Max() [2/2]

ValueType Max ( const ValueType & initial_value ) const

Max is an Action, which computes the maximum of all elements globally.

Parameters

initial_value Initial value of the max.

Definition at line 32 of file max.hpp.

◆ MaxFuture() [1/2]

Future< ValueType > MaxFuture ( ) const

Max is an ActionFuture, which computes the maximum of all elements globally.

Definition at line 43 of file max.hpp.

◆ MaxFuture() [2/2]

Future< ValueType > MaxFuture ( const ValueType & initial_value ) const

Max is an ActionFuture, which computes the maximum of all elements globally.

Parameters

initial_value Initial value of the max.

Definition at line 52 of file max.hpp.

◆ Min() [1/2]

ValueType Min ( ) const

Min is an Action, which computes the minimum of all elements globally.

Definition at line 22 of file min.hpp.

◆ Min() [2/2]

ValueType Min ( const ValueType & initial_value ) const

Min is an Action, which computes the minimum of all elements globally.

Parameters

initial_value Initial value of the min.

Definition at line 32 of file min.hpp.

◆ MinFuture() [1/2]

Future< ValueType > MinFuture ( ) const

Min is an ActionFuture, which computes the minimum of all elements globally.

Definition at line 43 of file min.hpp.

◆ MinFuture() [2/2]

Future< ValueType > MinFuture ( const ValueType & initial_value ) const

Min is an ActionFuture, which computes the minimum of all elements globally.

Parameters

initial_value Initial value of the min.

Definition at line 52 of file min.hpp.

◆ Print() [1/2]

void Print ( const std::string & name = std::string() ) const

Print is an Action, which collects all data of the DIA at the worker 0 and prints using ostream serialization.

It is implemented using Gather().

Definition at line 50 of file print.hpp.

Referenced by examples::suffix_sorting::ConstructWaveletTree(), examples::suffix_sorting::PrefixQuadrupling(), examples::suffix_sorting::PrefixQuadruplingDiscarding(), and Process().

◆ Print() [2/2]

void Print	(	const std::string &	name,
		std::ostream &	out
	)		const

Print is an Action, which collects all data of the DIA at the worker 0 and prints using ostream serialization.

It is implemented using Gather().

Definition at line 24 of file print.hpp.

◆ Sample()

auto Sample ( size_t sample_size ) const

Select up to sample_size items uniformly at random and return a new DIA<T>.

Examples:: examples/tutorial/k-means_step4.cpp.

Definition at line 247 of file sample.hpp.

Referenced by Process().

◆ Size()

size_t Size ( ) const

Computes the total size of all elements across all workers.

Definition at line 87 of file size.hpp.

Referenced by examples::select::Select().

◆ SizeFuture()

Future< size_t > SizeFuture ( ) const

Lazily computes the total size of all elements across all workers.

Definition at line 97 of file size.hpp.

◆ Sum() [1/2]

ValueType Sum ( const SumFunction & sum_function = SumFunction() ) const

Sum is an Action, which computes the sum of all elements globally.

Parameters

sum_function Sum function.

Definition at line 23 of file sum.hpp.

Referenced by PrepareNextTree().

◆ Sum() [2/2]

ValueType Sum	(	const SumFunction &	sum_function,
		const ValueType &	initial_value
	)		const

Sum is an Action, which computes the sum of all elements globally.

Parameters

sum_function	Sum function.
initial_value	Initial value of the sum.

Definition at line 57 of file sum.hpp.

◆ SumFuture()

Future< ValueType > SumFuture	(	const SumFunction &	sum_function = `SumFunction()`,
		const ValueType &	initial_value = `ValueType()`
	)		const

Sum is an ActionFuture, which computes the sum of all elements globally.

Parameters

sum_function	Sum function.
initial_value	Initial value of the sum.

Definition at line 92 of file sum.hpp.

◆ WriteBinary()

void WriteBinary	(	const std::string &	filepath,
		size_t	max_file_size = `128* 1024* 1024`
	)		const

WriteBinary is a function, which writes a DIA to many files per worker.

The input DIA can be recreated with ReadBinary and equal filepath.

Parameters

filepath	Destination of the output file. This filepath must contain two special substrings: `"$$$$$"` is replaced by the worker id and `"#####"` will be replaced by the file chunk id. The last occurrences of `"$"` and `"#"` are replaced, otherwise `"$$$$"` and/or `"##########"` are automatically appended.
max_file_size	size limit of individual file.

Definition at line 191 of file write_binary.hpp.

◆ WriteBinaryFuture()

Future< void > WriteBinaryFuture	(	const std::string &	filepath,
		size_t	max_file_size = `128* 1024* 1024`
	)		const

WriteBinary is a function, which writes a DIA to many files per worker.

The input DIA can be recreated with ReadBinary and equal filepath.

Parameters

filepath	Destination of the output file. This filepath must contain two special substrings: `"$$$$$"` is replaced by the worker id and `"#####"` will be replaced by the file chunk id. The last occurrences of `"$"` and `"#"` are replaced, otherwise `"$$$$"` and/or `"##########"` are automatically appended.
max_file_size	size limit of individual file.

Definition at line 203 of file write_binary.hpp.

◆ WriteLines()

void WriteLines	(	const std::string &	filepath,
		size_t	target_file_size = `128* 1024* 1024`
	)		const

WriteLines is an Action, which writes std::strings to multiple output files.

Strings are written using fstream with a newline after each entry. Each worker creates its individual file.

Parameters

filepath	Destination of the output file. This filepath must contain two special substrings: `"$$$$$"` is replaced by the worker id and `"#####"` will be replaced by the file chunk id. The last occurrences of `"$"` and `"#"` are replaced, otherwise `"$$$$"` and/or `"##########"` are automatically appended.
target_file_size	target size of each individual file.

Definition at line 176 of file write_lines.hpp.

◆ WriteLinesFuture()

Future< void > WriteLinesFuture	(	const std::string &	filepath,
		size_t	target_file_size = `128* 1024* 1024`
	)		const

WriteLines is an ActionFuture, which writes std::strings to multiple output files.

Strings are written using fstream with a newline after each entry. Each worker creates its individual file.

Parameters

filepath	Destination of the output file. This filepath must contain two special substrings: `"$$$$$"` is replaced by the worker id and `"#####"` will be replaced by the file chunk id. The last occurrences of `"$"` and `"#"` are replaced, otherwise `"$$$$"` and/or `"##########"` are automatically appended.
target_file_size	target size of each individual file.

Definition at line 192 of file write_lines.hpp.

◆ WriteLinesOne()

void WriteLinesOne ( const std::string & filepath ) const

WriteLinesOne is an Action, which writes std::strings to a single output file.

Parameters

filepath Destination of the output file.

Definition at line 111 of file write_lines_one.hpp.

◆ WriteLinesOneFuture()

Future< void > WriteLinesOneFuture ( const std::string & filepath ) const

WriteLinesOne is an ActionFuture, which writes std::strings to a single output file.

Parameters

filepath Destination of the output file.

Definition at line 126 of file write_lines_one.hpp.

Actions

Detailed Description

Function Documentation

◆ AllGather() [1/2]

◆ AllGather() [2/2]

◆ AllGatherFuture()

◆ AllReduce() [1/2]

◆ AllReduce() [2/2]

◆ AllReduceFuture() [1/2]

◆ AllReduceFuture() [2/2]

◆ Gather() [1/2]

◆ Gather() [2/2]

◆ HyperLogLog()

◆ Max() [1/2]

◆ Max() [2/2]

◆ MaxFuture() [1/2]

◆ MaxFuture() [2/2]

◆ Min() [1/2]

◆ Min() [2/2]

◆ MinFuture() [1/2]

◆ MinFuture() [2/2]

◆ Print() [1/2]

◆ Print() [2/2]

◆ Sample()

◆ Size()

◆ SizeFuture()

◆ Sum() [1/2]

◆ Sum() [2/2]

◆ SumFuture()

◆ WriteBinary()

◆ WriteBinaryFuture()

◆ WriteLines()

◆ WriteLinesFuture()

◆ WriteLinesOne()

◆ WriteLinesOneFuture()