Detailed Description

template<typename ValueType, typename Key, typename Value, typename KeyExtractor, typename ReduceFunction, typename Emitter, const bool VolatileKey, typename ReduceConfig_, typename IndexFunction, typename KeyEqualFunction = std::equal_to<Key>>
class thrill::core::ReduceTable< ValueType, Key, Value, KeyExtractor, ReduceFunction, Emitter, VolatileKey, ReduceConfig_, IndexFunction, KeyEqualFunction >

Common super-class for bucket and linear-probing hash/reduce tables.

It contains partitioning parameters, statistics, and the output files.

Definition at line 102 of file reduce_table.hpp.

Inheritance diagram for ReduceTable< ValueType, Key, Value, KeyExtractor, ReduceFunction, Emitter, VolatileKey, ReduceConfig_, IndexFunction, KeyEqualFunction >:

Collaboration diagram for ReduceTable< ValueType, Key, Value, KeyExtractor, ReduceFunction, Emitter, VolatileKey, ReduceConfig_, IndexFunction, KeyEqualFunction >:

#include <reduce_table.hpp>

Public Types
using	MakeTableItem = ReduceMakeTableItem< Value, TableItem, VolatileKey >

using	ReduceConfig = ReduceConfig_

using	TableItem = typename std::conditional< VolatileKey, std::pair< Key, Value >, Value >::type

Public Member Functions
	ReduceTable (Context &ctx, size_t dia_id, const KeyExtractor &key_extractor, const ReduceFunction &reduce_function, Emitter &emitter, size_t num_partitions, const ReduceConfig &config, bool immediate_flush, const IndexFunction &index_function, const KeyEqualFunction &key_equal_function)

	ReduceTable (const ReduceTable &)=delete
	non-copyable: delete copy-constructor More...

void	Dispose ()
	Deallocate memory. More...

void	InitializeSkip ()
	Initialize table for SkipPreReducePhase. More...

ReduceTable &	operator= (const ReduceTable &)=delete
	non-copyable: delete assignment operator More...

Accessors
Context &	ctx () const
	Returns the context. More...

size_t	dia_id () const
	Returns dia_id_. More...

const KeyExtractor &	key_extractor () const
	Returns the key_extractor. More...

const ReduceFunction &	reduce_function () const
	Returns the reduce_function. More...

const Emitter &	emitter () const
	Returns emitter_. More...

const IndexFunction &	index_function () const
	Returns index_function_. More...

IndexFunction &	index_function ()
	Returns index_function_ (mutable) More...

const KeyEqualFunction &	key_equal_function () const
	Returns key_equal_function_. More...

std::vector< data::File > &	partition_files ()
	Returns the vector of partition files. More...

size_t	num_partitions ()
	Returns the number of partitions. More...

size_t	num_buckets () const
	Returns num_buckets_. More...

size_t	num_buckets_per_partition () const
	Returns num_buckets_per_partition_. More...

size_t	limit_memory_bytes () const
	Returns limit_memory_bytes_. More...

size_t	limit_items_per_partition () const
	Returns limit_items_per_partition_. More...

size_t	items_per_partition (size_t id) const
	Returns items_per_partition_. More...

size_t	num_items () const
	Returns the total num of items in the table. More...

size_t	num_items_calc () const
	Returns the total num of items in the table. More...

common::Range	key_range (size_t partition_id)
	calculate key range for the given output partition More...

bool	has_spilled_data () const
	returns whether and partition has spilled data into external memory. More...

bool	has_spilled_data_on_partition (size_t partition_id)

Switches for VolatileKey
Key	key (const TableItem &t) const

TableItem	reduce (const TableItem &a, const TableItem &b) const

IndexFunction::Result	calculate_index (const TableItem &kv) const

Static Public Attributes
static constexpr bool	debug = false

Protected Attributes
Context &	ctx_
	Context. More...

size_t	dia_id_
	Associated DIA id. More...

Emitter &	emitter_
	Emitter object to receive items outputted to next phase. More...

IndexFunction	index_function_
	Index Calculation functions: Hash or ByIndex. More...

KeyEqualFunction	key_equal_function_
	Comparator function for keys. More...

KeyExtractor	key_extractor_
	Key extractor function for extracting a key from a value. More...

std::vector< data::File >	partition_files_
	Store the files for partitions. More...

ReduceFunction	reduce_function_
	Reduce function for reducing two values. More...

Fixed Operational Parameters
const size_t	num_partitions_
	Number of partitions. More...

ReduceConfig	config_
	config of reduce table More...

size_t	num_buckets_

size_t	num_buckets_per_partition_
	Partition size, the number of buckets per partition. More...

size_t	limit_memory_bytes_ = 0
	Size of the table in bytes. More...

size_t	limit_items_per_partition_
	Number of items in a partition before the partition is spilled. More...

bool	immediate_flush_

Current Statistical Parameters
size_t	num_items_ = 0
	Current number of items. More...

std::vector< size_t >	items_per_partition_
	Current number of items per partition. More...

Member Typedef Documentation

◆ MakeTableItem

using MakeTableItem = ReduceMakeTableItem<Value, TableItem, VolatileKey>

Definition at line 111 of file reduce_table.hpp.

◆ ReduceConfig

using ReduceConfig = ReduceConfig_

Definition at line 107 of file reduce_table.hpp.

◆ TableItem

using TableItem = typename std::conditional< VolatileKey, std::pair<Key, Value>, Value>::type

Definition at line 110 of file reduce_table.hpp.

Constructor & Destructor Documentation

◆ ReduceTable() [1/2]

ReduceTable	(	Context &	ctx,
		size_t	dia_id,
		const KeyExtractor &	key_extractor,
		const ReduceFunction &	reduce_function,
		Emitter &	emitter,
		size_t	num_partitions,
		const ReduceConfig &	config,
		bool	immediate_flush,
		const IndexFunction &	index_function,
		const KeyEqualFunction &	key_equal_function
	)

inline

Definition at line 113 of file reduce_table.hpp.

◆ ReduceTable() [2/2]

ReduceTable ( const ReduceTable< ValueType, Key, Value, KeyExtractor, ReduceFunction, Emitter, VolatileKey, ReduceConfig_, IndexFunction, KeyEqualFunction > & )

delete

non-copyable: delete copy-constructor

Member Function Documentation

◆ calculate_index()

IndexFunction::Result calculate_index ( const TableItem & kv ) const

inline

Definition at line 267 of file reduce_table.hpp.

◆ ctx()

Context& ctx ( ) const

inline

Returns the context.

Definition at line 171 of file reduce_table.hpp.

◆ dia_id()

size_t dia_id ( ) const

inline

Returns dia_id_.

Definition at line 174 of file reduce_table.hpp.

◆ Dispose()

void Dispose ( )

inline

Deallocate memory.

Definition at line 152 of file reduce_table.hpp.

◆ emitter()

const Emitter& emitter ( ) const

inline

Returns emitter_.

Definition at line 183 of file reduce_table.hpp.

◆ has_spilled_data()

bool has_spilled_data ( ) const

inline

returns whether and partition has spilled data into external memory.

Definition at line 243 of file reduce_table.hpp.

◆ has_spilled_data_on_partition()

bool has_spilled_data_on_partition ( size_t partition_id )

inline

Definition at line 250 of file reduce_table.hpp.

◆ index_function() [1/2]

const IndexFunction& index_function ( ) const

inline

Returns index_function_.

Definition at line 186 of file reduce_table.hpp.

◆ index_function() [2/2]

IndexFunction& index_function ( )

inline

Returns index_function_ (mutable)

Definition at line 189 of file reduce_table.hpp.

◆ InitializeSkip()

void InitializeSkip ( )

inline

Initialize table for SkipPreReducePhase.

Definition at line 158 of file reduce_table.hpp.

◆ items_per_partition()

size_t items_per_partition ( size_t id ) const

inline

Returns items_per_partition_.

Definition at line 216 of file reduce_table.hpp.

◆ key()

Key key ( const TableItem & t ) const

inline

Definition at line 259 of file reduce_table.hpp.

◆ key_equal_function()

const KeyEqualFunction& key_equal_function ( ) const

inline

Returns key_equal_function_.

Definition at line 192 of file reduce_table.hpp.

◆ key_extractor()

const KeyExtractor& key_extractor ( ) const

inline

Returns the key_extractor.

Definition at line 177 of file reduce_table.hpp.

◆ key_range()

common::Range key_range ( size_t partition_id )

inline

calculate key range for the given output partition

Definition at line 237 of file reduce_table.hpp.

◆ limit_items_per_partition()

size_t limit_items_per_partition ( ) const

inline

Returns limit_items_per_partition_.

Definition at line 212 of file reduce_table.hpp.

◆ limit_memory_bytes()

size_t limit_memory_bytes ( ) const

inline

Returns limit_memory_bytes_.

Definition at line 209 of file reduce_table.hpp.

◆ num_buckets()

size_t num_buckets ( ) const

inline

Returns num_buckets_.

Definition at line 202 of file reduce_table.hpp.

◆ num_buckets_per_partition()

size_t num_buckets_per_partition ( ) const

inline

Returns num_buckets_per_partition_.

Definition at line 205 of file reduce_table.hpp.

◆ num_items()

size_t num_items ( ) const

inline

Returns the total num of items in the table.

Definition at line 222 of file reduce_table.hpp.

◆ num_items_calc()

size_t num_items_calc ( ) const

inline

Returns the total num of items in the table.

Definition at line 227 of file reduce_table.hpp.

◆ num_partitions()

size_t num_partitions ( )

inline

Returns the number of partitions.

Definition at line 199 of file reduce_table.hpp.

◆ operator=()

ReduceTable& operator= ( const ReduceTable< ValueType, Key, Value, KeyExtractor, ReduceFunction, Emitter, VolatileKey, ReduceConfig_, IndexFunction, KeyEqualFunction > & )

delete

non-copyable: delete assignment operator

◆ partition_files()

std::vector<data::File>& partition_files ( )

inline

Returns the vector of partition files.

Definition at line 196 of file reduce_table.hpp.

◆ reduce()

TableItem reduce	(	const TableItem &	a,
		const TableItem &	b
	)		const

inline

Definition at line 263 of file reduce_table.hpp.

◆ reduce_function()

const ReduceFunction& reduce_function ( ) const

inline

Returns the reduce_function.

Definition at line 180 of file reduce_table.hpp.

Member Data Documentation

◆ config_

ReduceConfig config_

protected

config of reduce table

Definition at line 306 of file reduce_table.hpp.

◆ ctx_

Context& ctx_

protected

Context.

Definition at line 276 of file reduce_table.hpp.

◆ debug

constexpr bool debug = false

static

Definition at line 105 of file reduce_table.hpp.

◆ dia_id_

size_t dia_id_

protected

Associated DIA id.

Definition at line 279 of file reduce_table.hpp.

◆ emitter_

Emitter& emitter_

protected

Emitter object to receive items outputted to next phase.

Definition at line 288 of file reduce_table.hpp.

◆ immediate_flush_

bool immediate_flush_

protected

Whether to spill overfull partitions to disk or to immediately flush to next phase.

Definition at line 323 of file reduce_table.hpp.

◆ index_function_

IndexFunction index_function_

protected

Index Calculation functions: Hash or ByIndex.

Definition at line 291 of file reduce_table.hpp.

◆ items_per_partition_

std::vector<size_t> items_per_partition_

protected

Current number of items per partition.

Definition at line 334 of file reduce_table.hpp.

◆ key_equal_function_

KeyEqualFunction key_equal_function_

protected

Comparator function for keys.

Definition at line 294 of file reduce_table.hpp.

◆ key_extractor_

KeyExtractor key_extractor_

protected

Key extractor function for extracting a key from a value.

Definition at line 282 of file reduce_table.hpp.

◆ limit_items_per_partition_

size_t limit_items_per_partition_

protected

Number of items in a partition before the partition is spilled.

Definition at line 319 of file reduce_table.hpp.

◆ limit_memory_bytes_

size_t limit_memory_bytes_ = 0

protected

Size of the table in bytes.

Definition at line 316 of file reduce_table.hpp.

◆ num_buckets_

size_t num_buckets_

protected

Size of the table, which is the number of slots / buckets / entries available for items or chains of items.

Definition at line 310 of file reduce_table.hpp.

◆ num_buckets_per_partition_

size_t num_buckets_per_partition_

protected

Partition size, the number of buckets per partition.

Definition at line 313 of file reduce_table.hpp.

◆ num_items_

size_t num_items_ = 0

protected

Current number of items.

Definition at line 331 of file reduce_table.hpp.

◆ num_partitions_

const size_t num_partitions_

protected

Number of partitions.

Definition at line 303 of file reduce_table.hpp.

◆ partition_files_

std::vector<data::File> partition_files_

protected

Store the files for partitions.

Definition at line 297 of file reduce_table.hpp.

◆ reduce_function_

ReduceFunction reduce_function_

protected

Reduce function for reducing two values.

Definition at line 285 of file reduce_table.hpp.

The documentation for this class was generated from the following file:

thrill/core/reduce_table.hpp

Detailed Description

Public Types

Public Member Functions

Static Public Attributes

Protected Attributes

Member Typedef Documentation

◆ MakeTableItem

◆ ReduceConfig

◆ TableItem

Constructor & Destructor Documentation

◆ ReduceTable() [1/2]

◆ ReduceTable() [2/2]

Member Function Documentation

◆ calculate_index()

◆ ctx()

◆ dia_id()

◆ Dispose()

◆ emitter()

◆ has_spilled_data()

◆ has_spilled_data_on_partition()

◆ index_function() [1/2]

◆ index_function() [2/2]

◆ InitializeSkip()

◆ items_per_partition()

◆ key()

◆ key_equal_function()

◆ key_extractor()

◆ key_range()

◆ limit_items_per_partition()

◆ limit_memory_bytes()

◆ num_buckets()

◆ num_buckets_per_partition()

◆ num_items()

◆ num_items_calc()

◆ num_partitions()

◆ operator=()

◆ partition_files()

◆ reduce()

◆ reduce_function()

Member Data Documentation

◆ config_

◆ ctx_

◆ debug

◆ dia_id_

◆ emitter_

◆ immediate_flush_

◆ index_function_

◆ items_per_partition_

◆ key_equal_function_

◆ key_extractor_

◆ limit_items_per_partition_

◆ limit_memory_bytes_

◆ num_buckets_

◆ num_buckets_per_partition_

◆ num_items_

◆ num_partitions_

◆ partition_files_

◆ reduce_function_