Thrill  0.1
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
ReduceTable< ValueType, Key, Value, KeyExtractor, ReduceFunction, Emitter, VolatileKey, ReduceConfig_, IndexFunction, KeyEqualFunction > Class Template Reference

Detailed Description

template<typename ValueType, typename Key, typename Value, typename KeyExtractor, typename ReduceFunction, typename Emitter, const bool VolatileKey, typename ReduceConfig_, typename IndexFunction, typename KeyEqualFunction = std::equal_to<Key>>
class thrill::core::ReduceTable< ValueType, Key, Value, KeyExtractor, ReduceFunction, Emitter, VolatileKey, ReduceConfig_, IndexFunction, KeyEqualFunction >

Common super-class for bucket and linear-probing hash/reduce tables.

It contains partitioning parameters, statistics, and the output files.

Definition at line 100 of file reduce_table.hpp.

+ Inheritance diagram for ReduceTable< ValueType, Key, Value, KeyExtractor, ReduceFunction, Emitter, VolatileKey, ReduceConfig_, IndexFunction, KeyEqualFunction >:

#include <reduce_table.hpp>

Public Types

using MakeTableItem = ReduceMakeTableItem< Value, TableItem, VolatileKey >
 
using ReduceConfig = ReduceConfig_
 
using TableItem = typename std::conditional< VolatileKey, std::pair< Key, Value >, Value >::type
 

Public Member Functions

 ReduceTable (Context &ctx, size_t dia_id, const KeyExtractor &key_extractor, const ReduceFunction &reduce_function, Emitter &emitter, size_t num_partitions, const ReduceConfig &config, bool immediate_flush, const IndexFunction &index_function, const KeyEqualFunction &key_equal_function)
 
 ReduceTable (const ReduceTable &)=delete
 non-copyable: delete copy-constructor More...
 
void Dispose ()
 Deallocate memory. More...
 
void InitializeSkip ()
 Initialize table for SkipPreReducePhase. More...
 
ReduceTableoperator= (const ReduceTable &)=delete
 non-copyable: delete assignment operator More...
 
Accessors
Context & ctx () const
 Returns the context. More...
 
size_t dia_id () const
 Returns dia_id_. More...
 
const KeyExtractor & key_extractor () const
 Returns the key_extractor. More...
 
const ReduceFunction & reduce_function () const
 Returns the reduce_function. More...
 
const Emitter & emitter () const
 Returns emitter_. More...
 
const IndexFunction & index_function () const
 Returns index_function_. More...
 
IndexFunction & index_function ()
 Returns index_function_ (mutable) More...
 
const KeyEqualFunction & key_equal_function () const
 Returns key_equal_function_. More...
 
std::vector< data::File > & partition_files ()
 Returns the vector of partition files. More...
 
size_t num_partitions ()
 Returns the number of partitions. More...
 
size_t num_buckets () const
 Returns num_buckets_. More...
 
size_t num_buckets_per_partition () const
 Returns num_buckets_per_partition_. More...
 
size_t limit_memory_bytes () const
 Returns limit_memory_bytes_. More...
 
size_t limit_items_per_partition () const
 Returns limit_items_per_partition_. More...
 
size_t items_per_partition (size_t id) const
 Returns items_per_partition_. More...
 
size_t num_items () const
 Returns the total num of items in the table. More...
 
size_t num_items_calc () const
 Returns the total num of items in the table. More...
 
common::Range key_range (size_t partition_id)
 calculate key range for the given output partition More...
 
bool has_spilled_data () const
 returns whether and partition has spilled data into external memory. More...
 
bool has_spilled_data_on_partition (size_t partition_id)
 
Switches for VolatileKey
Key key (const TableItem &t) const
 
TableItem reduce (const TableItem &a, const TableItem &b) const
 
IndexFunction::Result calculate_index (const TableItem &kv) const
 

Static Public Attributes

static constexpr bool debug = false
 

Protected Attributes

Context & ctx_
 Context. More...
 
size_t dia_id_
 Associated DIA id. More...
 
Emitter & emitter_
 Emitter object to receive items outputted to next phase. More...
 
IndexFunction index_function_
 Index Calculation functions: Hash or ByIndex. More...
 
KeyEqualFunction key_equal_function_
 Comparator function for keys. More...
 
KeyExtractor key_extractor_
 Key extractor function for extracting a key from a value. More...
 
std::vector< data::Filepartition_files_
 Store the files for partitions. More...
 
ReduceFunction reduce_function_
 Reduce function for reducing two values. More...
 
Fixed Operational Parameters
const size_t num_partitions_
 Number of partitions. More...
 
ReduceConfig config_
 config of reduce table More...
 
size_t num_buckets_
 
size_t num_buckets_per_partition_
 Partition size, the number of buckets per partition. More...
 
size_t limit_memory_bytes_ = 0
 Size of the table in bytes. More...
 
size_t limit_items_per_partition_
 Number of items in a partition before the partition is spilled. More...
 
bool immediate_flush_
 
Current Statistical Parameters
size_t num_items_ = 0
 Current number of items. More...
 
std::vector< size_t > items_per_partition_
 Current number of items per partition. More...
 

Member Typedef Documentation

using MakeTableItem = ReduceMakeTableItem<Value, TableItem, VolatileKey>

Definition at line 109 of file reduce_table.hpp.

using ReduceConfig = ReduceConfig_

Definition at line 105 of file reduce_table.hpp.

using TableItem = typename std::conditional< VolatileKey, std::pair<Key, Value>, Value>::type

Definition at line 108 of file reduce_table.hpp.

Constructor & Destructor Documentation

ReduceTable ( Context &  ctx,
size_t  dia_id,
const KeyExtractor &  key_extractor,
const ReduceFunction &  reduce_function,
Emitter &  emitter,
size_t  num_partitions,
const ReduceConfig config,
bool  immediate_flush,
const IndexFunction &  index_function,
const KeyEqualFunction &  key_equal_function 
)
inline

Definition at line 111 of file reduce_table.hpp.

ReduceTable ( const ReduceTable< ValueType, Key, Value, KeyExtractor, ReduceFunction, Emitter, VolatileKey, ReduceConfig_, IndexFunction, KeyEqualFunction > &  )
delete

non-copyable: delete copy-constructor

Member Function Documentation

IndexFunction::Result calculate_index ( const TableItem kv) const
inline

Definition at line 265 of file reduce_table.hpp.

Context& ctx ( ) const
inline

Returns the context.

Definition at line 169 of file reduce_table.hpp.

size_t dia_id ( ) const
inline

Returns dia_id_.

Definition at line 172 of file reduce_table.hpp.

void Dispose ( )
inline

Deallocate memory.

Definition at line 150 of file reduce_table.hpp.

const Emitter& emitter ( ) const
inline

Returns emitter_.

Definition at line 181 of file reduce_table.hpp.

bool has_spilled_data ( ) const
inline

returns whether and partition has spilled data into external memory.

Definition at line 241 of file reduce_table.hpp.

bool has_spilled_data_on_partition ( size_t  partition_id)
inline

Definition at line 248 of file reduce_table.hpp.

const IndexFunction& index_function ( ) const
inline
IndexFunction& index_function ( )
inline

Returns index_function_ (mutable)

Definition at line 187 of file reduce_table.hpp.

void InitializeSkip ( )
inline

Initialize table for SkipPreReducePhase.

Definition at line 156 of file reduce_table.hpp.

size_t items_per_partition ( size_t  id) const
inline

Returns items_per_partition_.

Definition at line 214 of file reduce_table.hpp.

const KeyEqualFunction& key_equal_function ( ) const
inline

Returns key_equal_function_.

Definition at line 190 of file reduce_table.hpp.

const KeyExtractor& key_extractor ( ) const
inline

Returns the key_extractor.

Definition at line 175 of file reduce_table.hpp.

common::Range key_range ( size_t  partition_id)
inline

calculate key range for the given output partition

Definition at line 235 of file reduce_table.hpp.

size_t limit_items_per_partition ( ) const
inline

Returns limit_items_per_partition_.

Definition at line 210 of file reduce_table.hpp.

size_t limit_memory_bytes ( ) const
inline

Returns limit_memory_bytes_.

Definition at line 207 of file reduce_table.hpp.

size_t num_buckets ( ) const
inline

Returns num_buckets_.

Definition at line 200 of file reduce_table.hpp.

size_t num_buckets_per_partition ( ) const
inline

Returns num_buckets_per_partition_.

Definition at line 203 of file reduce_table.hpp.

size_t num_items ( ) const
inline
size_t num_items_calc ( ) const
inline

Returns the total num of items in the table.

Definition at line 225 of file reduce_table.hpp.

size_t num_partitions ( )
inline

Returns the number of partitions.

Definition at line 197 of file reduce_table.hpp.

ReduceTable& operator= ( const ReduceTable< ValueType, Key, Value, KeyExtractor, ReduceFunction, Emitter, VolatileKey, ReduceConfig_, IndexFunction, KeyEqualFunction > &  )
delete

non-copyable: delete assignment operator

std::vector<data::File>& partition_files ( )
inline

Returns the vector of partition files.

Definition at line 194 of file reduce_table.hpp.

TableItem reduce ( const TableItem a,
const TableItem b 
) const
inline

Definition at line 261 of file reduce_table.hpp.

const ReduceFunction& reduce_function ( ) const
inline

Returns the reduce_function.

Definition at line 178 of file reduce_table.hpp.

Member Data Documentation

ReduceConfig config_
protected

config of reduce table

Definition at line 304 of file reduce_table.hpp.

constexpr bool debug = false
static

Definition at line 103 of file reduce_table.hpp.

Emitter& emitter_
protected
bool immediate_flush_
protected

Whether to spill overfull partitions to disk or to immediately flush to next phase.

Definition at line 321 of file reduce_table.hpp.

Referenced by ReduceTable< TableItem, Key, Value, KeyExtractor, ReduceFunction, Emitter, VolatileKey, ReduceConfig_, IndexFunction, KeyEqualFunction >::ReduceTable().

KeyEqualFunction key_equal_function_
protected
size_t limit_items_per_partition_
protected

The documentation for this class was generated from the following file: