Thrill  0.1
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
thrill::common Namespace Reference

Contains many common tools and helpers shared by higher layers. See common. More...

Namespaces

 qsort_local
 

Classes

class  Aggregate
 Calculate running aggregate statistics: feed it with values, and it will keep the minimum, the maximum, the average, the value number, and the standard deviation is values. More...
 
struct  alias_cast_helper
 
class  AtomicMovable
 This is a derivative of std::atomic which enables easier and less error-prone writing of move-only classes by implementing a move constructor. More...
 
class  BinaryHeap
 
class  ComponentSum
 template for computing the component-wise sum of std::array or std::vector. More...
 
class  ComponentSum< std::array< Type, N >, Operation >
 Compute the component-wise sum of two std::array<T,N> of same sizes. More...
 
class  ComponentSum< std::vector< Type >, Operation >
 Compute the component-wise sum of two std::vector<T> of same sizes. More...
 
class  ErrnoException
 An Exception which is thrown on system errors and contains errno information. More...
 
struct  FunctionTraits< ReturnType(*)(Args...)>
 specialize for function pointers More...
 
struct  FunctionTraits< ReturnType(ClassType::*)(Args...) const >
 specialize for pointers to const member function More...
 
struct  FunctionTraits< ReturnType(ClassType::*)(Args...)>
 specialize for pointers to mutable member function More...
 
struct  HashCrc32Fallback
 
struct  HashDataSwitch
 Hashing helper that decides what is hashed. More...
 
struct  HashDataSwitch< std::string >
 
struct  Identity
 Identity functor, very useful for default parameters. More...
 
class  ItemReaderToolsBase
 CRTP class to enhance item/memory reader classes with Varint decoding and String decoding. More...
 
class  ItemWriterToolsBase
 CRTP class to enhance item/memory writer classes with Varint encoding and String encoding. More...
 
class  JsonBeginObj
 A special class to output verbatim text. More...
 
class  JsonEndObj
 A special class to output verbatim text. More...
 
class  JsonLine
 JsonLine is an object used to aggregate a set of key:value pairs for output into a JSON log. More...
 
struct  JsonLinePutSwitch
 
class  JsonLogger
 JsonLogger is a receiver of JSON output objects for logging. More...
 
class  JsonVerbatim
 A special class to output verbatim text. More...
 
class  Logger
 LOG and sLOG for development and debugging. More...
 
class  LoggerVoidify
 
class  Matrix
 A simple m x n dense matrix for generating statistics. More...
 
class  maximum
 
class  minimum
 
struct  NoOperation
 
struct  NoOperation< void >
 Specialized noop functor which returns a void. More...
 
class  OurConcurrentBoundedQueue
 This is a queue, similar to std::queue and tbb::concurrent_bounded_queue, except that it uses mutexes for synchronization. More...
 
class  OurConcurrentQueue
 This is a queue, similar to std::queue and tbb::concurrent_queue, except that it uses mutexes for synchronization. More...
 
class  ProfileTask
 
class  ProfileTaskRegistration
 
class  ProfileThread
 
class  RadixSort
 SortAlgorithm class for use with api::Sort() which calls radix_sort_CI() if K is small enough. More...
 
class  Range
 represents a 1 dimensional range (interval) [begin,end) More...
 
class  RunTimer
 RIAA class for running the timer until destruction. More...
 
class  Semaphore
 
class  SharedState
 
class  Singleton
 
class  SpacingLogger
 A logging class which outputs spaces between elements pushed via operator<<. More...
 
class  SplayzTree
 
class  StatsCounter
 This class provides a statistical counter that can easily be deactivated using a boolean template switch. More...
 
class  StatsCounter< ValueType_, false >
 
class  StatsCounter< ValueType_, true >
 
class  StatsTimerBase
 This class provides a statistical stop watch timer that can easily be deactivated using a boolean template switch. More...
 
class  StatsTimerBase< false >
 
class  StatsTimerBase< true >
 
class  StatsTimerBaseStarted
 
class  StatsTimerBaseStopped
 
class  StringView
 StringView is a reference to a part of a string, consisting of only a char pointer and a length. More...
 
class  SystemException
 An Exception which is thrown on system errors. More...
 
class  TabulationHashing
 Tabulation Hashing, see https://en.wikipedia.org/wiki/Tabulation_hashing. More...
 
class  ThreadBarrierLocking
 Implements a cyclic barrier using mutex locking and condition variables that can be used to synchronize threads. More...
 
class  ThreadBarrierSpinning
 Implements a cyclic barrier using atomics and a spin lock that can be used to synchronize threads. More...
 
class  TimedCounter
 
class  UIntPair
 Construct an 40-bit or 48-bit unsigned integer stored in five or six bytes. More...
 
class  Vector
 A compile-time fixed-length D-dimensional point with double precision. More...
 
class  VectorConcat
 Compute the concatenation of two std::vector<T>s. More...
 
class  VVector
 A variable-length D-dimensional point with double precision. More...
 
class  ZipfDistribution
 A class for producing random integers distributed according to the Zipf-Mandelbrot probability mass function: More...
 

Typedefs

template<typename Type >
using Array = Type[]
 A template to make writing temporary arrays easy: Array<int>{ 1, 2, 3 }. More...
 
template<typename T >
using ConcurrentBoundedQueue = OurConcurrentBoundedQueue< T >
 
template<typename T , typename Allocator >
using ConcurrentQueue = OurConcurrentQueue< T, Allocator >
 
using FakeStatsTimer = StatsTimerBase< false >
 
using FakeStatsTimerStart = StatsTimerBaseStarted< false >
 
using FakeStatsTimerStopped = StatsTimerBaseStopped< false >
 
template<typename T >
using hash = HashCrc32< T >
 Select a hashing method. More...
 
template<typename T >
using HashCrc32 = HashCrc32Fallback< T >
 
template<typename T >
using HashTabulated = TabulationHashing< sizeof(T)>
 Tabulation hashing. More...
 
template<typename T >
using is_trivially_copyable = std::is_trivially_copyable< T >
 
template<typename Key , typename T , typename Compare = std::less<Key>>
using logger_map = std::map< Key, T, Compare, LoggerAllocator< std::pair< const Key, T > > >
 deque without malloc tracking More...
 
template<typename Type >
using LoggerAllocator = mem::FixedAllocator< Type, g_logger_mem_manager >
 
template<typename Type , typename Allocator = std::allocator<Type>>
using RingBuffer = tlx::RingBuffer< Type, Allocator >
 
using StatsTimer = StatsTimerBase< true >
 
using StatsTimerStart = StatsTimerBaseStarted< true >
 
using StatsTimerStopped = StatsTimerBaseStopped< true >
 
using StringCount = std::pair< mem::by_string, size_t >
 
using ThreadBarrier = ThreadBarrierSpinning
 
using uint40 = UIntPair< uint8_t >
 Construct a 40-bit unsigned integer stored in five bytes. More...
 
using uint48 = UIntPair< uint16_t >
 Construct a 48-bit unsigned integer stored in six bytes. More...
 

Functions

template<typename IntegerType , unsigned bits = (8* sizeof(IntegerType))>
static IntegerType AddTruncToType (const IntegerType &a, const IntegerType &b)
 
template<typename To , typename From >
To & alias_cast (From &raw_data)
 
template<typename To , typename From >
const To & alias_cast (const From &raw_data)
 
static size_t CalcOneFactorPeer (size_t r, size_t p, size_t n)
 Calculate a Perfect Matching (1-Factor) on a Complete Graph. More...
 
static size_t CalcOneFactorSize (size_t n)
 Number of rounds in Perfect Matching (1-Factor). More...
 
static Range CalculateLocalRange (size_t global_size, size_t p, size_t i)
 
uint32_t crc32_slicing_by_8 (uint32_t crc, const void *data, size_t length)
 
template<typename... Args>
std::thread CreateThread (Args &&...args)
 create a std::thread and repeat creation if it fails More...
 
void FormatNameForThisThread (std::ostream &os)
 Outputs the name of the current thread or 'unknown [id]'. More...
 
template<typename T >
T from_cstr (const char *nptr, char **endptr=nullptr, int base=10)
 Number parsing helpers, wraps strto{f,d,ld,l,ul,ll,ull}() via type switch. More...
 
template<>
double from_cstr< double > (const char *nptr, char **endptr, int)
 
template<>
float from_cstr< float > (const char *nptr, char **endptr, int)
 
template<>
long from_cstr< long > (const char *nptr, char **endptr, int base)
 
template<>
long double from_cstr< long double > (const char *nptr, char **endptr, int)
 
template<>
long long from_cstr< long long > (const char *nptr, char **endptr, int base)
 
template<>
unsigned long from_cstr< unsigned long > (const char *nptr, char **endptr, int base)
 
template<>
unsigned long long from_cstr< unsigned long long > (const char *nptr, char **endptr, int base)
 
template<typename Type >
static bool from_str (const std::string &str, Type &outval)
 Template transformation function which uses std::istringstream to parse any istreamable type from a std::string. More...
 
std::string GetHostname ()
 get hostname More...
 
std::string GetNameForThisThread ()
 Returns the name of the current thread or 'unknown [id]'. More...
 
static uint64_t Hash128to64 (const uint64_t upper, const uint64_t lower)
 
static uint32_t Hash64to32 (uint64_t key)
 Returns a uint32_t hash of a uint64_t. More...
 
void LogCmdlineParams (JsonLogger &logger)
 
void MakePipe (int out_pipefds[2])
 create a pair of pipe file descriptors More...
 
template<typename Type , typename Functor >
auto MapVector (const std::vector< Type > &input, const Functor &f) -> std::vector< typename std::result_of< Functor(Type)>::type >
 
template<typename T >
static constexpr const Tmax (const T &a, const T &b)
 template for constexpr max, because std::max is not good enough. More...
 
template<typename T >
static constexpr const Tmin (const T &a, const T &b)
 template for constexpr min, because std::min is not good enough. More...
 
void NameThisThread (const mem::by_string &name)
 Defines a name for the current thread, only if no name was set previously. More...
 
static bool operator!= (const std::string &a, const StringView &b) noexcept
 
template<size_t D>
Vector< D > operator* (const double a, const Vector< D > &b)
 
VVector< double > operator* (const double a, const VVector< double > &b)
 
static bool operator== (const std::string &a, const StringView &b) noexcept
 
void PortSetCloseOnExec (int fd)
 set FD_CLOEXEC on file descriptor (if possible) More...
 
static JsonLinePut (JsonLine &line, bool const &value)
 
static JsonLinePut (JsonLine &line, int const &value)
 
static JsonLinePut (JsonLine &line, unsigned int const &value)
 
static JsonLinePut (JsonLine &line, long const &value)
 
static JsonLinePut (JsonLine &line, unsigned long const &value)
 
static JsonLinePut (JsonLine &line, long long const &value)
 
static JsonLinePut (JsonLine &line, unsigned long long const &value)
 
static JsonLinePut (JsonLine &line, double const &value)
 
static JsonLinePut (JsonLine &line, const char *const &str)
 
static JsonLinePut (JsonLine &line, std::string const &str)
 
template<typename Type , std::size_t N>
static JsonLinePut (JsonLine &line, const Type(&arr)[N])
 
template<typename Type >
static JsonLinePut (JsonLine &line, std::initializer_list< Type > const &list)
 
template<typename Type >
static JsonLinePut (JsonLine &line, std::vector< Type > const &vec)
 
template<typename Type , std::size_t N>
static JsonLinePut (JsonLine &line, std::array< Type, N > const &arr)
 
static JsonLinePut (JsonLine &line, JsonVerbatim const &verbatim)
 
template<typename Compare , typename Iterator >
void qsort_three_pivots (Iterator left, Iterator right, Compare cmp)
 
template<typename Compare , typename Iterator >
void qsort_two_pivots_yaroslavskiy (Iterator lo, Iterator hi, Compare cmp)
 
template<size_t MaxDepth, typename Iterator , typename Char , typename Comparator = std::less<typename std::iterator_traits<Iterator>::value_type>, typename SubSorter = NoOperation<void>>
static void radix_sort_CI (Iterator begin, Iterator end, size_t K, const Comparator &cmp, const SubSorter &sub_sort, size_t depth, Char *char_cache)
 Internal helper method, use radix_sort_CI below. More...
 
template<size_t MaxDepth, typename Iterator , typename Comparator = std::less<typename std::iterator_traits<Iterator>::value_type>, typename SubSorter = NoOperation<void>>
static void radix_sort_CI (Iterator begin, Iterator end, size_t K, const Comparator &cmp=Comparator(), const SubSorter &sub_sort=SubSorter())
 Radix sort the iterator range [begin,end). More...
 
template<typename RandomEngine = std::default_random_engine>
static std::string RandomString (std::string::size_type size, RandomEngine rng, const std::string &letters="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz")
 Generate a random string of given length. More...
 
void SetCpuAffinity (std::thread &thread, size_t cpu_id)
 set cpu/core affinity of a thread More...
 
void SetCpuAffinity (size_t cpu_id)
 set cpu/core affinity of current thread More...
 
template<typename Key , typename Tree , typename Compare >
Tree * splay (const Key &k, Tree *t, const Compare &cmp)
 
template<typename Tree , typename Compare >
void splay_check (const Tree *t, const Tree *&out_tmin, const Tree *&out_tmax, const Compare &cmp)
 check the tree order, recursively calculate min and max elements More...
 
template<typename Tree , typename Compare >
void splay_check (const Tree *t, const Compare &cmp)
 check the tree order More...
 
template<typename Key , typename Tree , typename Compare >
Tree * splay_erase (const Key &k, Tree *&t, const Compare &cmp)
 
template<typename Tree , typename Compare >
Tree * splay_erase_top (Tree *&t, const Compare &cmp)
 
template<typename Tree , typename Compare >
Tree * splay_insert (Tree *nn, Tree *t, const Compare &cmp)
 
template<typename Tree >
void splay_print (Tree *t, size_t d=0)
 print the tree More...
 
template<typename Tree , typename Functor >
void splay_traverse_preorder (const Functor &f, const Tree *t)
 traverse the tree in preorder More...
 
template<typename Key , typename Tree , typename Compare >
Tree * splayz (const Key &k, Tree *t, const Compare &cmp)
 
template<typename Tree , typename Compare >
void splayz_check (const Tree *t, const Tree *&out_tmin, const Tree *&out_tmax, size_t &out_size, const Compare &cmp)
 check the tree order, recursively calculate min and max elements More...
 
template<typename Tree , typename Compare >
void splayz_check (const Tree *t, const Compare &cmp)
 check the tree order More...
 
template<typename Key , typename Tree , typename Compare >
Tree * splayz_erase (const Key &k, Tree *&t, const Compare &cmp)
 
template<typename Tree , typename Compare >
Tree * splayz_erase_top (Tree *&t, const Compare &cmp)
 
template<typename Tree , typename Compare >
Tree * splayz_insert (Tree *nn, Tree *t, const Compare &cmp)
 
template<typename Tree >
void splayz_print (Tree *t, size_t d=0)
 print the tree More...
 
template<typename Tree >
Tree * splayz_rank (size_t r, Tree *t)
 
template<typename Tree >
size_t splayz_size (Tree *x)
 
template<typename Tree , typename Functor >
void splayz_traverse_preorder (const Functor &f, const Tree *t)
 traverse the tree in preorder More...
 
template<typename F >
static void SplitView (const std::string &str, char sep, F &&callback, std::string::size_type limit=std::string::npos)
 Split the given string at each separator character into distinct substrings, and call the given callback for each substring, represented by two iterators begin and end. More...
 
void StartLinuxProcStatsProfiler (ProfileThread &sched, JsonLogger &logger)
 launch profiler task More...
 
template<typename String = std::string>
String str_snprintf (size_t max_size, const char *fmt,...) TLX_ATTRIBUTE_FORMAT_PRINTF(2
 Helper for using sprintf to format into std::string and also to_string converters. More...
 
template<typename String >
String String str_snprintf (size_t max_size, const char *fmt,...)
 
template<typename String = std::string>
String str_sprintf (const char *fmt,...) TLX_ATTRIBUTE_FORMAT_PRINTF(1
 Helper for using sprintf to format into std::string and also to_string converters. More...
 
template<typename String >
String String str_sprintf (const char *fmt,...)
 
template<typename Type >
static std::string to_str (const Type &t)
 
template<typename T , size_t N>
static std::string VecToStr (const std::array< T, N > &data)
 Logging helper to print arrays as [a1,a2,a3,...]. More...
 
template<typename T >
static std::string VecToStr (const std::vector< T > &data)
 Logging helper to print vectors as [a1,a2,a3,...]. More...
 

Variables

const uint32_t crc_tableil8_o32 [256]
 
const uint32_t crc_tableil8_o40 [256]
 
const uint32_t crc_tableil8_o48 [256]
 
const uint32_t crc_tableil8_o56 [256]
 
const uint32_t crc_tableil8_o64 [256]
 
const uint32_t crc_tableil8_o72 [256]
 
const uint32_t crc_tableil8_o80 [256]
 
const uint32_t crc_tableil8_o88 [256]
 
static constexpr unsigned g_cache_line_size = 64
 Finding cache line size is hard - we assume 64 byte. More...
 
static constexpr bool g_debug_mode = !g_ndebug
 debug mode is active, if NDEBUG is false. More...
 
static constexpr bool g_debug_push_file = false
 
mem::Manager g_logger_mem_manager (nullptr,"Logger")
 memory manager singleton for Logger More...
 
static constexpr bool g_ndebug = false
 global ndebug flag as a boolean, NDEBUG means no debug in Release mode. More...
 
static constexpr bool g_profile_thread = true
 global flag to enable background profiler thread More...
 
static constexpr bool g_self_verify = g_debug_mode
 
static std::mutex s_logger_mutex
 mutex for log output More...
 
static std::mutex s_mutex
 mutex for s_threads map More...
 
static logger_map
< std::thread::id, StringCount
s_threads
 map thread id -> (name, message counter) More...
 
class thrill::common::UIntPair TLX_ATTRIBUTE_PACKED
 

Detailed Description

Contains many common tools and helpers shared by higher layers. See common.

Typedef Documentation

using Array = Type[]

A template to make writing temporary arrays easy: Array<int>{ 1, 2, 3 }.

Definition at line 64 of file json_logger.hpp.

using ConcurrentQueue = OurConcurrentQueue<T, Allocator>

Definition at line 127 of file concurrent_queue.hpp.

Definition at line 284 of file stats_timer.hpp.

Definition at line 285 of file stats_timer.hpp.

Definition at line 286 of file stats_timer.hpp.

using hash = HashCrc32<T>

Select a hashing method.

Definition at line 257 of file hash.hpp.

Definition at line 200 of file hash.hpp.

using HashTabulated = TabulationHashing<sizeof(T)>

Tabulation hashing.

Definition at line 253 of file hash.hpp.

using is_trivially_copyable = std::is_trivially_copyable<T>

Definition at line 54 of file defines.hpp.

using logger_map = std::map<Key, T, Compare, LoggerAllocator<std::pair<const Key, T> > >

deque without malloc tracking

Definition at line 41 of file logger.cpp.

Definition at line 36 of file logger.cpp.

using RingBuffer = tlx::RingBuffer<Type, Allocator>

Definition at line 21 of file ring_buffer.hpp.

using StatsTimer = StatsTimerBase<true>

Definition at line 280 of file stats_timer.hpp.

Definition at line 281 of file stats_timer.hpp.

Definition at line 282 of file stats_timer.hpp.

using StringCount = std::pair<mem::by_string, size_t>

Definition at line 33 of file logger.cpp.

Definition at line 165 of file thread_barrier.hpp.

using uint40 = UIntPair<uint8_t>

Construct a 40-bit unsigned integer stored in five bytes.

Definition at line 260 of file uint_types.hpp.

using uint48 = UIntPair<uint16_t>

Construct a 48-bit unsigned integer stored in six bytes.

Definition at line 263 of file uint_types.hpp.

Function Documentation

To& thrill::common::alias_cast ( From &  raw_data)
const To& thrill::common::alias_cast ( const From &  raw_data)
static size_t thrill::common::CalcOneFactorPeer ( size_t  r,
size_t  p,
size_t  n 
)
inlinestatic

Calculate a Perfect Matching (1-Factor) on a Complete Graph.

Used by collective network algorithms.

Parameters
rround [0..n-1) of matching
prank of this processor 0..n-1
nnumber of processors (graph size)
Returns
peer processor in this round

Definition at line 110 of file math.hpp.

References CalcOneFactorSize().

Referenced by Group::OneFactorPeer().

static size_t thrill::common::CalcOneFactorSize ( size_t  n)
inlinestatic

Number of rounds in Perfect Matching (1-Factor).

Definition at line 97 of file math.hpp.

Referenced by CalcOneFactorPeer(), and Group::OneFactorSize().

static Range thrill::common::CalculateLocalRange ( size_t  global_size,
size_t  p,
size_t  i 
)
inlinestatic

given a global range [0,global_size) and p PEs to split the range, calculate the [local_begin,local_end) index range assigned to the PE i.

Definition at line 81 of file math.hpp.

References min().

Referenced by Context::CalculateLocalRange(), Context::CalculateLocalRangeOnHost(), DistributeNode< ValueType >::Execute(), DuplicateDetection::WriteEncodedHashes(), and LocationDetection< thrill::api::JoinNode::HashCount >::WriteOccurenceCounts().

uint32_t crc32_slicing_by_8 ( uint32_t  crc,
const void *  data,
size_t  length 
)
std::thread thrill::common::CreateThread ( Args &&...  args)

create a std::thread and repeat creation if it fails

Definition at line 42 of file porting.hpp.

References LOG1.

Referenced by SortNode< ValueType, CompareFunction, SortAlgorithm >::MainOp(), and thrill::api::RunLoopbackThreads().

void thrill::common::FormatNameForThisThread ( std::ostream &  os)

Outputs the name of the current thread or 'unknown [id]'.

Definition at line 56 of file logger.cpp.

References s_mutex, and s_threads.

Referenced by GetNameForThisThread(), Logger::Logger(), and SpacingLogger::SpacingLogger().

T thrill::common::from_cstr ( const char *  nptr,
char **  endptr = nullptr,
int  base = 10 
)

Number parsing helpers, wraps strto{f,d,ld,l,ul,ll,ull}() via type switch.

double thrill::common::from_cstr< double > ( const char *  nptr,
char **  endptr,
int   
)
inline

Definition at line 132 of file string.hpp.

float thrill::common::from_cstr< float > ( const char *  nptr,
char **  endptr,
int   
)
inline

Definition at line 125 of file string.hpp.

long thrill::common::from_cstr< long > ( const char *  nptr,
char **  endptr,
int  base 
)
inline

Definition at line 149 of file string.hpp.

long double thrill::common::from_cstr< long double > ( const char *  nptr,
char **  endptr,
int   
)
inline

Definition at line 139 of file string.hpp.

long long thrill::common::from_cstr< long long > ( const char *  nptr,
char **  endptr,
int  base 
)
inline

Definition at line 163 of file string.hpp.

unsigned long thrill::common::from_cstr< unsigned long > ( const char *  nptr,
char **  endptr,
int  base 
)
inline

Definition at line 155 of file string.hpp.

unsigned long long thrill::common::from_cstr< unsigned long long > ( const char *  nptr,
char **  endptr,
int  base 
)
inline

Definition at line 169 of file string.hpp.

static bool thrill::common::from_str ( const std::string &  str,
Type &  outval 
)
inlinestatic

Template transformation function which uses std::istringstream to parse any istreamable type from a std::string.

Returns true only if the whole string was parsed.

Definition at line 107 of file string.hpp.

std::string GetHostname ( )

get hostname

Definition at line 142 of file porting.cpp.

std::string GetNameForThisThread ( )

Returns the name of the current thread or 'unknown [id]'.

Definition at line 79 of file logger.cpp.

References FormatNameForThisThread().

static uint64_t thrill::common::Hash128to64 ( const uint64_t  upper,
const uint64_t  lower 
)
inlinestatic

This is the Hash128to64 function from Google's cityhash (available under the MIT License).

Definition at line 63 of file hash.hpp.

Referenced by ReduceByHash< Key, HashFunction >::operator()().

static uint32_t thrill::common::Hash64to32 ( uint64_t  key)
inlinestatic

Returns a uint32_t hash of a uint64_t.

This comes from http://www.concentric.net/~ttwang/tech/inthash.htm

This hash gives no guarantees on the cryptographic suitability nor the quality of randomness produced, and the mapping may change in the future.

Definition at line 82 of file hash.hpp.

void LogCmdlineParams ( JsonLogger &  logger)

try to figure out the command line options of the current process and log it to json logger

Definition at line 67 of file porting.cpp.

References tlx::replace_all(), thrill::mem::to_string(), and tlx::unused().

Referenced by HostContext::HostContext().

void MakePipe ( int  out_pipefds[2])

create a pair of pipe file descriptors

Definition at line 51 of file porting.cpp.

References O_BINARY, and PortSetCloseOnExec().

Referenced by SelectDispatcher::SelectDispatcher(), thrill::vfs::SysOpenReadStream(), and thrill::vfs::SysOpenWriteStream().

auto thrill::common::MapVector ( const std::vector< Type > &  input,
const Functor &  f 
) -> std::vector<typename std::result_of<Functor(Type)>::type>
inline

apply the Functor to each item in a std::vector<T> and return a new std::vector<U> with different type.

Definition at line 98 of file functional.hpp.

static constexpr const T& thrill::common::max ( const T a,
const T b 
)
inlinestatic

template for constexpr max, because std::max is not good enough.

Definition at line 65 of file functional.hpp.

Referenced by Aggregate< Type_ >::Add(), Manager::add(), BlockPool::AllocateByteBlock(), CmdlineParser::calc_option_max(), CmdlineParser::calc_param_max(), MemoryFile::discard(), LocationDetection< thrill::api::JoinNode::HashCount >::Flush(), MixStream::GetWriters(), CatStream::GetWriters(), UIntPair< High_ >::high_max(), Range::Invalid(), StochasticGradientDescent< Vector >::is_converged(), UIntPair< High_ >::low_max(), main(), BlockPool::MapExternalBlock(), UIntPair< High_ >::max(), numeric_limits< thrill::common::UIntPair< HighType > >::max(), Pool::max_size(), HyperLogLogRegisters< p >::mergeDense(), BlockManager::new_blocks_int(), Aggregate< Type_ >::operator+(), Aggregate< Type_ >::operator+=(), OutputSVG(), examples::select::PickPivots(), examples::suffix_sorting::PrefixDoublingDementiev(), examples::suffix_sorting::PrefixDoublingDiscardingDementiev(), examples::suffix_sorting::PrefixQuadrupling(), CmdlineParser::print_result(), Context::PrintCollectiveMeanStdev(), ReadBinaryNode< ValueType >::ReadBinaryNode(), RunJoinPageRankEdgePerLine(), RunPageRankEdgePerLine(), SyscallFile::serve(), WincallFile::serve(), StatsCounter< ValueType_, true >::set_max(), MemoryFile::set_size(), Select::SetException(), Select::SetRead(), Select::SetWrite(), ReduceBucketHashTable< TableItem, Key, Value, KeyExtractor, ReduceFunction, Emitter, VolatileKey, ReduceConfig, IndexFunction, KeyEqualFunction >::SpillSmallestPartition(), Connection::SyncRecv(), Connection::SyncSend(), Connection::SyncSendRecv(), HyperLogLogRegisters< p >::toDense(), and SortNode< ValueType, CompareFunction, SortAlgorithm >::wanted_sample_size().

static constexpr const T& thrill::common::min ( const T a,
const T b 
)
inlinestatic

template for constexpr min, because std::min is not good enough.

Definition at line 59 of file functional.hpp.

Referenced by Aggregate< Type_ >::Add(), CalculateLocalRange(), UnionNode< ValueType >::consume_counter(), ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::DoScatter(), ZipWindowNode< ValueType, ZipFunction_, Pad_, UnequalCheck, UseStdArray, kNumInputs_ >::DoScatter(), OverlapWindowNode< ValueType, Input, WindowFunction, PartialWindowFunction >::Execute(), ConcatNode< ValueType >::Execute(), DisjointWindowNode< ValueType, Input, WindowFunction, PartialWindowFunction >::Execute(), ReduceProbingHashTable< TableItem, Key, Value, KeyExtractor, ReduceFunction, Emitter, VolatileKey, ReduceConfig_, IndexFunction, KeyEqualFunction >::GrowPartition(), ReduceProbingHashTable< TableItem, Key, Value, KeyExtractor, ReduceFunction, Emitter, VolatileKey, ReduceConfig_, IndexFunction, KeyEqualFunction >::Initialize(), Range::Invalid(), numeric_limits< thrill::common::UIntPair< HighType > >::lowest(), UIntPair< High_ >::min(), numeric_limits< thrill::common::UIntPair< HighType > >::min(), ReadBinaryNode< ValueType >::FileBlockSource::NextBlock(), ComponentSum< std::vector< Type >, Operation >::operator()(), Aggregate< Type_ >::operator+(), Aggregate< Type_ >::operator+=(), OutputSVG(), examples::select::PickPivots(), FlowControlChannel::Predecessor(), Context::PrintCollectiveMeanStdev(), ReadBinaryNode< ValueType >::ReadBinaryNode(), and SimpleVector< std::thread >::resize().

void NameThisThread ( const mem::by_string &  name)

Defines a name for the current thread, only if no name was set previously.

Defines a name for the current thread.

Definition at line 50 of file logger.cpp.

References s_mutex, and s_threads.

Referenced by thrill::api::RunLocalSameThread(), thrill::api::RunLoopbackThreads(), and DispatcherThread::Work().

static bool thrill::common::operator!= ( const std::string &  a,
const StringView &  b 
)
inlinestaticnoexcept

Definition at line 143 of file string_view.hpp.

Vector<D> thrill::common::operator* ( const double  a,
const Vector< D > &  b 
)

Definition at line 115 of file vector.hpp.

References gen_data::D, and Vector< D, Type >::x.

VVector<double> thrill::common::operator* ( const double  a,
const VVector< double > &  b 
)

Definition at line 211 of file vector.hpp.

References VVector< Type >::x.

static bool thrill::common::operator== ( const std::string &  a,
const StringView &  b 
)
inlinestaticnoexcept

Definition at line 138 of file string_view.hpp.

Referenced by Delegate< R(A...), Allocator >::operator!=().

void PortSetCloseOnExec ( int  fd)

set FD_CLOEXEC on file descriptor (if possible)

Definition at line 41 of file porting.cpp.

References tlx::unused().

Referenced by MakePipe(), thrill::vfs::SysOpenReadStream(), and thrill::vfs::SysOpenWriteStream().

static JsonLine& thrill::common::Put ( JsonLine &  line,
bool const &  value 
)
inlinestatic
static JsonLine& thrill::common::Put ( JsonLine &  line,
int const &  value 
)
inlinestatic

Definition at line 285 of file json_logger.hpp.

References JsonLine::os_, and gen_data::value.

static JsonLine& thrill::common::Put ( JsonLine &  line,
unsigned int const &  value 
)
inlinestatic

Definition at line 291 of file json_logger.hpp.

References JsonLine::os_, and gen_data::value.

static JsonLine& thrill::common::Put ( JsonLine &  line,
long const &  value 
)
inlinestatic

Definition at line 297 of file json_logger.hpp.

References JsonLine::os_, and gen_data::value.

static JsonLine& thrill::common::Put ( JsonLine &  line,
unsigned long const &  value 
)
inlinestatic

Definition at line 303 of file json_logger.hpp.

References JsonLine::os_, and gen_data::value.

static JsonLine& thrill::common::Put ( JsonLine &  line,
long long const &  value 
)
inlinestatic

Definition at line 309 of file json_logger.hpp.

References JsonLine::os_, and gen_data::value.

static JsonLine& thrill::common::Put ( JsonLine &  line,
unsigned long long const &  value 
)
inlinestatic

Definition at line 315 of file json_logger.hpp.

References JsonLine::os_, and gen_data::value.

static JsonLine& thrill::common::Put ( JsonLine &  line,
double const &  value 
)
inlinestatic

Definition at line 321 of file json_logger.hpp.

References JsonLine::os_, and gen_data::value.

static JsonLine& thrill::common::Put ( JsonLine &  line,
const char *const &  str 
)
inlinestatic

Definition at line 327 of file json_logger.hpp.

References JsonLine::os_, and JsonLine::PutEscapedChar().

static JsonLine& thrill::common::Put ( JsonLine &  line,
std::string const &  str 
)
inlinestatic

Definition at line 335 of file json_logger.hpp.

References JsonLine::os_, and JsonLine::PutEscapedChar().

static JsonLine& thrill::common::Put ( JsonLine &  line,
const Type(&)  arr[N] 
)
inlinestatic

Definition at line 345 of file json_logger.hpp.

References gen_data::N, JsonLine::os_, and Put().

static JsonLine& thrill::common::Put ( JsonLine &  line,
std::initializer_list< Type > const &  list 
)
inlinestatic

Definition at line 357 of file json_logger.hpp.

References JsonLine::os_, and Put().

static JsonLine& thrill::common::Put ( JsonLine &  line,
std::vector< Type > const &  vec 
)
inlinestatic

Definition at line 371 of file json_logger.hpp.

References JsonLine::os_, and Put().

static JsonLine& thrill::common::Put ( JsonLine &  line,
std::array< Type, N > const &  arr 
)
inlinestatic

Definition at line 385 of file json_logger.hpp.

References JsonLine::os_, and Put().

static JsonLine& thrill::common::Put ( JsonLine &  line,
JsonVerbatim const &  verbatim 
)
inlinestatic

Definition at line 398 of file json_logger.hpp.

References JsonLine::items_, JsonLine::os_, and JsonVerbatim::str_.

void thrill::common::qsort_three_pivots ( Iterator  left,
Iterator  right,
Compare  cmp 
)
void thrill::common::qsort_two_pivots_yaroslavskiy ( Iterator  lo,
Iterator  hi,
Compare  cmp 
)
static void thrill::common::radix_sort_CI ( Iterator  begin,
Iterator  end,
size_t  K,
const Comparator &  cmp,
const SubSorter &  sub_sort,
size_t  depth,
Char *  char_cache 
)
inlinestatic

Internal helper method, use radix_sort_CI below.

Definition at line 37 of file radix_sort.hpp.

References tlx::swap().

static void thrill::common::radix_sort_CI ( Iterator  begin,
Iterator  end,
size_t  K,
const Comparator &  cmp = Comparator(),
const SubSorter &  sub_sort = SubSorter() 
)
inlinestatic

Radix sort the iterator range [begin,end).

Sort unconditionally up to depth MaxDepth, then call the sub_sort method for further sorting. Small buckets are sorted using std::sort() with given comparator. Characters are extracted from items in the range using the at_radix(depth) method. All character values must be less than K (the counting array size).

Definition at line 118 of file radix_sort.hpp.

static std::string thrill::common::RandomString ( std::string::size_type  size,
RandomEngine  rng,
const std::string &  letters = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz" 
)
inlinestatic

Generate a random string of given length.

The set of available bytes/characters is given as the second argument. Each byte is equally probable. Uses the pseudo-random number generator from stdlib; take care to seed it using srand() before calling this function.

Parameters
sizelength of result
rngRandom number generator to use, e.g. std::default_random_engine.
letterscharacter set to choose from
Returns
random string of given length

Definition at line 220 of file string.hpp.

void SetCpuAffinity ( std::thread &  thread,
size_t  cpu_id 
)
void SetCpuAffinity ( size_t  cpu_id)

set cpu/core affinity of current thread

Definition at line 127 of file porting.cpp.

References LOG1, and tlx::unused().

Tree* thrill::common::splay ( const Key &  k,
Tree *  t,
const Compare &  cmp 
)
inline

Splay using the key i (which may or may not be in the tree.) The starting root is t, and the tree used is defined by rat size fields are maintained.

Definition at line 109 of file splay_tree.hpp.

Referenced by Pool::AllocateFreeArena(), Pool::deallocate(), splay_erase(), and splay_erase_top().

void thrill::common::splay_check ( const Tree *  t,
const Tree *&  out_tmin,
const Tree *&  out_tmax,
const Compare &  cmp 
)
inline

check the tree order, recursively calculate min and max elements

Definition at line 83 of file splay_tree.hpp.

Referenced by splay_check().

void thrill::common::splay_check ( const Tree *  t,
const Compare &  cmp 
)
inline

check the tree order

Definition at line 100 of file splay_tree.hpp.

References splay_check().

Tree* thrill::common::splay_erase ( const Key &  k,
Tree *&  t,
const Compare &  cmp 
)
inline

Erases i from the tree if it's there. Return a pointer to the resulting tree.

Definition at line 188 of file splay_tree.hpp.

References splay(), and gen_data::x.

Tree* thrill::common::splay_erase_top ( Tree *&  t,
const Compare &  cmp 
)
inline

Erases i from the tree if it's there. Return a pointer to the resulting tree.

Definition at line 214 of file splay_tree.hpp.

References splay(), and gen_data::x.

Referenced by Pool::deallocate().

Tree* thrill::common::splay_insert ( Tree *  nn,
Tree *  t,
const Compare &  cmp 
)
inline

Insert key i into the tree t, if it is not already there. Before calling this method, one MUST call splay() to rotate the tree to the right position. Return a pointer to the resulting tree.

Definition at line 168 of file splay_tree.hpp.

Referenced by Pool::AllocateFreeArena().

void thrill::common::splay_print ( Tree *  t,
size_t  d = 0 
)
inline

print the tree

Definition at line 73 of file splay_tree.hpp.

void thrill::common::splay_traverse_preorder ( const Functor &  f,
const Tree *  t 
)
inline

traverse the tree in preorder

Definition at line 231 of file splay_tree.hpp.

Tree* thrill::common::splayz ( const Key &  k,
Tree *  t,
const Compare &  cmp 
)
inline

Splay using the key i (which may or may not be in the tree.) The starting root is t, and the tree used is defined by rat size fields are maintained.

Definition at line 288 of file splay_tree.hpp.

References splayz_size().

Referenced by SplayzTree< Key >::exists(), SplayzTree< Key >::find(), SplayzTree< Key >::insert(), splayz_erase(), and splayz_erase_top().

void thrill::common::splayz_check ( const Tree *  t,
const Tree *&  out_tmin,
const Tree *&  out_tmax,
size_t &  out_size,
const Compare &  cmp 
)
inline

check the tree order, recursively calculate min and max elements

Definition at line 258 of file splay_tree.hpp.

Referenced by splayz_check().

void thrill::common::splayz_check ( const Tree *  t,
const Compare &  cmp 
)
inline

check the tree order

Definition at line 278 of file splay_tree.hpp.

References splayz_check().

Tree* thrill::common::splayz_erase ( const Key &  k,
Tree *&  t,
const Compare &  cmp 
)
inline

Erases i from the tree if it's there. Return a pointer to the resulting tree.

Definition at line 393 of file splay_tree.hpp.

References splayz(), and gen_data::x.

Referenced by SplayzTree< Key >::erase().

Tree* thrill::common::splayz_erase_top ( Tree *&  t,
const Compare &  cmp 
)
inline

Erases i from the tree if it's there. Return a pointer to the resulting tree.

Definition at line 423 of file splay_tree.hpp.

References splayz(), and gen_data::x.

Tree* thrill::common::splayz_insert ( Tree *  nn,
Tree *  t,
const Compare &  cmp 
)
inline

Insert key i into the tree t, if it is not already there. Before calling this method, one MUST call splayz() to rotate the tree to the right position. Return a pointer to the resulting tree.

Definition at line 370 of file splay_tree.hpp.

References splayz_size().

Referenced by SplayzTree< Key >::insert().

void thrill::common::splayz_print ( Tree *  t,
size_t  d = 0 
)
inline

print the tree

Definition at line 248 of file splay_tree.hpp.

Tree* thrill::common::splayz_rank ( size_t  r,
Tree *  t 
)
inline

Returns a pointer to the node in the tree with the given rank. Returns nullptr if there is no such node. Does not change the tree. To guarantee logarithmic behavior, the node found here should be splayed to the root.

Definition at line 446 of file splay_tree.hpp.

References splayz_size().

Referenced by SplayzTree< Key >::rank().

size_t thrill::common::splayz_size ( Tree *  x)
inline

Definition at line 242 of file splay_tree.hpp.

Referenced by splayz(), splayz_insert(), and splayz_rank().

void thrill::common::splayz_traverse_preorder ( const Functor &  f,
const Tree *  t 
)
inline

traverse the tree in preorder

Definition at line 466 of file splay_tree.hpp.

Referenced by SplayzTree< Key >::traverse_preorder().

static void thrill::common::SplitView ( const std::string &  str,
char  sep,
F &&  callback,
std::string::size_type  limit = std::string::npos 
)
inlinestatic

Split the given string at each separator character into distinct substrings, and call the given callback for each substring, represented by two iterators begin and end.

Multiple consecutive separators are considered individually and will result in empty split substrings.

Parameters
strstring to split
sepseparator character
callbackcallback taking begin and end iterator of substring
limitmaximum number of parts returned

Definition at line 160 of file string_view.hpp.

Referenced by examples::word_count::HashWordCountExample(), main(), WordCount(), and examples::word_count::WordCount().

void StartLinuxProcStatsProfiler ( ProfileThread &  ,
JsonLogger &   
)

launch profiler task

Definition at line 859 of file linux_proc_stats.cpp.

Referenced by HostContext::HostContext().

String thrill::common::str_snprintf ( size_t  max_size,
const char *  fmt,
  ... 
)

Helper for using sprintf to format into std::string and also to_string converters.

Parameters
max_sizemaximum length of output string, longer ones are truncated.
fmtprintf format and additional parameters
String String thrill::common::str_snprintf ( size_t  max_size,
const char *  fmt,
  ... 
)

Definition at line 44 of file string.hpp.

String thrill::common::str_sprintf ( const char *  fmt,
  ... 
)

Helper for using sprintf to format into std::string and also to_string converters.

Parameters
fmtprintf format and additional parameters

Referenced by examples::suffix_sorting::ConstructWaveletTree(), RunJoinPageRankEdgePerLine(), RunPageRankEdgePerLine(), and RunPageRankJoinGenerated().

String String thrill::common::str_sprintf ( const char *  fmt,
  ... 
)

Definition at line 69 of file string.hpp.

static std::string thrill::common::to_str ( const Type &  t)
inlinestatic

Use ostream to output any type as string. You generally DO NOT want to use this, instead create a larger ostringstream.

Definition at line 94 of file string.hpp.

static std::string thrill::common::VecToStr ( const std::vector< T > &  data)
static

Logging helper to print vectors as [a1,a2,a3,...].

Definition at line 194 of file string.hpp.

Variable Documentation

const uint32_t crc_tableil8_o32[256]

Definition at line 71 of file hash.cpp.

Referenced by crc32_slicing_by_8().

const uint32_t crc_tableil8_o40[256]

Definition at line 125 of file hash.cpp.

Referenced by crc32_slicing_by_8().

const uint32_t crc_tableil8_o48[256]

Definition at line 179 of file hash.cpp.

Referenced by crc32_slicing_by_8().

const uint32_t crc_tableil8_o56[256]

Definition at line 233 of file hash.cpp.

Referenced by crc32_slicing_by_8().

const uint32_t crc_tableil8_o64[256]

Definition at line 287 of file hash.cpp.

Referenced by crc32_slicing_by_8().

const uint32_t crc_tableil8_o72[256]

Definition at line 341 of file hash.cpp.

Referenced by crc32_slicing_by_8().

const uint32_t crc_tableil8_o80[256]

Definition at line 395 of file hash.cpp.

Referenced by crc32_slicing_by_8().

const uint32_t crc_tableil8_o88[256]

Definition at line 449 of file hash.cpp.

Referenced by crc32_slicing_by_8().

constexpr unsigned g_cache_line_size = 64
static

Finding cache line size is hard - we assume 64 byte.

Definition at line 38 of file config.hpp.

constexpr bool g_debug_mode = !g_ndebug
static

debug mode is active, if NDEBUG is false.

Definition at line 28 of file config.hpp.

constexpr bool g_debug_push_file = false
static

global flag to warn user when two DIANodes could push data directly via a File (skipping one data round trip) if the function stack were empty. This can be used to find lambda which could be be fused to reduce the number of data round trips.

Definition at line 44 of file config.hpp.

Referenced by ConcatNode< ValueType >::OnPreOpFile().

mem::Manager g_logger_mem_manager

memory manager singleton for Logger

Definition at line 27 of file logger.hpp.

constexpr bool g_ndebug = false
static

global ndebug flag as a boolean, NDEBUG means no debug in Release mode.

Definition at line 24 of file config.hpp.

constexpr bool g_profile_thread = true
static

global flag to enable background profiler thread

Definition at line 35 of file config.hpp.

Referenced by ProfileThread::ProfileThread(), and ProfileThread::~ProfileThread().

constexpr bool g_self_verify = g_debug_mode
static

global flag to enable code parts doing self-verification. Later this may be set false if NDEBUG is set in production mode.

Definition at line 32 of file config.hpp.

std::mutex s_logger_mutex
static

mutex for log output

Definition at line 88 of file logger.cpp.

Referenced by Logger::Output().

std::mutex s_mutex
static

mutex for s_threads map

Definition at line 44 of file logger.cpp.

Referenced by FormatNameForThisThread(), and NameThisThread().

logger_map<std::thread::id, StringCount> s_threads
static

map thread id -> (name, message counter)

Definition at line 47 of file logger.cpp.

Referenced by FormatNameForThisThread(), and NameThisThread().

class thrill::common::UIntPair TLX_ATTRIBUTE_PACKED