Thrill  0.1
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
Context Class Reference

Detailed Description

The Context of a job is a unique instance per worker which holds references to all underlying parts of Thrill.

The context is able to give references to the stream multiplexer, the netgroup" \ref common::Stats "stats" and \ref common::StatsGraph "stats graph". Threads share the stream multiplexer and the net group via the context object.

Definition at line 218 of file context.hpp.

+ Collaboration diagram for Context:

#include <context.hpp>

Public Member Functions

 Context (HostContext &host_context, size_t local_worker_id)
 
common::Range CalculateLocalRange (size_t global_size) const
 
common::Range CalculateLocalRangeOnHost (size_t global_size) const
 
bool consume () const
 return value of consume flag. More...
 
void enable_consume (bool consume=true)
 Sets consume-mode flag such that DIA contents may be consumed during PushData(). More...
 
void Launch (const std::function< void(Context &)> &job_startpoint)
 method used to launch a job's main procedure. it wraps it in log output. More...
 
const MemoryConfigmem_config () const
 host-global memory config More...
 
mem::Managermem_manager ()
 returns the host-global memory manager More...
 
net::Managernet_manager ()
 
size_t next_dia_id ()
 Returns next_dia_id_ to generate DIA::id_ serial. More...
 
template<typename Type >
void PrintCollectiveMeanStdev (const char *text, const Type &local)
 
Data Subsystem
data::File GetFile (size_t dia_id)
 Returns a new File object containing a sequence of local Blocks. More...
 
data::File GetFile (DIABase *dia)
 Returns a new File object containing a sequence of local Blocks. More...
 
data::FilePtr GetFilePtr (size_t dia_id)
 
data::FilePtr GetFilePtr (DIABase *dia)
 
data::CatStreamPtr GetNewCatStream (size_t dia_id)
 
data::CatStreamPtr GetNewCatStream (DIABase *dia)
 
data::MixStreamPtr GetNewMixStream (size_t dia_id)
 
data::MixStreamPtr GetNewMixStream (DIABase *dia)
 
template<typename Stream >
tlx::CountingPtr< Stream > GetNewStream (size_t dia_id)
 
data::BlockPoolblock_pool ()
 the block manager keeps all data blocks moving through the system. More...
 

Public Attributes

Shared Objects
std::default_random_engine rng_
 a random generator More...
 
Network Subsystem
net::FlowControlChannelnet
 
Logging System
common::JsonLogger base_logger_
 base logger exclusive for this worker More...
 
common::JsonLogger logger_
 

Private Attributes

data::BlockPoolblock_pool_
 data block pool More...
 
bool consume_ = false
 flag to set which enables selective consumption of DIA contents! More...
 
net::FlowControlChannelManagerflow_manager_
 net::FlowControlChannelManager instance that is shared among workers More...
 
size_t last_dia_id_ = 0
 the number of valid DIA ids. 0 is reserved for invalid. More...
 
size_t local_host_id_
 id among all local hosts (in test program runs) More...
 
size_t local_worker_id_
 number of this host context, 0..p-1, within this host More...
 
const MemoryConfigmem_config_
 memory configuration in HostContext More...
 
size_t mem_limit_
 memory limit of this worker Context for local data structures More...
 
mem::Managermem_manager_
 host-global memory manager More...
 
data::Multiplexermultiplexer_
 data::Multiplexer instance that is shared among workers More...
 
net::Managernet_manager_
 net::Manager instance that is shared among workers More...
 
size_t workers_per_host_
 number of workers hosted per host More...
 

System Information

size_t num_hosts () const
 Returns the total number of hosts. More...
 
size_t workers_per_host () const
 Returns the number of workers that is hosted on each host. More...
 
size_t my_rank () const
 Global rank of this worker among all other workers in the system. More...
 
size_t mem_limit () const
 memory limit of this worker Context for local data structures More...
 
size_t num_workers () const
 Global number of workers in the system. More...
 
size_t host_rank () const
 
size_t local_worker_id () const
 
size_t local_host_id () const
 id among all local hosts (in test program runs) More...
 
std::ostream & operator<< (std::ostream &os, const Context &ctx)
 Outputs the context as [host id]:[local worker id] to an std::ostream. More...
 

Constructor & Destructor Documentation

Context ( HostContext host_context,
size_t  local_worker_id 
)

Definition at line 1090 of file context.cpp.

Member Function Documentation

common::Range CalculateLocalRange ( size_t  global_size) const
inline
bool consume ( ) const
inline

return value of consume flag.

Definition at line 375 of file context.hpp.

References Context::consume_.

Referenced by Context::enable_consume(), DIA< ValueType_, Stack_ >::Keep(), and DIANode< StackInput >::RunPushData().

void enable_consume ( bool  consume = true)
inline

Sets consume-mode flag such that DIA contents may be consumed during PushData().

When in consume mode the DIA contents is destroyed online when it is transmitted to the next operation. This enables reusing the space of the consume operations. This enabled processing more data with less space. However, by default this mode is DISABLED, because it requires deliberate insertion of .Keep() calls.

Definition at line 385 of file context.hpp.

References Context::consume(), and Context::consume_.

Referenced by JoinTPCH4(), main(), RunHashWordCount(), RunHashWordCountGenerated(), RunJoinPageRankEdgePerLine(), RunPageRankEdgePerLine(), RunPageRankGenerated(), RunPageRankJoinGenerated(), RunWordCount(), and RunWordCountGenerated().

data::File GetFile ( DIABase dia)

Returns a new File object containing a sequence of local Blocks.

Definition at line 1107 of file context.cpp.

References Context::GetFile(), and DIABase::id().

data::FilePtr GetFilePtr ( size_t  dia_id)

Returns a new File, wrapped in a CountingPtr, containing a sequence of local Blocks.

Definition at line 1111 of file context.cpp.

References Context::block_pool_, and Context::local_worker_id_.

Referenced by JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::AddEqualKeysToVec(), and Context::GetFilePtr().

data::FilePtr GetFilePtr ( DIABase dia)

Returns a new File, wrapped in a CountingPtr, containing a sequence of local Blocks.

Definition at line 1116 of file context.cpp.

References Context::GetFilePtr(), and DIABase::id().

data::CatStreamPtr GetNewCatStream ( size_t  dia_id)
data::CatStreamPtr GetNewCatStream ( DIABase dia)

Returns a reference to a new CatStream. This method alters the state of the context and must be called on all Workers to ensure correct communication coordination.

Definition at line 1124 of file context.cpp.

References Context::GetNewCatStream(), and DIABase::id().

data::MixStreamPtr GetNewMixStream ( size_t  dia_id)

Returns a reference to a new MixStream. This method alters the state of the context and must be called on all Workers to ensure correct communication coordination.

Definition at line 1128 of file context.cpp.

References Multiplexer::GetNewMixStream(), Context::local_worker_id_, and Context::multiplexer_.

Referenced by Context::GetNewMixStream(), and SortNode< ValueType, CompareFunction, SortAlgorithm >::MainOp().

data::MixStreamPtr GetNewMixStream ( DIABase dia)

Returns a reference to a new MixStream. This method alters the state of the context and must be called on all Workers to ensure correct communication coordination.

Definition at line 1132 of file context.cpp.

References Context::GetNewMixStream(), and DIABase::id().

tlx::CountingPtr<Stream> GetNewStream ( size_t  dia_id)

Returns a reference to a new CatStream or MixStream, selectable via template parameter.

size_t host_rank ( ) const
inline

Returns id of this host in the cluser A host is a machine in the cluster that hosts multiple workers

Definition at line 254 of file context.hpp.

References Manager::my_host_rank(), and Context::net_manager_.

Referenced by Context::my_rank().

size_t local_host_id ( ) const
inline

id among all local hosts (in test program runs)

Definition at line 265 of file context.hpp.

References Context::local_host_id_.

size_t local_worker_id ( ) const
inline
const MemoryConfig& mem_config ( ) const
inline

host-global memory config

Definition at line 326 of file context.hpp.

References Context::mem_config_.

Referenced by Context::Launch().

size_t mem_limit ( ) const
inline

memory limit of this worker Context for local data structures

Definition at line 245 of file context.hpp.

References Context::mem_limit_.

mem::Manager& mem_manager ( )
inline

returns the host-global memory manager

Definition at line 329 of file context.hpp.

References Context::mem_manager_.

Referenced by DIABase::mem_manager().

size_t next_dia_id ( )
inline
void PrintCollectiveMeanStdev ( const char *  text,
const Type &  local 
)
inline
size_t workers_per_host ( ) const
inline

Returns the number of workers that is hosted on each host.

Definition at line 235 of file context.hpp.

References Context::workers_per_host_.

Referenced by Context::CalculateLocalRangeOnHost(), Context::my_rank(), and Context::num_workers().

Friends And Related Function Documentation

std ::ostream& operator<< ( std::ostream &  os,
const Context ctx 
)
friend

Outputs the context as [host id]:[local worker id] to an std::ostream.

Definition at line 269 of file context.hpp.

Member Data Documentation

common::JsonLogger base_logger_

base logger exclusive for this worker

Definition at line 454 of file context.hpp.

data::BlockPool& block_pool_
private

data block pool

Definition at line 416 of file context.hpp.

Referenced by Context::block_pool(), Context::GetFile(), and Context::GetFilePtr().

bool consume_ = false
private

flag to set which enables selective consumption of DIA contents!

Definition at line 422 of file context.hpp.

Referenced by Context::consume(), and Context::enable_consume().

net::FlowControlChannelManager& flow_manager_
private

net::FlowControlChannelManager instance that is shared among workers

Definition at line 413 of file context.hpp.

size_t last_dia_id_ = 0
private

the number of valid DIA ids. 0 is reserved for invalid.

Definition at line 425 of file context.hpp.

Referenced by Context::next_dia_id().

size_t local_host_id_
private

id among all local hosts (in test program runs)

Definition at line 392 of file context.hpp.

Referenced by Context::Launch(), and Context::local_host_id().

size_t local_worker_id_
private

number of this host context, 0..p-1, within this host

Definition at line 395 of file context.hpp.

Referenced by Context::GetFile(), Context::GetFilePtr(), Context::GetNewCatStream(), Context::GetNewMixStream(), Context::Launch(), and Context::local_worker_id().

Initial value:
{
&base_logger_, "host_rank", host_rank(), "worker_rank", my_rank()
}

public member which delivers key:value pairs as JSON log lines. this logger is local to this Context which is exclusive for one worker thread.

Definition at line 459 of file context.hpp.

Referenced by DIA< ValueType_, Stack_ >::Filter(), DIA< ValueType_, Stack_ >::FlatMap(), Context::Launch(), and DIA< ValueType_, Stack_ >::Map().

const MemoryConfig& mem_config_
private

memory configuration in HostContext

Definition at line 404 of file context.hpp.

Referenced by Context::mem_config().

size_t mem_limit_
private

memory limit of this worker Context for local data structures

Definition at line 401 of file context.hpp.

Referenced by Context::mem_limit().

mem::Manager& mem_manager_
private

host-global memory manager

Definition at line 407 of file context.hpp.

Referenced by Context::mem_manager().

data::Multiplexer& multiplexer_
private

data::Multiplexer instance that is shared among workers

Definition at line 419 of file context.hpp.

Referenced by Context::GetNewCatStream(), and Context::GetNewMixStream().

net::Manager& net_manager_
private

net::Manager instance that is shared among workers

Definition at line 410 of file context.hpp.

Referenced by Context::host_rank(), Context::Launch(), Context::net_manager(), and Context::num_hosts().

std::default_random_engine rng_

a random generator

Definition at line 432 of file context.hpp.

Referenced by MergeNode< ValueType, Comparator, kNumInputs >::SelectPivots().

size_t workers_per_host_
private

number of workers hosted per host

Definition at line 398 of file context.hpp.

Referenced by Context::workers_per_host().


The documentation for this class was generated from the following files: