Thrill  0.1
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
Stream Class Referenceabstract

Detailed Description

Stream - base class for CatStream and MixStream.

Definition at line 36 of file stream.hpp.

+ Inheritance diagram for Stream:
+ Collaboration diagram for Stream:

#include <stream.hpp>

Public Types

using Writer = StreamData::Writer
 
using Writers = StreamData::Writers
 

Public Member Functions

virtual ~Stream ()
 
void Close ()
 shuts down the stream, waits for all closing blocks More...
 
virtual StreamDatadata ()=0
 Return stream data reference. More...
 
virtual const StreamDatadata () const =0
 Return stream data reference. More...
 
virtual Writers GetWriters ()=0
 
virtual const StreamIdid () const =0
 Return stream id. More...
 
template<typename ItemType >
void Scatter (File &source, const std::vector< size_t > &offsets, bool consume=false)
 Scatters a File to many worker: elements from [offset[0],offset[1]) are sent to the first worker, elements from [offset[1], offset[2]) are sent to the second worker, ..., elements from [offset[my_rank - 1],offset[my_rank]) are copied locally, ..., elements from [offset[num_workers - 1], offset[num_workers]) are sent to the last worker. More...
 
template<typename ItemType >
void ScatterConsume (File &source, const std::vector< size_t > &offsets)
 Consuming Version of Scatter() see documentation there. More...
 
template<typename ItemType >
void ScatterKeep (File &source, const std::vector< size_t > &offsets)
 Keep Version of Scatter() see documentation there. More...
 
Statistics
size_t tx_items () const
 return number of items transmitted More...
 
size_t tx_bytes () const
 return number of bytes transmitted More...
 
size_t tx_blocks () const
 return number of blocks transmitted More...
 
size_t rx_items () const
 return number of items received More...
 
size_t rx_bytes () const
 return number of bytes received More...
 
size_t rx_blocks () const
 return number of blocks received More...
 
size_t tx_net_items () const
 return number of items transmitted via network excluding internal tx More...
 
size_t tx_net_bytes () const
 return number of bytes transmitted via network excluding internal tx More...
 
size_t tx_net_blocks () const
 return number of blocks transmitted via network excluding internal tx More...
 
size_t rx_net_items () const
 return number of items received via network excluding internal tx More...
 
size_t rx_net_bytes () const
 return number of bytes received via network excluding internal tx More...
 
size_t rx_net_blocks () const
 return number of blocks received via network excluding internal tx More...
 
size_t tx_int_items () const
 return number of items transmitted via internal loopback queues More...
 
size_t tx_int_bytes () const
 return number of bytes transmitted via internal loopback queues More...
 
size_t tx_int_blocks () const
 return number of blocks transmitted via internal loopback queues More...
 
size_t rx_int_items () const
 return number of items received via network internal loopback queues More...
 
size_t rx_int_bytes () const
 return number of bytes received via network internal loopback queues More...
 
size_t rx_int_blocks () const
 return number of blocks received via network internal loopback queues More...
 
- Public Member Functions inherited from ReferenceCounter
 ReferenceCounter () noexcept
 new objects have zero reference count More...
 
 ReferenceCounter (const ReferenceCounter &) noexcept
 coping still creates a new object with zero reference count More...
 
 ~ReferenceCounter ()
 
bool dec_reference () const noexcept
 Call whenever resetting (i.e. More...
 
void inc_reference () const noexcept
 Call whenever setting a pointer to the object. More...
 
ReferenceCounteroperator= (const ReferenceCounter &) noexcept
 assignment operator, leaves pointers unchanged More...
 
size_t reference_count () const noexcept
 Return the number of references to this object (for debugging) More...
 
bool unique () const noexcept
 Test if the ReferenceCounter is referenced by only one CountingPtr. More...
 

Member Typedef Documentation

Definition at line 39 of file stream.hpp.

Definition at line 40 of file stream.hpp.

Constructor & Destructor Documentation

~Stream ( )
virtual

Definition at line 23 of file stream.cpp.

Member Function Documentation

void Close ( )

shuts down the stream, waits for all closing blocks

Definition at line 26 of file stream.cpp.

References StreamData::Close(), and Stream::data().

virtual const StreamData& data ( ) const
pure virtual

Return stream data reference.

Implemented in CatStream, and MixStream.

virtual Writers GetWriters ( )
pure virtual

Creates BlockWriters for each worker. BlockWriter can only be opened once, otherwise the block sequence is incorrectly interleaved!

Implemented in CatStream, and MixStream.

Referenced by Stream::ScatterConsume(), and Stream::ScatterKeep().

virtual const StreamId& id ( ) const
pure virtual

Return stream id.

Implemented in CatStream, and MixStream.

size_t rx_blocks ( ) const

return number of blocks received

Definition at line 52 of file stream.cpp.

References Stream::rx_int_blocks(), and Stream::rx_net_blocks().

size_t rx_bytes ( ) const

return number of bytes received

Definition at line 48 of file stream.cpp.

References Stream::rx_int_bytes(), and Stream::rx_net_bytes().

size_t rx_int_blocks ( ) const

return number of blocks received via network internal loopback queues

Definition at line 104 of file stream.cpp.

References Stream::data(), and StreamData::rx_int_blocks_.

Referenced by Stream::rx_blocks().

size_t rx_int_bytes ( ) const

return number of bytes received via network internal loopback queues

Definition at line 100 of file stream.cpp.

References Stream::data(), and StreamData::rx_int_bytes_.

Referenced by Stream::rx_bytes().

size_t rx_int_items ( ) const

return number of items received via network internal loopback queues

Definition at line 96 of file stream.cpp.

References Stream::data(), and StreamData::rx_int_items_.

Referenced by Stream::rx_items().

size_t rx_items ( ) const

return number of items received

Definition at line 44 of file stream.cpp.

References Stream::rx_int_items(), and Stream::rx_net_items().

size_t rx_net_blocks ( ) const

return number of blocks received via network excluding internal tx

Definition at line 78 of file stream.cpp.

References Stream::data(), and StreamData::rx_net_blocks_.

Referenced by Stream::rx_blocks().

size_t rx_net_bytes ( ) const

return number of bytes received via network excluding internal tx

Definition at line 74 of file stream.cpp.

References Stream::data(), and StreamData::rx_net_bytes_.

Referenced by Stream::rx_bytes().

size_t rx_net_items ( ) const

return number of items received via network excluding internal tx

Definition at line 70 of file stream.cpp.

References Stream::data(), and StreamData::rx_net_items_.

Referenced by Stream::rx_items().

void Scatter ( File source,
const std::vector< size_t > &  offsets,
bool  consume = false 
)
inline

Scatters a File to many worker: elements from [offset[0],offset[1]) are sent to the first worker, elements from [offset[1], offset[2]) are sent to the second worker, ..., elements from [offset[my_rank - 1],offset[my_rank]) are copied locally, ..., elements from [offset[num_workers - 1], offset[num_workers]) are sent to the last worker.

The number of given offsets must be equal to the net::Group::num_hosts() * workers_per_host_ + 1.

/param source File containing the data to be scattered.

/param offsets - as described above. offsets.size must be equal to num_workers + 1

Definition at line 77 of file stream.hpp.

void ScatterConsume ( File source,
const std::vector< size_t > &  offsets 
)
inline

Consuming Version of Scatter() see documentation there.

Definition at line 89 of file stream.hpp.

References Stream::data(), File::GetConsumeReader(), Stream::GetWriters(), and StreamData::my_worker_rank().

void ScatterKeep ( File source,
const std::vector< size_t > &  offsets 
)
inline

Keep Version of Scatter() see documentation there.

Definition at line 149 of file stream.hpp.

References StreamData::Writers::Close(), File::GetKeepReader(), Stream::GetWriters(), and BlockReader< BlockSource >::HasNext().

size_t tx_blocks ( ) const

return number of blocks transmitted

Definition at line 40 of file stream.cpp.

References Stream::tx_int_blocks(), and Stream::tx_net_blocks().

size_t tx_bytes ( ) const

return number of bytes transmitted

Definition at line 36 of file stream.cpp.

References Stream::tx_int_bytes(), and Stream::tx_net_bytes().

size_t tx_int_blocks ( ) const

return number of blocks transmitted via internal loopback queues

Definition at line 92 of file stream.cpp.

References Stream::data(), and StreamData::tx_int_blocks_.

Referenced by Stream::tx_blocks().

size_t tx_int_bytes ( ) const

return number of bytes transmitted via internal loopback queues

Definition at line 88 of file stream.cpp.

References Stream::data(), and StreamData::tx_int_bytes_.

Referenced by Stream::tx_bytes().

size_t tx_int_items ( ) const

return number of items transmitted via internal loopback queues

Definition at line 84 of file stream.cpp.

References Stream::data(), and StreamData::tx_int_items_.

Referenced by Stream::tx_items().

size_t tx_items ( ) const

return number of items transmitted

Definition at line 32 of file stream.cpp.

References Stream::tx_int_items(), and Stream::tx_net_items().

size_t tx_net_blocks ( ) const

return number of blocks transmitted via network excluding internal tx

Definition at line 66 of file stream.cpp.

References Stream::data(), and StreamData::tx_net_blocks_.

Referenced by Stream::tx_blocks().

size_t tx_net_bytes ( ) const

return number of bytes transmitted via network excluding internal tx

Definition at line 62 of file stream.cpp.

References Stream::data(), and StreamData::tx_net_bytes_.

Referenced by Stream::tx_bytes().

size_t tx_net_items ( ) const

return number of items transmitted via network excluding internal tx

Definition at line 58 of file stream.cpp.

References Stream::data(), and StreamData::tx_net_items_.

Referenced by Stream::tx_items().


The documentation for this class was generated from the following files: