Detailed Description
Stream - base class for CatStream and MixStream.
Definition at line 36 of file stream.hpp.
Inheritance diagram for Stream: Collaboration diagram for Stream:#include <stream.hpp>
Public Types | |
| using | Writer = StreamData::Writer |
| using | Writers = StreamData::Writers |
Public Member Functions | |
| virtual | ~Stream () |
| void | Close () |
| shuts down the stream, waits for all closing blocks More... | |
| virtual StreamData & | data ()=0 |
| Return stream data reference. More... | |
| virtual const StreamData & | data () const =0 |
| Return stream data reference. More... | |
| virtual Writers | GetWriters ()=0 |
| virtual const StreamId & | id () const =0 |
| Return stream id. More... | |
| template<typename ItemType > | |
| void | Scatter (File &source, const std::vector< size_t > &offsets, bool consume=false) |
| Scatters a File to many worker: elements from [offset[0],offset[1]) are sent to the first worker, elements from [offset[1], offset[2]) are sent to the second worker, ..., elements from [offset[my_rank - 1],offset[my_rank]) are copied locally, ..., elements from [offset[num_workers - 1], offset[num_workers]) are sent to the last worker. More... | |
| template<typename ItemType > | |
| void | ScatterConsume (File &source, const std::vector< size_t > &offsets) |
| Consuming Version of Scatter() see documentation there. More... | |
| template<typename ItemType > | |
| void | ScatterKeep (File &source, const std::vector< size_t > &offsets) |
| Keep Version of Scatter() see documentation there. More... | |
Statistics | |
| size_t | tx_items () const |
| return number of items transmitted More... | |
| size_t | tx_bytes () const |
| return number of bytes transmitted More... | |
| size_t | tx_blocks () const |
| return number of blocks transmitted More... | |
| size_t | rx_items () const |
| return number of items received More... | |
| size_t | rx_bytes () const |
| return number of bytes received More... | |
| size_t | rx_blocks () const |
| return number of blocks received More... | |
| size_t | tx_net_items () const |
| return number of items transmitted via network excluding internal tx More... | |
| size_t | tx_net_bytes () const |
| return number of bytes transmitted via network excluding internal tx More... | |
| size_t | tx_net_blocks () const |
| return number of blocks transmitted via network excluding internal tx More... | |
| size_t | rx_net_items () const |
| return number of items received via network excluding internal tx More... | |
| size_t | rx_net_bytes () const |
| return number of bytes received via network excluding internal tx More... | |
| size_t | rx_net_blocks () const |
| return number of blocks received via network excluding internal tx More... | |
| size_t | tx_int_items () const |
| return number of items transmitted via internal loopback queues More... | |
| size_t | tx_int_bytes () const |
| return number of bytes transmitted via internal loopback queues More... | |
| size_t | tx_int_blocks () const |
| return number of blocks transmitted via internal loopback queues More... | |
| size_t | rx_int_items () const |
| return number of items received via network internal loopback queues More... | |
| size_t | rx_int_bytes () const |
| return number of bytes received via network internal loopback queues More... | |
| size_t | rx_int_blocks () const |
| return number of blocks received via network internal loopback queues More... | |
| Public Member Functions inherited from ReferenceCounter | |
| ReferenceCounter () noexcept | |
| new objects have zero reference count More... | |
| ReferenceCounter (const ReferenceCounter &) noexcept | |
| coping still creates a new object with zero reference count More... | |
| ~ReferenceCounter () | |
| bool | dec_reference () const noexcept |
| Call whenever resetting (i.e. More... | |
| void | inc_reference () const noexcept |
| Call whenever setting a pointer to the object. More... | |
| ReferenceCounter & | operator= (const ReferenceCounter &) noexcept |
| assignment operator, leaves pointers unchanged More... | |
| size_t | reference_count () const noexcept |
| Return the number of references to this object (for debugging) More... | |
| bool | unique () const noexcept |
| Test if the ReferenceCounter is referenced by only one CountingPtr. More... | |
Member Typedef Documentation
◆ Writer
| using Writer = StreamData::Writer |
Definition at line 39 of file stream.hpp.
◆ Writers
| using Writers = StreamData::Writers |
Definition at line 40 of file stream.hpp.
Constructor & Destructor Documentation
◆ ~Stream()
|
virtual |
Definition at line 23 of file stream.cpp.
Member Function Documentation
◆ Close()
| void Close | ( | ) |
shuts down the stream, waits for all closing blocks
Definition at line 26 of file stream.cpp.
References StreamData::Close(), and Stream::data().
◆ data() [1/2]
|
pure virtual |
Return stream data reference.
Implemented in CatStream, and MixStream.
Referenced by Stream::Close(), Stream::rx_int_blocks(), Stream::rx_int_bytes(), Stream::rx_int_items(), Stream::rx_net_blocks(), Stream::rx_net_bytes(), Stream::rx_net_items(), Stream::ScatterConsume(), Stream::tx_int_blocks(), Stream::tx_int_bytes(), Stream::tx_int_items(), Stream::tx_net_blocks(), Stream::tx_net_bytes(), and Stream::tx_net_items().
◆ data() [2/2]
|
pure virtual |
◆ GetWriters()
|
pure virtual |
Creates BlockWriters for each worker. BlockWriter can only be opened once, otherwise the block sequence is incorrectly interleaved!
Implemented in CatStream, and MixStream.
Referenced by Stream::ScatterConsume(), and Stream::ScatterKeep().
◆ id()
|
pure virtual |
◆ rx_blocks()
| size_t rx_blocks | ( | ) | const |
return number of blocks received
Definition at line 52 of file stream.cpp.
References Stream::rx_int_blocks(), and Stream::rx_net_blocks().
Referenced by Stream::ScatterKeep().
◆ rx_bytes()
| size_t rx_bytes | ( | ) | const |
return number of bytes received
Definition at line 48 of file stream.cpp.
References Stream::rx_int_bytes(), and Stream::rx_net_bytes().
Referenced by Stream::ScatterKeep().
◆ rx_int_blocks()
| size_t rx_int_blocks | ( | ) | const |
return number of blocks received via network internal loopback queues
Definition at line 104 of file stream.cpp.
References Stream::data(), and StreamData::rx_int_blocks_.
Referenced by Stream::rx_blocks(), and Stream::ScatterKeep().
◆ rx_int_bytes()
| size_t rx_int_bytes | ( | ) | const |
return number of bytes received via network internal loopback queues
Definition at line 100 of file stream.cpp.
References Stream::data(), and StreamData::rx_int_bytes_.
Referenced by Stream::rx_bytes(), and Stream::ScatterKeep().
◆ rx_int_items()
| size_t rx_int_items | ( | ) | const |
return number of items received via network internal loopback queues
Definition at line 96 of file stream.cpp.
References Stream::data(), and StreamData::rx_int_items_.
Referenced by Stream::rx_items(), and Stream::ScatterKeep().
◆ rx_items()
| size_t rx_items | ( | ) | const |
return number of items received
Definition at line 44 of file stream.cpp.
References Stream::rx_int_items(), and Stream::rx_net_items().
Referenced by Stream::ScatterKeep().
◆ rx_net_blocks()
| size_t rx_net_blocks | ( | ) | const |
return number of blocks received via network excluding internal tx
Definition at line 78 of file stream.cpp.
References Stream::data(), and StreamData::rx_net_blocks_.
Referenced by Stream::rx_blocks(), and Stream::ScatterKeep().
◆ rx_net_bytes()
| size_t rx_net_bytes | ( | ) | const |
return number of bytes received via network excluding internal tx
Definition at line 74 of file stream.cpp.
References Stream::data(), and StreamData::rx_net_bytes_.
Referenced by Stream::rx_bytes(), and Stream::ScatterKeep().
◆ rx_net_items()
| size_t rx_net_items | ( | ) | const |
return number of items received via network excluding internal tx
Definition at line 70 of file stream.cpp.
References Stream::data(), and StreamData::rx_net_items_.
Referenced by Stream::rx_items(), and Stream::ScatterKeep().
◆ Scatter()
|
inline |
Scatters a File to many worker: elements from [offset[0],offset[1]) are sent to the first worker, elements from [offset[1], offset[2]) are sent to the second worker, ..., elements from [offset[my_rank - 1],offset[my_rank]) are copied locally, ..., elements from [offset[num_workers - 1], offset[num_workers]) are sent to the last worker.
The number of given offsets must be equal to the net::Group::num_hosts() * workers_per_host_ + 1.
/param source File containing the data to be scattered.
/param offsets - as described above. offsets.size must be equal to num_workers + 1
Definition at line 77 of file stream.hpp.
◆ ScatterConsume()
|
inline |
Consuming Version of Scatter() see documentation there.
Definition at line 89 of file stream.hpp.
References Stream::data(), File::GetConsumeReader(), Stream::GetWriters(), and StreamData::my_worker_rank().
◆ ScatterKeep()
|
inline |
Keep Version of Scatter() see documentation there.
Definition at line 149 of file stream.hpp.
References StreamData::Writers::Close(), File::GetKeepReader(), Stream::GetWriters(), BlockReader< BlockSource >::HasNext(), Stream::rx_blocks(), Stream::rx_bytes(), Stream::rx_int_blocks(), Stream::rx_int_bytes(), Stream::rx_int_items(), Stream::rx_items(), Stream::rx_net_blocks(), Stream::rx_net_bytes(), Stream::rx_net_items(), Stream::tx_blocks(), Stream::tx_bytes(), Stream::tx_int_blocks(), Stream::tx_int_bytes(), Stream::tx_int_items(), Stream::tx_items(), Stream::tx_net_blocks(), Stream::tx_net_bytes(), and Stream::tx_net_items().
◆ tx_blocks()
| size_t tx_blocks | ( | ) | const |
return number of blocks transmitted
Definition at line 40 of file stream.cpp.
References Stream::tx_int_blocks(), and Stream::tx_net_blocks().
Referenced by Stream::ScatterKeep().
◆ tx_bytes()
| size_t tx_bytes | ( | ) | const |
return number of bytes transmitted
Definition at line 36 of file stream.cpp.
References Stream::tx_int_bytes(), and Stream::tx_net_bytes().
Referenced by Stream::ScatterKeep().
◆ tx_int_blocks()
| size_t tx_int_blocks | ( | ) | const |
return number of blocks transmitted via internal loopback queues
Definition at line 92 of file stream.cpp.
References Stream::data(), and StreamData::tx_int_blocks_.
Referenced by Stream::ScatterKeep(), and Stream::tx_blocks().
◆ tx_int_bytes()
| size_t tx_int_bytes | ( | ) | const |
return number of bytes transmitted via internal loopback queues
Definition at line 88 of file stream.cpp.
References Stream::data(), and StreamData::tx_int_bytes_.
Referenced by Stream::ScatterKeep(), and Stream::tx_bytes().
◆ tx_int_items()
| size_t tx_int_items | ( | ) | const |
return number of items transmitted via internal loopback queues
Definition at line 84 of file stream.cpp.
References Stream::data(), and StreamData::tx_int_items_.
Referenced by Stream::ScatterKeep(), and Stream::tx_items().
◆ tx_items()
| size_t tx_items | ( | ) | const |
return number of items transmitted
Definition at line 32 of file stream.cpp.
References Stream::tx_int_items(), and Stream::tx_net_items().
Referenced by Stream::ScatterKeep().
◆ tx_net_blocks()
| size_t tx_net_blocks | ( | ) | const |
return number of blocks transmitted via network excluding internal tx
Definition at line 66 of file stream.cpp.
References Stream::data(), and StreamData::tx_net_blocks_.
Referenced by Stream::ScatterKeep(), and Stream::tx_blocks().
◆ tx_net_bytes()
| size_t tx_net_bytes | ( | ) | const |
return number of bytes transmitted via network excluding internal tx
Definition at line 62 of file stream.cpp.
References Stream::data(), and StreamData::tx_net_bytes_.
Referenced by Stream::ScatterKeep(), and Stream::tx_bytes().
◆ tx_net_items()
| size_t tx_net_items | ( | ) | const |
return number of items transmitted via network excluding internal tx
Definition at line 58 of file stream.cpp.
References Stream::data(), and StreamData::tx_net_items_.
Referenced by Stream::ScatterKeep(), and Stream::tx_items().
The documentation for this class was generated from the following files:
- thrill/data/stream.hpp
- thrill/data/stream.cpp
