Detailed Description
template<typename StreamData>
class thrill::data::StreamSet< StreamData >
Simple structure that holds a all stream instances for the workers on the local host for a given stream id.
Definition at line 33 of file multiplexer.hpp.
Inheritance diagram for StreamSet< StreamData >: Collaboration diagram for StreamSet< StreamData >:#include <multiplexer.hpp>
Public Types | |
| using | StreamDataPtr = tlx::CountingPtr< StreamData > |
Public Member Functions | |
| StreamSet (Multiplexer &multiplexer, size_t send_size_limit, StreamId id, size_t workers_per_host, size_t dia_id) | |
| void | Close () final |
| Close all StreamData objects. More... | |
| template<> | |
| MagicByte | magic_byte () const |
| template<> | |
| MagicByte | magic_byte () const |
| MagicByte | magic_byte () const |
| size_t | my_host_rank () const |
| Returns my_host_rank. More... | |
| size_t | num_hosts () const |
| Number of hosts in system. More... | |
| void | OnWriterClosed (size_t peer_worker_rank, bool sent) |
| StreamDataPtr | Peer (size_t local_worker_id) |
| bool | Release (size_t local_worker_id) |
| size_t | workers_per_host () const |
| Returns workers_per_host. More... | |
| Public Member Functions inherited from StreamSetBase | |
| virtual | ~StreamSetBase () |
| Public Member Functions inherited from ReferenceCounter | |
| ReferenceCounter () noexcept | |
| new objects have zero reference count More... | |
| ReferenceCounter (const ReferenceCounter &) noexcept | |
| coping still creates a new object with zero reference count More... | |
| ~ReferenceCounter () | |
| bool | dec_reference () const noexcept |
| Call whenever resetting (i.e. More... | |
| void | inc_reference () const noexcept |
| Call whenever setting a pointer to the object. More... | |
| ReferenceCounter & | operator= (const ReferenceCounter &) noexcept |
| assignment operator, leaves pointers unchanged More... | |
| size_t | reference_count () const noexcept |
| Return the number of references to this object (for debugging) More... | |
| bool | unique () const noexcept |
| Test if the ReferenceCounter is referenced by only one CountingPtr. More... | |
Private Attributes | |
| StreamId | id_ |
| stream id More... | |
| Multiplexer & | multiplexer_ |
| reference to multiplexer More... | |
| std::mutex | mutex_ |
| mutex for working on the data structure More... | |
| size_t | remaining_ |
| countdown to destruction More... | |
| std::vector< StreamDataPtr > | streams_ |
| 'owns' all streams belonging to one stream id for all local workers. More... | |
| std::vector< size_t > | writers_closed_per_host_ |
| number of writers closed per host, message is set when all are closed More... | |
| std::vector< size_t > | writers_closed_per_host_sent_ |
| number of writers closed per host, message is set when all are closed More... | |
Additional Inherited Members | |
| Static Public Attributes inherited from StreamSetBase | |
| static constexpr bool | debug = false |
Member Typedef Documentation
◆ StreamDataPtr
| using StreamDataPtr = tlx::CountingPtr<StreamData> |
Definition at line 224 of file stream_data.hpp.
Constructor & Destructor Documentation
◆ StreamSet()
| StreamSet | ( | Multiplexer & | multiplexer, |
| size_t | send_size_limit, | ||
| StreamId | id, | ||
| size_t | workers_per_host, | ||
| size_t | dia_id | ||
| ) |
Creates a StreamSet with the given number of streams (num workers per host).
Definition at line 108 of file stream_data.cpp.
References StreamSet< StreamData >::num_hosts(), StreamSet< StreamData >::remaining_, StreamSet< StreamData >::streams_, StreamSet< StreamData >::workers_per_host(), StreamSet< StreamData >::writers_closed_per_host_, and StreamSet< StreamData >::writers_closed_per_host_sent_.
Member Function Documentation
◆ Close()
|
finalvirtual |
Close all StreamData objects.
Implements StreamSetBase.
Definition at line 140 of file stream_data.cpp.
References StreamSet< StreamData >::streams_.
◆ magic_byte() [1/3]
| MagicByte magic_byte | ( | ) | const |
Definition at line 211 of file stream_data.cpp.
References thrill::data::CatStreamBlock.
◆ magic_byte() [2/3]
| MagicByte magic_byte | ( | ) | const |
Definition at line 216 of file stream_data.cpp.
References thrill::data::MixStreamBlock.
◆ magic_byte() [3/3]
|
inline |
Referenced by StreamSet< StreamData >::OnWriterClosed().
◆ my_host_rank()
|
inline |
Returns my_host_rank.
Definition at line 247 of file stream_data.hpp.
Referenced by StreamSet< StreamData >::OnWriterClosed().
◆ num_hosts()
|
inline |
Number of hosts in system.
Definition at line 249 of file stream_data.hpp.
Referenced by StreamSet< StreamData >::StreamSet().
◆ OnWriterClosed()
|
virtual |
method called from StreamSink when it is closed, used to aggregate Close messages to remote hosts
Implements StreamSetBase.
Definition at line 146 of file stream_data.cpp.
References StreamMultiplexerHeader::all_workers, DispatcherThread::AsyncWrite(), Group::connection(), die_unless, Multiplexer::dispatcher(), StreamMultiplexerHeader::final_seq, Multiplexer::group(), StreamSet< StreamData >::id_, LOG, MultiplexerHeader::magic, StreamSet< StreamData >::magic_byte(), StreamSet< StreamData >::multiplexer_, StreamSet< StreamData >::mutex_, StreamSet< StreamData >::my_host_rank(), StreamMultiplexerHeader::receiver_local_worker, StreamMultiplexerHeader::sender_worker, StreamMultiplexerHeader::seq, StreamMultiplexerHeader::Serialize(), Buffer::size(), StreamMultiplexerHeader::stream_id, BufferBuilder::ToBuffer(), MultiplexerHeader::total_size, StreamSet< StreamData >::workers_per_host(), StreamSet< StreamData >::writers_closed_per_host_, and StreamSet< StreamData >::writers_closed_per_host_sent_.
◆ Peer()
| tlx::CountingPtr< StreamData > Peer | ( | size_t | local_worker_id | ) |
Returns the stream that will be consumed by the worker with the given local id
Definition at line 122 of file stream_data.cpp.
References StreamSet< StreamData >::streams_.
◆ Release()
| bool Release | ( | size_t | local_worker_id | ) |
Release local_worker_id, returns true when all individual streams are done.
Definition at line 128 of file stream_data.cpp.
References StreamSet< StreamData >::mutex_, StreamSet< StreamData >::remaining_, and StreamSet< StreamData >::streams_.
◆ workers_per_host()
|
inline |
Returns workers_per_host.
Definition at line 251 of file stream_data.hpp.
Referenced by StreamSet< StreamData >::OnWriterClosed(), and StreamSet< StreamData >::StreamSet().
Member Data Documentation
◆ id_
|
private |
stream id
Definition at line 259 of file stream_data.hpp.
Referenced by StreamSet< StreamData >::OnWriterClosed().
◆ multiplexer_
|
private |
reference to multiplexer
Definition at line 257 of file stream_data.hpp.
Referenced by StreamSet< StreamData >::OnWriterClosed().
◆ mutex_
|
private |
mutex for working on the data structure
Definition at line 269 of file stream_data.hpp.
Referenced by StreamSet< StreamData >::OnWriterClosed(), and StreamSet< StreamData >::Release().
◆ remaining_
|
private |
countdown to destruction
Definition at line 263 of file stream_data.hpp.
Referenced by StreamSet< StreamData >::Release(), and StreamSet< StreamData >::StreamSet().
◆ streams_
|
private |
'owns' all streams belonging to one stream id for all local workers.
Definition at line 261 of file stream_data.hpp.
Referenced by StreamSet< StreamData >::Close(), StreamSet< StreamData >::Peer(), StreamSet< StreamData >::Release(), and StreamSet< StreamData >::StreamSet().
◆ writers_closed_per_host_
|
private |
number of writers closed per host, message is set when all are closed
Definition at line 265 of file stream_data.hpp.
Referenced by StreamSet< StreamData >::OnWriterClosed(), and StreamSet< StreamData >::StreamSet().
◆ writers_closed_per_host_sent_
|
private |
number of writers closed per host, message is set when all are closed
Definition at line 267 of file stream_data.hpp.
Referenced by StreamSet< StreamData >::OnWriterClosed(), and StreamSet< StreamData >::StreamSet().
The documentation for this class was generated from the following files:
- thrill/data/multiplexer.hpp
- thrill/data/stream_data.hpp
- thrill/data/stream_data.cpp
