Bonus Step 6: Boost Qi

As Thrill is written in C++, one can draw from the vast universe of libraries available to C/C++ programs. Inside Map() lambda methods one can do just about anything.

As a simple example, we will show how to improve the parser using Boost.Spirit's Qi. Qi is a parser generator using C++ meta-template programming to generate highly optimized parsers. The problem with std::istringstream is that it uses way to many allocations and internal facet function calls to just parse two doubles.

thrill::DIA<Point> LoadPoints(thrill::Context& ctx, const char* path) {

    // shorthand namespace
    namespace qi = boost::spirit::qi;

    // load points from text file
    auto points =
        ReadLines(ctx, path)
        .Map(
            [](const std::string& input) {
                // parse "<x> <y>" lines
                Point p;
                std::string::const_iterator begin = input.begin(), end = input.end();

                qi::phrase_parse(
                    begin, end,                 // iterators
                    qi::double_ >> qi::double_, // parser grammar: two doubles
                    qi::ascii::space,           // skip grammar: spaces
                    p.x, p.y);                  // put directly into the Point

                if (begin != end)               // check that fully parsed
                    die("Could not parse point coordinates: " << input);
                return p;
            });
    return points.Cache();
}

The example above will generate a parser inline in the .Map() lambda which parses the input string directly into the Point struct.

See the complete example code examples/tutorial/k-means_step6.cpp

Author

Timo Bingmann (2016)