Sequence-zip function for c++11?

Question

With the new range-based for loop we can write code like

for(auto x: Y) {}

Which IMO is a huge improvement from (for ex.)

for(std::vector<int>::iterator x=Y.begin(); x!=Y.end(); ++x) {}

Can it be used to loop over two simultaneous loops, like Pythons zip function? For those unfamiliar with Python, the code:

Y1 = [1,2,3]
Y2 = [4,5,6,7]
for x1,x2 in zip(Y1,Y2):
    print x1,x2

Gives as output (1,4) (2,5) (3,6)

Solution 1

Warning: boost::zip_iterator and boost::combine as of Boost 1.63.0 (2016 Dec 26) will cause undefined behavior if the length of the input containers are not the same (it may crash or iterate beyond the end).

---

Starting from Boost 1.56.0 (2014 Aug 7) you could use boost::combine (the function exists in earlier versions but undocumented):

#include <boost/range/combine.hpp>
#include <vector>
#include <list>
#include <string>

int main() {
    std::vector<int> a {4, 5, 6};
    double b[] = {7, 8, 9};
    std::list<std::string> c {"a", "b", "c"};
    for (auto tup : boost::combine(a, b, c, a)) {    // <---
        int x, w;
        double y;
        std::string z;
        boost::tie(x, y, z, w) = tup;
        printf("%d %g %s %d\n", x, y, z.c_str(), w);
    }
}

This would print

4 7 a 4
5 8 b 5
6 9 c 6

---

In earlier versions, you could define a range yourself like this:

#include <boost/iterator/zip_iterator.hpp>
#include <boost/range.hpp>

template <typename... T>
auto zip(T&&... containers) -> boost::iterator_range<boost::zip_iterator<decltype(boost::make_tuple(std::begin(containers)...))>>
{
    auto zip_begin = boost::make_zip_iterator(boost::make_tuple(std::begin(containers)...));
    auto zip_end = boost::make_zip_iterator(boost::make_tuple(std::end(containers)...));
    return boost::make_iterator_range(zip_begin, zip_end);
}

The usage is the same.

Solution 2

std::transform can do this trivially:

std::vector<int> a = {1,2,3,4,5};
std::vector<int> b = {1,2,3,4,5};
std::vector<int>c;
std::transform(a.begin(),a.end(), b.begin(),
               std::back_inserter(c),
               [](const auto& aa, const auto& bb)
               {
                   return aa*bb;
               });
for(auto cc:c)
    std::cout<<cc<<std::endl;

If the second sequence is shorter, my implementation seems to be giving default initialized values.

Solution 3

So I wrote this zip before when I was bored, I decided to post it because it's different than the others in that it doesn't use boost and looks more like the c++ stdlib.

template <typename Iterator>
    void advance_all (Iterator & iterator) {
        ++iterator;
    }
template <typename Iterator, typename ... Iterators>
    void advance_all (Iterator & iterator, Iterators& ... iterators) {
        ++iterator;
        advance_all(iterators...);
    } 
template <typename Function, typename Iterator, typename ... Iterators>
    Function zip (Function func, Iterator begin, 
            Iterator end, 
            Iterators ... iterators)
    {
        for(;begin != end; ++begin, advance_all(iterators...))
            func(*begin, *(iterators)... );
        //could also make this a tuple
        return func;
    }

Example use:

int main () {
    std::vector<int> v1{1,2,3};
    std::vector<int> v2{3,2,1};
    std::vector<float> v3{1.2,2.4,9.0};
    std::vector<float> v4{1.2,2.4,9.0};
     zip (
            [](int i,int j,float k,float l){
                std::cout << i << " " << j << " " << k << " " << l << std::endl;
            },
            v1.begin(),v1.end(),v2.begin(),v3.begin(),v4.begin());
}

Solution 4

With range-v3:

#include <range/v3/all.hpp>
#include <vector>
#include <iostream>

namespace ranges {
    template <class T, class U>
    std::ostream& operator << (std::ostream& os, common_pair<T, U> const& p)
    {
      return os << '(' << p.first << ", " << p.second << ')';
    }
}

using namespace ranges::v3;

int main()
{
    std::vector<int> a {4, 5, 6};
    double b[] = {7, 8, 9};
    std::cout << view::zip(a, b) << std::endl; 
}

The output:

[(4, 7),(5, 8),(6, 9)]

Solution 5

See <redi/zip.h> for a zip function which works with range-base for and accepts any number of ranges, which can be rvalues or lvalues and can be different lengths (iteration will stop at the end of the shortest range).

std::vector<int> vi{ 0, 2, 4 };
std::vector<std::string> vs{ "1", "3", "5", "7" };
for (auto i : redi::zip(vi, vs))
  std::cout << i.get<0>() << ' ' << i.get<1>() << ' ';

Prints 0 1 2 3 4 5

View more solutions

Comments

[Xeo]

Why won't it work with range-based for? Combine it with Boost.Range and you should be set.

[Cat Plus Plus]

@Xeo: I don't know Range too well. I guess you could involve some boilerplate and make it work, but IMO just using for_each would be less hassle.

[Nicol Bolas]

+1: Boost.Range should probably incorporate this. In fact, I'll drop them a feature request on this.

[Alexandre C.]

@NicolBolas: You do well. This should be quite easy to implement with boost::iterator_range + boost::zip_iterator, even the variadic version.

[UncleBens]

You mean something like this is not hassle: std::for_each(make_zip_iterator(make_tuple(Y1.begin(), Y2.begin())), make_zip_iterator(make_tuple(Y1.end(), Y2.end())), [](const tuple<int, int>& t){printf("%d %d\n", get<0>(t), get<1>(t)); });?

[Cat Plus Plus]

@UncleBens: "Less hassle", not "not hassle". :P Again, I don't know Range very well, maybe it'll look better after all.

[UncleBens]

I should start a Lambda Does Not Make std::for_each Useful campaign. :)

[Xeo]

@UncleBens: Even C++ gurus (like Herb Sutter) state that std::for_each with a lambda is to be preferred over range-based for in most cases.

[UncleBens]

@Xeo: This should probably be a separate question, but why oh why??

[gnzlbg]

could you use this for sorting? i.e. std::sort(zip(a.begin(),...),zip(a.end(),...),[](tup a, tup b){a.get<0>() > b.get<0>()}); ?

[kennytm]

@gnzlbg: No you can't.

[Jonathan Wakely]

I believe this will never terminate (and have undefined behaviour) if the ranges are not the same length.

[Alexandre C.]

@JonathanWakely: Good point. However, I believe that boost::zip_iterator does the Right Thing here. If it does not, then the code above may need more work indeed.

[Jonathan Wakely]

boost::zip_iterator does not work with ranges of different lengths

[Yakk - Adam Nevraumont]

I would be tempted by optional elements for past-the-end iteration possibilities...

[Xeo]

You should check if any of the iterators is at the end.

[aaronman]

@Xeo all the ranges should be the same size as the first or greater

[Hooked]

Can you explain how [](int i,int j,float k,float l) works? Is this a lambda function?

[aaronman]

@Hooked yeah it's a lambda, it basically works just std::for_each but you can use an arbitrary number of ranges, the parameters in the lambda depend on how many iterators you give the function

[aaronman]

@Xeo could you explain your criticism if you still think it's valid, I would like to fix it

[Xeo]

A common need is to zip ranges of different size, or even with infinite ranges.

[aaronman]

@Xeo I see your point, it's just that stdlib functions like this usually just assume the first range is the smallest, that was the pattern I was going by

[Xeo]

The stdlib unfortunately doesn't always make the smartest choices. ;)

[Hooked]

This is neat! Can it take an arbitrary number of arguments are all those limited by your clever templating to a finite number?

[cshelton]

Currently it only handles up to 9 parallel containers. That would be simple to advance. While variadic macros allow for a single "zipfor" macro to handle different numbers of parameters, one still has to code up a separate macro for each (to be dispatched to). See groups.google.com/forum/?fromgroups=#!topic/comp.std.c/… and stackoverflow.com/questions/15847837/…

[Carneiro]

Any chance you can do this with std::make_tuple and std::tie ? I was trying to use this while minimizing the boost dependency but I couldn't make it work.

[kirill_igum]

you can also use boost/tuple/tuple_io.hpp to cout << i;

[Paul]

This should also work even in clean c++03 with pair instead of tuple. Still this wil also create problems when the lengths are not equal. Something might be done with the end() by taking the corresponding end() of the smallest container. This seems to be in the spec as it was in OPs question.

[coyotte508]

Does it handle arguments of different size well? (as described in the OP)

[cshelton]

@coyotte508, it assumes that the first container has the fewest number of elements (and ignores the extra elements in other containers). It would be easy to modify to not make this assumption, but that would slow it down (currently it is no slower than hand-written) when the number of elements match.

[Isac Casapu]

This is what worked for me. However, in my code I had to use the equivalent of boost::get<0>(i) and boost::get<1>(i). I'm not sure why the original sample could not be adapted directly, it might have to do with the fact that my code takes constant references to containers.

[javaLover]

link broken... would be useful if the post shows how to use it e.g. main() ?

[javaLover]

"like the c++ stdlib" Do you mind to show an example of stdlib? I am curious - never seen such function. (I am new to C++)

[winnetou]

@javaLover: you can use it the same way as cppitertools in @knedlsepp's answer. One notable difference is that with the above solution you can not modify the underlying containers as the operator* for seq::iterator returns a std::tuple of const references.

[Adrian]

If the 2nd sequence is shorter, then I'd expect that this is UB as you would be iterating off the end of b.

[Catskul]

@kennytm any idea why they decided to go with UB instead of just ending at the end of the shortest range in the bunch?

[Et7f3XIV]

If you implement an iterator then you could avoid creation of result and return as needed the next element. It need a bit more code as you need to define ++ * etc (all operator used by for (auto v : containers))

[Andrew]

@Et7f3XIV True, but looking at this code that '16 Andrew wrote I would nuke it from orbit and use one of the other answers as a starting point.

[c z]

@Adrian Partly - Should note that the UB is due to vector<>::iterator, not std::transform. The user should provide their own iterator to handle end-of-range if they expect it, e.g. by raising an error or returning zeros past the end.

[einpoklum]

I've posted a slight improvement - or what I believe is an improvement - to this answer; would you take a look?

[einpoklum]

Hundreds of lines of code, heavy use of the preprocessor, needs std::tuple (so even more code), plus still only supports zipping up to 9 things.

[einpoklum]

Please list your dependencies. Specifically, this requires Boost...