How to access/modify matrix element in OpenCV? Why at() is templatized?

Question

Should I know Mat element type for use at() correctly? For example, if I have

Mat rose = Mat(1,180, CV_64F, 0);

then can I call

rose.at<short>(i,j)++;

If not then which template argument should I use?

Why Mat::at is templatized while Mat itself is not?

UPDATE

This question contained sample code with another error, which is now here: How to fill Matrix with zeros in OpenCV?

Solution 1

As already properly pointed out by William, you should provide only the correct type as a template argument for at. I believe that cv::Mat itself is not made template only for simplification. However, OpenCV crew are trying to support C++ features, including templates. The interface became slightly heterogeneous in this way.

You could not deduce compiler type from a type variable at runtime for obvious reason. However, you can deduce it at compile time, if your type variable is known at this point, using a traits class:

template<int I>
struct CvType {};

template<>
struct CvType<CV_64F> { typedef double type_t; };
template<>
struct CvType<CV_32F> { typedef float type_t; };
template<>
struct CvType<CV_8U> { typedef unsigned char type_t; };
// Other types go here

void main()
{
  const int type = CV_64F;
  cv::Mat mat(10, 10, type);
  mat.at<CvType<type>::type_t>(1, 1);
}

In this case you can change the value of type and wouldn't need to change types manually for all the at or other methods calls.

Solution 2

Well now your edited post differs. Rectified:

Mat m1 = Mat(1,1, CV_64F, cvScalar(0.));
m1.at<double>(0,0) = 0;

or try different:

Mat m1 = cv::Mat::zeros(1,1, CV_64F);
m1.at<double>(0,0) = 0;

Solution 3

The Mat class is not a template class, which allows to change its "type" at runtime. Changing the type is useful, e.g. when reading from a file. Not being a template is convenient, when using a Mat as function parameter, since the function is not required to be a template function.

However, for accessing single elements (with pointers, at or iterators) you need the data type. I guess this is done for performance reasons. This contradicts to the runtime type system and makes it harder to write generic code, when you do not know the type at compile time. Nevertheless, you can do it with a workaround.

The simplest method is just to use an if-else-cascade:

Mat img = imread("test.png");
if (img.channels() == 1) {
    if (img.depth() == CV_8U) {
        cout << (int)(img.at<uint8_t>(0,0)) << endl;
    }
    else if (img.depth() == CV_8S) {
        /* ... */
    }
    /* ... */
}
/* ... */
else if (img.channels() == 3) {
    if (img.depth() == CV_8U) {
        auto p = img.at<array<uint8_t,3>>(0,0);
        cout << (int)(p[0]) << ";" << (int)(p[1]) << ";" << (int)(p[2]) << endl;
    }
    /* ... */
}
/* ... */

But you can imagine that this becomes cumbersome, if you write it out for all types and channels. You have to limit the number of channels anyway, since there is no hard limit by OpenCV. We choose 4 in the following.

I wrote a helper template metaprogram header, that does the job. You can provide a functor with a templated operator(). Then call the template metaprogram, which will call the functor with a compile time type. See this example for a functor that prints the first pixel and returns whether it is not all zero:

struct PrintPixel {
    Mat img;

    // this template function will be called from the metaprogram
    template<int cv_type> // compile time value e.g. CV_8UC3
    bool operator()() {
        using elem_t  = typename CvTypeTraits<cv_type>::base_type;
        using array_t = typename CvTypeTraits<cv_type>::array_type;
        // you could also do static_asserts here

        array_t pixel = img.at<array_t>(0,0);
        for (elem_t val : pixel)
            cout << (double)(val) << ", ";
        cout << endl;
        return any_of(pixel.begin(), pixel.end(), [](elem_t v){return v != 0;});
    }
};

Note, the return type of the operator() can be arbitrary, but may not depend on the image type cv_type unfortunately. This is because it is also used as the return type of the function that holds the if-else cascade (the run function, see below).

Here is the calling code, which can check for "all" channels (1-4) and types or for a specified set:

Mat img = imread("test.png");
int t = img.type();

// call functor, check for 1-4 channels and all 7 base types
bool not_zero = CallFunctor::run(PrintPixel{img}, t);

// call functor, check only for 1 or 3 channels and 8 bit unsigned int
CallFunctorRestrictChannelsTo<1,3>::AndBaseTypesTo<CV_8U>::run(PrintPixel{img}, t);

The latter call will throw an exception, if t is not CV_8UC1 or CV_8UC3. If you are often using the same restriction, you can abbreviate it, with a using declaration (see at the bottom of the header file below).

So, this is an easy-to-use solution, which allows you to use a compile time value "made" from a run time value. But remember, in the background an if-else-cascade is doing all the checks (in the order the channels and types were specified). This implies that for each combination of channel and type that is checked one concrete functor class is generated. If it is large, that might be bad. So it should only include the type dependent parts. It could also be a factory functor that instanciates a templated class with a non-template base with virtual functions to reduce code size.

It follows the header file, which contains the CvTypeTraits class and the template metaprogram functions. At the bottom you can see that the CallFunctor type is really just an abbreviation for a "restriction" of types and channels. You could also declare something like that with other restrictions.

#pragma once

#include <cstdint>
#include <type_traits>
#include <array>
#include <opencv2/core/types_c.h>


template<int> struct BaseType { };
template<> struct BaseType<CV_8S>  { using base_type = int8_t;   };
template<> struct BaseType<CV_8U>  { using base_type = uint8_t;  };
template<> struct BaseType<CV_16S> { using base_type = int16_t;  };
template<> struct BaseType<CV_16U> { using base_type = uint16_t; };
template<> struct BaseType<CV_32S> { using base_type = int32_t;  };
template<> struct BaseType<CV_32F> { using base_type = float;    };
template<> struct BaseType<CV_64F> { using base_type = double;   };


template<int t>
struct CvTypeTraits {
    constexpr static int channels = t / CV_DEPTH_MAX + 1;
    using base_type = typename BaseType<t % CV_DEPTH_MAX>::base_type;
    using array_type = std::array<base_type, channels>;
};


template<int currentChannel, int... otherChannels>
struct find_chan_impl {
    template<typename ret_type, int... types>
    struct find_type_impl {
        template<class Functor>
        static inline ret_type run(Functor&& f, int const& c, int const& t) {
            if (c == currentChannel)
                return find_chan_impl<currentChannel>::template find_type_impl<ret_type, types...>::run(std::forward<Functor>(f), c, t);
            else
                return find_chan_impl<otherChannels...>::template find_type_impl<ret_type, types...>::run(std::forward<Functor>(f), c, t);
        }
    };
};

template<>
struct find_chan_impl<0> {
    template<typename ret_type, int... types>
    struct find_type_impl {
        template<class Functor>
        [[noreturn]] static inline ret_type run(Functor&& f, int const& c, int const& t) {
            throw std::runtime_error("The image has " + std::to_string(c) + " channels, but you did not try to call the functor with this number of channels.");
        }
    };
};

template<int channel>
struct find_chan_impl<channel> {
    template<typename ret_type, int currentType, int... otherTypes>
    struct find_type_impl {
        static_assert(currentType < CV_DEPTH_MAX, "You can only restrict to base types, without channel specification");

        template<class Functor>
        static inline ret_type run(Functor&& f, int const& c, int const& t) {
            if (t == currentType)
                return find_type_impl<ret_type, currentType>::run(std::forward<Functor>(f), c, t);
            else
                return find_type_impl<ret_type, otherTypes...>::run(std::forward<Functor>(f), c, t);
        }
    };

    template<typename ret_type, int type>
    struct find_type_impl<ret_type, type> {
        template<class Functor>
        static inline ret_type run(Functor&& f, int const& c, int const& t) {
            return f.template operator()<CV_MAKETYPE(type,channel)>();
        }
    };

    template<typename ret_type>
    struct find_type_impl<ret_type, -1> {
        template<class Functor>
        [[noreturn]] static inline ret_type run(Functor&& f, int const& c, int const& t) {
            throw std::runtime_error("The image is of base type " + std::to_string(t) + ", but you did not try to call the functor with this base type.");
        }
    };
};

template<int... channels>
struct CallFunctorRestrictChannelsTo {
    template<int firstType, int... types>
    struct AndBaseTypesTo {
        template<class Functor>
        static inline auto run(Functor&& f, int t) -> decltype(f.template operator()<firstType>()) {
            using functor_ret_type = decltype(f.template operator()<firstType>());
            std::div_t d = std::div(t, CV_DEPTH_MAX);
            int c             = d.quot + 1;
            int const& base_t = d.rem;
            return find_chan_impl<channels..., 0>::template find_type_impl<functor_ret_type, firstType, types..., -1>::run(std::forward<Functor>(f), c, base_t);
        }
    };

    template<class Functor>
    static inline auto run(Functor&& f, int t) -> decltype(f.template operator()<CV_8S>()) {
        return AndBaseTypesTo<CV_8S, CV_8U, CV_16S, CV_16U, CV_32S, CV_32F, CV_64F>::run(std::forward<Functor>(f), t);
    }
};

template<int... types>
using CallFunctorRestrictBaseTypesTo = CallFunctorRestrictChannelsTo<1,2,3,4>::template AndBaseTypesTo<types...>;

using CallFunctor = CallFunctorRestrictChannelsTo<1,2,3,4>::template AndBaseTypesTo<CV_8S, CV_8U, CV_16S, CV_16U, CV_32S, CV_32F, CV_64F>;