Code coverage for Protractor tests in AngularJS

Question

I am running some e2e tests in my angularJS app with protractor (as recommended in the angularJS documentation). I've googled around and cannot find any information on how to measure coverage for my protractor tests.

I think I'm missing something here... is there any way to get a code coverage report for protractor e2e tests? Or is it simply a feature for unit tests?

Solution 1

This is achievable using Istanbul. Here is the process, with some example configurations that I've extracted from our project (not tested):

1. Instrument your code using the command istanbul instrument. Make sure that istanbul's coverage variable is __coverage__.

   // gulpfile.js
   
   gulp.task('concat', function () {
       gulp.src(PATH.src)
         // Instrument for protractor-istanbul-plugin:
         .pipe(istanbul({coverageVariable: '__coverage__'}))
         .pipe(concat('scripts.js'))
         .pipe(gulp.dest(PATH.dest))
   });
   

2. Configure Protractor with the plugin protractor-istanbul-plugin.

   // spec-e2e.conf.js
   var istanbulPlugin = require('protractor-istanbul-plugin');
   
   exports.config = {
       // [...]
       plugins: [{ inline: istanbulPlugin }]
   };
   

3. Run your tests.

4. Extract the reports using istanbul report.

This approach has worked for me and is easy to combine with coverage reports from unit tests as well. To automate, I've put step 1 into my gulpfile.js and step 3 and 4 in the test and posttest scripts in package.json, more or less like this:

// In package.json:
"scripts": {
  "test": "gulp concat && protractor tests/spec-e2e.conf.js",
  "posttest": "istanbul report --include coverage/**/.json --dir reports/coverage cobertura"
},

Solution 2

if you are using grunt - you can use grunt-protractor-coverage plugin, it will do the job for you. You will have to instrument the code first and then use the mentioned plugin to create coverage reports for you.

Solution 3

To add to ryanb's answer, I haven't tried this but you should be able to use something like gulp-istanbul to instrument the code and override the default coverage variable, then define an onComplete function on the jasmineNodeOpts object in your Protractor config file. It gets called once right before everything is closed down.

exports.config = {

  // ...

  jasmineNodeOpts: {
    onComplete: function(){
      browser.driver.executeScript("return __coverage__;").then(function(val) {
        fs.writeFileSync("/path/to/coverage.json", JSON.stringify(val));
      });
    }
  }
};

Solution 4

I initially tried the onComplete method suggested by daniellmb, but getting the coverage results only at the end will not include all the results if there were multiple page loads during the tests. Here's a gist that sums up how I got things working, but basically I had to create a reporter that added coverage results to the instanbul collector every time a spec finished, and then wrote the reports in the onComplete method. I also had to use a "waitPlugin" as suggested by sjelin to prevent protractor from exiting before the results were written.

https://gist.github.com/jbarrus/286cee4294a6537e8217

Solution 5

I managed to get it working, but it's a hack at the moment. I use one of the existing grunt istanbul plugins to instrument the code. Then I made a dummy spec that grabs the 'coverage' global variable and write it to a file. After that, you can create a report with any of the reporting plugins.

The (very over-simplified) test looks like:

describe('Output the code coverage objects', function() {
    it('should output the coverage object.', function() {
        browser.driver.executeScript("return __coverage__;").then(function(val) {
            fs.writeFileSync("/path/to/coverage.json", JSON.stringify(val));
        });
    });
});

View more solutions

Comments

[ezequielc]

Could you share a sample gulpfile.js? I can't get coverage variable in the browser. Thanks.

[Dag Høidahl]

@ezequielc Updated the answer with example configurations. Hope it helps!

[ezequielc]

Thanks @dag-høidahl. My instrumented files were not being generated until I removed .pipe(istanbul.hookRequire()) like in your example. Cheers

[Chetan Laddha]

Does this works with Angular Application also? It worked for me angularjs application but facing issue with Angular 5 application..?

[Tester]

@DagHøidahl if a have a lot of gulpfile.js in my project which is the right one to put the configuration?

[Dag Høidahl]

@Tester That's impossible for me to tell. You or your team needs to decide that according to how you want to structure your project.

[Tester]

But if I put it inside a guplfile that is inside node_modules/receipe or node_modules/reflect_metadata is there any difference or rule?

[Dag Høidahl]

@Tester Sorry, I can't help you with that.