Jacoco Unit and Integration Tests coverage - individual and overall

Question

I have a project (ProjectA) which contains some unit tests and integration tests.

Following is the structure.

ProjectA - src/java (java source code)
- test/java (Junit unit tests)
- test/resources (resources required for Junit Unit tests)
- src/java-test (Integration tests)
- conf (contains .xml files required for building/testing/compiling purposes)

I run the following commands -- All of them works but I have a doubt on how the configurations that I have in build.gradle / GRADLE_HOME/init.d/*.gradle files are affecting what I'm getting.

It seems like I'm missing something and not getting where I want what.

Commands:
- gradle clean build -- it works fine
- gradle clean build jacocoTestReport -- it works fine.
- gradle clean build integrationTest jacocoTestReport -- it works fine (if I have a tomcat instance up and running in another putty window for the same ProjectA).

After the 3rd bullet operation is complete, I see the extra folder "build" and its subfolders (other than what's checked out from source/version control) in my Jenkins jobs workspace.

i.e. under -- JenkinsWorkspace
/build
- classes (contains .class files for the following which are mentioned as one of the sourceSets section)
---- integrationTest
---- main
---- test

- resources (this contains all the files .properties/.xml files which were under "conf" folder in source control.

- reports (contains .xml/.html files for PMD/CheckStyle/Findbugs and Tests results for either Unit or IT Tests but NOT both). ---- checkstyle
---- findbugs
---- pmd
---- jacoco
---- tests (Note: this is plural i.e. it's not "test" which is defined as one entry in sourceSets)

- jacoco (This contains 2 .exec files i.e. test.exec and integrationTest.exec both have different file size)
---- test.exec
---- integrationTest.exec

- jacocoHtml (This folder contains lots of folders (containing .html files) and mainly "index.html" under it.
---- somexxxfolders
---- ---- somexxfolder's.html files
---- index.html
---- other etc files/folders

- test-results (This contains some .xml files BUT only for either Unit tests or Integration tests - but not for both of the test types at a given time).

i.e. if I run "gradle clean build", then you'll see Unit test related .xml files and if I run "gradle clean build integrationTest", then Unit test .xml files are overwritten and the .xml files I see are only related to/generated by integrationTest task.

Following is one of the common gradle (GRADLE_HOME/init.d/some.common.gradle file)

//
//Extra file can hold global Gradle settings so that these dont have to be inserted in project
//specific build.gradle file.
//Filename: extraN.common<anyname>.gradle
allprojects {
   apply plugin: 'java'
   apply plugin: 'pmd'
   apply plugin: 'findbugs'
   apply plugin: 'checkstyle'
   apply plugin: 'jacoco'
   apply plugin: 'sonar-runner'
   tasks.withType(Compile) {
     options.debug = true
     options.compilerArgs = ["-g"]
   }
   sourceSets {
      main {
         java {
            // MOTE: If your project's build.gradle specify the sourceSet section, the following
            // values will be overwritten by what project's build.gradle will set.
            //
            // If you project's sourceSet structure if different in each project, then in this
            // global common .gradle file, you can define srcDir for main or other sections like
            // test, integrationTest etc as shown below -- commented out. If that's the case, 
            // then uncomment the below line and comment out using // -- srcDir 'src/java' line
            // for sourceSets.main.java section. This rule applies to other sections as well.
            // srcDir 'no_src_dir_set_yet'

            srcDir 'src/java'
         }
         resources {
            srcDir 'conf'
         }
      }
      test {
         java {
            srcDir 'test/java'
         }
         resources {
            srcDir 'test/resources'
            srcDir 'conf'
         }
      }
      integrationTest {
         java {
            srcDir 'src/java-test'
         }
         resources {
            srcDir 'conf'
         }
      }
   }
   def sonarServerUrl = "dev.sandbox.server.com"
   sonarRunner {
      sonarProperties {
         property "sonar.host.url", "http://$sonarServerUrl:9000"
         property "sonar.jdbc.url", "jdbc:h2:tcp://$sonarServerUrl:9092/sonar"
         property "sonar.jdbc.driverClassName", "org.h2.Driver"
         property "sonar.jdbc.username", "sonar"
         property "sonar.jdbc.password", "sonar"
         properties ["sonar.sources"] += sourceSets.main.allSource.srcDirs
         //properties ["sonar.tests"] += sourceSets.test.java.srcDirs
         properties ["sonar.tests"] += sourceSets.integrationTest.allSource.srcDirs
      }
   }
   checkstyle {
        configFile = new File(rootDir, "config/checkstyle.xml")
        ignoreFailures = true
        //sourceSets = [sourceSets.main, sourceSets.test, sourceSets.integrationTest]
        sourceSets = [sourceSets.main]
   }
   findbugs {
        ignoreFailures = true
        sourceSets = [sourceSets.main]
   }
   pmd {
        ruleSets = ["basic", "braces", "design"]
        ignoreFailures = true
   }
   jacoco {
      toolVersion = "0.6.2.201302030002"
      reportsDir = file("$buildDir/customJacocoReportDir")
   }
   task testReport(type: TestReport) {
      destinationDir = file("$buildDir/reports/allTests")
   }
   test {
        jacoco {
            //destinationFile = file("$buildDir/jacoco/jacocoTest.exec")
            destinationFile = file("$buildDir/jacoco/test.exec")
            //classDumpFile = file("$buildDir/jacoco/classpathdumps")
            classDumpFile = file("$buildDir/build/classes/test")
        }
   }
   jacocoTestReport {
         group = "Reporting"
         description = "Generate Jacoco coverage reports after running tests."
         reports {
                xml{
                    enabled true
                    destination "${buildDir}/reports/jacoco/jacoco.xml"
                }
                csv.enabled false
                html{
                    enabled true
                    destination "${buildDir}/jacocoHtml"
                }
        }
        additionalSourceDirs = files(sourceSets.main.allJava.srcDirs)
        //additionalSourceDirs = files([sourceSets.main.allJava.srcDirs, xxxx, 'xxxxyyyy' ])
   }
}

build.gradle file looks like:

import com.tr.ids.gradle.CustomFileUtil
apply plugin: 'CustomSetup'
apply plugin: 'java'
apply plugin: 'customJarService'
apply plugin: 'customWarService'
sourceSets {
   main {
      java {
         srcDir 'src/java'
      }
   }
   test {
      java {
         srcDir 'test/java'
      }
      resources {
         srcDir 'test/resources'
         srcDir 'conf'
      }
   }
   integrationTest {
      java {
         srcDir 'src/java-test'
      }
   }
}
// Read dependency lists from external files. Our custom plugin reads a dep-xxx.txt file for compile/test/war related .jar file entries
// where each entry is like: groupid:artifactid:x.x.x
// and these artifact jars are available in Artifactory
List depListCompile = customFileUtil.readIntoList( "$projectDir/dep-compile.txt" )
List depListTest = customFileUtil.readIntoList( "$projectDir/dep-testArtifacts.txt" )
List depListWar = customFileUtil.readIntoList( "$projectDir/dep-war.txt" )
// Define dependencies
dependencies {
   // Compilation
   compile  depListCompile
   // Unit Tests
   testCompile depListTest
   // Integration tests
   // Everything from compile and testCompile targets
   integrationTestCompile configurations.compile
   integrationTestCompile configurations.testCompile
   // Output of compiling "main" files
   integrationTestCompile sourceSets.main.output
   // Additional dependencies from war and others
   integrationTestCompile depListTest, depListWar
   // All configuration files
   integrationTestRuntime files( 'conf' )
}
task deployTomcat( type: Copy, dependsOn: [ jar, compileIntegrationTestJava, warService ] ) {
   from "$buildDir/customWar/${project.name}.war"
   into "$projectDir/tomcat/webapps"
}
build {
  dependsOn deployTomcat
}
task integrationTest( type: Test, dependsOn: cleanTest ) {
   jacoco {
      //destinationFile = file("$buildDir/jacoco/jacocoTest.exec")
      destinationFile = file("$buildDir/jacoco/integrationTest.exec")
      //classDumpFile = file("$buildDir/jacoco/classpathdumps")
      classDumpFile = file("$buildDir/classes/integrationTest")
   }
   testClassesDir = sourceSets.integrationTest.output.classesDir
   classpath = sourceSets.integrationTest.runtimeClasspath
}
apply plugin: 'eclipse'
eclipse.classpath {
   // Define output directory so Eclipse does not accidentally clobber /bin
   defaultOutputDir = file( 'out/classes' )
   // Add integration test
   plusConfigurations += configurations.integrationTestCompile
   // Remove unnecessary files
   file.whenMerged { classpath ->
      classpath.entries.removeAll { entry -> ( entry.path.indexOf( '/build/classes/main' ) > 0 ) }
      classpath.entries.removeAll { entry -> ( entry.path.indexOf( '/build/resources/main' ) > 0 ) }
   }
}

My questions:

1) Why "gradle clean build integrationTest" -- which is working successfully, is overwriting test results in build/reports/tests and build/test-results folders.

2) It doesn't matter what name I give for .exec file under common gradle file for test and in build.gradle for integrationTest task for jacoco, it always create test.exec and integrationTest.exec file but the resultant build/jacocoHtml folder index.html file doesn't show coverage / files related to both Unit / Integration tests. To prove this, if I run "gradle clean build integrationTest jacocoTestReport sonarRunner", I see the workspace for the job, now contains, ".sonar" folder and build/reports/sonar folder which contains another file called "overall-xxx.exec" some file, but that file size is not close to the "sum" of Unit test.exec and IT integrationTest.exec file size. Though its bigger than test.exec file size by few bytes.

3) What configuration can I set to have overall coverage for both Unit and IT tests i.e. overall...exec file gets good size (after running sonarRunner task). During sonarRunner task, I do see SonarRunner task's "jacocoSensor step" does see both UT and IT .exec files and the overall .exec file as well automatically (a good feature from Sonar).

Solution 1

Found answer to my 2nd question. High level info:

1. Gradle 1.6 jacocoTestReport uses different variables, Gradle >=1.7 uses different.

   For ex: we can tweak Unit tests and Integration Tests .exec file creation by changing "test" or "integrationTest" task by using the CORRECT variables -or it wont work n generate "test.exec" and "integrationTest.exec" default file names. See example below.

task integrationTest(type: Test) OR test { ... } section can have the correct variables for the given Gradle version that we are using.

task integrationTest (type: Test) { 
testClassesDir = sourceSets.integrationTest.output.classesDir

classpath = sourceSets.integrationTest.runtimeClasspath

   testReportDir = file("$buildDir/reports/tests/IT")
   testResultsDir = file("$buildDir/test-results/IT")



  ignoreFailures = true
   jacoco {

 //This works with 1.6
 destPath = file("$buildDir/jacoco/IT/jacocoIT.exec")
 classDumpPath = file("$buildDir/jacoco/IT/classpathdumps")

 /*
 Following works only with versions >= 1.7 version of Gradle
 destinationFile = file("$buildDir/jacoco/IT/jacocoIT.exec")
 classDumpFile = file("$buildDir/jacoco/IT/classpathdumps")
 */
   }
}

Similarly, for test { .... } task, you can define it as ../../UT/jacocoUT.exec and ../../UT/classpathdumps...

1. .sonar folder gets created if I run "sonar-runner" Linux/Unix sonar-runner executable in my project's workspace, BUT if I run Jenkins job which calls Gradle "clean build integrationTest jacocoTestReport sonarRunner", then build/sonar folder is created and becomes the WORKING DIR for SONAR (this shows up during the output).

2. In Jenkins > under Post build section, I mentioned the following and NOW, jacoco code coverage report on Jenkins Dashboard for both jacoco files (.html and .xml) - includes Unit and Integration tests code coverage data.

Record Jacoco coverage report section in Jenkins has the following boxes:

I mentioned:

Path to exec files: /build/jacoco/UT/jacocoUT.exec, */build/jacoco/IT/jacocoIT.exec Path to class dirs: */build/jacoco//classpathdumps/com/thc (this is the location where Jacoco instrumented classes sit).. both build/jacoco/UT/classpathdumps/com/thc and build/jacoco/IT/classpathdumps/com/thc will be picked as ** will be replaced for any folder under build/jacoco. this value can be set to "build/classes" folder as well.

Path to source files: ** (if I use src/java, few of the links for source file don't work i.e. file contents don't show up).. used ** .. it works now.

rest boxes - left blank.

1. Still wondering - why overall-jacoco.exec is not having the file size = sum of both jacocoUT.exec and jacocoIT.exec

At this point, I'm able to see Jacoco code coverage for both Unit and Integration Tests i.e. via visiting the Jacoco code coverage image on job's dashboard and visiting source code links and also if you go and browse "build/reports/jacoco/html/index.html" or "build/jacocoHtml/index.html" file.

Still trying to find out - what needs to be done for SONAR to pick these 2 .exec files (I have valid values set for sonar.xxx variurs variables for sources, tests, binaries, ...reportsPath etc for UT / IT exec files... on SonarQube dashboard, Unit test coverage is showing up fine but Integration tests coverage is still 0.0%.

I'll paste the copy of both my common.gradle and project's build.gradle file soon .... to have a better look.

Solution 2

OK, found the solution to UT/IT folder issue and my question (1) in this post. It's actually the "cleanTest" task which is one of the default RULE in Gradle.

Running "gradle tasks -all" gives a big output of tasks that Gradle supports and this output at the end tells about such Rules where if we call clean, then it'll wipe those folders. As you see in my code above, integrationTest was dependent upon cleanTest, thus when I called "gradle clean build integrationTest" it ran units tests first (via build task as Unit tests runs by default with the build step in Gradle) and then integration tests via "integrationTest" task, therefore, while running integration tests, it called cleanTest task, which wiped out "UT" folder which i have mentioned in a common gradle script (/init.d/commmon-some-name.gradle file) like I mentioned IT folders for reports/results directories.

Removing cleanTest as a dependsOn from integrationTest task, resolved wiping out issue.

task integrationTest( type: Test, dependsOn: cleanTest ) {
//task integrationTest( type: Test ) {

Output of following command: showing only last few lines...

gradle tasks -all

integrationTest
    classes - Assembles binary 'main'.
    cleanTest
    compileIntegrationTestJava - Compiles source set 'integrationTest:java'.
    compileJava - Compiles source set 'main:java'.
    integrationTestClasses - Assembles binary 'integrationTest'.
    processIntegrationTestResources - Processes source set 'integrationTest:resources'.
    processResources - Processes source set 'main:resources'.
jarService
sonarRunner [test]

Rules
-----
Pattern: build<ConfigurationName>: Assembles the artifacts of a configuration.
Pattern: upload<ConfigurationName>: Assembles and uploads the artifacts belonging to a configuration.
Pattern: clean<TaskName>: Cleans the output files of a task.

BUILD SUCCESSFUL