How to organize full build pipeline with Gulp, Maven and Jenkins, all the way to integration tests?

Question

I have a project that has:

• JS client with somewhat interesting build process. That includes compiling CSS, catenating and minifying JS and CSS, generating and processing HTML, and some other steps. The Node tools like Grunt or Gulp are great at this.
• Java server that is a WAR deployed on Tomcat. It includes these assets as well as all the Java code. It has all kinds of tests: Unit tests, integration tests that may instantiate a DAO and talk to DB, and end-to-end API tests that actually talk to the app running on Tomcat.
• End-to-end tests with Protractor. If you're not familiar, it's another Node tool that wraps Selenium.

How can I organize this whole process in a sane, robust, and automated way?

What I have at the moment is Gulp and Maven, with Maven basically owning the whole process.

1. It calls Gulp asset generation in generate-sources using antrun (doh, third build tool!).
2. It runs the regular Java build.
3. It starts Tomcat with my WAR in pre-integration-test.
4. It runs Java E2E tests talking to that tomcat with failsafe plugin.
5. It calls Gulp again with antrun, this time to run Protractor tests.
6. It shuts down Tomcat in post-integration-test.
7. It's supposed to verify test results in verify.

That kind of works, except for that Maven is generally very rigid and I feel I'm taking it too far. Using antrun to call Gulp is an ugly trick. It's very hard to control dependencies between these steps and monitor their outcomes. It's hard to control order of things in the same phase. Failsafe verify does not seem to process the external JUnit report files that Gulp generates. I could go on.

I wonder if I should do more in my build server (Jenkins), maybe using a build pipeline or parameterized triggers - but I've never done it and I'm not sure if that's really better.

So, how would you implement it?

Solution 1

In my experience, the frontend maven plugin is far and away the best plugin for this type of build/deploy process. https://github.com/eirslett/frontend-maven-plugin . This is how i use it for Grunt but it supports Gulp just as well.

<plugin>
    <groupId>com.github.eirslett</groupId>
    <artifactId>frontend-maven-plugin</artifactId>
    <version>...</version>

    <!-- optional -->
    <configuration>
        <workingDirectory>src/main/frontend</workingDirectory>
    </configuration>

   <execution>
    <id>grunt build</id>
    <goals>
        <goal>grunt</goal>
    </goals>

    <!-- optional: the default phase is "generate-resources" -->
    <phase>generate-resources</phase>

    <configuration>
        <!-- optional: if not specified, it will run Grunt's default
        task (and you can remove this whole <configuration> section.) -->
        <arguments>build</arguments>
    </configuration>
</execution>
</plugin>

One thing to be aware of it is will download node for the system it is being run on, so if you have a different OS on your build server, you'll need to make sure that is the version you have checked into version control, your local version (for me OSX) will have to be maintained local to your project.

Solution 2

This project is about 2 years old but it does a lot of what you are looking for.

https://github.com/pankajtandon/PointyPatient/blob/master/pointy-web/pom.xml

Do checkout the parent pom which basically runs all the sub-projects together right from the web to the domain to the repo and fails if any (Jasmine or SpringMVC or SpringServices tests) fail. And it build a war for the server side deployment too.

This was pre-protractor, so that would be a nice addition. Altho the frontend maven plugin looks like the tool for the job now.

HTH Pankaj

Solution 3

I'd try to build kind of poor-man's-pipeline.

1. Let grunt/gulp do its work first (process assets, run frontend tests etc - prepare artifacts to be included in WAR). Fail entire build when this step fails (assets generation or tests).

2. Run regular maven build producing WAR file with assets created in step 1. It will run own set of tests with just regular WAR file. Doesn't need to know about grunt/gulp things.

You'll then have two places where e.g. tests are run (frontend, run by grunt/gulp and backend by maven) but configuring correct reporters will let CI servers to detect all of them (we use TeamCity and it handles it fine).

Script it up a bit and it should be better than calling node via antrun multiple times. Alternatively you can run first step from within maven build, but it may be hard to control stuff.

Comments

[Konrad Garus]

What about end-to-end tests with Protractor? They need the app running on web server, then the build needs to shut down the web server and only verify tests late.

[Alireza Mirian]

That's a good maven plugin. This way you will generate at least some resources inside static directory during your gulp/grunt build process. So it's a good idea to add the static directory (or only generated ones inside it) into ignore list of your IDE, in order to stop it from indexing files inside it whenever you run your frontend build process. It will be more problematic if you setup gulp source watchers for example.