Benchmarking Performance on Java VM vs .NET CLR

Question

Have you ever had to justify the choice over using .NET instead of Java based on performance?

For a typical high volume transaction processing system that can perform the following operations,

• Concurrent Database transactions
• Mathematical computations
• Interaction with other web services (SOAP/XML, XML-RPC)

My approach would be to code benchmark tests in both Java for the JVM and C# for .NET CLR that benchmark the above operations under various levels of load and compare the results.

Language and platform preferences aside, I am interested in hearing how you would go about doing a conclusive performance comparison between the Java VM and .NET CLR?

Are there any comprehensive and respected benchmarks that exist?

Solution 1

I don't have exact numbers on the efficiency of the JVM vs the CLR, but the difference, if any, is likely to be small.

However, on the language side, C# does have some more low level constructs than Java, which would allow for more optimization.

Constructs such as:

• User defined value types. Fast to allocate, no memory overhead (which is 12 bytes per reference type in both the CLR and JVM if I remember correctly). Useful for things that let themselves naturally be expressed as values, like vectors and matrices. So mathematical operations. Combine this with ref and out to avoid excessive copying of these large value types.

• Unsafe blocks of code that allow a little more 'close to the metal' optimization. For example, while the CLR and JVM can avoid array bounds checks in some situations, in a lot of cases, they can't and every array access requires a check whether or not the index is still within bounds of the array. Using unsafe code here allows you to access the memory of the array directly with pointers and circumvent any bounds checks. This can mean a significant saving. And on the very low level side, there's also stackalloc which allows you to allocate arrays directly on the stack, whereas a normal array is allocated on the heap, which is slower, but also more convenient. I personally don't know any practical applications of stackalloc.

• True generics, unlike the type erasing generics of Java, avoiding unneeded casting and boxing. But if this is a problem in your Java program, it can easily be solved with some extra work (switching from for example a ArrayList<Integer> to a custom type that internally uses an int[] buffer.)

This all seems biased towards C# and I do think C# has better low level language constructs available that can help with performance. However, I doubt these differences really matter (and they might not even apply in your case, using pointers gains you nothing if all you do is database access, where Java might be faster) if the choice impedes you in some other way (like going cross platform). Go for correctness, the platform that matches your requirements, rather than minor performance differences.

Solution 2

a conclusive performance comparison between the Java VM and .Net CLR is a pipe dream. You can always make a performance test that will make one look better than the other since engineering a complex system always involves compromises. If you narrow down the type of benchmarks to runtime speed and memory you might find articles like this: http://www.codeproject.com/KB/dotnet/RuntimePerformance.aspx that is by no means the end of the debate.

Solution 3

Language and platform preferences aside, I am interested in hearing how you would go about doing a conclusive performance comparison between the Java VM and .Net CLR?

I would write a suite of benchmarks designed to let you compare various different characteristics of the two VMs and their standard libraries. My expertise is in technical computing so I would recommend the following:

• In register integer arithmetic, e.g. Fibonacci.
• In register floating point arithmetic, e.g. mandelbrot.
• Array iteration, e.g. FFT.
• Allocation, e.g. purely functional red-black trees.
• Hash tables with int or float keys and values.
• Hash tables with string keys and values.
• Strings.
• Regular expressions.
• File IO.

Perhaps you can come up with something similar for databases and web services.

Don't forget that the languages sitting on top of the CLR have quite different properties. For example, inline in the F# language lets you automate optimizations that can give huge performance gains over C#. Conversely, goto in C# lets you do some things more efficiently than is possible in F# and the optimizations on structs were more effective in C# than F# (last I looked).

Are there any comprehensive and respected benchmarks that exist?

No but there are lots of scattered benchmarks that focus on the outliers because they are more interesting. For example, this blog post describes why a simple generic hash table benchmark can be 17× faster in F# on .NET than in Java on the JVM. In that case the reason was that value types and reified generics make it possible to write a much more efficient generic hash table implementation on .NET than on the JVM.

Solution 4

FYI you are specifically not allowed to use the .NET framework code for any form of benchmarking without contacting Microsoft first and getting their approval.

If you did want to publish something I thought I'd let you know.

From memory MS did some book store, pet store thing that Java did first to show off how their software could work more effectively at the same task. I'm trying to think of it.

.NET Pet Store

Solution 5

I have also wondered which would give better performance. But have no idea about how I might go about performing bench mark test for this.

So good question - hopefully we will all get some guidance here.

I would guess this benchmark testing would need to be a "discount" benchmark testing approach (easy to setup & run by 1 developer)?

I anyone has this kind of info it would be great. I'm often asked to evaluate technologies on my own within short time scales.

Nice one bunn_online!

View more solutions

Comments

[bunn_online]

In terms of complexity, what we are doing is pretty standard and well supported on both platforms. We want to determine though that we will go with the platform that will provide us the best performance and scalability.

[bunn_online]

Thanks that is interesting, probably why I am battling to find published results!

[bunn_online]

For those interested in the "pet store benchmark": theregister.co.uk/2001/11/12/pet_vs_pet_ms_opens

[Spence]

@MSalters I've yet to see a java app out perform .Net on windows where my customers are. Until Java has something even close to the expressiveness of linq and Linq to SQL I don't think they are sitting on high side of the fence in regards to readability.

[Spence]

To be clear to everyone, I greatly respect that java runs on phones, dvd players and any os you care to name, which is it's strength. It's strength is NOT performance in the average case due to the constraints on working away from x86 arch.

[bunn_online]

For the sake of this question, performance is the criteria I want to evaluate. I understand that there are many other criteria when chosing a language/platform (code complexity/readability, skills availability, etc,etc), the point of my question though is to determine how others would go about doing a performance evaluation between the two platforms.

[bunn_online]

@JulianR: Thanks for the objective view. I agree, C# has better low level constructs (amongst other improvements), but thats probably because it got an opportunity to learn from Java's quirks (mistakes?).

[Jé Queue]

@Spence, you sure about that? This isn't the 90s anymore. I've formally benchmarked Java against C++ against FORTRAN in a non-linear maths environment. Java and C++ were within 2% of each other in various tests. FORTRAN was consistently 20% faster. My point is Java is NOT slow.

[konrad.kruczynski]

"The only thing is it uses mono instead of visual studio but the performance difference between the two is now very small." - it depends. For example compare time of n-body test on Mono and .NET. It isn't small.

[Totti]

"the difference, if any, is likely to be small". Woah, the differences can be huge. For example, generic hash tables can be 17x faster on .NET than JVM. fsharpnews.blogspot.co.uk/2010/05/java-vs-f.html

[Totti]

"I think it is objectively superior". I agree that C# is generally superior but one place where Java has a big advantage is garbage collection. With value types and reified generics, .NET lets you evade GC entirely more often but when that is not possible the JVM tends to have the advantage from a significantly more highly optimized GC implementation. Indeed, JVMs are used to pioneer new GC algorithms...

[Totti]

FFI (e.g. PInvoke) is another place where the performance gap is apparently huge. I've heard that Java is 10x slower than C#/F# here.

[JulianR]

@JonHarrop - Well I did address that by mentioning Java's generics system. But while inconvenient, there's nothing stopping you from taking the source code of the Java generic hash map and search/replace 'T' with 'int' to avoid those boxing operations :) But yeah, a Dictionary<int, int> is going to be a lot faster than the generic Java equivalent.

[JulianR]

@JonHarrop - I haven't benchmarked the Java HotSPot VM, but I also have the impression it's more advanced than the CLR in general. But like I said, that's just my impression. No complaints on the CLR GC from me though, it hasn't really been a limitation for me. Just the Large Object Heap makes me think "eh... that's a little iffy" with regards to compacting and fragmentation.

[JulianR]

@JonHarrop - Isn't Mono's P/Invoke a lot faster still than .NET's?

[Totti]

@JulianR "there's nothing stopping you..." No. Consider Dictionary<int, float>. The JVM is fundamentally incapable of expressing the underlying array...

[Totti]

@JulianR "Isn't Mono's P/Invoke a lot faster still than .NET's". No idea but I can well believe Mono is faster here because .NET's FFI must work in harmony with an accurate garbage collector.

[Totti]

@JulianR I've had problems on real projects with .NET's GC leaking, causing massive (minutes) pauses and fragmenting until it runs out of memory. See here, for example: flyingfrogblog.blogspot.co.uk/2010/10/…

[Totti]

@JulianR Startup times might also vary significantly.

[Totti]

"The only thing is it uses mono instead of visual studio but the performance difference between the two is now very small". .NET was 50% faster than Mono on this number crunching benchmark. flyingfrogblog.blogspot.co.uk/2009/01/mono-22.html

[Totti]

-1 "You can prove almost every thing with test that suites your need". No, you cannot. I think you mean that you can try to deceive people but the OP is interested in collating useful information which is entirely feasible.

[Spence]

Yes I'm sure that the JAva vs .Net benchmark for a website based system was faster as the operating system HTTP.sys driver in IIS gives you excellent performance gains. I'm not making a blanket statement that C# rules the world, but in that specific benchmark, java got reamed.

[Leonardo Lobato]

Just to add to the topic: I found that article citeseerx.ist.psu.edu/viewdoc/… which give some others perspective to the discussion.