How to [politely?] tell software vendor they don't know what they're talking about

Question

Not a technical question, but a valid one nonetheless. Scenario:

HP ProLiant DL380 Gen 8 with 2 x 8-core Xeon E5-2667 CPUs and 256GB RAM running ESXi 5.5. Eight VMs for a given vendor's system. Four VMs for test, four VMs for production. The four servers in each environment perform different functions, e.g.: web server, main app server, OLAP DB server and SQL DB server.

CPU shares configured to stop the test environment from impacting production. All storage on SAN.

We've had some queries regarding performance, and the vendor insists that we need to give the production system more memory and vCPUs. However, we can clearly see from vCenter that the existing allocations aren't being touched, e.g.: a monthly view of CPU utilization on the main application server hovers around 8%, with the odd spike up to 30%. The spikes tend to coincide with the backup software kicking in.

Similar story on RAM - the highest utilization figure across the servers is ~35%.

So, we've been doing some digging, using Process Monitor (Microsoft SysInternals) and Wireshark, and our recommendation to the vendor is that they do some TNS tuning in the first instance. However, this is besides the point.

My question is: how do we get them to acknowledge that the VMware statistics that we've sent them are evidence enough that more RAM/vCPU won't help?

--- UPDATE 12/07/2014 ---

Interesting week. Our IT management have said that we should make the change to the VM allocations, and we're now waiting for some downtime from the business users. Strangely, the business users are the ones saying that certain aspects of the app are running slowly (compared to what, I don't know), but they're going to "let us know" when we can take the system down (grumble, grumble!).

As an aside, the "slow" aspect of the system is apparently not the HTTP(S) element, i.e.: the "thin app" used by most of the users. It sounds like it's the "fat client" installs, used by the main finance bods, that is apparently "slow". This means that we're now considering the client and the client-server interaction in our investigations.

As the initial purpose of the question was to seek assistance as to whether to go down the "poke it" route, or just make the change, and we're now making the change, I'll close it using longneck's answer.

Thank you all for your input; as usual, serverfault has been more than just a forum - it's kind of like a psychologist's couch as well :-)

Solution 1

I suggest that you make the adjustments they have requested. Then benchmark the performance to show them that it made no difference. You could even go so far to benchmark it with LESS memory and vCPU to make your point.

Also, "We're paying you to support the software with actual solutions, not guesswork."

Solution 2

Providing you are confident you are within the given system specs they document.

Then any claim they are making in regards to requiring more RAM or CPU they should be able to back up. As the experts in their system I hold people to account on this.

Ask them specifics.

• What information provided on the system indicates more RAM is needed and how did you interpret this?

• What information provided on the system indicates more CPU is needed and how did you interpret this?

• The data I have - at first glance - contradicts what you are telling me. Can you explain to me why I may be interpreting this incorrectly?

• I am interpreting this [obvious series of data] to mean [obvious interpretation]. Can you confirm I am interpreting it correctly with regards to my problem?

Having dealt with support in the past I have asked the same questions. Sometimes I was right and they were not focusing their attention on my problem properly. Other times however, I was wrong and I was interpreting the data incorrectly, or failing to include other data which was important in my analysis.

In any case, both of these situations were a net benefit to me, either I learnt something new I did not know before - or I have got their support teams to think harder about my problem to get a a decent root cause.

If the support team are unable to provide you with a logical expansion of their argument to a basis you can be satisfied with (you need to have an open mind to compromise yourself, be reasonable to accept your interpretation of the data is wrong) then it should become very present in their response. Even in the worst case scenario you can use this as a basis for escalating the problem.

Solution 3

For this specific situation (where you have VMware and application developers or a third party who does not understand resource allocation), I use a week's worth of metrics obtained from vCenter Operations Manager (vCops - download a demo if needed) to pinpoint the real constraints, bottlenecks and sizing requirements of the application's VM(s).

Sometimes, I've been able to satisfy the more stubborn consumers by modifying VM reservations or changing priorities to handle contention scenarios; "If RAM|CPU are tight, YOUR VM will take precedence!". Bad-bad things have happened when I've allowed software vendors to dictate their requirements on my vSphere clusters without real analysis.

But in general, numbers and data should win-out.

---

An example of something I used to justify VM sizing to the developer of a Tomcat application:

Dev: The VM needs MOAR cpu!

Me: Well, memory is your biggest constraint, and here's a heat map of your performance versus time... Wednesdays at 6pm are the most stressful periods, so we can spec around that peak period. Oh, and here's a sizing recommendation based on the past 6 weeks of production metrics...

Solution 4

The big thing is to be able to prove that you are using best practices for your system allocation, notably RAM and CPU reservations for your SQL server.

All this being said the easiest thing is to make the adjustments requested, at least temporarily. If nothing else it tends to get vendors over feet dragging. I can't count the number of times I've needed to do something crazy like this to satisfy a tech on the other end of the line that it really is their software not behaving.

Solution 5

I used to work in support - and part of what you're asking sounds highly rational (and probably is): but there are a few questions to ask yourself prior to just doing the "performance enhancement" they're requesting

• are you running at least at the vendor's stated minimum system requirements already?
• if you're at least at minimum sysreqs, are you already at their "recommended" system settings?

Vendors will 99 times out of 100 (in my experience - both on the support side and the customer/field side) not even deal with performance-related issues until/unless the systems match what their documentation calls for. Maybe it's a system that runs fine 99.5% of the time with 1 CPU and 512M RAM - but if the system requirements say 4 CPUs and 4G RAM and you've only got 2 CPUs and 1G RAM, they're well within their rights to demand more resources be assigned^*.

It is probable that they're asking you to increase system resources because of something they found in the lab/development wherein an issue magically disappears if you cross a specific threshold; if this is the case, yes it's an example of potentially-poor debugging on their end, but keep in mind they don't have time to eliminate every possible bug/issue that arises - some just need to be worked-around, and if that is the case here, just go with it.

There's also a not-insignificant chance that the issues you're seeing aren't even part of "their" software, but a component they rely on from some other source (vendor, OSS library, etc). I ran into this exact situation related to swap size, BEA WebLogic, and the Sun JRE at a customer a few years ago.

tl;dr:

In short, work with their support team, escalating as needed, until you find a resolution - but don't be surprised when some of the suggestions/debugging steps/fixes sound off-the-wall or pointless.

---

^{*_{If it truly doesn't "need" those extra resources, you're likely in a place to be able to file a doc bug / RFE for future versions - but don't push that route until you've demonstrated it's not the issue at hand}
^_{an eBook I wrote you may find helpful on the topic: Debugging and Supporting Software Systems}}

View more solutions

Comments

[Christopher Karel]

LART / Clue-by-four? (catb.org/jargon/html/L/LART.html) (catb.org/jargon/html/C/clue-by-four.html)

[Simon Catlin]

Clue by four - PMSL :-)

[Rohini]

Typo in your question I think. Should "give the production system for memory and vCPU" be "give the production system more memory and vCPU". I got ver puzzled the first time through and it was only the penultimate sentence that actually clarified what the problem was. :)

[Sobrique]

This remains my preferred LART: laughingsquid.com/cat-5-o-nine-tails-ethernet-cable-whip It's for network diagnostics. Honest.

[Ashigore]

Out of interest have you checked storage performance? Asking for more CPU/RAM might just be a layman response to poor performance which could easily be cause by high disk queue depth. Seems like a lot of folks forgot about SQL storage best practices when virtualisation came in.

[Sobrique]

grumble. That's right, blame storage! But more seriously - it's a good point. If there's a problem and RAM/CPU isn't helping, then it might be IO. Especially if we're talking VMWare, because it's not uncommon for ... well, the storage performance side of a system to be almost entirely ignored - whilst forgetting that you intrinsically get a massive bottleneck if you feed a lot of VMs on a limited number of HBAs.

[mustaccio]

May be the vendor has a point? Do you overcommit pCPUs, by chance? See this: zdnet.com/…

[Christopher Wirt]

Is HP your vendor in this case? Because I work there. I can confirm we don't care.

[user2338816]

As noted in comment from @Benubird below, can you clarify "vendor"? Not so much the identity, but at least the category. They supplied the server? An application that you run on the server? Specific hardware components? That is, are you being asked to spend more money to get more "stuff" from the same vendor? Or are they pushing blame away from their product onto your hardware? Need to know the specific area this vendor should have appropriate competence, especially the individuals you're dealing with in the Support group.

[jqa]

Inside a VM the usual tools (taskmanager, top etc) can act weird. Sometimes IMHO they appear to report on the resources the VM is currently given by the host, not necessarily what has been granted by the host. That aside, look closely at the peaks when they happen (a monthly CPU view is only a meaningless average), check IO and SAN. Check they configured their database and application properly to use more cores and memory? Performance problems are difficult to trace, I always start with the network cables and work up from there. A half duplex network connection or faulty cable can do it.

[Simon Catlin]

Thanks for comments folks. I will try to answer them in order. a) Have been looking at storage this week following above comments, and it appears to be OK (occasional disk queue blip, but generally healthy). b) We do have a 2:1 ratio of vCPU vs CPU, but, if the dev system isn't being used, it's 1:1. Hopefully, the CPU shares would limit the impact of any clash anyway. c) HP not involved, the vendor is a "systems integrator" of a large Oracle based system. When I say large, I mean an unnecessarily complex mix of Oracle middleware and "financial" products.

[user2338816]

"systems integrator" Ah. Not specialists, but generalists. (Specialist: Knows more and more about less and less until eventually knowing everything about nothing. Generalist: Knows less and less about more and more until eventually knowing nothing about everything.) Probably a maze working through Support before getting to an appropriate person (if exists). Still, are vendor recommendations to purchase more (memory, etc.) from them? And did they provide (and recommend) the current configuration? I.e., are they now saying they were not accurate (wrong) earlier?

[Lightness Races in Orbit]

ServerFault is not a forum!

[Simon Catlin]

...wise words. I reckon this might be the way forward, as much as it pains us to make the change. The good (?) thing is that the changes will require a reboot, and we can be clear to our business users that this is due to the vendor's request... which will almost certainly prove to be pointless. Sounds like I'm getting petty, but we're growing tired of the vendor's apparent lack of proper troubleshooting.

[Sobrique]

It's not unusual for vendors to play this sort of stunt. I think it's partly down to service level metrics - fob off, ask for more information and suggest a (pointless) workaround, because at least some of the time, the problem goes away/gets fixed in the meantime. If you've 'pull' with the vendor, having a chat with the account manager might do the trick. But don't hold your breath.

[Sobrique]

I'd have thought it'd give ammunition in a counter argument - you asked us to do this nonsensical thing last time; we did as a gesture of goodwill. This time we want some more detail as to your reasoning why this will make any difference.

[Sobrique]

Anything performance related takes a lot of time and resources to troubleshoot and diagnose. After all, there's nothing that's broken so you have to trace through painfully.

[warren]

@Sobrique absolutely - and they're usually in pretty remotely-related (even apparently unrelated) segments of the product at hand

[jojojoj]

Had a similar situation once with a SQL server for SCCM (system center config mgr) 4 CPU 30% util avg. Console terribly slow. Bumped to 8 CPU still 30% util, console finally responds in a normal manner.

[the-wabbit]

I should add that analysis based on averages might lead to wrong results. There are conditions where peak performance is important but you don't see the peaks in load statistics when they are significantly shorter than your collection / averaging interval. So you might have a nice colorful "your overall utilization is <60%" stats graph but see severe performance degradation in 1-minute peaks occurring 8 times an hour at the same time.

[Benubird]

@Sobrique That makes sense, and it might work out that way - I don't know enough psychology to say one way or another. My instinct though, is that if you've done something now just because they said to - effectively admitting they know more than you - they'll expect the same in the future. Either way, if you're having to argue with them (ammunition or not) you're already wasting time that could be spent solving the problem.

[Benubird]

Maybe I've completely misread the question, but isn't this the opposite of what the OP asked? I thought they were the dev, who knew they didn't need more cpu, which the vendor was trying to sell them - it sounds like you are describe the inverse, where a dev is asking for more cpu that they don't need.

[Sobrique]

"We did it your way last time. You were wrong. Are you prepared to accept that you might be wrong again? We do have precedent here."

[warren]

@Benubird - sadly some of these things come down to gut instinct or "it fixed it somewhere else ..." :(

[Benubird]

"it fixed it somewhere else" is a terrible reason to do something. True, sometimes there isn't time to properly debug a problem, and you have to go by gut instinct, but the thought of it still makes me shudder. I've seen plenty of bugs that "appeared" to be fixed by doing X, only to discover later that the problem was actually in something seemingly totally unrelated, which went onto cause more problems elsewhere until we figured it out.

[warren]

@Benubird - no argument ... just saying that's what happens :)

[Cosmic Ossifrage]

+1 for the recognition that human error can go two ways (and making support squirm a little when they have indeed tried to "fob off").

[Philip]

The whole first half of your suggestion seems to have been done already. The whole second half is exactly what the OP is asking.

[Paul Smith]

I would disagree. There is no evidence presented of problem analysis and the cpu and mem figures quoted are monthly aggregations that have no apparent relevance to the issue at hand.

[Floris]

Excellent suggestion. There's nothing quite like data to shut people up. "We will make the change you suggest. If it doesn't give the projected improvement, you eat the cost." Not sure how many systems are impacted here but your time proving them wrong QUICKLY becomes more expensive than plugging in some extra RAM.

[Simon Catlin]

Thanks for this. We don't have vCops, but I reckon our vSphere "estate" is now mature enough to require this level of detail. I'll add this to our Capex wish list for next year.

[ewwhite]

@SimonCatlin you don't need to buy it. You can download the demo for free and use it for 60 days. It's perfect for this type of situation.