Comparison between TestFlight Live, QuincyKit and Crashlytics

Question

I am going to launch my app on the AppStore and I would like to keep track of crashes and fix them as soon as possible. If possible, it would be nice to collect also some additional information about user activity and other useful stuff. In order to do so, I have looked for some crash reporting tools and the most interesting ones I have found are: TestFlight Live, QuincyKit and Crashlytics.

Among those three, QuincyKit should be the lightest one but the other two seem to be quite interesting since they provide more sophisticated reports and other interesting stuff.

My goal is to have as many information as I can on any issue the user can experience, but at the same time I don't want to make the app slower or consume more resources.

1. In your opinion and from your personal experience, which of these tools is the best one (taking into account my goal and my needs)?
2. By using TestFlight Live or Crashlytics I would make my app too slow?
3. Is there a risk to overload the device?
4. Reports provided by QuincyKit are precise enough? How many information can I retrieve from them?

Thanks!

Here is what I decided:

I am using Crashlytics for crash reporting (yes, it seems to be really great) and TestFlight for tracking user activity (checkpoints are really useful to find out what users generally do and figure out what the tendency is). I followed the instructions written here

Solution 1

I honestly think Crashlytics is a better solution than Testflight for crash reporting.

Here's what you get with Crashlytics that you don't get with the others.

• Duplicate culling (TF does this too, but its not too great at it, Crashlytics is damned near perfect)
• You can actually mark crashes as closed/resolved, and get them out of your list for a given version.
• Crashlytics does everything TF's Crash reporting does, but better and then some (logging, stack traces, etc.)
• Percentage of affected users, and the numbers that go with that. (ie: should i fix the bug that happened to one guy, or the one that's happening to 10k?) Testflight doesn't tell you this.
• Prioritization based on occurrence. This is probably the most important gain in my opinion.

These are just a few, but I figure they're probably the most important ones for you.

We used Testflight's crash reporting for close to 2 years on an extremely popular app (several million D/Ls). Its definitely better than nothing, and very convenient if you're using TF for distribution as well, however you get many more benefits from Crashlytics. We switched to Crashlytics this summer and now we're actually able to manage crashes and make smart decisions about what to fix and when, instead of just sifting through a giant never ending list.

I see you already accepted an answer, but I'd seriously give it another look even if you opt to continue with Testflight. I've found its hard to really grasp what you're missing until your app has shipped, at which point is even harder to change.

Solution 2

Crashlytics is second to none for crash reporting.

We were in the same boat as you trying to find the best crash reporting solution. After some thorough investigation and test runs of TestFlight, HockeyApp, and Crashlytics, we originally chose HockeyApp because they allowed us beta distribution along with crash reporting on both iOS and Android (we wanted both in one solution for both platforms). However, HockeyApp's exception backtracing was just not giving us any additional crash details. This is where Crashlytics shines. Their exception backtracing is amazing. Period.

So here's my summary of all 3 SDKs:

Crashlytics

• #1 crash reporting
• #1 exception backtracing, bar none (provides very useful extra crash details)
• Extremely fast and lightweight
• Custom key logging for additional crash context
• Best duplicate crash recognition and culling
• Automatic SDK updates (Their Mac app automatically updates the Crashlytics iOS SDK in your project)
• No beta distribution (I'd love a one stop solution for crash reporting and beta distribution)
• Automatic build server support

TestFlight

• Somewhat heavy, and adds bloat to your app package
• Great beta distribution
• No Android support (at least when we tested back 6+ months ago)

HockeyApp (HockeyKit - Beta distribution, QuincyKit - Crash reporting)

• Lightweight
• Crash reporting UI a bit confusing
• Exception backtracing severely limited (at least when we tested in March of 2011)
• Very good beta distribution

All that said, we chose Crashlytics for crash reporting, and HockeyApp for beta distribution. But you must chose what works best for your needs.

Solution 3

Definitely recommend Crashlytics as well.

TestFlight Live has given me issues in the past. It seems every time I go to use TestFlight, it is down anyway.

Crashlytics is awesome. Here's why:

• Adding it to your project couldn't be easier. There is a Mac app that does most of the hard work for you.
• Automatically updates itself
• Prioritizes crashes for you
• Provides handy stats like OS and device percentages as well as average memory available, etc

I use Crashlytics in all of my applications. I added it to Hipstamatic when I was there and the data we got was shocking. It really helped improve our product. I also tried TestFlight Live and quickly removed it after the first beta since it was causing crashes.

Crashlytics is awesome. You should use it.

Solution 4

If we're only talking about crash reporting, Crashlytics is far better than TestFlight. (Never tried QuincyKit, so I can't compare the 3 options)

We've been using Crashlytics for over a year on Weddar and it has been great. Having tried other solutions before I have to say that before installing it I was kind of suspicious of the great features they were stating, but the installation was indeed done in about 5 minutes and it only added around 40-45Kb to the app.

The crash reports are incredibly detailed making it really fast to pinpoint solutions for the bugs and the updates to the sdk are pretty steady and stable. The team is incredibly supportive too. I remember that we had a problem with the new ARM7s when iPhone5 came out and they solved it in about 30 minutes.

I use TestFlight for user beta testing management so I've tried TestFlight Live SDK in the summer just to see if it was a solution to have all integrated in one service, but we had a really bad experience with it. I had 2 updates rejected in the App Store Approval for the first time (Weddar was launched in April 2011) and we lost about a month trying to catch the bug. When LIVE beta testing, no user would complain about any problem, we "solved" it by removing the TF SDK. Never quite understood what was the problem. We contacted TestFlight team and never had contact back. (Another big big detail is that TF SDK added about 800Kb to our app.)

So, even if I still use TestFLight for beta testing, if you're looking for a great and lightweight crash reporting SDK, I definitely say that you should use Crashlytics.

Hope this helps.

Solution 5

I'd say go with TestFlight (Live)

In my experience the TestFlight SDK won't crash/slow-down your device and has very versatile crash reporting - allowing you to debug reported errors fairly accurately.

TestFlight also doubles-up as a feedback package for when you're testing in-development.

It's also a pretty light SDK.

To be more specific (in answering your list of questions):

1. TestFlight allows you to scalp for user 'checkpoints' and has its own version of NSLog that allows you to dynamically log events at runtime.
2. Your app wouldn't slow down as network requests are handled off the main thread.
3. I don't understand why a device would get overloaded by using either of the SDKs you've mentioned.
4. QuincyKit reports seem reasonably precise, however you need to make up your own mind about the precision you need - you can find QuincyKit docs here.

View more solutions