Million years

I once calculated how long it would take to manually QA one feature in Ambiki. The answer was 114 million years.

Last year we moved our plan of care onto Ambiki's custom form system. Before we released it, I sat down and tried to crudely and roughly calculate how many different scenarios a QA person could theoretically need to test.

The table below was my attempt to be extremely conservative. I took eight dimensions that could affect the behavior of a plan of care and gave each an absurdly conservative number of test variants.

A table with eight rows and two columns, Actual and Conservative. Custom form disciplines: 2^5 = 32 actual, 5 conservative. Custom form components: infinite actual, 20 conservative. Organizations / EMR settings: 6,000+ actual, 200 conservative. Therapist disciplines / credentials: 262,112 actual, 20 conservative. Patients: 70,000+ actual, 500 conservative. POC / Previous POC situation: 100+ actual, 20 conservative. Goal (ST, LT, Goal Status, etc.): 1,000+ actual, 50 conservative. Permissions: 2^242 = 7 x 10^72 actual, 10 conservative. Total: infinite actual, 5 x 20 x 200 x 20 x 10 x 500 x 20 x 50 = 2 trillion conservative.
Eight dimensions, absurdly conservative numbers, 2 trillion scenarios.

Setting up just one of those scenarios manually can easily take 30 minutes. You need the right patient. The right insurance. The organization configured a certain way. A therapist with the right disciplines, credentials, and permissions. A previous plan of care in the right state. Goals with the right statuses.

Then you can finally click the button you're trying to test. 30 minutes x 2 trillion scenarios = 60 trillion minutes. That's about 114 million years of manual QA. Even if a human could test one combination every second, it would take about 63,000 years.

The funny part is that I forgot permissions the first time I made this table. Which is also kind of the point. The variables you forget are usually the ones that bite you.

So how do you QA software when manual testing mathematically cannot cover it?

A few things we do (but I would love to hear what others do on complex systems):

  1. Prioritize using real usage data. A workflow that runs 500 times a day deserves more attention than one nobody has used in a year.
  2. Test real edge cases, not just ones developers imagine. We built a small open-source Ruby gem called real_data_tests that turns anonymized slices of actual production scenarios into automated fixtures.
  3. Get the data model right. Good structures make invalid states difficult to represent in the first place. Then automated tests can cover combinations in seconds that would take a human hours to set up.
  4. Roll out gradually. Put the feature behind a flag. Start with a handful of pilot practices. Let it encounter hundreds of scenarios before exposing it to more.
  5. Turn every production surprise into a permanent test. If something slips through, the fix includes a test reproducing exactly what happened. That edge case is now covered forever. The gem in step 2 makes this easy to do.
  6. Watch closely after release and move fast. We cannot test everything. Nobody can. The goal is to test the right things well and respond quickly to what you missed.