If you’re not load testing applications, you’re relying on end-users or customers to do the testing for you. And that — as Healthcare.gov very publicly demonstrated — can have disastrous results.
Whereas once IT could perhaps get away with top 20 testing, that approach is unworkable in today’s complex computing world. Users — whether internal or end customers — are using myriad devices, different operating systems, and hundreds of applications. Consumers may well choose to buy from your well-integrated competitor. Even employees may switch to unapproved, cloud-based applications instead of official corporate software or services if authorized tools are too unwieldy, complex, or don’t do the job well.
Education company Corinthian Colleges Inc. or CCI, best known for Everest University and trade school WyoTech, runs CampusView to support approximately 100,000 students in 130 campuses and 80,000 online students, lead quality assurance engineer Rey Biedgoly told me in an interview. In addition to student, faculty, and administrative portals, the software also notifies students about the status of their funding, he said. For example, if a student’s attendance record could jeopardize a government grant, the software must generate several notices to inform students so they can rectify the situation and hang onto their grants.
(Source: CCI)
Because CCI is for-profit and operates a trade school, it is subject to some more stringent rules than typical non-profit educational institutions, he said. And these rules can change frequently, meaning quarterly software updates can morph into three or update every three months. Although Campus Management Corp. develops CampusView software, CCI does its own testing to ensure compatibility across its vast user ecosystem, Biedgoly said.
They do their own testing but we do the ultimate UAT [user acceptance] testing and we do the testing because it affects our system. If there’s a bug and we didn’t know about the bug, it could affect us adversely. We go through a complete QA process here, as far as regression testing and load testing and enhancement testing, manually and automated, for every single release. It’s because we have seen issues that came up in the past with the releases we got from CMC and it affected us negatively. We’re the customer and we shouldn’t’ be finding anything. Sometimes we find enhancements that we kick back and say, ‘It doesn’t work.’ Sometimes we find fixes that don’t work. The burden’s on us because we are the ultimate customer with the Department of Education so we want to know whatever we’re doing, how it is affecting us. If there is an issue we find, we want to find a workaround for it, so when we do put it into production we know how to avoid it, how to work around it, or how to correct it. That’s why we do such thorough testing for every release we do. If we run into production and run into surprises, it’ll affect us negatively from a cash flow perspective.
When Biedgoly joined CCI about six years ago, the company performed top 20 testing: It checked the top 20 business functions of the new release. Today, however, the QA team performs about 340 tests for each release, plus manual enhancement testing, he said. A group of 40 to 60 people run these tests, using tools such as Automai’s AppLoader, added Biedgoly. This technology has streamlined the process and improved testers’ productivity, he said.
The way they were doing it here was calling all users in a conference call and being on the line: Everybody ready, everybody click on the link at the same time. I was like, all right, this is a primitive way of load testing. They were taking measurements and benchmarks like that. They could only do 20 or 30 users they had access to. With AppLoader, we can do up to 2,000 users: We have a license for up to 2,000 users so we can do a load test for up to that amount. The tool is fairly simple to use actually and fairly intuitive.
As a direct result of this load testing, CCI has more satisfied end-users, fewer IT support calls, and improved cash flow, putting Biedgoly and his team at the top of CCI’s class.