Sashi Ravi is an English teacher in Calicut, India. She walks 30 minutes every day to her school of 2,500 children, where she teaches kids ages 5 to 13.
But at night, Ravi logs onto her computer and joins thousands of other quality assurance testers, clicking through websites and combing for visible errors in the code.
Ravi has bosses at her school like any other teacher, but the boss at her second job isn’t a human—it’s the conglomerate of algorithms that hired her, trained her, doles out her work, monitors her productivity, pays her, and would send her back to training if her “laziness score” dipped below an acceptable level.
Rainforest QA, a quality assurance service that tests websites and apps for bugs, has a workforce of about 60,000 contracted employees that are managed exclusively through an interconnected series of algorithms. One percent of Rainforest testers work full-time, using it as their main source of income, and 10% work 20 hours per week. The other 89% take jobs more sporadically. Boston-based Kristen Hohenstein, the company’s tester ambassador, checks in with the most active workers to make sure they’re happy. There are a little more than 100 full-time employees in all, split across North America, South America, Africa, Asia, and Europe.
The algorithms are the cornerstone of the five-year-old San Francisco-based company’s ability to manage the horde of workers necessary to scour every nook and cranny of a website for potential bugs. The service works a lot like Uber’s automated system for matching drivers with passengers, except Rainforest has far more control and insight into the work the testers are doing, and ranks them accordingly. It’s as if Uber had cameras constantly judging how well every driver was performing, and those who took longer routes or texted while driving got less work.
The future of automated management is quickly becoming a reality. Large companies that employ hundreds in middle management are turning to automated alternatives, which could manage projects and deadlines with less overhead and increased productivity. It’s not just tech companies like Amazon— even hedge funds like Bridgewater Associates are working toward this goal. And without your boss being a real person, suddenly nobody knows (or cares) if you’re working from home.
Rainforest’s multi-part algorithm is designed to guide workers through their entire tenure at the company, from training to testing. But before it can manage, it has to recruit. When the algorithm detects more testers are needed for a job, based on metrics like overall productivity, it supplements the registered testers with temporary workers from automated online crowdsourcing companies CrowdFlower and Amazon Mechanical Turk. If it needs more workers overall, Rainforest staff are pinged, and they post job openings manually—though this only happens a few times per year. While anyone on the internet can apply, Rainforest’s highest densities of workers are located in Venezuela, Serbia, Russia, and India. The testers that spoke to Quartz mentioned that working online for an American company was particularly attractive because of the dollar’s strong conversion rate back to their own currency.
Once it has new candidates, a training program weeds out applicants who don’t meet Rainforest’s standards. The training consists of tests on best practices and sample jobs. After training—which lasts “until they graduate or fail,” says co-founder and CTO Russell Smith—testers begin to take on real work. They’re paid four cents per step it takes to complete the job, which varies depending on the type of task. Jobs can range up to more than 80 steps, and last anywhere from 5 minutes to an hour if not completed correctly. Multiple testers are used for each job, so the tests are redundant, and Rainforest claims that its average time to test a site is 33 minutes.
Quality assurance testing has typically been a full-time job. The average salary is around $53,000 per year, but can range past $93,000 according to Glassdoor, a job site that tracks data about careers and businesses.
Pricing on testing tasks varies based on how complex the site or software is and how often the testing needs to be done, but Rainforest’s online rate calculator estimates its service is one tenth the cost of hiring manual testers from another QA company, and half the cost of hiring a completely automated company to do the work.
These tasks might be clicking on a link to make sure it works, filling out a form to see if it submits responses to the right place, or testing that videos load correctly on multiple browsers.
Every action testers make is carefully recorded; they work in a virtual machine on their personal computer, so Rainforest can track keystrokes and mouse movement. Time spent per job, jobs per day, number of times a job needs to be tried before it’s correctly completed, and extraneous clicks and typing are all measured. A “laziness score” is invisibly calculated for each tester, raised by wasting time during an active job, and directly affects how much work a tester receives.
What happens if laziness reaches a critical level? Back to training, square one.
“We like to rescue [testers] if we can,” Smith says.
Despite the algorithm’s cold calculations, three of the platform’s most-active testers who spoke to Quartz (an admittedly small sample of the total 60,000) said they didn’t mind working for a machine—in fact, they each preferred it to a standard working environment. Ravi, the teacher from Calicut, wants to quit her job to work for Rainforest full-time, so she can care for her family and still bring home a paycheck.
Nikolai Berchev, a tester who works from Bulgaria, doesn’t see the algorithm as a boss. He gives an example of a job he completed three weeks prior. He tried to submit the finished work, but the algorithm wouldn’t accept it. He tried it a different way, 10 or more times, until the algorithm finally accepted the job as correct.
“I was wrong and the algorithm was right, or I was 90% right and the algorithm was 100% right,” Berchev said. “So I look at it as a teacher. And a patient one.”
Before joining Rainforest in 2014, he had worked in offices but struggled with interpersonal communication. Berchev also wanted to write a book, but says previous managers looked down on his writing during work hours. “But now when I’m waiting for a hit or a test to come, I can basically do whatever I want,” he said.
Berchev compares Rainforest’s algorithm to self-driving trucks, in the way he thinks this automation will change the way people work.
“This is the future of all the jobs,” Berchev (left) says. “There would be whole families working from home. I think it’s really where the future lies, the electronic house.”
Beyond gauging productivity, Rainforest is looking for more ways to use the massive swaths of data collected from its testers. Chiefly, the company is looking to further automate the process, using machine learning to train other algorithms to do the testing work.
This is a big ask. Humans typically find it easy to understand why an error has occurred, like a form not showing up on a webpage or a link being broken because there was no “.com” in the URL. An automated tester might be able to find problems, but the bigger challenge—and the one that still stands—is making the algorithm understand and be able to explain why one of the hundreds of potential errors on a website happened.
Further, while the algorithms have the benefit of learning from every human that has ever done a similar task, every job is different, and two testers that do the same job could produce different results. That makes the data difficult to standardize and learn from, says Rainforest data engineer Maciej Gryka.
“Given the same website and same scenario, you can get different results,” Gryka said. “We don’t have gold standards, because there is no gold standard.”
Rainforest’s challenge lies in creating algorithms that can learn how to navigate the myriad combinations of apps that humans have become accustomed to—from minimalist websites with hidden features to button-packed mobile apps—so engineers don’t have to recalibrate for each job.
Gryka says the overall goal is “generalizing human decision,” meaning being able to anticipate and test for every way a user would interact with a website or app.
Umesh Harugade, a tester with two degrees in computer science and a day job as a system administrator at a medical facility, says he’s not worried about automation taking his online work anytime soon.
To him, the tasks, which can include intuitively knowing how a human would use a new kind of application or website, are too complex for a machine.
“You’re always going to need a human in there,” Harugade said.
Gryka agrees. Despite his work to automate the system, the amount of data required to properly train each new algorithm and the intricacy of jobs mean not everything can be automated.
“We’ll probably never get rid of humans,” Gryka says.