Boeing’s 737 Max Software Outsourcing to $9-an-Hour Engineers
Software outsourcing to inexperienced or underpaid engineers can cause a monetary as well as human loss. It is yet surprising about how Boeing 737 Max Crisis took place. A big Chicago based planemaker like Boeing is famous for its narrow body design. It is still unknown how such famous Jet company can make basic software mistakes. It lead to 2 deadly crashes killing 346 people in Lion Air and Ethiopian Airlines.
Senior engineers who are with Boeing for many years consider Software outsourcing maintenance to lower paid contractors as reason. The Max Software was developed at a time when Boeing laid off experienced developers and pushed suppliers to cut costs. Owing to this Boeing and subcontractors have relied more on temporary contractor developers at $9 an hour. The developers were expected to develop and test the software.
Software Outsourcing to College Graduates:
“In Boeing’s Seattle office, recent college graduates hired by Indian Software Company HCL Technologies Ltd. occupied many desk rows” – said Mark Rabin, an ex-Boeing employee who worked in flight test group supporting Max software. “All the HCL coders were designing as per Boeing specifications, but still it was a big risk and inefficient as compared to other experienced Boeing developers. It took long hours to go back and forth to rework on the code owing to the code not written properly”.
Boeing won many Indian orders with Indian military and commercial aircraft like SpiceJet Ltd. The reason being its collaboration with Indian Software companies. Till Jan 2017, Airbus ruled Indian skies and then Boeing changed the Indian aerospace scene.
Boeing Max’s Flight-Display Software outsourcing happened to HCL to develop and test it. Developers from another Indian company, Cyient Ltd., were outsourced software development for flight-test equipment.
Software Outsourcing for Costly Delay:
In Jan 2016, a HCL employee who was working on Max software wrote the following to summarize his work:
“Provided quick workaround to resolve production issue which resulted in not delaying flight test of 737-Max (delay in each flight test will cost very big amount for Boeing).” Though Boeing has said that it did not rely on engineers from HCL and Cyient for the MCAS(Maneuvering Characteristics Augmentation System), which has been linked to the Lion Air crash last October and the Ethiopian Airlines disaster in March.
The Chicago-based company also said it didn’t rely on either firm for another software issue disclosed after the crashes: a cockpit warning light that wasn’t working for most buyers. “Boeing has many decades of experience working with supplier/partners around the world. Our primary focus is on always ensuring that our products and services are safe, of the highest quality and comply with all applicable regulations.”
HCL also released its statement “It has a strong and long-standing business relationship with The Boeing Company, and we take pride in the work we do for all our customers. However, HCL does not comment on specific work we do for our customers. HCL is not associated with any ongoing issues with 737 Max.”
Downfall from cost cutting:
Recent simulator testing by the Federal Aviation Administration suggests that software issues on Boeing’s best-selling model run deeper. The company’s stock fell after the regulator found a further problem with a computer chip that experienced a lag in emergency response when it was overwhelmed with data.
Engineers who worked on the Max, which Boeing began developing eight years ago to match a rival Airbus plane, have complained of pressure from managers to limit changes that might need extra time or cost.
“Boeing was doing all kinds of things, everything you can imagine, to reduce cost, including moving work from Puget Sound, because we’d become very expensive here,” said Rick Ludtke, a former Boeing flight controls engineer laid off in 2017. “All that’s very understandable if you think of it from a business perspective. Slowly over time it appears that’s eroded the ability for Puget Sound designers to design.”
Rabin, the ex-Boeing software engineer, remembered one manager saying at a meeting that Boeing didn’t need senior engineers because its products were mature. “I was shocked that in a room full of a couple hundred mostly senior engineers we were being told that we weren’t needed,” said Rabin, who was laid off in 2015.
Sales from India
The typical Boeing plane has millions of parts and lines of code. Boeing has long turned over large portions of the work to contractors who follow its detailed design blueprints. Starting with the 787 Dreamliner, launched in 2004, it increased profits by instead providing high-level specifications. Then they asked contractors to design more parts on their own. The thinking was “they’re the experts, you see, and they will take care of all of this stuff for us,” said Frank McCormick. Frank was an ex-Boeing flight-controls software engineer who later worked as a consultant to regulators and manufacturers. “This was just nonsense.”
Sales are another reason to send the work overseas. In 2005, Boeing promised to invest $1.7 billion in Indian companies if they get 11$ billion order from Air India. That was a boon for HCL and other software developers from India, such as Cyient. Their engineers were widely hired in computer-services industries but not yet experienced in aerospace.
Rockwell Collins, which makes cockpit electronics, had been among the first aerospace companies to outsource significant work in India. In 2000 HCL began testing software for the Iowa-based company. By 2010, HCL employed 400+ people at design, development and verification centers for Rockwell Collins in Chennai and Bangalore.
That same year, Boeing opened a “center of excellence” with HCL in Chennai. They wanted to partner “to create software critical for flight test”. In 2011, Boeing included Cyient to a list of its “contractors of the year”. Cyient won it for design, stress analysis and software engineering on 787 and 747-8 at their Hyderabad center.
Mistakes of outsourcing to Russia:
The Boeing rival also relied somehow on offshore software outsourcing. In addition to supporting sales, the planemakers say global design teams add efficiency as they work 24/7. But outsourcing has long been a pain point for many Boeing engineers. They fear job losses and admit it has led to miscommunications and bugs.
Boeing built a design center in Moscow. At a meeting with chief Boeing 787 engineer Cynthia Cole in 2008, she complained about re-sending drawings to Russia team. She had to send drawings 18 times before they understood the importance of smoke detectors. Detectors need to be connected to the electrical system.
Price the strong attraction :
Indian Engineers make 9-10$ an hour compared to 40 $ per hour of H1b Visa in US. HCL currently has 18000 employees in US and 15000 in Europe and has deep expertise in computing. HCL won the contract on basis of a strong R&D background as said by Sukamal Banerjee, HCL VP.
As the sources say The Max was an update of a 50-year-old design. They wanted the changes to be limited enough that Boeing could produce the new planes faster. Only they needed would be a few changes for either the assembly line or airlines. Rockwell Collins hired HCL and Cyient. Cyient engineers used to do the drawings and provide US team by each morning. They faced challenges still they delivered it, but then could have done better.
The issue in the flight crashes was due to MCAS system pushing planes into uncontrollable dives . All this was due to bad data from a single sensor. Proper testing was not done in this case which made 346 people lose their lives.
Software outsourcing should be done with quality in mind. Cost cutting upto certain level is fine, but if there are major casualties due to lack of experience. It is a big failure on long term businesses as well as trust.
Boeing faced a $5.6 billion fall in revenue for the 2 quarter of 2019 due to max crisis.
Courtesy : Bloomberg & Aerotime