New Uncommanded Dive Risks
Last week the FAA issued statement on a new risk that Boeing must mitigate. The new risk does not involve the MACS but could lead to similar results according to the Seattle Times.
The Federal Aviation Administration discovered that data processing by a flight computer on the jetliner could cause the plane to dive in a way that pilots had difficulty recovering from in simulator tests, according to two people familiar with the finding who asked not to be named discussing it.
While the issue didn’t involve the Maneuvering Characteristics Augmentation System linked to the two accidents since October that killed 346 people, it could produce an uncommanded dive similar to what occurred in the crashes, according to one person, who wasn’t authorized to speak about the matter.
David Learmount, consulting aviation-safety editor at Flight Global and a former Royal Air Force pilot, said details of the new issue are sketchy but it’s possible that it could further delay the MAX’s return. “The implication is that this is different software in a different control computer that’s presenting similar symptoms,” he said. “When you control an aircraft with computers, which we do now, you’ve always got potential for problems.”
Boeing agreed with the FAA’s findings but has not yet presented a fix to the FAA.
DoJ Probe Expands to Dreamliner
The Seattle Times reports DOJ probe expands beyond Boeing 737 MAX, includes 787 Dreamliner.
Federal prosecutors have subpoenaed records from Boeing relating to the production of the 787 Dreamliner in South Carolina, where there have been allegations of shoddy work, according to two sources familiar with the investigation.
The subpoena was issued by the Department of Justice (DOJ), the sources said. DOJ is also conducting a criminal investigation into the certification and design of the 737 MAX after two deadly crashes of that jetliner. The 787 subpoena significantly widens the scope of the DOJ’s scrutiny of safety issues at Boeing.
The grand-jury investigation into the MAX has been cloaked in secrecy, but some of the Justice Department’s activities have become known as prosecutors issued subpoenas for documents. Allegations relating to the 787 Dreamliner have centered on shoddy work and cutting corners at the company’s South Carolina plant.
Prosecutors are likely looking into whether broad cultural problems run throughout the company, according to the third source and a person in South Carolina, who also spoke on condition of anonymity because of the sensitive nature of the matter. That could include pressure to sign off on faulty work to avoid delays in delivering planes to customers, the source said.
The entire [Dreamliner] fleet was grounded in January 2013 after two battery-overheating incidents: a battery fire on an empty 787 parked at the gate at Boston airport, then a smoldering battery on a flight in Japan that forced an emergency landing. The FAA lifted the grounding in April 2013 after Boeing modified the jets with beefed-up batteries, containment boxes and venting tubes.
In the 737 MAX investigation, prosecutors appear to be getting information from someone with inside knowledge of the plane’s development based on the questions they are asking, the third source said.
$9 an Hour Programmers with No Aviation Experience
Shoddy work and cutting corners? Uh … Yes.
Bloomberg reports [Boeing’s 737 Max Software Outsourced to $9-an-Hour Engineers](Boeing’s 737 Max Software Outsourced to $9-an-Hour Engineers)
It remains the mystery at the heart of Boeing Co.’s 737 Max crisis: how a company renowned for meticulous design made seemingly basic software mistakes leading to a pair of deadly crashes. Longtime Boeing engineers say the effort was complicated by a push to outsource work to lower-paid contractors.
The Max software — plagued by issues that could keep the planes grounded months longer after U.S. regulators this week revealed a new flaw — was developed at a time Boeing was laying off experienced engineers and pressing suppliers to cut costs.
Increasingly, the iconic American planemaker and its subcontractors have relied on temporary workers making as little as $9 an hour to develop and test software, often from countries lacking a deep background in aerospace — notably India.
In offices across from Seattle’s Boeing Field, recent college graduates employed by the Indian software developer HCL Technologies Ltd. occupied several rows of desks, said Mark Rabin, a former Boeing software engineer who worked in a flight-test group that supported the Max.
The coders from HCL were typically designing to specifications set by Boeing. Still, “it was controversial because it was far less efficient than Boeing engineers just writing the code,” Rabin said. Frequently, he recalled, “it took many rounds going back and forth because the code was not done correctly.”
Double Dividends
Not only did Boeing benefit from cheap coders who did not know what they were doing, Bloomberg notes that Boeing’s cultivation of Indian companies appeared to pay other dividends.
Boeing won several orders for Indian military and commercial aircraft, such as a $22 billion one in January 2017 to supply SpiceJet Ltd. That order included 100 737-Max 8 jets and represented Boeing’s largest order ever from an Indian airline, a coup in a country dominated by Airbus.
Grounded for Cause
The Wall Street Journal reports Boeing 737 MAX Likely Grounded Until Late This Year.
Boeing Co.’s troubled 737 MAX fleet is expected to stay grounded until late this year as a result of the latest flight-control problem flagged by U.S. air-safety regulators, according to people briefed on the issue.
The setback, at the very least, is expected to prompt additional disruptions to airline schedules across the U.S. and overseas as some 500 of the planes remain idled for months longer than previously projected.
During simulator tests of certain emergency procedures, FAA pilots uncovered a potentially dangerous situation they hadn’t encountered before, according to people briefed on the issue. The crux of the problem, according to the Boeing official and company messages to airlines, is that if a chip inside the plane’s flight-control computer fails, it can cause uncommanded movement of a panel on the aircraft’s tail, pointing the nose downward.
Tests of the emergency procedures to cope with this so-called runaway stabilizer condition, the official said, revealed that it would take average pilots longer than expected to recognize and counteract the problem.
Darn those Simulators
When you use actual flight simulators instead of iPads problems turn up. But all along Boeing has insisted and still insists iPads are all the pilots need to train.
No New Parts Needed
“We believe this can be updated through a software fix,” a Boeing official said.
Of course it does.
It could take many more months if the 737 fleet needs new parts.
What Can Possibly Go Wrong?
Boeing took a base 1964 design, overloaded it with huge engines making the aircraft unstable, then depended on poorly designed software that cannot easily be overridden to keep the plane from nosediving in crashes, while insisting training can take place on an iPad.
What could possibly go wrong with that set of cost-cutting decisions?
Unfortunately, we just found out.
Yet, even after the second crash, Boeing begged the FAA to keep the plane in service.
Mike “Mish” Shedlock
I will tell you something about the compatibility of computer programming, what American programmers do with a product like MS Windows 10 is NOT the same thing charcter for character as what gets shipped to the EU or India. I bought an HP Pavillion laptop in Ireland in 2017 then moved back to the states in October of that year. It has not been updated since because American updates are incompatible with European versions of the program. The keyboards are not even mapped the same, I have downloaded updates but the computer will not load them as they are not compatible.
Now Boeing (and others) want the internal and CRITICAL features of their plane’s functions updated and farmed out to the cheapest possible call center/programmers India offers. Just don’t be surprised when those programs do not function as intended, or at all, in an emergency.
I don’t have much experience working with DAW but I am thinking to try this. When it comes to deal with critical code, I prefer to visit website for hiring the professionals and they guide me perfectly.
Wow guys, wow! You all seem to believe that only American Software engineer can write good code…I disagree! Certainly, it’s hard to believe that $9 /hrs software engineers of high quality exist. When we hired Indian, Mexican and other software companies (almost all for data processing) we found the error rate excellent. As to writing great software, that’s got nothing to do with the country and EVERYTHING to do with the actual engineer writing the software!
However, Boeing was is search for the ultimate cost-cutting and share buyback program. Rather than investing in the company, they decided they didn’t need the capital and returned it to their investors. By some happy coincidence, the share price rose from $150 to $450…who knew (sarcasm intended)
any quality internship pays more than $9/hour for a US citizen.
Hi Mish, in my opinion, this issue is more of a failure of the design team writing the software function than the salary of the programmers. Before a programmer saw a line of code, the scope of the project and function should have been well defined. Apparently , they forgot what action the software should take to insure the angle of attack sensor was giving the correct information and what to do if it had failed. The fact it happened twice , causing two crashes, it’s unforgivable.
“Before a programmer saw a line of code, the scope of the project and function should have been well defined. Apparently , they forgot what action the software should take to insure the angle of attack sensor was giving the correct information and what to do if it had failed. “
If “the scope of the project and function should have been well defined”, all you would need to make it run flawlessly, was a compiler.
The reason computer languages don’t allow for the kind of mumbo-jumbo requirements-document-writing “analysts” and other hacks mindlessly babble in; is exactly because those restrictions are what allow you to write things in a well defined manner in them. You can’t do that in a language where yahoos get to debate what the meaning of is is.
When an airliner becomes unflyable because a computer chip fails, we have huge problem in the industry. Lately I have been encountering similar problems in excessively complicated automotive computer systems, which always end up being very expensive to fix, and few places even know how to fix it.
The problem is not the 9 an hour programmers. An airplane has multiple SW units. The 9 an hour may have been involved writing SW for the entertainment unit. We just do not know. I worked in SW developing real time or near real time SW for control systems. It is totally different to say SW for financial applications. A control system piece of SW will normally have a very detailed requirements document, a very detailed code review phase, a very stringent unit test phase and finally a system test. If during those phases something is discovered then you will need to go back and do regression testing. All those phases normally capture faulty code. In essence code that does not match the requirement.
Also, SW for control systems is written in heavily structured language such as Ada which I think Boeing still uses. This avoids coding errors. 9 an hour programmers do not know Ada and Ill be surprise if they can write C or C++.
I think they are being used as scapegoats. I think the plane is a bad HW design that can not be completely fixed with SW. In other words more HW fixes are needed. As for the 787 well that is built in a nonunion plant as far as I can tell. I know this is not an audience that likes unions. However, unions do have a role keeping business manager in check.
Boeing doesn’t develop the software for the ‘entertainment’ units. That stuff is purchased from companies like Panasonic Avionics (the eX2 system, which is basically based on Linux), or Thales.
Mish is incorrect to say that Boeing’s changes to the original 737 made the aircraft “unstable”. The new versions of the 737 (such as the MAX) are in fact very stable. The problem is that the changes to the 737 changed the flight characteristics of the aircraft which should have required retraining and certification for pilots. In order to avoid requiring pilots to retrain on the new 737s (which customers don’t like having to pay for), Boeing created software to mimic the behavior of old 737s from the controls.
It is this classic 737 software mimicry (to avoid having to recertify pilots) that is the real problem. This software mimicry doesn’t always work and relies on sensors which can be faulty (e.g. such as having a bird hit an air speed sensor). It is telling that the pilot instructions for handling flight problems is to disable the mimic software and fly the plane in a native mode. Unfortunately, very few 737 pilots have experience with how to handle the plane without the mimic software, and they only get the chance to try when something is really bad, which is the exact worst time to start learning how a new model 737 really behaves.
At last. ONE commentator nails it….wow. this all did not start as a software error. OK? This was a PILOT(s) error. or perhaps to clarify for the rest of the poor folks here: PILOT TRAINING error.
In no way should Boeing be absolved of wrongdoing. Boeing’s error is in trying to pass off a brand new aircraft as being no different from an old model 737 and lead customers to think it doesn’t require significant retraining (actually making that a selling point). That’s pretty close to fraud. But it’s a VERY different problem from saying the aircraft was poorly designed and unstable.
Right on. So let’s spotlight Boeing. Where’s this arrogance come from?
Ans: monopoly. and now we come to the truly important element. What is the problem w/monopoly? Centralized control made uninhibited by any constraints.
The problem is not the programmers. You could have the programming done by Americans being paid $150/hr and the planes would still be accidents waiting to happen.
The problem is that the engineers designed an airframe that was aerodynamically unstable and was essentially unflyable, and rather than spend the time and money to fix the problem by redesigning the airframe to make it stable in flight, they spent far less money to impart a false stability by means of computer controls to override that aerodynamic instability. If the computer fails for any reason, bad programming being only one of many reason a computer can fail, the aerodynamic instability of the aircraft itself causes the airplane to crash.
I worked for a company that setup a development farm in India. It wasn’t to replace US workers. Rather, it was to augment US workers. At least that’s what the CEO told us. I managed a group or 6 or so in Bangalore. Not only were they extremely sub par. I spent my days fixing their mistakes. Their culture is not suited for team work. Every one of them was trying to claw their way to the top. In the end, we hired maybe 25 people in India and 1 US worker lost their job. I don’t know the exact economics of it, but I suspect we lost money.
Sounds like a pretty scammy argument on the part of management. If they didn’t send work to India, they would have needed to hire people in the US to do it. Maybe none of your current workgroup lost their jobs, but I doubt there was much if any new grad or entry level hiring. Thus, over time, the US talent pool of experienced people would be subject to depletion, and over time, those jobs would have been gradually and permanently moved offshore.
I worked over 20 years in programming. I made AVP at Harris Bank now BMO after 2 years.
We hired many contractors. The best were well-paid. The worst were not.
The problem is management. They assumed that at the bottom level, a programmer is a programmer is a programmer.
I was in finance. The best contractors knew finance. The worst did not know squat.
Boeing is aviation. I suspect the best know something about what they were doing but most don’t.
Expecting to get quality code out of poor designers and programmers is a huge mistake.
Yes, this is a management fuck-up but I seriously doubt those $9 programmers were any good.
If the team did not any experience in avionics control systems then this is a HUGE problem. I’m guessing at $9/hr, they weren’t exactly vetted that well by Boeing. The world deserves whatever happens because the world chases after profits at the expense of quality. Anyone who pines for profit at the expense of quality, gets what they deserve.
Indeed. Engineers working for a company like Boeing should be well-paid so that, if there is a fundamental disagreement with management over something like safety, they have the personal funds to be able to walk away, whistleblow, etc.
There also needs to be blame put on the move to South Carolina. Boeing moved there for tax breaks and cheaper labor and a right to work state. Lindsey Graham is at the center of this. This is where the road to lower regulations leads. You reap what you sow.
Cue the tape..
The only thing lower regulations would lead to, is those more competent setting up shop driving the current Boeing out of business. One role or subcontract after the other. It’s not as if Tupolev is some epitome of well ran airplane maker.
The morons making the decisions to hire substandard work, aren’t suddenly going to grow a functioning intellect by having to spend even more time wining and dining Lindsey and his regulator cousin party members. Instead, the companies the Fed bought the morons, should be gutted overnight: By competent people setting up competing shop across the and street hiring the best airplane builders and designers on terms beneficial to both. And by customers bearing the full cost of the decisions of whom to use. Leaving the rabble to either find work more suitable to their abilities, or to simply wither, and stop being in the, way.
9 dollar per hour engineer – Human Resource Department Translation:
Student engineer in 3rd year with no job experience.
In an earlier life, I was a systems designer and systems programmer, and a good one. I concur that a bad programmer is worse than no programmer at all. Nevertheless, I have no problem with $9/hr programmers from India. I don’t care what they make. I don’t care where they are from. I do care if they are competent. A bad programmer is a bad programmer, whether they make $9/hr, or $90/hr.
The problem here is bad code, not who wrote it. Blaming it on $9/hr contract programmers is all too easy. Where wrote the specs? Who supervised the programmers? Who tested the code? Who tested the entire system? There is far more than enough blame to go around.
Ummm you don’t create an environment of professionalism by paying so little. Or threatening the US employees with outsourcing to places where $9/hour is actually a decent salary. Pay peanuts, only attract monkeys.
If company pays 9 dollars an hour it only gets programmers to whom the 9 dollar Job is the only option.
So code is free and the $9 coder is like the minimum wage waiter-busboy?
Bad programmers don’t make $90/hr for long.
Because it tends to be abstract, basically engineering by writing, there are few automated tests for bad v.s. good code, but a good coder can easily spot it. And usually have written it for less total cost.
You also can’t ISO9000 your way out of it or use MISRA or some other standard.
But current “management” seems to think you can economize this way.
Would you prefer a minimum wage Doctor, Lawyer, or financial adviser?
I’ve known some bad programmers who were very well paid. I’ve also known bad doctors, bad lawyers, and bad financial advisors who were very, very well paid. I’ve also known very good people who weren’t paid well at all. For example, we had a programmer trainee who was from backwoods, Kentucky, and he was very good. By now, I’m sure he’s an outstanding programmer and very well paid, but you have to start somewhere.
Again, I don’t care if these were $9/hr contract programmers from India. They didn’t just write code, and stick it into the planes. Someone read their code. Someone tested it. If bad code got into the finished product, there is more wrong than just the guy that wrote it, and it’s all too easy to put all the blame on him. If that were the way it worked, though, what if there was a good programmer, who was well paid, but had a bad day? Could he make a mistake, and no one checks it, so it goes right into the plane? Surely not. There is supposed to be a system in place, one that protects against mistakes, whether that mistake is made by a trainee or someone with experience. And, by the way, I suspect some of the $9/hr programmers were probably good programmers, and they are probably still programming today, and with a substantially higher salary.
Sechel has it right. The $9/hr is just click-bait. It’s designed to attract attention, and get people excited. But, if bad code is getting into finished airplanes, there is far more wrong than a few contract programmers. The whole department, from top to bottom, is suspect.
Especially the management
Guys living in the jungles of India and magically being Google grade without knowing it, are by now as common as athletic 9 footers living in villages in China willing to play ball for a pittance. They just don’t exist anymore, if they ever did.
The reason good people get paid, is because there is demand for them. And hence, demand for finding them. Weird anomalies of genuises in mudhuts are just that, anomalies. Even more so in fields as specialized as airplane control software.
None of which means the “wrong” is with the programmers themselves. I’m sure they delivered work more than justifying $9/hour. Just as a $9/hour basketball player, or surgeon, would.
Instead, the “wrong” is with the ones who believing cutting costs by skimping on important work and those who perform it, in order to hand more resources to those who provide little to nothing to downright negative value. Which is exactly how the once-was West ended. Boeing is now seemingly little different from any other once-were technology leader: Ran largely by clowns who wouldn’t know a $9/hr burger flipper from Einstein even if they tried, as long as both were labeled, by some clueless “recruiter”, as a “programmer.”
You’ll always end up in that predicament, once decision making authority, in complex fields and over complex organizations, is granted arbitrarily. Whether on account of being the King’s cousin, a High Priest’s favorite choirboy, or one of The Fed’s, or Government’s, favored welfare queens. There simply is no other possible outcome.
Resources obtained by Boeing, is now being allocated, not to those who build and design airplanes, narrowly according to proven merit at doing exactly that; but rather to clowns doing anything but. It’s akin to the guy who are handed the surgeons’ tools at a hospital, being the HR guy. Or some insurance hack, PE monkey or just simple but kissing career climber.
Anyone but the guy who has actually performed similar surgery before. Who is, instead, being outsourced to someone willing to give it a shot for $9/hour. And people act as if him losing the patient is some sort of surprise, just because some rhesus monkeys who “made money on his house” hired a certified moron to “supervise” him…..
I also was a good systems programmer and systems designer, and I’ve also met some excellent junior (and therefore cheap) programmers who were poorly paid (although not for long) and many who were atrocious and highly paid.
I wonder what systems you could have worked on where you subscribe to the idea that bad code can be identified and corrected by testers, QA engineers, or anybody else other than another excellent programmer—and then not easily. I have never, never seen it done. The kind of error the Boeing contractor probably made was likely exactly the kind that only an idiot would have made—and (ironically) exactly because of that probably not a situation that most testers would have thought of testing. It’s a real Catch-22. Testing is necessary (it would be truly stupid to skip it) but mostly serves as little more than CYA for management.
I’ll also disagree (and here I enter deep water) with your assertion that it doesn’t matter where the programmers were educated: it does. Where they were educated imprints an attitude on the mind that is hard to alter. Indian education (and Asian more broadly) emphasises following rules one foot after another. US education (although much less so than formerly) emphasises independence and initiative. Good enough just isn’t. I’ve seen the loss of this stress on individual thought and responsibility in the US with regret, in Germany with astonishment, and in the UK in recent years with sadness.
Political correctness in designing airplanes kills.
Ah yes!
And when the $9 an hour outsourcing {engineering} finally gets to effect fund managers, bankers, medical doctors, on and on, everyone will FINALLY understand why the middle Amerika {those steel workers , coal miners, general factory workers, etc} voted for Trump. A non-adiabatic economy only works for few at the expense of many others.
S H O D D Y
One word says it all. Look to the leadership to find out how shoddy creeps in everywhere. It seems the emblem of our times. Shoddy leaders with shoddy morals and shoddy vision, everywhere throughout the West.
It’s also more than shoddy. It’s fraud. Not resubmitting the design to the FAA after changing the limits (4× earlier design) of MCAS is fraud. And the relationship between the FAA/Boeing, also shoddy and arguably fraudulent. As always, there will no charges or trials.
Meanwhile real US citizen engineers, software, computer, hardware, sent their applications to Boeing for jobs. And were ignored. While Boeing hired a firm that no self-respecting talented US citizen would even bother to apply to, to staff the positions. Effectively creating a self-fulfilling prophecy that foreign nationals were required for jobs “Americans wouldn’t do”.
The solution is to ban the H-1B visa. It has destroyed so many US citizen STEM jobs, and left so much great talent “on the couch” or “in the basements” of the nation.
How many other American companies have massive problems and liabilities currently hidden that have come from saving a few dollars by outsourcing software development to India or bringing cheap indian programmers to USA on H1B-visas who are NOT More talented than American programmers they are just cheaper and allow company executives to extract more bonuses and options from the company just like happened with boeing?
It’s scary to think but it’s damn true. Lots of software developers aren’t even considering corporate jobs because they can do better on their own.
BofA, chase, wells fargo, american express, visa, mastercard)… all use Indian programmers to save $$$. Wall Street also, but not as much. Travel agencies such as all the airline sites. Everyone in the development team knows how inexperienced & bad these off-shore workers are, yet some of the banks have 30,000 Indians working for them (I work for a bank). They work really long hours, and management thinks that makes up for all the redo’s required. This article doesn’t say it, but another said that the back and forth for one fix on pre-release MCAS was 18 times. Boeing should have kept their experienced programmers; My sister had a good deal of experience & she was laid off a while ago. She stopped flying on any Boeing plane 15 years ago, because she was a software auditor. I haven’t flown Boeing in a long time also, preferring to use AirBus and Embraers (anything but Boeing!)
Somebody in Boeing leadership said :
“Hey, let’s outsource software development to India and we will save money ”
Every other executive at Boeing agreed.
Everything went fine for a bit and profits increased and executives got bigger bonuses and more money from their options because the failures and problems were hidden with workarounds.
Then reality entered and Boeing brand and image and products Were destroyed.
Every executive at Boeing should be FIRED and then sued and asset-stripped to their underwear by shareholders.
Being in the software development business, I can attest that having developers who suck on the team is way worse than having no developers at all.
Also in the software dev business. I used to work with a guy who used to say “if you think professionals are expensive, you should try hiring amateurs to write code for you”. I think this proves his maxim
Was anything actually wrong with the code in the 737Max?
is that a rhetorical question?
Its not a coder’s job to debug the entire overall system, and determine suitability of the code relative to the rest of the system.
Now if the code itself, or the computing system the code ran on, failed to operate, and didn’t implement the specified logic, then that would be a failure of the coders (ie: the alleged $9/hour software programmers). But MCAS was clearly a concoction of the systems and aerodynamics engineers, not the software programmers.