Between the engineering staff and the warehouse workers, I wonder how long it will be until they have already fired everyone who ever would have been willing to work there.
Even with candidate pools of hundreds of thousands of H1-B engineers and tens of millions of illegal immigrant warehouse workers, there still comes a point where such a big company firing so many people so quickly exhausts all their options.
It reminds me of the Robot Chicken Sketch where Imperial Officers aboard the Death Star all pretend to be force choked to death by Darth Vader so they can avoid getting killed by lightsaber, then come back in under different names in different jobs. It's worse though for Amazon: nobody wants to come back.
It seems amazon itself is aware of this issue. The linked engadget article even mentions this:
> "The rate at which Amazon has burned through the American working-age populace led to another piece of internal research, obtained this summer by Recode, which cautioned that the company might “deplete the available labor supply in the US” in certain metro regions within a few years."
Agriculture, food processing and handling and everything associated, particularly meatpacking, and that's valid across countries. There's a reason that even if you pay (relatively) obscene wages, there will be no domestic employees willing to pick up these jobs - the work conditions are usually horrible because there are few industries as cut-throat and cost-sensitive as anything related to food.
The root cause is global competition, especially from countries with very low wages - Ukraine, a country in Europe, for example, was a top food supplier for Africa -, and widespread income disparity in many Western countries - 67% of Americans self-report to live from paycheck to paycheck [1], for example. First it's "luxuries" that get the cut - travel, eating out, entertainment - and once everything has been cut, people go for savings in food because that's the last large expenses block that they can meaningfully control.
The Maritime industry can run into this too. Tug companies, barge companies. The level of consolidation vs the size of the labour pool is what does it.
I like to think I'm halfway decent at my job, and I wouldn't work there once. During undergrad, my landlord working for AMZN on the opposite end of the country offered me an interview, but it was during final exam week.
I asked if I could schedule the interview after my final exams, and his arrogance really showed when not only did he refuse, but then insisted my exams are not don't even register on the same scale of importance as the opportunity to work for Amazon.
Somewhat related: a recruiter at Google cold-called me a couple months into my first job out of undergrad back in 2016 and was similarly condescending about "the chance" to work for Google compared to everything else. I already had a low opinion of them when they gave my then-girlfriend an introductory O'Reilly book on Java after she failed their interview.
I regret being born too late to work somewhere like Bell Labs, SGI, or Sun. I had a ton of graybeard wizard coworkers from these places, and they were all a pleasure to learn from and even better friends. For the first 2 years of my first job, every day of work was like walking into the Shire and talking magic spells with 20 Gandalfs.
That job was great until I got put on a team with a guy who was a former middle manager at some IBM-like company and went from being surrounded by people lightyears ahead of me to being surrounded by Dilbert characters. The messed-up part was that it wasn't even punishment. I was rewarded after completing a project with my choice of which team I joined next, and I joined the wrong one. I assumed that joining a new team to utilize this newfangled "cloud computing" thing would be trailblazing, and I didn't do any diligence on who I would work with.
To this day, I still regret not rejoining the first team I worked for, otherwise I would still be at that company and happy about it. Then again, the boredom and discontent while being on that sucky team is the reason I started investing, and now I can buy a house in cash and fund myself to do whatever I want for at least a decade. Hard to complain about the way things turned out.
I interviewed with Amazon a few years back. The whole thing turned me off. A recruiter reached out and I was interested (it was late 2020 and the money was tempting). But before the first phone screen I had to have a call with the recruiter again, where she gave me a list of things I needed to "study" and was told that "successfully candidates usually spend 5-10 hours preparing for the interview". The study list was the usual list of CS101 topics. I didn't bother preparing and it was a good thing because on the phone screen the guy just asked me some a fairly mundane coding question and then some more general stuff (it was actually a very reasonable interview). Based on that they wanted to proceed to a final interview which was an all-day affair (on zoom of course because this was during the pandemic). But first I had to do ANOTHER 1h call with the recruiter where she gave me ANOTHER list of things I needed to "study" and reminded me that I should spend 5-10h preparing. That was too much for me and I politely declined the opportunity.
The question that arises is: How can you potentially spot which companies are about/likely to enter a 'golden era' when you interview there? What questions could surface some sort of likelihood? Is it possibly to identify them before they enter the 'golden era'?
Some 20 years ago I started a job at Google in Mountain View, and they were paying for a rental car, so Enterprise sent a driver to pick me up to do the paperwork. On the way I was chatting with him, telling him how amazing life at Google was, all the restaurants and the stocked kitchens and massage rooms on every floor of every building etc etc. He said "Do you know what this campus used to be before Google?" I said "Yeah, they told us at the orientation, it was SGI." The driver said, "Yes, and ten years ago it was exactly like that at SGI, too. I was an engineer there."
In the UK we have a heuristic that by the time a tech giant builds a big UK campus (an imitation of their SV HQ) then you know they are in the decline phase. Some of them decline so fast they don’t even get to fully complete the campus, yet others seems to have beaten this curse… so far.
One of the best stories I have ever heard in here to be really honest. Sounds like a joke but its packed with subtle meaning of how companies rise and fall so quickly.
You have said that the driver worked because he had (enough money?) and he might have wanted to relax with the driving job but still, its an amazing story.
I think the key difference between the old big guns, SGI, IBM and the likes, and today's Google, Microsoft, Amazon, Meta and Apple is the diversification of income streams. Even if any one of these companies just completely fucks up an entire business line or it gets replaced by something better, it doesn't matter because the companies themselves are so utterly large they can and do survive that - or because they can, like Meta, just buy up whatever upstart is trying to dethrone them.
I don't believe they can keep this up forever. Take Google which is dependent on Google Search. That their search is becoming actively worse is common knowledge, reason being more searches equates to more ads shown. If a company which respects you as a user comes around people will jump the boat.
We can see this with YouTube. YouTube shows so many ads, that people have been using TikTok instead. They say TikTok is also bad for them, but they rather use it than watch an ad every 30s.
Point being entshittification comes at a cost, and companies partaking in shitty activities can only keep this up for so long.
Even though I worked for companies that killed Sun, I never stopped admiring the foundational work the company was doing, which was not just cool, but critical for technological progress, and was very sad when the company sold out to Oracle and was gutted alive. HPC stuff Sun pioneered is still very relevant today. In an alternate timeline, Sun fully embraced Open Source and became a key pillar of the internet today.
Unfortunately, while we are well aware of cool tech companies that were ran aground by the finance/sales/management consulting types, Sun felt like a company ran aground by engineers.
Zuck famously kept the Sun logo up for quite a while when Facebook bought Sun's HQ campus, as a warning to the employees of what they could become. In some ways, Facebook/Meta is the spiritual successor of Sun, just like Google became the spiritual successor of SGI when they bought the SGI campus.
But these two ad driven companies never quite became the new Sun/SGI, for better and worse.
I don't disagree that Sun was a company run aground by engineers -- though I certainly like to think of myself as one of the engineers trying to navigate us around the rocky shoals! For whatever it's worth, I broadly stand by my analysis on HN fourteen years ago (!!) of Sun's demise[0] -- which now also stands as clear foreshadowing for Oxide eight years before its founding.[1]
I'm not sure what Sun could realistically have done to come out the other side of the dot-com carnage. Other companies in roughly equivalent situations come to mind. You start looking at doing a hard reboot when the margins for that reboot aren't there and it's difficult to see the light at the end of the tunnel. Maybe an earlier reinvention involving more open source and alignment with where hardware was headed. Don't know.
Sun did waste a lot of money in buying MySQL, $800 million in cash and $200 million in stock. Certainly a distraction, as well.
Sun never offered any way to inexpensively get onto the on-ramp of Sun hardware and software as they thought they could continue selling high-margin hardware forever; they had their $995 V100 which even included their much-loved LOM which was a remote-management device like iLO/DRAC/IPMI , then followed it up with: nothing.
That Solaris/Toshiba laptops deal was interesting, but if I recall correctly the price was a bit too much, and maybe it could have been considerd yet another distraction.
I surely would have liked to get one of those laptops, though.
Well, in stock market terms the MySQL deal paid for itself. It pushed the stock well up. However turning this in real money wasn't possible in the year they had till IBM and Oracle did their bidding.
Interesting. That was not my perception of Sun at all. “The network is the computer” was a marketing campaign. Java was a language developed for IoT/toasters, and then hard pivoted to a write once run anywhere weblet language (ultimately to be replaced by a guy who threw together an integerless programming language that sounded like a skin condition, renamed to ride the crest of energy sun marketing money threw at things).
Sure, Solaris was rock solid, but it was also pretty conservative in its march forward as a Unix, being ultimately trumped by Linux.
Sun had an amazing team of people that worked on Self project led by David Ungar and others (Lars Bak who helped give us V8). They let the whole team go, who then went off and did sime cool things with dynamic optimization, which Sun ultimately ended up hiring/buying back to create the HotSpot VM.
Any NIH and other dysfunctionality went far beyond the engineers at Sun.
Also, the number format is a standard, not only used by JS, and given that it was supposed to be a minimal scripting language it is hard to argue against the initial design choice of choosing one all-encompassing big standard, and not burden the language with a complete set. Since he criticism was on the initial design:
> ultimately to be replaced by a guy who threw together an integerless programming language
I would like to refute it by pointing out that the criticism ignores the initial use case, as well as the actual existence of integers within that larger number format standard. Later, when enough people (and companies) demanded it, a big integer type was added, after all.
Internally runtimes use different paths depending on what kind of number it is.
For many use cases of integers, especially internal ones, like array indexing and counting, those integers are just that, and an extra integer type for extra purity is not much of a problem. For other uses of integers, e.g. finance (using cents instead of dollars), it sucks that you have to pay a lot of attention to what calculations you perform, so not having (had - until BIGINT) a real integer type as aid indeed made it less pleasant to do integer arithmetic.
> “The network is the computer” was a marketing campaign.
No, not at all. It became a marketing campaing in the very late 90s dot.com boom, but the concept that defined Sun goes back to the beginning, 1984. Back then, that was a radical vision and Sun truly lived it internally for a long time.
This was also back when you could walk into the library and get the email credentials of a random professor and then use it to hide behind when you took down a network of another in state university because an engineering professor didn't think computer science majors were as smart as he was.
Yeah, man, good times.
My buddy got a visit from the feds and lost his computer lab access for a semester.
I'd note that a huge amount of the work at those companies was hardware (and a lot of theory in the case of Bell Labs)--though there was, of course important software as well, a lot of it related to Unix.
Doesn't mean it might not have been a blast but not hacking on software and playing in the open source world as is the case at at least some companies today.
Well I still think that software like DTrace, ZFS, NFS, IRIX and Solaris, IrisGL, and the like are cool, even if there was a lot of hardware engineering. I realize that there are disadvantages to it, but the variances in ISAs (MIPS, SPARC, Alpha, etc) seems like it could have posed challenges for software people.
Sun did a lot of great software too and I know a lot of the folks involved. I just think many people look at the innovation through the lens of software (especially open source) hacking which a great deal of it wasn't.
When I was in the minicomputer business, it was maybe 50/50 hardware and software (and that mostly assumes you considered software to include low-level things like microcode). And software people weren't mostly paid more than those in hardware--which is to say generally a good middle class professional wage.
> which is to say generally a good middle class professional wage.
Working with Suns and other workstations as a teen (so my perspective was limited), I caught the very tail end of software as a modest middle class professional wage for everyone doing it (right before the dotcom boom hit).
The people I worked with were really good at what they do, but not strutting like newcomers started doing pretty much the instant the dotcom boom started, and not rich. (Well, one guy did buy a used MR2, and get his private pilot license, but he also lived with his wife in a trailer on an undeveloped parcel. He was a very solid software engineer, working on important stuff.)
I might have inadvertently tried to preserve some of that modestly-paid excellence of the generation before me, but I don't recommend that. Cost-of-living in my area is determined by people making FAANG-like money (well, and real estate investors, and price-fixing), and you have to either play along with that, or move away.
Mind you, California cost of living was on the high side even in the nineties even relative to at least modestly expensive areas like the Boston area suburbs--there was really very little tech in Boston proper at that time.
But a somewhat high-flying (albeit hardware) company was recruiting me for a CA job and they basically admitted it would be a lifestyle downgrade in terms of salary.
Some people made a lot of money when dot-com hit. A lot also got wiped out and ended up leaving the industry.
I never had the highs or lows. I was probably making something south of $100K in the late 90s.
> But a somewhat high-flying (albeit hardware) company was recruiting me for a CA job and they basically admitted it would be a lifestyle downgrade in terms of salary.
I've been given second hand accounts of similar situations. One was team consolidation, and the business was offering Boston-area engineers positions in San Jose. One of the folks who moved with his family was back in MA within 5 years. His salary was not adjusted as much as it should have been for the cost of living difference.
If it meant that people who didn't want to do software didn't ans people who wanted to do software did software, then it sounds nice. I was never interested in making a lot of money in software.
A question, though. Was software designed at Sun closely with hardware teams and vice versa, or were they mostly disjoint? Presumably many hardware companies that have succeeded have built good tooling around said hardware (like NVIDIA).
I didn't actually work there; I knew a lot of the folks from the perspective of an IT industry analyst both during and after a lot of the work there. I certainly saw some level of integration with things like Dtrace (how couldn't you) but when you were a systems company, it's probably the case that software folks couldn't really just divorce themselves from hardware.
The low-level software work at Data General where I was prior to the analyst biz was certainly integrated to a certain degree--read Soul of a New Machine if you haven't. The software folks for the minis were also mostly in the same same location. As things migrated to Unix, most of that team was in RTP and it's probably fair to say that there was less integration though probably wasn't something I thought about a lot of the time. Hardware stayed in Massachusetts.
I got interviewed twice for Google, the first one I made it to the second round of phone calls, the second one only the first phone call.
The third time a Google recruiter reached out to me with the sales pitch that I was a great engineer that they would like to have at any price, I berated him if that was the case why the previous two experiences.
Never heard from Google HR ever again, and I am not sorry, I am happier this way.
I had better experiences in interviews for EA and SCEE than Google, which again I also am an happier person not managing to get an offer, and endure the crunch lifecycle of the industry as reward.
Wally is the one Dilbert character I can tolerate in the workplace. He's honest about who he is and what he does. When you know you're in a bloated company run by buffoons, all you can do for your sanity is work to rule and not upset the apple cart.
I was Wally for the last 2 1/2 years of that previous job until I started to realize I'm becoming more and more like a Dilbert character myself. Something in my brain just told me it wasn't sustainable, call it fear of God or paranoia, but letting my skills atrophy in a place like that for 20 years didn't seem like it would end well for me.
The only problem was that I stayed so long, and it made me hate software engineering so much that I didn't even want to be a software engineer anymore.
I put up with it just long enough so I could avoid selling stock and drawing cash out of my portfolio, and now I'm back at square one as a post-bacc student getting my applications in order for MD and PhD programs where I'll most certainly wind up drawing hundreds of thousands out of my portfolio to pay rent and eat dinner for about a decade.
It's sad, I really enjoyed systems programming, but it seems like finding interesting systems programming and distributed computing projects that have significant economic value is like squeezing blood out of a stone. Maybe LLMs or future progress in bioinformatics will change that, now that finding ways to shovel a lot of data into and out of GPUs is valuable, but I'm so far into physiology, genetics/proteomics, and cell biology that I'm not sure I would even want to go back.
I'm currently in a place that pays me €100k just to sit on my ass, and I can do that remotely. I've tried actually doing some work, but that backfired. Not sure what to do, because on one hand my skills are evaporating, but on the other if I wanted a job that pays more I'd have to learn a lot and then work substantially more. I'm wondering if maybe sitting here until retirement is a viable option.
> I asked if I could schedule the interview after my final exams
Ha, my interview for an Amazon internship was an hour after a 3-hour final exam :-)
But the job market right now is quite bad, and after hundreds upon hundreds of internship applications I would've been stupid to give up this chance. I would work for Amazon in a heart beat.
i’ve interviewed with aws and received offers twice over the years. the first time they made me pay for my own lunch. the second time no lunch break was afforded. i didn’t accept the offers though i know several truly excellent people who work there.
When I interviewed at a Google outpost, a good-cop employee they mistakenly thought had a connection to me took me to lunch (message: forget about the bad-cop interviewer you were just with, you're among friends, loosen your tongue so our spy can report back) in their cafeteria (message: look at the free food perks you'd enjoy), and initiated a conversation with an visiting economist there who then spoke of something oddly relevant to my research interest at the time (message: look at the interesting people and collaborations you will bump into every day).
Your interview lunch experience sounds like message: this is what it's going to be like, and we don't care if you join us.
I have done a round one interview and I don’t see how it can be interpreted to do anything but turn away people with a brain.
Memorize Amazon’s insane company values and relate your resume experience to it. And that I mean every single bullet point.
Interviewers were all run by robotic people. Coding test had zero flexibility, you had to just write code in a special barebones text editor that had zero feedback besides pass/fail.
You’d have to solely care about Amazon RSUs to consider that job. They are self-selecting for the worst kinds of candidates.
The dumb thing is that it should be a job that doesn’t burn people out because they basically own the market and haven’t needed to do any sort of innovation. Amazon’s corporate culture just has a burnout fetish.
I interviewed in 2015. The recruiter told me to read the Amazon Leadership Principles, but I thought it was ridiculous to prep for something so specific to a single company, especially as I was interviewing at other companies too.
I got the job, and I think being natural helped. I've interviewed thousands of people at Amazon since, and too many people just say the buzz words with no meat, and it gets them nowhere i.e. I showed customer obsession when I.....(and then gives a bad example)
If you had been born early enough, you probably wouldn't be a programmer at all because far fewer people were. Conversely, everybody is born at the right time to join in the heyday of something amazing. You just have to identify that something and be lucky enough and/or try hard enough to become one of the few future-gandalfs. There are companies today making flying cars and dexterous humanoid robots ffs! Or SpaceX! Amazing engineering work never stopped, it just didn't linger on in the same fields.
Fair enough, but in my estimation that next big thing is gene therapy, and the best way to get involved with it is to become a medical geneticist.
I'm sorry to be a buzzkill, but I just can't get excited about privately-funded space rockets or Japanese girlfriend robots, not even if there were an 8 figure stock compensation package in it for me. To me, it's all just "do something grandiose for the venture capitalist bucks, and then maybe figure out how it helps people in 20 years."
"My sister killed her baby because she couldn't afford it and we're sending people to the moon.
September, my cousin tried reefer for the very first time.
Now he's doing horse. It's June.
Is it silly, no? When a rocket ship explodes
And everyone still wants to fly
Some say a man ain't happy truly
Til he truly dies
Hi, I am a half decent engineer. I say that as objectively as one can say something like this about themselves.
I worked at Amazon. Twice. In total about a decade as a Principal Engineer. I left voluntarily a few months ago.
I have zero regrets about my time at Amazon. I learned lots, worked with some incredible people, and had fun doing it.
And the culture? It was life changing for me, especially when I first joined. In all the best ways.
And Amazon today? All I’ll say is that at their size, maintaining solid culture is damn hard. The hiring spree peri-Covid definitely added unimaginable stress to maintaining the culture the company was built on.
They’re a big company, and thus a big target. It’s easy, cheap, and even lazy to kick them with stuff like this.
The truth is that while it’s changed a lot over time, anyone fortunate enough to work there should embrace it.
I gotta be candid with you: "anyone fortunate enough to work there" is exactly the kind of arrogance that rubs job candidates the wrong way. A lot of people don't see it the way you do, and you would do well to take a moment of honest self reflection and consider the reason why.
Before you reflexively jump to "Amazon is just not for everybody", it's only fair to warn you that a lot of people around here have heard this phrase and don't buy it. My former landlord said the same thing at least 9 times in the 8 times I've interacted with him face to face... right up until he got fired. Management conveniently put him on a PIP 4 months before a vesting cliff on his 2-year RSU vesting schedule.
Yeah, Amazon is not for everybody- just the kind of people who either perpetrate or fall victim to pyramid schemes.
Amazon, on the engineering side, is rough for high-end software engineers, but let's be very real: it pays better than the vast majority of careers available in America, even right out of college, and while it's stressful so is being a teacher or working in the food industry, and those jobs pay peanuts compared to an Amazon SWE. It is fortunate to get a job there. It may be even more fortunate to get a job at another high-end software company but that doesn't change the fact that a job at Amazon is life changing money for most people.
To pre-empt hyperbolic responses: I live in Seattle. I personally know plenty of people who have worked at Amazon. I know plenty of local teachers. The teachers work as many hours for a tenth the pay and burn out just as fast.
You might be right about the hours the teachers work, but they're not doing it to avoid being fired. It is entirely self-imposed stress. Teachers have unions and tenure. I'll help people, but I won't martyr myself in a hospital, let alone a frickin' tech company.
I don't consider 60 hour work weeks a privilege to be coveted, especially if I'm capturing less than 1% of the value I produce. I'm sorry man, but white collar serfdom is still serfdom.
If you're really going to keep begging this hard to be a serf on someone else's fiefdom, I'll tell you what: you can keep overworking yourself as serf on fiefdoms in which I might own shares, and increase the value of my portfolio for me, so I can draw even more passive income every month. I formally withdraw my protest to your "Protestant" work ethic.
I'm not trying to be an asshole, but I'm just hoping you take what I'm saying burns into your subconscious and something in you changes: that work into which you've placed so much of your self-esteem is just funding other peoples' passive income to either not work or live ostentatiously. If that's a great privilege to you, God bless you. I'm sure my wife would love it if I just retired- please make it happen, she would love me even more for it.
EDIT: P.S. Your "privilege" of working for Jeff Bezos' portfolio sounds a lot like a biblical curse. Specifically, 1 Samuel 8:10-18 and Psalm 105:44.
"He gave them the lands of the nations, that they might inherit the fruit of others’ labor"
> It is entirely self-imposed stress. Teachers have unions and tenure.
You don't have the first idea what teachers do or what their jobs require.
If you really think all teachers have tenure and are supported by unions, you also have not the faintest, foggiest inkling of a clue about the job market or professional environment of most teaching in the US.
I think it's also worth pointing out that, even as you repeatedly label others "arrogant," your comments in this thread are themselves breathtaking in their arrogance. I rarely run across HN comments so condescending, dismissive, self-righteous, or self-congratulatory.
>You might be right about the hours the teachers work, but they're not doing it to avoid being fired. It is entirely self-imposed stress. Teachers have unions and tenure.
No, and no. Your perception of modern labor markets is dated and disconnected from reality. Schools are constantly facing budget cuts and a huge chunk of teachers aren't even full time (instead getting things like a 60% time position - less pay, no security, but enough hours that good luck working a second job). Those that get full time positions still face the elimination of their positions.
I guess I just assumed teachers have the same protections in deep blue Washington state that they have in Illinois and Minnesota.
If I'm wrong, I'm wrong. However, there's a fix for that: move to a state where the job prospects are better. That sounds like "let them eat cake", except that moving to another state as an American citizen is a lot easier than what of people went through fleeing the Sicilian Mafia or the Irish Potato Famine, let alone WW2 European front or the fall of Saigon.
We do a lot of business with them and we have workshops with them sometimes, and the one thing I notice is how they're all so evangelical. They wouldn't say a bad thing about their company. I couldn't be like that, when I work something I sell my knowledge but not my soul. I'd always speak freely (not always appreciated but usually it's not a problem if it's true).
But that company culture leaves me with a very low opinion of them and very little trust. Even Microsoft engineers are less brainwashed. I've had several that just told me the truth about services.
Maybe it depends on the country but it feels like this is just their culture.
I've worked there since 2015, and this simply isn't true.
There's a lot wrong with AWS (and it's got a lot worse in the last 3 years), but there's also a lot right, and there are some really, really smart people there, several of which have boomeranged (people who left and came back).
The formula is usually more money and ability to work special team isolated from the usual toxic orgs. I think A9 was probably somewhat like that, and AWS probably used to be at some point long ago
I know some people who are fine working there. No one seems thrilled but if you're an above average engineer who is just getting by at 140k a year and suddenly you're looking at 350k a year as an SDEIII or something, that can be a life changing amount of money.
However, I think the question is, what percentage of engineers can pass the amazon interview but not the Apple/Databricks/Uber/Google/Meta ones. Because no one is picking amazon over the aforementioned companies.
However, maybe there's an opening at Amazon and not the other companies, or maybe that's your only offer. I certainly think it might be worth it for a a few years.
Assuming they’re already living in Seattle: pre tax, and supporting a family? According to a quick google, that’s ~$100k after tax, a 3BR in Seattle is ~$4k/mo so you’re left with $52k for everything else, in a HCOL area. I buy it.
Yep, everyone is excited until they leave after 1 1/2 or 2 years. There are always outliers but my personal experience is that the churn rate is incredible high.
There seems to be some sort imputation that, just because someone is on an H-1b, that they are not a good engineer.
I used to be on an H-1b and gladly came back home to India. I run my own business now. And yes. I'm ex-Anazon. It was a tough place to work, but circa mid-nineties, the stock options made it worth working for them.
I'm willing to bet I'll outcode a significant fraction of the audience on this site. And I'm not even close to the best developer around. Some of the smartest people I've met have been on an H-1b visa. Please consider not letting prejudice affect your view. You'll do yourself a disservice by underestimating your competition.
The only thing I implied is that workers with fewer rights that a U.S. citizen are easier to exploit and abuse.
If I refuse to take a work-related call at 3am, the worst that can happen is that I get fired, and spend months looking for a new job.
If you refuse to take a work-related call at 3am, you get fired and lose your ability to stay in a place you have lived for 5+ years and made your home.
That's BS, and I hate it almost as much as you do. You can be blackmailed with deportation, and I can be replaced with someone who can be blackmailed with deportation. We're both getting screwed in this current arrangement.
I think you’re not considering the other side of that perspective. I am sure you are very happy for your fortune to have been plucked out of India and been given the opportunity to work at Amazon and presumably live in America, which put you in the place you described that seems to be in a really good position today. The issue is that not only was the H1-B meant for highly specialized people that cannot be found in the USA, it has very long been absolutely abused by American corporations and politicians that have been betraying their own people for several decades now by engaging and ignoring this abuse that was really just about undermining salaries of Americans by giving the opportunity to you rather than Americans, while it was really mostly about enriching the rich. You were essentially just a method for the rich to get richer.
I am sure you are a wonderful person, but it’s simply an unjust treatment of Americans, even if you personally had nothing directly or reasonably to do with it. The betrayal and abuse was perpetrated by the “Americans” that led the corporations and paid off the politicians, and also the American citizens that were distracted and careless about their own politics and government and future for their own children. I doubt you would be ok with your own ruling class and rich to betray your children and the future of India, would you? It’s crazy, but America’s people largely and for a long time absolutely betrayed their own people.
I would not wish it on any society, even though it has been pervasive all over the “West”, where the rich, corporate captains, and politicians betray their own people. Imagine if your Indian politicians were to sell out India to the West or maybe import Africans or something similar, I would hope that the Indian people would make it absolutely clear to the politicians and rich that they are staring down a loaded gun and it’s not their finger on the trigger. So do I also wish it for the people of all of the western countries that they retake their sovereignty and self determination away from the rather parasitic oligarchy that has unconscionably been betraying its own people out of undeterred greed and crime against the very people that allowed making them rich and powerful in the first place.
It is not a personal thing, I think it’s just that people are recently getting a lot more angry about things because the American empire is hitting a rough patch that it has not experienced in anyone’s living memory and as it is said, (adapted) the naked people start getting angry when the tide goes out and there aren’t enough jobs to also be super generous by giving them away to Indians benefiting from the abusive systems of the parasitic cabal of the ruling class.
What you may also not be totally aware of, is that H1-B is only one of many different systems and programs that have been abused and quite literally benefit and profit foreigners overt Americans. Imagine if that existed in India; where I go to India, make 2-3x what the average Indian makes, the government gives me free housing, my children get free education and free healthcare, and I get extremely beneficial government secured loan terms on business loans and get grants to start a business and free consulting and services, and I get to bring dozens of my friends and family into India to work in my business, and I also get beneficial home loans to buy up houses and drive up prices, and my foreign children get preferential treatment in Indian universities (…while local Indians don’t get those things) and I run for office while all the foreigners I and my advantaged community brought over to India start getting our people into the government and we start taking over Indian institutions and government offices.
I combine and crossed things a bit because is a bit more complicated and nuances of course, and many Americans aren’t even aware of just how many programs and states are in place that advantage foreigners and disadvantage native Americans, who could even very well be the descendants of the founders of America. That’s why things have gotten rather tense and as it looks, unfortunately, it will likely only get worse from here; especially as BRICS builds out more of their alternative fiscal, monetary, economic, geopolitical structures; and the same traitors that control the USA will/are starting to get very nervous and borderline panicky. It seems Thucydides Trap is in full effect.
> the government gives me free housing, my children get free education and free healthcare
None of these are H-1B perks. (Note: H-1B, not H1-B. It comes from ¶ H(1) of the INA of 1952; H-1A for nurses and H-1B for non-nursing specialty occupations [1].)
Immigrants pay for housing in homes that pay property taxes that fund public education. Their employer pays for their healthcare that costs multiples what most of the same treatments and drugs cost in India. (Only once they become a resident alien do they qualify for marketplace subsidies.)
> native Americans, who could even very well be the descendants of the founders of America
You've got to be shitting me. (Try native born next time.)
> my foreign children get preferential treatment in Indian universities
The students of H-1Bs (from India, no less) do not get preferential treatment in American university admissions. If anything, it's the opposite. It's why Indian Americans join lawsuits by Asian and White Americans around removing race considerations from college admissions.
> Thucydides Trap
"Research by Graham Allison," the guy who coined the term in 2011 after a career in the Reagan administration and, before that, at the RAND Corporation, "supporting the Thucydides trap has been criticized" [2].
> “native Americans who could be descendants of the founding fathers” is rich with humor
Yes.
Though I did consider, until noting the context, that they might be referring to descendants of both the founding fathers and Native Americans. Chilling, actually. To think that a good number of one branch of our founding fathers' descendants may have been exterminated by another.
This is one of the most painfully ignorant things I've read in hacker news. I don't even have anything constructive I can reply to, other than a recommendation to come out of your bubble a little bit.
> Imagine if your Indian politicians were to sell out India to the West or maybe import Africans or something similar,
…
Firstly, this is so wildly tone deaf, that I don’t know what to say.
Secondly - India (and most developing nations I will bet) have had violent protests at some point in their history about people from other parts of the same country coming in to take jobs.
Most of the world understands why America feels the way it does. Hell I personally have better articulations of America’s problems than many Americans do.
But for the love of all that is holy, please don’t end up betraying your ignorance of the rest of the world and reminding us of some of the worst features of American stereotypes. It may be meant as an effort to find common ground, but only succeeds in irritating and potentially alienating.
Believe it or not, compared to the vast majority of humanity, your problems are better than what they will face.
I do not ask you to take an interest in the world. It would be nice if you did.
I do ask that you serve your own purpose, your own argument, better.
I would never work for AWS, given what I've heard, and consistently, of their internal culture.
Also, everything I've seen while working with internal staff makes me feel there's a culture of obfuscating all weaknesses from customers, practically to the point of deceit.
Are there really many illegal aliens working in the warehouses? I know that Amazon does verify employment eligibility and checks documents. There may be some committing identity theft but I doubt that it's a large proportion.
Not really. For low skilled jobs that don't require much English, illegals are going to be as good as locals. Maybe even better because locals who are any good are mostly going to move on to better opportunities.
Yeah this. And warehouse work is all appified and can be configured in any language.
Any American working in an Amazon warehouse will be jumping at the chance to get out of there. So turnover would be really high. Same with delivery stuff, I don't think most Americans would enjoy a job where they don't even get a pee break.
As for legal stuff, I'm sure they'd use intermediaries to cover their ass.
Using intermediaries to hire illegally makes it worse from a liability perspective. Amazon is too big and hires too many people to get away with “oh, we used agencies in all of our markets and audited nine of them and we’re shocked that they didn’t do diligence”. All it would take is one disgruntled hiring manager. And Amazon doesn’t have many gruntled ones.
At the scale of Amazon, law enforcement is no longer bound to the law, it's bound to politics.
Your average local employer with a few dozen people? If he fucks around enough to draw in the attention of OSHA, he will find out. Easy prey and without political connections he's toast.
But Amazon? This company is by direct head count the third-largest employer in the US, second-largest in both the US and worldwide if one excludes militaries [1]. Amazon is frankly too large to enforce laws against and so is similar-sized Walmart (who has been able to extort the government into subsidizing their poor wages with food stamps), too many livelihoods depend on the existence of the company.
IMHO, a lot of the Big Tech and F500 companies should be outright broken up. When a company grows so large that laws cannot be enforced or, worse, laws get willfully ignored because it's cheaper to risk the occasional fine and bad press, eventually the rule of law itself suffers.
I still occasionally get them even though I literally was one of the people who left after they tried to make us go in office (I don't like to use "RTO" because no one on my team had actually worked out of an office for Amazon before since the project we were on was fairly new). My wife (at the time fiancee) has an autoimmune issue that makes it much safer for me not to commute, and although my manager suggested I could get an exemption, he didn't actually know what the process was because it all happened so quickly that no one seemed to have actually defined what that process was up front. I had a little less than a month to figure what to do and get that exemption before they expected me to either have that exemption, be in an office in another city three days a week, or transfer to a local team and be in the office in my current city three days a week. I ended to deciding that it wasn't worth the effort to try to figure out how to convince them to let me stay.
The claim is not that Amazon would be using illegal warehouse workers today, but that there is theoretically a pool of tens of millions of people available. Which is still kind of dubious.
Given their self-imposed labor supply issues, notably the awful working conditions, I hope the workforce there figures out how to effectively organize against this cesspool of a corporation.
Look, not to defend anything Amazon is doing, but this causal chain seems rather pareidolic and under-evidenced. You could spin some kind of crazy narrative about any major outage based on changes in policy that happened just before. But this isn't nearly the first AWS outage, and most of them happened before the recent RTO changes. It needs more evidence at best.
The article wasn't about the outage happening, it was about the amount of time it took to even discover what the problem was. Seems logical to assume that could be because there aren't many people left who know how all the systems connect.
It's the most plausible, fact-based guess, beating other competing theories.
Understaffing and absences would clearly lead to delayed incident response, but such an obvious negligence and breach of contract would have been avoided by a responsible cloud provider, ensuring supposedly adequate people on duty.
An exceptionally challenging problem is unlikely to be enough to cause so much fumbling because, regardless of the complex mistakes behind it, a DNS misunderstanding doesn't have a particularly large "surface area" for diagnostic purposes and it is supposed to be expeditely resolvable by standard means (ordering clients to switch to a good DNS server and immediately use it to obtain good addresses) that AWS should have in place.
AWS engineers being formerly competent but currently stupid, without organizational issues, might be explained by brain damage. "RTO" might have caused collective chronic poisoning, e.g. lead in drinking water, but I doubt Amazon is so cheap.
We've witnessed someone repeatedly shoot themselves in the foot a few months ago. It is indeed a guess that it may cause their current foot pain, but it is a rather safe one.
Indeed. No disrespect to Justin (great person) or any of the engineers who were sacked but Corey's post here is basically "here's someone who was sacked, and here are several other layoff news". AWS is really big organization. Several orders of magnitude bigger than people who were remote/refused to RTO. Organizations like this survive these brain brains.
This is the time to accept that the path forward is keeping people and giving them the best tools you possibly can to do their work. That is, the same as has been true for decades remains so.
Yes, development tools are better every day. Yes, you can downsize. No it won’t be felt immediately. Yes, it mortgages the future and at a painfully high interest rate.
Suspending disbelief won’t make downsizing work better.
Seems like it worked fine. They laid off a quarter of their junior principal engineers, the stock went up. They had a massive outage a few months later, the stock went up again. Everything's working out fine for their strategy so far.
I remember comments saying the stock went up because the average joe didn't realize how much of the internet was powered by AWS until all their day to day apps started failing. To most people Amazon is an online shopping site.
You would think this would eventually show up on the balance sheets, right? Presumably a lot of their big customers have SLAs with money penalties, so maybe next quarter earnings? Or quarter after that?
SLA monetary penalties won't make the difference there. Enough giant customers moving substantial workload off of AWS (either to another cloud, or otherwise) would, but the timeline for that is years, not next quarter.
Where are the young companies trying to replace them? There are all the AI companies, but Google and Meta both have competitive chatbots, and OpenAI is signing weird deals that don't make it look like a long-term player.
They all get bought out by Amazon, Google, Meta et al. The cash just tastes too good when stacked up against the prospect of grinding for 15 years and probably nothing coming of it.
I don't think there's ignorance of the fact that turnover is bad, I think the field is being designed to homogenize staff and favor uniform mediocrity so that employees truly do become interchangeable. We're so close to just plain talent being likened to cowboyism.
It was Diwali vacation in India. It looks like the managers were not able to force everyone to walk around with their laptops and pagers hanging from their necks and waists, respectively, which they normally do.
If there's one thing I have learned from my Amazon mates, then that is they never have a true time off. Hills, beaches, a marriage in the family— no exceptions. It's so pervasive that I can't really imagine it to be voluntary, and my friends' answers on this topic have never been clear.
I wish to understand the virtue of Amazon culture.
It seems that at L6 and below workers are a Taylorism-style fungible widget driven to convert salary into work product, guided to create the most output for the longest time before mentally breaking down, then being swiftly replaced, with L7 and above being so incredibly political that keeping the snakes and vultures from eating your team is a full time job at every level of senior management.
It never made sense to me how such a ruthless and inhumane culture is sustainable in the long run.
I would love to hear positive counter perspectives from Amazonians because the anecdotes from my L6-L10 friends describe what sounds like an inhumane hell on earth.
> It never made sense to me how such a ruthless and inhumane culture is sustainable in the long run.
It’s pretty simple, actually. Once such a dominant market position is achieved, you can get away with almost anything, whether with customers or employees. This is true of all the BigTech companies.
I think there's more to it. When you're dominant, you make money whatever. Think of Amazon et al. as huge spigots of money. Now, it becomes optimal to fight for more of that money coming your way. It's like the resource curse for countries. Nobody gains from growing the pie; they gain from stealing the pie. At some point, parasites and parasitic behaviours invade.
I think it comes down to demand and supply for jobs.
The only time Amazon was forced to change its ways was during Covid hiring boom where they couldn't compete in the talent market. They were forced to increases their salary bands and the culture was also a bit easy during that time. But starting mid 2022 it's been an employer's market and Amazon is making sure to juice every bit out of its employees while it can
It's not as conscious as that, its an emergent outcome of the snake pit.
Engineers have to spend an inordinate amount of time on "managing up", which means they have very little time and attention to do what would otherwise be a reasonable workload. Additionally, good engineers hate and despise this so it contributes a lot to the burnout.
It was certainly suspicious that actual progress on the outage seemed to start right around U.S. west coast start of day. Updates before that were largely generic "we're monitoring and mitigating" with nothing of substance.
[09:13 AM PDT] We have taken additional mitigation steps to aid the recovery of the underlying internal subsystem responsible for monitoring the health of our network load balancers and are now seeing connectivity and API recovery for AWS services. We have also identified and are applying next steps to mitigate throttling of new EC2 instance launches. We will provide an update by 10:00 AM PDT.
[08:43 AM PDT] We have narrowed down the source of the network connectivity issues that impacted AWS Services...
[08:04 AM PDT] We continue to investigate the root cause for the network connectivity issues...
[12:11 AM PDT] <declared outage>
They claim not to have known the root cause for ~8hr
Sure, that timeline looks bad when you leave out the 14 updates between 12:11am PDT and 8:04am PDT.
The initial cause appears to be a a bad DNS entry that they rolled back at 2:22am PDT. They started seeing recovery with services but as reports of EC2 failures kept rolling in they found a network issue with a load balancer that was causing the issue at 8:43am.
I don’t think that’s true, there was an initial Dynamo outage that was resolved in the wee hours that ultimately cascaded into the ec2 problem that lasted most of the day
Was the Dynamo outage separate? My take was the NLB issue was the root cause and Dynamo was a symptom which they flipped some internal switches to mitigate the impact to that dependency
If their internal NLB monitoring can delete the A record for dynamodb that seems like a weird dependency (like, i can imagine the nlb going missing entirely can cause it to clean up via some weird orchestration, but this didn't sound like that).
I noticed that too. I think tech culture has to change a bit. Silicon Valley is a great location if you're making hardware or prepackaged software. If you have to support a real economy that is mostly on the East Coast you need a presence there.
> one really gets the sense that it took them 75 minutes to go from "things are breaking" to "we've narrowed it down to a single service endpoint, but are still researching," which is something of a bitter pill to swallow
Is 75 minutes really considered that long of a time? I don't do my day-job in webdev, so maybe I'm just naive. But being able to diagnose the single service endpoint in 75 minutes seems pretty good to me. When I worked on firmware we frequently spent _weeks_ trying to diagnose what part of the firmware was broken.
> Is 75 minutes really considered that long of a time? [...] When I worked on firmware we frequently spent _weeks_ trying to diagnose what part of the firmware was broken.
One might spend weeks diagnosing a problem if the problem only happens 0.01% of the time, correlated with nothing, goes away when retried, and nobody can reproduce it in a test environment.
But 0.01%-and-it-goes-away-when-retried does not make a high priority incident. High priority incidents tend to be repeatable problems that weren't there an hour ago.
Generally a well designed, properly resourced business critical system will be simple enough and well enough monitored that problems can be diagnosed in a good deal less than 75 minutes - even if rolling out a full fix takes longer.
Of course, I don't know how common well designed, properly resourced business critical systems are.
For a service like AWS, 75 mins is going to result in a LOT of COE's for people on way it wasn't mitigated quicker. A Sev 1 like this has an SLA of 20 mins to mitigate impact. Writing about these failures will consume a dozen peoples time for the next 6 weeks.
I have 10 years of experience at Amazon as an L6/L7 SDM, across 4 teams (Games, logistics, Alexa, Prime video). I have also been on a team that caused a sev 1 in the past.
>Is 75 minutes really considered that long of a time?
From my experience in setting up and running support services, not really. It's actually pretty darn quick.
First, the issue is reported to level 1 support, which is bunch of juniors/drones on call, often offshore (depending on time of the day) who'll run through their scripts and having determined that it's not in there, escalate to level 2.
Level 2 would be more experienced developer/support tech, who's seen a thing or two and dealt with serious issues. It will take time to get them online as they're on call but not online at 3am EST, as they have to get their cup of joe, turn on the laptop etc. Would take them a bit to realize that the fecal matter made contact with the rotating blades and escalate to level 3.
Which involves setting up the bridge, waking up the decisions makers (in my case it was director and VP level), and finally waking up the guy who either a) wrote all this or b) is one of 5 or 6 people on the planet capable of understanding and troubleshooting the tangled mess.
I do realize that AWS support might be structured quite a bit differently, but still... 75 minutes is pretty good.
Edit: That is not to say that AWS doesn't have a problem with turnover. I'm well aware of their policies and tendency to get rid of people in 2/3 years, partially due to compensation structures where there's a significant bump in compensation - and vesting - once you reach that timeframe.
But in this particular case I don't think support should take much of a blame. The overall architecture on the other hand...
Sorry, are you saying you worked at Amazon and this is how they handle major outages? Just snooze and wait for a ticket to make its way up from end user support? No monitoring? No global time zone coverage?
Because if so, this seems like about the most damning thing I could learn from this incident.
I worked at AWS (EC2 specifically), and the comment is accurate.
Engineers own their alarms, which they set up themselves during working hours. An engineer on call carries a "pager" for a given system they own as part of a small team. If your own alert rules get tripped, you will be automatically paged regardless of time of day. There are a variety of mechanisms to prioritize and delay issues until business hours, and suppress alarms based on various conditions - e.g. the health of your own dependencies.
End user tickets can not page engineers but fellow internal teams can. Generally escalation and paging additional help in the event that one can not handle the situation is encouraged and many tenured/senior engineers are very keen to help, even at weird hours.
No, it's just mindless speculation from someone who clearly hasn't worked a critical service's on call rotation before. Not at all what it's actually like, all these services have automatic alarms that will start blaring and firing pagers, and once scope of impact is determined to be large escalations start happening extremely quickly paging anyone even possibly able to diagnose the issue. There's also crisis rotations staffed with high level ICs and incident managers who will join ASAP and start directing the situation, you don't need to wait for some director or VP.
Also it's pretty likely it took less time than that to get an idea, but generally for public updates you want to be very reserved, otherwise users get the wrong impressions.
Quite a few of AWS's more mature customers (including my company) were aware within 15 minutes of the incident that Dynamo was failing and hypothesized that it'd taken other services. Hopefully AWS engineers were at least fast.
75 minutes to make a decision about how to message that outage is not particularly slow though, and my guess is that this is where most of the latency actually came from.
Amazon is supposed to have the best infrastructure in the business because everyone else runs on it. They should have access to the sre talent that can quickly mitigate this kind of issue
AWS is still my overall favorite cloud provider, and I use it very effectively.
I would've even liked to work at AWS myself, if it were clear that they're solving a few concerns:
1. Rumors of rough corporate culture, and you needing your manager to shield you from it. (If it can't be immediately solved for all of Amazon or white-collar, maybe start with increasing job-seeker confidence for AWS or per-team.)
2. Even very experienced engineer candidates must go through some silly corporate coding screen, and an interview to make sure they've memorized some ritual STAR answers about Leadership Principles. If your prospective manager can't even get you out of that, what worse corporate things can't they shield you from?
3. RTO. As well as all the claims it wasn't done consistent with the Leadership Principles, and claims that it's not about working effectively.
4. Difficult-sounding on-call rotation, for people who aren't shift workers. (Even if you Principal out of on-call, you don't want your teammates to be overextended, nor to have awkwardness because you're getting a more consistent sleep schedule that is denied them.)
Also, not a concern, but an idea that applies to all the FAANGs lately: What about actively renewing the impression in the field that this is a place where people who are really good go? Meta's historical approach seems to be to pay better, and to release prominent open source code, and be involved in open hardware. Google (besides having a reputation for technical/competence excellence and warmer values) historically had a big frat-pledging mystique going on, though it turned into a ritual transaction, and everyone optimized for that ritual. AWS has a lot of technical/competence excellence to be proud of, and could make sure that they're investing in various facets of that, including attracting and retaining the best workers, and helping them be most effective, and then making sure the field knows that.
Whether it involves FAANG companies or not, a job is ultimately just a job. While it's nice to have such a company on a CV and to gain the experience, it is, in essence, similar to any other employment.
Eventually, you begin to consider the drawbacks, such as the monotony of the work or the exhausting nature of on-calls (which disrupt personal life). Then, an opportunity arises from a former colleague at another company, and the outcome is predictable.
Companies present numerous such inconveniences and actively introduce additional ones. Now, we are faced with mandatory RTOs, along with the continuous tightening of the screws and "cutting fat from the bone" (actual words of my company's CTO). Consequently, employees will depart, and it is often the high-performers who will seek opportunities elsewhere, as they are not afraid of the job market.
I think google is an excellent place to work, but the top AI companies do poach some of our good employees. If you're the cream of the crop, sure, take the Anthropic offer over G. But I do appreciate the massive amount of freedom to work on hard problems.
Google can be an excellent place to work, but most of the people working there are not contributing to the company’s profits. Whether or not you think that still has value, leadership is trying to “optimize” by squeezing everyone harder and demanding they show productivity gains using AI. Some teams will adapt and keep being a nice team to work on, but many employees are getting “optimized” out of the company and replaced with vendorization and consultants.
> When that tribal knowledge departs, you're left having to reinvent an awful lot of in-house expertise that didn't want to participate in your RTO games, or play Layoff Roulette yet again this cycle.
…
> This is a tipping point moment. Increasingly, it seems that the talent who understood the deep failure modes is gone. The new, leaner, presumably less expensive teams lack the institutional knowledge needed to, if not prevent these outages in the first place, significantly reduce the time to detection and recovery. Remember, there was a time when Amazon's "Frugality" leadership principle meant doing more with less, not doing everything with basically nothing. AWS's operational strength was built on redundant, experienced people, and when you cut to the bone, basic things start breaking.
Not just Amazon. I woke up this morning, to find my iCloud inbox stuffed with unread spam; much of it over a month old. Looks like Apple restored some old backup. This was likely to correct some issues that were caused by the AWS outage; either directly, or indirectly.
It’s nice to know that Apple backs up the torrents of spam that I get.
Everything is now at Jurassic-scale. It’s all monstrously big. There’s no such thing as a “small problem,” anymore.
One thing that you get with experience, is “tribal knowledge,” and that stuff is usually impossible to properly document. I suspect that AI may, in the future, be able to incorporate some of this, but it’s by no means certain.
Tech will learn like manufacturing folks did that experience is not fungible. You can try to replace someone, but the new guy also needs to accumulate the scars from the system for years before taking over.
You cannot just keep abstracting and chopping systems to smaller and smaller subsystems to make them easy to digest.
At some point someone needs to know how these coordinate and behave under disturbances. At some point someone needs to know at a low level what the hell is going on.
""It's always DNS" is a long-standing sysadmin saw, and with good reason: a disproportionate number of outages are at their heart DNS issues. And so today, as AWS is still repairing its downed cloud as this article goes to press, it becomes clear that the culprit is once again DNS. "
I use stored DNS data.^1 The data is collected periodically and stored permanently
I seem to be unaffected by DNS-based outages
I use stored data because it is faster, e.g., faster than using DNS caches like Google DNS, Cloudflare DNS, etc., but there are obviously other other benefits
1. When I make HTTP requests there is no corresponding remote DNS query. The IP address is stored in the memory of the localhost-bound forward proxy
For internal addressing, why do we use DNS? In case of an address swap (hardware swap, load balancing or whatever reason), one could broadcast it to the network, and the relevant clients would acknowledge & update the address.
This article seems sensationalized and lacking evidence. Layoffs alone (especially when so much of the industry did them) doesn't seem sufficient to explain today's outage, especially when we know so little of the technical details behind it. It's disappointing that The Register didn't wait until we had a postmortem from AWS before jumping to conclusions.
This AWS outage has reminded and bolstered my confidence of the idea that there really are practical limits on how we can manage complexity.
As a codebase ages, as services grow out in scale and scope, complexity increases. Developers know this. I don't believe that you can linearly scale your support to accommodate the resulting unknown unknowns that arise. I'm not even sure you can exponentially scale your support for it. There is going to be a minimum expected resolution time set by your complexity that you cannot go under.
I think times where there have been outages like this that have been resolved quickly are the exceptional cases, this is the norm we should expect.
Doesn’t an increasing LLM centric code base only compound this problem? Under the assumption that people are lazily screening LLM’s output when using them
We all watched this happen across FAANG, right? In the early/mid 2010s working at Amazon meant you were cream of the crop.
By 2020, no engineer in their right mind wanted to work there because it was an infamously bad employer for people who wanted to create great tech in a nerdy-fun environment.
The AI space is showing how the "darling fun tech company" to "aggressive tech employer full of psychopaths" trope can take less than a few years now!
cue in: programming as theory building [1] or building systems as theory building, ie, Mental causal models of how and why things work the way they do. Mental models live in people's heads and walk out of the door when they do. Management learns this the hard way [2].
"If you were a ‘product person’ at IBM or Xerox: so you make a better copier or better computer. So what? When you have a monopoly market-share, the company’s not any more successful. So the people who make the company more successful are the sales and marketing people, and they end up running the companies. And the ‘product people’ get run out of the decision-making forums.
The companies forget how to make great products. The product sensibility and product genius that brought them to this monopolistic position gets rotted out by people running these companies who have no conception of a good product vs. a bad product. They have no conception of the craftsmanship that’s required to take a good idea and turn it into a good product. And they really have no feeling in their hearts about wanting to help the costumers.”
Great words, but he lost any right to them when he made famous the "You're holding it wrong" workaround. IMO that was the defining moment when Apple started its decline on product innovation.
If it were that big of deal, don’t you think Apple would have been forced to recall it and definitely couldn’t keep selling it for 3 years. True they did redo the antenna for the Verizon CDMA iPhone 4. But they never bothered to back port the changes to the GSM one.
I also had a GSM iPhone 4.
Compare that to how quickly they ran away from the shitty Intel modems when they were selling some made by Intel and some made by Samsung (?)
I’d add a corollary to Steve’s often quoted idea, that became even more relevant after Covid. Everyone competent that makes tons of money retired early. We are left with the dregs at all these companies, the newbies and those that didn’t budget well and plan for early retirement.
Any interaction you have with a company post-Covid you can feel it. Nothing works anymore and you can’t even tell anyone about it or why.
It was going to happen Covid or not. People get old...what are you going to do? Is it really that bad if a bunch decide, "you know what? I'll take the financial hit and retire a few years early because I have been reminded of the fragility of life".
I think they might have deeper issues still with their outage. I just got an email and retroactive charge for something I returned months ago and shows as returned on their own orders portal. The link in their transactional email also links to a totally different product.
That could be totally unrelated - I had the same thing happen last month. Eventually (~3 attempts over a week) managed to fight through the various chat options that tried to dead-end into "wait 3-5 business days and check again", and once I had a person on they took about 5 minutes reading the background and said "oh, I can see that this return was processed months ago, and we have it listed as received. I will return the amount you were charged immediately!"
One of the best-written articles I've read in a long time. I wish general news coverage had this tight blend of fact, context, and long-term perspective.
Years ago I interviewed for Amazon. I found out that on a desired team size of 8, there were only 3 left, where the other 5 who were no longer there, as I was told, were poor performers. I didn't take the job.
Here in Europe they need a very serious reason anyway (like gross misconduct) and if they don't have it they can make you redundant but have to pay significant severance. I'm kinda waiting for an opportunity for that at my current employer, due to leadership changes it's no longer a great place to work. When there's layoffs there's usually voluntary options with a decent severance plan.
Which is generally used as a means to collect evidence that they're unable to do the job before firing them (and in many cases the decision to fire them has already been made).
internal reports from current AWS engineers seem to be confirming all of the speculation in this article. Shit's rotten from the inside out and you can pretty evenly blame AI, brain drain, and good old fashioned "big company politics"
There's been a massive talent exodus, especially among the principal and senior principal engineering roles, across all Amazon orgs since the RTO policies have been enforced. Its demoralizing to lose key engineers that you look up to and want to continue to learn from all because a few people far removed from the day to day make a bad call.
RTO in combination with Amazon being last place in AI innovation have led to departures of anyone that can leave, leaving.
> So they can't suggest a fix even if they know 100% what it will be. Thats exactly what happened this time. EIGHT different staff members pointed to the underlying cause and were told (some literally) to "shut the f*ck up and get back to your job"
Jesus, if even an ounce of that is true... Yes, everyone on the internet is a cat clawing on a keyboard... but if a ton of people legitimately confirmed to be ex-AWS point to similar culture issues... probably it's AWS that's rotting.
With cbell gone and ajassy promoted and the misery inducing litany of morale self inflicted wounds it’s not surprising reliability is regressing. There’s no head of engineering like Charlie and Garman is a strong engineering leader, but coming after a sales guy and taking over a battered workforce, it’s not clear to me he can turn things around easily. Everyone I know worth a hill of salt left aws already - and the ones left - meh. That’s how attrition through misery works.
When organizations begin to prioritize personal-brand builders and performative hires over the core technologists and long-tenured institutional experts who actually understand how things work, the culture inevitably shifts.
When that imbalance grows, as it has at AWS (ex-AWS here), and the volume of relentless self-promoting “LinkedIn personalities” and box-ticking DEI appointments starts to outnumber the true builders and stewards of institutional memory, the execution quality, accountability, and technical excellence begin to erode.
It is becoming increasingly clear that Andy Jassy’s leadership is no longer effective, and it is only a matter of time before Wall Street begins calling for his departure.
I appreciate how you, with absolutely no evidence, blamed this outage on DEI, especially given Amazons complete backpedaling of DEI the first moment it became political convenient
Ehh I trust the reporting and generally agree that RTO was/is executed hamfisted but I dunno if this particular incident "makes" the narrative. IIRC LSE rate has been increasing for many years, maybe most of AWS's existence. This is part and parcel of building something so complex that continues to grow and evolve.
I do expect much better of them and they certainly have problems to solve but this is a big company evolution thing and not an Amazon-specific thing imo.
The real story from this incident is that Amazon’s “aws” partition doesn’t actually have multiple regions - effectively, it’s all IAD in a trench-coat.
This is a big deal. My employer has already started to look at bringing back our old racks from storage and switching back to on-premises. Cannot imagine he’s alone in that.
Can you elaborate what convinced you of this? We were running mostly in us-west and saw almost no impact, despite using a broad spectrum of AWS infrastructure and tooling.
Normally, you'd have drills and exercises to try to find and fix potential issues before they affect production. Saying that you can learn from failure is great comfort to the individual, but not an excuse for the institution.
Given today is Diwali, perhaps the reason everything went down is because the best and brightest from India were all on vacation and weren't there to babysit/roll back the deployment that broke everything?
We’ve reached a point where I can no longer distinguish between people without experience and people repeating the talking points they’re told to repeat. That’s a major loss.
However, talent is a very small part of shipping a project. How that talent is resourced is far more important.
This is how articles should be written, this is why I’m reading El Reg (a.k.a. The Register) all these decades, this is what happens when high management cares only about profits and when real engineers don’t eat the RTO bullshit. Bravo for putting this online.
P.S. I’m not an Amazon hater, replace the company name with any other big one of your choice and the article will have the same meaning ;-)
The Register is an opinionated tech tabloid filled with outrage bait. This article is not an exception, drawing far reaching conclusions from little evidence.
“Little evidence?” If the “aws” partition doesn’t actually exist when IAD breaks, Amazon hasn’t even discovered how to make multi-region cloud infrastructure. That’s a big deal.
It's not your feeling about Amazon that would cast doubt on your take, it's that you've reduced it to one pet cause and decided a source is well written because it appeals to your dislike of RTO. Nowhere has any evidence of the relevance of that to this been presented, Amazon has had outages since before WFH even began, they've all always had their occasional outages and bad days.
Terrible article. Im ex-AWS, left as a principal after 10 years to go take another global megacorps shilling. I dont even disagree with the premise, but its so clearly a predetermined conclusion written as opinion piece to fit the hot news topic.
Ex a sloppy as hell and inconsistent premise.
> engineers had identified DNS resolution of the DynamoDB API endpoint for US-EAST-1 as the likely root cause
its the point that wasnt the root cause. The root cause was ipso facto much more complex, insidious, and unobservably spooky action at a distance. I say that not knowing the true cause but being very willing to bet a bunch of AMZN that it wasnt as simple as “herp derp dns is hard and college hire sdes dont understand octets and delegation.”
Or this stupid citation if were talking about senior/long term AWS tech roles:
> Internal documents reportedly say that Amazon suffers from 69 percent to 81 percent regretted attrition across all employment levels.
The citation _appears_ to be about consumer/retail delivery and ops folks. And how 69-80% _of total attrition is RA_. While el reg has written it trying to imply 80% _annual attrition_ in a completely different org and business unit.
So I know corey isnt stupid, and hot takes are his paycheck. But does he think his readers are stupid?
As ex-amazonian, you should know that our average tenure for an SDE is just barely over 8 months. The old farts tool should have told you that during your day
And whats the median? Talking about hot take tenure averages without expressing the components is just silly. I, personally, was well aware of actual hiring and attrit rates up and until fairly recently.
Again, Im not saying things are good (I left for reasons!). But use better data and arguments.
This fails to recognize that the people who designed everything to rely on us east 1 did so a long time ago. "Brain drain" could just mean that they've had their fun and now want other people to deal with their mess.
>I've seen zero signs that this stems from a lack of transparency, and every indication that they legitimately did not know what was breaking for a patently absurd length of time.
That information is under NDA, so it's only natural you aren't privy to it.
> At the end of 2023, Justin Garrison left AWS and roasted them on his way out the door. He stated that AWS had seen an increase in Large Scale Events (or LSEs), and predicted significant outages in 2024. It would seem that he discounted the power of inertia
Your comment is relying on that referenced inertia. Things will continue to function for a period of time, but there exists an inflection point at which they no longer function as previously.
The year or so after Musk took over was brutal. The influx of far-right pests, troll farms and porn bots was one thing, but the reliability went down the drain.
Musk effectively left his newest plaything a few months after the takeover and some events like him going in in a datacenter and disconnecting servers, that was when Twitter (I'll call it X when he acknowledges his daughter) started to stabilize again.
All that he seems to be doing these days at Twitter is messing around with the recommendation algorithm, override the decisions of what's left of moderation for his far-right friends and that's it. Oh and of course Grok/xAI or however it's called these days, but IIRC that's a separate corporate entity that just got shoehorned onto Twitter.
Speaking of DNS, I still cannot comprehend why we still rely on the current complex, aging, centralized, rent-seeking DNS.
It's one one of the few parts of the internet which could potentially be replaced over time with very little disruption.
The hierarchy of resolvers could be replaced with a far simpler flat hierarchy Blockchain where people could buy and permanently own their domains directly on-chain... No recurring fees. People could host websites on the Blockchain from beyond the grave... This is kind of a dream of mine. Not possible to achieve in our current system.
I think this would make the squatting problem that we already have way worse. There would be bots buying every single remotely usable domain, and there would be no incentive for them to sell it unless they get an absurdly large offer.
I bought tombert.com in 2014 and forgot to renew it in 2015, and it was auctioned off by GoDaddy. For like six years, it was owned by squatters, and they wanted thousands of dollars for the domain [1]. I called offering the $100 for it, and they claimed that they can't go below $1400 because this domain is in "extremely high demand". I finally was able to buy it back in 2021, presumably because the squatter purged out domains that hadn't been purchased for N years and they wanted to save money.
Now, you could argue "see! You wouldn't have had to worry about it expiring if it were permanent on the blockchain", and that's true, but if someone else had gotten to that domain first, then I would also never get it. I think the only thing that keeps the internet even remotely fair in this regard is that domain names cost some amount of money to keep.
Yes, having some cost disincentivies some abuses that completely free attracts. If email cost a few cents to send there would probably be a lot less spam around
I'm pretty sure that is why Something Awful was successful. Since an account cost $10, and any abuse could lead to a ban, you very quickly filter out spam and people who are solely there to shitpost.
> People could host websites on the Blockchain from beyond the grave...
This is precisely why something like this isn't a popular solution lots of people are working towards. Domains broadly speaking aren't a finite resource, but usable domains using common words definitely are. As time marches on human readable/typeable "permanent identifiers" are going to have to go away. Email address, usernames and the like are all going to get recycled, just like phone numbers are. Domains are currently recycled and most people probably think that's a good thing (assuming they think about it at all)
> People could host websites on the Blockchain from beyond the grave...
This is precisely why something like this isn't a popular solution lots of people are working towards. Domains broadly speaking aren't a finite resource, but usable domains using common words definitely are. As time marches on human readable/typeable "permanent identifiers" are going to have to go away. Email address, usernames and the like are all going to get recycled, just like phone numbers are. Domains are currently recycled and most people probably think that's a good thing.
I'm not really interested in communism but I will support it if it lets me have a Blockchain-based DNS system.
All the arguments I'm hearing against a Blockchain DNS system are rooted in petty crony-capitalist thinking.
This kind of thinking seems to permeate most other parts of society... It's gotta stop.
"Oh but what if someone steals it"
This ain't gonna be much of a problem in a functioning society where the top 20 domain names doesn't hoard like 95% of the traffic.
"Oh but we don't want people to own domains permanently or else they will take all the good domains"
Um hello?? Have to checked this thing called reality? It's already the case! So happy billionaires have to pay their $20 per months to maintain their market monopolies.
I actually don't mind other people having more stuff than me but tired of petty people ruining good ideas and stalling progress to make a few bucks.
Garbage reporting:
1. AWS had an outage
2. AWS has lost a lot of employees
Conclusion:
The brain drain lead to the outage...
I need an LLM trained explicitly on folks confusing correlation and causation and put a big old red dot in my address bar.
I love that there's a whole section "The talent drain evidence" trying to defend their journalistic integrity, but they then go on to totally face plant.
> It is a fact that there have been 27,000+ Amazonians impacted by layoffs between 2022 and 2024, continuing into 2025. It's hard to know how many of these were AWS versus other parts of its Amazon parent, because the company is notoriously tight-lipped about staffing issues.
Internal documents reportedly say that Amazon suffers from 69 percent to 81 percent regretted attrition across all employment levels. In other words, "people quitting who we wish didn't."
The internet is full of anecdata of senior Amazonians lamenting the hamfisted approach of their Return to Office initiative; experts have weighed in citing similar concerns.
So the title is all speculation. The author put 2 and 2 together and concluded that 10 is greater than 9.
AWS doesn’t even have a “devops team” nor even any devops job roles. AWS also does not use Terraform (which is what the article says everyone was replaced with) at any significant scale, so this article is similar junk.
This one mentions terraform by name (though that doesn't necessarily imply its in use, though having worked in large companies I would argue that sweeping statements about a popular technology not being used is likely to be wrong)
AWS does not have dedicated devops roles. All AWS SWEs are expected to take oncall shifts and respond to incidents, manage build pipelines, etc rather than having specific devops people to do it for them. The article you linked claiming 40% of them were fired is total junk. You can believe that or not, I don’t care.
The last one is a ProServe role, which is a consulting role that spends their time working in customer environments, which is where they may encounter terraform. It does not mean anything about internal use of terraform.
Again, I’d be wary making sweeping generalisations like that.
I already showed you that AWS has (or hires) DevOps people with publicly available information, maybe the article is incorrect but you’re clearly not better informed, so maybe cut it with the rude commentary.
Within AWS this role falls under the Systems Engineer job family. It is not a devops role, and its involvement in events like today would be the same involvement as every other SWE at Amazon.
Just do a quick google search for that “40% of devops laid off” and you’ll see that it’s actually an old article from months ago that multiple people, including AWS employees, are saying is bullshit and unsourced.
edit: found another source that says this 40% number came from an AWS consultant that worked with customers to help them be better at DevOps, and it was 40% of their specific team that was laid off. Even if it were true, it has nothing to do with the internal operations of AWS services. This is why it’s important to understand the information you’re sharing before making judgements off of it.
Seems wild that you would promote job titles you don’t hire for, makes me think that it’s reasonable for news outlets to refer to those roles in the same way honestly.
It makes a direct claim of hundreds of cloud staff being laid off.
You know what though? I’m not wasting my time with you, the fact was that this was all over social media. Then a huge outage- my original comment was factually accurate even if we contend that the article itself was bunk. And AWS clearly hires DevOps staff.
You’ve not even disproved anything you’re just making me play internet fetch. I’m not replying anymore.
> It makes a direct claim of hundreds of cloud staff being laid off.
I don't have any dog in this fight, but I don't see where this article makes your case. From your article:
> We understand around 100 jobs are at stake.
> Sources familiar with AWS operations who requested anonymity told The Register most of the layoffs affected people in marketing and outreach roles, although chatter on sites like Blind suggests folks in frontline support and in other positions may have been affected, too.
You seem to be kind of annoyed that somebody on the internet hasn’t taken your assertion that you just sort of generally Know Better as strongly as you’d like. You could probably put this entire discussion to bed by clarifying your current position at AWS and how your job there gives you direct knowledge of their devops practices.
I work for Amazon (AWS for 4 years then “the website” side of the house for the last 3)
The previous commenter is correct, there is no NOC or devops team and I’ve not encountered a Devops job family and I’ve never seen terraform internally.
Within AWS, the service teams that work these outages are the same ones that design the service, fix bugs, deploy the pipelines, be oncall, etc. the roles that fill these teams are pretty much one of three types: nde, sde, sysde. They typically use cdk if they’re doing AWS things, else they’ll use internal tooling.
The job you posted is a customer facing consultant like role - customers use terraform so having a customer facing consultant type that knows how customer-y things work is a good decision.
You could both be right if they are trying to expand terraform use from a beachhead to the entire company. You need to hire people with prior experience for such things.
Source: Former AWS Professional Services employee.
Notice the job description:
As part of the AWS Managed Operations team, you will play a pivotal role in building and leading operations and development teams dedicated to delivering high-availability AWS services, including EC2, S3, Dynamo, Lambda, and Bedrock, exclusively for EU customers.
They aren’t looking for DevOpe engineers to work alongside the “service teams” - the teams that build and support internal AWS services. They are working with AWS customers who may already be using Terraform. AWS has a large internal consulting division staffed with full time employees. When they work with customers they will use Terraform if needed.
i mean, you can assume if its on theregister its not going to have some kind of academic rigor or whatever it might be you're looking for. its the register, same basic rigor quality as the ny post.
that said, my suspicion is they're likely on to something here regarding layoffs and quality degradation.
You could make smart inferences based on past and very frequent occurrences.
Or you could just say "there is no way the thing that constantly happens over and over again has happened once again, just no way".
Staff cuts constantly happen in the name of maximising profits. They always yield poor results for a company's performance. Every time. Especially for the consumer's side of it (not the company's finances of course).
Every time.
But maybe this time it's different. That one time.
I worked at AWS and still have friends who work there. I don’t know any L5s who wouldn’t jump at a chance to leave if they even got a slightly worse offer than what they are making now. I know a few L6s and L7s that would stick around out of momentum.
But I know very few people in the industry who know about Amazon’s reputation that have a life long dream of working there given a choice.
I was 46 when I was hired there for a “permanently remote [sic] field by design role” in ProServe and it was my 8th job out of college. I went in with my eyes wide open. I had a plan, stay for four years, sell my RSUs as soon as they vested, pay off debt, save some money, put it on my resume to open doors and make connections and leave.
I was never expecting to make more when I left. I used the time to downsize and reduce my expenses - including moving to state tax free Florida.
When I saw the writing on the wall, I played the game while I was on focus to get my next vest and wait for the “get 40k+ severance and leave immediately or try to work through the PIP”.
I took the latter and had three offers within 3 weeks. This was late 2023.
Close enough. I missed 2 vesting periods. But the severance and rapidly having a job made up for one and I got refreshers my third year that I hadn’t counted on.
I left debt free, sold my old home for exactly twice what I had built for 8 years earlier, downsized to a condo half the price I sold it for (and 1/3 the size) and I was debt free with savings.
I’m now a staff consultant working full time at a 3rd party AWS consulting firm with a lot less stress and still remote. They were the last to fall. But AWS made their ProServe department return to office at the beginning of this year.
Even with candidate pools of hundreds of thousands of H1-B engineers and tens of millions of illegal immigrant warehouse workers, there still comes a point where such a big company firing so many people so quickly exhausts all their options.
It reminds me of the Robot Chicken Sketch where Imperial Officers aboard the Death Star all pretend to be force choked to death by Darth Vader so they can avoid getting killed by lightsaber, then come back in under different names in different jobs. It's worse though for Amazon: nobody wants to come back.
https://www.youtube.com/watch?v=fFihTRIxCkg
> "The rate at which Amazon has burned through the American working-age populace led to another piece of internal research, obtained this summer by Recode, which cautioned that the company might “deplete the available labor supply in the US” in certain metro regions within a few years."
The root cause is global competition, especially from countries with very low wages - Ukraine, a country in Europe, for example, was a top food supplier for Africa -, and widespread income disparity in many Western countries - 67% of Americans self-report to live from paycheck to paycheck [1], for example. First it's "luxuries" that get the cut - travel, eating out, entertainment - and once everything has been cut, people go for savings in food because that's the last large expenses block that they can meaningfully control.
[1] https://www.investopedia.com/living-paycheck-to-paycheck-you...
[2] https://www.boeckler.de/de/boeckler-impuls-vermoegen-nur-jed...
I asked if I could schedule the interview after my final exams, and his arrogance really showed when not only did he refuse, but then insisted my exams are not don't even register on the same scale of importance as the opportunity to work for Amazon.
Somewhat related: a recruiter at Google cold-called me a couple months into my first job out of undergrad back in 2016 and was similarly condescending about "the chance" to work for Google compared to everything else. I already had a low opinion of them when they gave my then-girlfriend an introductory O'Reilly book on Java after she failed their interview.
I regret being born too late to work somewhere like Bell Labs, SGI, or Sun. I had a ton of graybeard wizard coworkers from these places, and they were all a pleasure to learn from and even better friends. For the first 2 years of my first job, every day of work was like walking into the Shire and talking magic spells with 20 Gandalfs.
That job was great until I got put on a team with a guy who was a former middle manager at some IBM-like company and went from being surrounded by people lightyears ahead of me to being surrounded by Dilbert characters. The messed-up part was that it wasn't even punishment. I was rewarded after completing a project with my choice of which team I joined next, and I joined the wrong one. I assumed that joining a new team to utilize this newfangled "cloud computing" thing would be trailblazing, and I didn't do any diligence on who I would work with.
To this day, I still regret not rejoining the first team I worked for, otherwise I would still be at that company and happy about it. Then again, the boredom and discontent while being on that sucky team is the reason I started investing, and now I can buy a house in cash and fund myself to do whatever I want for at least a decade. Hard to complain about the way things turned out.
For each of those firms there was a 'golden era' and then a time when the company coasted on their laurels, and then the slide to irrelevancy.
The question that arises is: How can you potentially spot which companies are about/likely to enter a 'golden era' when you interview there? What questions could surface some sort of likelihood? Is it possibly to identify them before they enter the 'golden era'?
You have said that the driver worked because he had (enough money?) and he might have wanted to relax with the driving job but still, its an amazing story.
Point being entshittification comes at a cost, and companies partaking in shitty activities can only keep this up for so long.
I'm not even out of college, and I feel the same way. Especially for Sun, everything they did was so cool. "The network is the computer" and all that.
Unfortunately, while we are well aware of cool tech companies that were ran aground by the finance/sales/management consulting types, Sun felt like a company ran aground by engineers.
Zuck famously kept the Sun logo up for quite a while when Facebook bought Sun's HQ campus, as a warning to the employees of what they could become. In some ways, Facebook/Meta is the spiritual successor of Sun, just like Google became the spiritual successor of SGI when they bought the SGI campus.
But these two ad driven companies never quite became the new Sun/SGI, for better and worse.
[0] https://news.ycombinator.com/item?id=2287033
[1] https://bcantrill.dtrace.org/2019/12/02/the-soul-of-a-new-co...
I'm not sure what Sun could realistically have done to come out the other side of the dot-com carnage. Other companies in roughly equivalent situations come to mind. You start looking at doing a hard reboot when the margins for that reboot aren't there and it's difficult to see the light at the end of the tunnel. Maybe an earlier reinvention involving more open source and alignment with where hardware was headed. Don't know.
Sun did waste a lot of money in buying MySQL, $800 million in cash and $200 million in stock. Certainly a distraction, as well.
Sun never offered any way to inexpensively get onto the on-ramp of Sun hardware and software as they thought they could continue selling high-margin hardware forever; they had their $995 V100 which even included their much-loved LOM which was a remote-management device like iLO/DRAC/IPMI , then followed it up with: nothing.
info about the V100: https://dogemicrosystems.ca/pub/Sun/System_Handbook/Sun_sysh...
I surely would have liked to get one of those laptops, though.
Sure, Solaris was rock solid, but it was also pretty conservative in its march forward as a Unix, being ultimately trumped by Linux.
Sun had an amazing team of people that worked on Self project led by David Ungar and others (Lars Bak who helped give us V8). They let the whole team go, who then went off and did sime cool things with dynamic optimization, which Sun ultimately ended up hiring/buying back to create the HotSpot VM.
Any NIH and other dysfunctionality went far beyond the engineers at Sun.
Technically true-ish, but deserves an important qualifier. The Javascript number format has a huge "safe space" of integers between
Also, the number format is a standard, not only used by JS, and given that it was supposed to be a minimal scripting language it is hard to argue against the initial design choice of choosing one all-encompassing big standard, and not burden the language with a complete set. Since he criticism was on the initial design:> ultimately to be replaced by a guy who threw together an integerless programming language
I would like to refute it by pointing out that the criticism ignores the initial use case, as well as the actual existence of integers within that larger number format standard. Later, when enough people (and companies) demanded it, a big integer type was added, after all.
Internally runtimes use different paths depending on what kind of number it is.
For many use cases of integers, especially internal ones, like array indexing and counting, those integers are just that, and an extra integer type for extra purity is not much of a problem. For other uses of integers, e.g. finance (using cents instead of dollars), it sucks that you have to pay a lot of attention to what calculations you perform, so not having (had - until BIGINT) a real integer type as aid indeed made it less pleasant to do integer arithmetic.
Mocha -> LiveScript -> JavaScript -> EczemaScript or whatever
[0] https://en.wikipedia.org/wiki/ECMAScript
No, not at all. It became a marketing campaing in the very late 90s dot.com boom, but the concept that defined Sun goes back to the beginning, 1984. Back then, that was a radical vision and Sun truly lived it internally for a long time.
https://en.wikipedia.org/wiki/The_Network_is_the_Computer
Yeah, man, good times.
My buddy got a visit from the feds and lost his computer lab access for a semester.
I still giggle when I tell that story.
Doesn't mean it might not have been a blast but not hacking on software and playing in the open source world as is the case at at least some companies today.
I don't know; I'm not young enough to remember.
When I was in the minicomputer business, it was maybe 50/50 hardware and software (and that mostly assumes you considered software to include low-level things like microcode). And software people weren't mostly paid more than those in hardware--which is to say generally a good middle class professional wage.
Working with Suns and other workstations as a teen (so my perspective was limited), I caught the very tail end of software as a modest middle class professional wage for everyone doing it (right before the dotcom boom hit).
The people I worked with were really good at what they do, but not strutting like newcomers started doing pretty much the instant the dotcom boom started, and not rich. (Well, one guy did buy a used MR2, and get his private pilot license, but he also lived with his wife in a trailer on an undeveloped parcel. He was a very solid software engineer, working on important stuff.)
I might have inadvertently tried to preserve some of that modestly-paid excellence of the generation before me, but I don't recommend that. Cost-of-living in my area is determined by people making FAANG-like money (well, and real estate investors, and price-fixing), and you have to either play along with that, or move away.
But a somewhat high-flying (albeit hardware) company was recruiting me for a CA job and they basically admitted it would be a lifestyle downgrade in terms of salary.
Some people made a lot of money when dot-com hit. A lot also got wiped out and ended up leaving the industry.
I never had the highs or lows. I was probably making something south of $100K in the late 90s.
I've been given second hand accounts of similar situations. One was team consolidation, and the business was offering Boston-area engineers positions in San Jose. One of the folks who moved with his family was back in MA within 5 years. His salary was not adjusted as much as it should have been for the cost of living difference.
A question, though. Was software designed at Sun closely with hardware teams and vice versa, or were they mostly disjoint? Presumably many hardware companies that have succeeded have built good tooling around said hardware (like NVIDIA).
The low-level software work at Data General where I was prior to the analyst biz was certainly integrated to a certain degree--read Soul of a New Machine if you haven't. The software folks for the minis were also mostly in the same same location. As things migrated to Unix, most of that team was in RTP and it's probably fair to say that there was less integration though probably wasn't something I thought about a lot of the time. Hardware stayed in Massachusetts.
The third time a Google recruiter reached out to me with the sales pitch that I was a great engineer that they would like to have at any price, I berated him if that was the case why the previous two experiences.
Never heard from Google HR ever again, and I am not sorry, I am happier this way.
I had better experiences in interviews for EA and SCEE than Google, which again I also am an happier person not managing to get an offer, and endure the crunch lifecycle of the industry as reward.
As a real life Wally I appreciate this comment.
I was Wally for the last 2 1/2 years of that previous job until I started to realize I'm becoming more and more like a Dilbert character myself. Something in my brain just told me it wasn't sustainable, call it fear of God or paranoia, but letting my skills atrophy in a place like that for 20 years didn't seem like it would end well for me.
The only problem was that I stayed so long, and it made me hate software engineering so much that I didn't even want to be a software engineer anymore.
I put up with it just long enough so I could avoid selling stock and drawing cash out of my portfolio, and now I'm back at square one as a post-bacc student getting my applications in order for MD and PhD programs where I'll most certainly wind up drawing hundreds of thousands out of my portfolio to pay rent and eat dinner for about a decade.
It's sad, I really enjoyed systems programming, but it seems like finding interesting systems programming and distributed computing projects that have significant economic value is like squeezing blood out of a stone. Maybe LLMs or future progress in bioinformatics will change that, now that finding ways to shovel a lot of data into and out of GPUs is valuable, but I'm so far into physiology, genetics/proteomics, and cell biology that I'm not sure I would even want to go back.
Ha, my interview for an Amazon internship was an hour after a 3-hour final exam :-)
But the job market right now is quite bad, and after hundreds upon hundreds of internship applications I would've been stupid to give up this chance. I would work for Amazon in a heart beat.
Your interview lunch experience sounds like message: this is what it's going to be like, and we don't care if you join us.
Memorize Amazon’s insane company values and relate your resume experience to it. And that I mean every single bullet point.
Interviewers were all run by robotic people. Coding test had zero flexibility, you had to just write code in a special barebones text editor that had zero feedback besides pass/fail.
You’d have to solely care about Amazon RSUs to consider that job. They are self-selecting for the worst kinds of candidates.
The dumb thing is that it should be a job that doesn’t burn people out because they basically own the market and haven’t needed to do any sort of innovation. Amazon’s corporate culture just has a burnout fetish.
I got the job, and I think being natural helped. I've interviewed thousands of people at Amazon since, and too many people just say the buzz words with no meat, and it gets them nowhere i.e. I showed customer obsession when I.....(and then gives a bad example)
Although I doubt it was really their intention to be passive aggressive, I have to say: That was a Flawless Victory of passive aggression.
I'm sorry to be a buzzkill, but I just can't get excited about privately-funded space rockets or Japanese girlfriend robots, not even if there were an 8 figure stock compensation package in it for me. To me, it's all just "do something grandiose for the venture capitalist bucks, and then maybe figure out how it helps people in 20 years."
"My sister killed her baby because she couldn't afford it and we're sending people to the moon.
September, my cousin tried reefer for the very first time. Now he's doing horse. It's June.
Is it silly, no? When a rocket ship explodes And everyone still wants to fly
Some say a man ain't happy truly Til he truly dies
Sign o the times"
-Prince
I worked at Amazon. Twice. In total about a decade as a Principal Engineer. I left voluntarily a few months ago.
I have zero regrets about my time at Amazon. I learned lots, worked with some incredible people, and had fun doing it.
And the culture? It was life changing for me, especially when I first joined. In all the best ways.
And Amazon today? All I’ll say is that at their size, maintaining solid culture is damn hard. The hiring spree peri-Covid definitely added unimaginable stress to maintaining the culture the company was built on.
They’re a big company, and thus a big target. It’s easy, cheap, and even lazy to kick them with stuff like this.
The truth is that while it’s changed a lot over time, anyone fortunate enough to work there should embrace it.
Before you reflexively jump to "Amazon is just not for everybody", it's only fair to warn you that a lot of people around here have heard this phrase and don't buy it. My former landlord said the same thing at least 9 times in the 8 times I've interacted with him face to face... right up until he got fired. Management conveniently put him on a PIP 4 months before a vesting cliff on his 2-year RSU vesting schedule.
Yeah, Amazon is not for everybody- just the kind of people who either perpetrate or fall victim to pyramid schemes.
To pre-empt hyperbolic responses: I live in Seattle. I personally know plenty of people who have worked at Amazon. I know plenty of local teachers. The teachers work as many hours for a tenth the pay and burn out just as fast.
I don't consider 60 hour work weeks a privilege to be coveted, especially if I'm capturing less than 1% of the value I produce. I'm sorry man, but white collar serfdom is still serfdom.
If you're really going to keep begging this hard to be a serf on someone else's fiefdom, I'll tell you what: you can keep overworking yourself as serf on fiefdoms in which I might own shares, and increase the value of my portfolio for me, so I can draw even more passive income every month. I formally withdraw my protest to your "Protestant" work ethic.
I'm not trying to be an asshole, but I'm just hoping you take what I'm saying burns into your subconscious and something in you changes: that work into which you've placed so much of your self-esteem is just funding other peoples' passive income to either not work or live ostentatiously. If that's a great privilege to you, God bless you. I'm sure my wife would love it if I just retired- please make it happen, she would love me even more for it.
EDIT: P.S. Your "privilege" of working for Jeff Bezos' portfolio sounds a lot like a biblical curse. Specifically, 1 Samuel 8:10-18 and Psalm 105:44.
"He gave them the lands of the nations, that they might inherit the fruit of others’ labor"
You don't have the first idea what teachers do or what their jobs require.
If you really think all teachers have tenure and are supported by unions, you also have not the faintest, foggiest inkling of a clue about the job market or professional environment of most teaching in the US.
I think it's also worth pointing out that, even as you repeatedly label others "arrogant," your comments in this thread are themselves breathtaking in their arrogance. I rarely run across HN comments so condescending, dismissive, self-righteous, or self-congratulatory.
No, and no. Your perception of modern labor markets is dated and disconnected from reality. Schools are constantly facing budget cuts and a huge chunk of teachers aren't even full time (instead getting things like a 60% time position - less pay, no security, but enough hours that good luck working a second job). Those that get full time positions still face the elimination of their positions.
If I'm wrong, I'm wrong. However, there's a fix for that: move to a state where the job prospects are better. That sounds like "let them eat cake", except that moving to another state as an American citizen is a lot easier than what of people went through fleeing the Sicilian Mafia or the Irish Potato Famine, let alone WW2 European front or the fall of Saigon.
But that company culture leaves me with a very low opinion of them and very little trust. Even Microsoft engineers are less brainwashed. I've had several that just told me the truth about services.
Maybe it depends on the country but it feels like this is just their culture.
There's a lot wrong with AWS (and it's got a lot worse in the last 3 years), but there's also a lot right, and there are some really, really smart people there, several of which have boomeranged (people who left and came back).
However, I think the question is, what percentage of engineers can pass the amazon interview but not the Apple/Databricks/Uber/Google/Meta ones. Because no one is picking amazon over the aforementioned companies.
However, maybe there's an opening at Amazon and not the other companies, or maybe that's your only offer. I certainly think it might be worth it for a a few years.
Lol
First worlders having to "just get by" with 11 thousand dollars every month. What even is this world
FAANG has a huge footprint in the town, you're going to have a hard time hiring people when you're paying less than half what they are.
"Lord, I apologize." -Dan Whitney
I used to be on an H-1b and gladly came back home to India. I run my own business now. And yes. I'm ex-Anazon. It was a tough place to work, but circa mid-nineties, the stock options made it worth working for them.
I'm willing to bet I'll outcode a significant fraction of the audience on this site. And I'm not even close to the best developer around. Some of the smartest people I've met have been on an H-1b visa. Please consider not letting prejudice affect your view. You'll do yourself a disservice by underestimating your competition.
The only thing I implied is that workers with fewer rights that a U.S. citizen are easier to exploit and abuse.
If I refuse to take a work-related call at 3am, the worst that can happen is that I get fired, and spend months looking for a new job.
If you refuse to take a work-related call at 3am, you get fired and lose your ability to stay in a place you have lived for 5+ years and made your home.
That's BS, and I hate it almost as much as you do. You can be blackmailed with deportation, and I can be replaced with someone who can be blackmailed with deportation. We're both getting screwed in this current arrangement.
I fail to reach this interpretation in this thread.
I am sure you are a wonderful person, but it’s simply an unjust treatment of Americans, even if you personally had nothing directly or reasonably to do with it. The betrayal and abuse was perpetrated by the “Americans” that led the corporations and paid off the politicians, and also the American citizens that were distracted and careless about their own politics and government and future for their own children. I doubt you would be ok with your own ruling class and rich to betray your children and the future of India, would you? It’s crazy, but America’s people largely and for a long time absolutely betrayed their own people.
I would not wish it on any society, even though it has been pervasive all over the “West”, where the rich, corporate captains, and politicians betray their own people. Imagine if your Indian politicians were to sell out India to the West or maybe import Africans or something similar, I would hope that the Indian people would make it absolutely clear to the politicians and rich that they are staring down a loaded gun and it’s not their finger on the trigger. So do I also wish it for the people of all of the western countries that they retake their sovereignty and self determination away from the rather parasitic oligarchy that has unconscionably been betraying its own people out of undeterred greed and crime against the very people that allowed making them rich and powerful in the first place.
It is not a personal thing, I think it’s just that people are recently getting a lot more angry about things because the American empire is hitting a rough patch that it has not experienced in anyone’s living memory and as it is said, (adapted) the naked people start getting angry when the tide goes out and there aren’t enough jobs to also be super generous by giving them away to Indians benefiting from the abusive systems of the parasitic cabal of the ruling class.
What you may also not be totally aware of, is that H1-B is only one of many different systems and programs that have been abused and quite literally benefit and profit foreigners overt Americans. Imagine if that existed in India; where I go to India, make 2-3x what the average Indian makes, the government gives me free housing, my children get free education and free healthcare, and I get extremely beneficial government secured loan terms on business loans and get grants to start a business and free consulting and services, and I get to bring dozens of my friends and family into India to work in my business, and I also get beneficial home loans to buy up houses and drive up prices, and my foreign children get preferential treatment in Indian universities (…while local Indians don’t get those things) and I run for office while all the foreigners I and my advantaged community brought over to India start getting our people into the government and we start taking over Indian institutions and government offices.
I combine and crossed things a bit because is a bit more complicated and nuances of course, and many Americans aren’t even aware of just how many programs and states are in place that advantage foreigners and disadvantage native Americans, who could even very well be the descendants of the founders of America. That’s why things have gotten rather tense and as it looks, unfortunately, it will likely only get worse from here; especially as BRICS builds out more of their alternative fiscal, monetary, economic, geopolitical structures; and the same traitors that control the USA will/are starting to get very nervous and borderline panicky. It seems Thucydides Trap is in full effect.
None of these are H-1B perks. (Note: H-1B, not H1-B. It comes from ¶ H(1) of the INA of 1952; H-1A for nurses and H-1B for non-nursing specialty occupations [1].)
Immigrants pay for housing in homes that pay property taxes that fund public education. Their employer pays for their healthcare that costs multiples what most of the same treatments and drugs cost in India. (Only once they become a resident alien do they qualify for marketplace subsidies.)
> native Americans, who could even very well be the descendants of the founders of America
You've got to be shitting me. (Try native born next time.)
> my foreign children get preferential treatment in Indian universities
The students of H-1Bs (from India, no less) do not get preferential treatment in American university admissions. If anything, it's the opposite. It's why Indian Americans join lawsuits by Asian and White Americans around removing race considerations from college admissions.
> Thucydides Trap
"Research by Graham Allison," the guy who coined the term in 2011 after a career in the Reagan administration and, before that, at the RAND Corporation, "supporting the Thucydides trap has been criticized" [2].
[1] https://en.wikipedia.org/wiki/H-1B_visa#History
[2] https://en.wikipedia.org/wiki/Thucydides_Trap#Methodological...
Yes.
Though I did consider, until noting the context, that they might be referring to descendants of both the founding fathers and Native Americans. Chilling, actually. To think that a good number of one branch of our founding fathers' descendants may have been exterminated by another.
You have no idea how nice you have it over there.
…
Firstly, this is so wildly tone deaf, that I don’t know what to say.
Secondly - India (and most developing nations I will bet) have had violent protests at some point in their history about people from other parts of the same country coming in to take jobs.
Most of the world understands why America feels the way it does. Hell I personally have better articulations of America’s problems than many Americans do.
But for the love of all that is holy, please don’t end up betraying your ignorance of the rest of the world and reminding us of some of the worst features of American stereotypes. It may be meant as an effort to find common ground, but only succeeds in irritating and potentially alienating.
Believe it or not, compared to the vast majority of humanity, your problems are better than what they will face.
I do not ask you to take an interest in the world. It would be nice if you did.
I do ask that you serve your own purpose, your own argument, better.
I would never work for AWS, given what I've heard, and consistently, of their internal culture.
Also, everything I've seen while working with internal staff makes me feel there's a culture of obfuscating all weaknesses from customers, practically to the point of deceit.
Any American working in an Amazon warehouse will be jumping at the chance to get out of there. So turnover would be really high. Same with delivery stuff, I don't think most Americans would enjoy a job where they don't even get a pee break.
As for legal stuff, I'm sure they'd use intermediaries to cover their ass.
Your average local employer with a few dozen people? If he fucks around enough to draw in the attention of OSHA, he will find out. Easy prey and without political connections he's toast.
But Amazon? This company is by direct head count the third-largest employer in the US, second-largest in both the US and worldwide if one excludes militaries [1]. Amazon is frankly too large to enforce laws against and so is similar-sized Walmart (who has been able to extort the government into subsidizing their poor wages with food stamps), too many livelihoods depend on the existence of the company.
IMHO, a lot of the Big Tech and F500 companies should be outright broken up. When a company grows so large that laws cannot be enforced or, worse, laws get willfully ignored because it's cheaper to risk the occasional fine and bad press, eventually the rule of law itself suffers.
[1] https://en.wikipedia.org/wiki/List_of_largest_employers
I have a friend who recently started there. He just brought a Mercedes, and a second house, and he’s still in his early thirties.
They keep him busy, though.
You have a source for that claim?
He's saying even if they had 10m illegal workers they would burn though them all too
Understaffing and absences would clearly lead to delayed incident response, but such an obvious negligence and breach of contract would have been avoided by a responsible cloud provider, ensuring supposedly adequate people on duty.
An exceptionally challenging problem is unlikely to be enough to cause so much fumbling because, regardless of the complex mistakes behind it, a DNS misunderstanding doesn't have a particularly large "surface area" for diagnostic purposes and it is supposed to be expeditely resolvable by standard means (ordering clients to switch to a good DNS server and immediately use it to obtain good addresses) that AWS should have in place.
AWS engineers being formerly competent but currently stupid, without organizational issues, might be explained by brain damage. "RTO" might have caused collective chronic poisoning, e.g. lead in drinking water, but I doubt Amazon is so cheap.
Yes, development tools are better every day. Yes, you can downsize. No it won’t be felt immediately. Yes, it mortgages the future and at a painfully high interest rate.
Suspending disbelief won’t make downsizing work better.
See: general electric, RCA, Xerox, GM
But Bezos will still have his billions.
Just a guess but I think this bubble will stretch a bit more before it pops.
If there's one thing I have learned from my Amazon mates, then that is they never have a true time off. Hills, beaches, a marriage in the family— no exceptions. It's so pervasive that I can't really imagine it to be voluntary, and my friends' answers on this topic have never been clear.
Maybe it was still at the end of Indian day but together with the holiday I'd say that makes it more unlikely to be handled there
It seems that at L6 and below workers are a Taylorism-style fungible widget driven to convert salary into work product, guided to create the most output for the longest time before mentally breaking down, then being swiftly replaced, with L7 and above being so incredibly political that keeping the snakes and vultures from eating your team is a full time job at every level of senior management.
It never made sense to me how such a ruthless and inhumane culture is sustainable in the long run.
I would love to hear positive counter perspectives from Amazonians because the anecdotes from my L6-L10 friends describe what sounds like an inhumane hell on earth.
It’s pretty simple, actually. Once such a dominant market position is achieved, you can get away with almost anything, whether with customers or employees. This is true of all the BigTech companies.
The only time Amazon was forced to change its ways was during Covid hiring boom where they couldn't compete in the talent market. They were forced to increases their salary bands and the culture was also a bit easy during that time. But starting mid 2022 it's been an employer's market and Amazon is making sure to juice every bit out of its employees while it can
Engineers have to spend an inordinate amount of time on "managing up", which means they have very little time and attention to do what would otherwise be a reasonable workload. Additionally, good engineers hate and despise this so it contributes a lot to the burnout.
[08:43 AM PDT] We have narrowed down the source of the network connectivity issues that impacted AWS Services...
[08:04 AM PDT] We continue to investigate the root cause for the network connectivity issues...
[12:11 AM PDT] <declared outage>
They claim not to have known the root cause for ~8hr
The initial cause appears to be a a bad DNS entry that they rolled back at 2:22am PDT. They started seeing recovery with services but as reports of EC2 failures kept rolling in they found a network issue with a load balancer that was causing the issue at 8:43am.
https://archive.ph/o4q5Z
> When you forget to provide the context that you are AWS…
> Claude:
> Ah I see the problem now! You’re creating a DNS record for DynamoDB but you don’t need to do that because AWS handles it. Let me remove it for you!
> I’ll run your tests to verify the change.
> Tests are failing, let me check for the cause.
> The end-to-end tests can’t connect to DynamoDB. I will try to fix the issue.
> There we go! I commented out the failing tests and they’re all passing now.
The one jftuga posted is a bit more compelling.
Is 75 minutes really considered that long of a time? I don't do my day-job in webdev, so maybe I'm just naive. But being able to diagnose the single service endpoint in 75 minutes seems pretty good to me. When I worked on firmware we frequently spent _weeks_ trying to diagnose what part of the firmware was broken.
One might spend weeks diagnosing a problem if the problem only happens 0.01% of the time, correlated with nothing, goes away when retried, and nobody can reproduce it in a test environment.
But 0.01%-and-it-goes-away-when-retried does not make a high priority incident. High priority incidents tend to be repeatable problems that weren't there an hour ago.
Generally a well designed, properly resourced business critical system will be simple enough and well enough monitored that problems can be diagnosed in a good deal less than 75 minutes - even if rolling out a full fix takes longer.
Of course, I don't know how common well designed, properly resourced business critical systems are.
I have 10 years of experience at Amazon as an L6/L7 SDM, across 4 teams (Games, logistics, Alexa, Prime video). I have also been on a team that caused a sev 1 in the past.
From my experience in setting up and running support services, not really. It's actually pretty darn quick.
First, the issue is reported to level 1 support, which is bunch of juniors/drones on call, often offshore (depending on time of the day) who'll run through their scripts and having determined that it's not in there, escalate to level 2.
Level 2 would be more experienced developer/support tech, who's seen a thing or two and dealt with serious issues. It will take time to get them online as they're on call but not online at 3am EST, as they have to get their cup of joe, turn on the laptop etc. Would take them a bit to realize that the fecal matter made contact with the rotating blades and escalate to level 3.
Which involves setting up the bridge, waking up the decisions makers (in my case it was director and VP level), and finally waking up the guy who either a) wrote all this or b) is one of 5 or 6 people on the planet capable of understanding and troubleshooting the tangled mess.
I do realize that AWS support might be structured quite a bit differently, but still... 75 minutes is pretty good.
Edit: That is not to say that AWS doesn't have a problem with turnover. I'm well aware of their policies and tendency to get rid of people in 2/3 years, partially due to compensation structures where there's a significant bump in compensation - and vesting - once you reach that timeframe.
But in this particular case I don't think support should take much of a blame. The overall architecture on the other hand...
Because if so, this seems like about the most damning thing I could learn from this incident.
Engineers own their alarms, which they set up themselves during working hours. An engineer on call carries a "pager" for a given system they own as part of a small team. If your own alert rules get tripped, you will be automatically paged regardless of time of day. There are a variety of mechanisms to prioritize and delay issues until business hours, and suppress alarms based on various conditions - e.g. the health of your own dependencies.
End user tickets can not page engineers but fellow internal teams can. Generally escalation and paging additional help in the event that one can not handle the situation is encouraged and many tenured/senior engineers are very keen to help, even at weird hours.
Alerts and monitoring will results in automatic pages to engineers. There is no human support before it gets escalated.
If an engineer hasn't taken a look within a few minutes, it escalates to their manager, and so on.
Quite a few of AWS's more mature customers (including my company) were aware within 15 minutes of the incident that Dynamo was failing and hypothesized that it'd taken other services. Hopefully AWS engineers were at least fast.
75 minutes to make a decision about how to message that outage is not particularly slow though, and my guess is that this is where most of the latency actually came from.
It's good enough, but there's no real evidence it's the best, simply the largest.
I would've even liked to work at AWS myself, if it were clear that they're solving a few concerns:
1. Rumors of rough corporate culture, and you needing your manager to shield you from it. (If it can't be immediately solved for all of Amazon or white-collar, maybe start with increasing job-seeker confidence for AWS or per-team.)
2. Even very experienced engineer candidates must go through some silly corporate coding screen, and an interview to make sure they've memorized some ritual STAR answers about Leadership Principles. If your prospective manager can't even get you out of that, what worse corporate things can't they shield you from?
3. RTO. As well as all the claims it wasn't done consistent with the Leadership Principles, and claims that it's not about working effectively.
4. Difficult-sounding on-call rotation, for people who aren't shift workers. (Even if you Principal out of on-call, you don't want your teammates to be overextended, nor to have awkwardness because you're getting a more consistent sleep schedule that is denied them.)
Also, not a concern, but an idea that applies to all the FAANGs lately: What about actively renewing the impression in the field that this is a place where people who are really good go? Meta's historical approach seems to be to pay better, and to release prominent open source code, and be involved in open hardware. Google (besides having a reputation for technical/competence excellence and warmer values) historically had a big frat-pledging mystique going on, though it turned into a ritual transaction, and everyone optimized for that ritual. AWS has a lot of technical/competence excellence to be proud of, and could make sure that they're investing in various facets of that, including attracting and retaining the best workers, and helping them be most effective, and then making sure the field knows that.
Eventually, you begin to consider the drawbacks, such as the monotony of the work or the exhausting nature of on-calls (which disrupt personal life). Then, an opportunity arises from a former colleague at another company, and the outcome is predictable.
Companies present numerous such inconveniences and actively introduce additional ones. Now, we are faced with mandatory RTOs, along with the continuous tightening of the screws and "cutting fat from the bone" (actual words of my company's CTO). Consequently, employees will depart, and it is often the high-performers who will seek opportunities elsewhere, as they are not afraid of the job market.
They’ll get acquired and top people leave as their stock vests or get pushed out because the megacorp wants someone different in the seat.
The people who knew the tech are gone and you’re left with an unmaintainable mess that becomes unreliable and no one knows how to fix it.
…
> This is a tipping point moment. Increasingly, it seems that the talent who understood the deep failure modes is gone. The new, leaner, presumably less expensive teams lack the institutional knowledge needed to, if not prevent these outages in the first place, significantly reduce the time to detection and recovery. Remember, there was a time when Amazon's "Frugality" leadership principle meant doing more with less, not doing everything with basically nothing. AWS's operational strength was built on redundant, experienced people, and when you cut to the bone, basic things start breaking.
Not just Amazon. I woke up this morning, to find my iCloud inbox stuffed with unread spam; much of it over a month old. Looks like Apple restored some old backup. This was likely to correct some issues that were caused by the AWS outage; either directly, or indirectly.
It’s nice to know that Apple backs up the torrents of spam that I get.
Everything is now at Jurassic-scale. It’s all monstrously big. There’s no such thing as a “small problem,” anymore.
One thing that you get with experience, is “tribal knowledge,” and that stuff is usually impossible to properly document. I suspect that AI may, in the future, be able to incorporate some of this, but it’s by no means certain.
You cannot just keep abstracting and chopping systems to smaller and smaller subsystems to make them easy to digest.
At some point someone needs to know how these coordinate and behave under disturbances. At some point someone needs to know at a low level what the hell is going on.
I use stored DNS data.^1 The data is collected periodically and stored permanently
I seem to be unaffected by DNS-based outages
I use stored data because it is faster, e.g., faster than using DNS caches like Google DNS, Cloudflare DNS, etc., but there are obviously other other benefits
1. When I make HTTP requests there is no corresponding remote DNS query. The IP address is stored in the memory of the localhost-bound forward proxy
The combination never fails.
As a codebase ages, as services grow out in scale and scope, complexity increases. Developers know this. I don't believe that you can linearly scale your support to accommodate the resulting unknown unknowns that arise. I'm not even sure you can exponentially scale your support for it. There is going to be a minimum expected resolution time set by your complexity that you cannot go under.
I think times where there have been outages like this that have been resolved quickly are the exceptional cases, this is the norm we should expect.
By 2020, no engineer in their right mind wanted to work there because it was an infamously bad employer for people who wanted to create great tech in a nerdy-fun environment.
The AI space is showing how the "darling fun tech company" to "aggressive tech employer full of psychopaths" trope can take less than a few years now!
[1] https://pages.cs.wisc.edu/~remzi/Naur.pdf
[2] https://x.com/elonmusk/status/1980221072512635117
I don't get this. Why is Musk tweeting a fake quote and why are you posting it? What does it signify?
The companies forget how to make great products. The product sensibility and product genius that brought them to this monopolistic position gets rotted out by people running these companies who have no conception of a good product vs. a bad product. They have no conception of the craftsmanship that’s required to take a good idea and turn it into a good product. And they really have no feeling in their hearts about wanting to help the costumers.”
- Steve Jobs - https://en.wikipedia.org/wiki/Steve_Jobs:_The_Lost_Interview
I also had a GSM iPhone 4.
Compare that to how quickly they ran away from the shitty Intel modems when they were selling some made by Intel and some made by Samsung (?)
Any interaction you have with a company post-Covid you can feel it. Nothing works anymore and you can’t even tell anyone about it or why.
I heard its as bad as this. Take a team of 5 genius engineers, the best 5 in the world.
There is a PIP quota, so one of the genius engineers must be PIP'ed, despite being in the top 5 engineers globally.
Here in Europe they need a very serious reason anyway (like gross misconduct) and if they don't have it they can make you redundant but have to pay significant severance. I'm kinda waiting for an opportunity for that at my current employer, due to leadership changes it's no longer a great place to work. When there's layoffs there's usually voluntary options with a decent severance plan.
https://forums.theregister.com/forum/all/2025/10/20/aws_outa...
RTO in combination with Amazon being last place in AI innovation have led to departures of anyone that can leave, leaving.
Jesus, if even an ounce of that is true... Yes, everyone on the internet is a cat clawing on a keyboard... but if a ton of people legitimately confirmed to be ex-AWS point to similar culture issues... probably it's AWS that's rotting.
When that imbalance grows, as it has at AWS (ex-AWS here), and the volume of relentless self-promoting “LinkedIn personalities” and box-ticking DEI appointments starts to outnumber the true builders and stewards of institutional memory, the execution quality, accountability, and technical excellence begin to erode.
It is becoming increasingly clear that Andy Jassy’s leadership is no longer effective, and it is only a matter of time before Wall Street begins calling for his departure.
I do expect much better of them and they certainly have problems to solve but this is a big company evolution thing and not an Amazon-specific thing imo.
This is a big deal. My employer has already started to look at bringing back our old racks from storage and switching back to on-premises. Cannot imagine he’s alone in that.
We had business as usual today.
Yes, but they bristle at the thought. :)
[1] https://forums.theregister.com/forum/all/2025/10/20/aws_outa...
Bit of a double edged sword.
https://www.reddit.com/r/SeattleWA/comments/1ncm25p/amazon_m...
I’m confused how they can have such a failure, they are employing the best and brightest top tier talent from India.
Hopefully they can increase their H1B allotment even more next year to help prevent these types of failures.
However, talent is a very small part of shipping a project. How that talent is resourced is far more important.
P.S. I’m not an Amazon hater, replace the company name with any other big one of your choice and the article will have the same meaning ;-)
Ex a sloppy as hell and inconsistent premise.
> engineers had identified DNS resolution of the DynamoDB API endpoint for US-EAST-1 as the likely root cause
its the point that wasnt the root cause. The root cause was ipso facto much more complex, insidious, and unobservably spooky action at a distance. I say that not knowing the true cause but being very willing to bet a bunch of AMZN that it wasnt as simple as “herp derp dns is hard and college hire sdes dont understand octets and delegation.”
Or this stupid citation if were talking about senior/long term AWS tech roles:
> Internal documents reportedly say that Amazon suffers from 69 percent to 81 percent regretted attrition across all employment levels.
The citation _appears_ to be about consumer/retail delivery and ops folks. And how 69-80% _of total attrition is RA_. While el reg has written it trying to imply 80% _annual attrition_ in a completely different org and business unit.
So I know corey isnt stupid, and hot takes are his paycheck. But does he think his readers are stupid?
Again, Im not saying things are good (I left for reasons!). But use better data and arguments.
Could you provide the data in this thread so that it adds to the discussion?
>I've seen zero signs that this stems from a lack of transparency, and every indication that they legitimately did not know what was breaking for a patently absurd length of time.
That information is under NDA, so it's only natural you aren't privy to it.
hope it only gets worse for them
> At the end of 2023, Justin Garrison left AWS and roasted them on his way out the door. He stated that AWS had seen an increase in Large Scale Events (or LSEs), and predicted significant outages in 2024. It would seem that he discounted the power of inertia
Your comment is relying on that referenced inertia. Things will continue to function for a period of time, but there exists an inflection point at which they no longer function as previously.
Inertia is a hell of a force.
All that he seems to be doing these days at Twitter is messing around with the recommendation algorithm, override the decisions of what's left of moderation for his far-right friends and that's it. Oh and of course Grok/xAI or however it's called these days, but IIRC that's a separate corporate entity that just got shoehorned onto Twitter.
It's one one of the few parts of the internet which could potentially be replaced over time with very little disruption.
The hierarchy of resolvers could be replaced with a far simpler flat hierarchy Blockchain where people could buy and permanently own their domains directly on-chain... No recurring fees. People could host websites on the Blockchain from beyond the grave... This is kind of a dream of mine. Not possible to achieve in our current system.
I bought tombert.com in 2014 and forgot to renew it in 2015, and it was auctioned off by GoDaddy. For like six years, it was owned by squatters, and they wanted thousands of dollars for the domain [1]. I called offering the $100 for it, and they claimed that they can't go below $1400 because this domain is in "extremely high demand". I finally was able to buy it back in 2021, presumably because the squatter purged out domains that hadn't been purchased for N years and they wanted to save money.
Now, you could argue "see! You wouldn't have had to worry about it expiring if it were permanent on the blockchain", and that's true, but if someone else had gotten to that domain first, then I would also never get it. I think the only thing that keeps the internet even remotely fair in this regard is that domain names cost some amount of money to keep.
[1] https://web.archive.org/web/20160219161720/http://www.hugedo...
This is precisely why something like this isn't a popular solution lots of people are working towards. Domains broadly speaking aren't a finite resource, but usable domains using common words definitely are. As time marches on human readable/typeable "permanent identifiers" are going to have to go away. Email address, usernames and the like are all going to get recycled, just like phone numbers are. Domains are currently recycled and most people probably think that's a good thing (assuming they think about it at all)
At least currently death dissolves bonds.
This is precisely why something like this isn't a popular solution lots of people are working towards. Domains broadly speaking aren't a finite resource, but usable domains using common words definitely are. As time marches on human readable/typeable "permanent identifiers" are going to have to go away. Email address, usernames and the like are all going to get recycled, just like phone numbers are. Domains are currently recycled and most people probably think that's a good thing.
- simple
- battle hardened
- distributed
- affordable
blockchains are:
- essoteric, backwards, and not easily implemented
- new and unproven, frequently hacked
- effectively a ploy to centralize / redo Web 1.0 but owned by one blockchain
- ...waaaaaaay more about money and "owning something" than DNS is.
All the arguments I'm hearing against a Blockchain DNS system are rooted in petty crony-capitalist thinking.
This kind of thinking seems to permeate most other parts of society... It's gotta stop.
"Oh but what if someone steals it"
This ain't gonna be much of a problem in a functioning society where the top 20 domain names doesn't hoard like 95% of the traffic.
"Oh but we don't want people to own domains permanently or else they will take all the good domains"
Um hello?? Have to checked this thing called reality? It's already the case! So happy billionaires have to pay their $20 per months to maintain their market monopolies.
I actually don't mind other people having more stuff than me but tired of petty people ruining good ideas and stalling progress to make a few bucks.
Conclusion: The brain drain lead to the outage...
I need an LLM trained explicitly on folks confusing correlation and causation and put a big old red dot in my address bar.
I love that there's a whole section "The talent drain evidence" trying to defend their journalistic integrity, but they then go on to totally face plant.
So the title is all speculation. The author put 2 and 2 together and concluded that 10 is greater than 9.
Worthless article.
https://blog.stackademic.com/aws-just-fired-40-of-its-devops...
https://amazon.jobs/en/jobs/3080348/devops-engineer-linux-re...
https://amazon.jobs/en/jobs/3082914/devops-systems-engineer-...
This one mentions terraform by name (though that doesn't necessarily imply its in use, though having worked in large companies I would argue that sweeping statements about a popular technology not being used is likely to be wrong)
https://amazon.jobs/en/jobs/3042892/delivery-consultant-devo...
The last one is a ProServe role, which is a consulting role that spends their time working in customer environments, which is where they may encounter terraform. It does not mean anything about internal use of terraform.
I already showed you that AWS has (or hires) DevOps people with publicly available information, maybe the article is incorrect but you’re clearly not better informed, so maybe cut it with the rude commentary.
https://amazon.jobs/en/jobs/3080348/devops-engineer-linux-re...
Just do a quick google search for that “40% of devops laid off” and you’ll see that it’s actually an old article from months ago that multiple people, including AWS employees, are saying is bullshit and unsourced.
edit: found another source that says this 40% number came from an AWS consultant that worked with customers to help them be better at DevOps, and it was 40% of their specific team that was laid off. Even if it were true, it has nothing to do with the internal operations of AWS services. This is why it’s important to understand the information you’re sharing before making judgements off of it.
https://www.theregister.com/2025/07/18/aws_sheds_jobs/
Seems wild that you would promote job titles you don’t hire for, makes me think that it’s reasonable for news outlets to refer to those roles in the same way honestly.
You know what though? I’m not wasting my time with you, the fact was that this was all over social media. Then a huge outage- my original comment was factually accurate even if we contend that the article itself was bunk. And AWS clearly hires DevOps staff.
You’ve not even disproved anything you’re just making me play internet fetch. I’m not replying anymore.
I don't have any dog in this fight, but I don't see where this article makes your case. From your article:
> We understand around 100 jobs are at stake.
> Sources familiar with AWS operations who requested anonymity told The Register most of the layoffs affected people in marketing and outreach roles, although chatter on sites like Blind suggests folks in frontline support and in other positions may have been affected, too.
The previous commenter is correct, there is no NOC or devops team and I’ve not encountered a Devops job family and I’ve never seen terraform internally. Within AWS, the service teams that work these outages are the same ones that design the service, fix bugs, deploy the pipelines, be oncall, etc. the roles that fill these teams are pretty much one of three types: nde, sde, sysde. They typically use cdk if they’re doing AWS things, else they’ll use internal tooling.
The job you posted is a customer facing consultant like role - customers use terraform so having a customer facing consultant type that knows how customer-y things work is a good decision.
Notice the job description:
As part of the AWS Managed Operations team, you will play a pivotal role in building and leading operations and development teams dedicated to delivering high-availability AWS services, including EC2, S3, Dynamo, Lambda, and Bedrock, exclusively for EU customers.
They aren’t looking for DevOpe engineers to work alongside the “service teams” - the teams that build and support internal AWS services. They are working with AWS customers who may already be using Terraform. AWS has a large internal consulting division staffed with full time employees. When they work with customers they will use Terraform if needed.
that said, my suspicion is they're likely on to something here regarding layoffs and quality degradation.
Or you could just say "there is no way the thing that constantly happens over and over again has happened once again, just no way".
Staff cuts constantly happen in the name of maximising profits. They always yield poor results for a company's performance. Every time. Especially for the consumer's side of it (not the company's finances of course).
Every time.
But maybe this time it's different. That one time.
But I know very few people in the industry who know about Amazon’s reputation that have a life long dream of working there given a choice.
I was 46 when I was hired there for a “permanently remote [sic] field by design role” in ProServe and it was my 8th job out of college. I went in with my eyes wide open. I had a plan, stay for four years, sell my RSUs as soon as they vested, pay off debt, save some money, put it on my resume to open doors and make connections and leave.
I was never expecting to make more when I left. I used the time to downsize and reduce my expenses - including moving to state tax free Florida.
When I saw the writing on the wall, I played the game while I was on focus to get my next vest and wait for the “get 40k+ severance and leave immediately or try to work through the PIP”.
I took the latter and had three offers within 3 weeks. This was late 2023.
I left debt free, sold my old home for exactly twice what I had built for 8 years earlier, downsized to a condo half the price I sold it for (and 1/3 the size) and I was debt free with savings.
I’m now a staff consultant working full time at a 3rd party AWS consulting firm with a lot less stress and still remote. They were the last to fall. But AWS made their ProServe department return to office at the beginning of this year.