Also: Ditching Twitter? How to get started with Mastodon Why? Simple. You can’t lay off half of the staff of a cloud-based social network and expect things to keep running smoothly for Twitter’s 450 million monthly active users. Indeed, Twitter accounts are already failing in odd ways. For example, Benjamin Dreyer, author of “Dreyer’s English” and copy chief of Random House, found that the vast majority of replies to one of his tweets were vanishing into the aether. He wasn’t the only one. Even Musk appears to have realized that maybe firing every other person was a mistake. On Monday, November 7th, he tried to get workers, especially software engineers, to return. Good luck with that. According to my Twitter sources and tweets on the site, they’re not coming back. As Gergely Orosz, editor and author of the popular software engineering and management blog, The Pragmatic Engineer, said, “Several people who were let go on Friday, then asked to come back were given less than an hour as a deadline. Software engineers who got this call … all said ’no’ and the only ones who could eventually say ‘yes’ are on visas.” Managers, according to my sources and Orosz, are “getting desperate, trying to call back more people. People are saying ’no’ + more sr engineers are quitting.” Also: Elon Musk says Twitter usage is ‘at all-time high LOL’ Orosz added, “None of this is surprising. As a rule of thumb, after you lay off X% of people, you get an additional half attrition. Lay off 10%: expect another 5% to quit. Lay off 50%… not unreasonable to expect another 25% to quit.” And, you can’t expect to replace social network and cloud experts with Tesla embedded system engineers and get anything done. I’m a good technology and business writer, but no one in their right mind would hire me to write opera arias. Let’s look at Twitter’s technology, shall we? Twitter runs on CentOS 7. This free Red Hat Enterprise Linux (RHEL) clone comes to the end of its life at the end of June 2024. The leading choices for what to replace it with should be RHEL 9, Rocky Linux, or AlmaLinux. But instead of working on on that transition, what few system administrators Twitter has left are both trying to get the platform ready for Musk’s laundry list of new features and keeping it patched and up-to-date. That’s a problem. You see, unlike RHEL, where a big part of the attraction is that you can depend on Red Hat for first-rate support, CentOS, Rocky, and AlmaLinux are all primarily meant for companies with in-house staff who already know Linux servers backward and forward. That’s no longer the case at Twitter. As Terra Field, a Honeycomb Staff Platform Engineer, pointed out, “An infrastructure the size of Twitter will not fail overnight, but it is going to relatively quickly go into a degraded state. The few platform people that remain will have their necks exactly at the waterline. It will not take much to sink.” Field’s right. Also: How to delete your Twitter account Twitter also relies on its own homebrew software. Programs such as Apache Heron for stream processing and twemproxy for cache scaling are essential for Twitter to run at scale. If programs like these go awry, and they will, Twitter’s performance will slow down. That’s probably not what Twitter wants to happen since the US elections are guaranteed to give the social network a serious stress test. Finally, there’s the cloud. While Twitter relies a lot on in-house bare-metal for its Linux servers, it also needs cloud functionality to run as well. So, when it was leaked that Musk wanted to cut between $1.5 million and $3 million daily from servers and cloud services in his “Deep Cuts Plan,” people worried. They’re worried with good reason. As Corey Quinn, Chief Cloud Economist at Duckbill Group, explained, “Pursuing [a 50% cut] as the defined goal is courting disaster.” That’s because “a third of your bill is misconfigurations” doesn’t scale, because someone notices! It’s a variety of interconnected services, along with redundancy for systems, backups, etc. Very quickly, you’re not cutting fat, but muscle and bone." For example, if you delete the data warehouse, you probably haven’t broken Twitter, but you no longer have reliable reporting. Another “significant spend source,” Quinn remarked, “is observability. Turning all of that off will cut a giant portion of your bill – but the next time Twitter goes down, there’s a decent chance that it never comes back up again. You absolutely don’t need backups until suddenly you very much do.” Sure, Quinn continued, “Twitter has also been around for ~15 years. [So,] a lot of infrastructure ‘cruft’ has built up. “Is that important, or can it be turned off” is the kind of thing best answered via asking people. 50% of Twitter was let go today, and your oral tradition just walked out the door too.” Besides the technical aspects, Quinn pointed out, from a business standpoint, Twitter has contracts with “Google Cloud, AWS, and on-premise data centers. At this level of public spend, there are multi-year contracts in place guaranteeing certain spend levels; an awful lot of this is amortized over multiple years.” Put it all together, and it’s not “if” Twitter will crash soon; it’s “when.” I won’t be surprised if Election Day brings it down. If not, I give Twitter until the end of the month before its next major crash. And, when will it come back up? That’s a darn good question. I won’t be surprised if it takes days or even weeks. Replacing Twitter, and let’s face it, that’s what I predict most of us will end up doing, won’t be easy. Ed Bott recently pointed out that there are no good Twitter replacements yet. He expects that unless Musk “actually burns down the data center housing Twitter’s core assets, Twitter isn’t going away.” I fear that while Musk isn’t throwing gasoline and lit matches at the servers, he’s already destroying the people and software Twitter needs to run. Related Stories:
Elon Musk says Twitter rules will ’evolve’ as he talks up new featuresPay-to-play Twitter verification? No thanks, but here’s what I would pay forElon Musk promises not to make Twitter a ‘free-for-all hellscape’
