Convopage
See the entire conversation
War stories please.
What’s the jankiest piece of tech you’ve seen a company depend on?
1,442 replies and sub-replies as of Dec 17 2021
You all are making my day.
Many years ago, I worked at a research chair and was told to take care of its infrastructure. In the server room, there was an old SUN workstation running 24/7 and nobody knew why. All of my predecessors were too afraid to turn it off and simply called it "the heater".
Technical debt as performance art.
twitter.com/erindataviz/st…
Oooh do I have some stories. One group had to perform complex calculations for all our 60k+ clients, had no IT support and were entirely staffed by 22 yr old business majors who had never heard of databases. (1)
This is the scariest thing I've read today.
twitter.com/tdawry/status/…
Not all heroes wear capes.
(Although if you do wear a cape Simon that's cool and I don't judge.)
twitter.com/sbisson/status…
The booking service for a major shipping company that was a stack of three different generations of Oracle technologies.
Layer 1 and 2 were supported, layer 2 and 3 were supported. But the whole stack? Nope.
It was 3 years into a six month development program. I cancelled it.
The replies are an escalating hall of horrors.
twitter.com/jeanmarcalkazz…
Worked at an insurance company where all financial data was stored on 1 dusty Windows NT 4 server. The day before closing the fiscal year it crashed and couldn't connect to the AD. Had to search for drivers and OS patches on archive.org and use 3x💾 to boot from...
This is why we can't have nice things.
twitter.com/chrisalbon/sta…
Sometimes I get weighed down worrying about how far my systems are from ideal, but it's good to remember that my real goal is just to make them a little less horrible.
twitter.com/dbredesen/stat…
I worked at an online travel company, and their entire travel booking flow was implemented in a single function with thousands of lines of code, called DoAllTasks(). It somehow managed to process thousands of bookings per day.
@rayploski @cbredesen @KonnectedKris
On dark days when your own tech debt makes you cringe, you can come back to this thread for comfort.
twitter.com/BruceEdiger/st…
SPARCStation-20 with a SDLC sbus card to communicate between a western US baby bell and Dun & Bradstreet. All customer credit ratings checked via this system. No earlier than 2007. They had no spare SDLC card.
Responses keep blurring the line between entertaining and terrifying.
twitter.com/justinlmitchel…
Does the fact that basically all of the US healthcare system is run on EHRs that are using MUMPS count?
Rube Goldberg, hold @patio11’s beer.
Code or process that is written like an inauspicious game of Jenga where the removal or failure of any single piece brings it all crashing down. 😂
The actuarial dept for the workplace div of a very large insurance company relied on an app running on a old desktop under the developer’s desk. He had a stack of 3 and always forgot which one was “production”.
I worked at a health care company responsible for notifying millions of hospitalised patients of surgeries, prescription info, & emergencies. One engineer built the application complex enough where only he understood how to edit the main java files to guarantee his employment.
Best "If it ain't broke ..." - An IBM XT PC running key switches in a telephone exchange for a whole suburb. It was on 24/7 for 20 years before finally catching fire and burning down a chunk of the telephone exchange.
I hope one of you is currently writing a treatise titled
On the Causes of Computing System Fragility and Why It’s Never Ever Going Away
So, I worked somewhere, that the CEO and founder insisted to write code, about 7 years later after they migrated away from the system he developed, they printed the source code which had 4000 lines of switch cases.
first magazine I ever worked for rebooted itself after failing, moved to new office, had no internet but had to close first print issue night, do or die -- we stole open wifi from office next door and connected our LAN to it; intern had to stay up all night to keep it running
nice try, i have an NDA
At Bing years ago we had a service that ran 100s of classifiers on every query. There was 1 editable config file for everything. Once another team disabled our classifier when they accidentally deleted the last line of our classifier’s configuration while editing their config.
I used to have to remote control a desktop over Zoom to ssh into an on-prem server (the on-prem eng had to log me into several time-out-ing terminal windows w/ his creds because I didn't have ssh creds and he wasn't going to sit there and watch me work) and code in vim.
There is a bowling alley near my house that still runs their small business off of an Apple IIe with a dot matrix printer. No joke. When I asked about it, the owner looked at me, said "don't fix what ain't broke" and then then back to spraying shoes.
Actual war story about ANET, the system NATO forces were using to track their progress training the Afghan military.
From this piece on the Defense Digital Service's efforts to overhaul it: wired.com/2017/05/meet-n…
Serial null-modem cable made out of paper clips. FWIW, a paper clip will support 115.2Kbps.
you said companies, but the academic science lab duct-tape as solution game is very strong
I ran RS232 50m using a shielded twisted pair cable and we maintained 115kbps.
That seems highly appropriate.
OF COURSE there's a typo
Well done the bowl owner for not wasting more money on the Apple users pay 30% more rip-off. ;)
Very short treatise.
"People."
-fin
No time, gotta unscrew this vlookup.
We've all worked with that person before. Sadly they still get fired and the company moves on without them when they could have been part of the evolution and transformation to bigger bolder and better. #klingon
All the traders at the I-banks are wedded to Excel, so now all the prop models need an army of coders using Excel Visual Basic to force them through spreadsheets...
This is a great thread, Brandon. Thanks... I think(?) :)
You're most welcome.
You get a sleepless night!
And you get a sleepless night!
And you get a sleepless night!
MUMPS is underrated though
Hospitality company’s worldwide reservation system ran on a single instance of hardware and software that was n-4 beyond support and DBA’s would kill processes daily to keep the system from crashing. Nightmare support scenario.
Sometimes you get to use your consultancy powers for good.
NO CAPES DAHLINK
Some users still haven't done so – according to StatCounter, about 0.59% of Windows computers are still running XP. NetMarketShare puts that figure at 0.32%.- Aug 20, 2021
Plenty of elevators out there running XP Embedded, serviced with iPaqs
Not really if it an air gapped system on industrial hardware. Would be stable and predictable.
I’m sorry I read that. I preferred being blissfully ignorant.
IIRC Lot's of things still run on DOS (which I would trust more than XP).
Jesus Christ
At least the submarine does not need to wait every other morning for the next pointless and time consuming update from MS for Windows 10. See my profile.
Nice. It took my a while before I could stop giggling long enough to type this.
Thank you. Personal computers were a thing of the future when I was on subs. Very little digital computing was employed and that was very specialized.
Could be vista
"We are all going to die"
this is definitely better than running any modern OS with all the telemetry etc
It's actually stable
Is it? Imagine them running on Windows 10.
Echoing someone earlier; air-gap critical systems.
OS/2 in use by a municipal utility ~2005.
I'm assuming they still use AOL dial-up to connect to the internet as well.
Oh dear God!!
Thanks for saying something that doesn't need to be made public.
I'm sure this required the Chinese to teach a whole bunch of 20 yr old hackers how to work with XP. Starting with "what is XP"?
Which I suppose could be seen as a clever security gambit on the part of the Brits...
Best not to think about the old icbm silos that still need 5.25" floppy discs.
Also I supported WinXp a couple years ago. There was a version with real-time extensions for industrial automation. Likely similar reason for the Trident dependency.
(No comment)
If you were in the UK I'd punt a guess that it might have been a British Telecom or other supplier's data traffic logger. Nobody onsite would be likely to look at one of those :)
I have seen occurances in companies where the solution to identify the owner and purpose was to power off or disable networking on a device/unplug it till someone complains 🤣
Not mine but about 10 years ago someone posted a pic of a tire alignment shop that was using a device connected to a C64
In 2013, while working on telehealth, due to legacy systems compatibility, corporate compliance, and liability, we had to make sure our internal portal web application supported IE8...
In Japan there was this company whose internet access was solely through a Win98 computer. The was in 2011
Until a few years ago, the server that handled our phones at work was running on win 98. The phones regularly went out, you couldn't check voicemail from outside the building and it didn't usually work inside the building. New IT guys were afraid to touch it. It was wonderful.
en.wikipedia.org/wiki/Cullinet CIMS, still in service today
Cullinet - Wikipedia
(no description)
en.wikipedia.org
Pretty sure the train in Atlanta Hartsfield airport still runs on an onboard VIC-20, or at least did a few years ago.
The 22 column display on board gave it away.
I know what I’m checking out the next time I fly through there
I'm going to cheat on the word 'depend' because this is my best story - Depending on an SR-71 to test the antenna tracking on a LEO satellite program, until the air force got tired of lending it to us (also the sonic boom was sending the local bovines into counseling)
Whoa. That's like using a GPU as a paperweight.
To be fair, there was no obvious way to test how antennas would track stuff moving at 10,000 miles an hour (LEO speed). That said - even if we'd given the problem serious thought, SR-71 idea would have been on the 3-sigma end of ridiculous
To be additionally fair, and while they may leak profusely on the ground, SR-71s hardly seem to qualify as "janky" tech overall.
I didn't know, I think 'leaking fuel on the ground and requiring an immediate in-air refuel after a high speed run to close the gaps' qualifies as pretty janky to me...
Hard to top this one, indeed.
Me. I’m the jankiest piece of tech companies have relied on.
deployed my first model in COBOL at @AIGinsurance lmao
But. But. But. You’re too young!
I clicked on this thread knowing I’d see insurance or banking mention COBOL and it’s right at the top.
it was 10 years ago! to be fair i built the model and worked with the developer that did it and he was 20+ years my senior
my mans is a legend
had a poisson regression with 500K coefficients 😂
me - I’m like old grandpa who still knows how to turn off the faucet if it starts leaking… 🙄
laptop that used excel, vbscript, access and windows scheduled tasks to run their reports.
i was hired to cast that ancient demon into the void
When you need to call an old IT guy and a young IT guy.
Many years ago, I worked at a research chair and was told to take care of its infrastructure. In the server room, there was an old SUN workstation running 24/7 and nobody knew why. All of my predecessors were too afraid to turn it off and simply called it "the heater".
“… and there it has been, ever since. It is said that late some nights, its fans can be heard *humming* the Macarena tune, which was atop all the charts on the year it was first powered on.”
Yes, that's how it was. Sometimes, I could hear the tune at night.😂
There was an ancient Sun x86 workstation nicknamed 'roadrunner' after the beeps it made when it booted.
If there was anything that could run 24/7, it was a Sun. (You've never seen anyone try to heat a cubicle with Windows, have you?)
That wasn't even intended to be a play on words. Sun computers (running SunOS) were just really reliable (expensive, but reliable).
We ran Sun servers (with Solaris) specifically for their reliability.
we used to test our software on Sun, reboot once every two years
AS400 was another 24/7 workhorse.
I heated my basement home office with a desktop Pentium 4 running XP. Even worked through the worst snowstorm of the decade.
Truly elegant thermal design.
Air comes in one side, exits the other, and everything inside is aligned so air flows across it.
You can even sit the case on its end without blocking the outlets thanks to the ridged plastic molding.
PC/Mac airflow was a total mess by comparison.
Sounds like the computer in @losttvseries. They had to enter numbers into it every 108m it the island would explode.
Pushing the button | Lostpedia | Fandom
Pushing the button | Lostpedia | Fandom
lostpedia.fandom.com
Well... maybe somebody secretly did that and lived in the server room... 😉
This has such a mythical quality to it. Ancient beings from ages before brought a mystical object that no one knows what it is anymore, other than a source of mystery and some unknowable power
The way this tweet is taking off right now, I'm already thinking about making a movie from this story.
I would definitely pay $18 to watch it.
In the chilly server room,
Shrouded in a misty gloom,
Rests a source of ancient power—
Down we kneel before the tower
And we beg it not us shun;
Hear our prayers, oh god of SUN
But what did it doooooooo?
I still don't know. 🤷♂️
It ran yp/nis/nfs/rpc and gave hackers easy access to your network.
Is it still heating today?
Ok, but wouldn’t a person worth their salt do some form of pcap on it to see who it was talking with? The clients would then offer some insight?!
Well, the infrastructure of this chair was maintained by student assistants coming and going every one or two years. Of course, nobody did any further digging and every new assistant coming in was just happy that things were running more or less smoothly.
This exact thing applies to so much in academic research
Did anybody even still have login credentials for it?
Not that I'm aware of.
I’m pretty sure this is what the hacker kids found when they broke into the Cold War era missile silo.
Sounds like the origin story for “Deep Thought” .. u should check if 42 is on the screen
Did it get as hot as the sun?
's @AreciboRadar used to be controlled by a Sun box that was older than some of the graduate students.
That was considered an improvement over the original VAX, which I never had to use but which was still legendary among the observatory staff.
One day, someone somewhere is going to turn one of these machines off, and the Simulation will finally, mercifully, end.
This is stunning
they should have left it offerings
Chesterton’s fence 2.0
And one day they turned it off and the “Y2K” crash finally happened 😜
Some people say it's still running to this day
Reminds me of the time I was upgrading a load of Netware servers in a financial company - there were two we simply couldn't find. Still happily file and print serving, but invisible to the human eye.
Since you said "company" I am going to assume that doesn't include universities. Phew.
A nonprofit I worked for early in my career justified its entire existence with an 'economic impact calculator' (spreadsheet) provided by a vendor with all of the logic hidden behind password protected spreadsheet tabs. And nobody (including the vendor) could remember the 🔑
Ah yes, black-box data manipulation. The best kind of data manipulation.
Besides my code?
Scientologists and "e-meters"... 🤭
A massive set of VMWare clusters hosting identical VMs - each with MS SQL and a .net app being hosted for 10,000+ customers. Each licensed and manually maintained without AD because “they had to be separate for security reasons”
Separately, in 2007 a MAJOR national telecom did its quarterly network capacity planning by having local analyst export excel reports from individual controllers. Then perform manual roll up, cut and paste into regional excel files. Then same for three more levels to national.
I remember in my first adtech job, had to debug an ad request api error for the smoking remnants of one of the late 90's tech darlings... didn't find the problem, but they were running PHP 3 on their backend in 2008
Another one, when meetup let me go, I had a bunch of automated email reports, ETLs, fun bots etc. running on my dev box sitting at my desk that I used as a footrest. I did tell them how to access the code to salvage.
About a year later friends say they finally retired my box
If I’m honest, I have left a sea of technical debt at every job I’ve had. But in my defense, managers basically beg for it.
We were building an atmospheric obs system with several generations of receivers. Old ones had been running for yrs, so ppl left them alone. All of a sudden, they stopped talking. We rummaged through closets til we found the KERMIT protocol manual & learned some macgyver skills
[we had literally tried absolutely everything and were about to accept the loss of lots of expensive equipment... just couldn't get it to communicate... glad someone was like "maybe someone left some breadcrumbs in these dusty closets"]
trying kermit was the last-ditch-of-last-ditches effort, and we wouldn't have thought of it if we didn't find the old manual and say "what the hell"
Please do not keep me hanging...what was wrong? Did the Count overflow and needed Kermit to fix it? #RealQuestion #BadJoke
hahahhaa :) awesome
I worked IT support in college. I helped a professor convert a dataset that was collected by a local city. It needed to be converted because the city kept all their data in Lotus. In 2007.
you would be shocked at the level of continued dependence on lotus at many financial institutions to this day (I’m in my mid 20s I should not know what lotus even is)
Came here for the Lotus Notes, didn't fail!
Here's your .id file now crack on haha
Lotus 1-2-3 was released for 32-bit Windows in 2002 in Smartsuite 9.8 and got fixed until Fixpack 6 in 2008.
archive.ph/20120719070454…
It's not *that* long ago. Of course, nothing says that they have to be running 1-2-3/Win (which I always felt should have been named Win-2-3).
Lotus Notes, circa 2010-2012
Big 4 accounting firm I see you
5.25" floppies, running MS DOS 5, in the mid 2000s to drive manufacturing for a large multinational.
An old coworker told me that a large national retail company was doing its ordering and inventory entirely in Excel with no VBA or other code, and no backups.
I ran a 300-person brain training lab. Students did intense timed memory exercises on laptops. One day they started complaining: tiny ants crawling on their hands/screens, messing up data collection. That night I put laptops in cabinet, filled w bug spray and slammed the door.
If you’re wondering I did clean off the laptops the next day including the dead ants. This actually worked. No more ant infestation and the experiment went on for months without a hitch thereafter.
So you can proudly say you've done some "Live debugging"
Oooh do I have some stories. One group had to perform complex calculations for all our 60k+ clients, had no IT support and were entirely staffed by 22 yr old business majors who had never heard of databases. (1)
They created an insane Excel sheet, 1 copy per client. By the end there were TERABYTES of Excel sheets, each containing tons of sensitive personal info, stored unprotected on a giant shared drive. Management didn't see this as a problem. (2)
By "the end" I mean by the time I left the company. I am positive nothing has changed. (3/3).
sickening
‘terabytes of excel sheets’ is a four word horror story
I can also add “thousands of Access databases”
Same company
why use one db when thousand do trick
One db good, thousand is better
Information retention overlords are up in our biz on this stuff finally....I'm not saying I'm grateful, but something had to be done with those 15 year old shared drives no one had access to anymore
Er, which company? 😬
Not one I recommend working for 😉
At least they’re not stored in hundreds of boxes of badly labeled floppy disks.
(No comment)
they call me the king of the spreadsheets. got them all printed out on my bedsheets.
If I had to work with terabytes of excel sheets, I'd delete myself
I once had to do a bunch of sorts on a mini mainframe. At the rate we charged computer time that one run would have more than paid for the computer. I asked the vp if he wanted to adjust the bill. His only Q: would it take longer to do by hand?
Sub <finance teams> with any and all who are drawn to the greatest database of them all
I am a Business Major. I started learning how to code #rstats and Python because of a 30 sheet workbook that was intended to solve an timetabling optimization problem was not reproducible. It hit me, the optimizer is not optimal. How do you get to terabytes of Excel, how Sway?
Effectively it was a naive form of "sharding", they probably had ~60,000 separate Excel 97-2003 workbooks, the total storage of them in filesystem could conceivably add up to terabytes.
Oooh no this is the stuff of nightmares. I had to clean approximately 3000 Excel sheets, I told the team it isn't sustainable. We need a database.
I worked for a large corp whose revenue reporting depended on dozens of interconnected Excel workbooks. A handful of query workbooks pulled data from the POS system using an extension that only worked in Excel 2k. So they had VMs w/ Excel 2k for those...
All others sourced from the "query" workbooks (and each other).
There was a team of ~30 who came in at 4am to kick off the query workbooks on the VMs, then open the other workbooks in a very particular order & run the macros that populated data from their various dependencies.
Oh yes, this very much rings a bell
Shudder
this story made me feel kinda sick, tbh
thanks for sharing it!
You’re making me feel better about our current attempts to replace access and excel workarounds with an integrated information management system. Amazing that we get as much work done as we do 🤷♀️
business major here, i only know microsoft access but even then i literally do not know how it works
I know you wrote a thread, but just this one tweet is already the stuff of nightmares.
I was tasked with building a streamflow model at the @USGS in Urbana as an intern. Nice, modern stacks Had to fix the rainfall ETL once, the hydrologist on staff smirked and gave me his FORTRAN77 book said “good luck kid”
Been there.
Homegrown web interface for deployment that had no cli interface.
All of financial trading ran on AIM until they shut it down. Weirdest interface to use at work.
Technically not my former company, but we were monetizing access to government data. These data were technically available cheaply, but on top of being horrendously messy, were stored on magnetic tapes in a 1975 IBM mainframe. '75 because everything was encoded in EBCDIC.
library automation company used by dozens of major US universities rely on a 2 million line legacy Cobol code base that no one at the company understood or was capable of maintaining
*Definitely not me*, but a friend built:
Google Sheets -> SQL table -> Prod ML model
It's still running great. 🙃
cutover script 😅
Ooooof literally though
How do we revert the revert of the revert
Nightmares
Okay but how did they put the label on the cable?
Very carefully.
Pen & paper, Google Sheet + Zappier is a classic. For the itemised pay of thousands of people. It broke in creative ways _every_ time.
Otherwise, not a company but a public hospital the patient files are on a… Oh so, you know that story already?
Student-lead corp, ~2000. All data is on a MS Access db (very modern) with a clear data-object structure, etc. But every method, class, etc. named through to graphic adult-themed metaphor: contracts are… “movement”, subcontracting involves several parties, pay is “consuming”.
I remember having to ask “What does _that_ mean?” to horrified classmates because I didn’t want to google it, but I needed to understand what that class method did and I had no idea how many people were involved in… ::gesticulates:: at screen.
No one believed me.
I had a large airline customer years ago who migrated a lot of services from one VM to another but didn’t want to update all their service code that pointed to “localhost”. This is where I learned it’s sometimes possible to point localhost to… another server.
Took weeks to debug a problem and figure out wtf went where.
Same customer who mixed tabs and spaces in all their Python code. FML.
This project also included code comments in multiple languages. There was a _lot_ of Google translate involved in debugging.
Multi-lingual comments are a new one for me, but in retrospect, this should probably be way more common than it is.
yeah; I often talk about coding being anglocentric down to the var and const, but I hardly ever see processes for routinely translating code comments
I expect there is a lot of valuable non-English commentary that just doesn't get written.
Some people live to see the world burn.
Until I set my tab stop to 8 it realllly confused me as to how any of the code actually ran.
hairs standing up on my neck now
I’ve seen a firewall where every computer had to set its gateway to 127.0.0.2
it's just a change to one file. we'll do it once and not speak of it again.
In 2016 working for a company that made the complete billing (1k+ B2B customers) via MS Access running in Win 98 on a server next to the coffee machine. 😱
I vote for @MicrosoftTeams...
(probably not used at companies) but the jankiest pieces of tech I've seen are these MS Word plugins that come with reference/paper managing software (Mendeley, Endnote, Zotero, you name it)
Oh, and any software related to German banks (or maybe just DKB). It's so bad that it's not even funny. Like if you want to set up the mandatory TAN app, you have to use the TAN app to be able log in and let them send you a temporary code to your physical address.
Of course, this doesn't work, so you have to email them about it. The letter they will then send you takes like 10 business days, and you have to do it each time you get a new phone, because the app doesn't work if you restore the phone from a backup like any other app.
You're just talking about the Word plugins, right? Because standalone @zotero is awesome.
Yeah. They often screw up the numbering big time. Or at least that was my experience when I used them back then in collaborations (we had to use Word). The standalone app is okay. That is, it is not amazing but it does what it is supposed to do.
Those software companies employ enterprise sales professionals. These plugins are indeed used at companies.😂
The funny thing about these plugins is you can literally see the code running in slow-motion in real time
The booking service for a major shipping company that was a stack of three different generations of Oracle technologies.
Layer 1 and 2 were supported, layer 2 and 3 were supported. But the whole stack? Nope.
It was 3 years into a six month development program. I cancelled it.
Safety system built on Lotus Notes.
O. M. G. Lotus 🤦♀️🤦♀️🤦♀️ Yes I’m old. My first portable computer was the size of a carry-on bag, ran on DOS and weighed 30 pounds. 🤣🤣🤣
Early on I worked a networked Wang PC. Then I had the Compaq that was the size of a sewing machine, double floppy drive.
The first Compaq computer
The first Compaq computer was the Compaq Portable, a suitcase-sized clone of the IBM PC which was the first legal clone with a high degree of compatibility.
dfarq.homeip.net
Omg that was the one! The keyboard was attached by an old-fashioned looking phone cord, and it snapped onto the bottom of the unit. There was an actual luggage handle on top! Floppy disks - TWO!! 🤦♀️🤣
I actually may still have it in the garage. 😂
My company at the time tried to upgrade to Oracle 11. Inbound and outbound loading docks were offline for 5 months, developers had cots in the halls and when asked for a backout plan, the VP said "we don't have one"
Luckily I wasn't in that division
A company with several thousand installed locations ran their entire payments clearance system (multiple parties get paid for each transaction) on a single FoxPro database/app sitting on a Windows server box under a guy's desk. As late as 2015. Maybe *still*.
In all honesty, that seems far more stable than a modern Javascript front end talking to, say, mongodb.
I think there was some pipeline at Goldman Sachs that involved sending a fax somewhere in the middle.
A git repo within which were nested several dozen more git repos.
A Quip machine.
Previous company calculated all of their new sales data (new contracts) with a team that entered the contract details into an Access db. There were four or five members but they could only work in the DB one at a time. They finally upgraded to SQL server a year after I joined.
Worked at an insurance company where all financial data was stored on 1 dusty Windows NT 4 server. The day before closing the fiscal year it crashed and couldn't connect to the AD. Had to search for drivers and OS patches on archive.org and use 3x💾 to boot from...
(No comment)
"Had to search for drivers and OS patches on archive.org" .... dear sweet baby Jesus. I didn't even know that was a thing.
Now everyone reading this go donate to archive.org
NT4 was a great system for its day. However, that day was before the turn of the millennium. 1/
My war story: I was responsible for the care and feeding of a System 360/65 and a Burroughs B6700. One fine day, a Grand Pooh-Bah dropped off 8 boxes of cards, directing me to feed them through the 360 and submit the generated reports. 2/
There was no printing at the top of the cards, but I was able to read ALGOL code. I asked my manager to tell milord what I had found, and I could hear the blowback 3 offices away. (He quit the next week.) I fed the cards to the IBM, with two boxes of error dumps as result. 3/
I then fed the cards to the B6700, which produced a series of entirely legible reports. I sent both to milord, along with his original cards, and asked why he had me feed the wrong computer. He and his retinue appeared 30 min later, shouting that he never made 4/
such mistakes and it was obviously my fault. He fired me then and there, watching me clear my desk.
I have never been so happy to be fired, and told him so. Everyone around him was certain he’d have a stroke. Karma is a wonderful thing.
That was 35 years ago now. 5/5
Javascript?
hey, we ported it rather quickly to 5
I was porting Perl4 to Perl5 in 2006. After I’d started playing with Moose:
ouch
This was around 1996. There was some 4isms in it.
Mostly module'submodule stuff.
I think we finally got rid of our Vax in 2015 or so? Can't believe how long they limped that thing along for very minimal use.
I remember watching two rows of VAX equipment beying decommissioned at a Dutch hospital in 2001/2002 and that stuff was ancient back then 🥺😳
Way back when I was in high school (80's) I got a job that was basically writing Lotus 1-2-3 macros to make Lotus 1-2-3 into a database. Fortunately they listened to me that they needed a real database or they'd probably still be running those macros today.
Lotus 1-2-3 *was* a database. That was the "2". "1" was spreadsheet mode; "2" was the DBMS functionality (queries, sorting, etc.) "3" was because it could draw graphs, too.
That was the entire point of its name.
Far from the worst in this thread, but in high school (circa 2005) I had a job shadow with a temp agency that had all of their payroll and scheduling running in an MS-DOS virtual machine inside of Windows XP.
a friend of a friend told me her soon to be retired dad was one of the few remaining COBOL programmers who was responsible for keeping social security disability checks going out, and they had no plans to either upgrade the system or train new programmers.
This is the dirty little secret of most US federal and many state agencies.
My Dad worked for a company whose in-house medical insurance was run from a Borland Paradox database some guy wrote in his spare time. I met the guy at his retirement party that my Dad attended 1/
Nearly a decade later, I ended up working for the same company & met the supposedly retired guy, still working… the company was *still* running their medical insurance on his little Paradox DB & had him now working as a contractor! He must have been well into his 70s! 2/2
Saw at Fortune XX company a piece of mission critical software running on MS DOS 6.22
My ADHD evaluator last year had a set of custom programs & database he runs on a Windows 2000 virtual machine. Has only keyboard/mouse/display out of the vm.
He would manually edit the sql tables if some data entry was wrong.
Does the JVM count?
A hospital had a pill dispenser in their pharmacy based on a Compaq 286.
In 2014.
Optometrist had an eye laser surgery machine hooked up to a Win XP host with internet access in 2011. 😬
It would be offline, is what.
I’d also accept not a desktop OS with a mountain of known vulnerabilities.
Gotta get images off of it somehow, but “email them to myself using the web browser” ain’t it. 😬
In 2012, I saw a whole hospital on XP with Novell directory service. At least they didn't have to worry about Active Directory getting hacked....
Sonogram machines - custom OS
MRI - custom OS
CT Scanner - custom OS
A desktop OS with a web browser and local admin for the user was definitely not “the way to go” for anything involving patient safety.
XP Embedded was / *is* on a lot of IV infusers.
I’ve seen a few embedded Win things, but at least they weren’t local admin with desktop apps 😬
There was a blood analysis thing that ran on unpatched Win2k. The vendor refused to patch and would cut support if anyone patched (as actually is the fucked up standard by most med vendors using Win), but at least that couldn’t do direct harm if something went wrong.
FDA changed authorization rules about 5 years ago to allow security updates/ OS upgrades, IIRC, but manufacturers still claim "costs too much to recert"
I once worked for a multibillion dollar corporation that was running its entire datacenter's border traffic through a five port Linksys switch from Best Buy.
I still have a 98 PC in production, can't be virtualized. Runs a CNC saw whose maker went out of business shortly after it was purchased.
Cost like half a mill new, would cost about 4-5mil to replace now.
They are aware "eventually this will die, and that's it"
This is a common problem with old CNC machines. Pharmacy robots as well. Hell any robot. I know of a certain popular humanoid robot that runs admin through https with no ssh, but is still vulnerable to heartbleed!
A friend's custom guitar business used a Commodore with a 5 1/4" drive to drive a CNC.
One day I will tell a story of a Windows XP machine that hung on far too long in life for somewhat similar reasons.
I recommend a qualified therapist. I know a trauma expert if you need.
We have an entire production line dependent on our ability to keep this PC operational. The clone died recently and this is all that is holding it together. Our attempts to duplicate the output have failed but management says the line will be upgraded next year
And I thought needing to have PCI slots was bad (I need PCs with PCI slots for one specific fieldbus card).
These days you could probably duplicate it with a USB converter box and microcontrollers.
IIRC you can still get motherboards for modern CPUs with ISA and/or PCI
Yeah, there are options, but they often kinda suck, and may not line up with smooth "fleet sales" through IT. In fact, there is a PCIe version, but I'd have to write some new driver wrappers for it. It's all a matter of saving the effort and hoping the equipment hangs on :D
So is the whole system proprietary or just the connections?
Controller card. ISA...
Parallel port? Serial port?
What I am asking is how is the computer connected to the machine?
Don't remember, I've never had hands on it myself. MSP, different engineer's account. I just consulted.
Ahh, was under the wrong impression here. Thought I could help with a solution.
My Uncle runs a bunch of nearly dinosaur era machines. It bothers me because he won’t let me dev a more modern control solution. Something about not knowing how to operate them.
I'm sure a hardware hacker with the proper skills an a couple hundred in Pi bits and pieces could work out a new controller... but that would require taking it out of production for rev-eng and testing, which means it may fucking break 😸😼
And that still leaves the totally custom software package, which won't even start under emulation (we tried...)
I would suggest actually waiting until the machine is stopped and interface the connections to data log and track all of the operations to decode the signal under normal operating conditions.
If there is an emergency protocol, you only would need the code once or so.
Ya, not really my area. I'm more of a communications infrastructure and security kitty.
But I certainly have some hardware hackers in my circle if/when the time comes I'll make the suggestion.
That’s a lot of money to lose when that machine has unexpected down time.
As far as controllers, there’s a few that are better suited than Raspberry. You could even write software to signal the supervisor in case of a real problem.
Wood and metal fabrication shops are the worst. Windows xp and earlier systems for days.
I worked at a place that relied on one person who operated "the machine" that was used in processing contract initiation paperwork. If they were out sick or busy, you couldn't start work. You took your paperwork, if you were lucky they'd do it while you waited but probably not.
Anyway, "the machine" turned out to be an electric typewriter and I revolutionized the job opening process by showing people how to use a pdf editor. The only drawback was that I became the operator of "the machine".
We had a product which depended on learning from responses provided by humans, if you can believe such a thing!
I worked at an online travel company, and their entire travel booking flow was implemented in a single function with thousands of lines of code, called DoAllTasks(). It somehow managed to process thousands of bookings per day.
@rayploski @cbredesen @KonnectedKris
DoAllTasks(). Wow, I blocked that from my memory.
Iiiits back!
An abomination of 30KLOC (if memory serves) of VisualBasic. 😅
By the time we all left, DoAllTasks() simply orchestrated to functional areas. Took lots of effort if I remember correctly.
I thought we scrapped it with the Java™ migration, but if you told me it was still powering cruise bookings somewhere, I’d believe you.
Let’s be honest, if we saw the Java code that replaced things, we’d probably be equally horrified today as we were by DoAllTasks() in 2001 - if not more so.
BroadVision anyone? What a PoS!
But JFC what. a. nightmare.
Honestly though this isn’t even in the same ball park when compared to a project that my old company worked on … a legacy system rescue project for a FTSE 100 company… 🧵
Imagine a stock control system written in Pick Basic (look it up) that has been hacked over and over again for two and half decades to support the acquisition and merger of 100s of small business over it is life time. A reference implementation every known anti-pattern to man.
It ran (by ran I mean melted the CPUs constantly at 100%) on two HP Mainframe Servers whose hardware and O/S were both obsolete and unsupported. It’s database was NoSQL before SQL became the trendy new thing. All environments, including dev ran on the prod hardware.
There was one dev that had been repeated bribed to continue working part time on the system having reached retirement age and a couple of Pick Basic contractors who knew just about enough to recover from crashes and make very small changes.
The “pricing engine” of the stock system (which has a myriad of unique pricing code implemented with liberal usage of if/then and goto statements) was such a convoluted mess of spaghetti code that nobody knew how the business logic it worked, or how it was meant to work (specs)
The whole thing was a ticking time bomb and an existential risk to the biz. The CFO confessed that the only way they knew they were pricing to make a profit is that there was more money in the bank account at the end of each month than they started with.
Sure it's noSQL, because the Pick Operating System predated IBM's invention of SQL by nearly a decade.
Pick: 1965 (as Generalized Information Retrieval Language System, GIRLS, on S/360).
secure28.securewebsession.com/jes.com/cdp/cd…
SQL: 1974 (as SQUARE).
LINQ vs SQL. Learn the difference - Chubby Developer
If you are confused between LINQ and SQL before starting your web or desktop application, we are here to shed some light on it, which will going to help you in understanding the concepts of both terms. To make a long story short, LINQ is a framework used to develop applications mainly rapid applications, which […]
chubbydeveloper.com
The database is an integral part of the OS, as is the BASIC. It was a very powerful system in its day & a strong competitor to early UNIX™.
Several derivatives are still around:
rocketsoftware.com/products/rocke…
Home
(no description)
openqm.com
(No comment)
what is DoAllTasks() but int main(void) persevering
Encapsulation is overrated.
I worked at a different travel booking company where the website talked to the booking service using Java RMI. Every time we deployed to the website, if the shared code had changed, we had to stop taking bookings until we also updated the booking service 😂
another travel booking company, we had agent terminals on the GDS mainframes whose interface was written by a company out of business for years, given to us as 32 bit ELF libraries compiled on RHEL6 (1999!) Friend got them running 15yr later in a 64bit JVM by hand relinking them
Impressive but...
I work at yet another online booking company. It's not as bad as these others, but I am very fond of the name of the login function that does SetUserStuff().
Well, maybe not as bad. We do still have classic asp vbscript running in 2021
I’ll see you that and raise you classic ASP with *JScript*
My first BASIC interpreter didn’t have local variables. Subroutines weren’t so terribly useful in this case. Maybe those guys never advanced from that level.
If only we’d had GOTO statements and line numbers…
At least it's implicitly documented with a relevant function name
Hey, at least they named it accurately. Good documentation is the first step to good code?
AAAAAAAAAAHHHHH!!!!
An ex coworker told me he went to work at IBM and they had a .NET WCF endpoint which was about 80k lines. 1 function/endpoint, with 100s of if statements to manage financial transactions. He tried to break it up into something manageable but management shot it down.
I came here to say WPF, WCF, and Silverlight as a train wreck of a tech stack choice I saw coming a mile away.
Sadly, our director drank the kool-aid. We know how that turned out.
at one the places I worked at, the entire billing, invoicing and finance system - was one PHP file, which was a giant tree of if/else statements, all relying on register_globals and there was one comment in the entire file at the very top, which was //filename.php
I respect the naming convention here
I think I would have nightmares if I had to work on something like this 😱
Sweet Jesus
I ran a wind power and weather monitoring station in the Guatemalan highlands on an eepc for years that the client in Hong Kong communicated with via Skype chat. You'd chat with it asking what's the power output, send me logs for, or what's the wether and it dutifully responded.
all gold. Workd at a smb firm, we once virtialized a win95 box because it handled the accounting database, and it was held together with dust, cobwebs, and dumb luck. not sure how the original hd was still spinning. they refused to pay for db upgrade, so they still use win95.
In the year of our lord 2017 I decommissioned a Windows NT4 VM because I was able to replace an Access Database with a module in ServiceNow. It was our inventory.
Fed gov requires online form to be filled out. Won't share db with state gov. So submitter takes screenshot of fed web form and faxes to state. State hand enters into Excel file. Turnover at state, so every year or two a new resource modifies the Excel format.
List of items need to be entered for each site. Too hard to repeat site id for each row, so they enter once and leave other rows with blank id. Depending on mood, they fill in the first or last row, with no way after the fact to tell which.
Wait, what? In what programming language?
Learning how to use this must have been such a pain for them!
I once worked on an Italian computer. (Yeah bad idea). Among other things one of the packages that came w the computer (converting ascii into some other format maybe? ) crashed w prob 1.0 if the input included one particular ascii character 🙄 😅
Sorry for the bad ethnic joke. Turns out it was a British computer with an Italian name
This last detail is the cherry on top
a mix and match open source data platform which was supposed to do real time operations reporting at customer sites... but ran batch processing on a nightly cadence.
oh no. no no no.
I once had a process break 1/3rd of the time.
Turned out it was because at some point there was .NET framework pretending to be an IBM 3720 terminal talking to a mainframe, and it didn't like lowercase letters.
Realise you wanted more given the other replies 😂
Oh, we've seen *many* votes for Excel. You're in good company.
I’m glad I’m not alone, but I’m sad at how broken the world is …
Excel disguised as an ERP.
Hard same.
Excel to fill the gaps of your bad ERP...
Blow it up. Blow it ALL up.
SPARCStation-20 with a SDLC sbus card to communicate between a western US baby bell and Dun & Bradstreet. All customer credit ratings checked via this system. No earlier than 2007. They had no spare SDLC card.
If you know who these folks are...I have a spare SPARCStation-20 and SDLC card.
How much?
Does using Great Plains as a technical support ticketing system for techs (field and internal) supporting telco for a couple thousand standalone facilities make your skin crawl, or is it just me?
Worked at a company in the mid-2000s running accounts payable off a FORTRAN-based ERP called MANMAN on a 1970's era HP 3000. We didn't have a FORTRAN programmer on staff and just hoped that it all just kept working.
If I knew COBOL, I'd damn well not be writing it for free.
I worked at a place where data queries we're done by calling an overseas team who looked at a dataset, typed rows into Excel, then sent it over email 1-8 days later, depending on size.
No metrics were stable over any pd. Constant source of internal arguments
Machine learning pipelines
That looks about right 😂😂
whatever your answer, it is outdone by what appears on this website, which has been around for years.
thedailywtf.com
US Courts using Lotus Notes….
Used to work with a guy who worked at well funded VC dot com company back in the day that was cratered b/c someone changed a string literal in the code that went on hardware to be shipped but didn’t update the checksum the hardware validates. Every unit off the line was bricked.
To be fair: the company was running out of $ already, but had enough to produce the units. Also, clearly the company had horrible test and build processes, so it’s hardly that one devs fault. Still, most people would think changing a string literal would be innocuous...
I'm always excited to hear about the computer under someone's desk that can't be turned off because it's responsible for running some automated script that keeps the business running and isn't checked into source control
I've had former employer call me like "hey I know it's been a year but do you remember the server number that python app was running on"
My personal favorite as well. How about an Excel PowerQuery masterpiece built as a POC that ended up powering the Sr. Executive dashboards for a year. Btw — Took an entire IT team exactly one year to reprogram it with Informatica and Oracle.
My account at a company I left was active for more than 12 months because they never got around to creating a service account for our CAD license server when I was there and disabling me meant breaking it. They didn’t even change my password, meaning the server was never patched
My favorite is that computer, containing some secret keys and scripts, and is reachable over ssh but noone remembers where it is, and the computer’s owner has quit the company and then died.
Beyond cloud computing there is ghost computing
All of the explosives detectors that magically appeared on Army bases within a few days of 9/11 and didn't make a peep when you pointed them at a hand grenade.
Thought you would appreciate this thread.
We ran badging for the entire company off of the old laptop of the security guy when the company was a wee baby. We could not patch the laptop in a way that required downtime.
it was the ldap server wasn't it? but like the old Apple ODS version of ldap
Was it the google glass thing?
oh god. I think the ODS thing may have been mac mini too but apparently there was a mini that was used to tunnel access to prod network from the office
And we all miss the whale. :p
Have worked somewhere where we only found out that two clients' websites had been running from a PC in the corner when we threw it out.
Happened to us to. Something like 80 static pages with a bunch of Perl.
No backups.
I recreated the site using IE caches from PC’s that had visited the site. It took me a couple of days.
That was clever!
I think I installed that server, and I think it was an early openldap test machine moved to 795 Folsom. But I forget. That was 15 years ago.
There was also a pile of servers running test scripts in the design department, high-jank.
Or hey, maybe we did that a few times!
I think this was called “the load-bearing Mac Mini”
the best part was we couldn’t file a ticket about it because the jira or whatever was hosted on the mac mini so we had to just figure out where it was every time
heh i am sure nothing like that exists at twitter any more
I was asked to extend Layer 2 because a piece of legacy code called PowerBuilder (circa 1998; literally last millennium) didn't support routing of any kind
After about 4 months of struggle with the PM they gave up and retired the code because I wasn't about to span L2
The flight computer(Collins aerospace FCC-730) in brand new 737’s first debuted in 1996. Part of the reason the 737 MAX software fix took so long is they have to be careful with the code not to overload the computer.
I worked for a microchip manufacturer last year whose servers were still running Windows XP.
I was not high ranking enough for details but my understanding is a hugely important piece of code signing for BlackBerry was a single machine.
Timeslips
Wrong, Adam Schiff evidence 😂
A bespoke (non-IT) employee-created Access database in runtime mode which was essential to the daily running of retail operations & created millions in annual revenue. Employee left, took the source code, still relying on it ten years later. It was exactly as bad as it sounds
ran fulfillment automation for 2,000+ daily orders through an ngrok server on my laptop for 8 months bc i couldnt figure out a "it works on my machine" bug
Access databases.
Word Perfect. In 2018.
A CMM machine from 1995 that's still in use today.
When I was a kid in high school (2010 or so) my mom worked at a small accounting firm that still used a main computer and dummy terminal set up. I love fallout
Honestly though this isn’t even in the same ball park when compared to a project that my old company worked on … a legacy system rescue project for a FTSE 100 company… 🧵
does an elevator controlled by an Apple ][+ count?
Oh god.
Does the fact that basically all of the US healthcare system is run on EHRs that are using MUMPS count?
I mean, when the CIOs of health systems will send out email saying that you can't put anything in the EHR during the "extra hour" from daylight savings time, that feels pretty "janky"
Seriously!? 😳
To be fair, they do shut the systems down during that extra hour to ensure no changes are made during that time 😂
Thanks!! (I’ve worked mainly on scientific & engineering software — including my share of time issues 🙃 — but not for healthcare, and I’d never heard of this.) Very entertaining!
link.medium.com/mpJltG352lb
What's MUMPS?
A coding language and database system from the mid-60s
When liking a tweet is combination of laughter and horror.
Thanks
Lol I wrote the white paper related to handling DST for one of the big EHR providers. Janky? Absolutely. Still better than paper? Yup!
I worked somewhere recently that was working on integrating a MUMPS database into their system, because apparently it's commonly used for financial stuff in other countries. Somehow it's supposed to be the fastest database for that?!
That's a newer version of M (MUMPS) than the one most of the EHRs still use. I've heard it's fast, but I can't believe that something based on tech from the 60's is still the fastest for financial stuff. Sounds like a copout answer to avoid migrating billions of transactions
I never measured it personally. I just heard about it being the main database behind every large Asian bank because it was the one that could handle the most transactions per second. (they have a LOT more TPS than in the US from what I understood)
Canada too!
Epic does, Cerner does not.
Speaking of which… bizjournals.com/kansascity/new…
Oh come on, many of them moved to the jvm hosted mumps, jumps.
It's exactly as you think....
You think its just healthcare?
Last time I was at one hites their reservation system wasn't even GUI.
Came here to say MUMPS. Also, in terms of frontend, a lot of surgeons prefer to use Cerner and refuse to use Epic. So nurses and support staff have to toggle back and forth between the two often-incompatible monopoly EHR systems. (This happened at 3 hospital systems I worked for)
A combination of low-RAM local machines used to run long training jobs, combined with a clear-desk hot-desking policy. Would put laptop in a locker and hope the battery held out till the job completed. …And hoped the fire alarms worked.
Wooooow. Very Mission Impossible.
An Access VBA app for order tracking and invoicing on a shared folder that would break anytime more than one person tried to use it at the same time
I work in television, and the answer is the teleprompter. Easily.
We scroll through every script before each show just to make sure there isn’t something janky with the copy. Even still, there’s always a chance of a malfunction. On air. For a live program.
Three pages of Perl I wrote
some debian based firewalls I built :)
off the shelf: radware load balancers
Did some work with life insurance company whose millions of variable policies, requring a daily value adjustment tied to equity and bond markets could not be managed well with the decades old software and hardware. Their solution was to hire battalions of people to do...
hand calculations to keep thumbs in the dike, with a tiny number of old timers who could keep the ancient crap from collapsing. One mid level person recognized that if she were to get hit by a bus, it would be a catastrophe for the company. It reached the point where it....
didn't matter who called her when she wasn't in a good mood; she'd pick up the phone,& snap "WHAT the F*CK do YOU want!!" I was only a tourist passing through, so it was all entertainment to me. Eventually, of course, they updated, and she quit with a pile before they canned her
An identity solution from some random vendor in a strip mall in Florida.
Pulse operating systems. Absolute garbage.
In the early 1990's I operated a Data General Nova computer from the 1970s for a publisher of catalogs. The president wrote the code and would not part with it, though at one point he started to port it over... to a Commodore 64. Paper tape, reel to reel tape, massive hard discs.
Microsoft Windows, prior to Windows 11 (I think Microsoft finally got it right).
ROFLMAO.
There's enough COBOL legacy codes in various companies that COBOL programmers can earn 6 figures salaries.
We ran an Excel sheet “model” in production with Apache POI.
We "developed" what basically amounted to a cardboard box with an LED inside. You would set your phone on top and it made it "easier" to scan documents by taking a picture.
All I can say is, at least it was many years ago. 🤷♂️
I know several race horse breeders that have all their data in an MS-DOS Ryan McFarland COBOL system. I know, because I reverse-engineered the data structures in 2004 after the original developer committed suicide.
Unfortunately, after I spent weeks reverse-engineering the entire system and building a data extractor to move the history to SQL, those breeders decided it was too much trouble, and I spent the next several years supporting that system in a nesting doll ...
... of Windows 7 (later 10), running VMs of Windows XP so we could get a CMD close enough to DOS for RM-COBOL to run.
I'm not in that biz anymore, but I got an emergency call 3 months ago from a veterinarian in Texas who trashed his system and needed to restore the backup.
They still use it. EDMS - The Equine Data Management System.
I'm one of probably 10 people on the planet that understand it well enough to recover a broken one, and large swaths of the top Thoroughbred race horse breeders would be totally screwed without it.
Wow. I knew about the hospitality and banking industries, but horse breeding IT teetering on the brink of collapse is a new one.
Two of these same farms had me install fiber optic lines between the office and the breeding shed to improve the multi-user network performance of the COBOL system.
That might be the most high-tech/low-tech thing I've ever done.
This is ludicrous. Do they have no concern for what happens if suddenly NO ONE knows how—or wants—to support it?
Trust me, I’m not disagreeing with you. This situation is nuts, but it’s not even in the top 10 list of most ludicrous things I’ve seen in the horse breeding industry.
That should be it’s own thread, but the industry is small. Even if I don’t name names, somebody would guess.
For that matter, I'm not arguing with you either. It sounds like you tried to help them migrate and they turned it down, so what can you do.
Rhymes with “mob maffert”?
Let’s not speculating, or I’ll need to delete this whole thread.
But to quash any rumors, I’ve never worked for him and have no idea what software he runs. EDMS isn’t the only system available, it’s just one of the last available options that isn’t a subscription.
Dang, what a tail (tale)!
OS/2 Wrap that ran voicemail and fax machines for a legal firm back in 2010.
Got hired as the computer guy for a boutique hotel.
Was shown the IBM tape backup machine used for all system data.
Was shown a stack of backup tapes.
Every one of them had been reused *countless* times.
When I tested the archives not a single one was readable.
Laptop under someone’s desk that ran code written by an account manager
yahoo finance in a fintech startup
I use Yahoo Fin all the time as a retail and when am not at the TF
But the straddle view in vol was revolutionary!
more common than you’d think lol
A $1B fund on an algorithm I wrote on my blog six years ago.
I kid. At least my code is readable--unlike the PWC consultants' drivel that wrote Deutsche Bank's CCAR analytics code.
you bite your tongue
Pretty tough to top.
i mean... i know investment banks that depend on yahoo finance
I work in a company where we're encouraged to do research from it if we need "public" data. It's a 1T+ asset manager.
entire pitchbook industry would grind to a halt without yahoo finance.
This puts into perspective just how much I oblige my product and RFP people's requests in the middle of a work day. I should tell them to look it up on fucking yahoo finance from now on.
MS Access at one of the world's largest banks
Id pay for another dependency but I don’t know they aren’t using yfin!
A login/authentication server to a very popular MMO went completely dark.
That's when I discovered it was an i386 dell PC under someone's desk.
old school linksys wireless router for firewall at top 5 university.
Jenkins job DSL & groovy scripts produced from a series of Go templates, which in turn were described using another internal YAML DSL. Groovy loaded via shell exec caused one of the worst String memory leaks I've seen in my career, on the order of 100s of gb
A few companies ago our DB setup was so messed up that we had a script we needed to regularly run to fix things locally called db_sadness.sh
We ran the front offices of a busy FBO with rental aircraft on an apple I, and the air ambulance co on pirated win95. Maintenance for both was a microfiche in the hangar.
Until the day the new avionics guy called MS tech support... stern letter and we had to pirate a new OS.
A Gestetner
The server inside the wall.
Outside of Ansible / Red Hat I've never seen a company package their Python code.
but every single company depended on Python code for critical tasks like provisioning machines, backing up databases, verifying schema changes, etc.
They had an online form people could use to report product issues. The forms were sent to a printer and someone took the hard copies and typed the info into a database.
They should have used ocr.
The issue they were solving for was a complex 8-bit coding system for data. Even if we ignore updating their database, they could have replaced all that effort with some drop downs and a touch of code.
(Possibly sensitive)
I really hope it was just running the coffee machine
No not the coffee machine, it was a bona fide secondary or tertiary control system (it was proudly describe in one of my .de computer mags in the 1980s) but not related to the rods or IIRC safety of the nuke plant.
Small printshop using a Silicon Graphics O2 workstation as a print server in 2010.
Those O2s had STYLE, though.
Developer (who had long since left the company) had code running on his personal box in the dev area that was part of the production site via reverse ssh. Now, you're assuming this was a small company. It Was Not.
I worked in a place that had a custom Java messaging framework. This was before Kafka or even RabbitMQ and the best option at the time sucked. So they built one. We would get stuck or missing messages, data was inconsistent between data centers, etc. Could have been worse though.
The folks who built it were very smart and they did what they could I think. Just a big problem to tackle with a small team who also had products and features to build.
Also, the best commercial option (if I've gotten the time period right) was tibco, licenses of which cost more than I made as an engineer
Lotus Notes. IBM ran everything on it.
Absolutely everything. I was there when they bought Lotus and golly that conversion was fun
Network of @Windows 3.1 machines running recursive Excel VBA macros
[REDACTED] 5th amendment claim against self incrimination.
One desktop running as headless server, with windows 2000, running beside the foot one random employee. This system runs at the time all part of the erp of an entire multinational steel company.
I once worked for a company that paid teenagers to rip capacitors off of our $120 servers with pliers to make them work. They did thousands of them
Come again???
I worked at a health care company responsible for notifying millions of hospitalised patients of surgeries, prescription info, & emergencies. One engineer built the application complex enough where only he understood how to edit the main java files to guarantee his employment.
My hero
Air conditioning for the whole building relying on an ISA card in an ancient 486 PC running Windows 95.
Adobe Text Search Engine that had a lockout password that required a Board Member at both companies to call support to get the website back up.
A compac brand desktop PC from the late 1990s, still mounted on a shelf in a rack, which contained a hardware locked licensed piece of software required for a core business process, in 2017. Software vendor was out of business.
~2000 line SQL query maintained by a dedicated contractor
That’s a dream job right there.
Right, now imagine that spagetti is under every tile of a raised-floor office, covering 2 or 3 storeys. That was my 1st contract job, in 1992 in the City of London. A mixture of RS-232 lines to VAXes & terminals, UTP Ethernet, & coax or twinax for IBM minis & mainframes.
In the end, the company moved to a new office across the river & recabled from scratch. It took a year, but it was cheaper & easier than sorting out the underfloor pasta.
I had actual nightmares about cables-as-snakes for years afterwards.
Microsoft Teams
Not sure it qualifies as tech but the thing that drives me insane is to watch medium to large asset managers use a single bbg terminal to access market data across the entire firm.
Omg. So before I get started… are you familiar with VG-LAN?
One major UK bank processes hundreds of millions worth of transactions on code which was hand crafted in IBM mainframe assembly with some code from 1960s.
Only *one?*
On the bright side Google Sheets is unlikely to download & run external code while you're writing logs to it.
Inherited the corporate website for a 200mm post IPO company. They had a former employee on retainer who updated it, $2500 a month. He’d built some janky Rails CMS on his own and half of it didn’t even work. Fired him and rebuilt the whole site in Wordpress.
One name: jsonata
Perl
FoxPro 2.7 on DOS for an insurance company. Not as bad as the other examples in this thread but...it was janky to me in the 21st century.
Around 2012-2016, a HP desktop PC that ran Windows Server 2000. That “server” ran proprietary billing software for public customers that couldn’t be virtualized based on licensing 😬
Lotus notes.. Scary
Oh! Forgot about when ppl were still trying to release an alpha version of a homegrown C++ data analysis system... 17 years into the project, fully staffed. That ship was not resteerable. The tech was released as a beta at ~21 yrs and is now 30. Quality takes time! ;)
[credit where credit is due: it turned out OK, pretty much. should not have taken an entire human's childhood and adolescence, imo]
One of the largest <redacted> companies main product was over 300 servers with /servername NFS exported & mounted to all others. The app was a collection of bash & Perl. Made millions/yr. Was insanely painful to maintain.
Every server had every other servers NFS share mounted to it.
next summer camp over beers I'll have you tell you about the impact from NotPetya. Spoiler: not pretty.
Humans
True Root Cause.
Company which I won’t name built a version of Angular for website. Was horribly obtuse and hard to work with.
The org didn't depend on it by any means, but I once got paged that something random on an ancient/well-protected server had gone down. It was a (now well-known) researcher's web scraper perl script that was written c2001. It had a helper script to restart it as needed.
Best "If it ain't broke ..." - An IBM XT PC running key switches in a telephone exchange for a whole suburb. It was on 24/7 for 20 years before finally catching fire and burning down a chunk of the telephone exchange.
Telco Systems AC-211. It was a full-featured telephony adapter with no cooling - they failed often, sometimes melting and on a few occasions literally catching on fire. And there was a model which was made for a small form factor that got too hot to touch.
The same goes for oil and gas supply chains.
And, tbh, most tech company supply chains.
Same for aerospace...
That’s awesome
Imagine Excel, but with the addition of VBA using COM-wrapped Java classes to send sealed objects over JMS to the trading platform.
Friends of friends as managers
At my old bank we priced & traded around $10bn of bonds a day, using an excel add-on we downloaded from the internet lol
Coworkers told me of an auction system for short-term loans totaling 100s of millions of $, implemented in Excel 97. Office update wrecked the macros. Current staff didn’t know the manual process, had to frantically restore.
Somewhere up there is a guy who wrote the Excel Macros banking still depends on!
nice try, i have an NDA
When my Grandfather build a new building for his Ford dealership in the late 50s, someone talked him into installing a pneumatic tube system for moving paperwork. It ran into the 90s. At least once a year a mechanic would send a live mouse to accounting.
log4j lol
AS 400 with several used components from eBay since the entire system was out of support and spares were no longer available
I work on a legacy application that’s hosted on a Windows 2003 Server and it’s database is FoxPro.
Serial null-modem cable made out of paper clips. FWIW, a paper clip will support 115.2Kbps.
This... this is the one that broke my spirit.
Man, that's hacking in its most pragmatical, refined way. My most sincere respect!
this looks like something you'd do
And it is something I have
Of course you have.
Load bearing paperclip
I’m reminded of this abomination:
«
DB25 Killer
Thanks to a major network manufacturer who ships every device with a serial to RJ-45 connector, we are able to make serial toast. DB-9's are laying around too.
»
The Etherkiller
fiftythree.org/etherkiller/
Thank you all
Daley Center in Chicago, at least til the pandemic hit, used Carbon Paper to make copies of handwritten official court orders.
NYC subway is held together by duct tape and 80+ year old switches:
Key to Improving Subway Service in New York? Modern Signals (Published 2017)
New York’s subway is struggling with old infrastructure and overcrowding. The M.T.A.’s failure to modernize its signal system is a crucial example.
nytimes.com
I'd say the fact that many major business systems (financial, travel, IRS) still run on COBOL and all of the programmers who know it are reaching retirement age is pretty risky.
Windows
Can I have an NT with that 🤣 But seriously, in 1999 we were hired to do an inter-BU EDW. VPMkt didn’t feel her dept needed it. Called in a marketeer who explained his “Lake of Data” —Access on every computer that he would sync nightly — worked
I've worked in a company where the entire payment flow was a single class with nearly 10k lines of codes.
Debugging was hell everytime modifications where made due to extremely but painful regression flow. 😢, I did a lot of refactoring before my time was done
So there was this TSQL script being called by a Payment Recon job, it was like a textbook 😂
Views, sub procedures calls, functions, cursors, switches, try catches locks, db links.. file containing all the possible sql commands I have seen in a lifetime 😭
1/
Every time something goes wrong with reconciliation you literally would spend a lot of time trying to figure out where this a possibly a rounding, truncation, wrong joins etc happening.
Sadly the second half of gymnast happens on the code side (story for an other time 😪)
decade out of date filemaker pro database
I wrote an "xml parser" in a day for an old mobile platform (BREW) to parse one file (a Winamp style UI skin). A week later I got bugs that it wouldn't parse some other file that came from a server. I fixed the bug and it became our "official" parser for that platform :'(
A mishmash of .NET, Erlang and Node.js/React installed on premise on customer machines.
#IIoT
Yay Erlang!
That is literally an XKCD joke.
Actually I think I was remembering this article! (Which is probably what xkcd was referencing)
How one programmer broke the internet by deleting a tiny piece of code
A man in Oakland, California, disrupted web development around the world last week by deleting 11 lines of code.
qz.com
Could be. Article: March 27, 2016; comic August 17, 2020.
Use a bunch of if-else and regex matching and call it `AI` 😂
Running an early 90s SCO Unix app in RH6 (circa 2000) using Linux ABI/ibcs emulation, to perform the most critical revenue generating functions, & it was only phased out a few years ago. Ohh, and I'm sure there's a mid-90s ColdFusion beta instance running a govt website still.
COBOL on DEC, in 2005, for a national healthcare provider. They were verifying the listings (yes, the printer was old-school too) by hand to make sure all reimbursements were processed.
In the early days I was working on online store and in one of the perl scripts I left the debugger option on the bang line #!/usr/bin/perl -d. Each time when hit,the script waited for a debugger to attach in production in a way that hung the whole server,hosting 1000s of websites
Has to be this one soon-retired guy still doing maintenance and updates on the 68hc12 microcontroller software written in assembly, on a development setup involving an ISA expansion board connected to a machine no one was allowed to turn off - we were not sure if it would reboot
Or in the same company we also had one part of our hardware based on an obsolete SX48 microcontroller, they just had bought a few thousands of them from the final production batch and planned to use that for the next few years of production
I worked at a firm that relied on proprietary benchmarking software for all its client reports. You'd upload client data and it would churn out client vs peer group benchmarks. If anybody ever uploaded a zero the entire thing would break, for everyone.
My first job out of college had a daily script that ran on a windows machine. That machine would automatically connect via PCAnywhere to a remote machine (owned by a partner company), and copy some generated files from that machine to an FTP site that our code could access 🤯
The U.S. Copyright Office database of post-1978 copyright records.
The web
When I worked at an ad agency one major airlines backend was down so often that one of our leads stubbed out a complete replacement for their API in ruby just so we could finish on time. Worked brilliant! Would recommend
our school ran a version of ASP .NET that had a critical known vulnerability since like 2014 that allowed you to gain admin access to their whole system.
i proved it to them and i *think* they’ve updated it since then, not sure tho
Napster’s first “search engine” was just a strcmp() against its entire index. Didn’t scale beyond a few hundred users. 🤣
An auto parts logistics company went with VAX, but decided to save money and not use VMS, but instead RSTS/E. They wrote all their code in BASIC PLUS. When that platform went obsolete, they had a problem. So naturally,
They commissioned a RSTS/E and BASIC PLUS emulator for Motorola UNIX running on 68k to future proof themselves. Then they had a dispute with the consultants around pay and lost access to the emulator code. So naturally,
When Motorola UNIX and 68k went obsolete, they wrote a 68k emulator to emulate the RSTS emulator on top of SCO UNIX on 486 boxes. Probably that is still responsible for running your local auto parts warehouse and someone is still writing BASIC PLUS.
Someone certainly could do that, but there are multiple PDP-11 emulators, including several FOSS ones which are free. pdp-11.org.ru/emulation.pl?l…
I don't know when this happened, but the best-known, SimH, has been around since 1993:
simh.trailing-edge.com
I am fairly sure – say 95% – that it's impossible to run RSTS/E on a VAX.
The only way to run RSTS/E programs on VAX-VMS was via a 3rd party RSTS OS emulator, called ROSS/V from Evans, Griffiths & Hart.
RSTS/E - Wikipedia
(no description)
en.wikipedia.org
I think it was WSO2
HTML strings with inline styles stored in a database. Heard of SSR? This was DB-SR.
e-commerce distribution company: client pricing was decided each year on a per sales manager/client basis. Managed in glorified Excel.
Central data comms for whole business ran over telnet
Late ‘90’s was running test rigs and IRL game testing for Namco. Every single one of those machines was running off software written to EEPROM’s. I hadn’t seen that sort of thing since the late 80’s. Was shocked to see a modern video game being rendered by soldered together IC.
Worked with a co. who built power generators for satellites. They had a defect tracking system written in PHP1 running on an ancient HP "unit" that was a full sized cabinet. Upgrade to PHP7 running on a RPi with backup to S3. They were blown away by the performance improvement🤣
Love the stories 😂😂😂 bookmarking
Their website was running on a Win NT desktop on the floor of a middlemanager's office. An analog timer plug did a hard power cycle because Dr. Watson kept flooding the machine with popups. So the site was offline every evening 4PM-8AM. (This a respectable listed company in 1999)
That I'm allowed to speak about, robots at heavy industry based on DR-DOS. Floppies used to program the suffered a lot with plastic vapours and debris.
No joke. In 2010, a small white power strip on the windowsill in the server room over which some main switches run. When the thing accidentally went out, it was a party.
The shaft balancing machine that has a 1990's PC built into it. :)
pics or it didnt happen. =D
Large FTSE100 company relying on an access database to do all it's invoicing, that nobody knew how it worked...
Built an excel model to calculate wages with raw time data for a labour hire company.Included fairly convoluted overtime and bonus rules. Someone copy pasted it for IT contractors on day rates and paid out 10x in wages for 100+ people. One dude got a monthly paycheck of $113K
This incredible web frontend spun up a complete java environment per active user. Hit the memory ceiling after 10 users.
The biggest insurance company still runs off an MS-DOS system
The actuarial dept for the workplace div of a very large insurance company relied on an app running on a old desktop under the developer’s desk. He had a stack of 3 and always forgot which one was “production”.
jesus, the war stories from actuarial are atrocious. I worked on some actuarial data tools in python last year. my predecessor formatted several 400mb csvs with nested dicts and opened a new interpreter to generate each of its 200 columns... O(n^♾)
Mgmt must have loved always busy hard worker
I've seen some hairy situations where a dev desktop running some beta feature branch accidentally got added to the production VIP. Actually caused customer data loss one time. Major protections added after that to prevent test servers in prod VIPs.
A friend told me a similar story also from an insurance company in .hu: a dev pulled data from a huge MS SQL server and ran an overnight batch job on his PC. Took around 7 hours. Then my pal rewrote the queries, moved the workload to the server and squeezed it into 30 mins.
Code or process that is written like an inauspicious game of Jenga where the removal or failure of any single piece brings it all crashing down. 😂
Multiple logic statements in an if-then-else statement
Chain enough together - say thirty or 40 - no way can you refactor that sob
Or calculated goto states in Fortran which go up and down
200 on a goto (100, 500, 200)
Where a is variable
If you're lucky the 100 label is before 200
Stop poking fun of AWS US-East
Cics transaction gateway running on OS/2 on an ancient physical server, discovered during a critical data center power down. No support, no replacement parts and a lot of breath holding as it was power cycled 😅
I’m breaking into a cold sweat just reading this.
I worked at a major bank who’s SIEM didn’t index IP addresses.
Excel ... it's always Excel
Uhhhh a 20 year old roland sticker printer. It was acquired used. It had a lot of trouble color matching orange and purple and threatened to break down completely every time there was a big job. One time we had a tech in from Roland and he'd nevrr even seen the model before.
I actually compiled a list of all of the janky tech I’ve used professionally.
Former employer: student system running off a Mac from the 1980s.
Lotus Notes network run on two servers in different buildings using modems on internal phone extensions.
Corporate IT wouldn't let them replicate on the pipe between the buildings... 🤷
Have you heard of access, good sir?
So I’ve got a couple. Worked for a financial org during a massive security overhaul, consolidating security tools wherever we can. We get a report that a business unit cannot run their model to asses whether or not we should be insuring someone
The person they put us in contact with paints it as if the sky is falling and that this is a extremely complex analytical model and that they’re currently unable to process any policies because of it. Finally get hands on with the “app”.
Find out that it’s an extremely complex macro that’s been carried over the last 15-20 years with the person who created it long since retired. I’m talking thousands of lines of “code” that we had to figure out what part of it was being stopped by our new endpoint security tool
Guess what funding was nixed year after year because it was just working? Credit to the person trying to “support” this beast, he barely knew how to even execute the damn thing. Okay, 2nd story!
You know where folks are crowing about that they’re going zero waste and all that jazz? Well a lot of those orgs utilize what are called “waste to energy” facilities, basically a fancy phrase for incinerators.
Now what most people don’t know are that a lot of those facilities are 20+ years old because who wants one of those in your backyard? Now imagine all the things you’ve only read about in tech history books, running the things within these facilities.
And all the technical debt, bubble gum, and tape to go along with it, Token ring networks, Windows XP, random wires running into walls into old switch hardware. Now, I don’t know how many of you can imagine this, but imagine the smell of garbage turned up to 11
And having to hunt down one of these broken switches because of one of two things. The system that monitors the emissions can’t operate or the systems that operate the 20+ year old plc can’t communicate with said PLC. All running extremely legacy tech.
Both of these things caused or had the potential to cause a facility shut down because of thousands of dollars in fines for not monitoring what you’re putting into the air or not being able to control the burning of garbage happening in the boiler. Both extremely important things
So yea, legacy tech spread all around makes the money machine go brrrrrr while running some extremely important systems that very large facilities or financial institutions rely on!
I also realize that many people deal with this in their day to day jobs still, you folks are the real unsung heroes!!
I worked at a company whose main product was screen scraping a client’s desktop site to build a mobile app dynamically.
Too many to mention in large enterprise IT in the communications and software sectors. Software-wise using Excel as a database or an out-dated version of Microsoft Access as they wouldn't pay for a dev to put it on a more suitable platform.
Our desktop IT people used a Windows bat file to connect PCs to printers rather than let people sue the Windows printer interface.
A private university. Refused to listen to the videographer about media storage recommendations for their growing video library. 3 years later they are still using an assortment of disconnected hard drives
Now, they are hiring a new person to organize the hard drives
(The videographer left 2 years ago, and the uni is flush with cash)
I worked at a company that kept reusing code between clients, well eventually they ran into a problem. the maximum number of joins in a sql statement was exceeded. 255. Thats right production code with a 255 tables joined. They were using views heavily.
I worked for a company that automated legal filings. The codebase was 4.5 GB of PHP. No framework. It was that big because each form for each state was copied to make the next one. The original source material was Sam's Tech Yourself PHP in 24 hours. The year was 2010.
Reading this makes my eyeballs hurt.
Probably why I have glasses now.
Reading this make me realize things aren't so bad where I work
Software made in 1995, could not run the installer any more (I had to manually update the Windows registry). It only supported 5 connections to the server, and records locked when editing, causing issues when reading and writing at the same time.
My job was to keep this thing working in 2015, and migrate it to Windows 10. The worst part of this thing was that we had a couple remote employees (using RDP to connect to the local network, the server couldn’t be put on the web directly)…
Occasionally they would walk away while still in edit mode, causing the entire office to lock up.
Sometimes the server would soft freeze itself too, no interaction required but would have to be rebooted.
A halon button that had no cover. And, someone set it off by leaning on the wall. There was no fire.
So, I worked somewhere, that the CEO and founder insisted to write code, about 7 years later after they migrated away from the system he developed, they printed the source code which had 4000 lines of switch cases.
Thats high performance oriented design right?
High performance, high maintenance. That'd how they rolled.
Now, mix for loops and goto statements for completeness
A homegrown CRM from 1998 that was still being used in the 2010s and only one developer knew how to maintain. It froze and crashed so frequently that there was an app on everyone's desktop that was basically a kill command for it.
A system full of over 4000 stored procs, some of which were over 10k lines long with 1k line where clauses performing operations on non-normalized tables with 100+ columns. Some of those cols were incorrectly named. Like a date column would contain an id with a foreign key.
Code hell… Satan awaits you somewhere between those SProcs, proceed with caution ⚠️ 😂🤣
Lol 🤣🤣 as soon as they made it clear that they weren't interested in improving the codebase, I proceeded to leave.
1. SQLJ
2. Murex
3. Homegrown Excel addins with state
first magazine I ever worked for rebooted itself after failing, moved to new office, had no internet but had to close first print issue night, do or die -- we stole open wifi from office next door and connected our LAN to it; intern had to stay up all night to keep it running
JIRA...🥺
worked at one of the nation's biggest banks who's mainframe relies on a COBOL programmed core that only 2 people know about. Same company (EU) has outsourced 60%+ of tech to India
We had a distributed job queue. A grad student wrote some reporting software for the system. To avoid divide by zero issues, he put a job in the queue with requirements that always evaluated to false so it would never run and there was always at least one job in the queue.
Using regexp to parse ton of json from rest APIs as json decoder was not available.
A glorified phonebook system for marketing. Core functionality was making searches with multiple parameters. Problem: searching was slow. Solution: cache results. As dynamically created tables in the db. Result: production db with over 70K tables in it.
The kicker: since the underlying data could change at a moment's notice, the cached results would be often invalid. Solution: run the search on every pageview anyway, recreating the "cache" tables over and over.
pure gold buddy
I'm afraid to look. 🙈
Excel for complex $$MM project management, including budgets.
A significant portion of US Government satellites have had their course design trade offs worked out on a set of linked spreadsheets running across ~20 workstations.
Broken laptop that required an external keyboard and mouse (or RDP) that was used for controlling door key software and "hidden" on top of the server rack.
Ill take resp and admit i was the agent of chaos occasionally. My first "real" commercial work didnt work at deploy at client site but did at my machine bc i had crosswired username and password fields, but all tests had them as "test:test" so theyd pass.
A dot matrix line printer connected to a web server in the office, receiving POSTed payloads from a web app off-site, in order to print triplicate contractual agreements onto special preprinted carbon paper.
This might not sound that bad but it was like 2013, and the payloads were generated with Perl's ancient "format" magic.
Oh god I keep remembering the horror, the "web server" was a repurposed desktop sat next to the reception desk because it needed a parallel port to drive the printer.
In 2013.
Gold veins everywhere, omg.
Friend used to work in a legal aid firm, back when data stored on tape drives. Tape backed up last thing on a Friday, and operation controlled via keyboard. The t key tensioned the tape, r key rewound it, and e key erased (without confirmation). Check the key proximity.
A consultancy whose whole business depended on analyzing telephone Co traffic data. They did all sorts of consulting but that was the foot-in-the-door.
And it all ran on a Vax or some very old mainframe.
And one guy understood the program. And he knew he had them over a barrel
filemaker in 2021
Once upon a time a major derivatives index, which was published daily by one of the major finance data providers, came from a bank's trading division - where it was calculated on an old developer workstation, running an unpatched version of Windows, that sat under a trader's desk
Live in New York? Ridden the subway?
You survived through jank.
This will blow your whole mind. Just look at those control handles:
Key to Improving Subway Service in New York? Modern Signals (Published 2017)
New York’s subway is struggling with old infrastructure and overcrowding. The M.T.A.’s failure to modernize its signal system is a crucial example.
nytimes.com
Its all well and good until you have to interview the creator of a whole ERP system who is now in hospice so you can understand it enough to export it.
A server under a desk that, once disconnected, showed to host the actual active AD .
Network ablation study
Financial institutions had to use a WORM drive. Early tech was built on writeable CDs in a cartridge. When you requested a file from the filesystem, a picker arm would pull the cartridge, load it and read the file.
In 2008, frequently had to have someone jiggle it when it stuck.
In the late 2000s, there was a place that was using a machine running Windows 97 because of a legacy database in Access 97, which was only slightly more stable than Charles Manson.
Um. It can't have been, because there never *was* a Windows 97. It went 95 → 95A → 95 OSR2 (AKA 95B) → 95 OSR 2.5 (95C) → 98 → 98SE → ME.
Maybe the OS was Windows 98 then? I was told not to touch that computer and the back story.
I do know that the database was eventually migrated to something else.
Well, yes, fair, but it's 90 less. 😉
Well, the Polish Social Security Institution was still dependant on 3.5" floppy discs a decade ago. And apparently, Boeing 737's required floppy discs for updates as late as 2020, perhaps still do.
They now use a floppy emulator running on a laptop
I worked at a graphics company that had two servers, both of which had login and password set to "ADMIN" and "ADMIN" for decades leading up to my working there.
One server had all the business-side software on it, for the accountants, etc. The other was for all the work files.
When I pointed out the vulnerability, my boss asked why anyone would want to hack little old THEM.
I don't know. Financial information like routing numbers and stuff from clients? I offered.
They were unconvinced.
Also, the work server had two partitions: one for active work in the last year and the other an archive of all the work that had been done since the company went digital.
Often we'd pull files from the archive because people would hire us to work on them again for new projects.
Like...every day this happened.
However, every account in the building had both read and write access to 100% of both servers.
Literally, the entire archive server could be deleted with a couple keystrokes on accident.
I think the passwords and permissions are still this way!
(By "a couple' I mean "Ctrl-A, Delete.")
OH!
And there was also a thing where someone had come in to salvage files after a ransomware attack on the server a couple years before I got there!
In turn, EVERY FILE before that date had a non-corrupt duplicate on the server, sitting right beside it. Inconvenient.
Hey you want to got fix I recommend you to contact cybertech121 on IG his legit and fast
Worked for a newspaper where the terminals we used to access the CMS were so old they caught fire periodically during the workday, often enough that we were familiar with the smell of “burning computer.”
Updated jpeg assets for a big hotel chain & file locations were kept in an Excel workbook w/ about 10k records over 30 sheets. It took longer to find each record than I est. to do the retouch so I had my Excel wiz friend write a program to find the records. Use a GD database.
Node and react
An entire floor to ceiling wall of 480v mechanical relays circa 1960s.
That was in 2017.
A cluster of two MySQL servers sympathetically replicated across Atlantic
Pretty genius MacGyver level monitoring
School district running windows xp because they havnt upgraded their computers since core duo era i want to say compaqs at that. they want to upgrade them to windows 10 when new administrators came on board. but i also saw gateways still in the wild running hvac systems.
I saw a credit card processing company operating on an old 3174 mainframe controller that was running a cooling fan extracted from another old, canibalized unit, because they hadn’t been produced for years
My code
I think of two particular examples 1) an Access DB solution that turned out not to be 1998-compliant and 2) a rogue ‘open tran’ that turned a 20-person customer services order system single-user.
I developed logic to control the upper set point on a 300 mw wind farm, and the consultant implemented it as a series of SQL queries over a SOAP API. Drove our CTO crazy, but they still haven't been able to replace it.
When I was 17 (around 1999) I wrote a small tool in VB6 that connected to the company sales db and generated various report emails (like sales per team per month etc.). It still generates these reports to this day, and presumably business decisions are made upon these mails...
I wrote something which would screenscrape a DOS application and then convert the output to HTML. For every screen in a system with tens of thousands of records in a flat file database.
But hey the performance of the web app was insane.
Wrote an unauthenticated Valentine messaging app; worried about profanity, used by insurgents.
Years ago, a major telco had a customer network that was completely dependent on a routing table algo that ran on a single SPARC pizza box sitting in an empty cubicle, with the code compiled into the SunOS kernel so it could get system priority. No one had the source, of course.
Oh for sure Adobe ColdFusion. But not even that, we used a different open sourced version of CFML that’s popular with ColdFusion devs called Lucee.
Keep in mind this is circa like 2019.
I’ll give you one guess who got it wrapped in Docker and working on OpenShift.
What's janky about any of that? @lucee_server is a modern JVM scripting lang and I've been running it on Docker Swarm since circa 2017. There's been official Lucee docker images out since 2015.
The docs were bad at best, there wasn’t an active community that I could find at the time, and there was no information anywhere on how to get it containerized (much less working on K8s) when I started that project in 2018-ish.
Plus I didn’t think an open sourced ColdFusion alternative had any business powering a dashboard to help federal judges across the country get oriented for the day.
But it was the language my boss at the time wrote the proof of concept in so we kept moving forward with it.
I will say that DataSources were fun, though.
They were like a little proto-GraphQL.
That was retweeted way out of context, @bdw429s.
Has the community improved at all? I really felt like my options at the time were to dig through a lot of stale forum posts or struggle through finding my own solutions.
And docs were seemingly directly lifted from Adobe.
Hmm, there's the dev.lucee.org Discourse forum which is very active, a 4k person Slack team, facebook groups, and the Docker images are right there on Docker Hub! hub.docker.com/r/ortussolutio… You're also the first person who I've heard dog a lang for BEING open source, lol.
Lucee Dev
Lucee Dev Forum https://dev.lucee.org/
dev.lucee.org
Good that this stuff is growing. At the time it felt like I was on an island.
And normally I’m all about open source but federal government has some weird requirements and I was surprised we went with OSS for something deemed mission critical.
Yeah, I guess that’s more in line with what was going on.
We tended not to touch anything if we couldn’t get a support contract. Even for parts of small side projects.
Did Lucee have a commercial support contract available at the time?
Yes, Lucee has had support available since day 1, but it comes from the LAS member companies like Rasia. My company (Ortus) provides paid Lucee support too. For Ex, I've fixed bugs in the engine via pull request on behalf of customers. Here's one: wwvv.codersrevolution.com/blog/improving…
Gotcha.
Unsure if that would have satisfied our requirements. Like I said, fed gov gets wonky when it comes to support contracts.
I’ve had perfect tools for projects turned down because there weren’t first party support options.
Could be it.
I also just didn’t want to be working with that technology at the time. I was just getting into web dev and learning React.
Having my day job be an offshoot of an Adobe created language felt like a step backwards when all my free time stuff was latest and greatest.
Heh, I see Lucee as a step FORWARD in the lang Adobe created. It's faster, smaller, modular, FOSS, & has unique features like extensions, caching abstractions. Heck, I use Lucee for hardware automation & RPi servers. pi.bradwood.com It's like ACF, but BETTER 🤓
Glad it works for you!
I had a bad taste in my mouth from DreamWeaver in high school and would rather have been exploring Node or RoR at the time.
I was also only a few years out of college and wanted to be working with technologies that I knew folks were hiring for.
They tryna migrate off of CF at work right now. Rough stuff.
Depending on what you’re doing I might have a repo for you.
At one point I added Vue to the mix to slowly start taking over bits and pieces with custom JS components, rather than one monolithic overhaul.
Could that be done with Angular?
Pretty much every AWS service is launched by a chunk of black box Perl. Requires a specific older version of Perl. Ask most engineers there what it does, or whether its safe to get rid of it, and you get hushed replies about the deep and important magic this package provides
Once I got annoyed enough to go poke under the hood, and it turns out the only relevant thing this package does is auto-relaunch the service when it crashes. The other thousand or so lines of perl all concern setup for Oracle databases that haven't been used in at least a decade
One of the Apollo 11 astronauts accidentally broke the circuit breaker on eagle that controlled the ascent engine. Essentially would’ve been stuck there. Saved it with a pen. Don’t know why this is never spoken about.. eetimes.com/how-a-felt-tip…
Earlier company had a critical application used to pay commission to traders which was acquired from different company without entire source code. Binaries running fine on Windows XP when I last saw :)
Automotive IoT startup. The entire backend server was a single 9,000-line PHP script running on the command line. The founder just liked PHP.
The electronic medical record—more time spent charting than caring for patients.
When I worked there the canonical New Yorker Magazine article archive was stored in a FileMaker Pro db, I want to say version 8. That meant zero support for unicode encoding, all those diereses (not umlauts!) needed entity encoding. I’d be surprised if this has changed.
Wordpress
A large intl corp had a demo room for their future product, and because of VPN/firewall, parts of it connected to their inner servers through OUR 8yr old workstation that we used to try out stuff. They would call us the day before big presentations so we didn't crash the computer
At a Fortune 100 some twenty years ago. Instead of making UIs for the CRM people used MS Access with the DBO login to modify the tables directly
... and that table was used for *all* apps. Someone accidentally locked it, went to lunch, and brought down the public website
angular
An Access 95 database to handle all of the billing for the company that wouldn’t be upgraded beyond Access 97 that required the users to still have Windows XP workstations in 2018. They may still be using it, but I left the company.
I've personally installed & run Access 97 💯% fine on Win 7. I routinely write in Word 97, running under WINE on 64-bit Ubuntu. It does all I need & more, & it's lightning-fast on a 21st century laptop.
Both work perfectly including installing all Office 97 service releases.
& of course Microsoft offered Windows XP Mode as a free download for Win 7 (Pro & above). Run 16- & 32-bit apps, on real XP, integrated into the Win7 desktop, completely legal & free.
theregister.com/2009/05/01/win…
Could be coaxed onto Win8.x but doesn't natively work on W10.
Windows 7’s XP Mode — Virtually worth the effort
What’s real and what’s not?
theregister.com
LOG4J
outlook
I had assembly code tutorials on an LSI-11 back in 1983 😁 missed my calling?
I've written PDP-11 assembly more recently than that, but not by much.
This makes me think that most RPA is just tech debt waiting to happen.
Major well-regarded tech co, production database with client rollups, 'isDeleted' field. All good so far... isDeleted type CHAR. isDeleted values "yes"/"no".
This is the story of how I learned not to just use table.isdeleted in a where clause...
Our data center ceiling had two tarps that fed into buckets to contain the leaking roof (right over the UPS). The entire records system was on an antiquated version of PeopleSoft and they weren't sure exactly which machines it ran on.
For a number of days one summer many years ago, a local government's entire IT operation relied on:
1) Two large desk fans
2) A lack of rain
Unnamed company had relational data.
But it used MongoDB.
Uh-oh, broken data because no txns.
Found a fork of MongoDB with txns.
Maintainers got bought; project scrapped.
We forked it before it disappeared.
Took 1-3 more years before we could migrate to PG.
At Bing years ago we had a service that ran 100s of classifiers on every query. There was 1 editable config file for everything. Once another team disabled our classifier when they accidentally deleted the last line of our classifier’s configuration while editing their config.
The most frightening part of this story is that I’ve been the person that makes that tragic edit.
Windows 2000 email server... this was three months ago.
A pre-Pentium Dell used to configure the internal telephone system's call routing to desk phones. It used a phone jack and didn't have wifi.
Last year I took a job as HelpDesk Manager and nothing was modern, they were doing things like it was 1999. Worst of all was IT data drive was on a “linux server” which turned out to be a 9 year old Desktop with unpatched Ubuntu 14 and an HDD hot glued to the drive tray.
log4j2
I used to convert 8 in floppies to 1650 reels so the data could be printed to microfiche via a 20+ year old COM machine. This was in the 90s but even then each of these media were obsolete.
We identified a huge leak in the systems of a major US broker.
Leaking live client orders + account info for all clients out to *any* API user. Had been going on for 10+ years.
We helped them identify it, they fixed it a couple of weeks later and never told client base.
An excel spreadsheet for contracts greater than $10MM. No macros, just spread the formula across dozens of cells using concat(). Each concat was at least 5 more cells, each with their own concats. Usually this went down 3 levels before you got to a number.
I once worked into an IT Security company where the CRM/Accountancy/Invoices system was a Visual FoxPro 98 behemoth running in Windows Server 2003 SBS with SQL Server 2005.
Running in a janky server that barely booted, when it did. So it was scary to turn off.
What was magic of this server is that the spinning disks actually made sound, you almost could hear the read/write arm scratching the platters.
Anyways, the virtual machine running the WS2003SBS finally died of a VirtualBox VMDs inconsistency between snapshots.
So it was time to decommission the server. I rebuilt the machine in a Windows Server 2016 vm and turns out SQL Server 2005 is just too old.
I had to do some hex magic over DLLs to make it run, replacing files just at the right time, starting the service many times.
The company was 4 days without the system, and I happened to have turned in my quit notice 2 weeks earlier, as I had accepted a job at another company.
I managed to fix the whole thing on my last day on the job. The system was passed into production on 4 p.m. EST May 31st, 2017.
One hour later, I said goodbye to my workmates.
That system exactly as it is, is still running alive today and kicking ass. I got lucky.
We had a MUMPS-based system running on a disused SUN server that was essentially the backbone of a ~400 person data analysis company. And then I started building (MUMPS!) NLP applications on it... I think it eventually caught on fire. Ahh, the dot-com boom.
I appreciate the drama of the “hardware caught fire” subgenre here.
Years ago I worked at a company that had an expensive computer, on loan for rendering. The computer overheated a lot. So it was put it next to the bathrooms, a portable air conditioner next to it, and the water outflow to a bucket next to this $1,000,000 computer.
I am a bit surprised the Windows-XP-64 ~2010 era precision workstation I cobbled as a VM host at work is still running...combined 2 surplus machines into one, dual CPU sockets so transplanting one I had no thermal paste available. Hasn't burnt up yet, on 24x7...
And has 16GB RAM (combined from 8GB per box), RAID of scavenged workstation 250GB drives (even took out floppy and CD-ROM to plug in more). For a while it had too much storage capacity to be backed up too...yay development "servers"?
Its got dual Xeon 5160 3.00GHz if anyone was that curious. 4 whole cores! And no hyperthreading. Yes, a VM host. No, not XP...its running RHEL.
Crowdsourced from around @ZusHealthHQ:
"The parts management system at Lockheed Martin in 2006 was still a IBM System V mainframe. First (and last) exposure to PL/1 and JCL"
@bryanjknight
"Large e-commerce store had a homebrewed Jira/Salesforce clone. They ran everything through it: sprint planning, customer service, etc
Oh, and also HR tickets. And the database was readable by any of the 1000 or so engineers and PMs who had database read access"
"Large EHR company with homegrown everything. RATmail & RATtrax. It looked like a game of Minesweeper from the 90s, but less functionality.
Except for the part where you could read anyone in the company's email"
"Another large EHR company’s expense system In the last few years was something similar. Garbage Oracle app that gave the impression of a student project"
"The whole application for Student Registration written with Oracle Pl/SQL web package (never knew that existed) at an Ivy League college. The scariest code I've ever seen"
Basically all the crap I used in the military. Amazing when you got replacements... That were 8 years old
In a large public institution, I wrote a backup script for their payroll system in 2007. It was running on a Solaris 2.3 workstation from '93 with 4 2 GB SCSI disks, lying on a small table and under a thick coat of dust in the HR office... They had more than 500 employees.
10 years after I left the company , a former colleague phoned to tell me that they were moving building and no-one knew how to switch off the ICL DRS6000. I told him, got the course notes from the attic and mailed them to him with deepest sympathy.
The building they were leaving was a former factory. The Big Tension cables from the road were insulated in tar. Because of the power demand it was hot and liquid, had to be topped up every a.m by the caretaker. If he ever left the whole place would have gone up.
What with old Unix machines and crumbling infrastructure I was glad to be out of there. [pauses to survey home office] Anyway....
As an integration partner, I had admin level access to a MSSQL db of hidden persons in my country. I did not even need that data nor did I want it. Because the DBA did not care enough to create a restricted account for me.
interactive voice response system would screen scrape customer data from a 3270 terminal emulator session.
It had all the reliability and low latency/response time you would expect it to have.
And in those days, it was SNA over token ring natively on Cisco gear.
Accessibility overlays.
I worked for a company that is actively using MS-DOS. 🤦🏽♂️🤦🏽♂️
A friend worked for a well known product safety testing and standards non-profit. They hosted their code on a networked mac mini and used Time Machine for version control
A reality company using Windows XP and IE as a browser for secure banking transactions. And using a KVM switch to switch desktops to login to each bank scanner. What a nightmare to migrate all of that to VM for them
If you make two subfolders (called bins) in Avid Media Composer (used for almost all movies and tv shows) with the same name, you are forbidden to open both at the same time. Even if you rename one, because it still secretly has the same original name.
Microsoft Teams.
Microsoft Dynamics 365.
Microsoft Outlook.
Microsoft SharePoint.
Oracle.
Adobe.
Hmm, I’m detecting a theme here…
An entire SMTP MTA written in Perl that offered so much custom functionality to customers (that depended on Perl) that it was impossible to replace.
Same place also used Perl based IP failover.
Once had a client that used the server room as a "smoke-break" room. And yes, the servers did crash eventually and it was disgusting.
A company was shocked when I trained them in using CMS I've built for them. Like you edit stuff on the same webpage? Turned out previous solution was using plain @phpmya for content and data management and somebody taught non-tech folks how to use it
I was called in to do IR for a business in a unique industry. They had a compromise but needed special attention to analyzing ... an Internet-accessible iMac running macOS 9 that had a critical business app.
I nope'd right out and handed off to a team mate
The server closet at work. It's just... why are there so MANY?
All that needs is a dust over maybe in 2030. Nothing wrong with that.
Apart from that abomination of a switch. Ugh.
A manufacturing line using a crockpot to keep the label glue at right temperature before application.
My own small business relies entirely on a complicated Google sheet built by a guy named Dave for $100. I don't know how any of it works. Dave didn't want me to tag him, but hey Dave! 👋
Knew of retailer who up until 2 years ago still had a Novell 3 server handling their POS data collection from stores over dial-up
I once worked for a company that had a single database that housed all of the companies PCI data that gave out "Keys to the kingdom" the password was 5 characters long. It was open to the world. The company dealt with school students data all over the country.
Virtually everything at certain-large-game-company with regard to BI (at least when I was there) relied entirely on a cobbled together mess of python scripts dumping things into AWS from peoples' desktops at random places around the company.
This.
Yep, that is the CPU fan hanging off. It still worked. Ran for another 5 years before I got to kill it.
Ten years ago I still occasionally had to BRUSys PDPs for a household name. There were still law firms running on bugged Pentium Pro machines on SCO Unix and a local authority on an ICL 3900
I love this photo, I want to marry it.
Not my story, but former roomie once unplugged an old Pentium3 that ran a version of openbsd that hasn’t seen updates in years, and nobody knew what it did. It took the whole building, housing several smaller companies, down. Turned out it was the firewall in front of the modem.
A scary few years ago / a bank in England still running IE6 internally
200 car diesel electric ferrys with Windows 98 PCs for the alarm and monitoring system
There was this time when everyone used AWS…. What could go wrong?
A box-cutting and folding machine controlled (in 2016) by an embedded 486 ISA mainboard with 4 4MB 30-pin SIMM modules, booted and operated entirely by a program on a DOS 3.5” floppy (Heaven forbid that disk goes bad).
Why was I called in? In far older AMI BIOS font than this:
At my old company. They do some very important cleanroom & pharmacy validation and emcironmental monitoring testing, in accordance with a lot of laws. The offic8ally have over 500 employees nationwide (they run a skeleton crew, it should really be 750+), and their big database...
... that everyone used for everything... is Microsoft Access. In case you're unfamiliar, access is basically a glorified Excel spreadsheet.
They're headquartered less than an hour from Silicon Valley, too. The pain is real.
Investment bank, in Hong Kong, running a multi-million pound *intra* day P&L off a s/sheet that took 6 hours to run. Not to mention the MS-Access programs built by novices in the 1990s hanging off mainframe feeds, which then became mission-critical.
The glory days or the gory days ?
A well known bank with its largest fund £10bn running on an access database.
ah good times. just needs some cobol.
I manually de-dupped large merged mailing lists with excel sorts & my eyeballs for a political campaign on early 2000s
1990s: the payment card processing for a major airport car parking firm in the UK was running on an RS6000 in a disused portakabin toilet.
Not my story but a heard of a medium size animation studio that had an important part of their render pipeline that was not designed for headless rendering. So they had one poor little SGI box that did nothing except every other machine in the farm would set X11 DISPLAY to it.
I think I remember part of the Oracle Application Server installer requiring something similar... we needed to repeatedly reinstall it on headless rack boxes for reasons and bodged together something similar.
In the era before video recording but after video broadcasting, NBC would time delay the daily news for the west coast by optically filming a TV screen, then doing a live broadcast of the projection screen.
Wow so they had like a 3 hour window to develop and print the whole hour worth of film? that must have been stressful.
I wonder if they could’ve used X virtual framebuffer (Xvfb) instead? Probably not if they depended on hardware accelerated graphics.
en.wikipedia.org/wiki/Xvfb
Xvfb - Wikipedia
Xvfb - Wikipedia
(no description)
en.wikipedia.org
There was probably a hundred better solutions. It didn’t even open a window, just tried to create one as part of their initialization code. But it worked, at least until a sysadmin noticed an “idle” machine…
I used to have to remote control a desktop over Zoom to ssh into an on-prem server (the on-prem eng had to log me into several time-out-ing terminal windows w/ his creds because I didn't have ssh creds and he wasn't going to sit there and watch me work) and code in vim.
What's wrong with coding in vim ?
Not a goddamn thing.
There is a bowling alley near my house that still runs their small business off of an Apple IIe with a dot matrix printer. No joke. When I asked about it, the owner looked at me, said "don't fix what ain't broke" and then then back to spraying shoes.
My hat tips to this IIe and its dedicated user.
WebEOC gang, rise up.
IBM SPSS CADS. Just the most haunted piece of crap, poorly conceived, poorly supported, mysterious failures. Terrible.
I once wrote a terminal emulator to do automated data entry from a spreadsheet source. It was used for years after I left even though not a single person would have been able to make even a minimal change had the green-screen terminal UI changed a tiny bit.
Mail order pharmacy that I worked for in 2000 had an old vax/vms server that everything ran on. tape drive and everything. All of our computers were dumb terminals, the old orange or green screened text only ones. as the terminals broke down and became expensive to replace
they ended up buying windows computers, and running software on them to emulate the old terminals.
Had a questionable IT guy who decided the best way to set up a email system for the company in the mid/late-2000’s was to run it through his personal server at home. Guess what happened when he got fired and nobody knew it was setup that way but him…
"questionable IT guy" :)
Oh, that was just the start with him. He was also reading emails between the CTO and the CEO, among other things
in-house xml parser implemented as a gigantic single perl function "because xml::libxml is slow." (lol, no) the place handled about 80% of all hotel reservations in the united states at the time.
I will just say this: old unpatched windows XP (SP1) desktop under a dusty desk. Login was "admin". Potential consequence of failure: deadly ammonia cloud enveloping a city.
This feels like the premise of a summer blockbuster.
There are pieces of Top Secret military stuff that still run off Win 95 or Win 7.
This is why I just don’t buy into 90% of the conspiracy theories involving the Gov….
A utilities company I worked for ran their billing system on an IBM mainframe that required a piece of kit, a cluster controller IIRC, in each remote location for 3270 terminals to talk to the head office. We had the job of updating the controllers to talk over X.25 1/
At one location we visited, the controller was in a store room, surrounded by junk & balanced on an upended wastebasket. Updating it required loading a new config file from an 8”(!) floppy. It took several attempts because the drive was basically scrap at this point 2/
Despite the network being necessary for billing & newer controllers being available with 3.5” drives (woo), the beancounters wouldn’t authorise their purchase because it was “too expensive” 🙄 3/3
Circa 2005, white box that controlled a very old testing tool. Took a minute to figure it out, but it had a 1992 release of Solaris installed. The nerds from the comp science department came over to take pictures. Original hard drive was installed and still working.
A COBOL to PostgreSQL to both MSSQL/Java ecommerce and Perl/MySql ecommerce systems which then fed back through a SaxBasic script to post back to the COBOL system.
A cluster of three servers, all of them 17 years old, 1 still working with all but one root disk not broken down. Yet.
I don't work there, but SpaceX's crew dragon touch panels (yes, those panels) are run on Chromium and built with React.
As it's the future, they've also done away with old-fashioned things like joysticks. There are no joysticks on Crew Dragon.
I will let that sink in.
That's gonna have to be the most stable react app ever. Let's hope it doesn't freeze or smth
To those wondering about the accuracy of my statement, here it is straight from the horse's mouth!
We are the SpaceX software team, ask us anything!
1. Yes we use Chromium and we do use a reactive library that we developed in house. - Sofian 2. The docking simulator is completely separate...
old.reddit.com
They actually used web components, not React (which didn’t support custom elements at the time) as part of their custom “reactive” library 🙂
r/spacex - Comment by u/spacexfsw on ”We are the SpaceX software team, ask us anything!”
23,708 votes and 7,329 comments so far on Reddit
reddit.com
My top nine answers are all Airflow.
We were talking to the head of a law a office about Wi-Fi security, and why the Walmart Linksys router wasn’t ideal for his clients’ and his own sensitive data. We asked what his clients’ security and confidentiality was worth. He deadass replied, “About seventy bucks.”
Every state of Maryland website that collects payment and SSNs:
beacon.labor.maryland.gov/ReBEACONClaima…
velocitypayment.com/vbills/mdccu/l…
interactive.marylandtaxes.gov/individuals/pa…
egov.maryland.gov/com/TaxPayment…
Also nearly every federal government staff tools are god awful and compromisable.
I once ran a multi-camera video livestream of a concert on a single old PC laptop I reformatted to keep it light. Cameras plugged into Firewire on an ExpressCard. Soundboard into a mic jack. The laptop was plugged into a very long extension cord & series of Ethernet cables + hubs
US border patrol was still using Windows XP back in 2015. The officer that was interrogating me for 5 hours (and ultimately deported me) at some point complained about all the popups that kept appearing out of nowhere…
And yeah, that was the same desktop computer that processed all my info and even the video of my interrogation.
Not a company, but I remember an astronomer at a South African observatory who explained that their Internet connection was so slow they sent photos to their university by memory sticks transported by carrier pigeons.
Major university medical lab. The electron microscope's vendor is out of business and it only runs proprietary software on a Win2K box - that is infested with malware and popups.
Solution? Drop from the network, no fix/reimage. No one knows where the drivers for the scope are.
Sonicwall
Blade server stood on its side on the window sill was the main product build machine. It had a USB Christmas tree light that was an indicator that the server is on and alive.
A company who’s entire business relied on 4th Shift CRM. That software required nightly processing but their business was build-to-ship that day. They hacked the software to pieces using MSAccess to extract data from manufacturing and give it to shipping and accounting.
Banyan vines
At my previous job there was a domain controller that hadn't been patched in ten years.
A multi-billion dollar online retailer using a tampermonkey script written by a dispatcher to manage their -entire- driver/route assignment workflow for a given business space.
A wheelie trolley with a router on that had bare wires connected to a frame relay circuit. If the trolley moved 1 cm the WAN went down
WAN down?
Jog the trolley
Application service provider and we used to house some of our own servers (email, for e.g.) on site.
Sometimes we would find them rebooted inside the locked server/sysadmin office.
One outlet also had power to the OUTSIDE wall (we found).
Cleaners were rebooting them vacuuming.
I once had to write a bunch of JS embedded in Adobe PDF’s to make a particularl form dynamic enough to adjust based on how certain questions were answered. This was in 2012-2013 for Obama Care, so you can probably guess why we didn’t just build a gd website for it
I am still traumatized
Company with a factory and HQ in different states had LOB app that ran some old server in the HQ. To print their factory work orders they had to print via serial>USB>USBoverEthernet>VPN>USBoverEthernet>USB>serial>server 1/2
Then the print job went from the server>serial>USB>USBoverEthernet>VPN>USBoverEthernet>USB>Serial>Printer. It broke one time and it brought the whole factory to a stop... They were a MSP client so it took me 6+ hours to figure out WTF I was looking at and fix it...
From a few jobs ago:
Boss: "We've been running a rcpd on Windows machines with targeted access control lists---"
Me: *pokes head up over cubicle wall*
Boss: "---yeah, yeah, you just sit down."
excuse me did you say _rcp_ as in the BSD4.4 protocol? D:
That's exactly why I poked my head over the cubical wall. I wasn't even listening to the conversation, I just sensed a disturbance in the Force.
Consultant for the health ministry in a Spanish speaking country to check why their database was so slow.
well it was build on Visual Fox Pro 9x, no indexes, 3 tables with 100 columns each, all the fields were strings.
I gave them options..
Their solution was to add more ram.
Saw the excel workbook a small utility company ran everything off once. And I mean everything. They'd somehow built a variety of auth stuff into it and a bunch of SQL queries. It was like an internal webapp if all you know is Excel.
Mine arent as bad as some.. But I am continually amazed how many "enterprise" solutions are Access "databases"
My first database was Excel. Don’t @ me.
Every industrial system older than 20 years is held together by chewing gum and twine. HMIs running windows nt with no spares and it bring the whole line down if it dies.
Wrote a DOS-Program in the 90s for a customer. Printing by sending ESC-P sequences to LPT:. Worked fine, until they bought a Windows GDI Printer. Sources no longer existent. Hex-edited LPT: to LPTR and built a Java-Parser for those sequences to render the pages in AWT.
Teach people to use ZZ next time. Works in vi mode, not ex mode, so not :ZZ, no colon.
Avoids that particular pitfall, and is faster because two times the same key.
A GPS NTP appliance with wired aerial, on a public IP address (but internal network) which noone had any login details for - and no support supplier - which provided authoritative time for the entire SCADA and Corporate network for a power company.
I remember a "magic" node_modules zip file on the company shared drive that was the only way the production JS app would build 🙂, it was something like 5 years old and we had to copy and paste it into the codebase instead of npm install
A domino server hosting Lotus Notes 🤢
Lightning struck a communications bank in the 80s and burned the half the board to a melted blank lump. The other half remained in use until about two months ago
A red coloured firewall.
Dosbox was business critical software as it was the only way we could run 2 critical systems that were written in 1995 & 1998 respectively...
I wrote an adhoc piece of SAS code. I moved on to different firms and 7 years later came back. Heard rumor that no one knows how to update this Janky but important code cause everyone only knows Python. I volunteered, and to my surprise- it was mine. Still had my old directory.
Felt like I made a real impact by leaving a legacy 👍
Windows
Extended the life of their existing cabling by installing Ethernet to Serial adaptors all throughout the network
WSUS Server ;)
The entire UK court service running bulk case issuing via 56k modems and windows 98 clients in the late 2000s. They then replace it with a soap service that accepts fixed width strings in a string array.
A large pharmacy PBM still has their primary plans and claims system running on a DOS green screen system (as of 2020)
FTP file transfer into a bank of "virtual" modems to copy files to & from an old Prime system
An IFRAME loaded JavaScript to refresh page content. It loaded financial data. Early 2000s
"Janky" means something good, right? Right?! 😟
I’m using a mainframe computer to do part of my current job.
Windows 98SE desktop and a Windows 2000 server for gold pricing, tracking, and reports. This was in 2012.
The server technically was connected to the internet, and that scared the shit out of me.
Worked at an org that had a 10 yr old air-gapped desktop pc running inventory software for their sword check-in/check-out (yes swords). IT didn't know it existed, it wasn't backed up, no one knew where the software came from, no one knew the last time it was turned off.
A custom workflow scheduler that had the worst code base I've ever seen with more bugs than features, but it was the CTO's pet project (even though he didn't have the time to maintain it), so there was no getting rid of it and pushing for its removal got me fired.
The CTO, while clearly intelligent (Has a PhD, though not in compsci), was not all that competent and is responsible for some pretty bad security holes that I'm not going to talk about as I doubt that they have fixed any of them.
I offer this screen cap from a major banking company customer service portal, just a couple of days ago:
I bet you could contribute to this thread
When I worked at Sun, most WOS ("wad of stuff") contributors had source control, but that did not include the hardware groups. We derisively called them the "home directory consolidation" and it was common to pull a production driver image from something like ~author/tmp/ce.2.12
I remember those. Stats store kinda started off that way tbf
I interned at a small shop that self hosted client websites back in the day. They had an array that would fail once a week and alarm across the whole office. Random employees would go up to the rack, eject the drive tray and slam it back in to 'fix' it.
Classic macos era Applescript that pre rendered html websites.
I thought it was clever. We ran a reports server for a 400 employee company on a desktop computer. We later turned it into a farm of ten desktop "servers" standing behind a load balancer, pushing the limits of the vendors licensing definitions.
When the farm couldn't keep up with the real-time demand, we added a report scheduler tool that would launch reports with a delay then notify the user when it was ready. The tool monitored server load and kicked off reports when it wasn't pinned at capacity.
Company was using an actual mainframe that you interacted with solely by text-mode interface on a dedicated console that showed “COPYRIGHT 1985…” at the bottom of the screen well into the 2000s. Like, past 2010.
All transactions were processed via this computer.
It was an actual IBM AS/400 running OS/400.
Part of the nightly duties included swapping the tapes on the tape drive…
In, like, 2012
I once had a backup server running a really old hard drive enclosure.
The enclosure had two separate power supplies failing constantly.
We bought a modular power supply and had molex connectors running into each drive for years before I was allowed to replace the enclosure.
Hope this qualifies: I had a team lead that was sitting in our FX trading group. Asked him if he was trading Euros? He said, “ never had one” :he thought I was talking about Gyros! Could have been close to lunch time! Hold the tzatziki sauce.
MS Excel to keep track of a 2,000 person workforce
... paying some schmuck $15 an hour to design and maintain it, and produce accurate pop reports
... and the schmuck had never used Excel before but really needed the work, so claimed he was an expert and learned via online help
One windows XP pc in 2015 that ran the payroll for the entire business with no back up.
Also an examinations authority that used a dot matrix printer to print all public examinations results and the “data” was gathered by a woman with a highlighter row by row. This was 2018.
I know that as of around 10 years ago, a significant element of the UK flight booking systems were dependent on a digital video camera pointed at a 1980s monochrome CRT monitor to literally screen scrape live flight information.
There had been several attempts to replace this feed and none of them were successful it would seem. I always wanted to hear about the attempted to shore up this system but no one was around anymore that had been involved. I'll always associate it with this Larson cartoon...
I was contracted to work on this system.
It's exactly as described. Fortunately my job was to pop the screen-scraped data in a database, although actually booking flights involved a weird terminal program (also screen scraped).
I'm certain there are plenty of others who've had to deal with it and I don't recall signing an NDA so could be some info out there on it. I'm afraid it's been so long now that I'm hazy on the details so don't have a huge amount more to add
I think the one I was working on was called JET2 - but the Travel Agents I worked for has since gone out of business...
Is there more information on this anywhere? Sounds wild
In the late 90's I worked at a dial access ISP that served the Eastern seaboard.
The user tracking (and therefore financial) system was Magic.
I'm not joking, there was a database company and product in the 90's called Magic which ran on DOS (Windows) machines.
So, predictably, there was a PC hardware machine in the back room that had the hostname "magic" and this company spared no expense in its construction.
By which I mean it was put together with leftover parts from the company's PC building side of the business.
Too important to give it something as useful as a "tower case," this machine was a motherboard laid out on a towel sitting on someone's desk. A desk fan nearby blew air on it. All its internal components just out for the viewing.
I worked morning shift, so I only got to read the reports about this later, but apparently towards the end of a workday the system went down hard. The first person on the scene shows up to find out all its RAM has been plucked out of its SIMM slots.
Someone just yanked em hot.
The back room this machine was in was between the technical support bullpen and the break room, so they immediately close the doors, shut down the phone lines, and ask everyone to line up.
One tech support staff is wearing a jacket. It's the south and it's summer. Only one.
The NOC super walks up to this person and just straightens their collar, right.
The SIMMs all fall out around the dude's feet.
Needless to say, they got fired.
The jankiest thing here, of course, is how the company treated tech support. It was a "loss leader." Techs were ranked by their ability to terminate phone calls the quickest without Karen escalations. You don't need to be helpful you just need to get the customer off the phone.
We tried over the years I was there to try and repair the company culture, but in the end the best we could do was just provide a promotion system to get out and into one of the engineering roles; we promoted a lot of techs into low level ops roles.
A system so old that it didn't have a real GUI and used the employee's initials for authentication. It was like an unholy marriage between a username and a password.
Worked at a company in the UK which had one ancient server running a critical app. If it he app failed the company would lose millions. It was so flaky that if powered off it may not power on again. The person who wrote the software had died and left limited documentation.
Our own products.
there was a time where all Square transactions had to be routed though a single TCP connection to the credit card network, emulating some old mainframe system that expected hardwired copper connections
it literally only allowed one connection, so if you lost the pipe you couldn’t reconnect until the other end was convinced your old one was truly dead
meantime the whole company’s service was down. so my team would get paged 📟. we got paged literally every night for a couple weeks, usually at 3am, with nothing to do except wait for
the server to let us reconnect
Three 42U racks full of servers, switches, and a random assortment of other networking gear, held together with batch files running in logged in admin sessions behind very simple passwords. All to provide telephony for a 400 person site.
Was eventually replaced with a resilient solution consisting of two 1U boxes. Sadly the replacement project managed to exclude getting rid of the old system from their project scope so it's all still sat there waiting for someone to find the time to dispose of it all.
It was a 14yo tech stack that we had to migrate to the cloud - everything was EOL, the Dev VP was running XP, no one really knew how it worked, and code deploys took a week (and rarely worked).
Then 1999 NetScreen firewall in production was the least janky component.
A cluster of over 300 servers running in a basement office hanging on Home Depot racks with zip ties and dripping thermal grease, the A/C was so over taxed we had to flush the system weekly via a hose on the floor to a toilet.
I client I was doing very occasional work for was still using Netware 4.1x in 2013.
a mysql trigger which answered phone calls
The “programmer” guy 😬
Feb 2000, my first job as an IT pro. Startup infosec company. We did automated security checks (port scanning, protocol identification, vuln checks). No server room, scanning machines were consumer desktops stacked on wooden IKEA racks in a small closet-size room without a fan.
When we upgraded all PCs to Office 2007 we got a call from one of our abrasives science labs. Their machine for testing abrasives durability had stopped working. It was a robot hand connected to some Lego Mindstorms kit, controlled by an Access database that didn’t work on 2007.
It had worked fine for years, after being made by one of the team’s kids when he was on work experience during high school. It was better than most industrial kit, so continued to serve a critical purpose.
This explanation is better than the description!
It was definitely better than the question: "can you drive up to Coventry to fix this Access database, so our Lego hand can start working again?"
I've worked at several organizations that used Oracle products
I saw a "data layer" sitting between the web server and database. It was a single class with over 20,000 lines of code and not a single unit test. We also had exclusive lockout source control at the file level so everyone had to wait until the locking engineer's work checked in.
Spotted in a file that hadn't been touched in years:
public String pluralize(String word) {
// TODO: find a better pluralize algorithm
return word + "s";
}
It’s a perfect storm of sketchiness
that brought me back 10 years
An entire high-performance database engine built entirely in PHP. As a single 10k line PHP file. Complete with a custom SQL parser. They wrote it because MySQL was too slow. 🤣 Oh and it was PHP 4 - all functions...
I believe it's still in use today in an enterprise product.
I worked for a n event booking company, with clients paying big money to plan 1000+ attendees itineraries and accommodations.
Our job was to split off a “new product” from a huge monolith - basically extracting code to make a separate deployable artifact with a fresh frontend…
The process was fascinating, we were sending code samples to the chief architect, because no one could understand what’s happening inside, except this single guy who spent ~10 years with the system.
Why doing it like that you would ask?
We had codebase consisting of god object and java methods of 5000 lines each.
5k was not even so bad, we could comb through it. But you can’t really understand whether this should be a part of a new system and where you should make a seam.
That was the only time when I’ve met a really unreadable code in production, where no one can be sure of what it does and whether it is still relevant.
If it’s get executed then it’s needed, right?
I’ve hit the JSP size limit for the first (and only) time in my career on that project.
Also, I’ve faced the first real deal prod incident - an SQL perf degradation made a service really slow (basically DOS) for 30 mins. Costed ~$500k in lost revenue.
I’ve never seen an email from a CEO written with 72pt font size ever before or after 😄
One of the greatest wins I’ve personally contributed was an “optimization” of a method chain call:
It was modifying hundreds of thousands records in several tables… and then just deleting all of that!
Shaved off ~10 secs of a use case processing time.
What a time to be alive!
I spent like 1.5 months trying to figure out what it was doing and whether there is anything to optimize at all.
That was a wild bet, but a rather cheap one: I was a cheap overseas outsourcing “resource”, and they can afford me sitting and reading through legacy code 😇
The product was a success at the end: it worked in IE6, and the main spreadsheet-like page loaded in less than 10 sec!
It was impossible with the main monolith and was considered a great success!
Overachieving the goals 😅
Tech? Ser - we use Excel only.
PPO I worked for handled data transactions with another company via a 486 machine running Corel Linux. The machine was under the Manager's credenza. He built it because he wanted to learn Linux.
That was one of jankier pieces of tech at that company.
When I first started working for my company a few years back they had a lotus Domino(?) database (I could be wrong about that) running on an optiplex G1 with Windows 2000 . It had customer records going back to the mid 90s if not earlier. It was not decommissioned until 2014.
The critical pager system at a hospital I used to work at ran on this old DOS box. On the back of it was a null modem cable that connected the two serial ports together. This was needed so two pieces of software could communicate with each other on the same system.
The APEX laboratory information system @NewcastleHosps @DXCTechnology @DedalusUKI ... Thankfully definitely being replaced over the new few years.
A custom built Java to .Net transpiler that has become unmaintainable because the developer no longer works for the company
A week after I was hired at a small, but multi-state, healthcare provider, another developer was let go and when his desktop workstation was decommissioned we learned that machine was the central processing unit for the company wide reporting application.
But wait, there's more: the database record schema was a column indicating what kind of data (string,float,integer,etc) the other text column contained.
At another job we used SCCS, so...
1990's... all stock exchange market data used by eveey trading terminal at investment banks was reported to the exchanges via 2400 baud modem uploaded to a Wildcat BBS running in a closet at the NYSE. I had to save data to a floppy and sneaker net to a diff machine for billing.
Bob from IT... Keeping the world turning since 1943!
Earlier this year, I found SQLi and XSS vulnerabilities in a PHP time management software from 2006.
Doing OSINT I found several “prestigious” colleges used the system, exposed to the internet.
It stopped being maintained in ‘13…
Disclosure was fun 😩
PHP TimeClock 1.04 SQLi/XSS Disclosure
A Deep Dive into Vulnerability Research with Docker and BurpSuite
tbutler.org
A calculator to determine pressure capacities for critical infrastructure written in Fortran.
Applicant Trackinf Systems
Goverment agency printed case files on a Windows server by invoking Word headlessly.
I worked at a telecomm company that's customer service system and provisioning system was underpinned by COBOL72 (as in 1972). When Y2K came, they had to get all of the retired programmers to come in for one last score and fix Y2K issues.
Sysadmins came to and said: "we're turning the whole system on and off for reasons"
I said no, they did and this old (crucial) AIX box dies and it backed up to one of its own partitions.
So I had to guess the last patches applied to the scripts as they weren't in CMS
I can say that Comcast relied WAY too long on mailing beta tape around for in-house production. Also as the singular junior dev who wasn’t paid much and who wrote the catalog software for tracking licenses of properties on those tapes they extremely got what they paid for.
I started working at a K-12 school district in the early 2000s and when I walked in on the first day their payroll server was running in RAID0, no one knew the Netware admin password, and when I hit the button to eject the backup tape the drive fell into the chassis. 🤷🏻♀️
and @IceMupppet have some good ones. most memorable was @arobodude when he'd edit the site on the fly (no version control or backups) and one day we lost it all
The time the firmware server got deleted and there were no backups. I had to code it again from memory in a day.
The flat json files hosted as a CMS for hundreds of thousands of users.
OrbBasic…
I could go on for awhile, it was all duct tape and bubble gum in the beginning.
Workday
My old PHP code
A Java SOAP web service that generates templated shell scripts that execute informix-4gl code and then capture and parse the text outputs of said shell scripts to return to the client
I will note this was in a year that started with 201x
In an internship I did there was a product that was an enterprise class java application, you name it - it had that design pattern, even ones that didn't make sense and there was one file with 500+ ifs that figured out what the hell it was meant to do.
Is it the 88k line C++ file with no classes and a .c extension? Is it using ClearCase?
Or is it the onrushing disaster of being forced to use a vendor’s Windows-only “MFA” app on our Linux system?
Windows 2000 Server in 2020
Best thing I've ready today. =)
Dell Computers
Hospital imaging system (think x-ray) server kept crashing because staff kept installing a Thomas Kincaid screen saver. Oh, and it's not a server but a desktop on the floor under a nurses station running XP. Hospital deems it mission critical in business continuity docs.
A payment processor's list of transactions in their own UI easily falls out of sync with the backend. If you click to Void the first in the list, but a new one was made since you loaded the page, bam you just voided the one that wasn't even visible.
We had a central automation system that controlled several television stations. Scheduled commercial breaks were easy, but ones rolled manually were done with a system lashed together with a nest of hand-etched circuit boards full of PIC microcontrollers (1/n)
(2/n) hand-coded in assembler. It absolutely had to work, because any lost break was many thousands of dollars of lost revenue, but the only person in the station who knew how it worked was the guy that built it.
(3/3) I was, of course, that guy.
A vax with a tiny amount of memory. Can’t tell you how many times we had to run in and find files to remove because there was not enough space left for the operating system to function.
A desktop Java app that queried an ancient Informix DB attempting to find an optimal distribution of shippable items through a process referred to as “the spin cycle.” This ultimately output thousands of shipping labels and maps telling warehouse workers where to find items.
Bonus that the output of “the spin cycle” was fed to MS Access for formatting and the print job would lock up the system for 30 minutes or so when sending the job to the printer. The label printing took hours.
386 tower PC running Netware. Case off, with a house fan blowing on it. The server ran a hospital around 1986.
Only way to download a file needed from the state, daily was over a modem connection, tbf it was secure connection.
They had to install a special drop for this line right next to the emergency POTS line when they switched everything to VOIP.
Wrote custom software just for this.
I was brought in to redesign a high traffic website that performed complex calculations for users by using Microsoft OLE to leverage Excel running on the same box as the IIS web server.
FileMaker. It was our CRM, billing, HR, sales, quoting and briefly also project tracking. Ugly as fuck and an absolute mess.
In a call center with about 30 people, the PABX had a serial cable to an SCO box that ran their billing program. I needed to tap that data but the PABX had only one serial, so I made a cable that had 2 heads, one w flow control to the SCO box, one w just TX from PABX to my system
It worked for at least 5 years after I left.
My half-ass deployment tool that I wrote because Ansible didn’t exist yet.
The sales org at a public global SaaS company running on a single dell tower under a desk
I went out to fix a non-booting NT4 server once.
I was told to take a USB floppy drive and some disks.
It was 2012.
Pure genius
A multi-billion dollar, major brand company runs on an ERP system from the 70s. It is non (pre?) relational. It’s literally one super-wide, single table where precious columns are re-purposed everywhere.
ETL pulls data into SQL. Then NoSQL then pulls it back to non-relational.
HORRIFIC!
Last month received an automated request by an ATS co complete a background check, when I clicked the location it opened Mapquest … Yes Mapquest
One place I worked we had a development “server” that was a desktop computer cobbled together with leftover parts shoved into a server rack. One day it stopped working. The CPU fan and heat sink were facing down and popped out along with the CPU.
Email server running on a laptop in a supply closet. No circulation and so the laptop melted and email was down for a day
marketing agency's airtable base (with the "over capacity" warning for 1+ year) with zapier hooks to send emails to clients or @ mention team members in Slack when item statuses were changed. this would have been ok-ish were it not for the GUI on top that ... didn't really work
Bank anti-fraud run by giving three people a green bar printout of all transactions for a day, each person would mark what they thought was "suspicious".
Phones as the only way to get internal tech support at a major Enterprise co., up until 2016 or so (as opposed to using a ticketing system). Drove me absolutely nuts.
University IT in the mid 2000s had a 15+ year old 70ppm 2 color Xerox laser printer which nearly all printing, including time-critical stuff like W-2s. A critical part of it was ASCII -> EBCDIC translation software which ran on a 386 shoved in a corner.
Am I understanding correctly that the printer was an EBCDIC printer? Because that is hilarious all on its own.
(My dad was an IBM sales engineer)
I'm not sure if it was but because they had bought it with the previous system (IBM mainframe) the controller was.
Gotcha. Hilarious and head-desking both. I'm assuming there was some sort of hard coding for printing as opposed to pure institutional inertia?
A long story of university budgets, etc. It was reliable.
My last experience with spending a very small part of a university budget (in the early '90s) multi-part carbonless form with each layer having various cutouts:
"Press very hard, you are making 8 copies"
I've written EBCDIC to ASCII converters too.
A company that ran old ibm token ring mainframe COBOL database - most of it written before many of the current employees were born. Everything interacted through a screen scraped terminal emulator via MS comti.
Tbh as a user one of the scariest things to me is medical clinics using paper forms that have been photocopied over & over & no one has original soft copy anymore. They make patients fill out by hand, then someone presumably types it into a totally diff electronic record
I keep thinking abt medical errors happening purely due to data entry issues.
Email server running on a desktop pc, manager ignored warnings, motherboard died, in an attempt to recover data we had a similar mobo, psu, etc spread out on a desk and it booted,mail came back up. We put caution tape around the desk until replacement came. Ran for a couple weeks
Novell 4.11
Doing some sales engineering for a network management company. National-scale mobile operator had whole network model in an XL sheet. I poked about and wrote a function to validate circuit routing with a bit if VBA.
Made a huge mistake: I showed it to the client to demonstrate they had some serious data issues and should buy our product. They stopped being interested in our product and became fixated on my ghastly VBA hack.
They were still using my 10-minute hack to plan routing for their entire network three years later. They might still be using it.
One day there's going to be a massive network outage in a medium-size European country and it will be my fault.
Siebel
*cries in Club 33 Member Services*
A few years ago, I worked support for POS software. Many locations didn’t have high speed internet so we connected to their systems using dialup. Their entire business hinges on me being able to dial in or walk a non-tech person through Linux terminal cmds
Worked at a place whose entire GIS mapping and work management tooling ran on Windows NT and to "modernise" it the entire server simply got @VMware converted so it could be run on modern hardware.
Same company had its customer billing system written in COBOL and we had to pay an external developer to maintain it who charged for features by the keystroke... 🤑
Oh sweet summer children: the correct and terrifying answer is of course the US power grid
This one might win the prize.
In 2005, the software used to track friendly forces in Iraq could only run on windows NT, on barely functional toughbooks. We couldn’t patch or even troubleshoot them. People would literally get killed
A company I worked for had an ancient x386 running their huge parts picker tower, because it had a software/hardware combo licence tied to something inside, which couldn't be replaced without significant expense.
When my husband does rounds at the hospital he writes his patient notes on index cards. He then drops them off at a random building where a woman enters them into the computer. The cost to link the his medical practice's system with the hospital is an unaffordable upgrade.
That sounds like something that would happen in the 1950’s. Wow.
Yeah, one day our young dog got hold of a stack of patient cards and shredded them. He had to go back to the hospital and essentially redo his rounds.
This is how the music in our lobby plays. No one knows how it got there. No one knows where the music originated from.
80% of veterinary clinics in the United States rely on record keeping software that started as CRM for piano tuners. And is written in Delphi and requires running your own local server.
This one is good 😳
I don’t know if this qualifies but in 1999 Akamai shipped us nine rack-mounted real-time encoders through the US mail which more or less disassembled themselves in transit.
2005 model OKK PLC unit. Older than dirt and covered in oil and wonder why there’s so many problems.
Microsoft Mail server for the Mac. Circa 1995.
Every two weeks it would crash the Quadra it was running on, so I eventually started going in on Monday morning about five minutes before everyone else came in to reboot it.
I was offshore and a sales guy had promised that we could send data from the rig to town. (1993). We didn't have a working satellite. I connected via a bag cell phone, through a 1200 baud modem. I had to telnet in, start a program called kermit to rcv, jump back and send data
I had to use AT commands to dial the modem. To send a MByte of data it took an hour, and if anything moved, the kluge of cables going from the phone box to a serial port would disconnect.
I started to explain to the client everything, he said "not interested"
Wooooooow. Hang in there little buddy.
Co I work for used to have a control station for sending runs in to process. It was a bank of 9, 9 position switches to enter run numbers, this had to match what was in the computer. 2/7 worked, and the 7 dead ones may not show up as the # in the window (1/2)
If anyone lost the post-it with the info for the 7 bad ones it had to be figured out by looking at the lights on the mid 80s PLC input card and figuring out how to convert binary to decimal
My first job, we sold an Enterprise management system for a very specific sector. On the surface a Windows client, server backend, Oracle DB.
REST API with JSON? SOAP XML? Wrong!
Pipe delimiteted strings streamed directly over TCP!
The user acceptance test for the new policy and claims system we were building at an insurance company — literally the backbone tech of the business — ran on a laptop next to my desk.
Note that the new system was built in some weird proprietary vendor framework built on Delphi. It was 2013.
We were replacing something written in Visual Basic 6 (main system) and the most complex Excel macro setup I’ve ever seen.
Again, literally the backbone of the business.
Was asked to “onboard” the AOL Mac client into our centralized CM/RM. Walked to the guy’s desk and saw the G3 under his desk. It held all the source code. Above it was his personal refrigerator. I imagined the impact of a power outage. I powered down and took it to a safe place.
Mimeograph! Lining up to make purple, damp, & smelly copies.
Lol, stop that! He will have no idea what you're talking about!
Classic. I remember my mom making something like these at home in a cake pan on gelatin.
SAP R3. Software designed to ensure perpetual employment of SAP consultants while nobody can do anything except things they do wrong because the UI is hostile to interpretation.
Windows PCs. I’m serious. And I’ve been using them since DOS 1.0 up through windows vista. Every time I try the new version of windows I just get nauseous.
Not a company but in the 90s my geologist boss had this calculator on his desk. Legend had it that it was the gold standard of its day (?)
Are those nixie tubes? 😻
I had to look up what those are but yes, I think so!
At a biz event in 2010, I met a guy who used Windows 1.0 for inventory and invoicing. He said, "Software is like a Crescent wrench. Unless I break it, why do I need a new Crescent wrench?"
Working for a state contractor, the agency that our site handles is literally run off a COBOL mainframe. It’s cheaper for them to pay a 10k fine per year than it is to move into the 21st Century.
This is why we don’t take direct payments & can’t change typos in names…
The logistics program SAP!….
The single comm LCpl who built the network.
Facebook circa 2010
I know for a fact that a major US financial company’s production system is a single mainframe written entirely in COBOL. The code is undocumented. They have a backup, that they’ve never tested.
Oh, also, the NYC subway’s credit card processing is dependent on a system that only one guy named Miguel knows how to operate
the MTA's fare-processing system relying on a password only known by some guy named Miguel seems like a bad idea. newyorker.com/magazine/2018/…
A Twix machine receiving data from overseas at 110 baud in Baudot code onto paper tape. (1978)
Due to HIPAA regulations in the mid-2000s, our process for sending files between offices for a state agency was:
Office 1: Print out PDF, fax to Office 2, shred printout.
Office 2: Receive fax, scan as a PDF, shred printout.
It terrifies me how much of our government still relies on fax machines.
I once worked at a place where everything, and I mean everything, ran on Lotus Notes.
There's been rumors for years that my universities class scheduling requires punch cards when it reboots
AS400 that ran compiled code that was blindly transferred off a burrows punch card machine. I was not realtime and needed data entered in a sequence. As the company grew the process the result drifted and the staff tweaked the numbers. They didn’t have the source code
Mission Command in the Army.
A very large print shop for a large media company used early NT servers.
You had one chance out of 50ish that the SCSI drivers would load at startup.
Servers had to be rebooted every 2 weeks.
I worked at a startup office with a single shared networking/janitorial closet.
One afternoon, Internet suddenly goes down. We spend 30 mins on phone with ISP. I eventually open the closet and start laughing because the fiber cable had been pulled by a falling mop pole.
FormFlow
Did a project in 2009 for a major newspaper publisher that had a 1975 FACOM mainframe that controlled large parts of their daily print runs. They had over 1,000 programs that they dared not touch...
Our landlords ran a recreational soccer league. Until they retired in 2015, they had a scheduling system where teams faxed in their field and schedule preferences, which were then entered manually into a LOTUS 123 sheet running on a Windows 3.1 box. 1/2
It then ran some set of miraculous scripts that managed to spit out a season schedule after 15 to 20 hrs. They wrote into our rental contract that we had to be willing to work for them - I made a Win 3.1 VM with the sheet and software so they had a backup (previously none). 2/2
We had a legacy app (desktop and web using OCX and Active X) that ran in windows server 2003 and they were unable to migrate it bc someone lost the source code. All hail to reverse engineering and visual studio debugger.
For years, the entire provisioning of 911 services for Rogers ran on a desktop. Since there was no one to restart it on the weekend, it had a ~30% failure rate.
Finding and fixing that was “fun.”
While being a plane captain on F-14s for the Navy, after the engines were started on the first $40 million dollar aircraft, we would have to use a $11 bottle jack to close the engine doors. After it landed we’d have to use the $75 rubber chocks to open the doors.
A good one is that apparently a lot of CT scanners can only be calibrated by taking a photo with an embedded Polaroid camera.
When Polaroid went under, stocks of film cartridges were stored in fridges at many hospitals for years …
Windows 3.1, this was in the last decade. An entire business was built around a specific industrial printer. Never upgraded, too costly. Now once it goes it goes the whole business needs to be redesigned. Last I knew they had three physical boxes left, then just drives & prayers
A script that ran every 8 hours to reboot the server and restart the app because that was easier than fixing and figuring out whatever was causing the memory buffer over run.
A box fan to keep the servers cool
This doesn’t count. Because improvising an antenna like that is brilliant!
challenge to see if you can beat ‘em all ^^
Windows ME.
...wait, I know: java.
I mean the answer to end all is... the Hubble still runs on the original computers.
Hubble Space Telescope sidelined by issue with its 1980s computer
"There is no definitive timeline yet as to when this will be completed, tested, and brought back to operational status," the Hubble operations team said.
space.com
Xymon 😂
Late 70s. Diplomatic Comms Network on an IBM 360 Mainframe. 2 non-tech users figured out the commands to crash the system. They were part-time farmers.If they were working on Thurs, when The Land newspaper came out, they would crash the system so they could read in peace
Hand copying product data from a mainframe terminal into an excel file that dynamically loaded/resized image files to output product labels with parts information to locate parts on an assembly floor. I wrote all the custom code. Paid less than a janitor.
I'll confess one: Had a switch bridging the FDDI and Fast Ethernet parts of the network (>20 yrs ago). Sometimes it would stop forwarding traffic. Wrote a script that pinged a server on the other side, and ran an expect script to bounce the interface if no response.
Ran for years that way. A <2 second outage wasn't really noticeable back then.
Is that jankier than the electrical outlets screwed into light sockets to power switches?
Do I not recall this being a thing?
It might have been after your time. And it's not like I was telling everyone about it. It worked really well.
I meant I remember doing the light socket thing someplace. It might have been on another old-ass campus…
I don't remember specifically where we might have done that but I'm also not going to rule out we didn't.
Home grown code generator. 100,000 lines of horrible C code that consumed handcrafted XML and output indecipherable Smalltalk code.
The solution was to spend a couple million to make it output even worse C# code.
Original WindowsNT SERVER and the (1) inability to stay “up” and (2) the “domain” system
I made tons of billable hours resurrecting WinNT servers 😂😂😂
Worked for a school district early 2010s. Our disk imaging setup was a Dell tower with two SATA cables hanging out of it, one red, one blue. Golden image HDD connects to red, new disk to blue, pop in a floppy disk with Symantec Ghost on it and power on. Don't mix up the colors!
We also used a lot of unmanaged switches with no loopback protection, so it was a regular issue where some student would plug a switch into itself and take down the network for half the building. We had to go room by room checking every switch to find it.
Best one was when a power surge killed a whole swath of drives in the districts SAN and we discovered that the former network admin that set it up, who had left for a new job just a few weeks prior, never put any sort of backup system in place. -
This happened the day before the start of the school year and we used roaming profiles, so when the teachers came in the next day for the first day of school they found that all their profiles, with all their curriculum materials, we're gone. They couldn't even take attendance.
The district had to spend a ton of money to send the drives off to a data recovery firm to get everything back up, took about two weeks and not everything was recovered. I was really glad I was just a lowly admin and not one of our two engineers, they didn't sleep for a few days
A firm that executed about 10% of the trades on NASDAQ ran on PDP-11s until about ten years ago (20 years after DEC shipped the last model of the PDP-11)
I have worked at two separate newspapers in the 21st century that used modern page layout software to print pages that were then physically pasted up with a waxer like it was 1970.
An emulation of a minicomputer. Users would connect with an IBM terminal.
The IBM 3151 terminal was TOO NEW FOR THIS SYSTEM, so had too put a card in it to emulate an EVEN OLDER terminal 😬
Early 2000's, small company needed to encrypt/decrypt credit card numbers they stored. They had a proprietary executable to do this. Other code would invoke the exe, thread sleep until it spit out encrypted/decrypted value to stdout, pipe to file, read in file & then delete it.
I feel like @Foone probably has a few for this bin.
At one job, we had a system that: had a graphical editor over a JSON database => which it then iterated over from nodejs => indexing into a hand-rolled dictionary nested 5 layers deep =>
1/
=> to access format strings => which it concatenated together into a ruby program => saved to disk => forked a separate process to run that ruby program => which depended on a hand-built library => that spun up a headless chrome docker image => 2/
that made selenium calls => in order to write automated UI tests for our application.
We called it the Ruby Goldberg machine, and I rejoiced so much the day we killed it.
Commodore 64 Still Powering Auto Repair Shop
Commodore 64 Still Powering Auto Repair Shop in Poland
Why throw out what works?
popularmechanics.com
Aside from janky power bricks, I bet my childhood C64 would still boot if I had it
I worked for a (now-defunct) retail pharmacy store in 2000, with over 4500 stores whose ONLY means of ordering new inventory was to dial into a bank of eight OS2 Warp desktops with 3600 baud modems.
No one on the entire IT staff knew anything about OS2 Warp, so each desktop had a DVD taped to the front and if anything went wrong, you simply re-ghosted it.
I was brought in as a short term hired gun to a medical company in about 2001. The problem was that each page on their website took 60s to load (auth system was *slow*). By caching several tables (and thus eliminating a quintuple join) I got the auth to around 10 ms.
1/2
But local management hated that, as they felt the long auth time “increased security”. They declined to put in my fix. Got a call 6 months later saying that upper management had declared “thou shalt fix this”… and presto, my solution was in place.
In a cubicle long, long ago, a PC sat unsecured. It had millions in e-commerce transactions passing through it. I was given “easy to follow” instructions in the event it was ever down. The command prompt was readily available to all who passed by.
[this is a joke] objective-c
Glad you clarified it was a joke…
How about Clarion?
A programming environment where huge chunks cannot be under version control..
Or maybe Visual Source Safe…
softvelocity.com
I worked for a major US Class 1 Railroad about 10 years ago. And they LIVED in Lotus Notes. Gawd, what a craptastic platform. I have never seen Lotus Notes since leaving the RR. And come to think of it had never seen it in use before the RR either.
I only read about this one, but it still keeps me up at night.
r/sysadmin - A Dropbox account gave me stomach ulcers
4,805 votes and 609 comments so far on Reddit
reddit.com
A former employer stored all their banking info on a shared excel sheet that ANYONE could access from the network drive. I was responding to a ticket for excel and stumbled upon it. Yikes is all I gotta say
I’m not tech enough to contribute, but talked to a nuclear engineer one time & he told me about the punch cards still being used at a nuclear plant… he was retired, but still had to go in twice a week to, per his terms, “fix shit.”
physical: a single server rack with all company critical operations with a single plug into the wall, directly below the ethernet trunk which was precariously loose
logical: if Facebook didn't ship for a week, all serving nodes would have been marked out of date and drained
An old windows box running a vb6 app that transferred data from a few hundred stores to the erp system at HQ. It sat under his desk plugged into the wall. This was a $billion+ yearly revenue company.
SQL Server 2000 RTM. Actually that was a great product in 2000, but the company was intentionally using it on new deployments in 2016 (last I knew).
They were seriously coding around known bugs because they could not update it.
Lotus notes for email. It would bug out in any number of random ways. Menus stopped opening, email text would look like size 3 or 47, and general crashes and stalls. Was responsible for the vast majority of all support calls.
I deployed a desktop shortcut for all users called "fix lotus notes". What it really did was kill the process, delete temp or cache files, then open it again.
Once I trained users to click that, I bet our support calls dropped 30%.
We kept a 20-year-old personal printer and a nearly as old laptop in the IT closet for printing the Christmas gift checks using an ancient version of Quicken because no one knew how to set up the checks to print on any of the new corporate laser printers
We FTPS (not a typo) a csv of credit cards to our payment processor every night. At the end of the next day we'd FTPS an excel spreadsheet off of the same server.
Said excel sheet would often have typos, sometimes in the credit cards.
We passed a PCI audit.
(just to clarify, all passed tense. Not that they don't do that anymore I just don't work there)
Oh man... I think that would have to be NICE, later bought by RSA and renamed Envision. Worst attempt at a SIEM in history... Windows only, 100% poorly written java, accessed by an even worse Java thick client, and both were 15 or so years ago (so you know how bad Java was)
I once rented space in a datacenter... this was in maybe 2010?, that depended on an old HP workstation running windows 2000 to run the doors.
The thing was, nobody was monitoring it. It was setup well to begin with, with a hardware RAID5, and it was solid hardware, but
I got asked to look at the thing, 'cause it wouldn't boot. BIOS wouldn't boot 'cause the fan behind the hot swap power supplies had died. I couldn't figure the connection, (and there would be many other parts required) so I had him buy a donor box from ebay.
I swapped the fan, got it booted up, but... red lights on the drives. I replaced the drives out of my own collection and as soon as you replaced one drive and the rebuild finished, it failed another drive. I replaced all five drives, and it seemed happy?
and I'm like "you should probably get a new keycard system" and found him some quotes. Nope! that thing was in there, as far as I know, until the next big remodel, many years later.
In Manhattan in the 90’s, I saw an 8086 PC with a custom card installed that ran a mission-critical function for a Fortune 500 company. The engineer who built the card had been retired for years. They had it running with the case removed to reduce overheating.
I have a hunch @TinkerSec or @TryCatchHCF might have a good one (:
Decades ago I was hired by a Wall St firm as software lead to fix product line.
I found their self-taught programmer had hand-tweeked custom builds for each customer. No baseline, no CM.
No customer had same mix of binaries. All same version #.
We processed $billions daily.
Mine's simple silly shit. Server 2000 and Windows NT in a "modern" shop... had to find some old hacks to bust into them.
Split them open and pivoted into their code base to seed the code with my malware.
The jankiest is in a place I can't talk about. Ask me when we get whiskey.
When the company grew large enough to move out of the CEOs basement... the new location didn't have reliable internet.
We built a 4 mile line-of-site network to the CEOs home in order to survive the 1 year until fiber arrived
JIT too before an apt complex was built breaking LoS
The law firm that used Dropbox as its file server.
A pizza shop it do IT for still uses their RapidFire Pizza POS on old ATX computers. Those things are more then 20 years old and seen HD replacements, power supply failures, and more. Still working though. I would like to one day get the customer data and put it on something else
When I worked on Second Life we had a problem where people couldn't download the installer. The download config ran on a web interface running on a PC under someone's desk. The web interface didn't respond so I restarted the machine...and then all of Second Life login went down
Vista for Airbus software ...
Last job was at a pizzeria. Closet in the back room had telecom equipment branded Bell Atlantic. Lots of thin wires with paper tags hanging on em. Status/activity lights were on. No one knew what it did, exactly, because it was installed in the early 80s. Couldn't be turned off.
To say nothing of the mess of cables actually running the place. Two routers and a hardware firewall visible identical to a router. Cables running next to gas lines. I never knew it was possible to have multiple single points of failure.
I have in my little pile of keepsakes a Ticketeer key from my days as an Amtrak ticket clerk in Detroit. The Burroughs Ticketeer was a motorized but mechanical device for quickly producing railroad tickets for set fares...
Someone would ask for a one-way ticket from Detroit to Chicago, and you're insert the so-called matrix for DET-CHI into the machine, which would spit out the ticket and add an entry to a paper journal for accounting purposes.
The matrix was a metal object that might have been 12 or 14 inches long, with a small handle at one end as a label. These were prepared elsewhere and had to be updated when fares changed, which wasn't that often in Detroit circa 1973.
It was a fast way to produce a lot of unreserved tickets; just keep shoving the matrix in. For a time the Detroit office also produced pre-printed Ticketeer tickets for some of the other stations in Michigan, since we had the only two Ticketeers in the state.
We had matrices from Detroit to Ann Arbor ($2 one way), Jackson ($4), Battle Creek (maybe $5), Kalamazoo, Niles, and Chicago ($13).
I had worked in Ann Arbor and had to hand-write and hand-stamp every single ticket till Detroit got the Ann Arbor matrices.
Ticket stock (accountable) went into an interior hopper, as did the journal paper.
There was also a crank in there; in theory if the power failed you could turn the crank to put the matrix through its paces and get a ticket. I don't think we ever tried.
Detroit got ARTS capability (reservations and ticketing terminals and printer) around 1976; I was the first ticket clerk trained to operate them. They made the Ticketeers obsolete; much faster ticketing, more accurate accounting.
Four years later, more or less, I was at Amtrak HQ; I was the person in charge of creating online training for ARROW, the replacement for the ARTS system.
*** Knowing the deep knowledge of rail historians and fans, I am astonished not to find a picture of a Ticketeer machine. They were heavily used in Amtrak's Northeast Corridor.
I do have the Ticketeer key my ticket-clerk brother sent me when the Detroit station was shut down.
This promotional film from Burroughs circa 1956 says in the archive notes that it has footage of the Ticketeer in use.
…troithistorical.pastperfectonline.com/archive/27A886…
The Ticketeer is at 12:00 - see
The Power to Serve, A Presentation of Burroughs Corporation (1956)
2015.011.01216mm black and white promotional film about the Burroughs Corporation, entitled "The Power to Serve." The film covers the company's scope, its hi...
youtube.com
Just funny…worked on a healthcare app, clinical trials. Pearl. 20 years old. Found 20yo comments like…not sure if we want to do it like this, should do it differently…App worked. Code has to work, not be inspirational.
A regional phone company that made all their profit with enhanced phone switches that ran on an operating system custom designed by the CEOs college roommate who was learning C for the first time. It stopped working if you patched bugs that other parts of the system depended on.
1st company I worked at held office in souped up shipping containers. All production servers were located in-office too. This became problematic during summer, because these containers get _hot_. The solution was to just hose down the roof every 45 mins during business hours 🥵
So, let me tell you the story of the Bar Exam...
#lawtwitter
That is spot on
I worked for an org that skimped on switches, so the head of IT put together a map of ports to machines so they could keep track.
They forgot to put servers on that map and wondered why the whole network suddenly went down.
Am not sure of it qualifies as 'tech'..
6 years ago I worked for a client, who asked us to fill paper forms for approving creation of tables, queues, etc. in dev environment..
10 year old computer in a closet with water heater. This ran the banks internet banking and bill pay system. No firewall or router, internet modem straight into the computer.
I know of assembly lines in multi-million companies, which have critical parts running on a code, made by some worker in the 80s-90s, who has since retired. If something goes wrong, they'll call the guy who coded it to help. And don't even consider replacing the ancient tech.
It’s a toss-up between the cheapest big enough safe found at Costco or something to contain the keys to customer racks in a data center which we eventually had to call a locksmith to drill open… or the text file that controlled auth for a whole isp.
Windows
1/n
At NASA in the 90's, we had to use a commercial database maintained by ops to map disk inodes to data on robot-managed tape. Ops was new to db mgt, and the journal file filled all of main disk such that the Amdahl mainframe wouldn't boot..
2/n
due to license restrictions of some sort. It came down to the most-expensive contractor editing the OS binary in hex, making one change that let everything recover over about 3 days. Our group decided to write our own db solution, since we found suspect partial writes..
3/n
on recovery. The goal was a db that only required word-level write consistency. The odd part (not janky, I cheated) was that I found a B-tree index package archived from the code-download dialups that were reappearing on the (pre-web) internet. It had a license, and..
4/n
it took me months to track down the surprised author and tell him NASA wanted to give him money. The code was written for DOS on PC, and worked great on the SysV Unix we had.
Windows NT server that handled phone registration for a college. The reg software had a memory leak so the server had to be rebooted every Monday or it would stop working at some point later in the week.
Another winner:
HR timecard software that used an n^2 search algorithm. The org created a fresh database every year so it didn’t get too slow to use. They also kept a Win98 machine to print reports from because it wouldn’t print from XP.
Kronos.
University: when SUN 3/260 server had severe harddrive/hd controller problems & no money 2 fix, I turned it into a diskless client, turned a diskless 3/60 client into a diskless server & spread filesystem over several heterogeneous systems since no 1 system had enough hd space.
& if I remember correctly, did it with very little down time. Obviously not ideal but it worked, teachers were able 2 edit/print exams students were able 2 finish term projects etc. Eventually (I think after I graduated (& not SA anymore)) they bought a Sparcstation 2 replace it.
also everyone continued to be able to send/receive email (that's one of the reasons I made the 3/60 a server - needed a mail server for the subdomain)
Air Force Nuclear arsenal is intentionally old and janky...can't hack something that dumb. 🤣
Giant global company using a proprietary database entry tools for import/export papers. Legally required to be kept available for 5 years. Except they had a "per major revision" license... And the database could not be migrated from v.x to v.x+1 /1
So they had 5 licenses in order to comply with the 5 year records window.
Except they "slimmed down" to only the latest 2 licenses... Except one department of 3 people in a different country got to keep theirs. /2
So the workflow now moved from "boot v.x of software to get data" to "how old is the data requested? Send an email and call as there is a ticking clock".
Did I mention the data was usually requested by gov. Revenue and customs authorities?
Fun times.
I worked in a safeway store years ago where the functional register display was 2 lines tall by about 40 chars wide, the customer saw a modern led monitor which was a skin for this register.
A typical operation on this system was like this '1', 'override', 'price', '1000', 'upc'(this could be entered via barcode scanner. The display would show "O 1000 'upc'" You then had to hit 'override' and scan in the magic overide key to force the item's price to 10$
Downloaded torrents of 2009 Iran protest videos for a major media company from the Starbucks across the street because IT wouldn't let me get through the firewall
SMS alerting system for 5000+ client companies was a Perl script hooked up to a cheap mobile phone sitting in a rack in the corner of our office, with a battery that was ready to melt. Operational for more years than my employment… and I was there for over 10 years.
Damnit you stole mine!
I presume @mowog was responsible for that 😂
I have NO IDEA what you’re talking about, Senator.
Also is that REALLY the jankiest piece of tech you could come up with from that hole? Whole place was BUILT on jank. I ought to know 😂
In house MySQL replication being the shining exception…
It was the first thing that came to mind because it was one of the last things I fixed before I left 😂
Does an underpaid undergrad running and servicing the department servers count? We also have computers running Windows 7, because some software's website hasn't updated their requirements.
Well, there are the billion dollar telescopes controlled by Forth running on a PDP8 using 8” floppies still out there ;)
My dad ran lights for a strip club in London in the 60s, and if the fuses blew he was expected to put on a pair of industrial rubber gloves and hold a six inch nail across the fuse contacts.
The dial-up ISP I at when I was a teenager had a radius server running on a Pentium 90. Someone had to kick start the CPU fan a few times a day or it would reboot and cause authentication to fail.
In the 1980s a hospital’s lab interface stopped communicating to the Hospital Information System. We decided to trace the connection, a serial cable under the DC floor that turned out to be 2 cables held together with a rubber band and friction.
Once my whole job was transferring data from the Windows 95 computer that ran the proprietary bit of software that ran their operations to a modern computer that could process and invoice the order, had the sense not to connect the 95 to I've Internet but still
I've seen a lot of companies depend on Facebook. Does that count?
I don’t have technical details, but in 2008, my husband was a contract software developer for a bank machine company. He was, and remains, profoundly disturbed by the sheer ignorance of basic information security protocols in that industry.
So much NT4 in production
45 year old COBOL code which encoded basically every piece of British pensions legislatory change in that time.
8-inch floppies in the university history department office when we were using 3.5s, back in the day. Like working in a museum.
Hotfixes that involved copying and pasting compiled .class files onto a running server
1500 lines of if/elseif searching for literal hashtags in a string to determine what to display on a CMS page. The database also had plain text passwords and varchar for every field (including dates)
I worked at Novell from 2001 to 2007 and believe me there were endless stories of customers who had a Netware box running undisturbed in a closet somewhere, a box that never rebooted and no one dared unplug.
Without scrolling through all these I do hope the old BBC News 24 master time clock, that was overlaid onto the live screen image, that was generated from VERY old PC - literally stuffed in cupboard - is mentioned somewhere....
This has been a very entertaining read. It’s not just IT where stuff like this happens. And it’s true what they say about duct tape.
Mail handling in bash(!), formatter creates pages long '}', needs a lexer to understand what happens when under what condition?
Worked at a museum, the DVRs for the security system would break constantly and were so old the only place to find replacements / parts were on ebay.
One of the chemistry machines (HPLC) at a university ran off a computer on Windows 98, because the company that made the machine only does whole system upgrades (££££) rather than just the software.
You’d be depressed at how many critical desktop business apps still use MS Access 97 as their database
Dude in Y2K I worked for a company where all we did was write Jankware integrations to screen-scrape people's IBM 3270 apps, pull CICS codebooks into XML, keep old EDI shit running, etc. etc. This is all so familiar.
I once worked at a failing company where one of the NT4 servers wouldn't ping *itself*, they couldn't afford the maintenance contract for the OS/2 software (mission critical), and their dot matrix printer was *essential* for certification. It failed. No hot spare.
Ex-sysadmin here. A Netware 4 server with all the financial data of the company on it, but with RAID 5 array running in degraded mode for 5+ years. Finances repeatedly rejected the purchase order of a replacement SCSI disk for being "too expensive". (year: ~2002)
I left the company, ~3 years later it finally died. But before I left, I put together a Linux backup server from old hardware about to be scrapped, and hid it in a closet in the IT office. It still ran, my passwords still worked. So I could help them recover from the disaster.
That was some nice pocket money, and a _really_ good laugh when they called me desperately pleading for help. Right, @rATdrAg0n? (He was the sysadmin there before me, we did the recovery together.)
Loads of horrifying software engineering stories written here 😱. thedailywtf.com
Galaxy Tab 3 and Galaxy Tab 4; Horribly underpowered things, very prone to frame skipping, with different specs across a generation. Laggy UI out of the box + rendering issues galore + Multiple logcat spam per frame sucked any fun out of development. Slower than the Tab 2 iirc.
A shoddy oId un-maintained forklift. I’d have been proper fucked for loading & unloading trucks if it broke down. But it was me that would’ve been fucked, not the boss, so he was slack about the maintenance schedule on it :/
Digiboards on SCO Unix.
My budy was data entry guy for QC packages into a database in late 80s
He took a vacation
While he was gone they removed a filing cabinet next to his desk
Someone was nice enough to take the refrigerator magnets off the cabinet & place them on his PC in the area of the hard drive
He spent four weeks working late to restore all the data that went "missing" on his PC after he got back from vacation
He told me I am never again taking a vacation while I am here, I can't afford the cost of it 😜
Someone I know was involved in renovations at a major airport, during the course of which an undocumented token ring network was discovered, attached to a wall that was to be demolished. Because it was passing traffic, and nobody knew if it was critical to the operation of (1/2)
the airport, they had to very carefully demolish the wall around the thing and reroute it above the suspended ceiling. To this day, it is still there beep-booping away, doing something unknown but possibly critical. (2/2)
All computer backups were reliant upon a third party tool to map a unc path so the backup service account would think its locally attached disk
This all reminds me of an article from a few years back where NASA needed programmers who knew . . . FORTRAN.
Why NASA Needs a Programmer Fluent In 60-Year-Old Code
To keep the Voyager 1 and 2 crafts going, NASA's new hire has to know FORTRAN and assembly languages.
popularmechanics.com
3M post-it's
Also.
El Torito CD-ROM images: "let's boot off a CD by writing an image of a floppy disk to the CD, which gets loaded into RAM at boot time so we can emulate a floppy disk and we don't have to reinvent DOS booting"
Software, hardware or either?
It was before my time, but I once worked at an online property auction company, and their entire property management system before I joined was an Excel spreadsheet called 'the masterlist'.
We had a Sony Discman running on repeat with a classical music CD in it for over 10 years. It hung on the side of the server rack and provided the hold music for our phone system. It spun for at least 100,000 hours before a power surge killed it.
My first television truck which the previous set of engineers put together themselves. God I loved that Truck.
The first company I worked for in Italy was using a programming language that was basically PHP meets Reverse Polish Notation 🤯🥲
Zero tests, silos knowledge, it would take weeks to build things that usually takes hours.
My cousin worked at a company that built planetariums. When I visited maybe 6 years ago, for legacy/compatibility reasons they had a whole team on Commodore type workstations with the big floppy disks & magnetic tape cassettes
As recently as 2017 a banking customer was running their main database on an NT4 server hosted on Virtual Server 2005. It bluescreened 9 out of every 10 boots.
A programer write a module or app that became an integral part of the business. It lived on his development sun workstation at his desk so Everytime he rebooted, updated, etc the app would go offline, sometimes for days.
I’ve worked in several government offices that operate using HP laptops and Windows OS
A major French investment bank relying on Excel 2007 (and giving traders second PCs just to run it) to help decide on key positions in one of their trading desks.
In the 90s the database that managed smogchecks for CA ran in an oracle db. Every car in CA had a record that had the last three checks recorded. Back in the day there were no live backups, you needed to take the db down....
The db was so large that it could not be backed up during off hours.
Worked at a call center where a radio tuned to a local station was the source of the hold music. Agents needed to manually tune to another station for an hour every Sunday morning while Christian programming was airing. Heaven forbid callers would hear something about Jesus!
Microsoft Lync 2010’s underlying APIs