See the entire conversation

War stories please. What’s the jankiest piece of tech you’ve seen a company depend on?
1,442 replies and sub-replies as of Dec 17 2021

You all are making my day.
The hits keep coming.
Many years ago, I worked at a research chair and was told to take care of its infrastructure. In the server room, there was an old SUN workstation running 24/7 and nobody knew why. All of my predecessors were too afraid to turn it off and simply called it "the heater".
Technical debt as performance art. twitter.com/erindataviz/st…
Oooh do I have some stories. One group had to perform complex calculations for all our 60k+ clients, had no IT support and were entirely staffed by 22 yr old business majors who had never heard of databases. (1)
This is the scariest thing I've read today. twitter.com/tdawry/status/…
The UK's Trident (nuclear weapon) submarines still run on Windows XP
Not all heroes wear capes. (Although if you do wear a cape Simon that's cool and I don't judge.) twitter.com/sbisson/status…
The booking service for a major shipping company that was a stack of three different generations of Oracle technologies. Layer 1 and 2 were supported, layer 2 and 3 were supported. But the whole stack? Nope. It was 3 years into a six month development program. I cancelled it.
The replies are an escalating hall of horrors. twitter.com/jeanmarcalkazz…
Worked at an insurance company where all financial data was stored on 1 dusty Windows NT 4 server. The day before closing the fiscal year it crashed and couldn't connect to the AD. Had to search for drivers and OS patches on archive.org and use 3x💾 to boot from...
This is why we can't have nice things. twitter.com/chrisalbon/sta…
I once wrote a Python module that allowed non-engineer staff members to edit a Google Sheet and then those changes would overwrite the production database. I wonder if it is still running.
Sometimes I get weighed down worrying about how far my systems are from ideal, but it's good to remember that my real goal is just to make them a little less horrible. twitter.com/dbredesen/stat…
I worked at an online travel company, and their entire travel booking flow was implemented in a single function with thousands of lines of code, called DoAllTasks(). It somehow managed to process thousands of bookings per day. @rayploski @cbredesen @KonnectedKris
On dark days when your own tech debt makes you cringe, you can come back to this thread for comfort. twitter.com/BruceEdiger/st…
SPARCStation-20 with a SDLC sbus card to communicate between a western US baby bell and Dun & Bradstreet. All customer credit ratings checked via this system. No earlier than 2007. They had no spare SDLC card.
Responses keep blurring the line between entertaining and terrifying. twitter.com/justinlmitchel…
Does the fact that basically all of the US healthcare system is run on EHRs that are using MUMPS count?
This is now in my head playing out as a Bugs Bunny cartoon.
I once ran lights for a live community theatre production where impending light cues were frantically mapped onto a giant plug board wall with cloth-insulated patch cables, and then executed by throwing a Frankenstein blade switch into the opposite position.
The House of Cards IRL
I would have been thrilled with a single janky piece of tech everything depended on instead of a precariously built non-interoperable tangle of many systems built out of fragments of janky tech held together by nothing more than hopes, where any one piece could bring things down.
Under the category of “happened in a company you’ve definitely heard of”.
In the very early days of Twitter, a production facing service was deployed on a Mac Mini that just lived on the floor next a bunch of random things and cables. Everytime someone tripped over the cables, it would disconnect the Mac Mini and the service would go down. Fun times!
Rube Goldberg, hold @patio11’s beer.
The first version of the Bingo Card Creator CMS needed to create PNGs and PDFs of bingo cards given word lists, but it was my first Ruby/Rails project. I did not yet actually have a functioning SaaS version of the app, just the downloadable Java swing app. So here’s workflow:
This is the best definition of “janky” I’ve seen.
Code or process that is written like an inauspicious game of Jenga where the removal or failure of any single piece brings it all crashing down. 😂
The entire computing world is held together by duct tape and bailing wire.
The actuarial dept for the workplace div of a very large insurance company relied on an app running on a old desktop under the developer’s desk. He had a stack of 3 and always forgot which one was “production”.
You may not like it but this is what peak performance looks like.
Global financial markets will crumble without Excel spreadsheets.
Some sins are unforgivable. But still kind of understandable.
I worked at a health care company responsible for notifying millions of hospitalised patients of surgeries, prescription info, & emergencies. One engineer built the application complex enough where only he understood how to edit the main java files to guarantee his employment.
“…before finally catching fire…”
Best "If it ain't broke ..." - An IBM XT PC running key switches in a telephone exchange for a whole suburb. It was on 24/7 for 20 years before finally catching fire and burning down a chunk of the telephone exchange.
I hope one of you is currently writing a treatise titled On the Causes of Computing System Fragility and Why It’s Never Ever Going Away
So, I worked somewhere, that the CEO and founder insisted to write code, about 7 years later after they migrated away from the system he developed, they printed the source code which had 4000 lines of switch cases.
We’re living Mad Max but digital
first magazine I ever worked for rebooted itself after failing, moved to new office, had no internet but had to close first print issue night, do or die -- we stole open wifi from office next door and connected our LAN to it; intern had to stay up all night to keep it running
And these are only the tip of the iceberg
nice try, i have an NDA
It’s looking more and more like these stories are the rule, not the exception
At Bing years ago we had a service that ran 100s of classifiers on every query. There was 1 editable config file for everything. Once another team disabled our classifier when they accidentally deleted the last line of our classifier’s configuration while editing their config.
This backbone thread is just a taste. Get a drink and read the other replies. They are exquisite.
I used to have to remote control a desktop over Zoom to ssh into an on-prem server (the on-prem eng had to log me into several time-out-ing terminal windows w/ his creds because I didn't have ssh creds and he wasn't going to sit there and watch me work) and code in vim.
If this thread were a Western
There is a bowling alley near my house that still runs their small business off of an Apple IIe with a dot matrix printer. No joke. When I asked about it, the owner looked at me, said "don't fix what ain't broke" and then then back to spraying shoes.
Literal war stories
Actual war story about ANET, the system NATO forces were using to track their progress training the Afghan military. From this piece on the Defense Digital Service's efforts to overhaul it: wired.com/2017/05/meet-n…
paper clips
Serial null-modem cable made out of paper clips. FWIW, a paper clip will support 115.2Kbps.
Soapbox time What is to be learned from this catalog of monstrosities? Takes of all temperatures welcome.
you said companies, but the academic science lab duct-tape as solution game is very strong
I ran RS232 50m using a shielded twisted pair cable and we maintained 115kbps.
Maybe not on this level, but I train my AIs with a bottle opener.
Doing deep learning with a bottle opener...
That seems highly appropriate.
Well done the bowl owner for not wasting more money on the Apple users pay 30% more rip-off. ;)
Very short treatise. "People." -fin
No time, gotta unscrew this vlookup.
We've all worked with that person before. Sadly they still get fired and the company moves on without them when they could have been part of the evolution and transformation to bigger bolder and better. #klingon
All the traders at the I-banks are wedded to Excel, so now all the prop models need an army of coders using Excel Visual Basic to force them through spreadsheets...
pretty sure that's how AWS works
This is a great thread, Brandon. Thanks... I think(?) :)
You're most welcome. You get a sleepless night! And you get a sleepless night! And you get a sleepless night!
Hospitality company’s worldwide reservation system ran on a single instance of hardware and software that was n-4 beyond support and DBA’s would kill processes daily to keep the system from crashing. Nightmare support scenario.
Sometimes you get to use your consultancy powers for good.
NO CAPES DAHLINK
Some users still haven't done so – according to StatCounter, about 0.59% of Windows computers are still running XP. NetMarketShare puts that figure at 0.32%.- Aug 20, 2021
Plenty of elevators out there running XP Embedded, serviced with iPaqs
Not really if it an air gapped system on industrial hardware. Would be stable and predictable.
sure but at least they are water gapped
That's much more recent than I would've guessed.
War story, indeed.
As a professional programmer, they may be better off than with newer technology especially from Microsoft!
I’m sorry I read that. I preferred being blissfully ignorant.
IIRC Lot's of things still run on DOS (which I would trust more than XP).
At least the submarine does not need to wait every other morning for the next pointless and time consuming update from MS for Windows 10. See my profile.
Nice. It took my a while before I could stop giggling long enough to type this.
Thank you. Personal computers were a thing of the future when I was on subs. Very little digital computing was employed and that was very specialized.
"We are all going to die"
this is definitely better than running any modern OS with all the telemetry etc
It's actually stable
Is it? Imagine them running on Windows 10.
Echoing someone earlier; air-gap critical systems. OS/2 in use by a municipal utility ~2005.
I'm assuming they still use AOL dial-up to connect to the internet as well.
Thanks for saying something that doesn't need to be made public.
Adds new meaning to the BSOD... 🤣
I'm sure this required the Chinese to teach a whole bunch of 20 yr old hackers how to work with XP. Starting with "what is XP"? Which I suppose could be seen as a clever security gambit on the part of the Brits...
Best not to think about the old icbm silos that still need 5.25" floppy discs.
Also I supported WinXp a couple years ago. There was a version with real-time extensions for industrial automation. Likely similar reason for the Trident dependency.
If you were in the UK I'd punt a guess that it might have been a British Telecom or other supplier's data traffic logger. Nobody onsite would be likely to look at one of those :)
I have seen occurances in companies where the solution to identify the owner and purpose was to power off or disable networking on a device/unplug it till someone complains 🤣
Not mine but about 10 years ago someone posted a pic of a tire alignment shop that was using a device connected to a C64
In 2013, while working on telehealth, due to legacy systems compatibility, corporate compliance, and liability, we had to make sure our internal portal web application supported IE8...
In Japan there was this company whose internet access was solely through a Win98 computer. The was in 2011
Until a few years ago, the server that handled our phones at work was running on win 98. The phones regularly went out, you couldn't check voicemail from outside the building and it didn't usually work inside the building. New IT guys were afraid to touch it. It was wonderful.
Pretty sure the train in Atlanta Hartsfield airport still runs on an onboard VIC-20, or at least did a few years ago. The 22 column display on board gave it away.
I know what I’m checking out the next time I fly through there
I'm going to cheat on the word 'depend' because this is my best story - Depending on an SR-71 to test the antenna tracking on a LEO satellite program, until the air force got tired of lending it to us (also the sonic boom was sending the local bovines into counseling)
Whoa. That's like using a GPU as a paperweight.
To be fair, there was no obvious way to test how antennas would track stuff moving at 10,000 miles an hour (LEO speed). That said - even if we'd given the problem serious thought, SR-71 idea would have been on the 3-sigma end of ridiculous
To be additionally fair, and while they may leak profusely on the ground, SR-71s hardly seem to qualify as "janky" tech overall.
I didn't know, I think 'leaking fuel on the ground and requiring an immediate in-air refuel after a high speed run to close the gaps' qualifies as pretty janky to me...
Hard to top this one, indeed.
Me. I’m the jankiest piece of tech companies have relied on.
But. But. But. You’re too young!
I clicked on this thread knowing I’d see insurance or banking mention COBOL and it’s right at the top.
it was 10 years ago! to be fair i built the model and worked with the developer that did it and he was 20+ years my senior my mans is a legend had a poisson regression with 500K coefficients 😂
An (ancient) MSAccess - running on a VM - used for data entry into the MSSQL for content to be presented on the web. It may have been a good idea at the time, when Microsoft support VBA in MSAccess...
me - I’m like old grandpa who still knows how to turn off the faucet if it starts leaking… 🙄
laptop that used excel, vbscript, access and windows scheduled tasks to run their reports. i was hired to cast that ancient demon into the void
When you need to call an old IT guy and a young IT guy.
Many years ago, I worked at a research chair and was told to take care of its infrastructure. In the server room, there was an old SUN workstation running 24/7 and nobody knew why. All of my predecessors were too afraid to turn it off and simply called it "the heater".
“… and there it has been, ever since. It is said that late some nights, its fans can be heard *humming* the Macarena tune, which was atop all the charts on the year it was first powered on.”
Yes, that's how it was. Sometimes, I could hear the tune at night.😂
There was an ancient Sun x86 workstation nicknamed 'roadrunner' after the beeps it made when it booted.
If there was anything that could run 24/7, it was a Sun. (You've never seen anyone try to heat a cubicle with Windows, have you?)
That wasn't even intended to be a play on words. Sun computers (running SunOS) were just really reliable (expensive, but reliable).
We ran Sun servers (with Solaris) specifically for their reliability.
we used to test our software on Sun, reboot once every two years
AS400 was another 24/7 workhorse.
I heated my basement home office with a desktop Pentium 4 running XP. Even worked through the worst snowstorm of the decade.
Truly elegant thermal design. Air comes in one side, exits the other, and everything inside is aligned so air flows across it. You can even sit the case on its end without blocking the outlets thanks to the ridged plastic molding. PC/Mac airflow was a total mess by comparison.
Well... maybe somebody secretly did that and lived in the server room... 😉
This has such a mythical quality to it. Ancient beings from ages before brought a mystical object that no one knows what it is anymore, other than a source of mystery and some unknowable power
The way this tweet is taking off right now, I'm already thinking about making a movie from this story.
I would definitely pay $18 to watch it.
Ok, I'll take your word for it. The theatrical poster is already done:
In the chilly server room, Shrouded in a misty gloom, Rests a source of ancient power— Down we kneel before the tower And we beg it not us shun; Hear our prayers, oh god of SUN
I still don't know. 🤷‍♂️
It ran yp/nis/nfs/rpc and gave hackers easy access to your network.
Is it still heating today?
Ok, but wouldn’t a person worth their salt do some form of pcap on it to see who it was talking with? The clients would then offer some insight?!
Well, the infrastructure of this chair was maintained by student assistants coming and going every one or two years. Of course, nobody did any further digging and every new assistant coming in was just happy that things were running more or less smoothly.
This exact thing applies to so much in academic research
Did anybody even still have login credentials for it?
Not that I'm aware of.
I’m pretty sure this is what the hacker kids found when they broke into the Cold War era missile silo.
Sounds like the origin story for “Deep Thought” .. u should check if 42 is on the screen
Did it get as hot as the sun?
's @AreciboRadar used to be controlled by a Sun box that was older than some of the graduate students. That was considered an improvement over the original VAX, which I never had to use but which was still legendary among the observatory staff.
One day, someone somewhere is going to turn one of these machines off, and the Simulation will finally, mercifully, end.
This is stunning they should have left it offerings
Chesterton’s fence 2.0
And one day they turned it off and the “Y2K” crash finally happened 😜
Some people say it's still running to this day
Reminds me of the time I was upgrading a load of Netware servers in a financial company - there were two we simply couldn't find. Still happily file and print serving, but invisible to the human eye.
Since you said "company" I am going to assume that doesn't include universities. Phew.
A nonprofit I worked for early in my career justified its entire existence with an 'economic impact calculator' (spreadsheet) provided by a vendor with all of the logic hidden behind password protected spreadsheet tabs. And nobody (including the vendor) could remember the 🔑
Ah yes, black-box data manipulation. The best kind of data manipulation.
Scientologists and "e-meters"... 🤭
A massive set of VMWare clusters hosting identical VMs - each with MS SQL and a .net app being hosted for 10,000+ customers. Each licensed and manually maintained without AD because “they had to be separate for security reasons”
Separately, in 2007 a MAJOR national telecom did its quarterly network capacity planning by having local analyst export excel reports from individual controllers. Then perform manual roll up, cut and paste into regional excel files. Then same for three more levels to national.
I remember in my first adtech job, had to debug an ad request api error for the smoking remnants of one of the late 90's tech darlings... didn't find the problem, but they were running PHP 3 on their backend in 2008
Another one, when meetup let me go, I had a bunch of automated email reports, ETLs, fun bots etc. running on my dev box sitting at my desk that I used as a footrest. I did tell them how to access the code to salvage. About a year later friends say they finally retired my box
If I’m honest, I have left a sea of technical debt at every job I’ve had. But in my defense, managers basically beg for it.
We were building an atmospheric obs system with several generations of receivers. Old ones had been running for yrs, so ppl left them alone. All of a sudden, they stopped talking. We rummaged through closets til we found the KERMIT protocol manual & learned some macgyver skills
[we had literally tried absolutely everything and were about to accept the loss of lots of expensive equipment... just couldn't get it to communicate... glad someone was like "maybe someone left some breadcrumbs in these dusty closets"]
trying kermit was the last-ditch-of-last-ditches effort, and we wouldn't have thought of it if we didn't find the old manual and say "what the hell"
Please do not keep me hanging...what was wrong? Did the Count overflow and needed Kermit to fix it? #RealQuestion #BadJoke
I worked IT support in college. I helped a professor convert a dataset that was collected by a local city. It needed to be converted because the city kept all their data in Lotus. In 2007.
you would be shocked at the level of continued dependence on lotus at many financial institutions to this day (I’m in my mid 20s I should not know what lotus even is)
Came here for the Lotus Notes, didn't fail! Here's your .id file now crack on haha
Lotus 1-2-3 was released for 32-bit Windows in 2002 in Smartsuite 9.8 and got fixed until Fixpack 6 in 2008. archive.ph/20120719070454… It's not *that* long ago. Of course, nothing says that they have to be running 1-2-3/Win (which I always felt should have been named Win-2-3).
Lotus Notes, circa 2010-2012
Big 4 accounting firm I see you
5.25" floppies, running MS DOS 5, in the mid 2000s to drive manufacturing for a large multinational.
An old coworker told me that a large national retail company was doing its ordering and inventory entirely in Excel with no VBA or other code, and no backups.
I ran a 300-person brain training lab. Students did intense timed memory exercises on laptops. One day they started complaining: tiny ants crawling on their hands/screens, messing up data collection. That night I put laptops in cabinet, filled w bug spray and slammed the door.
If you’re wondering I did clean off the laptops the next day including the dead ants. This actually worked. No more ant infestation and the experiment went on for months without a hitch thereafter.
It sounds like you found some hardware bugs
So you can proudly say you've done some "Live debugging"
Oooh do I have some stories. One group had to perform complex calculations for all our 60k+ clients, had no IT support and were entirely staffed by 22 yr old business majors who had never heard of databases. (1)
They created an insane Excel sheet, 1 copy per client. By the end there were TERABYTES of Excel sheets, each containing tons of sensitive personal info, stored unprotected on a giant shared drive. Management didn't see this as a problem. (2)
By "the end" I mean by the time I left the company. I am positive nothing has changed. (3/3).
‘terabytes of excel sheets’ is a four word horror story
I can also add “thousands of Access databases” Same company
why use one db when thousand do trick
One db good, thousand is better
Information retention overlords are up in our biz on this stuff finally....I'm not saying I'm grateful, but something had to be done with those 15 year old shared drives no one had access to anymore
The journey of a single database begins with a thousand
Not one I recommend working for 😉
At least they’re not stored in hundreds of boxes of badly labeled floppy disks.
they call me the king of the spreadsheets. got them all printed out on my bedsheets.
If I had to work with terabytes of excel sheets, I'd delete myself
I once had to do a bunch of sorts on a mini mainframe. At the rate we charged computer time that one run would have more than paid for the computer. I asked the vp if he wanted to adjust the bill. His only Q: would it take longer to do by hand?
Sub <finance teams> with any and all who are drawn to the greatest database of them all
I am a Business Major. I started learning how to code #rstats and Python because of a 30 sheet workbook that was intended to solve an timetabling optimization problem was not reproducible. It hit me, the optimizer is not optimal. How do you get to terabytes of Excel, how Sway?
Effectively it was a naive form of "sharding", they probably had ~60,000 separate Excel 97-2003 workbooks, the total storage of them in filesystem could conceivably add up to terabytes.
Oooh no this is the stuff of nightmares. I had to clean approximately 3000 Excel sheets, I told the team it isn't sustainable. We need a database.
I worked for a large corp whose revenue reporting depended on dozens of interconnected Excel workbooks. A handful of query workbooks pulled data from the POS system using an extension that only worked in Excel 2k. So they had VMs w/ Excel 2k for those...
All others sourced from the "query" workbooks (and each other). There was a team of ~30 who came in at 4am to kick off the query workbooks on the VMs, then open the other workbooks in a very particular order & run the macros that populated data from their various dependencies.
Oh yes, this very much rings a bell
this story made me feel kinda sick, tbh
thanks for sharing it!
You’re making me feel better about our current attempts to replace access and excel workarounds with an integrated information management system. Amazing that we get as much work done as we do 🤷‍♀️
business major here, i only know microsoft access but even then i literally do not know how it works
I know you wrote a thread, but just this one tweet is already the stuff of nightmares.
I was tasked with building a streamflow model at the @USGS in Urbana as an intern. Nice, modern stacks Had to fix the rainfall ETL once, the hydrologist on staff smirked and gave me his FORTRAN77 book said “good luck kid”
Homegrown web interface for deployment that had no cli interface.
All of financial trading ran on AIM until they shut it down. Weirdest interface to use at work.
Technically not my former company, but we were monetizing access to government data. These data were technically available cheaply, but on top of being horrendously messy, were stored on magnetic tapes in a 1975 IBM mainframe. '75 because everything was encoded in EBCDIC.
library automation company used by dozens of major US universities rely on a 2 million line legacy Cobol code base that no one at the company understood or was capable of maintaining
*Definitely not me*, but a friend built: Google Sheets -> SQL table -> Prod ML model It's still running great. 🙃
cutover script 😅
Ooooof literally though
How do we revert the revert of the revert
A very famous website that you’ll definitely have heard of used to have a data center in SFO. I started working there and they took me to the data center, and there was a cable in the middle labelled “do not TOUCH the cable” Never mind unplugging it, you weren’t allowed to touch
Okay but how did they put the label on the cable?
Very carefully.
Pen & paper, Google Sheet + Zappier is a classic. For the itemised pay of thousands of people. It broke in creative ways _every_ time. Otherwise, not a company but a public hospital the patient files are on a… Oh so, you know that story already?
Student-lead corp, ~2000. All data is on a MS Access db (very modern) with a clear data-object structure, etc. But every method, class, etc. named through to graphic adult-themed metaphor: contracts are… “movement”, subcontracting involves several parties, pay is “consuming”.
I remember having to ask “What does _that_ mean?” to horrified classmates because I didn’t want to google it, but I needed to understand what that class method did and I had no idea how many people were involved in… ::gesticulates:: at screen. No one believed me.
I had a large airline customer years ago who migrated a lot of services from one VM to another but didn’t want to update all their service code that pointed to “localhost”. This is where I learned it’s sometimes possible to point localhost to… another server.
Took weeks to debug a problem and figure out wtf went where.
Same customer who mixed tabs and spaces in all their Python code. FML.
This project also included code comments in multiple languages. There was a _lot_ of Google translate involved in debugging.
Multi-lingual comments are a new one for me, but in retrospect, this should probably be way more common than it is.
yeah; I often talk about coding being anglocentric down to the var and const, but I hardly ever see processes for routinely translating code comments
I expect there is a lot of valuable non-English commentary that just doesn't get written.
the localhost one was pretty bad, but I groaned out loud when I read this
Some people live to see the world burn.
Until I set my tab stop to 8 it realllly confused me as to how any of the code actually ran.
hairs standing up on my neck now
I’ve seen a firewall where every computer had to set its gateway to 127.0.0.2
it's just a change to one file. we'll do it once and not speak of it again.
In 2016 working for a company that made the complete billing (1k+ B2B customers) via MS Access running in Win 98 on a server next to the coffee machine. 😱
(probably not used at companies) but the jankiest pieces of tech I've seen are these MS Word plugins that come with reference/paper managing software (Mendeley, Endnote, Zotero, you name it)
Oh, and any software related to German banks (or maybe just DKB). It's so bad that it's not even funny. Like if you want to set up the mandatory TAN app, you have to use the TAN app to be able log in and let them send you a temporary code to your physical address.
Of course, this doesn't work, so you have to email them about it. The letter they will then send you takes like 10 business days, and you have to do it each time you get a new phone, because the app doesn't work if you restore the phone from a backup like any other app.
You're just talking about the Word plugins, right? Because standalone @zotero is awesome.
Yeah. They often screw up the numbering big time. Or at least that was my experience when I used them back then in collaborations (we had to use Word). The standalone app is okay. That is, it is not amazing but it does what it is supposed to do.
Those software companies employ enterprise sales professionals. These plugins are indeed used at companies.😂
The funny thing about these plugins is you can literally see the code running in slow-motion in real time
The booking service for a major shipping company that was a stack of three different generations of Oracle technologies. Layer 1 and 2 were supported, layer 2 and 3 were supported. But the whole stack? Nope. It was 3 years into a six month development program. I cancelled it.
Safety system built on Lotus Notes.
O. M. G. Lotus 🤦‍♀️🤦‍♀️🤦‍♀️ Yes I’m old. My first portable computer was the size of a carry-on bag, ran on DOS and weighed 30 pounds. 🤣🤣🤣
Omg that was the one! The keyboard was attached by an old-fashioned looking phone cord, and it snapped onto the bottom of the unit. There was an actual luggage handle on top! Floppy disks - TWO!! 🤦‍♀️🤣
I actually may still have it in the garage. 😂
My company at the time tried to upgrade to Oracle 11. Inbound and outbound loading docks were offline for 5 months, developers had cots in the halls and when asked for a backout plan, the VP said "we don't have one" Luckily I wasn't in that division
shopping App! So uncomfortable and useless
A company with several thousand installed locations ran their entire payments clearance system (multiple parties get paid for each transaction) on a single FoxPro database/app sitting on a Windows server box under a guy's desk. As late as 2015. Maybe *still*.
In all honesty, that seems far more stable than a modern Javascript front end talking to, say, mongodb.
I think there was some pipeline at Goldman Sachs that involved sending a fax somewhere in the middle.
A git repo within which were nested several dozen more git repos.
A Quip machine.
Previous company calculated all of their new sales data (new contracts) with a team that entered the contract details into an Access db. There were four or five members but they could only work in the DB one at a time. They finally upgraded to SQL server a year after I joined.
Worked at an insurance company where all financial data was stored on 1 dusty Windows NT 4 server. The day before closing the fiscal year it crashed and couldn't connect to the AD. Had to search for drivers and OS patches on archive.org and use 3x💾 to boot from...
"Had to search for drivers and OS patches on archive.org" .... dear sweet baby Jesus. I didn't even know that was a thing.
Now everyone reading this go donate to archive.org
NT4 was a great system for its day. However, that day was before the turn of the millennium. 1/
My war story: I was responsible for the care and feeding of a System 360/65 and a Burroughs B6700. One fine day, a Grand Pooh-Bah dropped off 8 boxes of cards, directing me to feed them through the 360 and submit the generated reports. 2/
There was no printing at the top of the cards, but I was able to read ALGOL code. I asked my manager to tell milord what I had found, and I could hear the blowback 3 offices away. (He quit the next week.) I fed the cards to the IBM, with two boxes of error dumps as result. 3/
I then fed the cards to the B6700, which produced a series of entirely legible reports. I sent both to milord, along with his original cards, and asked why he had me feed the wrong computer. He and his retinue appeared 30 min later, shouting that he never made 4/
such mistakes and it was obviously my fault. He fired me then and there, watching me clear my desk. I have never been so happy to be fired, and told him so. Everyone around him was certain he’d have a stroke. Karma is a wonderful thing. That was 35 years ago now. 5/5
hey, we ported it rather quickly to 5
I was porting Perl4 to Perl5 in 2006. After I’d started playing with Moose:
ouch This was around 1996. There was some 4isms in it. Mostly module'submodule stuff.
I think we finally got rid of our Vax in 2015 or so? Can't believe how long they limped that thing along for very minimal use.
I remember watching two rows of VAX equipment beying decommissioned at a Dutch hospital in 2001/2002 and that stuff was ancient back then 🥺😳
Way back when I was in high school (80's) I got a job that was basically writing Lotus 1-2-3 macros to make Lotus 1-2-3 into a database. Fortunately they listened to me that they needed a real database or they'd probably still be running those macros today.
Lotus 1-2-3 *was* a database. That was the "2". "1" was spreadsheet mode; "2" was the DBMS functionality (queries, sorting, etc.) "3" was because it could draw graphs, too. That was the entire point of its name.
Far from the worst in this thread, but in high school (circa 2005) I had a job shadow with a temp agency that had all of their payroll and scheduling running in an MS-DOS virtual machine inside of Windows XP.
a friend of a friend told me her soon to be retired dad was one of the few remaining COBOL programmers who was responsible for keeping social security disability checks going out, and they had no plans to either upgrade the system or train new programmers.
This is the dirty little secret of most US federal and many state agencies.
My Dad worked for a company whose in-house medical insurance was run from a Borland Paradox database some guy wrote in his spare time. I met the guy at his retirement party that my Dad attended 1/
Nearly a decade later, I ended up working for the same company & met the supposedly retired guy, still working… the company was *still* running their medical insurance on his little Paradox DB & had him now working as a contractor! He must have been well into his 70s! 2/2
Saw at Fortune XX company a piece of mission critical software running on MS DOS 6.22
My ADHD evaluator last year had a set of custom programs & database he runs on a Windows 2000 virtual machine. Has only keyboard/mouse/display out of the vm. He would manually edit the sql tables if some data entry was wrong.
Does the JVM count?
A hospital had a pill dispenser in their pharmacy based on a Compaq 286. In 2014.
Optometrist had an eye laser surgery machine hooked up to a Win XP host with internet access in 2011. 😬
What else would it be? Win XP EOL was 2014...
It would be offline, is what.
I’d also accept not a desktop OS with a mountain of known vulnerabilities. Gotta get images off of it somehow, but “email them to myself using the web browser” ain’t it. 😬
idk what y’all are talking about but win XP was the way to go for that kind of equipment in 2011. Win 7 SP1 had just shipped in feb 2011, you could not expect medical equipment to run on that immediately.
In 2012, I saw a whole hospital on XP with Novell directory service. At least they didn't have to worry about Active Directory getting hacked....
Sonogram machines - custom OS MRI - custom OS CT Scanner - custom OS A desktop OS with a web browser and local admin for the user was definitely not “the way to go” for anything involving patient safety.
I’ve seen a few embedded Win things, but at least they weren’t local admin with desktop apps 😬
There was a blood analysis thing that ran on unpatched Win2k. The vendor refused to patch and would cut support if anyone patched (as actually is the fucked up standard by most med vendors using Win), but at least that couldn’t do direct harm if something went wrong.
FDA changed authorization rules about 5 years ago to allow security updates/ OS upgrades, IIRC, but manufacturers still claim "costs too much to recert"
I once worked for a multibillion dollar corporation that was running its entire datacenter's border traffic through a five port Linksys switch from Best Buy.
I still have a 98 PC in production, can't be virtualized. Runs a CNC saw whose maker went out of business shortly after it was purchased. Cost like half a mill new, would cost about 4-5mil to replace now. They are aware "eventually this will die, and that's it"
This is a common problem with old CNC machines. Pharmacy robots as well. Hell any robot. I know of a certain popular humanoid robot that runs admin through https with no ssh, but is still vulnerable to heartbleed!
A friend's custom guitar business used a Commodore with a 5 1/4" drive to drive a CNC.
One day I will tell a story of a Windows XP machine that hung on far too long in life for somewhat similar reasons.
I recommend a qualified therapist. I know a trauma expert if you need.
We have an entire production line dependent on our ability to keep this PC operational. The clone died recently and this is all that is holding it together. Our attempts to duplicate the output have failed but management says the line will be upgraded next year
And I thought needing to have PCI slots was bad (I need PCs with PCI slots for one specific fieldbus card).
These days you could probably duplicate it with a USB converter box and microcontrollers.
IIRC you can still get motherboards for modern CPUs with ISA and/or PCI
Yeah, there are options, but they often kinda suck, and may not line up with smooth "fleet sales" through IT. In fact, there is a PCIe version, but I'd have to write some new driver wrappers for it. It's all a matter of saving the effort and hoping the equipment hangs on :D
So is the whole system proprietary or just the connections?
Parallel port? Serial port?
What I am asking is how is the computer connected to the machine?
Don't remember, I've never had hands on it myself. MSP, different engineer's account. I just consulted.
Ahh, was under the wrong impression here. Thought I could help with a solution. My Uncle runs a bunch of nearly dinosaur era machines. It bothers me because he won’t let me dev a more modern control solution. Something about not knowing how to operate them.
I'm sure a hardware hacker with the proper skills an a couple hundred in Pi bits and pieces could work out a new controller... but that would require taking it out of production for rev-eng and testing, which means it may fucking break 😸😼
And that still leaves the totally custom software package, which won't even start under emulation (we tried...)
I would suggest actually waiting until the machine is stopped and interface the connections to data log and track all of the operations to decode the signal under normal operating conditions. If there is an emergency protocol, you only would need the code once or so.
Ya, not really my area. I'm more of a communications infrastructure and security kitty. But I certainly have some hardware hackers in my circle if/when the time comes I'll make the suggestion.
That’s a lot of money to lose when that machine has unexpected down time.
As far as controllers, there’s a few that are better suited than Raspberry. You could even write software to signal the supervisor in case of a real problem.
Wood and metal fabrication shops are the worst. Windows xp and earlier systems for days.
I worked at a place that relied on one person who operated "the machine" that was used in processing contract initiation paperwork. If they were out sick or busy, you couldn't start work. You took your paperwork, if you were lucky they'd do it while you waited but probably not.
Anyway, "the machine" turned out to be an electric typewriter and I revolutionized the job opening process by showing people how to use a pdf editor. The only drawback was that I became the operator of "the machine".
We had a product which depended on learning from responses provided by humans, if you can believe such a thing!
I worked at an online travel company, and their entire travel booking flow was implemented in a single function with thousands of lines of code, called DoAllTasks(). It somehow managed to process thousands of bookings per day. @rayploski @cbredesen @KonnectedKris
DoAllTasks(). Wow, I blocked that from my memory.
An abomination of 30KLOC (if memory serves) of VisualBasic. 😅
By the time we all left, DoAllTasks() simply orchestrated to functional areas. Took lots of effort if I remember correctly.
I thought we scrapped it with the Java™ migration, but if you told me it was still powering cruise bookings somewhere, I’d believe you.
Let’s be honest, if we saw the Java code that replaced things, we’d probably be equally horrified today as we were by DoAllTasks() in 2001 - if not more so. BroadVision anyone? What a PoS!
But JFC what. a. nightmare.
Honestly though this isn’t even in the same ball park when compared to a project that my old company worked on … a legacy system rescue project for a FTSE 100 company… 🧵
Imagine a stock control system written in Pick Basic (look it up) that has been hacked over and over again for two and half decades to support the acquisition and merger of 100s of small business over it is life time. A reference implementation every known anti-pattern to man.
It ran (by ran I mean melted the CPUs constantly at 100%) on two HP Mainframe Servers whose hardware and O/S were both obsolete and unsupported. It’s database was NoSQL before SQL became the trendy new thing. All environments, including dev ran on the prod hardware.
There was one dev that had been repeated bribed to continue working part time on the system having reached retirement age and a couple of Pick Basic contractors who knew just about enough to recover from crashes and make very small changes.
The “pricing engine” of the stock system (which has a myriad of unique pricing code implemented with liberal usage of if/then and goto statements) was such a convoluted mess of spaghetti code that nobody knew how the business logic it worked, or how it was meant to work (specs)
The whole thing was a ticking time bomb and an existential risk to the biz. The CFO confessed that the only way they knew they were pricing to make a profit is that there was more money in the bank account at the end of each month than they started with.
The database is an integral part of the OS, as is the BASIC. It was a very powerful system in its day & a strong competitor to early UNIX™. Several derivatives are still around: rocketsoftware.com/products/rocke…
Home
(no description)
openqm.com
(No comment)
what is DoAllTasks() but int main(void) persevering
Encapsulation is overrated.
I worked at a different travel booking company where the website talked to the booking service using Java RMI. Every time we deployed to the website, if the shared code had changed, we had to stop taking bookings until we also updated the booking service 😂
another travel booking company, we had agent terminals on the GDS mainframes whose interface was written by a company out of business for years, given to us as 32 bit ELF libraries compiled on RHEL6 (1999!) Friend got them running 15yr later in a 64bit JVM by hand relinking them
Impressive but...
I work at yet another online booking company. It's not as bad as these others, but I am very fond of the name of the login function that does SetUserStuff().
Well, maybe not as bad. We do still have classic asp vbscript running in 2021
I’ll see you that and raise you classic ASP with *JScript*
My first BASIC interpreter didn’t have local variables. Subroutines weren’t so terribly useful in this case. Maybe those guys never advanced from that level.
If only we’d had GOTO statements and line numbers…
At least it's implicitly documented with a relevant function name
Hey, at least they named it accurately. Good documentation is the first step to good code?
An ex coworker told me he went to work at IBM and they had a .NET WCF endpoint which was about 80k lines. 1 function/endpoint, with 100s of if statements to manage financial transactions. He tried to break it up into something manageable but management shot it down.
I came here to say WPF, WCF, and Silverlight as a train wreck of a tech stack choice I saw coming a mile away. Sadly, our director drank the kool-aid. We know how that turned out.
at one the places I worked at, the entire billing, invoicing and finance system - was one PHP file, which was a giant tree of if/else statements, all relying on register_globals and there was one comment in the entire file at the very top, which was //filename.php
Ok, but that's actually kinda cool.
I respect the naming convention here
I think I would have nightmares if I had to work on something like this 😱
I worked at a big one where they decided to use a single db table for all the data (?!) so columns were named as codes (a1, c5, d7, ?!?) and there was a document mapping them to what they mean (?!?!) + a Hibernate mapping. All queries were done app-side, so no joins or counts
I ran a wind power and weather monitoring station in the Guatemalan highlands on an eepc for years that the client in Hong Kong communicated with via Skype chat. You'd chat with it asking what's the power output, send me logs for, or what's the wether and it dutifully responded.
all gold. Workd at a smb firm, we once virtialized a win95 box because it handled the accounting database, and it was held together with dust, cobwebs, and dumb luck. not sure how the original hd was still spinning. they refused to pay for db upgrade, so they still use win95.
In the year of our lord 2017 I decommissioned a Windows NT4 VM because I was able to replace an Access Database with a module in ServiceNow. It was our inventory.
Fed gov requires online form to be filled out. Won't share db with state gov. So submitter takes screenshot of fed web form and faxes to state. State hand enters into Excel file. Turnover at state, so every year or two a new resource modifies the Excel format.
List of items need to be entered for each site. Too hard to repeat site id for each row, so they enter once and leave other rows with blank id. Depending on mood, they fill in the first or last row, with no way after the fact to tell which.
I've seen actual production code written in French. Not "if then else" but "SI ALORS SINON". I'm French myself but I wasn't ready for that.
Wait, what? In what programming language?
It was an ad-hoc language within an homemade software, to define the behavior of some front-end elements. It wasn't a full-fledged language, basic instructions, you would write a couple of lines per element, <100 for the most complex behaviors.
Of course all the software was in French. Funny enough, I was taught how to use it by non-French speakers, who knew by heart where was which menu.
Learning how to use this must have been such a pain for them!
I once worked on an Italian computer. (Yeah bad idea). Among other things one of the packages that came w the computer (converting ascii into some other format maybe? ) crashed w prob 1.0 if the input included one particular ascii character 🙄 😅
Sorry for the bad ethnic joke. Turns out it was a British computer with an Italian name
This last detail is the cherry on top
a mix and match open source data platform which was supposed to do real time operations reporting at customer sites... but ran batch processing on a nightly cadence.
oh no. no no no.
I once had a process break 1/3rd of the time. Turned out it was because at some point there was .NET framework pretending to be an IBM 3720 terminal talking to a mainframe, and it didn't like lowercase letters.
Realise you wanted more given the other replies 😂
Oh, we've seen *many* votes for Excel. You're in good company.
I’m glad I’m not alone, but I’m sad at how broken the world is …
Excel disguised as an ERP.
Excel to fill the gaps of your bad ERP...
Blow it up. Blow it ALL up.
SPARCStation-20 with a SDLC sbus card to communicate between a western US baby bell and Dun & Bradstreet. All customer credit ratings checked via this system. No earlier than 2007. They had no spare SDLC card.
If you know who these folks are...I have a spare SPARCStation-20 and SDLC card.
Does using Great Plains as a technical support ticketing system for techs (field and internal) supporting telco for a couple thousand standalone facilities make your skin crawl, or is it just me?
Worked at a company in the mid-2000s running accounts payable off a FORTRAN-based ERP called MANMAN on a 1970's era HP 3000. We didn't have a FORTRAN programmer on staff and just hoped that it all just kept working.
If I knew COBOL, I'd damn well not be writing it for free.
I worked at a place where data queries we're done by calling an overseas team who looked at a dataset, typed rows into Excel, then sent it over email 1-8 days later, depending on size. No metrics were stable over any pd. Constant source of internal arguments
That looks about right 😂😂
whatever your answer, it is outdone by what appears on this website, which has been around for years. thedailywtf.com
US Courts using Lotus Notes….
Used to work with a guy who worked at well funded VC dot com company back in the day that was cratered b/c someone changed a string literal in the code that went on hardware to be shipped but didn’t update the checksum the hardware validates. Every unit off the line was bricked.
To be fair: the company was running out of $ already, but had enough to produce the units. Also, clearly the company had horrible test and build processes, so it’s hardly that one devs fault. Still, most people would think changing a string literal would be innocuous...
I'm always excited to hear about the computer under someone's desk that can't be turned off because it's responsible for running some automated script that keeps the business running and isn't checked into source control
I've had former employer call me like "hey I know it's been a year but do you remember the server number that python app was running on"
My personal favorite as well. How about an Excel PowerQuery masterpiece built as a POC that ended up powering the Sr. Executive dashboards for a year. Btw — Took an entire IT team exactly one year to reprogram it with Informatica and Oracle.
My account at a company I left was active for more than 12 months because they never got around to creating a service account for our CAD license server when I was there and disabling me meant breaking it. They didn’t even change my password, meaning the server was never patched
My favorite is that computer, containing some secret keys and scripts, and is reachable over ssh but noone remembers where it is, and the computer’s owner has quit the company and then died.
Beyond cloud computing there is ghost computing
All of the explosives detectors that magically appeared on Army bases within a few days of 9/11 and didn't make a peep when you pointed them at a hand grenade.
Thought you would appreciate this thread.
We ran badging for the entire company off of the old laptop of the security guy when the company was a wee baby. We could not patch the laptop in a way that required downtime.
In the very early days of Twitter, a production facing service was deployed on a Mac Mini that just lived on the floor next a bunch of random things and cables. Everytime someone tripped over the cables, it would disconnect the Mac Mini and the service would go down. Fun times!
it was the ldap server wasn't it? but like the old Apple ODS version of ldap
Was it the google glass thing?
oh god. I think the ODS thing may have been mac mini too but apparently there was a mini that was used to tunnel access to prod network from the office
And we all miss the whale. :p
Have worked somewhere where we only found out that two clients' websites had been running from a PC in the corner when we threw it out.
Happened to us to. Something like 80 static pages with a bunch of Perl. No backups. I recreated the site using IE caches from PC’s that had visited the site. It took me a couple of days.
That was clever!
I think I installed that server, and I think it was an early openldap test machine moved to 795 Folsom. But I forget. That was 15 years ago. There was also a pile of servers running test scripts in the design department, high-jank. Or hey, maybe we did that a few times!
I remember the load-bearing Mac Mini. I’m also pretty sure you installed it :)
I think this was called “the load-bearing Mac Mini”
the best part was we couldn’t file a ticket about it because the jira or whatever was hosted on the mac mini so we had to just figure out where it was every time
heh i am sure nothing like that exists at twitter any more
I was asked to extend Layer 2 because a piece of legacy code called PowerBuilder (circa 1998; literally last millennium) didn't support routing of any kind After about 4 months of struggle with the PM they gave up and retired the code because I wasn't about to span L2
The flight computer(Collins aerospace FCC-730) in brand new 737’s first debuted in 1996. Part of the reason the 737 MAX software fix took so long is they have to be careful with the code not to overload the computer.
I worked for a microchip manufacturer last year whose servers were still running Windows XP.
I was not high ranking enough for details but my understanding is a hugely important piece of code signing for BlackBerry was a single machine.
Wrong, Adam Schiff evidence 😂
A bespoke (non-IT) employee-created Access database in runtime mode which was essential to the daily running of retail operations & created millions in annual revenue. Employee left, took the source code, still relying on it ten years later. It was exactly as bad as it sounds
ran fulfillment automation for 2,000+ daily orders through an ngrok server on my laptop for 8 months bc i couldnt figure out a "it works on my machine" bug
I give you not a company but the Web!
Word Perfect. In 2018.
A CMM machine from 1995 that's still in use today.
When I was a kid in high school (2010 or so) my mom worked at a small accounting firm that still used a main computer and dummy terminal set up. I love fallout
In memory render farm so scenes and their progress was lost if it crashed, required starting everyone's jobs over if that happened.
Posting this back into the prime thread:
Honestly though this isn’t even in the same ball park when compared to a project that my old company worked on … a legacy system rescue project for a FTSE 100 company… 🧵
does an elevator controlled by an Apple ][+ count?
Does the fact that basically all of the US healthcare system is run on EHRs that are using MUMPS count?
I mean, when the CIOs of health systems will send out email saying that you can't put anything in the EHR during the "extra hour" from daylight savings time, that feels pretty "janky"
Seriously!? 😳
To be fair, they do shut the systems down during that extra hour to ensure no changes are made during that time 😂
Thanks!! (I’ve worked mainly on scientific & engineering software — including my share of time issues 🙃 — but not for healthcare, and I’d never heard of this.) Very entertaining! link.medium.com/mpJltG352lb
What's MUMPS?
A coding language and database system from the mid-60s
When liking a tweet is combination of laughter and horror.
Lol I wrote the white paper related to handling DST for one of the big EHR providers. Janky? Absolutely. Still better than paper? Yup!
I worked somewhere recently that was working on integrating a MUMPS database into their system, because apparently it's commonly used for financial stuff in other countries. Somehow it's supposed to be the fastest database for that?!
That's a newer version of M (MUMPS) than the one most of the EHRs still use. I've heard it's fast, but I can't believe that something based on tech from the 60's is still the fastest for financial stuff. Sounds like a copout answer to avoid migrating billions of transactions
I never measured it personally. I just heard about it being the main database behind every large Asian bank because it was the one that could handle the most transactions per second. (they have a LOT more TPS than in the US from what I understood)
Epic does, Cerner does not.
Oh come on, many of them moved to the jvm hosted mumps, jumps. It's exactly as you think....
You think its just healthcare?
Last time I was at one hites their reservation system wasn't even GUI.
Came here to say MUMPS. Also, in terms of frontend, a lot of surgeons prefer to use Cerner and refuse to use Epic. So nurses and support staff have to toggle back and forth between the two often-incompatible monopoly EHR systems. (This happened at 3 hospital systems I worked for)
A combination of low-RAM local machines used to run long training jobs, combined with a clear-desk hot-desking policy. Would put laptop in a locker and hope the battery held out till the job completed. …And hoped the fire alarms worked.
Wooooow. Very Mission Impossible.
An Access VBA app for order tracking and invoicing on a shared folder that would break anytime more than one person tried to use it at the same time
I work in television, and the answer is the teleprompter. Easily. We scroll through every script before each show just to make sure there isn’t something janky with the copy. Even still, there’s always a chance of a malfunction. On air. For a live program.
Three pages of Perl I wrote
some debian based firewalls I built :)
off the shelf: radware load balancers
Did some work with life insurance company whose millions of variable policies, requring a daily value adjustment tied to equity and bond markets could not be managed well with the decades old software and hardware. Their solution was to hire battalions of people to do...
hand calculations to keep thumbs in the dike, with a tiny number of old timers who could keep the ancient crap from collapsing. One mid level person recognized that if she were to get hit by a bus, it would be a catastrophe for the company. It reached the point where it....
didn't matter who called her when she wasn't in a good mood; she'd pick up the phone,& snap "WHAT the F*CK do YOU want!!" I was only a tourist passing through, so it was all entertainment to me. Eventually, of course, they updated, and she quit with a pile before they canned her
An identity solution from some random vendor in a strip mall in Florida.
Pulse operating systems. Absolute garbage.
In the early 1990's I operated a Data General Nova computer from the 1970s for a publisher of catalogs. The president wrote the code and would not part with it, though at one point he started to port it over... to a Commodore 64. Paper tape, reel to reel tape, massive hard discs.
Microsoft Windows, prior to Windows 11 (I think Microsoft finally got it right).
There's enough COBOL legacy codes in various companies that COBOL programmers can earn 6 figures salaries.
We ran an Excel sheet “model” in production with Apache POI.
We "developed" what basically amounted to a cardboard box with an LED inside. You would set your phone on top and it made it "easier" to scan documents by taking a picture. All I can say is, at least it was many years ago. 🤷‍♂️
I know several race horse breeders that have all their data in an MS-DOS Ryan McFarland COBOL system. I know, because I reverse-engineered the data structures in 2004 after the original developer committed suicide.
Unfortunately, after I spent weeks reverse-engineering the entire system and building a data extractor to move the history to SQL, those breeders decided it was too much trouble, and I spent the next several years supporting that system in a nesting doll ...
... of Windows 7 (later 10), running VMs of Windows XP so we could get a CMD close enough to DOS for RM-COBOL to run. I'm not in that biz anymore, but I got an emergency call 3 months ago from a veterinarian in Texas who trashed his system and needed to restore the backup.
They still use it. EDMS - The Equine Data Management System. I'm one of probably 10 people on the planet that understand it well enough to recover a broken one, and large swaths of the top Thoroughbred race horse breeders would be totally screwed without it.
Wow. I knew about the hospitality and banking industries, but horse breeding IT teetering on the brink of collapse is a new one.
Two of these same farms had me install fiber optic lines between the office and the breeding shed to improve the multi-user network performance of the COBOL system. That might be the most high-tech/low-tech thing I've ever done.
This is ludicrous. Do they have no concern for what happens if suddenly NO ONE knows how—or wants—to support it?
Trust me, I’m not disagreeing with you. This situation is nuts, but it’s not even in the top 10 list of most ludicrous things I’ve seen in the horse breeding industry. That should be it’s own thread, but the industry is small. Even if I don’t name names, somebody would guess.
For that matter, I'm not arguing with you either. It sounds like you tried to help them migrate and they turned it down, so what can you do.
Rhymes with “mob maffert”?
Let’s not speculating, or I’ll need to delete this whole thread. But to quash any rumors, I’ve never worked for him and have no idea what software he runs. EDMS isn’t the only system available, it’s just one of the last available options that isn’t a subscription.
Dang, what a tail (tale)!
major e-commerce site making price changes for big sales (Labor Day, black Friday)...by me manually running a Jenkins job that in normal times failed ~weekly. If it failed I ran a SQL script in prod instead.
OS/2 Wrap that ran voicemail and fax machines for a legal firm back in 2010.
Got hired as the computer guy for a boutique hotel. Was shown the IBM tape backup machine used for all system data. Was shown a stack of backup tapes. Every one of them had been reused *countless* times. When I tested the archives not a single one was readable.
Laptop under someone’s desk that ran code written by an account manager
yahoo finance in a fintech startup
I use Yahoo Fin all the time as a retail and when am not at the TF
But the straddle view in vol was revolutionary!
Import yfinance “””Solve all the problems”””
more common than you’d think lol
A $1B fund on an algorithm I wrote on my blog six years ago. I kid. At least my code is readable--unlike the PWC consultants' drivel that wrote Deutsche Bank's CCAR analytics code.
you bite your tongue
Pretty tough to top.
i mean... i know investment banks that depend on yahoo finance
I work in a company where we're encouraged to do research from it if we need "public" data. It's a 1T+ asset manager.
entire pitchbook industry would grind to a halt without yahoo finance.
This puts into perspective just how much I oblige my product and RFP people's requests in the middle of a work day. I should tell them to look it up on fucking yahoo finance from now on.
MS Access at one of the world's largest banks
bloomberg anywhere isn’t any better once you adjust for@the difference in price
Id pay for another dependency but I don’t know they aren’t using yfin!
A login/authentication server to a very popular MMO went completely dark. That's when I discovered it was an i386 dell PC under someone's desk.
old school linksys wireless router for firewall at top 5 university.
A long-running rewards scheme advertised on national TV being supported by VBA macros in MS Word & Excel documents running on a desktop PC in the marketing department.
Jenkins job DSL & groovy scripts produced from a series of Go templates, which in turn were described using another internal YAML DSL. Groovy loaded via shell exec caused one of the worst String memory leaks I've seen in my career, on the order of 100s of gb
A few companies ago our DB setup was so messed up that we had a script we needed to regularly run to fix things locally called db_sadness.sh
We ran the front offices of a busy FBO with rental aircraft on an apple I, and the air ambulance co on pirated win95. Maintenance for both was a microfiche in the hangar. Until the day the new avionics guy called MS tech support... stern letter and we had to pirate a new OS.
A dev team of ~8 people sharing a single 64kbps connection to remote desktop (citrix) into a client site overseas in order to develop a product because we couldn't afford an oracle license to run locally.
The server inside the wall.
Whipping up a linux box to ARP-poison our local network and pretend to be the gateway in order to rate-limit the guys downstairs who were trying to download movies on our 128kbps feed while the rest of us were trying to work.
Outside of Ansible / Red Hat I've never seen a company package their Python code. but every single company depended on Python code for critical tasks like provisioning machines, backing up databases, verifying schema changes, etc.
They had an online form people could use to report product issues. The forms were sent to a printer and someone took the hard copies and typed the info into a database.
They should have used ocr.
The issue they were solving for was a complex 8-bit coding system for data. Even if we ignore updating their database, they could have replaced all that effort with some drop downs and a touch of code.
I really hope it was just running the coffee machine
No not the coffee machine, it was a bona fide secondary or tertiary control system (it was proudly describe in one of my .de computer mags in the 1980s) but not related to the rods or IIRC safety of the nuke plant.
Small printshop using a Silicon Graphics O2 workstation as a print server in 2010.
Those O2s had STYLE, though.
Developer (who had long since left the company) had code running on his personal box in the dev area that was part of the production site via reverse ssh. Now, you're assuming this was a small company. It Was Not.
I worked in a place that had a custom Java messaging framework. This was before Kafka or even RabbitMQ and the best option at the time sucked. So they built one. We would get stuck or missing messages, data was inconsistent between data centers, etc. Could have been worse though.
The folks who built it were very smart and they did what they could I think. Just a big problem to tackle with a small team who also had products and features to build.
Also, the best commercial option (if I've gotten the time period right) was tibco, licenses of which cost more than I made as an engineer
Lotus Notes. IBM ran everything on it.
Absolutely everything. I was there when they bought Lotus and golly that conversion was fun
Network of @Windows 3.1 machines running recursive Excel VBA macros
[REDACTED] 5th amendment claim against self incrimination.
One desktop running as headless server, with windows 2000, running beside the foot one random employee. This system runs at the time all part of the erp of an entire multinational steel company.
I once worked for a company that paid teenagers to rip capacitors off of our $120 servers with pliers to make them work. They did thousands of them
I worked at a health care company responsible for notifying millions of hospitalised patients of surgeries, prescription info, & emergencies. One engineer built the application complex enough where only he understood how to edit the main java files to guarantee his employment.
Air conditioning for the whole building relying on an ISA card in an ancient 486 PC running Windows 95.
Adobe Text Search Engine that had a lockout password that required a Board Member at both companies to call support to get the website back up.
A compac brand desktop PC from the late 1990s, still mounted on a shelf in a rack, which contained a hardware locked licensed piece of software required for a core business process, in 2017. Software vendor was out of business.
~2000 line SQL query maintained by a dedicated contractor
One or two of these cables were still active...
That’s a dream job right there.
Right, now imagine that spagetti is under every tile of a raised-floor office, covering 2 or 3 storeys. That was my 1st contract job, in 1992 in the City of London. A mixture of RS-232 lines to VAXes & terminals, UTP Ethernet, & coax or twinax for IBM minis & mainframes.
In the end, the company moved to a new office across the river & recabled from scratch. It took a year, but it was cheaper & easier than sorting out the underfloor pasta. I had actual nightmares about cables-as-snakes for years afterwards.
Microsoft Teams
Not sure it qualifies as tech but the thing that drives me insane is to watch medium to large asset managers use a single bbg terminal to access market data across the entire firm.
Omg. So before I get started… are you familiar with VG-LAN?
One major UK bank processes hundreds of millions worth of transactions on code which was hand crafted in IBM mainframe assembly with some code from 1960s.
A production system that used Google sheets as a logging service. The service translating log lines to spreadsheet rows ran on one of the devs private home server, because we couldn’t get the right server image deployed in prod
On the bright side Google Sheets is unlikely to download & run external code while you're writing logs to it.
Google’s campus has as many glass walls as every other tech company.
Inherited the corporate website for a 200mm post IPO company. They had a former employee on retainer who updated it, $2500 a month. He’d built some janky Rails CMS on his own and half of it didn’t even work. Fired him and rebuilt the whole site in Wordpress.
One name: jsonata
FoxPro 2.7 on DOS for an insurance company. Not as bad as the other examples in this thread but...it was janky to me in the 21st century.
Around 2012-2016, a HP desktop PC that ran Windows Server 2000. That “server” ran proprietary billing software for public customers that couldn’t be virtualized based on licensing 😬
Lotus notes.. Scary
Oh! Forgot about when ppl were still trying to release an alpha version of a homegrown C++ data analysis system... 17 years into the project, fully staffed. That ship was not resteerable. The tech was released as a beta at ~21 yrs and is now 30. Quality takes time! ;)
[credit where credit is due: it turned out OK, pretty much. should not have taken an entire human's childhood and adolescence, imo]
One of the largest <redacted> companies main product was over 300 servers with /servername NFS exported & mounted to all others. The app was a collection of bash & Perl. Made millions/yr. Was insanely painful to maintain.
Every server had every other servers NFS share mounted to it.
next summer camp over beers I'll have you tell you about the impact from NotPetya. Spoiler: not pretty.
True Root Cause.
Company which I won’t name built a version of Angular for website. Was horribly obtuse and hard to work with.
The org didn't depend on it by any means, but I once got paged that something random on an ancient/well-protected server had gone down. It was a (now well-known) researcher's web scraper perl script that was written c2001. It had a helper script to restart it as needed.
Best "If it ain't broke ..." - An IBM XT PC running key switches in a telephone exchange for a whole suburb. It was on 24/7 for 20 years before finally catching fire and burning down a chunk of the telephone exchange.
I laughed way to loud at this. 😂
Telco Systems AC-211. It was a full-featured telephony adapter with no cooling - they failed often, sometimes melting and on a few occasions literally catching on fire. And there was a model which was made for a small form factor that got too hot to touch.
Global financial markets will crumble without Excel spreadsheets.
The same goes for oil and gas supply chains.
And, tbh, most tech company supply chains.
Same for aerospace...
That’s awesome
Imagine Excel, but with the addition of VBA using COM-wrapped Java classes to send sealed objects over JMS to the trading platform.
Friends of friends as managers
At my old bank we priced & traded around $10bn of bonds a day, using an excel add-on we downloaded from the internet lol
See also: Healthcare
Coworkers told me of an auction system for short-term loans totaling 100s of millions of $, implemented in Excel 97. Office update wrecked the macros. Current staff didn’t know the manual process, had to frantically restore.
Somewhere up there is a guy who wrote the Excel Macros banking still depends on!
Running BPICS on an AS400 using "screen captures" from the terminal to update reports.
nice try, i have an NDA
You'll be back to tell us...
When my Grandfather build a new building for his Ford dealership in the late 50s, someone talked him into installing a pneumatic tube system for moving paperwork. It ran into the 90s. At least once a year a mechanic would send a live mouse to accounting.
Lotus Notes. In 2021. As a database.
a trigger warning before you speak that name, please
AS 400 with several used components from eBay since the entire system was out of support and spares were no longer available
An email server for a company with 5 offices around the US running on a computer wedged under a cubical desk with minimal airflow and no UPS.
I work on a legacy application that’s hosted on a Windows 2003 Server and it’s database is FoxPro.
Serial null-modem cable made out of paper clips. FWIW, a paper clip will support 115.2Kbps.
This... this is the one that broke my spirit.
Man, that's hacking in its most pragmatical, refined way. My most sincere respect!
this looks like something you'd do
Load bearing paperclip
I’m reminded of this abomination: « DB25 Killer Thanks to a major network manufacturer who ships every device with a serial to RJ-45 connector, we are able to make serial toast. DB-9's are laying around too. » The Etherkiller fiftythree.org/etherkiller/
Daley Center in Chicago, at least til the pandemic hit, used Carbon Paper to make copies of handwritten official court orders.
I'd say the fact that many major business systems (financial, travel, IRS) still run on COBOL and all of the programmers who know it are reaching retirement age is pretty risky.
Can I have an NT with that 🤣 But seriously, in 1999 we were hired to do an inter-BU EDW. VPMkt didn’t feel her dept needed it. Called in a marketeer who explained his “Lake of Data” —Access on every computer that he would sync nightly — worked
I've worked in a company where the entire payment flow was a single class with nearly 10k lines of codes. Debugging was hell everytime modifications where made due to extremely but painful regression flow. 😢, I did a lot of refactoring before my time was done
So there was this TSQL script being called by a Payment Recon job, it was like a textbook 😂 Views, sub procedures calls, functions, cursors, switches, try catches locks, db links.. file containing all the possible sql commands I have seen in a lifetime 😭 1/
Every time something goes wrong with reconciliation you literally would spend a lot of time trying to figure out where this a possibly a rounding, truncation, wrong joins etc happening. Sadly the second half of gymnast happens on the code side (story for an other time 😪)
decade out of date filemaker pro database
I wrote an "xml parser" in a day for an old mobile platform (BREW) to parse one file (a Winamp style UI skin). A week later I got bugs that it wouldn't parse some other file that came from a server. I fixed the bug and it became our "official" parser for that platform :'(
A mishmash of .NET, Erlang and Node.js/React installed on premise on customer machines. #IIoT
Every saas company has at least ONE rickety outdated npm package being maintained by some dude in Arkansas that is unknowingly holding their business together 😭
That is literally an XKCD joke.
Holy shit you’re right, I felt like that was coming from some recess of my memory 🤣
Could be. Article: March 27, 2016; comic August 17, 2020.
Use a bunch of if-else and regex matching and call it `AI` 😂
My very first network troubleshooting job the company complained of dropped connections. 4 pairs of desks each with a hub, wires up in the ceiling run over high voltage lights all back to a 5th hub to a cable modem/router combo. Textbook networking nightmare.
Running an early 90s SCO Unix app in RH6 (circa 2000) using Linux ABI/ibcs emulation, to perform the most critical revenue generating functions, & it was only phased out a few years ago. Ohh, and I'm sure there's a mid-90s ColdFusion beta instance running a govt website still.
COBOL on DEC, in 2005, for a national healthcare provider. They were verifying the listings (yes, the printer was old-school too) by hand to make sure all reimbursements were processed.
In the early days I was working on online store and in one of the perl scripts I left the debugger option on the bang line #!/usr/bin/perl -d. Each time when hit,the script waited for a debugger to attach in production in a way that hung the whole server,hosting 1000s of websites
Knew a company that still had a machine running Win NT 3.1 only a couple of years ago (and probably still do). Only reason they had it was to test the software they produced didn't break on old systems that their clients had (it was for old computerised machinery).
A large array of old HDDs in RAID (over 20 random sized drives), seated on a home-made pvc pipe rig, with jury rigged power connections, that the company owner used as the main file server. It had about a 2ft square footprint and was completely open.
Has to be this one soon-retired guy still doing maintenance and updates on the 68hc12 microcontroller software written in assembly, on a development setup involving an ISA expansion board connected to a machine no one was allowed to turn off - we were not sure if it would reboot
Or in the same company we also had one part of our hardware based on an obsolete SX48 microcontroller, they just had bought a few thousands of them from the final production batch and planned to use that for the next few years of production
I worked at a firm that relied on proprietary benchmarking software for all its client reports. You'd upload client data and it would churn out client vs peer group benchmarks. If anybody ever uploaded a zero the entire thing would break, for everyone.
My first job out of college had a daily script that ran on a windows machine. That machine would automatically connect via PCAnywhere to a remote machine (owned by a partner company), and copy some generated files from that machine to an FTP site that our code could access 🤯
The U.S. Copyright Office database of post-1978 copyright records.
When I worked at an ad agency one major airlines backend was down so often that one of our leads stubbed out a complete replacement for their API in ruby just so we could finish on time. Worked brilliant! Would recommend
our school ran a version of ASP .NET that had a critical known vulnerability since like 2014 that allowed you to gain admin access to their whole system. i proved it to them and i *think* they’ve updated it since then, not sure tho
Someone coded a "csv parser". It just read text line by line and split on commas. Someone coded a "csv export". It exported an actual excel file with .csv extension. You could open it in excel, but not with a text editor.
Napster’s first “search engine” was just a strcmp() against its entire index. Didn’t scale beyond a few hundred users. 🤣
An auto parts logistics company went with VAX, but decided to save money and not use VMS, but instead RSTS/E. They wrote all their code in BASIC PLUS. When that platform went obsolete, they had a problem. So naturally,
They commissioned a RSTS/E and BASIC PLUS emulator for Motorola UNIX running on 68k to future proof themselves. Then they had a dispute with the consultants around pay and lost access to the emulator code. So naturally,
When Motorola UNIX and 68k went obsolete, they wrote a 68k emulator to emulate the RSTS emulator on top of SCO UNIX on 486 boxes. Probably that is still responsible for running your local auto parts warehouse and someone is still writing BASIC PLUS.
Someone certainly could do that, but there are multiple PDP-11 emulators, including several FOSS ones which are free. pdp-11.org.ru/emulation.pl?l… I don't know when this happened, but the best-known, SimH, has been around since 1993: simh.trailing-edge.com
I am fairly sure – say 95% – that it's impossible to run RSTS/E on a VAX. The only way to run RSTS/E programs on VAX-VMS was via a 3rd party RSTS OS emulator, called ROSS/V from Evans, Griffiths & Hart.
RSTS/E - Wikipedia
(no description)
en.wikipedia.org
I think it was WSO2
HTML strings with inline styles stored in a database. Heard of SSR? This was DB-SR. e-commerce distribution company: client pricing was decided each year on a per sales manager/client basis. Managed in glorified Excel. Central data comms for whole business ran over telnet
Late ‘90’s was running test rigs and IRL game testing for Namco. Every single one of those machines was running off software written to EEPROM’s. I hadn’t seen that sort of thing since the late 80’s. Was shocked to see a modern video game being rendered by soldered together IC.
Worked with a co. who built power generators for satellites. They had a defect tracking system written in PHP1 running on an ancient HP "unit" that was a full sized cabinet. Upgrade to PHP7 running on a RPi with backup to S3. They were blown away by the performance improvement🤣
Love the stories 😂😂😂 bookmarking
Their website was running on a Win NT desktop on the floor of a middlemanager's office. An analog timer plug did a hard power cycle because Dr. Watson kept flooding the machine with popups. So the site was offline every evening 4PM-8AM. (This a respectable listed company in 1999)
That I'm allowed to speak about, robots at heavy industry based on DR-DOS. Floppies used to program the suffered a lot with plastic vapours and debris.
No joke. In 2010, a small white power strip on the windowsill in the server room over which some main switches run. When the thing accidentally went out, it was a party.
The shaft balancing machine that has a 1990's PC built into it. :)
pics or it didnt happen. =D
Large FTSE100 company relying on an access database to do all it's invoicing, that nobody knew how it worked...
Built an excel model to calculate wages with raw time data for a labour hire company.Included fairly convoluted overtime and bonus rules. Someone copy pasted it for IT contractors on day rates and paid out 10x in wages for 100+ people. One dude got a monthly paycheck of $113K
This incredible web frontend spun up a complete java environment per active user. Hit the memory ceiling after 10 users.
The biggest insurance company still runs off an MS-DOS system
The actuarial dept for the workplace div of a very large insurance company relied on an app running on a old desktop under the developer’s desk. He had a stack of 3 and always forgot which one was “production”.
jesus, the war stories from actuarial are atrocious. I worked on some actuarial data tools in python last year. my predecessor formatted several 400mb csvs with nested dicts and opened a new interpreter to generate each of its 200 columns... O(n^♾)
Mgmt must have loved always busy hard worker
I've seen some hairy situations where a dev desktop running some beta feature branch accidentally got added to the production VIP. Actually caused customer data loss one time. Major protections added after that to prevent test servers in prod VIPs.
A friend told me a similar story also from an insurance company in .hu: a dev pulled data from a huge MS SQL server and ran an overnight batch job on his PC. Took around 7 hours. Then my pal rewrote the queries, moved the workload to the server and squeezed it into 30 mins.
Code or process that is written like an inauspicious game of Jenga where the removal or failure of any single piece brings it all crashing down. 😂
Multiple logic statements in an if-then-else statement Chain enough together - say thirty or 40 - no way can you refactor that sob
Or calculated goto states in Fortran which go up and down 200 on a goto (100, 500, 200) Where a is variable If you're lucky the 100 label is before 200
Stop poking fun of AWS US-East
Cics transaction gateway running on OS/2 on an ancient physical server, discovered during a critical data center power down. No support, no replacement parts and a lot of breath holding as it was power cycled 😅
I’m breaking into a cold sweat just reading this.
I worked at a major bank who’s SIEM didn’t index IP addresses.
Excel ... it's always Excel
Uhhhh a 20 year old roland sticker printer. It was acquired used. It had a lot of trouble color matching orange and purple and threatened to break down completely every time there was a big job. One time we had a tech in from Roland and he'd nevrr even seen the model before.
I actually compiled a list of all of the janky tech I’ve used professionally.
Former employer: student system running off a Mac from the 1980s.
Lotus Notes network run on two servers in different buildings using modems on internal phone extensions. Corporate IT wouldn't let them replicate on the pipe between the buildings... 🤷
Have you heard of access, good sir?
So I’ve got a couple. Worked for a financial org during a massive security overhaul, consolidating security tools wherever we can. We get a report that a business unit cannot run their model to asses whether or not we should be insuring someone
The person they put us in contact with paints it as if the sky is falling and that this is a extremely complex analytical model and that they’re currently unable to process any policies because of it. Finally get hands on with the “app”.
Find out that it’s an extremely complex macro that’s been carried over the last 15-20 years with the person who created it long since retired. I’m talking thousands of lines of “code” that we had to figure out what part of it was being stopped by our new endpoint security tool
Guess what funding was nixed year after year because it was just working? Credit to the person trying to “support” this beast, he barely knew how to even execute the damn thing. Okay, 2nd story!
You know where folks are crowing about that they’re going zero waste and all that jazz? Well a lot of those orgs utilize what are called “waste to energy” facilities, basically a fancy phrase for incinerators.
Now what most people don’t know are that a lot of those facilities are 20+ years old because who wants one of those in your backyard? Now imagine all the things you’ve only read about in tech history books, running the things within these facilities.
And all the technical debt, bubble gum, and tape to go along with it, Token ring networks, Windows XP, random wires running into walls into old switch hardware. Now, I don’t know how many of you can imagine this, but imagine the smell of garbage turned up to 11
And having to hunt down one of these broken switches because of one of two things. The system that monitors the emissions can’t operate or the systems that operate the 20+ year old plc can’t communicate with said PLC. All running extremely legacy tech.
Both of these things caused or had the potential to cause a facility shut down because of thousands of dollars in fines for not monitoring what you’re putting into the air or not being able to control the burning of garbage happening in the boiler. Both extremely important things
So yea, legacy tech spread all around makes the money machine go brrrrrr while running some extremely important systems that very large facilities or financial institutions rely on!
I also realize that many people deal with this in their day to day jobs still, you folks are the real unsung heroes!!
I worked at a company whose main product was screen scraping a client’s desktop site to build a mobile app dynamically.
Too many to mention in large enterprise IT in the communications and software sectors. Software-wise using Excel as a database or an out-dated version of Microsoft Access as they wouldn't pay for a dev to put it on a more suitable platform.
Our desktop IT people used a Windows bat file to connect PCs to printers rather than let people sue the Windows printer interface.
A private university. Refused to listen to the videographer about media storage recommendations for their growing video library. 3 years later they are still using an assortment of disconnected hard drives
Now, they are hiring a new person to organize the hard drives (The videographer left 2 years ago, and the uni is flush with cash)
I worked at a company that kept reusing code between clients, well eventually they ran into a problem. the maximum number of joins in a sql statement was exceeded. 255. Thats right production code with a 255 tables joined. They were using views heavily.
I worked for a company that automated legal filings. The codebase was 4.5 GB of PHP. No framework. It was that big because each form for each state was copied to make the next one. The original source material was Sam's Tech Yourself PHP in 24 hours. The year was 2010.
Reading this makes my eyeballs hurt.
Probably why I have glasses now.
Reading this make me realize things aren't so bad where I work
Software made in 1995, could not run the installer any more (I had to manually update the Windows registry). It only supported 5 connections to the server, and records locked when editing, causing issues when reading and writing at the same time.
My job was to keep this thing working in 2015, and migrate it to Windows 10. The worst part of this thing was that we had a couple remote employees (using RDP to connect to the local network, the server couldn’t be put on the web directly)…
Occasionally they would walk away while still in edit mode, causing the entire office to lock up. Sometimes the server would soft freeze itself too, no interaction required but would have to be rebooted.
A halon button that had no cover. And, someone set it off by leaning on the wall. There was no fire.
So, I worked somewhere, that the CEO and founder insisted to write code, about 7 years later after they migrated away from the system he developed, they printed the source code which had 4000 lines of switch cases.
Thats high performance oriented design right?
High performance, high maintenance. That'd how they rolled.
he brought this meme to life
Now, mix for loops and goto statements for completeness
A homegrown CRM from 1998 that was still being used in the 2010s and only one developer knew how to maintain. It froze and crashed so frequently that there was an app on everyone's desktop that was basically a kill command for it.
A system full of over 4000 stored procs, some of which were over 10k lines long with 1k line where clauses performing operations on non-normalized tables with 100+ columns. Some of those cols were incorrectly named. Like a date column would contain an id with a foreign key.
Code hell… Satan awaits you somewhere between those SProcs, proceed with caution ⚠️ 😂🤣
Lol 🤣🤣 as soon as they made it clear that they weren't interested in improving the codebase, I proceeded to leave.
1. SQLJ 2. Murex 3. Homegrown Excel addins with state
first magazine I ever worked for rebooted itself after failing, moved to new office, had no internet but had to close first print issue night, do or die -- we stole open wifi from office next door and connected our LAN to it; intern had to stay up all night to keep it running
worked at one of the nation's biggest banks who's mainframe relies on a COBOL programmed core that only 2 people know about. Same company (EU) has outsourced 60%+ of tech to India
We had a distributed job queue. A grad student wrote some reporting software for the system. To avoid divide by zero issues, he put a job in the queue with requirements that always evaluated to false so it would never run and there was always at least one job in the queue.
Using regexp to parse ton of json from rest APIs as json decoder was not available.
A glorified phonebook system for marketing. Core functionality was making searches with multiple parameters. Problem: searching was slow. Solution: cache results. As dynamically created tables in the db. Result: production db with over 70K tables in it.
The kicker: since the underlying data could change at a moment's notice, the cached results would be often invalid. Solution: run the search on every pageview anyway, recreating the "cache" tables over and over.
I'm afraid to look. 🙈
Excel for complex $$MM project management, including budgets.
A significant portion of US Government satellites have had their course design trade offs worked out on a set of linked spreadsheets running across ~20 workstations.
In 2012 the only way to book train tickets online in China was through the government’s website. It goes down for *daily* scheduled maintenance every night for *hours.* I wonder if it’s humans taking my orders.
Broken laptop that required an external keyboard and mouse (or RDP) that was used for controlling door key software and "hidden" on top of the server rack.
Ill take resp and admit i was the agent of chaos occasionally. My first "real" commercial work didnt work at deploy at client site but did at my machine bc i had crosswired username and password fields, but all tests had them as "test:test" so theyd pass.
A dot matrix line printer connected to a web server in the office, receiving POSTed payloads from a web app off-site, in order to print triplicate contractual agreements onto special preprinted carbon paper.
This might not sound that bad but it was like 2013, and the payloads were generated with Perl's ancient "format" magic.
Oh god I keep remembering the horror, the "web server" was a repurposed desktop sat next to the reception desk because it needed a parallel port to drive the printer. In 2013.
Gold veins everywhere, omg.
Friend used to work in a legal aid firm, back when data stored on tape drives. Tape backed up last thing on a Friday, and operation controlled via keyboard. The t key tensioned the tape, r key rewound it, and e key erased (without confirmation). Check the key proximity.
A consultancy whose whole business depended on analyzing telephone Co traffic data. They did all sorts of consulting but that was the foot-in-the-door. And it all ran on a Vax or some very old mainframe. And one guy understood the program. And he knew he had them over a barrel
filemaker in 2021
Once upon a time a major derivatives index, which was published daily by one of the major finance data providers, came from a bank's trading division - where it was calculated on an old developer workstation, running an unpatched version of Windows, that sat under a trader's desk
Crazy that the subway *still* runs on these
Its all well and good until you have to interview the creator of a whole ERP system who is now in hospice so you can understand it enough to export it.
A server under a desk that, once disconnected, showed to host the actual active AD .
Network ablation study
Pop the cover off any ATM that’s been in service for a while and behold the beautiful Windows NT server. Popular in 1996!
Financial institutions had to use a WORM drive. Early tech was built on writeable CDs in a cartridge. When you requested a file from the filesystem, a picker arm would pull the cartridge, load it and read the file. In 2008, frequently had to have someone jiggle it when it stuck.
In the late 2000s, there was a place that was using a machine running Windows 97 because of a legacy database in Access 97, which was only slightly more stable than Charles Manson.
Um. It can't have been, because there never *was* a Windows 97. It went 95 → 95A → 95 OSR2 (AKA 95B) → 95 OSR 2.5 (95C) → 98 → 98SE → ME.
Maybe the OS was Windows 98 then? I was told not to touch that computer and the back story. I do know that the database was eventually migrated to something else.
Well, yes, fair, but it's 90 less. 😉
Well, the Polish Social Security Institution was still dependant on 3.5" floppy discs a decade ago. And apparently, Boeing 737's required floppy discs for updates as late as 2020, perhaps still do.
I worked at a graphics company that had two servers, both of which had login and password set to "ADMIN" and "ADMIN" for decades leading up to my working there. One server had all the business-side software on it, for the accountants, etc. The other was for all the work files.
When I pointed out the vulnerability, my boss asked why anyone would want to hack little old THEM. I don't know. Financial information like routing numbers and stuff from clients? I offered. They were unconvinced.
Also, the work server had two partitions: one for active work in the last year and the other an archive of all the work that had been done since the company went digital. Often we'd pull files from the archive because people would hire us to work on them again for new projects.
Like...every day this happened. However, every account in the building had both read and write access to 100% of both servers. Literally, the entire archive server could be deleted with a couple keystrokes on accident. I think the passwords and permissions are still this way!
(By "a couple' I mean "Ctrl-A, Delete.")
OH! And there was also a thing where someone had come in to salvage files after a ransomware attack on the server a couple years before I got there! In turn, EVERY FILE before that date had a non-corrupt duplicate on the server, sitting right beside it. Inconvenient.
Hey you want to got fix I recommend you to contact cybertech121 on IG his legit and fast
Worked for a newspaper where the terminals we used to access the CMS were so old they caught fire periodically during the workday, often enough that we were familiar with the smell of “burning computer.”
Updated jpeg assets for a big hotel chain & file locations were kept in an Excel workbook w/ about 10k records over 30 sheets. It took longer to find each record than I est. to do the retouch so I had my Excel wiz friend write a program to find the records. Use a GD database.
Node and react
An entire floor to ceiling wall of 480v mechanical relays circa 1960s. That was in 2017.
A cluster of two MySQL servers sympathetically replicated across Atlantic
I saw a Nokia handset jimmied up to a server in a rack in a data centre. It sent SMS messages to the owner if the server was knocked off the network.
Pretty genius MacGyver level monitoring
School district running windows xp because they havnt upgraded their computers since core duo era i want to say compaqs at that. they want to upgrade them to windows 10 when new administrators came on board. but i also saw gateways still in the wild running hvac systems.
I saw a credit card processing company operating on an old 3174 mainframe controller that was running a cooling fan extracted from another old, canibalized unit, because they hadn’t been produced for years
I think of two particular examples 1) an Access DB solution that turned out not to be 1998-compliant and 2) a rogue ‘open tran’ that turned a 20-person customer services order system single-user.
I developed logic to control the upper set point on a 300 mw wind farm, and the consultant implemented it as a series of SQL queries over a SOAP API. Drove our CTO crazy, but they still haven't been able to replace it.
When I was 17 (around 1999) I wrote a small tool in VB6 that connected to the company sales db and generated various report emails (like sales per team per month etc.). It still generates these reports to this day, and presumably business decisions are made upon these mails...
I wrote something which would screenscrape a DOS application and then convert the output to HTML. For every screen in a system with tens of thousands of records in a flat file database. But hey the performance of the web app was insane.
Wrote an unauthenticated Valentine messaging app; worried about profanity, used by insurgents.
Years ago, a major telco had a customer network that was completely dependent on a routing table algo that ran on a single SPARC pizza box sitting in an empty cubicle, with the code compiled into the SunOS kernel so it could get system priority. No one had the source, of course.
Oh for sure Adobe ColdFusion. But not even that, we used a different open sourced version of CFML that’s popular with ColdFusion devs called Lucee. Keep in mind this is circa like 2019. I’ll give you one guess who got it wrapped in Docker and working on OpenShift.
What's janky about any of that? @lucee_server is a modern JVM scripting lang and I've been running it on Docker Swarm since circa 2017. There's been official Lucee docker images out since 2015.
The docs were bad at best, there wasn’t an active community that I could find at the time, and there was no information anywhere on how to get it containerized (much less working on K8s) when I started that project in 2018-ish.
Plus I didn’t think an open sourced ColdFusion alternative had any business powering a dashboard to help federal judges across the country get oriented for the day. But it was the language my boss at the time wrote the proof of concept in so we kept moving forward with it.
I will say that DataSources were fun, though. They were like a little proto-GraphQL.
That was retweeted way out of context, @bdw429s. Has the community improved at all? I really felt like my options at the time were to dig through a lot of stale forum posts or struggle through finding my own solutions. And docs were seemingly directly lifted from Adobe.
I hit the RT button on accident, sry. I've been in the CF community since 2000 & I think it's always been great. If you asked a CF question anywhere online, I've prob helped you. The entire lang was lifted from Adobe, so Adobe's docs all apply obviously. No use recreating it.
Hmm, there's the dev.lucee.org Discourse forum which is very active, a 4k person Slack team, facebook groups, and the Docker images are right there on Docker Hub! hub.docker.com/r/ortussolutio… You're also the first person who I've heard dog a lang for BEING open source, lol.
Lucee Dev
Lucee Dev Forum https://dev.lucee.org/
dev.lucee.org
Good that this stuff is growing. At the time it felt like I was on an island. And normally I’m all about open source but federal government has some weird requirements and I was surprised we went with OSS for something deemed mission critical.
I'm honestly surprised to hear that. I've been on the Lucee mailing lists, fb groups, conferences, and forums since day 1 and no one ever goes un-helped. Perhaps you just didn't get plugged in. The CFML Slack team is huge and started in 2015. I'm in the #lucee channel daily.
What lang would you expect them to use? Java, Ruby, .NET, Python, Golang, Node, PHP-- they're ALL open source, lol. Really, it's a matter of support, not the licensing.
Yeah, I guess that’s more in line with what was going on. We tended not to touch anything if we couldn’t get a support contract. Even for parts of small side projects. Did Lucee have a commercial support contract available at the time?
Yes, Lucee has had support available since day 1, but it comes from the LAS member companies like Rasia. My company (Ortus) provides paid Lucee support too. For Ex, I've fixed bugs in the engine via pull request on behalf of customers. Here's one: wwvv.codersrevolution.com/blog/improving…
Gotcha. Unsure if that would have satisfied our requirements. Like I said, fed gov gets wonky when it comes to support contracts. I’ve had perfect tools for projects turned down because there weren’t first party support options.
I understand that. We have DOD contracts and it takes a LONG time to get new software. Nasa/JPL uses Lucee (the Mars Rover site, for ex), and my Lucee-powered CLI tool, CommandBox is on the DADMS and used to deploy off-grid classified apps. It's a very mature technology.
Could be it. I also just didn’t want to be working with that technology at the time. I was just getting into web dev and learning React. Having my day job be an offshoot of an Adobe created language felt like a step backwards when all my free time stuff was latest and greatest.
Heh, I see Lucee as a step FORWARD in the lang Adobe created. It's faster, smaller, modular, FOSS, & has unique features like extensions, caching abstractions. Heck, I use Lucee for hardware automation & RPi servers. pi.bradwood.com It's like ACF, but BETTER 🤓
Glad it works for you! I had a bad taste in my mouth from DreamWeaver in high school and would rather have been exploring Node or RoR at the time. I was also only a few years out of college and wanted to be working with technologies that I knew folks were hiring for.
They tryna migrate off of CF at work right now. Rough stuff.
Depending on what you’re doing I might have a repo for you. At one point I added Vue to the mix to slowly start taking over bits and pieces with custom JS components, rather than one monolithic overhaul.
Could that be done with Angular?
Pretty much every AWS service is launched by a chunk of black box Perl. Requires a specific older version of Perl. Ask most engineers there what it does, or whether its safe to get rid of it, and you get hushed replies about the deep and important magic this package provides
Once I got annoyed enough to go poke under the hood, and it turns out the only relevant thing this package does is auto-relaunch the service when it crashes. The other thousand or so lines of perl all concern setup for Oracle databases that haven't been used in at least a decade
One of the Apollo 11 astronauts accidentally broke the circuit breaker on eagle that controlled the ascent engine. Essentially would’ve been stuck there. Saved it with a pen. Don’t know why this is never spoken about.. eetimes.com/how-a-felt-tip…
Earlier company had a critical application used to pay commission to traders which was acquired from different company without entire source code. Binaries running fine on Windows XP when I last saw :)
Automotive IoT startup. The entire backend server was a single 9,000-line PHP script running on the command line. The founder just liked PHP.
The electronic medical record—more time spent charting than caring for patients.
us-east-1
When I worked there the canonical New Yorker Magazine article archive was stored in a FileMaker Pro db, I want to say version 8. That meant zero support for unicode encoding, all those diereses (not umlauts!) needed entity encoding. I’d be surprised if this has changed.
Factory needed a way to read production data from PLCs for parts produced, MTTF, etc. Data collection application was an Excel spreadsheet that only ran in Excel 2003 because it accessed some COM library. All it did was output charts via HTML tables. This was in prod until 2018.
A large intl corp had a demo room for their future product, and because of VPN/firewall, parts of it connected to their inner servers through OUR 8yr old workstation that we used to try out stuff. They would call us the day before big presentations so we didn't crash the computer
At a Fortune 100 some twenty years ago. Instead of making UIs for the CRM people used MS Access with the DBO login to modify the tables directly ... and that table was used for *all* apps. Someone accidentally locked it, went to lunch, and brought down the public website
An Access 95 database to handle all of the billing for the company that wouldn’t be upgraded beyond Access 97 that required the users to still have Windows XP workstations in 2018. They may still be using it, but I left the company.
I've personally installed & run Access 97 💯% fine on Win 7. I routinely write in Word 97, running under WINE on 64-bit Ubuntu. It does all I need & more, & it's lightning-fast on a 21st century laptop. Both work perfectly including installing all Office 97 service releases.
& of course Microsoft offered Windows XP Mode as a free download for Win 7 (Pro & above). Run 16- & 32-bit apps, on real XP, integrated into the Win7 desktop, completely legal & free. theregister.com/2009/05/01/win… Could be coaxed onto Win8.x but doesn't natively work on W10.
Windows 7’s XP Mode — Virtually worth the effort
What’s real and what’s not?
theregister.com
My first "real job" was a startup during the first dotcom boom. Before we got data center space, the servers were in the office. Storage was a consumer RAID device that used to overheat, so we kept a desk fan pointed at it. If the fan moved, the site would go down.
A client of mine has a Unix program that's so old they need to run it on a VM. It's their main ticketing system, and the company that made it is still around but believes that running a 25 year old Unix command line program in 2021 is A-OK.
I had assembly code tutorials on an LSI-11 back in 1983 😁 missed my calling?
I've written PDP-11 assembly more recently than that, but not by much.
This makes me think that most RPA is just tech debt waiting to happen.
Major well-regarded tech co, production database with client rollups, 'isDeleted' field. All good so far... isDeleted type CHAR. isDeleted values "yes"/"no". This is the story of how I learned not to just use table.isdeleted in a where clause...
Our data center ceiling had two tarps that fed into buckets to contain the leaking roof (right over the UPS). The entire records system was on an antiquated version of PeopleSoft and they weren't sure exactly which machines it ran on.
For a number of days one summer many years ago, a local government's entire IT operation relied on: 1) Two large desk fans 2) A lack of rain
Unnamed company had relational data. But it used MongoDB. Uh-oh, broken data because no txns. Found a fork of MongoDB with txns. Maintainers got bought; project scrapped. We forked it before it disappeared. Took 1-3 more years before we could migrate to PG.
At Bing years ago we had a service that ran 100s of classifiers on every query. There was 1 editable config file for everything. Once another team disabled our classifier when they accidentally deleted the last line of our classifier’s configuration while editing their config.
The most frightening part of this story is that I’ve been the person that makes that tragic edit.
Windows 2000 email server... this was three months ago.
Worked with a BIG software company (revenue then was ~$20Bnyear) where all product activation was via s/w a licensing key. We needed to make update, discovered system was written by one contractor, who was off on holidays, and no one else knew how it worked!
A pre-Pentium Dell used to configure the internal telephone system's call routing to desk phones. It used a phone jack and didn't have wifi.
Last year I took a job as HelpDesk Manager and nothing was modern, they were doing things like it was 1999. Worst of all was IT data drive was on a “linux server” which turned out to be a 9 year old Desktop with unpatched Ubuntu 14 and an HDD hot glued to the drive tray.
I used to convert 8 in floppies to 1650 reels so the data could be printed to microfiche via a 20+ year old COM machine. This was in the 90s but even then each of these media were obsolete.
We identified a huge leak in the systems of a major US broker. Leaking live client orders + account info for all clients out to *any* API user. Had been going on for 10+ years. We helped them identify it, they fixed it a couple of weeks later and never told client base.
An excel spreadsheet for contracts greater than $10MM. No macros, just spread the formula across dozens of cells using concat(). Each concat was at least 5 more cells, each with their own concats. Usually this went down 3 levels before you got to a number.
raspberry pi feeding patient vitals into a cloud platform during surgery for the one person that needs to keep you alive
i can do better, i once saw a raspberry pi hooked to a vitals monitor to send data to some cloud platform for anesthesiologists during surgery
I once worked into an IT Security company where the CRM/Accountancy/Invoices system was a Visual FoxPro 98 behemoth running in Windows Server 2003 SBS with SQL Server 2005. Running in a janky server that barely booted, when it did. So it was scary to turn off.
What was magic of this server is that the spinning disks actually made sound, you almost could hear the read/write arm scratching the platters. Anyways, the virtual machine running the WS2003SBS finally died of a VirtualBox VMDs inconsistency between snapshots.
So it was time to decommission the server. I rebuilt the machine in a Windows Server 2016 vm and turns out SQL Server 2005 is just too old. I had to do some hex magic over DLLs to make it run, replacing files just at the right time, starting the service many times.
The company was 4 days without the system, and I happened to have turned in my quit notice 2 weeks earlier, as I had accepted a job at another company. I managed to fix the whole thing on my last day on the job. The system was passed into production on 4 p.m. EST May 31st, 2017.
One hour later, I said goodbye to my workmates. That system exactly as it is, is still running alive today and kicking ass. I got lucky.
I wrote a service that rendered svgs to pdf by embedding the svg into an html page, opening it up in headless Chrome, and taking a screenshot. I would 100% do it again rather than learn anything about pdfs.
We had a MUMPS-based system running on a disused SUN server that was essentially the backbone of a ~400 person data analysis company. And then I started building (MUMPS!) NLP applications on it... I think it eventually caught on fire. Ahh, the dot-com boom.
I appreciate the drama of the “hardware caught fire” subgenre here.
Years ago I worked at a company that had an expensive computer, on loan for rendering. The computer overheated a lot. So it was put it next to the bathrooms, a portable air conditioner next to it, and the water outflow to a bucket next to this $1,000,000 computer.
I am a bit surprised the Windows-XP-64 ~2010 era precision workstation I cobbled as a VM host at work is still running...combined 2 surplus machines into one, dual CPU sockets so transplanting one I had no thermal paste available. Hasn't burnt up yet, on 24x7...
And has 16GB RAM (combined from 8GB per box), RAID of scavenged workstation 250GB drives (even took out floppy and CD-ROM to plug in more). For a while it had too much storage capacity to be backed up too...yay development "servers"?
Its got dual Xeon 5160 3.00GHz if anyone was that curious. 4 whole cores! And no hyperthreading. Yes, a VM host. No, not XP...its running RHEL.
Crowdsourced from around @ZusHealthHQ: "The parts management system at Lockheed Martin in 2006 was still a IBM System V mainframe. First (and last) exposure to PL/1 and JCL" @bryanjknight
"Large e-commerce store had a homebrewed Jira/Salesforce clone. They ran everything through it: sprint planning, customer service, etc Oh, and also HR tickets. And the database was readable by any of the 1000 or so engineers and PMs who had database read access"
"Large EHR company with homegrown everything. RATmail & RATtrax. It looked like a game of Minesweeper from the 90s, but less functionality. Except for the part where you could read anyone in the company's email"
"Another large EHR company’s expense system In the last few years was something similar. Garbage Oracle app that gave the impression of a student project"
"The whole application for Student Registration written with Oracle Pl/SQL web package (never knew that existed) at an Ivy League college. The scariest code I've ever seen"
Student registration and housing is a magnet for jank. I've seen this at multiple schools.
Basically all the crap I used in the military. Amazing when you got replacements... That were 8 years old
In a large public institution, I wrote a backup script for their payroll system in 2007. It was running on a Solaris 2.3 workstation from '93 with 4 2 GB SCSI disks, lying on a small table and under a thick coat of dust in the HR office... They had more than 500 employees.
10 years after I left the company , a former colleague phoned to tell me that they were moving building and no-one knew how to switch off the ICL DRS6000. I told him, got the course notes from the attic and mailed them to him with deepest sympathy.
The building they were leaving was a former factory. The Big Tension cables from the road were insulated in tar. Because of the power demand it was hot and liquid, had to be topped up every a.m by the caretaker. If he ever left the whole place would have gone up.
What with old Unix machines and crumbling infrastructure I was glad to be out of there. [pauses to survey home office] Anyway....
As an integration partner, I had admin level access to a MSSQL db of hidden persons in my country. I did not even need that data nor did I want it. Because the DBA did not care enough to create a restricted account for me.
If you submit a medical claim to the Cigna website, they fax your uploaded document to another office. I only know this because they messed up one of my claims and they mailed me a copy of the fax.
interactive voice response system would screen scrape customer data from a 3270 terminal emulator session. It had all the reliability and low latency/response time you would expect it to have. And in those days, it was SNA over token ring natively on Cisco gear.
Accessibility overlays.
I worked for a company that is actively using MS-DOS. 🤦🏽‍♂️🤦🏽‍♂️
A friend worked for a well known product safety testing and standards non-profit. They hosted their code on a networked mac mini and used Time Machine for version control
A reality company using Windows XP and IE as a browser for secure banking transactions. And using a KVM switch to switch desktops to login to each bank scanner. What a nightmare to migrate all of that to VM for them
If you make two subfolders (called bins) in Avid Media Composer (used for almost all movies and tv shows) with the same name, you are forbidden to open both at the same time. Even if you rename one, because it still secretly has the same original name.
Microsoft Teams. Microsoft Dynamics 365. Microsoft Outlook. Microsoft SharePoint. Oracle. Adobe. Hmm, I’m detecting a theme here…
An entire SMTP MTA written in Perl that offered so much custom functionality to customers (that depended on Perl) that it was impossible to replace. Same place also used Perl based IP failover.
Once had a client that used the server room as a "smoke-break" room. And yes, the servers did crash eventually and it was disgusting.
A company was shocked when I trained them in using CMS I've built for them. Like you edit stuff on the same webpage? Turned out previous solution was using plain @phpmya for content and data management and somebody taught non-tech folks how to use it
I was called in to do IR for a business in a unique industry. They had a compromise but needed special attention to analyzing ... an Internet-accessible iMac running macOS 9 that had a critical business app. I nope'd right out and handed off to a team mate
The server closet at work. It's just... why are there so MANY?
All that needs is a dust over maybe in 2030. Nothing wrong with that.
Apart from that abomination of a switch. Ugh.
We had to replace a switch a couple years ago, it was brutal.
A manufacturing line using a crockpot to keep the label glue at right temperature before application.
My own small business relies entirely on a complicated Google sheet built by a guy named Dave for $100. I don't know how any of it works. Dave didn't want me to tag him, but hey Dave! 👋
Knew of retailer who up until 2 years ago still had a Novell 3 server handling their POS data collection from stores over dial-up
I once worked for a company that had a single database that housed all of the companies PCI data that gave out "Keys to the kingdom" the password was 5 characters long. It was open to the world. The company dealt with school students data all over the country.
Virtually everything at certain-large-game-company with regard to BI (at least when I was there) relied entirely on a cobbled together mess of python scripts dumping things into AWS from peoples' desktops at random places around the company.
This. Yep, that is the CPU fan hanging off. It still worked. Ran for another 5 years before I got to kill it.
Ten years ago I still occasionally had to BRUSys PDPs for a household name. There were still law firms running on bugged Pentium Pro machines on SCO Unix and a local authority on an ICL 3900
I love this photo, I want to marry it.
Not my story, but former roomie once unplugged an old Pentium3 that ran a version of openbsd that hasn’t seen updates in years, and nobody knew what it did. It took the whole building, housing several smaller companies, down. Turned out it was the firewall in front of the modem.
A scary few years ago / a bank in England still running IE6 internally
200 car diesel electric ferrys with Windows 98 PCs for the alarm and monitoring system
There was this time when everyone used AWS…. What could go wrong?
A box-cutting and folding machine controlled (in 2016) by an embedded 486 ISA mainboard with 4 4MB 30-pin SIMM modules, booted and operated entirely by a program on a DOS 3.5” floppy (Heaven forbid that disk goes bad). Why was I called in? In far older AMI BIOS font than this:
At my old company. They do some very important cleanroom & pharmacy validation and emcironmental monitoring testing, in accordance with a lot of laws. The offic8ally have over 500 employees nationwide (they run a skeleton crew, it should really be 750+), and their big database...
... that everyone used for everything... is Microsoft Access. In case you're unfamiliar, access is basically a glorified Excel spreadsheet. They're headquartered less than an hour from Silicon Valley, too. The pain is real.
Investment bank, in Hong Kong, running a multi-million pound *intra* day P&L off a s/sheet that took 6 hours to run. Not to mention the MS-Access programs built by novices in the 1990s hanging off mainframe feeds, which then became mission-critical.
The glory days or the gory days ?
A well known bank with its largest fund £10bn running on an access database.
ah good times. just needs some cobol.
I manually de-dupped large merged mailing lists with excel sorts & my eyeballs for a political campaign on early 2000s
1990s: the payment card processing for a major airport car parking firm in the UK was running on an RS6000 in a disused portakabin toilet.
Not my story but a heard of a medium size animation studio that had an important part of their render pipeline that was not designed for headless rendering. So they had one poor little SGI box that did nothing except every other machine in the farm would set X11 DISPLAY to it.
I think I remember part of the Oracle Application Server installer requiring something similar... we needed to repeatedly reinstall it on headless rack boxes for reasons and bodged together something similar.
In the era before video recording but after video broadcasting, NBC would time delay the daily news for the west coast by optically filming a TV screen, then doing a live broadcast of the projection screen.
Wow so they had like a 3 hour window to develop and print the whole hour worth of film? that must have been stressful.
I wonder if they could’ve used X virtual framebuffer (Xvfb) instead? Probably not if they depended on hardware accelerated graphics. en.wikipedia.org/wiki/Xvfb Xvfb - Wikipedia
Xvfb - Wikipedia
(no description)
en.wikipedia.org
There was probably a hundred better solutions. It didn’t even open a window, just tried to create one as part of their initialization code. But it worked, at least until a sysadmin noticed an “idle” machine…
I used to have to remote control a desktop over Zoom to ssh into an on-prem server (the on-prem eng had to log me into several time-out-ing terminal windows w/ his creds because I didn't have ssh creds and he wasn't going to sit there and watch me work) and code in vim.
What's wrong with coding in vim ?
Not a goddamn thing.
There is a bowling alley near my house that still runs their small business off of an Apple IIe with a dot matrix printer. No joke. When I asked about it, the owner looked at me, said "don't fix what ain't broke" and then then back to spraying shoes.
They clearly need a z15.
"Dad, what is a bowling alley?"
My hat tips to this IIe and its dedicated user.
IBM SPSS CADS. Just the most haunted piece of crap, poorly conceived, poorly supported, mysterious failures. Terrible.
I once wrote a terminal emulator to do automated data entry from a spreadsheet source. It was used for years after I left even though not a single person would have been able to make even a minimal change had the green-screen terminal UI changed a tiny bit.
Mail order pharmacy that I worked for in 2000 had an old vax/vms server that everything ran on. tape drive and everything. All of our computers were dumb terminals, the old orange or green screened text only ones. as the terminals broke down and became expensive to replace
they ended up buying windows computers, and running software on them to emulate the old terminals.
Had a questionable IT guy who decided the best way to set up a email system for the company in the mid/late-2000’s was to run it through his personal server at home. Guess what happened when he got fired and nobody knew it was setup that way but him…
"questionable IT guy" :)
Oh, that was just the start with him. He was also reading emails between the CTO and the CEO, among other things
in-house xml parser implemented as a gigantic single perl function "because xml::libxml is slow." (lol, no) the place handled about 80% of all hotel reservations in the united states at the time.
I will just say this: old unpatched windows XP (SP1) desktop under a dusty desk. Login was "admin". Potential consequence of failure: deadly ammonia cloud enveloping a city.
This feels like the premise of a summer blockbuster.
A single .bat file running on a Novell NetWare 6.5 network (separated for "legal reasons") is used to print about 10k paychecks each month for at least 12 years
There are pieces of Top Secret military stuff that still run off Win 95 or Win 7.
This is why I just don’t buy into 90% of the conspiracy theories involving the Gov….
A utilities company I worked for ran their billing system on an IBM mainframe that required a piece of kit, a cluster controller IIRC, in each remote location for 3270 terminals to talk to the head office. We had the job of updating the controllers to talk over X.25 1/
At one location we visited, the controller was in a store room, surrounded by junk & balanced on an upended wastebasket. Updating it required loading a new config file from an 8”(!) floppy. It took several attempts because the drive was basically scrap at this point 2/
Despite the network being necessary for billing & newer controllers being available with 3.5” drives (woo), the beancounters wouldn’t authorise their purchase because it was “too expensive” 🙄 3/3
Circa 2005, white box that controlled a very old testing tool. Took a minute to figure it out, but it had a 1992 release of Solaris installed. The nerds from the comp science department came over to take pictures. Original hard drive was installed and still working.
A COBOL to PostgreSQL to both MSSQL/Java ecommerce and Perl/MySql ecommerce systems which then fed back through a SaxBasic script to post back to the COBOL system.
A cluster of three servers, all of them 17 years old, 1 still working with all but one root disk not broken down. Yet.
I don't work there, but SpaceX's crew dragon touch panels (yes, those panels) are run on Chromium and built with React. As it's the future, they've also done away with old-fashioned things like joysticks. There are no joysticks on Crew Dragon. I will let that sink in.
That's gonna have to be the most stable react app ever. Let's hope it doesn't freeze or smth
They actually used web components, not React (which didn’t support custom elements at the time) as part of their custom “reactive” library 🙂
r/spacex - Comment by u/spacexfsw on ”We are the SpaceX software team, ask us anything!”
23,708 votes and 7,329 comments so far on Reddit
reddit.com
My top nine answers are all Airflow.
We were talking to the head of a law a office about Wi-Fi security, and why the Walmart Linksys router wasn’t ideal for his clients’ and his own sensitive data. We asked what his clients’ security and confidentiality was worth. He deadass replied, “About seventy bucks.”
Also nearly every federal government staff tools are god awful and compromisable.
I once ran a multi-camera video livestream of a concert on a single old PC laptop I reformatted to keep it light. Cameras plugged into Firewire on an ExpressCard. Soundboard into a mic jack. The laptop was plugged into a very long extension cord & series of Ethernet cables + hubs
Maconomy.
US border patrol was still using Windows XP back in 2015. The officer that was interrogating me for 5 hours (and ultimately deported me) at some point complained about all the popups that kept appearing out of nowhere…
And yeah, that was the same desktop computer that processed all my info and even the video of my interrogation.
Not a company, but I remember an astronomer at a South African observatory who explained that their Internet connection was so slow they sent photos to their university by memory sticks transported by carrier pigeons.
Major university medical lab. The electron microscope's vendor is out of business and it only runs proprietary software on a Win2K box - that is infested with malware and popups. Solution? Drop from the network, no fix/reimage. No one knows where the drivers for the scope are.
A CRM with PII for internal use, hosted on a public Azure server using http. The dev had rolled his own encryption in js: a function called ENCRYPT(), with the key hardcoded in plaintext inside. This was shipped with the rest of the js and was already loaded at the login prompt
Blade server stood on its side on the window sill was the main product build machine. It had a USB Christmas tree light that was an indicator that the server is on and alive.
A company who’s entire business relied on 4th Shift CRM. That software required nightly processing but their business was build-to-ship that day. They hacked the software to pieces using MSAccess to extract data from manufacturing and give it to shipping and accounting.
Banyan vines
At my previous job there was a domain controller that hadn't been patched in ten years.
A multi-billion dollar online retailer using a tampermonkey script written by a dispatcher to manage their -entire- driver/route assignment workflow for a given business space.
A wheelie trolley with a router on that had bare wires connected to a frame relay circuit. If the trolley moved 1 cm the WAN went down WAN down? Jog the trolley
Application service provider and we used to house some of our own servers (email, for e.g.) on site. Sometimes we would find them rebooted inside the locked server/sysadmin office. One outlet also had power to the OUTSIDE wall (we found). Cleaners were rebooting them vacuuming.
I once had to write a bunch of JS embedded in Adobe PDF’s to make a particularl form dynamic enough to adjust based on how certain questions were answered. This was in 2012-2013 for Obama Care, so you can probably guess why we didn’t just build a gd website for it
I am still traumatized
Company with a factory and HQ in different states had LOB app that ran some old server in the HQ. To print their factory work orders they had to print via serial>USB>USBoverEthernet>VPN>USBoverEthernet>USB>serial>server 1/2
Then the print job went from the server>serial>USB>USBoverEthernet>VPN>USBoverEthernet>USB>Serial>Printer. It broke one time and it brought the whole factory to a stop... They were a MSP client so it took me 6+ hours to figure out WTF I was looking at and fix it...
From a few jobs ago: Boss: "We've been running a rcpd on Windows machines with targeted access control lists---" Me: *pokes head up over cubicle wall* Boss: "---yeah, yeah, you just sit down."
excuse me did you say _rcp_ as in the BSD4.4 protocol? D:
That's exactly why I poked my head over the cubical wall. I wasn't even listening to the conversation, I just sensed a disturbance in the Force.
Consultant for the health ministry in a Spanish speaking country to check why their database was so slow. well it was build on Visual Fox Pro 9x, no indexes, 3 tables with 100 columns each, all the fields were strings. I gave them options.. Their solution was to add more ram.
Lotus Notes 😰
Saw the excel workbook a small utility company ran everything off once. And I mean everything. They'd somehow built a variety of auth stuff into it and a bunch of SQL queries. It was like an internal webapp if all you know is Excel.
Mine arent as bad as some.. But I am continually amazed how many "enterprise" solutions are Access "databases"
My first database was Excel. Don’t @ me.
Every industrial system older than 20 years is held together by chewing gum and twine. HMIs running windows nt with no spares and it bring the whole line down if it dies.
Early 2010's, a P233MMX clone sitting in a basement, running SCO Unix 5 and processing 2M USD credit card payments a month.
How about the waste water treatment plant running off a Windows XP - no vendor support. Multi-million yearly revenue, critical to business operation, yet ignored by Plant Management for more than a decade. The amount of times we cloned and ported that XP OS...
Wrote a DOS-Program in the 90s for a customer. Printing by sending ESC-P sequences to LPT:. Worked fine, until they bought a Windows GDI Printer. Sources no longer existent. Hex-edited LPT: to LPTR and built a Java-Parser for those sequences to render the pages in AWT.
A small ISP stored their customer database as a CSV text file. You'd update it by hand editting it; worked until someone flailing about trying to exit vi ended up with it being encrypted...
They were used to using ":x" - save file if changed and exit - to exit vi Unfortunately ":X" is the command to encrypt the file with a password.
Thankfully, we worked this out & then managed to get our colleague to remember what he'd typed after the :X as the password - so we could recover the file. Was kind of tense for a bit! The password turned out to be ":x"
Teach people to use ZZ next time. Works in vi mode, not ex mode, so not :ZZ, no colon. Avoids that particular pitfall, and is faster because two times the same key.
A GPS NTP appliance with wired aerial, on a public IP address (but internal network) which noone had any login details for - and no support supplier - which provided authoritative time for the entire SCADA and Corporate network for a power company.
I remember a "magic" node_modules zip file on the company shared drive that was the only way the production JS app would build 🙂, it was something like 5 years old and we had to copy and paste it into the codebase instead of npm install
A domino server hosting Lotus Notes 🤢
Lightning struck a communications bank in the 80s and burned the half the board to a melted blank lump. The other half remained in use until about two months ago
A red coloured firewall.
Dosbox was business critical software as it was the only way we could run 2 critical systems that were written in 1995 & 1998 respectively...
Debian 3 server used as the main firewall for the office (in 2013). Iptables with comments like “let’s stop some hackers”, and blocking /24s coming from Russia. Uptime was insane. The name of the host was “tequila” because anyone who had to work on it wanted to take a shot after.
I wrote an adhoc piece of SAS code. I moved on to different firms and 7 years later came back. Heard rumor that no one knows how to update this Janky but important code cause everyone only knows Python. I volunteered, and to my surprise- it was mine. Still had my old directory.
Felt like I made a real impact by leaving a legacy 👍
First marketing agency I worked at in 1997 used some old version of File Maker Pro. Every time we wanted to give a change-order to the creative team, we had to print out a "green sheet" and the default in FMP was to print all records (of which there were thousands). 1/2
If you didn't click "current record only" it printed every request ever logged. It was basically a rite of passage for every new account person to have at least 1x where they forgot to click the button and would go to the printer and try slamming "cancel" in full panic mode. 2/2
Extended the life of their existing cabling by installing Ethernet to Serial adaptors all throughout the network
WSUS Server ;)
The entire UK court service running bulk case issuing via 56k modems and windows 98 clients in the late 2000s. They then replace it with a soap service that accepts fixed width strings in a string array.
A large pharmacy PBM still has their primary plans and claims system running on a DOS green screen system (as of 2020)
FTP file transfer into a bank of "virtual" modems to copy files to & from an old Prime system
An IFRAME loaded JavaScript to refresh page content. It loaded financial data. Early 2000s
"Janky" means something good, right? Right?! 😟
I’m using a mainframe computer to do part of my current job.
Windows 98SE desktop and a Windows 2000 server for gold pricing, tracking, and reports. This was in 2012. The server technically was connected to the internet, and that scared the shit out of me.
Worked at an org that had a 10 yr old air-gapped desktop pc running inventory software for their sword check-in/check-out (yes swords). IT didn't know it existed, it wasn't backed up, no one knew where the software came from, no one knew the last time it was turned off.
A custom workflow scheduler that had the worst code base I've ever seen with more bugs than features, but it was the CTO's pet project (even though he didn't have the time to maintain it), so there was no getting rid of it and pushing for its removal got me fired.
The CTO, while clearly intelligent (Has a PhD, though not in compsci), was not all that competent and is responsible for some pretty bad security holes that I'm not going to talk about as I doubt that they have fixed any of them.
I offer this screen cap from a major banking company customer service portal, just a couple of days ago:
I bet you could contribute to this thread
When I worked at Sun, most WOS ("wad of stuff") contributors had source control, but that did not include the hardware groups. We derisively called them the "home directory consolidation" and it was common to pull a production driver image from something like ~author/tmp/ce.2.12
I remember those. Stats store kinda started off that way tbf
I interned at a small shop that self hosted client websites back in the day. They had an array that would fail once a week and alarm across the whole office. Random employees would go up to the rack, eject the drive tray and slam it back in to 'fix' it.
Classic macos era Applescript that pre rendered html websites.
I thought it was clever. We ran a reports server for a 400 employee company on a desktop computer. We later turned it into a farm of ten desktop "servers" standing behind a load balancer, pushing the limits of the vendors licensing definitions.
When the farm couldn't keep up with the real-time demand, we added a report scheduler tool that would launch reports with a delay then notify the user when it was ready. The tool monitored server load and kicked off reports when it wasn't pinned at capacity.
Observatory looking for Earth impacting asteroids. We used a 15+ year old IBM proto-RAID to store all images that crapped out if the room temperature got >65. As a result we kept the AC at 60 and just bundled up.
Company was using an actual mainframe that you interacted with solely by text-mode interface on a dedicated console that showed “COPYRIGHT 1985…” at the bottom of the screen well into the 2000s. Like, past 2010. All transactions were processed via this computer.
It was an actual IBM AS/400 running OS/400. Part of the nightly duties included swapping the tapes on the tape drive… In, like, 2012
I once had a backup server running a really old hard drive enclosure. The enclosure had two separate power supplies failing constantly. We bought a modular power supply and had molex connectors running into each drive for years before I was allowed to replace the enclosure.
The entire UI ran on an extremely user-unfriendly CMS with HTML, CSS, and JQuery written in random places like an unorganized word document
Hope this qualifies: I had a team lead that was sitting in our FX trading group. Asked him if he was trading Euros? He said, “ never had one” :he thought I was talking about Gyros! Could have been close to lunch time! Hold the tzatziki sauce.
MS Excel to keep track of a 2,000 person workforce ... paying some schmuck $15 an hour to design and maintain it, and produce accurate pop reports ... and the schmuck had never used Excel before but really needed the work, so claimed he was an expert and learned via online help
One windows XP pc in 2015 that ran the payroll for the entire business with no back up.
Also an examinations authority that used a dot matrix printer to print all public examinations results and the “data” was gathered by a woman with a highlighter row by row. This was 2018.
I know that as of around 10 years ago, a significant element of the UK flight booking systems were dependent on a digital video camera pointed at a 1980s monochrome CRT monitor to literally screen scrape live flight information.
There had been several attempts to replace this feed and none of them were successful it would seem. I always wanted to hear about the attempted to shore up this system but no one was around anymore that had been involved. I'll always associate it with this Larson cartoon...
I was contracted to work on this system. It's exactly as described. Fortunately my job was to pop the screen-scraped data in a database, although actually booking flights involved a weird terminal program (also screen scraped).
I could stand to hear a bunch more about this! Any write-ups?
I'm certain there are plenty of others who've had to deal with it and I don't recall signing an NDA so could be some info out there on it. I'm afraid it's been so long now that I'm hazy on the details so don't have a huge amount more to add
I think the one I was working on was called JET2 - but the Travel Agents I worked for has since gone out of business...
Is there more information on this anywhere? Sounds wild
In the late 90's I worked at a dial access ISP that served the Eastern seaboard. The user tracking (and therefore financial) system was Magic. I'm not joking, there was a database company and product in the 90's called Magic which ran on DOS (Windows) machines.
So, predictably, there was a PC hardware machine in the back room that had the hostname "magic" and this company spared no expense in its construction. By which I mean it was put together with leftover parts from the company's PC building side of the business.
Too important to give it something as useful as a "tower case," this machine was a motherboard laid out on a towel sitting on someone's desk. A desk fan nearby blew air on it. All its internal components just out for the viewing.
I worked morning shift, so I only got to read the reports about this later, but apparently towards the end of a workday the system went down hard. The first person on the scene shows up to find out all its RAM has been plucked out of its SIMM slots. Someone just yanked em hot.
The back room this machine was in was between the technical support bullpen and the break room, so they immediately close the doors, shut down the phone lines, and ask everyone to line up. One tech support staff is wearing a jacket. It's the south and it's summer. Only one.
The NOC super walks up to this person and just straightens their collar, right. The SIMMs all fall out around the dude's feet. Needless to say, they got fired.
The jankiest thing here, of course, is how the company treated tech support. It was a "loss leader." Techs were ranked by their ability to terminate phone calls the quickest without Karen escalations. You don't need to be helpful you just need to get the customer off the phone.
We tried over the years I was there to try and repair the company culture, but in the end the best we could do was just provide a promotion system to get out and into one of the engineering roles; we promoted a lot of techs into low level ops roles.
AS/400s in 2018?
A system so old that it didn't have a real GUI and used the employee's initials for authentication. It was like an unholy marriage between a username and a password.
Worked at a company in the UK which had one ancient server running a critical app. If it he app failed the company would lose millions. It was so flaky that if powered off it may not power on again. The person who wrote the software had died and left limited documentation.
Worked at a gaming company where the database of 100K+ players and results of millions of games was running on a generic beige box sitting under a dev’s desk. Discovered this over a long weekend when IT packed up desks for office remodel and everything stopped working.
there was a time where all Square transactions had to be routed though a single TCP connection to the credit card network, emulating some old mainframe system that expected hardwired copper connections
it literally only allowed one connection, so if you lost the pipe you couldn’t reconnect until the other end was convinced your old one was truly dead
meantime the whole company’s service was down. so my team would get paged 📟. we got paged literally every night for a couple weeks, usually at 3am, with nothing to do except wait for the server to let us reconnect
Three 42U racks full of servers, switches, and a random assortment of other networking gear, held together with batch files running in logged in admin sessions behind very simple passwords. All to provide telephony for a 400 person site.
Was eventually replaced with a resilient solution consisting of two 1U boxes. Sadly the replacement project managed to exclude getting rid of the old system from their project scope so it's all still sat there waiting for someone to find the time to dispose of it all.
It was a 14yo tech stack that we had to migrate to the cloud - everything was EOL, the Dev VP was running XP, no one really knew how it worked, and code deploys took a week (and rarely worked). Then 1999 NetScreen firewall in production was the least janky component.
A cluster of over 300 servers running in a basement office hanging on Home Depot racks with zip ties and dripping thermal grease, the A/C was so over taxed we had to flush the system weekly via a hose on the floor to a toilet.
I client I was doing very occasional work for was still using Netware 4.1x in 2013.
a mysql trigger which answered phone calls
The “programmer” guy 😬
Feb 2000, my first job as an IT pro. Startup infosec company. We did automated security checks (port scanning, protocol identification, vuln checks). No server room, scanning machines were consumer desktops stacked on wooden IKEA racks in a small closet-size room without a fan.
When we upgraded all PCs to Office 2007 we got a call from one of our abrasives science labs. Their machine for testing abrasives durability had stopped working. It was a robot hand connected to some Lego Mindstorms kit, controlled by an Access database that didn’t work on 2007.
It had worked fine for years, after being made by one of the team’s kids when he was on work experience during high school. It was better than most industrial kit, so continued to serve a critical purpose.
This explanation is better than the description!
It was definitely better than the question: "can you drive up to Coventry to fix this Access database, so our Lego hand can start working again?"
I've worked at several organizations that used Oracle products
There used to be web server software called Quid Pro Quo and I got to try to keep it running for a while when it hosted a department's website at a university. The professors were on OS X and I believe this was still running on a 68k.
I saw a "data layer" sitting between the web server and database. It was a single class with over 20,000 lines of code and not a single unit test. We also had exclusive lockout source control at the file level so everyone had to wait until the locking engineer's work checked in.
Spotted in a file that hadn't been touched in years: public String pluralize(String word) { // TODO: find a better pluralize algorithm return word + "s"; }
Mission-critcal, out-of-support Sun 450 with no redundancy running an in-house application and unsupported database with with no backups running in a "IT closet." Still makes me twitchy.
It’s a perfect storm of sketchiness
A large-ish pile of autoit scripts written by yours truly ...
that brought me back 10 years
Jenkins. But seriously, when I worked at Macromedia, one of the Flash compilers (yes, one of...) had a 10k line switch statement in the parser. The whole codebase only had one comment: `// These are not the droids you're looking for`
I would find my own code by searching for comments: literally searching for "//".
An entire high-performance database engine built entirely in PHP. As a single 10k line PHP file. Complete with a custom SQL parser. They wrote it because MySQL was too slow. 🤣 Oh and it was PHP 4 - all functions... I believe it's still in use today in an enterprise product.
I worked for a n event booking company, with clients paying big money to plan 1000+ attendees itineraries and accommodations. Our job was to split off a “new product” from a huge monolith - basically extracting code to make a separate deployable artifact with a fresh frontend…
The process was fascinating, we were sending code samples to the chief architect, because no one could understand what’s happening inside, except this single guy who spent ~10 years with the system. Why doing it like that you would ask?
We had codebase consisting of god object and java methods of 5000 lines each. 5k was not even so bad, we could comb through it. But you can’t really understand whether this should be a part of a new system and where you should make a seam.
That was the only time when I’ve met a really unreadable code in production, where no one can be sure of what it does and whether it is still relevant. If it’s get executed then it’s needed, right?
I’ve hit the JSP size limit for the first (and only) time in my career on that project. Also, I’ve faced the first real deal prod incident - an SQL perf degradation made a service really slow (basically DOS) for 30 mins. Costed ~$500k in lost revenue.
I’ve never seen an email from a CEO written with 72pt font size ever before or after 😄
One of the greatest wins I’ve personally contributed was an “optimization” of a method chain call: It was modifying hundreds of thousands records in several tables… and then just deleting all of that! Shaved off ~10 secs of a use case processing time. What a time to be alive!
I spent like 1.5 months trying to figure out what it was doing and whether there is anything to optimize at all. That was a wild bet, but a rather cheap one: I was a cheap overseas outsourcing “resource”, and they can afford me sitting and reading through legacy code 😇
The product was a success at the end: it worked in IE6, and the main spreadsheet-like page loaded in less than 10 sec! It was impossible with the main monolith and was considered a great success! Overachieving the goals 😅
Tech? Ser - we use Excel only.
PPO I worked for handled data transactions with another company via a 486 machine running Corel Linux. The machine was under the Manager's credenza. He built it because he wanted to learn Linux. That was one of jankier pieces of tech at that company.
When I first started working for my company a few years back they had a lotus Domino(?) database (I could be wrong about that) running on an optiplex G1 with Windows 2000 . It had customer records going back to the mid 90s if not earlier. It was not decommissioned until 2014.
The critical pager system at a hospital I used to work at ran on this old DOS box. On the back of it was a null modem cable that connected the two serial ports together. This was needed so two pieces of software could communicate with each other on the same system.
The APEX laboratory information system @NewcastleHosps @DXCTechnology @DedalusUKI ... Thankfully definitely being replaced over the new few years.
A custom built Java to .Net transpiler that has become unmaintainable because the developer no longer works for the company
A week after I was hired at a small, but multi-state, healthcare provider, another developer was let go and when his desktop workstation was decommissioned we learned that machine was the central processing unit for the company wide reporting application.
But wait, there's more: the database record schema was a column indicating what kind of data (string,float,integer,etc) the other text column contained. At another job we used SCCS, so...
1990's... all stock exchange market data used by eveey trading terminal at investment banks was reported to the exchanges via 2400 baud modem uploaded to a Wildcat BBS running in a closet at the NYSE. I had to save data to a floppy and sneaker net to a diff machine for billing.
Bob from IT... Keeping the world turning since 1943!
Earlier this year, I found SQLi and XSS vulnerabilities in a PHP time management software from 2006. Doing OSINT I found several “prestigious” colleges used the system, exposed to the internet. It stopped being maintained in ‘13… Disclosure was fun 😩
PHP TimeClock 1.04 SQLi/XSS Disclosure
A Deep Dive into Vulnerability Research with Docker and BurpSuite
tbutler.org
A calculator to determine pressure capacities for critical infrastructure written in Fortran.
Applicant Trackinf Systems
Goverment agency printed case files on a Windows server by invoking Word headlessly.
I worked at a telecomm company that's customer service system and provisioning system was underpinned by COBOL72 (as in 1972). When Y2K came, they had to get all of the retired programmers to come in for one last score and fix Y2K issues.
Sysadmins came to and said: "we're turning the whole system on and off for reasons" I said no, they did and this old (crucial) AIX box dies and it backed up to one of its own partitions. So I had to guess the last patches applied to the scripts as they weren't in CMS
I can say that Comcast relied WAY too long on mailing beta tape around for in-house production. Also as the singular junior dev who wasn’t paid much and who wrote the catalog software for tracking licenses of properties on those tapes they extremely got what they paid for.
I started working at a K-12 school district in the early 2000s and when I walked in on the first day their payroll server was running in RAID0, no one knew the Netware admin password, and when I hit the button to eject the backup tape the drive fell into the chassis. 🤷🏻‍♀️
Sounds about right
and @IceMupppet have some good ones. most memorable was @arobodude when he'd edit the site on the fly (no version control or backups) and one day we lost it all
The time the firmware server got deleted and there were no backups. I had to code it again from memory in a day. The flat json files hosted as a CMS for hundreds of thousands of users. OrbBasic… I could go on for awhile, it was all duct tape and bubble gum in the beginning.
A Java SOAP web service that generates templated shell scripts that execute informix-4gl code and then capture and parse the text outputs of said shell scripts to return to the client
I will note this was in a year that started with 201x
In an internship I did there was a product that was an enterprise class java application, you name it - it had that design pattern, even ones that didn't make sense and there was one file with 500+ ifs that figured out what the hell it was meant to do.
Is it the 88k line C++ file with no classes and a .c extension? Is it using ClearCase? Or is it the onrushing disaster of being forced to use a vendor’s Windows-only “MFA” app on our Linux system?
Windows 2000 Server in 2020
Best thing I've ready today. =)
Hospital imaging system (think x-ray) server kept crashing because staff kept installing a Thomas Kincaid screen saver. Oh, and it's not a server but a desktop on the floor under a nurses station running XP. Hospital deems it mission critical in business continuity docs.
A payment processor's list of transactions in their own UI easily falls out of sync with the backend. If you click to Void the first in the list, but a new one was made since you loaded the page, bam you just voided the one that wasn't even visible.
We had a central automation system that controlled several television stations. Scheduled commercial breaks were easy, but ones rolled manually were done with a system lashed together with a nest of hand-etched circuit boards full of PIC microcontrollers (1/n)
(2/n) hand-coded in assembler. It absolutely had to work, because any lost break was many thousands of dollars of lost revenue, but the only person in the station who knew how it worked was the guy that built it.
(3/3) I was, of course, that guy.
A vax with a tiny amount of memory. Can’t tell you how many times we had to run in and find files to remove because there was not enough space left for the operating system to function.
A desktop Java app that queried an ancient Informix DB attempting to find an optimal distribution of shippable items through a process referred to as “the spin cycle.” This ultimately output thousands of shipping labels and maps telling warehouse workers where to find items.
Bonus that the output of “the spin cycle” was fed to MS Access for formatting and the print job would lock up the system for 30 minutes or so when sending the job to the printer. The label printing took hours.
386 tower PC running Netware. Case off, with a house fan blowing on it. The server ran a hospital around 1986.
Only way to download a file needed from the state, daily was over a modem connection, tbf it was secure connection. They had to install a special drop for this line right next to the emergency POTS line when they switched everything to VOIP. Wrote custom software just for this.
I was brought in to redesign a high traffic website that performed complex calculations for users by using Microsoft OLE to leverage Excel running on the same box as the IIS web server.
FileMaker. It was our CRM, billing, HR, sales, quoting and briefly also project tracking. Ugly as fuck and an absolute mess.
In a call center with about 30 people, the PABX had a serial cable to an SCO box that ran their billing program. I needed to tap that data but the PABX had only one serial, so I made a cable that had 2 heads, one w flow control to the SCO box, one w just TX from PABX to my system
It worked for at least 5 years after I left.
ChangeGear. I worked at a higher education institution and they used the ChangeGear ticketing system. it was literally duct-taped together. constantly crashed, could barely handle the most basic of tasks. constantly overloaded. it was absolutely miserable.
My half-ass deployment tool that I wrote because Ansible didn’t exist yet.
The sales org at a public global SaaS company running on a single dell tower under a desk
I went out to fix a non-booting NT4 server once. I was told to take a USB floppy drive and some disks. It was 2012.
At one place I worked, stock orders were typed in via an old green-screen terminal. For each record to save, it *had* to print to a connected dot-matrix printer. There was no ink, but you still had to occasionally take the blank continuous paper it spat out and feed it back in.
(I revolutionised productivity by tearing the stack of continuous paper down to three or four sheets and taping the ends together, so the same loop fed through the machine indefinitely without someone having to faff around picking up and aligning the stack of paper every 15 mins)
A multi-billion dollar, major brand company runs on an ERP system from the 70s. It is non (pre?) relational. It’s literally one super-wide, single table where precious columns are re-purposed everywhere. ETL pulls data into SQL. Then NoSQL then pulls it back to non-relational.
Last month received an automated request by an ATS co complete a background check, when I clicked the location it opened Mapquest … Yes Mapquest
One place I worked we had a development “server” that was a desktop computer cobbled together with leftover parts shoved into a server rack. One day it stopped working. The CPU fan and heat sink were facing down and popped out along with the CPU.
Email server running on a laptop in a supply closet. No circulation and so the laptop melted and email was down for a day
marketing agency's airtable base (with the "over capacity" warning for 1+ year) with zapier hooks to send emails to clients or @ mention team members in Slack when item statuses were changed. this would have been ok-ish were it not for the GUI on top that ... didn't really work
Bank anti-fraud run by giving three people a green bar printout of all transactions for a day, each person would mark what they thought was "suspicious".
Phones as the only way to get internal tech support at a major Enterprise co., up until 2016 or so (as opposed to using a ticketing system). Drove me absolutely nuts.
University IT in the mid 2000s had a 15+ year old 70ppm 2 color Xerox laser printer which nearly all printing, including time-critical stuff like W-2s. A critical part of it was ASCII -> EBCDIC translation software which ran on a 386 shoved in a corner.
Am I understanding correctly that the printer was an EBCDIC printer? Because that is hilarious all on its own. (My dad was an IBM sales engineer)
I'm not sure if it was but because they had bought it with the previous system (IBM mainframe) the controller was.
Gotcha. Hilarious and head-desking both. I'm assuming there was some sort of hard coding for printing as opposed to pure institutional inertia?
A long story of university budgets, etc. It was reliable.
My last experience with spending a very small part of a university budget (in the early '90s) multi-part carbonless form with each layer having various cutouts: "Press very hard, you are making 8 copies"
I've written EBCDIC to ASCII converters too.
A company that ran old ibm token ring mainframe COBOL database - most of it written before many of the current employees were born. Everything interacted through a screen scraped terminal emulator via MS comti.
Tbh as a user one of the scariest things to me is medical clinics using paper forms that have been photocopied over & over & no one has original soft copy anymore. They make patients fill out by hand, then someone presumably types it into a totally diff electronic record
I keep thinking abt medical errors happening purely due to data entry issues.
Email server running on a desktop pc, manager ignored warnings, motherboard died, in an attempt to recover data we had a similar mobo, psu, etc spread out on a desk and it booted,mail came back up. We put caution tape around the desk until replacement came. Ran for a couple weeks
Toss-up working at a major ISP; between a Juniper Steel-Belted Radius deployment that hooked into Oracle and LDAP with no clustering (out of band sync) that enabled authentication to the entire network infra, and the software they tried to replace it with; "KnownAccess", the -
Latter of which I wouldn't be surprised if you've never heard of them. Who has? They're a company owned by a single person who contracts work out on Fiverr... I surmised likely an under-the-table deal with someone in leadership.
The KnownAccess product is a NodeJS server that runs in root context on a server appliance, and gets patches only when you pay the company to provide a specific update to the platform.
Doing some sales engineering for a network management company. National-scale mobile operator had whole network model in an XL sheet. I poked about and wrote a function to validate circuit routing with a bit if VBA.
Made a huge mistake: I showed it to the client to demonstrate they had some serious data issues and should buy our product. They stopped being interested in our product and became fixated on my ghastly VBA hack.
They were still using my 10-minute hack to plan routing for their entire network three years later. They might still be using it. One day there's going to be a massive network outage in a medium-size European country and it will be my fault.
*cries in Club 33 Member Services*
A few years ago, I worked support for POS software. Many locations didn’t have high speed internet so we connected to their systems using dialup. Their entire business hinges on me being able to dial in or walk a non-tech person through Linux terminal cmds
Worked at a place whose entire GIS mapping and work management tooling ran on Windows NT and to "modernise" it the entire server simply got @VMware converted so it could be run on modern hardware.
Same company had its customer billing system written in COBOL and we had to pay an external developer to maintain it who charged for features by the keystroke... 🤑
Oh sweet summer children: the correct and terrifying answer is of course the US power grid
This one might win the prize.
In 2005, the software used to track friendly forces in Iraq could only run on windows NT, on barely functional toughbooks. We couldn’t patch or even troubleshoot them. People would literally get killed
A company I worked for had an ancient x386 running their huge parts picker tower, because it had a software/hardware combo licence tied to something inside, which couldn't be replaced without significant expense.
When my husband does rounds at the hospital he writes his patient notes on index cards. He then drops them off at a random building where a woman enters them into the computer. The cost to link the his medical practice's system with the hospital is an unaffordable upgrade.
That sounds like something that would happen in the 1950’s. Wow.
Yeah, one day our young dog got hold of a stack of patient cards and shredded them. He had to go back to the hospital and essentially redo his rounds.
Ever heard of a gun loop?
This is how the music in our lobby plays. No one knows how it got there. No one knows where the music originated from.
80% of veterinary clinics in the United States rely on record keeping software that started as CRM for piano tuners. And is written in Delphi and requires running your own local server.
This one is good 😳
I don’t know if this qualifies but in 1999 Akamai shipped us nine rack-mounted real-time encoders through the US mail which more or less disassembled themselves in transit.
2005 model OKK PLC unit. Older than dirt and covered in oil and wonder why there’s so many problems.
Microsoft Mail server for the Mac. Circa 1995. Every two weeks it would crash the Quadra it was running on, so I eventually started going in on Monday morning about five minutes before everyone else came in to reboot it.
I was offshore and a sales guy had promised that we could send data from the rig to town. (1993). We didn't have a working satellite. I connected via a bag cell phone, through a 1200 baud modem. I had to telnet in, start a program called kermit to rcv, jump back and send data
I had to use AT commands to dial the modem. To send a MByte of data it took an hour, and if anything moved, the kluge of cables going from the phone box to a serial port would disconnect. I started to explain to the client everything, he said "not interested"
Wooooooow. Hang in there little buddy.
Co I work for used to have a control station for sending runs in to process. It was a bank of 9, 9 position switches to enter run numbers, this had to match what was in the computer. 2/7 worked, and the 7 dead ones may not show up as the # in the window (1/2)
If anyone lost the post-it with the info for the 7 bad ones it had to be figured out by looking at the lights on the mid 80s PLC input card and figuring out how to convert binary to decimal
My first job, we sold an Enterprise management system for a very specific sector. On the surface a Windows client, server backend, Oracle DB. REST API with JSON? SOAP XML? Wrong! Pipe delimiteted strings streamed directly over TCP!
The user acceptance test for the new policy and claims system we were building at an insurance company — literally the backbone tech of the business — ran on a laptop next to my desk.
Note that the new system was built in some weird proprietary vendor framework built on Delphi. It was 2013. We were replacing something written in Visual Basic 6 (main system) and the most complex Excel macro setup I’ve ever seen. Again, literally the backbone of the business.
Was asked to “onboard” the AOL Mac client into our centralized CM/RM. Walked to the guy’s desk and saw the G3 under his desk. It held all the source code. Above it was his personal refrigerator. I imagined the impact of a power outage. I powered down and took it to a safe place.
Mimeograph! Lining up to make purple, damp, & smelly copies.
Lol, stop that! He will have no idea what you're talking about!
Classic. I remember my mom making something like these at home in a cake pan on gelatin.
SAP R3. Software designed to ensure perpetual employment of SAP consultants while nobody can do anything except things they do wrong because the UI is hostile to interpretation.
Windows PCs. I’m serious. And I’ve been using them since DOS 1.0 up through windows vista. Every time I try the new version of windows I just get nauseous.
Not a company but in the 90s my geologist boss had this calculator on his desk. Legend had it that it was the gold standard of its day (?)
Are those nixie tubes? 😻
I had to look up what those are but yes, I think so!
At a biz event in 2010, I met a guy who used Windows 1.0 for inventory and invoicing. He said, "Software is like a Crescent wrench. Unless I break it, why do I need a new Crescent wrench?"
Working for a state contractor, the agency that our site handles is literally run off a COBOL mainframe. It’s cheaper for them to pay a 10k fine per year than it is to move into the 21st Century. This is why we don’t take direct payments & can’t change typos in names…
The logistics program SAP!….
The single comm LCpl who built the network.
Facebook circa 2010
I know for a fact that a major US financial company’s production system is a single mainframe written entirely in COBOL. The code is undocumented. They have a backup, that they’ve never tested.
Oh, also, the NYC subway’s credit card processing is dependent on a system that only one guy named Miguel knows how to operate
the MTA's fare-processing system relying on a password only known by some guy named Miguel seems like a bad idea. newyorker.com/magazine/2018/…
A Twix machine receiving data from overseas at 110 baud in Baudot code onto paper tape. (1978)
Due to HIPAA regulations in the mid-2000s, our process for sending files between offices for a state agency was: Office 1: Print out PDF, fax to Office 2, shred printout. Office 2: Receive fax, scan as a PDF, shred printout.
It terrifies me how much of our government still relies on fax machines.
I once worked at a place where everything, and I mean everything, ran on Lotus Notes.
There's been rumors for years that my universities class scheduling requires punch cards when it reboots
AS400 that ran compiled code that was blindly transferred off a burrows punch card machine. I was not realtime and needed data entered in a sequence. As the company grew the process the result drifted and the staff tweaked the numbers. They didn’t have the source code
Mission Command in the Army.
A very large print shop for a large media company used early NT servers. You had one chance out of 50ish that the SCSI drivers would load at startup. Servers had to be rebooted every 2 weeks.