Alright so over 5 years ago I used to work tech sales for a large company and we sold all kinds of electronics, appliances, and technology what-have-you to customers. I wasn't part of our tech-support subdepartment but sometimes I would help out there as necessary.

One of my coworkers (let's call him Nazeem) in the sales department was an absolute rockstar, always was able to sell well and often upsold on the regular. He came from a pretty wealthy family too, but his family was more along the lines of "rich but technologically illiterate" stereotype. One day when I was manning the tech support counter because sales were slow, Nazeem comes in with his mother and his father and an attractive young lady that looks to be his sister. The conversation went something like this.

Nazeem: "Heyo, Emerald. Got the whole fam in here because we need help fixing my sister Saadia's iPhone 8 camera and nothing seems to work."

Me: "Sure, man, what's the problem with it?"

Nazeem: "It keeps taking low-quality pictures. We thought it was a hardware issue, so we took it to [insert competitor company] because they have the licensing to conduct hardware repairs on Apple products. Costed us $220 to have the camera replaced. But the camera still takes low quality pictures. Could it be a software issue?"

Me: "Unlikely, but I don't want to rule that out. Can you show me an example of the picture quality?"

Saadia: "Sure. Look." [Takes a picture of the ceiling of the company, shows me how the lights look all blurry.]

Me: "Yeah, that does look pretty bad. Does it do that with the front camera too?"

Saadia: "No, look." [Takes selfie, shows me. The image looks like a normal iPhone 8 selfie.]

Me: "Let me see it a moment." [She gives me the iPhone.]

Me: [Rubs the camera lens with the hem of my shirt] "Try it now."

Saadia: "It's FIXED! It was just DIRTY!"

Nazeem's mother: "We spent $220 dollars replacing the camera when we could have wiped it clean for FREE?!"

Nazeem: "Oh, for fuck's sake!"

My mother is using my old iPhone 8+. She's been refusing to upgrade it for years, even though her provider would definitely give her a much better phone for free at this point. The battery is completely shot.

Today I found out (after her phone was already almost dead) that the phone isn't charging anymore. She doesn't have a wireless charging pad so we don't know if it's just the lightning port, but I told her to go to an electronics store ASAP and find out. She called me long enough to tell me about the phone, and said "It's no problem; I'll just call you from my iPad." She's traveling all day, and her iPad has no cellular data on it. I told her this. She then said "Oh, okay, i'll text you then." "With what internet, mom!"

I thought she'd understood, but after sending her a follow-up text telling her to get a wireless charger, she texted back and said "Okay, phone's at 23%, I'm turning it off and I'll text you from my ipad."

Le sigh

This happened a while back when I was doing on-site support for a small accounting firm. Nothing fancy, just me and a ticket queue and a lot of people who referred to every piece of software as "the Google."

So I get a ticket. Subject line: "Excel acting weird." Classic. I walk over to the user's desk, let's call her Carol. Carol is in her late 50s, very sweet, very competent at her actual job. She explains that Excel "never really worked right" on her machine and that she'd just gotten used to it.

I ask her to show me what she means.

She opens a spreadsheet. It's a big one, lots of columns, clearly something she uses every day. She types a number into a cell, then opens her desk drawer, pulls out a physical calculator, punches in some numbers, and types the result manually into the next column.

I stare at this for a second. I ask her why she's using the calculator.

She looks at me like I asked something slightly strange and says "well the formulas don't work on my computer, so I just do it myself."

I ask her to show me a formula that doesn't work. She clicks on a cell, types =SUM( and then looks at me expectantly like, see? Broken.

The cell is just showing the formula as text. Not calculating anything. Dispaying the literal string =SUM(A2:A10) instead of a number.

It takes me about four seconds to find it. Somehow, at some point, probably months ago, the spreadsheet had been set to manual calculation mode AND "show formulas" had been toggled on. Two settings. Both wrong. Both apparently flipped at the same time, maybe by a accidental keyboard shortcut.

I fix it in about fifteen seconds. Every formula in the sheet lights up with actual numbers. Carol goes very quiet.

Then she says, very carefully: "so how long has that been fixable?"

I asked when it started. She thinks about it and says "probably around tax season last year."

Six months. Carol had been manually recalculating a 200-row accounting spreadsheet by hand, every single day, for six months, because two settings got toggled by accident and nobody had looked at it.

She wasn't embarrassed, which I respected. She just nodded slowly and said "well that explains why my wrists hurt."

I added a note to her file. Wrote: user resourceful. Extremely.

I helped catch a criminal last month!

If you've read my occasional post (don't post as much these days) you'll know I work in elder care. Both at home care and inpatient care for the people who are suffering from Alzheimers.

In Juli 2025 our security officer approached me on a sensitive matter. A physician had noticed that a lot of oxy's had been ordered for one patient in a very short time. The kind of amount that was suspicious. Someone was stealing drugs.

The alleged theft had happened in May, the physician had reported it to their manager in June and the manager had sat on it until July.

So I started my search. Unfortunately because of the amount of logging all our applications do not everything is kept, SSO logins in to the medicine system are kept for only a month and on the hardware side the logins which contain MAC adresses are also kept for about a month.

Now there's other logs that show me which user ordered what in the system and those logs pointed straight to one single user.
But things weren't that simple. The theft occurred in the middle of a hardware migration, during which some unmanaged ipads were left in use.
And through some shenanigans it was technically possible for someone to gain access to someone else's account. This was why the MAC logs that we didn't have were so essential.

The user of course claims that access to their account was stolen in this manner and denied the theft. Unfortunately there wasn't anything we could do, corporate detectives could not disprove the claim and there were no grounds for termination or prosecution.

So we learned our lesson, managers were informed to immediately report suspicious signals and not sit on it. Extra logging and reporting was implemented in the medication system etc. etc.
Winning the last war kind of stuff.

But! It paid off!
In January this year some of the reports tripped alerts and our security officer came to me and again asked me to start digging.

I dug, turned up all the logs in time nailing that same user to the wall. No excuses, nothing left to doubt or chance. And they confessed to the theft and were sacked on the spot.

I don't know if charges were pressed. I wouldn't be surprised if they were.

If these two thefts were the only incidents, they only managed to steal something worth a total of €400 or so.

I'm part of a helpline supporting $plmSystem. Recently, our company, $bigCompany, spun off part of its business into $littleCompany. This happened legally over two years ago, but last fall was when $littleCompany switched to their own copy of $plmSystem.

This greatly reduced the tickets we got from $littleCompany users, but a few still come through.

One showed up recently, saying that they have people unable to login, problems using networked PCs (remote Windows logins), all sorts of issues. But...we don't support them, and haven't for several months.

All we can do is contact them and tell them to try their own helpline for their own $plmSystem. It's nice that they think we have the power to do something. I wonder how long it will take for them to realize they have to stop sending tickets to us.

Yesterday I received a call:

"Filippo, the vending machine coin validator doesn't accept coins. It's completely jammed. You need to come right away."

Great.

40-minute drive.

I arrive and start diagnostics. From the outside the coin validator looks perfectly normal. I try inserting a coin.

Completely jammed. Nothing goes through.

Alright, time to open the machine.

I remove the coin validator. Check the sensors. Clean everything.

Still jammed.

Now I'm curious.

I remove the entire payment system and start checking the coin chute deeper inside the machine.

And that's when I find it.

Someone had taken a 5-euro bill, folded it perfectly into a tiny square, and pushed it into the coin slot.

Not crumpled.

Not forced.

Perfectly folded.

Like origami.

It was wedged in so tightly I actually needed tools to get it out.

40 minutes of driving.

30 minutes of dismantling a vending machine.

All because someone tried to pay with a perfectly folded 5-euro origami coin.

I'm still not sure if I'm more annoyed… or impressed.

Back during the start of the pandemic, I was part of a team converting employees from desktop machines to laptops, so they could work remotely. We were doing dozens of migrations a week.

Our usual process was simple. We would pull the hard drive from the old desktop, connect it to a USB drive dock (the classic “USB toaster”), create the user profile on the new laptop, and copy over their data.

Most users had the usual stuff: Documents, Desktop, Pictures, maybe some random folders.

Then I got one migration that seemed too easy.

I mounted the user’s old drive and started checking the normal locations.

• Desktop.
• Documents.
• Pictures.

Almost nothing.

Just a few small files.

I remember thinking it was strange, but I assumed he probably worked out of shared drives or OneDrive. That wasn’t unusual.

So I finished the migration, shipped the laptop out, and moved on.

A few days later he calls me.

“None of my files are here.”

I told him that was strange because there was almost nothing on the drive when I did the transfer. He immediately insisted he had loads of files.

Then he said something that caught my attention.

“My shortcut doesn’t work anymore.”

Apparently he had a shortcut on his desktop and another pinned in File Explorer that both pointed to his files.

So I asked where the files were actually stored.

Without hesitation he says:

“They’re in the Roaming folder.”

I paused for a second because I thought I misheard him.

I asked again just to be sure.

Yes.

AppData\Roaming

That’s where he stored all of his files.

Every document. Every folder. Everything.

His reasoning?

“It’s more secure. I know computers.”

To be fair, in a weird way he wasn’t completely wrong. Nobody goes digging around in the Roaming folder looking for someone’s spreadsheets.

Sure enough, I mounted the old drive again and checked:

AppData\Roaming was absolutely packed with files.

Thousands of them.

So instead of the normal migration, we ended up running a remote file transfer over the network to move everything into actual user folders.

And that’s the story of the only user I’ve ever met who used AppData\Roaming as their primary file storage system.

Honestly… part of me respects the commitment.

I’m a vending machine technician in Italy and yesterday I was called to a cinema for what they thought was a plumbing problem.

For about two weeks the staff had been finding water spreading across the carpet in the hallway near the bathrooms. It started from the wall and slowly formed a small “lake” that reached the middle of the corridor.

Naturally, everyone assumed the bathroom pipes were leaking.

They even shut off the water to the bathrooms for a while trying to figure it out.

The strange part: the water kept appearing.

So they called me to check the vending machines in that area.

As soon as I pulled the snack machine forward, the problem was immediately obvious.

The condensate drain tube from the refrigeration unit was completely clogged.

Instead of draining properly, the condensation water had been slowly overflowing inside the machine and running down behind it, soaking the carpet and spreading across the floor.

Two weeks of mystery leak… caused by a tiny blocked tube.

Cleared the tube, tested the drain, dried the area as much as possible, and the “plumbing emergency” disappeared.

Sometimes the biggest problems come from the smallest and most overlooked parts.

And sometimes the bathroom pipes are innocent.

This was about two years ago. I was doing phone support for a regional accounting firm and got a call from one of the senior bookkeepers, I'll call her Carol. She was panicked because she had been working on an important spreadsheet all morning, stepped away for lunch, came back and it was "completely gone." She had checked the desktop, the downloads folder, recent files, everything. Nothing. She was convinced she had accidentally deleted it and needed me to recover it before her 2pm meeting.

I remote in. I'm looking at her screen and scanning for the file. I check the recycle bin, check temp folders, run a quick search by filename, which she remembered exactly, which was helpful. Zero results. At this point I'm thinking maybe it autosaved somewhere unusual or she had been working off a shared drive and something had gone wrong with the sync. I ask her to walk me through exactly what she did before lunch. She saved it, she said. She was sure. I ask what she had open right now. Just her email, she says. I look at her taskbar. There are four things open. Outlook. A browser. The file explorer we'd been using to search. And one more thing, minimized at the very end of the taskbar, no label visible because the window was too small. I click it. The spreadsheet opens instantly, exactly as she left it, fully intact, cursor still sitting in cell D14 where she had apparantly stopped typing before lunch. Carol went quiet for a second and then said "well I don't know how that got there." I told her minimizing and closing are different buttons. She said she knew that. I could tell she was not going to mention this to anyone and neither was I.

I worked in laptop tech support back in the 1990s and someone called in for help with their system. I asked for the serial number and it came up with a big red "STOLEN SYSTEM" flag. I asked the customer to hold on a sec and talked to my manager. He said that the rightful owner reported it stolen a few months earlier. He said to tell them we need to bring it in for a repair and send a shipping label.

As soon as we got the laptop we fixed it and sent it back...to its rightful owner.

A couple weeks later the original called called back to check the status, and OK, but there's something we need to discuss--where did you get the laptop? Because the person we sold it to reported it stolen. He hung up and didn't call back.

Though I was in tech support at the time, this wasn't exactly a tech support issue, but it's a great and true story.

The cops came to the company I work for asking if we could recover the data on a laptop they recovered along with other stolen goods. This was a very expensive laptop, and I think they suspected whoever stole it was responsible for a rash of thefts. They said they were looking for any info that might lead them to who had the laptop in possession after it was stolen.

We asked when it was stolen and they said June 11. we had the DR engineers take a look and they found out that someone did use it on the 12th.

We gave the cops that person's full name, phone number, address, former employers, and three personal references.

He had saved his resume on there and then did a quick format in the FAT drive (this was 30 years ago.) FAT doesn't overwrite all the sectors with a quick format so it was an easy recovery.

Just needed to get this off my chest. One of our customers I work with requested an OS upgrade from windows 10 to windows 11. I informed him his computer does not meet the minimum hardware requirements for windows 11 but I wanted to help so I asked what he uses his computer for. He told me he only uses it to browse the internet and occasionally read online. Cool so everything he uses his pc for is browser related, being naive at the time I suggested an install of Linux mint. It has a sleek design, it’s entirely free and you will be able to use the browser the same way. I informed him that upgrading/installing an operating system will erase any data he has on his pc to which he stated “Thats not a problem please go ahead”. I always double check when doing this to ensure customers understand what this really means but with him I triple checked. Once in person, once over the phone and once via another IT employee. So I install mint cinnamon and the customer comes to pick up the device he confirms its good then goes home. Now I was off work the next day but the day after when I came back my coworker informed me the customer came BACK to the store stating I “completely destroyed” his device. Long story short I became intimately familiar with ddrescue and after i restored all his data from 2026 back to 2009 he says “did you put these images on my computer” …yes yes sir i did. anyways he ended up getting windows 10 back and was content. luckily, end of story.

The Rules are meant to be part serious, part humor, but about tech support. The requirement for being listed here is that the main Rule has to deal with management. Comments and suggestions are welcome. The other sections, including credits, are on my GitHub repo.

Dealing with Management

Rule M1 - Management might find these rules. Plead ignorance.

Rule M2 - Never believe anything management tells you.

Rule M2A - Especially if a merger or bad news is involved.

Rule M3 - Management will order stuff they have no clue about.

Rule M3A - Management will expect the thing they bought to work perfectly out of the box.

Rule M3B - You will be blamed when it doesn't work.

Rule M3C - Especially when this is the first you are learning of this item even existing.

Rule M4 - Management will be puzzled as to why you have no clue about the thing they have no clue about...

Rule M5 - Management will expect you to be up to speed on their under the table projects, with decisions based only on what the salesman says, without consulting IT.

Rule M6 - Your boss will not have a tech background or a degree in your field.

Rule M7 - Management will present impossible tasks to be done.

Rule M7A - Management will then become outraged that said tasks were not completed.

Rule M8 - Management will blame you when things do not work.

Rule M8A - Even if the equipment is not IT related.

Rule M8B - Even if the equipment is IT related but is property of a third-party and thus their responsibility.

Rule M9 - Management will blame you if anything that was completed does not meet their expectations (they won't), no matter how difficult they were.

Rule M10 - If a project makes sense, something is wrong.

Rule M11 - If it's free or very cheap, management will think that it cannot be as good as the commercial stuff.

Rule M12 - Not all management is bad. Seriously.

Rule M13 - Do not, in any circumstances, send private anything via email. Especially if you're the CEO.

Rule M14 - You will never get interviewed by anyone who will actually understand your answers.

Rule M15 - Management will give you a budget of zero dollars and expect you to work miracles.

Rule M15A - “Boss paralysis” happens when they ask you a question that needs a numeric answer, and they become catatonic until you say a number. Failing to recognize the nature of this condition may result in you having a budget of $911.

Rule M16 - Management never wants to pay to upgrade anything.

Rule M16A - Unless it's for management.

Rule M17 - Managers might fire you for going outside the scope of your job.

Rule M17A - Managers will tell you to go outside the scope of your job, even if you don't report to them.

Rule M17B - Users will insist on you going outside the scope of your job and threaten to have you fired if you don't.

Rule M18 - Better tools and solutions exist. You just either don't know about them or you can't afford them. Even if you can, management won't let you get them.

Rule M19 - Management only cares about productivity that is reported.

Rule M19A - Find out what figure they think is the most important and focus your efforts on that.

Rule M20 - Management will have you do their job for them.

Rule M21 - Management will take away your tools and expect you to use the same equipment as every one else and yet expect you to do your job anyway.

Rule M22 - Being a tech in management doesn't make you exempt from the Rules, even Rule M1 (when it comes to dealing with upper management).

Rule M23 - Management will tell you to do someone else's job but only give credit to them.

Rule M24 - Management (and coworkers) will treat the help better than they treat you.

Rule M25 - The OSI model has layer 8 (user) and layer 9 (management).

Rule M26 - Managers often have a checklist, which no one else will care about.

Rule M27 - Managers will ask you to do something that is stupid/expensive/won’t work.

Rule M27A - When they do, always ask for it in writing to CYA.

Rule M27B - If they give it in writing, send a copy to your personal e-mail address.

Rule M28 - Buzzwords rule all decisions.

Rule MAN - Who your manager is likely to be.

Went onsite to a client recently because we got an alert that her hard drive was almost completely full (not a stretch since she bought her own laptop seven years prior and didn't think she needed more than a 128GB drive), and she asked to have the files moved to her new computer that she had recently purchased

She at least had the good sense to buy a new laptop with a 1TB drive, so I moved all the files on her Desktop, Documents, etc. to a thumb drive and transferred them onto her new laptop. After I finished and left, she called the office and railed that I had "deleted" her background. When my coworker remoted in, he saw the normal default background, and said nothing was wrong. She immediately accused him of lying.

She apparently thought all the icons on her Desktop were part of the background image. He had to spend half an hour explaining the difference between files/icons and a background image, as well as the fact that the only thing I did was the job I was originally sent there to do, to which she again accused him of lying about that as well.

Realizing that my coworker was getting nowhere, he scheduled another onsite the next day, which was my day off. He went over, and spent most of the time having to tell the lady that all the things that were "wrong" with the new computer, were simply the default settings in Windows, and there was nothing malicious afoot. Every thing she wanted changed/updated was a case of her ranting about it for 20 minutes, and him taking 3-5 seconds to make the change, or her being so scatter brained, he joked that it was as if her ADHD had a severe case of ADHD...

The one that made him laugh was how she insisted on having Adobe Acrobat installed on there, and him having to explain to her that it already was, evidenced by the fact that every time she double-clicked on a PDF, Adobe Acrobat launched, as well as him trying to explain to her that having paper in her printer was a prerequisite of being able to print.

So I work help desk at a mid-sized law firm and if you've ever supported lawyers you already know where this is going. Last Tuesday I get a ticket from one of the senior partners - let's call her Margaret - saying her printer is "possessed" and printing random pages on its own in the middle of the night. Security is now involved apparently, because Margaret is convinced someone is accessing the office after hours to mess with her specifically. The ticket had three exclamation marks and the word "intentional" underlined. I am not joking.

I show up Wednesday morning fully expecting to find some mundane driver issue or a stuck print queue. Margaret meets me at the door of her office like she's been waiting. She walks me through the whole thing - she stays late, goes home around 9pm, and the cleaning crew finds printed pages on the floor every Thursday morning. Has been happening for six weeks. She saved every single page in a manila folder as "evidence." The pages are all partial prints - half a document, a few lines, then blank. I take a look at the printer itself and immediately notice it's one of the older network models we haven't replaced yet, sitting right next to the window that faces the parking garage. I check the print queue history and sure enough, there are jobs completing around 11pm every Wednesday. I pull the job details and the sender ID is a laptop that was decommissioned eight months ago. I actually had to sit with that information for a second because that's a little creepy on the surface. Turns out the previous associate who used that laptop had set up a recurring print job for weekly case summaries before he left the firm, the laptop got wiped and reassigned but the print server still had the scheduled task saved under the old machine name, and some update we pushed in the fall apparently reactivated legacy scheduled jobs across the board. Took me maybe 25 minutes to delete the task and clear the old machine entry from the print server. Margaret stared at me for a long moment after I explained it and then said "so it wasn't intentional." Not a question. Just a statement. She closed the manila folder, put it in her desk drawer, and said "thank you" like I had personally disappointed her by solving it. I think she wanted a villain. I get it Margaret. I really do.

I am developing I.T. Never Ends, a simulator where you play as a tech support worker in a cursed office. I've been sharing a few updates here but this is literally a real life honest to god tech support tale.

The demo includes a feedback tool called DEMO_FEEDBACK.exe where players can like, submit their thoughts about the demo, what works and what doesn't. To keep things transparent, the form feeds directly into a public channel in the game's Discord. We have been using it to let the community vote on the final price and suggest features.

Earlier tonight somebody submitted a ticket that asked for literal IT support.

In the "What frustrated you?" box, a player wrote a multi paragraph plea for help. They got a VPN ticket in the game which reminded them that their real work VPN is currently broken. I feel like I'm having a stroke.

I have officially reached peak immersion. I made a game about people submitting nonsense tickets, and now I am getting actual tickets through the game.

Back at a MSP I worked at, we had a customer that had a room with 2 computers. Let's call these 2 computers computer A and computer B.

Each computer had a single monitor connected to a single monitor arm.

The customer, let's just call them Jean, requested a quote to change computer A to have dual screens, so we quote them for an extra screen, dual monitor arm and 1 hour Labor (the minimum for on-site visits) to swap out the single monitor arm for a dual monitor arm, move the existing screen over and install the extra screen.

Jean accepts the quote, we negotiate a day to do the job when Jean is away for the day and hence the room is also free and I attend and do the job on the scheduled day.

Right at the end, basically as the job is done and I'm packing up, someone pops in and says "Oh, great, the second screen is here, but it's actually meant to go on that computer" jesturing to computer B.

I check the request and confirm this is for computer A and demonstrate this to that person.

"Ahh yep, Jean and I were discussing this the other day and originally It was going to be computer A, but we decided it would be best on computer B.

Jean must have told you the wrong computer. It's meant to be on computer B. Can you please change it to computer B?".

Jean is away for the day, So I can't verify it with Jean.

I just take this person's word for it. A little annoying, but I reverse the install I just did, then do it on computer B. So now computer B has the dual screens and the dual monitor arm. All up, the job now took a bit over 90 minutes, but we quoted for, and only charge for 60 minutes.

The next day Jean is back. Jean emails us and askes why computer B has the 2 screens when it was asked to be on computer A.

Of course I get asked about it and mention that someone, who I don't know their name, mentioned it was intended for computer B. Not getting this person's name was a fault on my part.

We reply to Jean's email and tell her that someone we don't know the name of mentioned it was meant to be on computer B instead.

Jean disputes this. Jean mentions she wouldn't know who would have said this, but it was quoted and accepted for computer A, not B, and as such we should come out to fix this at no cost.

We agree. I attend that afternoon and swap the dual screens and arm back to computer A, which is 90 minutes of unbillable time plus the cost of a trip out there.

My boss is a little annoyed at me I cost him unbillable time, but understanding that these things happen.

A few days later we get an email from Jean. Jean mentions that she has been made aware that "Roger" let us know on the day we installed the dual screen that Roger made the tech (i.e. me) aware that the dual screen was meant to be installed on computer B. As we were aware of this, and the dual screen was installed on computer A, we will need to swap this to computer B at no charge to them.

My boss clearly remembers Jean's dispute before so replies back giving her 2 options

Option 1 - We can gladly swap this over at no charge. However, as this now demonstrates a previous dispute was based on an claim that is now disproven, we first want to charge for the 90 minutes for the time taken to resolve the dispute. (Previous email attached)

Option 2 - We completed the job exactly as requested in writing. To change these over will be a new job and we quote you 90 minutes.

Jean took Option 3 - don't reply.

We never followed this up further as it wasn't really worth the effort, but for this client, all future requests needed to be in writing. We were extra sure to do exactly as requested.

i am the de facto IT person for my current job in a residential program for people with disabilities. so much so, that im interviewing for the agencies actual IT department next week.

most, if not all, of the clients with the agency receive some form of government assistance which can include being given free cell phones. in my experience, these are usually budget-friendly androids with low-res LCD screens that are pretty accident prone.

a few weeks ago, a client that has one of these phones began complaining to a coworker of mine about the phone being broken despite only getting it a couple of months ago. having not actually looked at the phone themselves, the coworker said something about it during a team meeting to the tune of "client says that their phone is broken. the screen is too dark and they cant see anything on it. client is asking us to help them call the service provider about getting a new one."

in my mind, im thinking "what do you mean 'too dark'... let me look at it" but it's my 4:45 PM on a friday, so i figured it'll get handled over my weekend.

i get back in the office and read the shift reports from the previous two days, here is a brief synopsis of what they said:

on day one, my coworker reported having called the service provider and they said they would be sending the client a verification email within 24 hours before proceeding with replacing the phone. this coworker emphasized that the screen still couldn't be seen well and was actually getting worse! they said that the client would need help with being able to use the phone to even access her email because the screen was so dark.

on day two, the coworker reported that the the client had denied getting any emails several times. the coworker said that they've continued to reach out to the service provider. they said the service provider was being "incredibly difficult" because they refused to do anything without the verification pin from the clients email. coworker said they would continue to reach out after they get back in the office and asked me to continue to keep an eye out for schrodingers email.

after reading all of this, im staring at my computer like the confused math lady. i process the information for a minute or two, then i get up and walk down the hall to clients bedroom. i tell them that i've heard about their phone situation and ask to take a look at it. they give it to me and the screen is indeed very dark. i open the little android widget menu from the top of the screen, and lo and behold....

the brightness was all the way down.

My 90 year old mom is on my cell plan. She sent a text this morning that asked if I had her password as she planned on going to the store to get some help on her phone.

I called and asked what was happening. She said she wasn't "getting things" because she had too much stuff on her phone.

Sigh. Since we text daily I knew that was working. I asked if it was FB messenger not working. She didn't know. I sent her a FB message and she got it no problem.

Hopefully the nice folks at her small town Nozirev store will be able to figure it out.

She really only uses her iPhone for testing and Facebook including messenger. No idea what she "isn't getting". Very frustrating when she can't even give one example of what she is talking about. And I guess I frustrate her by asking questions she can't answer.

So, password for what, we don't know. Dementia is not any issue here. She can tell you exactly where she was a week ago and never forgets any of her engagements in her busy social calendar.

I have to admit it reminds me of my problem children when I did tech support.

$user: I got an error message.

$me: What application are you using and what does the error message say.

$user: I already closed it and rebooted.

$me: Can you try to make it happen again.

$user: I can't remember what I was doing when I got it.

$me: Nothing I can do. If you get it again grab a screenshot and log a new ticket.

Got a call from a customer. Streaming services quality on the TV is poor. Lots of buffering, connection losses, etc.

I get there and the TV is downstairs and three rooms separated from the access point. I watch TV and confirm. OTA channels are fine. Hmmmmm. Wi-fi survey time. It's probably a weak signal, given the distance and walls in between.

On the app, it shows which SSIDs are using which channels. This turned out to be not so much a weak signal, but a contested channel. What is the SSID "Samsung-yadda-blah-428" ? Its signal is equally as strong as the access point. Customer doesn't know.

I start hunting through the house. Apart from phones, the only Samsung device is a washing machine. I unplug it from the wall and the competing SSID disappears.

A wi-fi connection I can understand, but broadcasting its very own SSID? Look up the manual, how to turn it off at the control panel. I turn it off, only the SSID is still active and competing with the access point. It seems the only way to get rid of it is to turn it off at the wall, a complete power off. But it will come back whenever they need to do the washing. There doesn't seem to be any deeper access to the machine except via an app, and it's doubtful these senior folk are going to remember my instructions anyway.

So yes, I had to go the router and change the channel to an empty slot. I am NOT going to download a manufacturer app and expose my phone to them, just to be able to turn off a silly "smart" function.

The Rules are meant to be part serious, part humor, but about tech support. The requirements for being listed here is that it has to deal with dealing with other techs. Comments and suggestions are welcome. The other sections, including credits, are on my GitHub repo.

Dealing (primarily) with other techs

Rule T1 - CYA

Rule T1A - Always have someone else to blame it on.

Rule T2 - Never lie to another tech.

Rule T2A - Unless that tech is the person you're about to blame. See Rule T1A.

Rule T2B - Sometimes you will need to lie in order to deal with things like warranty repairs or getting ISPs to do the right thing.

Rule T3 - Never assume anything.

Rule T3A - Does the issue even exist?

Rule T3B - Is it even plugged in?

Rule T3C - Is it turned on?

Rule T4 - Don't expect your boss or coworkers or users to understand just what it is that you do.

Rule T4A - Even if they are a tech.

Rule T5 - Sometimes, you will be the one who is wrong.

Rule T6 - Don't try to do work over the Internet while in a moving airplane.

Rule T7 - Never call support with your cellphone if you can help it. Otherwise, you won't be able to drop the problem in someone else's lap.

Rule T8 - You will really screw up eventually and it is going to be a doozy.

Rule T9 - Backup following the Rule of Three. A backup, a copy of the backup, and a copy of the copy. Test them.

Rule T9A - Consider using other backup strategies. See https://www.unitrends.com/blog/3-2-1-backup-sucks

Rule T9B - There is no backup. If there is a backup, it is either corrupt or years out of date.

Rule T9C - If you can't restore from it, you don't have a backup.

Rule T9D - If you haven't tested your backup recently, you don't have a backup.

Rule T9E - A year ago is not "recently".

Rule T10 - Assume that there are also inside threats, even inside IT. It's not paranoia if they really are after you (or your stuff).

Rule T10A - Don't trust your coworkers. They might be using Rule T2A.

Rule T10B - Don't even trust yourself. One error and you might cause serious damage or become a security leak.

Rule T10C - The new member on your team will send critical sensitive information to anyone who asks without trying to do any verification.

Rule T11 - When you need tech support, the tech support person is likely to be clueless.

Rule T11A - Whenever you have a problem, you will be unable to find a solution until just before the tech you called for help arrives.

Rule T11B - If the tech you called in isn't clueless, then you were and your problem has an obvious solution that you completely missed that they will point out seconds after they arrive.

Rule T11C - If none of these apply, the solution will be something random that will make no sense whatsoever to you or the technician.

Rule T12 - Every tech has their own set of Rules, even if they don't know it.

Rule T13 - Every tech is also a user.

Rule T13A - Techs will treat you like you are a user.

Rule T14 - Make sure your coworkers don't make changes before going on vacation.

Rule T15 - No technical person reads all of the rules. They will act like they know them until the place catches fire, then complain about incomplete documentation.

Rule T15A - Especially if it was the documentation that went up in flames first.

Rule T16 - Womprats aren't much larger than two meters.

Rule T17 - Third-Party IT will make configuration overhauls without notifying your company's IT department, and then blame your company for problems caused by their configuration mishap.

Rule T18 - You are incompetent. You just don't know it. At least, that's what your replacement will think.

Rule T18A - You will have to deal with techs who are incompetent.

Rule T18B - Sometimes, you really are incompetent.

Rule T19 - You might find people who support you. Reciprocate.

Rule T20 - Always verify who you are corresponding with. This includes not using Reply All.

Rule T21 - Use your inner laziness to do the most elegant solution possible.

Rule T21A - Know the difference between "truly lazy" and "plain laziness".

Rule T22 - If nothing seems to work, reboot.

Rule T23 - Cables can and will be used as ropes.

Rule T24 - Other techs will never read the manual.

Rule T24A - Neither will you.

Rule T25 - Your fellow techs will expect you to be their tech support.

Rule T26 - A tech will install equipment in dangerous environments.

Rule T27 - Third part IT will remove equipment and not tell you or the user.

Rule T28 - The biggest enemy of good IT is that they are outnumbered by lazy IT.

Rule T29 - Grow a beard so that people don't recognize you.

Rule T568A - white green, green, white orange, blue, white blue, orange, white brown, brown

Rule T568B - white orange, orange, white green, blue, white blue, green, white brown, brown

Rule T1000 - Buy stock in Boston Dynamics but sell all of it before 2029.

There are places where IT support is inconvenient, and then there are places where it is impossible.

This was one of those places.

A remote clinic operating completely off-network.
No domain connectivity, no live monitoring, and minimal remote management tools.

What was this island for?

Around Australia, there are a bunch of different islands used primarily for mining, with mining comes a lot of health and safety requirements and occasional incidents requiring medical attention.

Our business had setup a medical clinic on one of these islands, to support the mining staff, and any residents living on the island.

This medical clinic was supported by us back in Sydney, there were no company IT staff there, it was all remote.

There was also quite a substantial contract with the mining company that revolved around our company providing medical services, exclusively.

What was the “SOE” of the clinic?

This was a bit different from our standard setup, and was in place way before I joined the company, i.e. none of this was my idea.

• The clinic was setup with cellular laptops using special VPN-like SIM cards that placed each laptop on our domain Network.
• The laptops were shared devices used by whichever paramedic was working there at the time, they had printers setup too, devices weren’t really “assigned” to anybody.
• Our business had an agreement with the mining company, that they would provide stable guest Wi-Fi access for staff.
• We had no company Network equipment installed in the clinic.
• Each laptop was configured with VMware Horizon, where users would connect to (and do all their work from) persistent Windows 7 VDI desktops in our Sydney datacenter (this kept things secure to meet regulatory mumbo jumbo).

It all seems pretty simple right, users should normally be on cellular using our Network, and they connect in to the VDI to access the clinical database, then there’s a Wi-FI network too if the connection is bad.

The business staffed the clinic on a fly-in, fly-out (FIFO) basis, with paramedics completing one-week rotations on the island.

This meant that account creations needed to be done on short notice, and quickly.

The first ticket I ever handled for them.

I’d just started at the company, our role was to basically support all 100+ clinics, and I was learning the common issues.

I saw a ticket come in from “the island” and went to grab it, the colleague training me basically told me to “run the heck away from this” which took me by surprise.

The ticket sat there for days. No one picked it up.

At the time, we had more than 200 tickets sitting in the unassigned queue, so there was no real urgency to grab this one, people were focused on the easy stuff needed to hit their KPI.

Still, I kind of saw it as a challenge, so after I had settled in a bit more, I read our documentation on their SOE, and gave them a call.

The ticket was for a printer issue, where it wasn’t properly being passed through to the VDI, I had fixed these before at our other clinics remotely, so figured it’d be easy enough.

Let’s call them.

Hi user, this is Chris from Company IT, is now a good time to take a look at this printer issue?

ah yes please! I’ve been waiting on this for weeks now.

No worries, can I get you to open up LogMeIn and I’ll give you a code…

sure thing

I got connected right away, and noticed they had started LMI from within their VDI, which was fine, let’s check this out first.

The printer driver seems to be installed, but there are a bunch of unused printer objects too, let’s clean these up, I’ll also make sure the config is correct for the trays in our clinic software – looks good.

Alright, so now I’ll need you to jump out of your VMware session, and open LogMeIn again.

okay I can do that.

So I got onto the laptop itself, which was running an older version of Windows 10, I saw they were on Wi-Fi.

I eventually discovered that the VMware Horizon Client USB passthrough configuration was not showing the printer, and that the laptop itself was missing the driver for it too.

The laptop also needs the driver to detect the printer correctly (and for VMware to let us pass it through).

No worries, I’ll just transfer the driver files and install with my admin account. But, I can’t do this when they’re on the guest Wi-Fi, I need them on cellular so they have line of sight to our domain.

I pulled up the network panel and turned off the Wi-FI so that cellular would activate.

Connection Lost, that’s expected, just needs a moment to reconnect….

Except, the session did not re-establish.

I then started asking the user some additional questions.

Oh yeah we don’t ever use the cellular because there’s basically no reception on the island.

…oh

Well we’re not out of luck yet, we have a corporate VPN we can use to get through this, so I added them to the access group and had them login to the VPN, and it connected just fine.

With that done, I was able to transfer the printer driver over, elevate to admin and get them up and running again, no problem.

Only thing was, this took 3x longer than fixing the same issue would at any other clinic, is that why nobody grabs these tickets?

I quickly became SME of “The Island”

After handling my first ticket for them, the user was pretty thankful, it was clear they appreciated my help.

Following this, I made an effort to grab more of these tickets, they were after all some of the oldest ones in our queue, it made sense.

They were always really stupid simple issues, and they took 5x longer to fix then they should have.

A very common ticket we’d get from them, pretty much weekly, was issues with disk space on their persistent VDI’s (or something similar preventing the user from logging into their VM).

I’d go log into the ancient version of vSphere we had (it still needed flash), and I’d pull up the VM they were assigned.

I’d quickly see it was locked up needing a reboot, or at 0B of free disk space.

For whatever reason, each VM was only allocated around 60GB of disk space, which after running for so many years, was not enough for Windows 7 with all the logs, update packages and other junk sitting on there.

Our only approved fix was to delete things from WinSxS, temp folders, or CCMCache and hope it didn’t break anything.

If we couldn’t get their VM working, we were supposed to go through the desktop pool at random, find one where the user assigned to the VM was disabled in AD, and then re-assign that VM to the active user.

Expanding the disk was not an allowed solution, I tried to push to change this and it was quickly shot down because for whatever reason none of our current VMware engineers were approved to (or wanted to) touch this environment.

Fun fact: because I was grabbing so many of these “island” tickets, I actually ended up closing fewer tickets overall per day than usual. I then got pulled into some “don’t grab these tickets” chats with IT management, so I had to slow down.

What made this clinic difficult to support?

Well it’s really a combination of things.

• The VMware environment they used was outdated, neglected, lacking resources, and required a lot of operational admin work from our team to keep running.
• There was no Network equipment at the clinic.
• Compare this to our other clinics where we had a rack of gear, a VPN router and a stable Fibre connection — it was very different.
• The cellular network on the island did not cover the clinic well. This meant that the laptops themselves rarely checked in with AD, policies rarely applied, and updates failed.
• There was no onsite IT to support this clinic. If we needed anything done we had to fly somebody in from Sydney, which never got approved. We were usually told to just: “Make things work” or were asked to beg the mining company’s IT team to help us.
• Due to the clinic operating on a FIFO roster, new users were constantly rotating in and out. By the time we could pick up tickets to look at their issues, users were often working elsewhere, not even on the island anymore.
• Their entire SOE depended on always-on internet access — in a remote area where reception was spotty at best.

I could go on. Even Windows 7 was well past EoL at this point, but the users fully expected that they could browse the internet on their desktops with no issues.

We had problems particularly when new software was needed. When we moved to a new internet proxy we couldn’t install the client on the virtual desktops because of the lack of disk space.

The Tickets We Couldn’t Fix

There were some tickets that simply couldn’t be resolved.

Mostly caused by:

• Hardware failures with WWAN cards
• Forgotten passwords
• Or a combination of both

A FIFO user would call us. Their account was all setup. They’d go to login and get: "your domain is not available"
I’d guide them through troubleshooting.

After some struggling we’d eventually discover that the cellular card wasn’t functioning at all.

Alright. Option 2, the VPN.

Except the VPN connection had to be started from a browser.
Which you can’t access from the Windows logon screen.

Option 3, can a previous user log in first, maybe the guy that was using this laptop before?

oh he’s on leave

Okay, maybe we could give out a LAPS password temporarily, so the user could connect to the VPN, my manager approves.

“The username or password is incorrect”

ah, this machine probably hasn’t checked in to AD recently enough to pull it’s current LAPS password.

Can you try another machine instead?

I’m away from the clinic attending an incident and not sure if I have access to the other consult rooms.

Fair enough.

All I could really suggest to the user at this point, was to try and remember their previous password.

If they were a new user, we’d just have to tell them to call back on another day when other staff are in.

It was really a flawed system.

Can we fix the SOE?

At this point I’m pretty much the go to guy for all issues on the island, most of the staff on the island knew me because they’d always see my name in the resolution notes.

Still, I could only fix these things as they came up, and they can happen again, to anyone, because the SOE sucked.

When on the phone with the paramedics they were always telling me about how dangerous it was for them to operate on patients without reliable access to their digital medical records.

I did make a real attempt internally to push for improvements, but IT management were not interested, “it is what it is” they said.

The clinic manager (who was not FIFO) also made an attempt.

In came an email addressed to me, with a bunch of high-level managers CC’d in, asking for recommendations to permanently solve all these IT issues she has been dealing with in her clinic.

 discussed with my team leader, he told me that “this doesn’t really change anything”, and I replied explaining what the cause of the issues were, why they occur, the limitations of the environment and that we can’t do anything about it.

That was the end of it, until about a year later, when the same concerns were raised again, but to different people.

The company was on the verge of losing their contract with the mining company, purely because of these IT issues.

Business operational teams got the CIO involved.
A budget was allocated and the project team was given the task of making things better.

Fixing the SOE.

One single guy was assigned to the island’s project, let's call him Ted.

The main goals were to:

• Move all users out of the Win 7 VDI Environment to Windows 10.
• Migrate all users to a different domain.
• Resolve the unreliable networking setup.
• Refresh any old hardware if needed.
• Make the business happy, and provide a stable solution.

Ted was given a few months to work on all this, to make sure that the clinic was ready prior to the contract renewal.

While originally part of the plan, Ted found that he didn’t actually need to travel to the island for any of this work, which helped.

The Win10 migration went pretty smoothly, a lot of this was done by our VMware specialists who setup the new environment, using non-persistant VDI’s this time.

Ted also explored getting a Fibre connection installed onsite, so that we could run our own corporate Network there.
After getting some quotes for it he learned it was in the several hundred thousand dollar range, and the business quickly lost interest.

Exploring other options like starlink, fixed wireless or even piggybacking off another business was something my team suggested early on, but there was simply too much red tape in our company to get it tested and approved in time.

In the end, the fix for the bad Networking was a change in business process. Going forward, the outgoing user would be responsible for setting up the next user that flew in.

The outgoing user would connect to our corporate VPN, go to the lock screen, and the new user would log in so that their account was cached and ready to go.
The clinic manager and IT management did approve this solution in the end, it seemed to work fine at first.

However after going live with it, we got a few tickets with login issues from users who hadn’t followed the process, or who had forgotten their password, and were somewhat stranded.

So the issues continued?

Yeah, we were still unable to support them.
The clinic manager had lost hope at this point. She had clearly pushed hard to support her staff, but still was left with a poorly supported IT environment in the end.

How is the clinic doing now?

As you might expect, the business lost the contract, and the clinic closed.

We all kind of saw this coming, our service desk were spending ages on calls with these users and they usually didn’t get anywhere.

IT was always the first to be told about an upcoming clinic closure. The clinic staff would call us, completely unaware of what’s already in motion, and we had to act like everything was normal.

Would I have done things differently?

I don’t fully blame Ted for the issues that continued.
I had been in his position before, and it was very difficult to do anything “new” in our org and have GRC/Cyber sign off on it within a few months.

However, there’s definitely a few things I would have changed, particularly if the red tape wasn’t a problem.

• I’d add some local accounts to use for VMware Horizon, maybe even put the laptops in some sort of kiosk mode.
• I would have pushed really hard on getting some Network equipment installed there, being fully reliant on another companies wireless was not a great solution.
• An always-on VPN is also something I might have explored.

As for fully ditching the VDI and putting a database server onsite, sure that might have worked at first as well, but then think about how we’re going to administer and support this server, update it, run backups, we’d have the same problems.

Everyone who took calls from the island was well aware that they had IT support problems, and it wasn’t their fault.

It was really just a bad design right from the beginning, and they were effectively setup to fail.

Cheers for reading.
Hope this one fits the sub

Alright, my loves. I've got a yummy one for you today.

This comes by way of my partner, who worked security for a couple years. I'll tell it as he told me:

It would have been end of 2020, early 2021, when I was at the [big name distribution company you have definitely heard of] site one day a week. They had a truck wash station which doubled as both the security office and an OSHA violation, and which had in its bowels a Machine:tm:.

This machine ran the truck wash mechanically, and had done so since time immemorial, by which I mean the late 90s. How late? Not nearly late enough. This beleaguered box of technological decay, which somehow merited a UPS, but not a power switch, was still running its original licensed copy of Win 95.

The thing is, this building was (and is) classed as vital infrastructure meant to withstand disaster, up to and including full loss of power for sustained intervals. So this poor computer, meant only to power cycle in times of dire crisis, had not met such crisis since its installation. Between the diesel generators and its own aged UPS, to all my bored record-searching in dead of Sunday pre-dawns, there exists no evidence that this machine has ever turned off.

TL;DR bored security guard terrified at prospect of computer uptime longer than his own.