October 31st seems like the perfect time to ask you about moments that gave you shivers.
Please share. Anything from the customer who said “What do you mean 'Where’s your backup?” after you told them their server disk had fried, to the co-workers pulling weird pranks on you.
1 Like
I was starting to develop in FM, didn’t really understand what duplication of tables in the graph did or adding links and removing them caused and FM native warning messages aren’t the most helpful. My boss had just given me full access and I was tinkering on the production files. I was “testing” (probing, not testing in the sense we use it in development) to try to understand what did what. I duplicated a table and linked stuff to it. It didn’t do what I thought it was doing so I deleted the links or table but probably the wrong TOC or links.
Back to the records…they were gone. I was about to pass out! I went to see my boss, flushed and mortified to asked if we had backups. He was very calm. Thank goodness one thing was set up right with our solution: a very good backup schedule. We still lost that afternoon’s work but it could have been much worst. From then on, I learned to do my probing on copies and imports early in the morning so that we could revert before the others arrived if I sc****up.
This summer a cloud backup provider had a glitch in the new client software that effectively bricked encrypted back-up files. As the concerned customer had recently moved office to a new building and an all-new infrastructure was set up, local backup was not in place at the time - which was very quickly amended.
Watching all newly created cloud (off-site) backups going brick at once was a spooky cloud experience and reminded me of the true amount of control we really have when cloudy things go awry. 
First day with a new client can be daunting. Deleting records is the scariest. One day I was sent out to a client for the first time. I was accessing a database across a very bad network. It was notoriously bad, so we used VNC to login to the machine rather than use FMP as a remote client. On this day the screen display was not refreshing accurately and the cursor wasn’t aligned with the mouse position. So I would click and nothing would happen. I would type and nothing would happen. The cursor would slide away across the screen without me moving the mouse and I would shake the mouse and try to move it back. This went on for about five minutes then suddenly the screen burst alive with movement. All the mouse clicks and keystrokes were suddenly being performed, but at random positions on the screen. I watched as a dialog popped up saying “Permanently Delete all Records?” then saw the focus shift to the “Delete All” button and the dialog disappeared. Seconds later I was staring at blank fields and the status bar showing that there were no records left in the table. I had just deleted their entire contacts table and had no idea whether they had backups or how to access them.
What, your going to end it there? You need to tell us the rest of the story! Were you able to retrieve the records? This is so spooky, I’ll have nightmares far a week, and won’t ever get close to a vnc connection again.
How did it end? While I was staring at the screen in horror my point-of-contact walked over and asked how everything was going. I steeled myself for the worst and said that I wanted to ensure that their backup routines were OK and that I wanted to do that before I got on with the rest of the job. Apparently the backup routines were working perfectly. I told her that I wanted to be able to run a couple of checks on them and asked if she would show me which drive they were stored on. So the story ends happily. While I was in transit back to the office my boss had called the client. They’d told him how pleased they were, not only had I done the job they needed but I’d also tested their backup systems and verified the backups manually.
4 Likes
Omg.
What does bricked mean in this context?
Bricked here means corrupted beyond recovery.
2 years ago, a customer called for help who did not call us for years…
I went to that location and found a blue G3 Mac with classic os 8, FileMaker 3 Server, USB1, etc. There was a firewire disk connected, probably for some backups
Since I had to change something in the structure, I asked for a backup. ‘backup? wadda?’
This Mac was running 24/7 for YEARS, they restarted that machine one time as they moved to a new location
The machine did not show any plugged usb-devices, needs a restart (was somewhat common at that time) - but who wants to restart a machine like that?
I informed the boss of that shop, telling him that we need to restart the Mac, unsure if everything will be working again
He was cool, he just unpowered the Mac, power-on again. Restarting took forever, usb was working again after that, copying a backup took forever as well (usb1, quite big db files…)
I asked them to change that situation, urgent! Did not hear anything from them during the last 2 years. Assuming that there is a wonderful blue G3 running still 24/7
1 Like
This story is not for the faint-hearted…
1 Like
This approach to IT is not uncommon in SME management. Use it until it dies, don’t spend any time and/or money on it. It is hard making it a talking point. I observed that actual IT accidents are the most powerful mind-changers.
2 Likes
well… that system was really internet-savy. No way to get into from outside (anymore). Not connected to any WAN
If that system would die, the company would have serious problems because the accounting/billing/etc. goes via filemaker
2 Likes
A customer had 2 Runtimes on his Mac, each for a different company with different customers, invoices and so on…
I agreed to take care of the update process for him via teamviewer. Unfortunately I somehow managed to import the first company into the second one, deleting all of his first company data. The regular backup of my solution had been overwritten by myself(!) with the new data as part of the update process (don’t ask!)…
That moment, when we both found out that he didn’t have a Time Machine backup was the most horrific in my entire career as a developer. I was more than lucky that he wasn’t upset at all. I eventually decided to waive my fee for the support… 
3 Likes
@Bobino
Great topic btw 
1 Like
Can’t forget a weekend Novell NetWare upgrade project in the 1990s where we were relying on an employee for most of the knowledge. 3 of us started the project on a Friday night, we ensured we had 2 x tape and 2 (new) disk backups.
We commenced the basic upgrade project, but upon first data restoration, discovered not 1, but both new hard disks were faulty (the days of SCSI and SCSI terminators). Reverting to the tape backups (now into Saturday morning), we were just about finished when there was a power failure, which corrupted the tape and the NetWare installation, the UPS had failed as well.
We now had a single tape backup left and by now the NetWare knowledge ran out - of energy, his head hit the table and that was him done (for the rest of the project, as he’d gradually panicked himself into a negative resource by now).
This left 2 of us and a NetWare manual and eventually 3 days without sleep, as we’d not recovered the situation (or finished the relevant pages in the manual) by Monday. However, by Tuesday we did have a working server back up and running.
Another memory was when we (me and Val) were members of the team setting up the 100 Mac installation (or large installation, it may have been the year after the 100 Mac install) for the public to use at the UK MacWorld exhibition at Olympia in London; a chance for the general public to view and try all the latest hardware and software (days before the Internet). Val was very pregnant with Kirsty our first child and she wore a ‘I’m not fat I’m pregnant’ badge throughout the show. The rest of us were wearing ‘Windows 98 equals Macintosh 95’ badges (think that’s what they said).
We were gifted ‘Frog’ (again SCSI) external hard disks, and we’d built up golden masters so that all Macs could run off these external drives and all have identical software available, which could be reverted to standard if needed. However, the Frogs arrived late and were far from being princes as they failed at an alarming rate. We had tens of these things mirroring and failing throughout the night. I believe we’d just about got most working after sunrise, went back for a shower and breakfast then did the first day manning the stand at the show. One of a number of stories from various Mac shows from Olympia, including networking the whole exhibition area in the early days of WiFi complete with a networking control room, got legless on the last night and turned up (late) to find the whole exhibition centre just about cleared except for our network room with all the cables falling from the ceiling into it.
I’m very glad to be doing FileMaker development and building/maintaining cloud virtual servers these days!
3 Likes
in 2009 or 2010, we were lucky to aquire a quite big project that needed 3 of us working together on that project.
For setting up that project, I prepared a Mac Mini with fms and NAT, so we could all do our job from different locations.
That Mini was not the newest, but at that time, this was the only server available. We could start the project in october, going live had to be per 1.1.
while working, a disk began to be noisy and I checked that Mini (it had an external firewire disk attached for backups, a new disk), it hat some problems.
I created a backup and saved it on the firewire disk, copied that one on iDisk, just to be double save - and reformatted the Mini’s disk.
Just ready with that, our cat dropped in (the ‘rack’ was just aside my desk) and jumped on the desk, snooked toward the Mini and started tapping - on the firewire disk, not the Mini (hear ears are way better…).
I tried checking that disk… it was that disk that made the noise, the Mini was fine (disk-utility always finds something…)
That disk died…
Was never again so happy to save data on a third media…
(sounds more dramatic than it was - I had the backup intervall on 1 hour on the internal disk of the mini and every couple of hours on iDisk per chronosync, but anyway…)
1 Like
I want a fixit-cat 
2 Likes
A very instructive tale about Murphy’s law. Glad that @AndyHibbs and team survived.
1 Like
btw. She also creates perfect passwords…
…when walking across my keyboard 
5 Likes