I could literally write a book on this but I’ll try to convey the super short version here.

It was around 8’o clock at night and I was at one of the most beautiful places I have ever been – Lake Atitlán, Guatemala. It’s a really beautiful crater lake surrounded by 3 volcanic peaks and it was formed around 80,000 years ago when there was an eruption.

So i’m sitting at the common area with some people having drinks playing board games when I got a message that none of the videos in the app I was currently working on are playing. It’s an eLearning app so videos are an essential part of it. Plus this app is live with about 100,000 users. Basically, the app was close to unusable. So we had a mechanism in place where we could show a “temporarily unavailable” message in the app and we did that.

The culprit here was the storage – where we stored our videos and streamed them.

I’m not going to go into details what the issue was but the short version is that the company basically sold us out and gave our account to someone else. 21st century and this is a really huge organisation, about 70% of projects use it for storage. And we’re talking TeraBytes of data here. We couldn’t access our account because someone got a hold of them (on call) and changed all the login information, keys etc.

Now you would think it would be easy to get your account back. But hold on … not so easy …. they wanted us to prove that we are in fact the rightful owners of the account and they did nothing wrong and there is nothing they can do.

3 points I’d like to yell WTF !!!!!

Anyways, after about 8 hours of going back and forth with them we were finally able to get our account back. Do the needful so the videos start working in the platform etc. But now we have another challenge – getting the Apps submitted and approved as soon as possible in the Apple and Google Stores.

Luckily Google reviews apps fairly quickly, could be even just a couple of hours. For Apple there is something called “Expedited App Review” that you can request. You can only request it a handful number of times and ofc we need it. It took about a couple of hours for Apple to approve the update.

So about 14 hours later all was back to normal. But the lessons to take away from these are –

1. We developed our own storage servers and started moving all data to it. We no longer trusted them. Plus when you’re moving the data you have to pay something they call “migration fees”. And trust me that was ALOT.
2. Their support team didn’t really care about us. They wouldn’t listen and wouldn’t admit why it happened and that they were at fault.
3. Not to mention the 100s of helpdesk tickets that came in from frustrated users.
4. The whole team was there for 16 hours straight.

So basically our app was down, the whole team was working double shifts, had to develop another solution fairly quickly, frustrated users and more money spent.

I’ll end this by a vision of the great Steve Jobs – “If you’re an organisation that makes software, to succeed you will have to make your own hardware as well.”