Sorry! We’re having technical issues *update* Announcements

Sorry! We’re having technical issues *update*

Our hosting provider, Amazon Web Services, suffers currently severe technical issues in one of their data centers (see this list of startups with connectivity issues). Due to these issues we’re unfortunately not able to access your most recent test data at the moment. Amazon’s engineers are working hard to solve this problem. In the meanwhile we’re preparing to kickstart our servers again when we regain access to our data again.

Follow us on Twitter for updates or check this blogpost for updates on the Amazon outage.
Contact us if you got any questions: support@usabilla.com

Update: 0:30 AM PST
We’re currently working on a workaround. The last currently available snapshot of our data is from yesterday 3:00 AM EST. Our servers went offline at 10:30 AM EST. We’re going to restore our servers with the last available snapshot. That means that most of your data will be available any time soon. We trust that Amazon will be able to recover access to our most recent data as well. When we regain access, we will switch back to this data and merge any updates you make today. In short: we’re recovering and trying to get everything up and running as soon as possible, in the meanwhile we’re making very sure that you will not loose any of your data.

Update: 1:15 AM PST
We’re testing our workaround setup at the moment. When we pass these tests, we can enable access again. This would mean that you can continue your current test cases and use Usabilla as you would normally do. The only drawback is that you’re not able to access 7 hours of yesterday’s data yet. We will import this data when Amazon makes it’s available again.

Update: 1:35 AM PST
Our workaround is available! We’re back up again. Everything should be working like it supposed to be. Unfortunately yesterday’s data (3:00AM – 10:30AM EST) is not available yet. We will import this data as soon as it’s available again.

Update: 8:25 AM PST
Yes! We’ve been able to retreived the missing data from yesterday between 3:00 AM and 10:30 AM EST. We’re now working on a clean import.

Update: 1:55 PM PST
Everything currently works like a charm. We’re preparing an insert of  yesterday’s data. We’re taking our time for this, since we don’t want to schedule any downtime. We plan to insert the 6 hours of unavailable data during the weekend.

Update: April 26, 7:00 AM PST
Rik made good progress on a script to import the missing data. Since we want to be extremely careful not to mix up or corrupt any data we’re testing this update extensively. Tonight we’ll be doing some test runs on our development server. If this works as planned we can insert the missing data tonight at around 1:00 AM PST.

Update: April 27, 6:00 AM PST
Unfortunately we haven’t been able to run the update yet. We ran a couple of test batches and are getting closer to the final import. Please stay tuned a little longer…

Read more about how we recovered and will improve our infrastructure to prevent future outages.

Leave a Reply

Your email address will not be published. Required fields are marked *