On 9 December 2019 at 15:58 (CET, UTC+1) both Recras servers were hit by a major failure. Recras runs synchronously on two servers at two physically different locations. If one server does not work or is overloaded, the other takes over. The major failure, however, concerned data corruption on both servers.
Backups and small malfunctions during the day
Every night all of Recras is backed up. In addition, all changes (everything that is changed or added in all single Recras applications) are tracked per server. In the event of a malfunction, such a log can normally be used to ensure that no data is lost. However, during the day there had already been a number of small malfunctions that could be solved by restarting the server. Because of the restarts on both database servers these logs were no longer correct. Our hosting partner Osso has tried to restart the database servers in several ways and restore the corrupted data, but unfortunately without success. Therefore, in consultation with Osso we decided to restore the backup that was made on 9 december 2019 at 01:19 AM.
Between what times did the data in Recras get lost?
All data entered in Recras between 01:19 and 15:58 CET on 9 December 2019 has been lost.
How can you restore it?
What have you lost?
First of all it is good to realise that every action that happened in Recras on December 9th between 01:19 and 15:58 has been lost. Here are a few things you need to take into account.
- New requests/bookings (from your website)
- Invoices, payments or quotations sent or generated
- Automatically sent mails (which are sent again the next day) are also reminders.
- Changes in settings, such as products, materials, start moments, PDF templates and booking proposals.
- Scheduled and registered hours
Let all users (staff) know that they have to repeat all the actions they did on 9 December between 01:19 and 15:58 in Recras.
How can you find out what you sent?
If you use Office 365 and use this in Recras, all sent mails can be found under “Sent mails” in the account that you have set as SMTP user under Company settings.
Possibly, this is also the case for other mail providers.
BCC email address
If you’re lucky, you’d have set up a BBC e-mail address. You can do this under Settings → Other → Other settings -> BCC address.
If you have set an e-mail address here, you can go through the following steps:
- Go to the mailbox that is set as BCC address.
- View all mails sent between 01:19 and 15:58 on December 9.
- Check them one by one and make sure all the information of the mail, booking and contact details are correct in your Recras. If necessary add them manually.
- If these include invoices, you must email the customer that the invoice will be resent, in order to keep your billing correct in your Recras. They will probably get a different invoice number.
- After that, recreate the invoice at the customer in Recras and if it was an online booking that has already been paid with Mollie, you also have to manually add the payment.
If you have not set up a BCC email address
- Look in Mollie for online payments made between 01:19 and 15:58 on 9 December 2019.
- If this is not the case, no online bookings with payment have been made.
- If this is the case, the description in Mollie will say for example ‘Booking reference 55-4’, search in Recras for the booking number. If this does not exist, there is a good chance that the booking has disappeared. But your customer has received a confirmation with an invoice. The name of the customer is in Mollie, so we advise to create a customer for that in your Recras with a booking of that amount.
- Look in your mailbox (possibly in your archived mails) and see / figure out which mails you had processed on Monday 9 December and answered from Recras. You have to process all these mails again in Recras.
- Look in the history of your phone. See who you called on December 9th and try to trace the phone calls and if necessary make the right bookings / changes in Recras.
- If you have an integration with accounting software you may be able to find the invoices in your accounting package. These invoices will have to be created again in Recras. There is a big chance they will get a different invoice number.
Questions and follow-up
We find it really annoying this was able to happen. We will research today (Tuesday 10 December) if we can recover important data from our logs. We will get back to that.
In addition we are researching the cause of the malfunction extensively and will take measures to lower the chance of such a malfunction in the future.
We will also come with recommendations to reduce the impact of such a malfunction.
If you have any questions, don’t hesitate to contact us.