Last updated 26/02.
While some features have been/were impacted, tracking is not affected. All tracking on the Awin network is working correctly, no data was lost, and all sales have been recorded.
Our engineering teams are working hard to resume normal service across all features and we expect everything to be resolved by this coming Monday. The teams are also working on fixing the root cause of the issue, as well as taking steps to prevent the infrastructure issues reoccurring.
The infrastructure issues, the cause of which we’re still investigating, caused a delay in reporting by blocking systems being able to process transactions. Following this, there was an additional infrastructure issue where data from our master database was not being replicated to other databases. This meant features that relied on getting data from these other databases were not able to access up-to-date data.
The following services were impacted:
Over the weekend, transaction reporting on the Awin platform was delayed up to three days. All reports apart from the click ref and product performance report on the Awin platform are now up-to-date. While we are experiencing some technical difficulties in displaying data within click ref report, no data has been lost. Publishers are still able to get click ref data from individual transactions from the transaction report export. We estimate that the click ref report will be fully functioning again by the end of this week.
Additionally, there was an extended delay in reporting of cross device transactions. This has now been resolved. Advertiser and publishers are able to view these transactions within the transactions report.
Since Monday there was a delay on sending out transaction notifications. We have since processed this backlog and normal service has resumed.
Short links created on Monday and Tuesday were redirecting to a blank page. The creation of new shortened links both in the UI link builder and the MyAwin chrome plugin have since been disabled. We've made further fixes to any links created within this period so they are now functioning correctly again. We will re-enable the short-link functionality once the master database issue has been resolved.
Today's payment run was successfully completed with no issues. We also expect tomorrow's scheduled invoice run to be completed with no issues. Reporting on programme balances are currently affected and may display out of date data, we are working to correct this.
We apologise for any inconvenience this has caused to clients during this period, and that issues weren't communicated to clients in a timely manner. We have reiterated our crisis communication plan to ensure that in future any impact on services are communicated as quickly as possible. If you have any questions or feedback, feel free to get in touch with your account representative or contact us here.
We experienced another issue with transaction reporting in the evening on Sunday 18th. This was caused by an automated pause in our transaction processing system. While our engineers continue to investigate the root cause of the trigger, they are working on restarting the processing system and reprocessing all transactions that have taken place between Sunday evening and now.
As a result, reporting will not be up to date until we have successfully reprocessed this data. As a side effect, you may also see a duplication of click data in your reports. This is temporary, our engineers are aware of this and any duplicate records will be removed once the process is complete. We aim to have our systems up to speed again as quickly as possible and would like to apologise for any inconvenience caused during this period.
We are currently experiencing delays with our transaction reporting across the interface, with incorrect data showing from around 10th February onwards. Click ref reporting and product performance reports are also showing no data from 6th February onwards.
Our engineers have worked through the night and continue to work on bringing our databases back up to speed as quickly as possible, we will continue to post further updates as to when this looks to be rectified and we should have a clearer view on this by tomorrow (Wednesday 21st). We are also aware there is a duplication of click tracking which will be causing further incorrect data to show, however, our engineers are aware of this and will rectify the figures as soon as transactions are up to date and the databases stable again. We can confirm tracking is not affected by these processing delays.
Our team continue to investigate the root cause for these issues and we would like to apologise for the ongoing inconvenience this has caused.
We have made considerable progress over the last 24 hours to bring our databases back up to speed however there will remain some discrepancies within our transaction reporting until all systems have been fully recovered and stabilised. This may cause some fluctuations to be seen in the data shown in reports during this period however we can confirm tracking is not affected by these processing delays and no data has been lost. Click ref reporting and product performance reports continue to show no data from 6th February onwards, however our engineers are working to fix this as soon as possible.
We anticipate that the majority of reports will return to normal by Friday afternoon (23rd February).
Our team continue to investigate the root cause for these issues and we would like to apologise for the ongoing inconvenience this has caused. If you have any further queries or concerns, please get in touch with firstname.lastname@example.org.
Update 22/02/18 :
Our systems continue to bring themselves back up to speed and our databases have made significant progress. Transactions are now fully caught up however there will remain some discrepancies in reporting until all databases have returned to normal function. Our engineers are working on a fix for click ref reporting to be back up and running as soon as possible, alongside active deduplication of click data from 1st – 20th February. We anticipate that the majority of reporting will return to normal by Friday afternoon, and click ref reporting to return to normal by Monday.
Our engineers continue to work through deduplication of click data and we estimate this to be restored to normal early next week. Transaction processing also continues to update in reporting however we expect to see some discrepancies until all databases have returned to normal function. Our engineers are working on restoration of the click ref report as soon as possible however tracking has remained unaffected and no data has been lost. We anticipate the majority of reporting to be resolved this afternoon. We would like to apologise for the ongoing inconvenience and frustration caused during this period and thank you for your patience.
Transactional reporting and click ref reporting are now up to date and running as normal. Product performance reporting is not currently showing data from 21st February onwards, however this should return to normal later today. Tracking has been unaffected during this period however our engineers are continuing to double check data today. Duplicate click data has been removed and conversion rates and volumes should also have returned to normal. We would like to apologise for the prolonged inconvenience this has caused and thank you for your patience during this period.
Q. What is the date range that is currently affected?
A. We are currently seeing reporting delays from 10th February onwards for transaction reports, click ref and product performance reports have no data showing from 6th February onwards.
Q. Which reports are behind and/or contain incorrect figures?
A. All reporting is currently delayed. Click reference and product performance reports are not currently functional for data after the 6th February.
Q. What is the ETA on all issues and reports being fixed?
A. We are working to rectify the issues as quickly as possible and anticipate that the majority of reports will be working as normal by Friday afternoon (23rd February).
Q. Why is my conversion rate so low?
A. At the moment we are aware that clicks are overreporting in the interface, and with transactions still catching up to our processing systems this has resulted in an abnormally low conversion rate for many partners. Rest assured we are working around the clock with our engineering teams to remove the additional click data, and once transactions are up to date, conversion rates should return to normal.
Q. Sales still look low, are you sure you've tracked all sales?
A. Sales tracking is run through a different system, and this was unaffected by the issues we've seen over the past 2 weeks. Only our reporting databases have been affected, however these are now catching up and once back to normal, will display all sales from the period.