Visa hires external party to conduct review of latest system failure, replies to UK Treasury Committee

Charlotte Hogg, Chief Executive Office (Europe) of Visa, has replied to questions posed by Rt Hon. Nicky Morgan MP, Chair of the Treasury Committee, regarding the system failure from June 1, 2018, which saw 5.2 million transactions fail to process correctly. Let’s recall that Nicky Morgan has asked Ms Hogg to provide an answer to 10 questions about the incident.

The answers, made public earlier today, provide some assessment of the disruption scale. Overall, for cards issued both in the UK and elsewhere, during the period of the service disruption, 51.2 million Visa transactions were initiated and sent to Visa’s European systems for processing. Of these, 5.2 million (10%) failed to process correctly. In the UK, during the period of disruption (about 10 hours), a total of 16 million cards were used and around 1.7 million were impacted (10.4%). During the incident window, there were 27.6 million Visa transactions initiated on UK-issued cards and sent to Visa for processing. Of these, 25.2 million transactions were processed properly (91 %) and 2.4 million (9%) failed to process.

Visa also elaborated on the “hardware failure” that allegedly cause the problems.

The incident was caused by the failure of a switch in one of Visa’s data centres, says Ms Hogg.

“We do not yet understand precisely why the switch failed at the time it did, but we are working with the manufacturer to conduct a forensic analysis of the switch”, Charlotte Hogg, Chief Executive Office (Europe) of Visa.

In particular, Visa processes payments in Europe in two data centres in the UK. The system works as an “active-active” service so both sites are continually processing transactions. The two sites are designed to continuously synchronise with one another, so that if one site encounters issues, transactions are routed to the other for processing.

In this instance, the switch being used in our primary data centre experienced a very rare, partial failure, which impacted the secondary site and prevented it from automatically processing all transactions, as it should have. As a result, it took far longer than expected to isolate the primary site; in the interim, the malfunctioning system at the primary data centre continued to try to synchronise messages with the secondary site. This created a backlog of messages at the secondary data centre, which, in turn, slowed down that site’s ability to process incoming transactions.

It took until approximately 19:10 to fully deactivate the system which was causing the transaction failures at the primary data centre. By that time, the secondary site began processing almost all transactions normally. The impact was largely resolved by 20:15, and we were processing at normal service levels in both data centres by Saturday morning at 00:45, and have been since that time.

Among the measures that Visa aims to implement after the incident is hiring an expert to conduct an external review. Visa has engaged an external party, EY, to conduct an independent review of agreed aspects of the incident. The Terms of Reference have been reviewed by our UK regulators. This will include: o Review of the root-cause analysis; o Evaluation of the leading factors in the incident; o Review of the effectiveness of Visa Europe’s disaster recovery, its response to the incident and its communications with stakeholders.

The Treasury Committee said it was satisfied with Visa’s answers regarding its system failure and it expects to see the findings of the independent review, which will examine the lessons to be learned from the incident, in full.