Visa hires external party to conduct review of latest system failure, replies to UK Treasury Committee

Maria Nikolova

EY will conduct a review of the root-cause analysis, major factors of the incident and the effectiveness of Visa’s disaster recovery.

Charlotte Hogg, Chief Executive Office (Europe) of Visa, has replied to questions posed by Rt Hon. Nicky Morgan MP, Chair of the Treasury Committee, regarding the system failure from June 1, 2018, which saw 5.2 million transactions fail to process correctly. Let’s recall that Nicky Morgan has asked Ms Hogg to provide an answer to 10 questions about the incident.

The answers, made public earlier today, provide some assessment of the disruption scale. Overall, for cards issued both in the UK and elsewhere, during the period of the service disruption, 51.2 million Visa transactions were initiated and sent to Visa’s European systems for processing. Of these, 5.2 million (10%) failed to process correctly. In the UK, during the period of disruption (about 10 hours), a total of 16 million cards were used and around 1.7 million were impacted (10.4%). During the incident window, there were 27.6 million Visa transactions initiated on UK-issued cards and sent to Visa for processing. Of these, 25.2 million transactions were processed properly (91 %) and 2.4 million (9%) failed to process.

Visa also elaborated on the “hardware failure” that allegedly cause the problems.

The incident was caused by the failure of a switch in one of Visa’s data centres, says Ms Hogg.

“We do not yet understand precisely why the switch failed at the time it did, but we are working with the manufacturer to conduct a forensic analysis of the switch”, Charlotte Hogg, Chief Executive Office (Europe) of Visa.

In particular, Visa processes payments in Europe in two data centres in the UK. The system works as an “active-active” service so both sites are continually processing transactions. The two sites are designed to continuously synchronise with one another, so that if one site encounters issues, transactions are routed to the other for processing.

In this instance, the switch being used in our primary data centre experienced a very rare, partial failure, which impacted the secondary site and prevented it from automatically processing all transactions, as it should have. As a result, it took far longer than expected to isolate the primary site; in the interim, the malfunctioning system at the primary data centre continued to try to synchronise messages with the secondary site. This created a backlog of messages at the secondary data centre, which, in turn, slowed down that site’s ability to process incoming transactions.

It took until approximately 19:10 to fully deactivate the system which was causing the transaction failures at the primary data centre. By that time, the secondary site began processing almost all transactions normally. The impact was largely resolved by 20:15, and we were processing at normal service levels in both data centres by Saturday morning at 00:45, and have been since that time.

Among the measures that Visa aims to implement after the incident is hiring an expert to conduct an external review. Visa has engaged an external party, EY, to conduct an independent review of agreed aspects of the incident. The Terms of Reference have been reviewed by our UK regulators. This will include: o Review of the root-cause analysis; o Evaluation of the leading factors in the incident; o Review of the effectiveness of Visa Europe’s disaster recovery, its response to the incident and its communications with stakeholders.

The Treasury Committee said it was satisfied with Visa’s answers regarding its system failure and it expects to see the findings of the independent review, which will examine the lessons to be learned from the incident, in full.

Read this next

Retail FX

X Open Hub expands multi-asset offering with new cryptocurrencies and indices

X Open Hub, a multi-asset liquidity provider, announced today that it has extended its offering with inclusion of 30 new cryptocurrencies and two emerging market indices.

Retail FX

Interactive Brokers now offers 24/7 access to crypto trading

Interactive Brokers Group, Inc. (NASDAQ: IBKR) has widened access to cryptocurrency trading for its clients. Specifically, the discount broker extended trading hours for its cryptocurrency products to be traded 24/7, including on the weekends.

Inside View

Natural Language Generation for Multi-Language Social Media Strategies 

Natural Language Generation (NLG) is a crucial growth area in the digital landscape, with the unique potential to be used across multiple industries.

Market News, Technology

The B2Core Android App is Now Available For Download

The first version of the B2Core Android app is available for download, and it has many beneficial features for users.

Digital Assets

XBO taps custodian Fireblocks ahead of crypto exchange’s launch

XBO.com has integrated with digital asset and crypto technology platform Fireblocks ahead of the crypto exchange’s upcoming launch.

Industry News

Appital raises another £1.7m after integrating with FactSet, FlexTrade, and TS Imagine

“Appital has a proven track record in securing partnerships with major EMS providers, executing brokers and a trading venue to address the lack of efficiency and control over existing bookbuilding and deal distribution processes.”

Digital Assets

Tim Draper backs crypto-focused asset manager OSD in seed round

“The Old Street Digital team is well-positioned to provide access to a range of research-backed, crypto-based investment funds, and we at Draper Associates are excited to support them.”

Industry News

Swissquote launches mobile payments app TWINT

Swissquote has announced the launch of its own TWINT mobile payment app, effective today.

Institutional FX

Apex Group subsidiary EDB launches institutional banking solution

EDB’s solution enables clients to open multi-currency accounts and manage their day-to-day transactions, including global payments, FX requirements, and cash management through automated money market sweeps from a single online dashboard.

<