Senior management to be responsible for processes based on big data and AI, German regulator says

Maria Nikolova

It is the responsibility of supervised firms to guarantee the explainability of BDAI-based decisions, BaFin says.

There will be no black box excuses for the management of financial services companies in Germany, in case they want to shift the responsibility for a decision involving big data and artificial intelligence (BDAI) to machines rather than human beings, the Federal Financial Supervisory Authority of Germany stated in a report published earlier today.

The report challenges the traditional approach to algorithms as black boxes. BaFin insists that senior management remains responsible for BDAI-based processes and decisions.

BaFin agrees that BDAI will create additional opportunities for automating standard market processes – from chatbots to liquidity management. Examples include the applications of automation for the lending business at banks or for claims management at insurance companies. Further automation is aimed at increasing efficiency and effectiveness – and thus cutting costs.

However, the regulator notes that this does not mean that responsibility for the results of BDAI-supported processes should be shifted to machines. When designing (partially) automated processes, it is therefore important to ensure that they are embedded in an effective, appropriate and proper business organisation.

Ultimately, responsibility for automated processes has to remain with the senior management of the supervised firm, says BaFin.

According to the regulator, supervised firms are the ones that have to guarantee the explainability/traceability of BDAI-based decisions. Supervisory and regulatory authorities will not accept any models presented as an unexplainable black box.

In the case of black boxes a user has no direct possibility of finding out why or how the algorithm has made its decision and thus produced the result. This is mainly due to the fact that the input and output values of neural networks are linked in a very non-linear and complex way.

According to the regulator, a better understanding of models provides an opportunity to improve the analysis process – for example, the users are then in a position to recognise problems such as overfitting and data bias.

As far as the intelligibility of AI systems is concerned, a distinction is made between transparency and explainability. Transparency describes the behaviour of the system being completely comprehensible. But this requirement can almost never be fulfilled since many models are necessarily complex. Put otherwise, a black box can hardly be made transparent.

In contrast, explainability describes being able to list the major factors of influence for a concrete individual decision. This is much easier to fulfill in terms of technology.

There are two main approaches for the creation of “transparent” systems:

  • An attempt can be made to generate comprehensible approximations for any models. This approximation should be characterised by being able to reproduce the behaviour of the original model with as little deviation as possible, despite being able to be depicted in a way that humans can understand. The TREPAN algorithm is an example of such an approach.
  • There are also approaches for the generation of directly comprehensible models from data, such as decision trees or subgroups. Such models are not always ideal for the learning task, yet if transparency is more important than the quality of the result, these methods are more appropriate.

The generation of explanations is currently a fast-developing field of research in machine learning. One of the most famous approaches is the LIME algorithm, which uses simpler methods to form a local explanation model for the individual case to be explained and similar data points. Other approaches are prototype-based and deliver representative individual cases to explain decisions.

The German regulator stresses that new approaches could provide at least some insight into how the models work and the reasons for decisions, even with very complex models. This is set to prevent models from being seen and treated purely as black boxes.

A consultation has been open into the detailed report which raises many important questions around data mining, privacy, and innovation. Feedback is expected not later than September 30, 2018.

Read this next

Retail FX

Eurotrader taps MahiMarkets’ cross-asset trading technology for pricing and risk management

Eurotrader has tapped MahiMarkets’ cross-asset trading technology to manage its pricing and risk management on its fully licensed and regulated retail brokerage operation covering forex, stocks, indices, and commodities. 


STT doubles user base as Sterling OMS adopted by TradeZero, Avenue, Banco C6

Sterling Trading Tech (STT) has announced record growth of its order management system (OMS) offering, doubling its user base throughout the second half of 2022.

Retail FX

CMC Invest announces flexible stocks and shares ISAs amid changes in UK policy

In addition to a flexible ISA, the Plus plan provides members with a USD currency wallet for their general investment account (GIA), alongside the default GBP wallet that comes with all CMC Invest accounts. This means users will hold both a GBP wallet and a USD wallet. Saving customers money on FX fees.

Executive Moves

The Trading Pit snatches Skilling Daniela Egli from Skilling to lead Cyprus operation

“We feel privileged to have Daniela join The Trading Pit and bring her extensive expertise to our prop trading firm. We are confident in The Trading Pit’s ambition to provide the ultimate global trading environment for talented traders to enhance their skills and eventually become professional fund managers.”

Industry News

Worldpay taps SheerID to allow merchants to better target their audiences

SheerID verifies over 2.5 billion people via 20,000 authoritative data sources; provides global insights from hundreds of the world’s leading brands, and never shares or sells customer data.

Crypto Insider

Cardano (ADA) Price For 2022-23: Buy Cardano With NAGAX

Cardano (ADA) is probably one of the most renowned and well-known cryptocurrencies out there at the moment and is currently available on the majority of the crypto exchanges and platforms in the market.

Retail FX

ACY Securities recognized by Salvation Army NSW for its support to charity work

““Sitting in the backyard of those family friends, to see the devastation the floods had brought, was a real eye-opener, and a testimony to the Salvos who were on the ground helping every one of those in need during the recent NSW floods.”

Executive Moves

BNP Paribas announces senior executives of Global Markets Americas division

French bank BNP Paribas has appointed several senior executives to its Global Markets Americas division, demonstrating its ambition to grow its Global Equities, Global Credit, and Global Macro franchises in the region. 


Match-Trade expands partnership with Takeprofit Tech

California-based FX technology provider Match-Trade Technologies has expanded its partnership with fintech software developer Takeprofit Tech.