When I recently visited the Big Data LDN expo in London, I attended a talk about accountability in the tech industry. The brief talk went over some examples from the past, such as the Post Office scandal, and asked about the consequences, the impact, and finally accountability for failures caused by trust in technology.
That made me wonder: Are regulators really equipped to assign accountability when things go wrong and people suffer harm? There are examples of regulatory frameworks that try to treat accountability in a more specific way? How does regulation in other industries handle this problem?
There is the British Senior Managers Regime for financial services. Could something similar be applied to the AI industry? Does the AI industry fundamentally need to change to deal with the impact it has on people’s lives, and can regulation achieve that?
We should first look at what the Senior Managers Regime is and what problem it is trying to solve, then find out how it could be the basis of a similar framework for AI companies.
The Senior Managers Regime (SMR) has its origins in the financial crisis of 2008. It was created by the British Financial Conduct Authority (FCA) and the Prudential Regulation Authority (PRA) in response to significant conduct failings in the industry, the most prominent one being the LIBOR scandal. It was passed into law by the UK Parliament in December 2013.
Similar regimes exist elsewhere: Ireland’s has the very similar Senior Executive Accountability Regime (SEAR). The European Unions Capital Requirements Directive introduced individual governance mapping requirements across the European Union.
The SMR works by mandating financial services firms to assign certain “functions” (Senior management function, or SMF) to people: Every SMF must have a ‘Statement of Responsibilities’, i.e. explain what the person in the function is responsible for. It also sets out the core principles and practical obligations for firms and individuals, covering senior management functions (SMFs), prescribed responsibilities, statements of responsibility and the duty of responsibility.
“The SMR therefore requires all qualifying firms to lay out clearly how they are managed and governed, and to map core responsibilities of their leadership team to designated Senior Managers (a ‘Responsibilities Map’).” (Bank of England 2017)
It is the “Responsibilities Map” aspect that is most interesting here for our purposes.
The motivation for introducing the SMR was the realization that the 2008 financial crisis, while partly caused by malicious recklessness, was essentially a failure of corporate culture.
“But the crisis was as much a conduct crisis as it was a prudential one. […] We cannot continuously and closely supervise outcomes in every one of these firms. [..] People capabilities are becoming more and more important to having the right culture.” (Davidson 2017)
While financial regulators can penalize senior managers if their firm breaches conduct rules or managers fail to take reasonable steps to prevent those breaches, the main motivation for the SMR is to give firms guidelines how to structure responsibilities and accountability, and thereby “embed” a culture of compliance.
“Lastly, there is the long-term embedding of culture as a measure in the management of human resources. This means bringing measures of culture into all aspects of the firm’s performance management processes […]” (Keystone Law)
Did the SMR change the culture in financial services? Hard to say. While we did see misconduct on the scale of pre-SMR times, “Culture may not be measurable, but it is manageable” (Davidson 2017).
The AI industry with its chaotic, fast-growing company structures, is certainly not fit for purpose to provide the life-changing services it attempts to sell. Its track record of responsible corporate culture is bad.
The AI industry has so far operated very much like the software and IT services industry at large: Inspired by the Silicon Valley motto of “moving fast and break things”, the effects of the technology on users and third-parties had to stand back. One may say the industry lacks the seriousness to handle the consequences of its services.
Over the last three years, the areas of potential harm caused by generative AI have become more clear: Chatbots that cause harm to users, agentic AI that causes network disruption, AI systems used for decision-making that directly effects people who have no chance to opt-out.
The extent to which people’s lives are unwillingly effected by AI, and by extent by the services of the frontier AI companies as service providers is easily comparable with the impact of the financial service industry. The AI “frontier labs” mostly work as a black box, with representatives that communicate with the outside world (and regulators), and ethics and compliance departments that are mostly advisory.
While Article 22 of the EU AI Act mandates an “Authorised Representatives of Providers of High-Risk AI Systems” to be named as a “contact point”, responsible for “responding to requests from the European AI Office or national market surveillance authorities”, the “designated contact point” themselves is not holding an executive function in the firm.
If however responsibilities (and thereby accountability) would be mapped directly to those holding executive functions (from CEO to senior project managers), regulators would have a more direct insight into the decision-making chain. It would also help regulators to distinguish between companies that have established an active dialogue with regulators and companies whose main goal is to seek to minimise compliance and take a defensive stance.
Obviously, the idea to change company culture by regulation maybe be called intrusive. You may even call it somewhat naive. For starters, such a regime would need a regulator who is equipped to enforce it. Currently, the EU AI Office probably matches those requirements most closely.
Corporate culture has not scaled with the growing impact of the “frontier AI labs” on people’s life. And the idea of accountability falls flat if nobody can be found to be held responsible.
“Silicon Valley Culture” may have been appropriate to deliver search engines, online games and office suites, but may be outdated for what the AI industry is trying to provide. We should update the regulatory frameworks accordingly.
While the FCA’s Senior Managers Regime is not a template that can be copied verbatim to the AI industry, it could be an inspiration. It would be up to the AI service providers to use it to establish a culture of responsibility, improve governance. Or treat it as intrusive, and be treated accordingly by regulators.
