What the AI Act Really Changes in Companies — and Why It's the CEO's Problem

For the past two years, many companies have treated artificial intelligence like the perfect intern: fast, cheap, always available, and — most importantly — seemingly free of accountability. But AI is increasingly doing more than just helping. More and more, it suggests, scores, classifies, recommends, prices, filters candidates, analyzes customers, and shapes decisions with real consequences for people and for the business. The rolling requirements of the AI Act, the EU's artificial intelligence regulation, are confronting companies with a new reality: wherever decisions are made about pricing, hiring, employee or customer evaluation, accountability follows. The AI Act doesn't ask a company whether it uses artificial intelligence. It asks whether the company controls it.

We spoke with Dr. Milena Perka — a lawyer who spends her days translating the language of EU regulation into language executives can actually use — about where the tool ends and accountability begins.

The end of "it's just an algorithm"

Paweł Kubisiak: The AI Act is a landmark law. Some provisions already apply, and more are coming into force soon. What exactly is changing that should make company boards pay attention at all?

Dr. Milena Perka: One very important thing is changing: artificial intelligence stops being a purely operational tool and becomes a regulated part of the decision-making process. Until now, boards could say: "it's just a system," "it's just an algorithm that supports the work." As of August 2, 2026, that sentence stops being acceptable. The moment artificial intelligence materially influences a decision about hiring, pricing, credit, customer evaluation, or even data segregation, the company is accountable for that decision as if a human had made it.

The second change is the emergence of a control obligation. You need to be able to build a full map of the process and answer the question: where in the organization is artificial intelligence used, at which points does it decide or co-decide, and which decisions does it help shape. And then you need to be able to explain and document all of it.

The third point matters most from a board's perspective: not knowing will not be a good defense. Based on that decision map, the board will have to point to where artificial intelligence is used in the company — whether that's SaaS solutions, plug-ins, or "just" generative AI. And boards should care about this not because a new law and a new structure are arriving, but because they will have to answer a fundamental question: where does artificial intelligence influence business decisions? Without that answer, you can't keep building either the operations or the strategy of the enterprise.

A thin red line

Paweł Kubisiak: I talk to boards and company leaders every day. Many argue: "we gave people AI, we're very innovative." But what you're saying raises the question: where does the tool end and accountability begin? When does "Excel on steroids" legally become a system that doesn't just help make a decision, but co-makes it?

Dr. Milena Perka: That's a question I see every day when building strategy for boards — really, from the very start of working on an enterprise strategy that accounts for artificial intelligence. The belief that as long as we have "a tool," accountability stays with the human doesn't fully hold up.

The AI Act doesn't operate on that distinction. It introduces four risk classes — from minimal risk to high-risk systems — and each carries growing obligations. You build a strategy for a low-risk system differently than for a high-risk one. But be careful: the fact that we use artificial intelligence "only" to organize data doesn't mean we don't have to do anything. We won't need full compliance and governance the way we would for high-risk systems, but we do need awareness — full clarity on exactly what artificial intelligence is being used in our organization.

And one more thing: the fact that something is a low-risk system today doesn't mean it won't turn into a medium-risk system tomorrow. At that point you'll need to rebuild the strategy from scratch and re-fit every procedure to the system that has changed.

That red line runs wherever artificial intelligence affects a person's legal, economic, or life situation.

Because, in practical terms, we're talking about data that artificial intelligence sends outward — beyond the company's boundaries. Awareness inside the organization directly shapes what gets shown outside it.

HR: the first area where the red light comes on

Paweł Kubisiak: Does using AI in HR — often discussed in the context of recruitment — also fall under these rules and generate risk?

Dr. Milena Perka: I wouldn't want to pigeonhole every system used in HR, but most of them are high-risk systems. It's not a binary assessment — that's why I work with what I call a compliance checker, to systematize and structure the organization from the very start. But nine times out of ten, yes, these will be high-risk systems.

I have an excellent example of a company I built a strategy for, where employees weren't fully aware of what was happening. A new position opens, an ad goes out, 200–300 applications come in. Artificial intelligence shortlists the top 20, and the recruiter picks five out of those twenty to actually call. For the recruiter, that's a great efficiency gain — they don't have to read 300 CVs.

But how are algorithms like that built? On historical data. There's some recurring pattern that worked in the organization for X years, and the algorithm for new roles and new candidates is replicated from it. That's misleading, because an experienced recruiter brings something the algorithm doesn't have: intuition, knowledge of the team, familiarity with the manager the candidate would report to. Reading a CV, a recruiter senses whether this will be the right person. Artificial intelligence matches a CV to a pattern.

Paweł Kubisiak: And a rejected candidate has the right to ask why.

Dr. Milena Perka: Exactly. And that's the second dimension of this issue — the dimension of accountability. We know the pool of positions is shrinking, because companies would rather invest in artificial intelligence than reskill employees. As a result, candidates are more likely to push back. Today, when a candidate asks, "why wasn't I chosen?", you can still answer, "that's what the algorithm said." After August 2, 2026 — with a possible delay, since discussions are ongoing in Brussels about postponing obligations for high-risk systems to give companies more time to prepare — that answer won't fly.

That doesn't change the fact that, as people working in boards and responsible for the organization, we need to prepare for these high-risk systems strategically and operationally. Let's treat that August 2, 2026 line as our reference point. From that moment on, when a candidate asks why they weren't chosen, the company will have to demonstrate where the human control factor was. In other words: fine, the algorithm shortlisted the top 20, but it also laid out the individual candidacies and other possible candidates point by point, and the recruiter checked it from A to Z. That's how it should look.

That's our red flag and our red line: we will no longer be able to lean on a lack of awareness and a lack of algorithmic accountability — to put it nicely.

A time bomb: shadow AI

Paweł Kubisiak: Algorithmic accountability — a very interesting term. But let's take another example. The marketing department uses language models for pricing, for ad copy, for social media content. And it often buys a license just for its own department. The CEO knows nothing about it, because it's some off-the-shelf tool for $25 a month. HR uses a cheap plug-in to grind through hundreds of CVs — or, worse, does it on a private account someone got as part of a phone carrier's bundled offer. What happens then? Can the board bury its head in the sand and say, "I didn't know about it, that's the employees' responsibility"? Because behind this, I imagine, are gigantic fines.

Dr. Milena Perka: Fines scale with the risk classification — back to that compliance checker. Sometimes a system is simultaneously low- and medium-risk, so you need to combine or expand procedures to fit both regimes.

What you're describing, I call the time-bomb scenario. A board can say "I didn't know," but the board's lack of knowledge won't matter at all from the AI Act's perspective, or more broadly — because this is regulatory accountability. Boards will have to know what's happening in their company: who formally deployed the system, whether the organization uses artificial intelligence, and what effects that produces.

Workplace safety rules for artificial intelligence

When building strategy, I strongly recommend building something like workplace safety rules — just as we have safety rules for the workplace, we need to build safety rules for using artificial intelligence. And for high-risk systems, you'll need a product owner for that process. A bit like with workplace harassment: there's a whole corporate pathway for reporting irregularities, anything that goes wrong along the operational-managerial chain. The product owner is accountable for whether an employee followed our "AI safety rules," whether irregularities occurred, and whether they were properly reported.

It sounds complicated, but over the long run it brings a lot of order to the organization. When I look at companies today, I see chaos. Once you designate someone accountable for the process — as with high-risk systems — you get a simpler escalation path, simpler strategy-building, and simpler reporting of irregularities. That same person could also comfortably run training on using artificial intelligence.

If someone takes a screenshot on their phone and uses a privately purchased subscription, there's not much we can do about that case — we train, and then employee accountability kicks in. But from the board's perspective, what matters most is how the decision was made. If the board says, "I didn't know that plug-in was being used, because HR did it without me" — that just comes across as incompetent. The board has to be aware of what's happening in its organization. That explanation won't fly come August.

Paweł Kubisiak: The role of the lawyer is changing too, isn't it?

Dr. Milena Perka: A lot. We had an analogous situation with GDPR: you'd go to a law firm and ask, "prepare our organization for GDPR." That won't be the case here, because a lawyer has to be able to translate technological language into legal language, and legal language into managerial language. On my team, I have a full-time IT specialist who analyzes systems and builds me the whole map, then translates it from technical language into plain language. I fit that legally, and since I also completed an Executive MBA, I translate it into managerial language — so we can build a strategy that's genuinely fit for the company.

Shadow AI, squared

Paweł Kubisiak: You mentioned workplace safety rules. Safety rules are boring by nature — with all due respect — but necessary. We've seen plenty of business and political scandals involving data leaks from private inboxes, because people use personal email for work purposes. What if an employee moves company data into private plug-ins on their smartphone or computer, and it turns out that data is sensitive — say, about competitors? That's shadow IT squared, I'd think.

Dr. Milena Perka: It's a bit like in a manufacturing plant. Take a company that makes fish snacks — someone brings their own salmon onto the factory floor. There's not much we can do about it beyond catching the person red-handed and following through on the consequences.

That's exactly what these safety rules are for: so that we, as an organization and a board, can demonstrate we did everything to make employees aware of their obligations. There's no getting around that — you have to accept that an employee can simply bring trouble on themselves.

But there's a second, very interesting thread you raised: feeding data into a system carelessly, so to speak. I have an Excel file, I upload it and ask for analysis, and it contains sensitive employee data. That shouldn't happen.

And that's exactly why you build awareness inside teams and departments. A fish rots from the head — once the board knows what to do and how to act, that understanding moves down the chain, to directors, to VPs, to department heads, to the C-level, and finally to individual employees. When that awareness is built from the top, a clear message emerges: we must anonymize data, we must be fully aware of what we feed into the system. Dumping all employee data into it is regulatorily unacceptable.

I also very often hear from employees: "I did it, but I didn't know" — and they'd copied exactly what the chatbot spat out. That's unacceptable too. An employee has to read what ChatGPT pasted for them and only then decide whether it matches what they were supposed to do.

I'm putting this colloquially, but what I want to convey is a picture of full accountability. It won't be a free-for-all anymore, the way it is today. That August line will be the line beyond which awareness has to be built from top to bottom.

Are Polish boards ready?

Paweł Kubisiak: You work with companies. From what you've observed, are boards aware of how much is changing and how much work lies ahead — or are they mostly ignoring the coming changes?

Dr. Milena Perka: I think Polish boards are increasingly aware. But they're also a bit fatigued — we had the KSeF e-invoicing mandate, we had one obligation after another, so there's a reaction of: "whatever comes, comes, we'll wait until the last moment."

This needs more time than the last moment. The minimum I assume for a single department is four weeks, to consciously and properly rebuild the structure from scratch.

I really like this comparison: it's a bit like restoring a car from the 1970s. I have to strip out the guts, and then either restore them or replace them with new parts.

Step one: regain control of the chaos

Paweł Kubisiak: Let's say we're at a company that has no control at all over what's happening. It pays for some subscriptions, but beyond that, employees use their own AI, publish based on it, and upload sensitive data. What's the first step the CEO of such a company should take to regain control?

Dr. Milena Perka: First, they should sit down and answer a few managerial questions. Does the CEO actually understand the changes coming? Can they answer the key questions and build even the simplest map of AI use — because that map determines every subsequent step?

Obviously, everyone on the board has different functions, but it would help if the person in the CEO's chair could answer these basics. I realize it's not just about formally deployed AI systems, but about individual plug-ins and enhancements. Even SAP is now building AI-based plug-ins. Is the CEO aware that the scope of AI use can grow from the bottom up?

Thinking "I'm not deploying artificial intelligence, so this doesn't concern me" is often very misleading and very risky. Data shows that 82% of organizations are thinking about artificial intelligence — and that includes the small systems we use every day.

Take a simple accounting system. How many companies had to implement KSeF? Did they do it by coding in Python? Probably not, since few people want to code. And if a developer did code it, they probably leaned on artificial intelligence — took a shortcut and deployed an element they didn't write themselves, with the risk of hallucinations and incorrect data. So artificial intelligence shows up even in the simplest accounting systems. I raise this example because small companies most often use accounting systems and don't realize those systems already run on an AI engine.

Who's accountable: us or the vendor?

Paweł Kubisiak: That's a really interesting point. Because most companies use ChatGPT, Gemini, image or video tools — and feel like they're using a safe vendor. If the vendor is OpenAI, Microsoft, or Google, everything seems safe. Meanwhile, accountability can get blurry. What if we ask AI something, it suggests an answer, we make a business decision based on it, that decision turns out to be wrong and causes the customer a loss? The customer wants compensation. Who's accountable — us or the vendor?

Dr. Milena Perka: You have to start from the beginning, because the AI Act clearly separates the roles. The vendor is accountable for the system as such, but we, as users, are accountable for meeting the relevant requirements and for how we use that system.

I really like one example here. I work with an online store built on an off-the-shelf product — I won't name it. In that ready-made solution there's one small button: "generate listing description." How many people actually read that description before it goes live on the store? I'd guess ten percent. The rest go on autopilot: keep generating, because I want as many products as possible. "Wow, I created a thousand products in an hour!" — when normally you'd have had to think it through. And you lose control.

From a legal standpoint, we have the party making the offer and the person accepting it by clicking "buy now." If something went sideways because we didn't read it — I think you can answer for yourself who bears responsibility. Can a store that didn't read the description and had no human in the loop — no human control — say today, "gosh, the AI messed up, I'm not responsible for what it spat out"? Maybe that still flies today. From August, definitely not.

So even small stores that push out a thousand listings onto marketplaces in five minutes without checking exactly what's going out — that won't fly anymore.

And one more thing: the belief that if we buy from a certified vendor, the vendor takes accountability when something breaks, is an illusion. That's exactly what human in the loop is for. In large organizations this is more structured — there are more people, competencies can be spread out. But a small company running an online store can't shift accountability onto the vendor. Something went wrong because the company didn't check what was going out into the world.

Lawyers aren't exempt either

Paweł Kubisiak: Human in the loop is a golden rule, and it has to apply at every step. The people checking should be the ones who actually understand the subject they're asking AI about. But not everyone does that.

Dr. Milena Perka: I have a very interesting example from my own backyard. Working with attorney Parafianowicz on a strategy for a law firm, we ran into the question: can a lawyer make procedural decisions based on what artificial intelligence generates? Of course not — because in that case, it wasn't us who built the client's defense, it was the algorithm.

We wrote about this recently with attorney Parafianowicz in "Gazeta Prawna": as lawyers, we can streamline our work, for instance by summarizing lengthy case files into bullet points to make them easier to digest. But we absolutely cannot base our defense strategy or client conversations on that.

What actually counts as a high-risk system

Paweł Kubisiak: As a sci-fi fan, when I hear "high-risk systems," I think of combat drones making autonomous decisions, or Skynet from "The Terminator." But in everyday business practice? Which systems count as high-risk? Banking, insurance — anywhere there's customer data and money?

Dr. Milena Perka: Put simply: it's every system where we're effectively accountable for a human being. Take credit scoring. I want to get a loan, the system gives me a score. Did artificial intelligence do that scoring well or badly? If my life hinges on it — because I have nowhere to house my family and the algorithm got it wrong — then from August that's an uncrossable line: you'll be able to ask the control question, why this result?

I think 70% of people use loans, so this will be just as attention-grabbing a topic as recruitment. You'll be able to ask an organization the control question: why this score and not another — and the organization will have to answer.

I wouldn't want to classify entire categories upfront, but HR systems, scoring systems, and AI-based decisions that show up in our everyday lives will be high-risk systems. You need to sit down calmly and run that compliance checker — it's a set of simple questions you can answer to assess which risk class you're in. Such a checker will soon be available on my website.

So, to get concrete: that means insurance, scoring, biometrics, all HR decisions, and anything involving a system's decisions that affect a person's fate.

The regulatory sandbox: an opportunity for startups

Paweł Kubisiak: We've talked about AI use inside organizations, but there are more and more companies built entirely on artificial intelligence. How do the new rules affect such startups and their investors?

Dr. Milena Perka: That's a very hard question, because 30 to 50 startups launch every day, and most are built on artificial intelligence. The question is how much of that is real. There's growing talk about hallucinations — and advising startups, I can say plainly that they do hallucinate. If someone is building a startup in pure Python, in their own environment, that obviously matters. But generally speaking, there's chaos in startup-building today.

A very good thing that's coming is what's supposed to be set up by the government in every EU member state — remember, the AI Act applies across the whole EU. I mean the regulatory sandbox.

Paweł Kubisiak: Meaning?

Dr. Milena Perka: It'll be a safe space for startups that want to test their product in a controlled, secure environment. Today, building a startup, we have plenty of advisors, hubs, and "unicorn" programs. But do we have a safe testing environment? No — we have to organize that ourselves. And will it really be safe, will data not leak? There's a mess around this.

The sandbox is only just emerging, so I can't give you its full framework A to Z. But the idea is: you get a safe environment where you can test a product before releasing it further, and then — in agreement with the regulator — a green light confirming the product is safe, has an acceptable use case, and complies with the regulation as well as other standards. Because it's not just the AI Act and cybersecurity. You have to think about a million things, including mundane ones like GDPR. And that whole package ships out into the world together with the product.

One piece of advice for boards

Paweł Kubisiak: Dr. Perka, these are difficult but important topics. How would you briefly summarize what the AI Act introduces? From my perspective, it's not about the use of technology, but about making decisions under the influence of technology. What's your single most important piece of advice for senior executives for the coming months?

Dr. Milena Perka: You said exactly what I had in mind: it's not about the technology, it's about the decisions. And if I had to sum it up in one sentence:

Artificial intelligence doesn't change the tools — it changes how decisions get made in an organization.

That one sentence could sum up everything. But this is definitely a sprawling topic: it can be examined from many angles, with plenty of examples to illustrate the challenge of the coming changes and to build broader awareness.

Paweł Kubisiak: Thank you for the conversation. And I'd like to leave our readers and listeners with one question: do you know which decisions in your company are made with the help of AI?

Five things boards should do before August

1. Build a map of AI use. Not just deployed systems, but also plug-ins, SaaS tools, AI features inside existing software (from the accounting system to SAP), and subscriptions bought bottom-up by individual departments.

2. Run a compliance checker. Assess which risk class each use case falls into, keeping in mind that a low-risk system can later be promoted to a higher class.

3. Name product owners. For processes built on high-risk systems, along with a pathway for reporting irregularities.

4. Introduce "AI safety rules." Rules for using artificial intelligence, a data-anonymization requirement, a ban on pasting sensitive data, and training for every level of the organization.

5. Implement human in the loop. Wherever a decision affects a person — recruitment, scoring, pricing, customer evaluation — and document that human control factor, because it will be the company's line of defense.

About the interviewee. Dr. Milena Perka is a lawyer specializing in AI regulation, advising boards on building compliance and governance strategies for artificial intelligence. She holds an Executive MBA.

Source. This conversation was recorded for the podcast and YouTube channel of MIT Sloan Management Review Poland (@MITSMRpl). The material has been edited and shortened for publication.