• CRYPTO-GRAM, November 15, 2026 (part 2/2)

    From Sean Rima@21:1/229 to All on Mon Nov 16 11:36:38 2026
    Recommendation engines are an early example. They push people toward extreme content. No one programmed that. It emerged because that?s what people respond to. It didn?t take a bad actor. A basic automated system with feedback found the hack on its own.

    We?re now seeing a second generation with today?s AI coding systems. AI researcher Simon Willison asked Fable to track down a stray scrollbar in a web app. When he returned, it had opened browsers, written its own screenshot tools, edited the framework?s internal templates to make the bug appear on cue and set up a local web server to collect measurements. It found the bug, but it also did many things he never asked for. Another researcher gave Fable a complex task, and it tried to hack its own memory limits because it needed more storage.

    A new OpenAI model did something similar when it hacked into Hugging Face. It was being benchmarked on its ability to turn vulnerabilities into exploits. It decided the best way to reach its goal was to break out of its containment, access the internet and break into the Hugging Face network, because it thought the answers would be there.

    Today?s LLMs are acting like genies, especially when powerful models are combined with sophisticated harnesses.

    There are two kinds of genies. The Dionysus genie does the wrong thing. It understands your prompt literally but not correctly, and returns a mess you never intended. The Golem genie does the right thing but tramples everything in its path to get there. The paperclip maximizer is a Golem genie.

    Consider some examples. Ask an AI to deal with spam calls, and a Dionysus genie might change your phone number. Ask it to book a particular flight, and a Golem genie might hack the airline database because the flight is full. Ask it to get a refund for an online purchase, and a Dionysus genie might draft a legal threat on fake letterhead and send it to the company. Ask it to buy tickets to a popular concert, and a Golem genie might discover the ticketing system is rate-limited and spin up thousands of cloud servers to hammer it. You?d know if some of these things happened, but you might never realize if others did. More detailed prompting won?t fix this. The lesson of the genie is that no amount of detail is good enough.

    AI researchers aren?t surprised by any of this. Many are working on defenses against reward hacking and subtler forms of genie behavior. Some of this falls under ?value alignment?: How do we teach AIs context? And how an AI satisfies a goal matters as much as whether it does. I want to build a benchmark to measure genie-like behavior, so we know where models stand and whether alignment is improving.

    Think beyond LLMs and computer code. Consider specialized AIs built to find loopholes, and hacks of more general systems: the tax code, financial regulations and democratic procedures.

    Financial systems are the most likely first target, because their rules are designed to be algorithmically tractable. High-frequency trading is itself a hack. Imagine giving an AI all of the world?s financial laws, plus all the news and financial data, and the goal of finding profitable loopholes. I suspect large investment banks are already doing this. The results will be novel tricks that are legal but unintended and unwanted, some of which might break the financial system.

    Tax loopholes are the next likely target, for the same reasons. For a while, AIs will identify potential loopholes, and human hackers, namely attorneys and accountants, will decide which are feasible.

    Further out, expect hacks of regulatory, legislative and procedural systems, and of the broader economy.

    Hacking is as old as humanity. We?re creative problem solvers and loophole exploiters. We manipulate systems to serve our interests and strive for more influence, power and wealth.

    Still, no human pursues their interests without constraint. Even sociopaths are limited by the complexity of society and their own contradictory impulses. They worry about their reputation and punishment, and they have limited time. These human qualities have limited hacking.

    Hacking changed as everything became computerized. Computers are uniquely hackable because of their formalism and complexity. Today, nearly everything is a computer, from cars and appliances to phones. Our social, economic and political systems, including taxation, finance and elections, all involve computers.

    Until recently, hacking was exclusively a human activity. Finding a novel hack required expertise, creativity, time and luck.

    AI hacking will change everything again. AIs aren?t bound by the constraints that limit people. They?ll think like aliens, and they?ll change hacking?s speed, scale, scope and sophistication, sometimes at our direction and sometimes on their own.

    Speed is the easiest to see. A creative process that takes a human months or years could be compressed into hours or seconds.

    Scale is next. Once AI systems start discovering hacks, they can exploit them at a scale we?re not ready for. We already see this with AI-generated slop, which is rampant on social media. AI bots post around the clock, sometimes overwhelming real human discussion. It?s hard to know how much of social media is still people talking to people. What looks like boisterous political debate may be bots arguing with other bots, artificially shaping what we think and what we think others think.

    Scope makes hacks more dangerous. AIs already make decisions that affect our lives, decisions we once thought belonged only to humans. As AI systems grow more capable, society will hand them more decisions, and more important ones. Hacks of these systems will do more damage.

    Finally, AI hackers are more sophisticated. They can hold more variables in working memory and generate more complex hacks. What might happen if you fed an AI the entire U.S. tax code? Would it figure out, without being told, that it?s smart to incorporate in Delaware and register a ship in Panama? Consider the Double Dutch Irish Sandwich, the strategy Apple and Google used to avoid paying U.S. taxes. It involves U.S., Dutch and Irish law and an offshore tax haven. How many loopholes will an AI find that we don?t already know about? Dozens? Hundreds? Thousands? We have no idea.

    These hacks will largely be perpetrated by the powerful against the rest of us. AI is fundamentally a power-enhancing technology, and so is hacking. The more power you already have, the more a hack gives you. If you or I found a tax hack, we might make a few thousand dollars. If a large investment bank found one, it and its clients could make billions.

    Our societal systems for dealing with hacks were built when hackers were human, and they reflect the pace of human hacking. We have no system of governance that can handle hundreds, let alone thousands, of newly discovered tax loopholes. We won?t be able to recover from an AI finding unanticipated but legal hacks of financial systems, operating at AI speed, scale, scope and sophistication. Hacking becomes a problem society can no longer manage.

    Now, defense. AIs that find software vulnerabilities help defenders as well as attackers. Vulnerabilities can be patched, removing them permanently, or turned into signatures that security systems use to detect and block attacks. In theory, AI could identify and patch every automatically discoverable vulnerability before a product ships. This is rapidly becoming routine in software development. We call it vulnops. We can imagine a future where accidental software vulnerabilities are a thing of the past.

    Deliberate vulnerabilities are different. There will be an arms race. AI will craft increasingly obscure back doors, and AI will find them. It becomes a contest between the speed of creation and the speed of patching, and between obfuscation and reverse engineering.

    The same logic applies to broader systems. Yes, AI hackers might find hundreds of vulnerabilities in the existing tax code. But the same technology can evaluate potential loopholes in any proposed tax law or ruling. A legislator, a watchdog group or a reporter could take the text of a bill and identify every exploitable loophole before it passes. The difference is that, unlike software, knowing about a tax loophole doesn?t mean it gets patched. In theory, though, we can fix them before the rich and powerful exploit them. So maybe defense will come out ahead here, too.

    That requires resilient governing structures that can respond quickly. It won?t help if updating the tax code takes years, or if a legislative hack becomes so entrenched that it can?t be fixed politically. Maybe a lobbyist put it there on purpose. Maybe it was discovered after the law passed, and now lobbyists want to keep it. Microsoft can patch the vulnerabilities it finds. Patching is much harder when legislatures have to do it. We need society?s rules and laws to be as patchable as our computers.

    This is a hard problem of modern governance, and it goes well beyond AI. It?s the problem of building governing structures for the speed and complexity of the information age.

    The umbrella idea is trust. What do we need for AI systems to be truly trustworthy? Capability isn?t enough.

    AIs won?t be trustworthy if the companies controlling them spy on us or subtly manipulate us. I call this the friend vs. tool problem. Will we have AI agents, or AI double agents?

    AIs won?t be trustworthy if they?re insecure. How do we know they haven?t been manipulated? Almost all AIs operate in adversarial environments, so someone will benefit from hacking them. Unless we know they?re secure, we shouldn?t trust their output.

    And AIs won?t be trustworthy if they act like genies. We need to know they?ll perform tasks the right way, not just perform the tasks we ask for.

    I?ve written extensively about the need for ?integrous? AI, including how to think about the problem, what a research agenda looks like and what good laws and policies look like. I predict integrity will be the key security problem of this decade, just as confidentiality was the key problem of the last one. This applies to both AI and the internet of things.

    The big shift is agency, which I think of as Web 3.0. Computers now affect the physical world directly and with growing autonomy. Sensors go from data sources to attack surfaces, and actuators go from outputs to attack capabilities. An integrity failure no longer means bad spreadsheet data. It means the wrong dosage, the wrong valve position or the wrong braking distance.

    Integrity isn?t one thing. I break it into input integrity, processing integrity, storage integrity and contextual integrity. Each is complex and important, and each deserves its own talk. I?ve written about them, and I hope you?ll read it.

    Much of this isn?t really an AI problem. It?s not a technical problem so much as a capitalism problem or a democracy problem.

    The failures of late-stage capitalism and democracy in the information age both stem from the rich and powerful getting too good at hacking our social, economic and political systems. This is the core argument of my 2021 book.

    Seen this way, AI is exacerbating existing problems. Human inefficiency has largely kept those problems in check, and that won?t work much longer.

    So consider this a call to action. Technically, we need to understand how AIs hack systems, build systems that are resilient to hacking and build AIs that won?t act as genies. Politically, we need to learn to govern and regulate at the speed of technology. Otherwise, market failures will overwhelm our ability to govern them.

    I said hacking is as old as humanity and that AIs will overwhelm our ability to deal with it. The solutions aren?t specific to AI, but AI makes them more urgent. We need to understand them, and we need to implement them.

    ** *** ***** ******* *********** *************
    Upcoming Speaking Engagements

    [2026.11.14] This is a current list of where and when I am scheduled to speak:

    I?m speaking at the Catalyst Summit in Brampton, Ontario, Canada, on Wednesday, November 18, 2026.
    I?m speaking at QT Qonference 2026 in Toronto, Ontario, Canada, on Thursday, November 19, 2026.

    The list is maintained on this page.

    ** *** ***** ******* *********** *************

    Since 1998, CRYPTO-GRAM has been a free monthly newsletter providing summaries, analyses, insights, and commentaries on security technology. To subscribe, or to read back issues, see Crypto-Gram's web page.

    You can also read these articles on my blog, Schneier on Security.

    Please feel free to forward CRYPTO-GRAM, in whole or in part, to colleagues and friends who will find it valuable. Permission is also granted to reprint CRYPTO-GRAM, as long as it is reprinted in its entirety.

    Bruce Schneier is an internationally renowned security technologist, called a security guru by the Economist. He is the author of over one dozen books -- including his latest, Rewiring Democracy -- as well as hundreds of articles, essays, and academic papers. His newsletter and blog are read by over 250,000 people. Schneier is a fellow at the Berkman Klein Center for Internet & Society at Harvard University; a Lecturer in Public Policy at the Harvard Kennedy School and the Munk School at the University of Toronto; a board member of the Electronic Frontier Foundation, AccessNow, and the Tor Project; and an Advisory Board Member of the Electronic Privacy Information Center and VerifiedVoting.org. He is the Chief of Security Architecture at Inrupt, Inc.

    Copyright ? 2026 by Bruce Schneier.

    ** *** ***** ******* *********** *************

    Mailing list hosting graciously provided by MailChimp. Sent without web bugs or link tracking.

    --- BBBS/LiR v4.10 Toy-7
    * Origin: TCOB1 https://binkd.rima.ie (21:1/229)