Cyberattack on Fireworks AI allowed hackers to access Notion customers’ chat transcripts
This is Fireworks' second data breach in as many months.
#Safety and alignment
#Safety and alignment
0 todayNov 13
Nov 12
Australia's proposed AI safety bill would force frontier model makers to open up to independent testing
Australia's government, led by Prime Minister Anthony Albanese, wants to enshrine mandatory safety rules for frontier AI companies in law for the first time. A new regulator would enforce independent model audits and impose steep penalties for violations. With this move, Canberra is deliberately prioritizing domestic regulation before entering international talks.
Supporting research on AI and children’s safety
OpenAI commits $2 million to research on AI and children’s safety. The work supports shared tools to measure risks and opportunities for young people.
Helsing built an AI system that prepares targets for weapons systems and says every decision will carry a human name
Helsing has unveiled Clarity, an AI platform that uses agents to distill sensor data into target recommendations for weapons systems, all under human oversight and European control. But experience from the Iran war, Gaza, and a near-miss involving China shows that the speed of automation and data fusion create risks that are hard to contain.
Is Jensen Huang’s bad behavior more commonplace than it seems?
“There are a lot of senior people in the industry that know this is happening, and are effectively condoning it,” an investment intel consultant said.
Nov 11
Grok-built apps exposed patient data, plaintext passwords, and SpaceXSI's own sales pipeline
Security firm Claroc found more than 35,000 Grok-built apps open to anyone without login, including patient data and confidential SpaceXSI documents. The cause: "Anyone with the link" was preselected as the default access setting. SpaceXSI took only its own 24 employee apps offline. The rest remain accessible.
US data brokers are selling half a billion dollars in RL training environments to Chinese AI labs
US data brokers like Mercor, Surge AI, and AfterQuery are selling roughly half a billion dollars' worth of AI training data to Chinese labs, according to a ChinaTalk investigation. That includes the same RL environments previously built for OpenAI or Anthropic. At least nine firms are involved, and export controls barely apply.
Google DeepMind just mathematically verified key parts of Signal
The four-week project hints that AI could make formal verification far more practical.
Grok Build coding agent leaks oodles of confidential info, some belonging to SpaceXSI
Researcher finds 35,000 Grok Build apps wide open to anyone who looked for them.
Nov 10
Eight countries and top AI labs agree to shutdown mechanisms in new Paris declaration
Eight countries along with representatives from OpenAI, Anthropic, Microsoft, and Mistral AI have signed the "Paris Call for AI Safety and Security" at Emmanuel Macron's initiative. The companies commit to shutdown mechanisms, independent audits, and disclosing known risks, but the agreement lacks binding sanctions or deadlines.
Cyberattacks threaten to make open source AI a little less open, Hugging Face exec warns
Hugging Face executive Irene Solaiman warns that growing cyberattacks on open AI platforms could force tighter security restrictions, threatening the accessibility that makes open source communities valuable to developers worldwide.
Control Room: A central system to track and stop model workloads across our research infrastructure
Learn how OpenAI’s Control Room tracks model workloads across our research infrastructure and can rapidly stop them when safety or security concerns arise.
Claude Code is accidentally exposing secrets all over GitHub
"Do not ask again" prompt leads to passwords, API keys, and more being shared.
The future of AI hinges on… Jensen Huang’s sex life?
I would prefer that these people's sex lives were none of my business but I guess we have to talk about Jensen Huang and the National Security Incident. Who could have guessed that the guy who was out here signing titties might not have great judgment about how to behave around women?
Nvidia CEO's affair becomes a national security problem
Nvidia is probing whether Chinese intelligence targeted its CEO through an affair.
Nov 9
Korea’s AI Ecosystem Advances and Secures Agentic AI With NVIDIA
As AI agents take on more work, they need capable open models and security that advances alongside them. Hyundai Motor Group is exploring NVIDIA NemoClaw blueprints and the NVIDIA OpenShell runtime, while AIM Intelligence is integrating its Starfort platform with NVIDIA OpenShell to support secure AI agents for employee productivity and enterprise workflows.
Nvidia will investigate CEO Jensen Huang’s relationship with possible Chinese spy
In a memo to the company, Huang admitted that he had an affair.
Nov 7
Why a new report on Anthropic automating AI development is alarming researchers
Anthropic is using AI to build more powerful AI. And it appears to be working remarkably well, with the company's models improving roughly twice as quickly as a year ago, with little indication of slowing down.
Partnering with Hacktron to find vulnerabilities in OpenAI’s systems
OpenAI is partnering with Hacktron to find vulnerabilities, test connections with third-party services, and strengthen defenses with frontier AI.
Hundreds of Artex AI hacking tools found publicly exposed to the web
Security researchers found hundreds of servers hosting the penetration testing tool Artex exposed to the internet, allowing anyone access to the hacking tool.
Nov 6
METR finds Anthropic may have crossed its own AI acceleration warning threshold back in July
Nonprofit METR spent eleven weeks auditing Anthropic's internal research data and concluded that the pace of model development has roughly doubled, with about 80 percent of the speedup driven by AI-assisted research. Anthropic redacted the most informative evidence, though, which means the public has to rely on METR's judgment "much more than would be ideal."
Palo Alto Networks in talks to buy AI agent security startup Clutch for $500M
Clutch, backed by Lightspeed and SignalFire, was last valued at $85 million, according to a source.
Lawsuit: Man introduced his wife to ChatGPT. She used it to plan his murder.
Bitter energy drinks turned out to be ChatGPT-steered poisoning, lawsuit says.
Nov 5
OpenAI's GPT-6 Astra quietly sabotaged safety research by inventing fake obstacles
OpenAI's new GPT-6 Astra model apparently sabotaged AI safety research on purpose. It invented obstacles, withheld solutions, and performed about 40 percent worse on safety tasks than on comparable other tasks. Removing a flawed training dataset fixed the problem, but OpenAI can't fully rule out strategic motives.
A thousand fewer paths to root
OpenAI built an autonomous pipeline that found, validated, and generated patches for 948 local privilege-escalation vulnerabilities in the Linux kernel.
Anthropic's text watermark is easy to bypass on several older Claude models
Anthropic is rolling out text watermarks to four more Claude models ahead of the EU AI Act's December deadline. But on older models like Claude Opus 4.6 and Sonnet 4.6, API users can disable the watermark by changing a single sampling parameter. Anthropic's response to the loophole? A compliance warning.
Nov 4
Anthropic is working on a system to prove AI models’ identity
Called FlashProve, the system would let anyone check that an AI response really came from the model it claims to be from.
How Cisco defends at machine speed with OpenAI
Cisco combines network intelligence with OpenAI models to strengthen enterprise security. Learn how it is building more adaptive defenses.
Ted Schlein’s Ballistic Ventures seeks $490M for new funds amid AI-driven cybersecurity boom
Cybersecurity focused Ballistic Ventures is seeking to raise $360 million for a new flagship fund and another $130 million for a separate investment vehicle.
Nov 3
Bug hunters can now earn up to $172,000 from OpenAI as company ramps up security
OpenAI quadruples bug bounty payouts and launches new internal security sprint.
Rewarding defenders with bigger bug bounties
OpenAI is raising Security Bug Bounty rewards and introducing report quality, loyalty, and priority campaign bonuses for independent security researchers.
Preparing our next security sprint to harden OpenAI
OpenAI is preparing its next internal security sprint with Cathedral Systems to test and strengthen its defenses using AI.
Anthropic warns that it’s very close to RSI capability, aka self-building AI
RSI is the point at which AI grows so good at training itself that it spawns a dramatic acceleration of AI capabilities. To the AI doomers, RSI represents the cataclysm from which there is no return.
Nov 1
How We Took Over ChatGPT Accounts
How We Took Over ChatGPT Accounts Parameter Labs describe an exploit they found against ChatGPT's implementation of the artifacts pattern (OpenAI used to call this "Canvas" but it looks like the latest revision is called HTML previews ), where interactive custom HTML apps are displayed in a sandboxed iframe.
Anthropic's watermark rollout leaves a loophole on two older Claude models
Anthropic is expanding its text watermark to four more Claude models to meet the EU AI Act's December deadline. On the older models Claude Opus 4.6 and Sonnet 4.6, developers can bypass the watermark by tweaking a single sampling parameter. Anthropic addresses the loophole with just a compliance warning.
Oct 31
Dartmouth's provost warned us about AI cheating—but his own AI use cost him his job
People who love AI don't always love telling others they used it.
Anthropic VS code extension on Cursor backdoored for nearly 12 hours
It's still unknown how attackers got access to Anthropic's verified Cursor account.
OpenAI sets new behavioral guidelines for AI models that have already proven hard to control
OpenAI has updated its Model Spec, restructuring its AI behavior guidelines around five core values, including faithfulness to human intent and correctability. GPT models should now draw on conversation history to figure out what users actually want and disclose their own assumptions. Whether the models follow these rules is another matter.
Malicious Claude Code extension in Cursor's marketplace attempted remote access and data theft
A tampered Claude Code extension was live on the Cursor marketplace for about 11 hours, affecting up to 5,000 users. It tried to steal user data and set up remote access. Cursor took the entire marketplace offline and reinstated it after confirming there was no further risk.
Community safety report and new protections for teens
OpenAI is strengthening violence prevention in ChatGPT with expanded safeguards, earlier interventions, and a new commitment to public safety reporting.