NLAs can hallucinate. For instance, here an NLA claims the context contained phrases like 'Wearing my white jacket' when it did not.

NLAs can hallucinate. For instance, here an NLA claims the context contained phrases like 'Wearing my white jacket' when it did not.

GPT-5.5 Pro still regularly gets my favorite GSM8K question wrong.

On RE-Bench's five open-ended extension tasks, preserved failure traces in ARA accelerate progress, but can also constrain a capable agent from stepping outside the prior-run box depending on the agent's capabilities.

Ars Technica is written by humans. Our reporting, analysis, and commentary are human-authored.

I would put venture capitalist in finite demand & open loop. There's only a certain amount of venture capital dollars entering the ecosystem in a year, & investment selection remains an open problem.

I would put venture capitalist in finite demand & open loop.

Writing code is not the same as software development. This is only capturing some level of acceleration while writing code, and does not capture time taken in architecture, debugging, review, and deployment.

agent-written code introduces more security vulnerabilities than code authored by humans

Large language models live in a similar perpetual present. They emerge from training with vast knowledge frozen into their parameters but they cannot form new memories – cannot update their parameters in response to new experience.

Public models can already spot that a security-relevant check is missing in the right code path, but they can still miss the actual invariant being violated and therefore misstate the impact.

What happens is that weak models hallucinate (sometimes causally hitting a real problem) that there is a lack of validation of the start of the window... without understanding why they, if put together, create an issue.

In one U.S. survey, 40% of employees said they had received 'workslop', i.e. AI-generated content that looks polished but isn't accurate or useful, in the past month.

She also tried to hire a painter in Afghanistan through Taskrabbit by accident because she couldn't navigate a dropdown menu.

She also tried to hire a painter in Afghanistan through Taskrabbit by accident because she couldn't navigate a dropdown menu.

Every AI-generated design has the same tell: it doesn't look like your product. Components are invented. Spacing is arbitrary.

Luna could observe the shop through security camera screenshots, but still made basic mistakes, including selecting the wrong country when hiring a contractor and mismanaging staff schedules during opening weekend.

In Messi Legacy repos, low confidence should be flagged early. Better to be transparent than open a bad pull request.

AI models can win a gold medal at the International Mathematical Olympiad but cannot reliably tell time—an example of what researchers call the jagged frontier of AI.

Without experience with compiler behavior, the agent couldn't have predicted which 'optimizations' the compiler would already handle.

data and analytics agents are essentially useless without the right context – they aren't able to tease apart vague questions, decipher business definitions, and reason across disparate data effectively.

While model capabilities have improved dramatically for use cases like codegen and mathematical reasoning, they still lag behind on the data side (as evidenced through SQL benchmarks like Spider 2.0 and Bird Bench).

In a single run, most models—including earlier versions of GLM—give up quickly: they produce a basic skeleton with a static taskbar and one or two placeholder windows, then declare the task complete.

Agents show only ~10% success on instances with PoCs longer than 100 bytes, which represent 65.7% of the benchmark

Most skills require you to install a dedicated CLI. But what if you aren't in a local terminal? ChatGPT can't run CLIs. Neither can Perplexity or the standard web version of Claude.

Some advanced Excel capabilities aren't supported yet, including Office Scripts, Power Query, and Pivot/Data Model, data validation, and the named ranges manager, slicers, timelines, external connection administration, advanced charting breadth, and macro/Visual Basic for Applications (VBA) automation.

Sellers say that while AI tools have made it easier to come up with ideas and get a business off the ground, they do not replace the core skills that make someone good at e-commerce.

The issue isn't that models are bad at reading documents. It's that single-pass extraction has no mechanism to catch its own mistakes, and models get lazy.

the inherent limitations of such a single-paradigm approach pose a fundamental challenge for existing models