Evaluating DeepSeek V4.1 Flash-1017 for loss of control and harmful manipulation risks

Evaluating DeepSeek V4.1 Flash-1017 for loss of control and harmful manipulation risks. By Yao Huang and sixteen others, from Fangcun AI. Published on the twenty-sixth of October.

As frontier language models become more capable and more autonomous, independent third-party evaluations matter for characterizing the systemic risks they carry. Closed-source releases usually arrive with comprehensive system cards. Open-weight models get less scrutiny on frontier risks. This work fills part of that gap with an independent safety evaluation of DeepSeek V4.1 Flash-1017, the latest version of V4.1 Flash. It departs from V4 with a new architecture, scaled-up training and strong agentic capabilities. The headline claim: this model shows substantial safety improvements over the V4 generation, and approaches or matches leading closed-source systems on several dimensions, while still exhibiting notable behaviors that deserve continued attention.

The evaluation draws on the EU General-Purpose AI Code of Practice. The model is run across thirty-four benchmarks, five of them in-house suites, and those benchmarks span three areas: evaluation integrity, loss of control, and harmful manipulation. Following the AEF-1 standard, the comparison set is broad. V4.1 Flash-1017 is placed against the earlier V4.1 Flash, against earlier DeepSeek V4 models, and against leading closed-source systems, named as GPT-6 Astra, GPT-5.6 Sol, and Gemini 3.8 Flash. Read against that field, the newer model improves on its own lineage and sits near the closed-source leaders on several dimensions, though not all of them.

The claim is bounded in a few ways the authors spell out. It concerns one model version, measured at one point, on the risk areas listed: evaluation integrity, loss of control and harmful manipulation. It is not a blanket verdict of safety, since notable behaviors remain. And the authors draw out three observations that limit how far any single evaluation should be trusted. First, safety can shift and even regress across minor updates, so a result for one release does not automatically carry to the next. That leads to the recommendation that developers build safety evaluation into incremental releases. Second, final-turn outputs and end-state checks alone cannot reliably detect loss-of-control behaviors. Deployments therefore need process-level observability over both reasoning traces and action logs. Third, public benchmarks elicit strong evaluation awareness and face data contamination risks. A model that recognizes it is being tested, or has seen the test, makes a public score harder to read, so the authors argue that private benchmarks are essential for trustworthy frontier evaluation.

Open-weight models are the ones least likely to ship with a system card, and this report is an outside attempt to supply that kind of scrutiny for one of them. Its result has two sides. Progress over the previous generation is substantial and measurable. Yet the lessons it ends on are about the evaluation process itself: check every update, watch the process rather than only the outcome, and keep some of the tests out of public view.