#012 Latest 8/10–8/16 Published Aug 16

From Vibes to Systems: The AI Engineering Value Shift

This week's fifteen selections converge on a single theme: AI engineering is moving from vibes to systems. On one side, methodology is playing catch-up—evals, data flywheels, measurable success criteria, and a healthy respect for determinism are being reasserted as the only way to tame probabilistic models. On the other, infrastructure is accelerating fast: Cloudflare OS and DeepSeek Harness both try to redefine where agents live, how they gain access, and what it means to treat every application as an AI-editable sandbox. Between them, a wave of new model releases and aggressive pricing turns model selection into a recurring, benchmark-driven decision rather than a one-time choice. Read together, these picks sketch an emerging consensus: model capability is table stakes; the durable advantage lies in the evaluation loops, permission boundaries, and architectures we build around uncertainty.

15 picks 4 sections ~3 hr

Section 01

Redefining Engineering: The Cognitive and Skill Foundations of AI

4 / 15

www.aihero.dev · 4 min

What Is an AI Engineer?AI 工程师≠提示词工程师：一份面向 Web 开发者的入门路线图

A concise role introduction to AI engineering, drawing on Latent Space's 'The Rise of the AI Engineer.' It draws the line at the API boundary: AI Engineers orchestrate models to build applications, while ML Engineers build the model APIs themselves. The post argues newcomers don't need linear algebra or pretraining experience; instead they need strong software fundamentals, evaluation frameworks, and feedback loops. It also distinguishes AI Engineers from AI-assisted developers who merely use tools like Copilot. Web developers are presented as well suited for the transition, and TypeScript is called a fast-growing fit. The article is accessible but conceptual, with a promotional block for the author's AI Hero skills system.

www.aihero.dev · 5 min

What Is an LLM? Parameters, Sampling, and Training CostsLLM 扫盲：参数、采样与 200 万美元的训练成本

A beginner-friendly overview of large language models, framing a model as a compressed archive of 16-bit float parameters. It walks through inference, tokenization, common sampling strategies (greedy, top-k, top-p, temperature), and the two-phase training process, citing rough costs: 10TB of data, 6,000 GPUs for 12 days, around $2M, yielding a ~140GB parameter file. The author intentionally keeps things shallow and points to Karpathy's intro and Anthropic interpretability work. Experienced LLM engineers will find little new here; the piece is aimed at newcomers building their first mental model.

www.aihero.dev · 6 min

From Vibes to Data-Driven Development: The AI Engineer MindsetAI 工程师心态：接受概率，用可量化标准替代“感觉”

This article argues that building with LLMs requires a fundamental mindset shift from deterministic to probabilistic systems. The author introduces two concepts: the 'Vibes-Only Trough' (where demos appear to work but lack real-world coverage) and the 'Data-Driven Slope' (where systematic evaluation and user data guide improvements). Success criteria must be specific and measurable, such as 92% agreement with human labels for sentiment analysis, 95% of classifications within 500ms, and 90% match on high-confidence predictions. The key practice is collecting data from every user interaction - starting with simple thumbs up/down feedback and evolving into richer pipelines. The article honestly states that first versions usually fail due to insufficient data, and that performance volatility is inherent to probabilistic systems. The trick is tracking changes systematically and building a culture of iterative improvement based on real metrics. Aimed at engineers moving LLM apps to production and teams needing an evaluation-first culture.

x.com · 6 min

The AI Engineering Skills MapAI 工程技能图谱：从 1 万条招聘数据提炼的四大核心技能

In this post, Andrew Ng unveils the AI Engineering Skills Map, synthesized from 10,000+ job postings, dozens of expert interviews, and surveys. It identifies four core skills: building and deploying AI applications, software engineering fundamentals, using coding agents, and shaping the build. Because AI systems produce unpredictable outputs, developers must rely on statistical evals and error analysis. Strong software fundamentals help you steer agents effectively, while the rise of agentic coding shifts engineering value from implementation to product sense and tradeoff decisions. A useful guide for developers setting priorities and employers hiring AI-capable engineers.

Section 02

Taming Probability: Boundaries, Evals, and the Optimization Ladder

3 / 15

www.aihero.dev · 8 min

What Can You Use LLMs For? Four Use Cases and a Rule of ThumbLLM 能做什么：四个用途与一条确定性原则

This article is a practical guide to where LLMs actually shine. It walks through four common use cases: converting unstructured data into structured tables, labeling and classification, question answering, and agents that take actions in the world. It also warns against shipping naive chatbots, pointing out that guardrails are never perfect and citing the infamous Gemini incident where the model told a user to 'please die'. The core argument is a simple rule of thumb: if a system can be built deterministically, it should be. LLMs are only worth the complexity for tasks that are either too expensive for humans to do at scale or too ill-defined for deterministic code to handle. Examples include 19th-century hawk migration logs, The Prompt Report's suicide-risk classification case, and the DeepResearch pattern.

www.aihero.dev · 9 min

Your App Is Only As Good As Its EvalsAI 应用的质量上限，取决于你的 Evals

In LLM apps, inputs no longer map to deterministic outputs—any small change can remake the entire behavior. This post argues that evals are the AI engineer's unit tests, the primary way to impose predictability on a probabilistic system. It walks through three evaluation types: deterministic assertions, human review, and LLM-as-a-judge, noting the cost trade-offs and recommending split cadences (fast local set vs. daily full set). It then explains the data flywheel: converting user downvotes into new eval cases and feeding them back into the system. Tooling compares Braintrust, a cloud eval platform, with Evalite, the author's Vitest-based local runner. Useful for engineers moving LLM prototypes to production.

www.aihero.dev · 23 min

17 Techniques for Improving Your LLM-Powered App从改提示词到微调：一条按成本排序的 LLM 应用优化阶梯

This guide walks through 17 techniques for improving LLM-powered applications, deliberately ordered from cheapest to most expensive on what the author calls the Staircase of Complexity Hell. It starts with prompt tweaks — role prompting, XML tags, structured outputs, chain-of-thought, multishot, temperature, and tool calling — before moving to system-level patterns: RAG, chunking, agentic loops, parallelization, evaluator-optimizer, LLM routers, and fine-tuning. The core advice: work down the staircase only after simpler options are exhausted. Concrete evidence includes Claude 4.6 returning 400 on prefilling, parallel processing cutting 10-document analysis from ~10s to 2-3s, routers bypassing the common ~30-tool limit per model, and fine-tuned smaller models beating larger ones on specific tasks. Each section names real trade-offs, such as CoT increasing latency and agentic loops adding decision overhead. Useful as a practical checklist for engineers iterating on LLM systems.

Section 03

The Model Arena: Selection Frameworks and This Week's Price War

4 / 15

www.aihero.dev · 8 min

5 Questions to Ask Before Choosing an LLM模型选型五问：开源与闭源、Token 成本、延迟、上下文窗口怎么权衡

Choosing an LLM is not a one-time decision—it must be revisited as models and your app evolve. This guide breaks the choice into five questions: open vs. closed source, cost, latency, performance, and context window. Open models require self-hosting or an API provider like Hugging Face or Groq; closed models are hosted by vendors and priced per token. Latency should be measured with TTFT and TPOT. Benchmarks like Chatbot Arena and Open LLM Leaderboard are useful early indicators, but prone to overfitting—the only real test is running your own evals within your application. Reasoning models such as o1 excel at planning-heavy tasks but cost more and respond slower. Context windows count both input and completion tokens, which is why RAG chunking exists. A practical primer for developers starting model selection.

x.com · 6 min

Grok 4.6 and DeepSeek V4 Pro: near-Fable power, far lower priceGrok 4.6 与 DeepSeek V4 Pro 同日实测：准 Fable 级能力，价格打到底

A heavy Agent user shares a first-hand model-selection update after DeepSeek V4 Pro and Grok 4.6 launched within hours of each other. Key data: DeepSeek V4 Pro costs $0.87 per million output tokens, roughly 1/57 the price of Claude Fable 5, scores 87.9 on Terminal Bench 2.1 versus Fable 5's 88.0, and lifts DeepSWE from 12.8 to 62.7. Grok 4.6, priced at $2/$6 per million tokens, finished most development tasks in under 20 minutes; four hours of heavy use consumed only 2% of the author's weekly SuperGrok Heavy quota. The post also admits the impossible triangle is not fully broken: DeepSeek still lacks multimodal support and is not fast, while Grok lags Fable 5 on complex engineering and agent work. Useful for engineers comparing model cost, speed, and agent capability.

x.com · 10 min

Grok 4.6 Field Guide: Verification Loops Beat Long PromptsGrok 4.6 使用指南：短提示词加验证循环胜过冗长规格

The author used Grok 4.6 as a daily driver for weeks across coding and knowledge work, running side-by-side comparisons against 4.5 with identical prompts. Key finding: short prompts plus a clear preference match two-page specs, while adding a single sentence demanding post-implementation verification and iteration had the highest leverage. 4.6 performs steadily on browser automation, visual QA, inbox triage, and editing a real Excalidraw codebase, but 3D and video work still need human oversight because a screenshot cannot confirm time-based behavior. Probably written by an xAI team member, so treat the launch framing skeptically; the methodology and prompt examples are useful for engineers working with AI coding agents.

z.ai · 22 min

GLM-5.3: Post-Training Scaling Boosts Coding and CyberGLM-5.3：只靠后训练扩展，代码与网络攻防能力同步提升

Z.ai announces GLM-5.3, claiming all gains come from post-training scaling on the same base model as GLM-5.2. The release post details environment-synthesis pipelines, reward verification, and the slime RL stack. Public coding/agent benchmarks improve markedly: Terminal-Bench 3.0 rises from 4.6 to 28.3, DeepSWE v1.1 from 46.2 to 66.9. The standout claim is an "emergent cyber capability": ExploitBench jumps from 24.4 to 54.4, and real-world testing across 269 projects found 2,436 vulnerabilities, the oldest from 1981, now tracked in a public disclosure ledger. On the systems side, slime adds local storage caching, 1e-7 training-rollout logprob alignment, and workload-aware scheduling, yielding 2.3x throughput for long-horizon coding RL. The API removes thinking disabled and introduces reasoning_effort low/high/max. Weights arrive in two weeks after safety hardening. All scores are self-reported, and closed models still lead on several cyber and coding suites.

Section 04

The New Shape of Agent Infrastructure: From Vocabulary to Workspaces

4 / 15

www.aihero.dev · 10 min

AI Coding Dictionary: A Vocabulary for Agent EngineeringAI 编程词典：为 Agent 工程术语建立共识

AI Hero publishes an AI coding dictionary that condenses the working vocabulary of agent engineering into terse entries. Instead of merely defining terms, it draws the conceptual lines that matter in practice: parametric vs contextual knowledge, the stateless model vs the stateful harness, input/output/cache tokens, and handoff mechanisms like compaction and handoff artifacts. It also introduces a few useful distinctions of its own — progressive disclosure as a way to load context on demand, Spec and Ticket as the building blocks of multi-session work, and DX/AX as separate measures of how well an environment serves humans and agents. Each definition is kept to a sentence or two, making the dictionary a useful reference for engineers debugging context-window waste, permission modes, or agent memory. For teams standardizing on terms like MCP, AGENTS.md, or subagents, this is a solid shared glossary. The piece is a lead magnet for AI Hero's newsletter, so treat it as a reference rather than a deep investigation.

github.com · 23 min

Cloudflare OS: an AI workspace with per-user sandboxed apps企业级 AI 工作台：每个应用都是可让 AI 改代码的私有沙箱

Cloudflare OS is an open-source AI productivity environment that Cloudflare built for its own workforce. It acts as an operating system for both company operations and AI workloads: instead of fixed document types, every file is a 'Gadget'—a private, sandboxed application that an agent can create, modify, and debug on the user's behalf. A capability-based security layer called Gatekeepers brokers access to external services, logging side effects and letting users approve queued actions in bulk later instead of blocking the agent mid-task. The entire platform runs on Cloudflare Workers and workerd, leveraging Durable Objects, Dynamic Workers, and Facets, and every Gadget has real-time multiplayer collaboration by default. Blueprints let users share the code of an app so others can spawn their own copies. It is early access but substantial, and a valuable reference for engineers exploring agent infrastructure, per-user app sandboxing, and edge-based AI tooling.

blog.cloudflare.com · 16 min

Cloudflare OS: an open source agent workspace with capability-based governanceCloudflare OS：让每个员工拥有带权限治理的 Agent 工作台，开源可自部署

Cloudflare has open-sourced Cloudflare OS, the agent workspace it uses internally, to bring AI leverage beyond programmers. A workspace is grounded in company-curated context and skills, and starts each agent with zero access. Resources are exposed as typed bindings, while Gatekeepers—service-specific Workers—hold credentials, enforce policy, and log every observed resource. Sharing a dashboard does not become a way to leak an underlying table, because the platform re-checks the viewer against the observation log. Apps are full-stack Workers with their own SQLite state, and the same server methods are callable by both users and agents. Model routing and spend run through AI Gateway, so admins can set budgets and pick models per task. The code is on GitHub and deployable into a Cloudflare account, with a reference deployment included. Written for engineers working on agent infrastructure, access control, and internal development platforms.

x.com · 7 min

DeepSeek Harness: Everything Is a Plugin on the Cordis KernelDeepSeek Harness 速通：一切皆插件，Agent 在运行中插拔自己

DeepSeek launched DeepSeek Harness right after V4 Pro, built around the idea that everything is a plugin. The core, Cordis, only handles plugin loading, unloading, and dependency management, allowing hot-swapping during agent runs; an 88-page paper describes temporal and spatial composability. UI, tools, skills, storage, and the agent loop are all exposed as plugins, with 100+ first-party plugins and a community plugin marketplace. Setup is a single `npx @deepseek-ai/dsh web` command, and the harness is not locked to DeepSeek models—custom providers, base URLs, protocols, and model lists are supported. The post walks through four modes: Standard, PTC (packing multiple tool round-trips into one run_code), Minimal, and Create, where the agent can inspect its own Cordis environment, build a plugin, and attach it mid-run. Sessions are append-only event logs, making failure traces observable, auditable, and reproducible. It also flags steep price increases for V4 Pro (cache hits up 12x, peak output ¥27) and admits the UX is rough for non-developers. Useful for engineers interested in agent-harness design and composability.

From Vibes to Systems: The AI Engineering Value Shift

Redefining Engineering: The Cognitive and Skill Foundations of AI

Taming Probability: Boundaries, Evals, and the Optimization Ladder

The Model Arena: Selection Frameworks and This Week's Price War

The New Shape of Agent Infrastructure: From Vocabulary to Workspaces

Mondays · one email