用奖励函数替代标注数据：GRPO 将 Qwen3-8B 的 JSON 结构输出有效性从 62% 提升至 82%

You ask a language model for JSON, and most of the time it works.

Then one response comes back broken. A number shows up as a string, or the model wraps the JSON in a line of explanation, and the parser chokes.

That small failure rate is the whole problem. The output feeds a function call or a database write, and looking like valid JSON is not the same as being valid JSON.

The code downstream only finds out which one it got when it breaks.

That is what structured output means. The model returns data in a fixed shape that matches a schema, not free text that happens to read right.

Agents, tool calls, and data pipelines all run on this now. The model is writing for code to run, not for a person to read.

Getting this reliable is the hard part, and the usual answer is to give the model more. You add examples, tighten the prompt, and fine-tune on correct outputs.

It helps a little, then stops. The ceiling was never the amount of data. It is the objective.

DeepSeek-R1 showed a way around this. Training a strong model used to mean annotation pipelines, preference pairs, and a team of labelers.

DeepSeek replaced all of it with one Python function that checks whether an answer is right. If you can define correctness via code, you do not need the rest.

That is the idea behind GRPO. Instead of learning from examples, the model learns from a reward function you write.

For each prompt, it generates a few candidate answers. The reward function scores them, and the model is pushed toward the ones that score higher.

In this walkthrough, I use it to fine-tune Qwen3-8B for JSON extraction. The loop runs from a local notebook while the model trains on remote H200s.

The reward function does one thing: it checks whether each output parses and matches the schema.

Schema-valid output went from 62% on the base model to 82% after training, past GPT-4.1 on the same eval at 58%.

Before the build, it helps to see why the obvious approach stalls. That is what makes everything after it click.

SFT learns by copying examples. Show it correct completions, and it gets good at producing output that looks like them.

But looking like valid JSON and being valid JSON are different goals. SFT only ever chases the first one.

The loss is measured token by token. A completion with one field typed wrong scores almost the same as a perfect one.

So you add more examples. The number ticks up, then flattens, because the limit is the objective, not the data.

Once you see the problem, the fix is clear. You define correct in code, and train the model against that definition directly.

This is what GRPO does. It swaps labeled examples for a reward function.

For each prompt, the model generates a small group of answers, usually four to eight. Your reward function scores every one of them.

The scores are normalized inside the group. The update then reinforces the answers that scored above the group's average.

So the model is always comparing its own outputs against each other. It learns what "more correct" means for your task, not what "more similar to an example" means.

Here is how the reward function scores three different outputs for the same prompt.

• Output that doesn't parse as JSON scores 0.0.
• Output that parses but fails the schema scores 0.5.
• Output that parses and matches the schema scores 1.0.

That middle score matters more than it looks. Without it, valid JSON with the wrong field types would score zero, the same as complete garbage.

The model would lose an important signal, that valid structure is already progress. The 0.5 is what gives training something to climb toward.

GRPO is heavier than SFT. On an 8B model it needs H200s and runs for hours.

Every step, it generates several completions per prompt, scores all of them, and updates the weights. That repeats across the whole dataset, many times over.

This is not something you run on your laptop.

There is also a timing problem SFT never has. During rollout, the model samples answers from its current weights. During training, those same weights are changing underneath it.

If the inference side and the trainer fall out of sync, you sample from a stale model and train on answers the current one would never give. This is where most custom RL setups fall apart.

Fireworks' Training API handles both sides. You write the training logic in Python on your own machine.

Their infrastructure does the rest. It provisions the GPUs, runs the forward and backward passes, saves checkpoints, and resyncs the inference deployment after every step.

The setup is three steps. You write the reward function, upload the dataset, and configure the run.

Let's go through each one.

This is the only place your task is defined. The schema says what a correct output looks like, and score() checks each completion against it.

For invoice extraction, I pull four fields from raw text: vendor, date, amount, and currency.

import json
from jsonschema import validate, ValidationError

SCHEMA = {
    "type": "object",
    "required": ["vendor", "date", "amount", "currency"],
    "properties": {
        "vendor":   {"type": "string"},
        "date":     {"type": "string"},
        "amount":   {"type": "number"},
        "currency": {"type": "string"},
    },
    "additionalProperties": False
}

def score(completion: str) -> float:
    try:
        parsed = json.loads(completion.strip())
    except (json.JSONDecodeError, ValueError):
        return 0.0
    try:
        validate(instance=parsed, schema=SCHEMA)
        return 1.0
    except ValidationError:
        return 0.5

jsonschema handles the type checks, the required fields, and any nested rules in a single call.

For a different task, like SQL or tool-call formatting, you swap in a new schema. The score() function stays the same.

GRPO does not need labeled outputs. The dataset is just the prompts you would send in production.

The model writes its own completions during training, and score() grades them as they come.

I used 200 training prompts. They cover different vendor names, date formats, amount styles, and currency codes.

I also set aside 50 eval prompts that the model never trains on.

Variety matters more than volume here. Prompts that all look alike produce a model that breaks on real invoice variation.

{"messages": [{"role": "user", "content": "Extract the following fields from this invoice:\n\nBill from Acme Corp, dated 2024-03-15, total $1,250.00 USD.\n\nReturn valid JSON only."}]}
{"messages": [{"role": "user", "content": "Extract the following fields from this invoice:\n\nReceived from TechSupplies Inc on January 8 2024, amount due: 340 euros.\n\nReturn valid JSON only."}]}

Upload the dataset and wait for the READY state before the training job can use it.

from fireworks import Fireworks

client = Fireworks()
client.datasets.create(dataset_id="invoice-extraction-grpo-v1",
                       dataset={"exampleCount": "200"})
client.datasets.upload(dataset_id="invoice-extraction-grpo-v1",
                       file="./train_prompts.jsonl")

# poll client.datasets.get(...).state until "READY" before proceeding

rl_loop runs the whole thing. It provisions the trainer, schedules the rollouts, syncs the weights, and cleans up when the run is done.

You connect your score() function by wrapping it and assigning it to rl_loop.reward_fn. The wrapper gets both the completion and the dataset row, so you can reach ground-truth metadata if your reward needs it.

from training.recipes.rl_loop import Config, main
from training.utils import DeployConfig, InfraConfig, WeightSyncConfig
import training.recipes.rl_loop as rl_loop

# Wire your reward function to the training loop
def invoice_reward(completion: str, row: dict) -> float:
    return score(completion)

rl_loop.reward_fn = invoice_reward

cfg = Config(
    base_model="accounts/fireworks/models/qwen3-8b",
    dataset="invoice-extraction-grpo-v1",
    max_rows=200,
    epochs=1,
    completions_per_prompt=4,
    max_completion_tokens=256,
    temperature=1.0,
    max_seq_len=4096,
    policy_loss="grpo",
    output_model_id="invoice-extractor-v1",
    infra=InfraConfig(
        training_shape_id="accounts/fireworks/trainingShapes/qwen3-8b-128k",
    ),
    deployment=DeployConfig(
        deployment_id="invoice-extractor-v1",
        tokenizer_model="Qwen/Qwen3-8B",
    ),
    weight_sync=WeightSyncConfig(weight_sync_interval=1),
)

main(cfg)

A few of these settings are worth a note.

• dataset points to the dataset_id you uploaded in Step 2. Fireworks pulls it from their storage directly.
• completions_per_prompt=4 sets the group size for GRPO. Production runs often use 8 to 16, which gives more signal per step at higher compute cost. Four is enough here. The reward is clear-cut enough that even a small group shows real variance between answers.
• weight_sync_interval=1 resyncs the inference deployment after every step. That keeps rollout sampling from the exact model being trained. Production runs often set this to 4 or 8 for speed. For a short 200-step run, 1 gives the tightest feedback loop, which is what you want.
• One Qwen3 quirk to handle. It defaults to thinking mode and adds thinking... response blocks before the answer. Strip them at eval time with content.split(" response")[-1].strip(). Suppress them in training by adding /no-think to the system prompt. Otherwise the reward function reads the reasoning text instead of the JSON, and scores everything 0.0.

Base Qwen3-8B scores 62% schema-valid on the 50 held-out prompts. After GRPO training on Fireworks H200s, the fine-tuned model hits 82%.

That is past GPT-4.1 on the same eval, which lands at 58%.

Here is the baseline run first, on the 50 prompts the model never trained on.

And here is the same eval after training.

The trained model runs on a Fireworks serverless endpoint, at a fraction of GPT-4.1's per-token cost. Latency is lower too, since the output is short and predictable.

The real difference shows up on messy inputs. A prompted general-purpose model starts to slip, while the trained model holds, because it learned the constraint instead of the examples.

This pattern works for any task where you can check correctness in code. SQL that has to parse, API responses that must match a shape, tool calls, code that has to pass a linter.

If you can score an output, you can train a model to chase that score.

What DeepSeek-R1 proved at frontier scale holds for your own small task. The model you get has practiced your definition of correct, not memorized examples of it.

On inputs it has never seen, that is the difference that holds up.

The full code is in the repo below. That includes the reward function, the training config, the dataset builder, and the eval script.

Training API docs → Finetuning Code →

(don't forget to star 🌟)

Thanks for reading, and to Fireworks for partnering on today's article.