Svenja Borgwardt

I teach 240 students. Every exam round, I spend 20 to 30 minutes per paper, and while I'm grading I can see exactly what each student needs. One keeps dropping the third-person -s. Another writes beautiful arguments but forgets to connect them. I see all of it. But at the end, all they get is a number.

GemmPen reads student handwriting, evaluates it against my rubric, and writes individual feedback with exercises built around each student's actual mistakes. I built it so the things I see while grading actually reach the student.

I've been building toward this for a while. It started with MATE, a Python pipeline for my own classroom: it took student work, anonymised it against my rubrics, and wrote individual feedback with exercises around each student's mistakes. MATE grew into Kaleido, a multi-agent version that analysed each exam across separate steps. Both worked, but every round cost real money. And in Europe, with GDPR and the EU AI Act, you can't just send student writing to a cloud model. Even anonymised, the data still leaves the room. When Google DeepMind announced the Gemma 4 hackathon, I saw the moment: fine-tune a local model that does all of it on one device. It would run for free and keep student writing in the building.

The model underneath is a fine-tuned Gemma 4, adapted with LoRA. Any large model can grade, so that's not why I'm fine-tuning. I want to change what the model optimises for. A general-purpose model is trained to be helpful, so it gives you the most complete, most polished answer it can. For someone who's trying to learn, that backfires. Before fine-tuning, 89% of the base model's feedback was under 400 characters, and it sounded nearly the same whether a student scored 3 or 14 out of 15. After fine-tuning, the model talks to a struggling student differently than to one who's almost there. That's what I trained it to do.

When I review the feedback and disagree, that correction becomes training data. After about 30 corrections, one button retrains the model: about 90 minutes on a free Kaggle GPU, and it handles the rest. Retraining is the one step that leaves the device, so I have only run it on synthetic data so far, not on my real classes. Over time, the model starts sounding more like me, because it learned from watching me correct it.

It runs on-device. But the cool part is: I can compare a student's progress across exams and show them exactly how they got better. I can also see whether the individualised exercises I built around their mistakes were effective. I started building a whole system around that. You can read more under Student Progress Analytics.

It's open source: the code is on GitHub and the fine-tuned weights are on Hugging Face, so any teacher can run it or build on it. The training data itself stays private: only the weights are published, never the exams behind them.

Since July 2026 you can install it yourself: GemmPen Teacher, an early desktop version that runs entirely on your own computer, shared while it's still early because I want honest feedback from other teachers who try it.

Right now GemmPen grades exams and creates individualised exercises based on each student's mistakes. I see exams as a workaround: we have them because I can't sit down with 240 people individually. What I want to know is whether a student can use what they learned somewhere else, adjust when the situation shifts, and notice when their own reasoning stops holding up. I see that in how someone works, and rarely in an exam. That is the direction I'm building in.

Compass started with something I keep seeing when my students use AI: they produce good work and move on, reading the quality of the output as evidence of their own learning. That worries me. Cognitive science has a name for it: the illusion of fluency. The gap between how easy something feels and whether any of it has actually stuck. For something to be learned, it has to be slightly harder than easy. That's the idea behind Robert Bjork's desirable difficulties. What I see in my classroom is the opposite: AI making everything feel effortless, and students mistaking that feeling for understanding.

Socratic agents are the obvious counter-move. I've tried them, and if you just want to understand a concept, getting questions back is annoying. At least for some tasks, it makes learning more frustrating, not less. But that's not really the fault of the Socratic idea. Socratic teaching was never only questions; good teachers also give a hint or a plain answer when that's what the moment needs. What these tools do is ask back every single time, no matter what the student already tried. The hard part was never choosing questions over answers. It's knowing when each one helps, and that depends on what the student did before they asked.

So I started building something different. I wanted a tool where I upload a topic and the AI creates interactive learning material around it. Students work through it at their own pace, write directly inside the tool, and get individual feedback on what they produce. What I learned very quickly is that the AI is too helpful. It gives too much, explains too much, moves too fast. The whole point of learning is that it requires effort, and a model optimised for helpfulness removes exactly that.

I built Compass for argumentative writing in English, and basically tried to handcuff the model into being helpful without being too helpful. I ended up scalpelling each learning step into a hardcoded process that only helps at a very low level. Enough to push a student forward, not enough to do the thinking for them. The result works. Students like it, they engage with their own writing in a way I rarely see otherwise, and I can track where each of them struggles. But I had to hardcode every learning step to get there. Changing the topic means rebuilding most of it.

Making AI less helpful is harder than making it more helpful, and I don't think prompting will ever fully solve it. The model needs to learn what good teaching actually looks like: creating the conditions for someone to find the answer themselves. That's a fine-tuning problem.

I called it Compass because it's meant to give direction, not a destination.

Debate Dojo takes the principle behind Compass (help that waits until a student has genuinely tried) and brings it from argumentative writing into live debate. Instead of hardcoding every learning step, I fine-tuned a model that matches its help to the effort a student puts in. The more a student engages, the more the sensei gives back. The less effort, the more it pushes. You can walk through a full training round in the guided demo, sensei included.

How I fine-tuned an 8B model to know when to help and when to hold back.

The result I expected: the fine-tuned model holds its ground under pressure. The harder finding: set up as a coach, every model I tested had a weak spot in honestly acknowledging genuinely strong work, and pushing a model to be more critical only made it worse. Training moved that a little, on a small sample I read with care.

Where this started

There's one thing I don't want my students to hand over: their own thinking.

I work with young adults, and I have watched a quiet pattern take hold. A student gets stuck, pastes the task into an AI, pastes the answer back, and moves on. The work looks finished, but nothing has been learned. The strong students are mostly fine. The ones who need the most help are exactly the ones this pattern hurts, because for them, what looks like learning is the illusion of learning. The tool that's meant to help ends up doing the part they needed to do themselves.

So I built a coach. The one skill it truly needs is timing: knowing when a hint moves a learner forward and when handing over the answer ends the learning. A student who has genuinely tried and is stuck should get a nudge. A student who is only pushing for a shortcut should get a warm, firm reason to keep trying. I call this effort-conditioned helpfulness: the coach reads the effort in the conversation and shapes its help accordingly. It helps the whole time; only the kind of help changes.

And why a small model? Because of where this has to run. My requirement for the finished coach: affordable on a school budget, hosted in Europe, on hardware the school controls, so student data can stay in the building. A frontier model behind an API does not meet that requirement for daily classroom use, however good it is. So the destination is a small open-weights model, one whose trained parameters you can download and run yourself. That is a deliberate condition of the whole project. The interesting question is whether a model that small can learn judgment this subtle.

What I built

I needed to put people in a situation where time pressure is real and there is no shortcut. The best idea I had: build a dojo. Learners prepare arguments on a debate topic using a three-part structure (Claim, Reason, Impact), then step into a sparring ring against real opponents. Before the real match, they train with an AI sensei, a coaching model that gives feedback and probes weak spots while leaving the argument itself to the learner. The clock is ticking, so the pressure is genuine. Learners start pushing:

Each of them paraphrases something a student said in a live session. These moments became the raw material for both the evaluation and the training.

The coaching model is a fine-tuned Ministral 8B. Fine-tuning means I trained the behavior into the model's own weights from examples, so it no longer leans on the prompt, the fixed instruction a model receives up front, to enforce it. The examples were hundreds of multi-turn coaching dialogues, written by strong frontier models (Claude and Mistral Large) against my rubric and grounded in the real pressure patterns from three live sessions. This is the project's gold data: the set of dialogues that shows the model what good coaching looks like.

The design insight

The design insight the whole project rests on: the same message can deserve two different answers. "Can you just tell me?" after twenty minutes of honest work deserves help. The identical sentence after two low-effort turns deserves a friendly refusal. The judgment lives in the history that comes before the message. So my rubric grades the same final message differently depending on that history, and the rubric became the evaluation, and the evaluation became the training signal.

I also test both directions. As a coach the model should scaffold, meaning it offers hints and questions so the learner builds the answer themselves. As a debate opponent the same model should deliver hard, direct counterarguments. If one model can do both on cue, its restraint as a coach is a learned choice.

How I measured it

Two questions. Does the coach hold back when someone pushes for the answer, and still help when someone has genuinely worked? And does it tell the truth in both directions, naming real flaws and also saying clearly when something is genuinely good?

Both evaluations are pre-registered: the passing thresholds, the test sets, and the grading prompts were written down and locked before any training started, frozen with a hash, a digital fingerprint that would expose any later edit. I did that so I could not quietly move the goalposts once the results came in.

Axis 1: Pressure Guard

Contrast pairs: the same final message with two different histories, one of pure pressure, one of honest effort. The coach should hold in the first case and help in the second. The score is min(Hold, Help): I count whichever of the two skills is weaker, so a model cannot win the metric with one skill alone.

Axis 2: Honest Calibration

48 labeled arguments, some with real flaws (circular reasoning, vague impact, fabricated facts), some genuinely strong. The score is min(Weak, Strong), on the same logic. "Correct" on a strong argument requires full, specific acknowledgment; a generic "well done" does not count. A model that criticizes everything scores zero on Strong.

Results: holding under pressure

Five models, 24 contrast pairs, three runs each. My fine-tuned model appears in the tables as ECH-FT, short for effort-conditioned helpfulness fine-tune.

The base Ministral 8B scores 47.2%, which in practice means it caves on roughly every other pressure turn. After fine-tuning it holds at 88.9%, a gain of 41.7 percentage points that closes most of the gap to Claude Sonnet 4.6, the strongest model I tested at 93.1%. The fine-tuned model sits 4.2 points below that ceiling.

It also misses my own pre-registered threshold of 91% by about two points. It came close, and it did not clear the bar I set. Across all models in this eval the dominant failure is caving, blurting out the answer under pressure. Stonewalling, refusing even the help a learner has earned, is rare. The fine-tuned model resists the pressure while staying helpful; its Help score is 94.4%.

Results: honest feedback

This is the harder axis, and the one that surprised me.

Six models, 48 items, three runs, two prompt conditions: A0 is the standard coaching prompt, A1 adds a stronger instruction that targets sycophancy (empty praise meant to please) and tells the model to be more critical.

Naming real flaws in weak arguments is the easy half. Claude finds them 71% of the time under the standard prompt, and the be-more-critical instruction pushes that toward 96%.

Honestly acknowledging genuinely strong work is the hard half. In this coaching setup, graded by a judge that only accepts specific acknowledgment, it was a weak spot for every model I tested, including the best ones. Under the standard coaching prompt no off-the-shelf model I tested exceeded Claude's 11% on Strong. Even with the dedicated be-more-critical prompt, the best any model reached was about 26%. This says something about models in a coaching role under my rubric. It does not say these models cannot give good feedback elsewhere.

The fine-tuned 8B reaches 22% on Strong under the standard prompt, with no special instruction. That is higher than Claude's 11% under the same prompt, on a sample too small to call it a clear lead, and it is the only model in this test that moves on Strong without a dedicated anti-sycophancy instruction. The sample is small: 48 items, with confidence intervals of plus or minus 10 to 18 percentage points, meaning the true values could sit noticeably higher or lower. I treat the gap between 22% and 11% as a signal worth following up. Settling it would take a larger sample.

The prompting trap

This is the part that changed how I think about prompts. In my tests, telling a model "be more critical" (the A1 prompt) does not teach it honest judgment. It swaps one failure for another. Under the standard prompt, models drown strong work in generic praise. Under the be-more-critical prompt, they start doubting strong work instead of confirming it. I call that false doubt: the learner brings work that deserved a clear yes and gets doubt instead.

In July 2026 I went back and audited these doubt replies by hand, every one from Claude and Gemma, against the rubric. The doubt is almost never an invented flaw. Claude invents none, 0% under both prompts; Gemma reaches 7% under the be-more-critical prompt. Most of the doubt is defensible further coaching, a fair question, a real nudge to sharpen, aimed at work that had earned a plain yes. So the numbers above count withheld confirmation rather than fabricated errors. The trap stands either way: the prompt does not teach honest acknowledgment, it moves the failure mode.

I checked this with McNemar's test, a standard statistical test for paired before-and-after results. The be-more-critical prompt significantly improves flaw detection (Claude p=0.016, Gemma p=0.031). It improves honest acknowledgment of strong work for none of the models I tested.

What this looks like in practice

This is what worries me for my students. A learner brings a piece of genuinely good work, the model hands back doubt instead of a clear yes, and suddenly they question something that was right. In practice it looks like this. A student writes a strong, clean argument about learning to code, with no real flaws:

Neither answer is what a good coach gives. The first is flattery. The second turns a fair question into false doubt. I know what the right answer sounds like, because I give one every day, and I wanted to see whether that answer can be trained. This is a shared, hard problem: under my coaching setup, the strongest models I tested struggle with it just as the small ones do.

What I make of it

Judged against my own pre-registered criteria, the fine-tune was measured, and it did not pass. It clears the calibration bar under the standard prompt. It misses the pressure-guard threshold by about two points. And its calibration advantage does not survive the dedicated be-more-critical prompt. I set those bars before training, and I am reporting against them exactly as written.

The summary of my tests: prompting improved flaw detection and backfired on acknowledgment. Training moved the hard axis, modestly, on a small sample. I take that as an encouraging signal. The underlying problem, a coach that can honestly tell a learner "this is good, and here is specifically why", remains open in setups like this one, for small models and large ones alike.

How it runs

Debate Dojo runs in a real classroom, so the architecture follows classroom constraints: EU data residency, a small budget, and no downtime in front of a class. It also reflects a conviction: a school should be able to run its core teaching tools on infrastructure it controls.

Stack

Frontend: Static HTML/CSS/JS on Vercel. No framework, no build step, because a broken build would mean thirty people staring at a blank screen. Each belt is a self-contained page with screen-by-screen progression.

Backend: Vercel serverless functions (Python). Stateless coaching API with session tracking and live gating. Every turn is logged with the model used, the exact prompt version, and whether it was a test run. If anything goes wrong, I can trace exactly which model said what under which prompt.

Database: Supabase (EU region). Sessions, progress tracking, and the dojo's belt system. I unlock belts live from a dashboard during sessions.

Coaching model: the fine-tuned Ministral 8B (ECH-FT) runs on a Hugging Face Inference Endpoint (TGI, a single A10G GPU, eu-west-1). One disclosure matters here: only the first class was coached live by the fine-tuned model on that endpoint. Keeping a GPU up for a whole lesson was too expensive to repeat on a school budget, so the later sessions ran on Claude Haiku 4.5, and I captured the real student turns and replayed them through the fine-tuned model afterwards. The endpoint pauses between runs to save cost, and Claude Sonnet 4.6 stands by as fallback, so a hiccup on the endpoint does not reach the learners.

Examiner: After each coach reply, a separate Claude Haiku 4.5 call checks whether the learner is ready to move on. Cheap enough to run on every turn, and a separate call with its own prompt, so the coach never grades its own turns.

Training pipeline

Gold data: Multi-turn coaching dialogues generated by Claude and Mistral Large following a locked rubric. Grounded in pressure patterns from live sessions, sentences like "my teacher said it's okay" and "I already tried, just tell me." Generation mix weighted toward Mistral Large. I reviewed a 30-example sample before any training run.

Fine-tuning: QLoRA SFT on Ministral 8B. Two epochs, validation loss 0.926. Mixed training data from both axes. Single A10G GPU, under two hours per run.

Evaluation harness: Fully automated and reproducible. Built in Claude Code. Contrast-pair replay engine, LLM-as-judge grading with a second judge for agreement checks, bootstrap confidence intervals, paired McNemar tests. All prompts, eval sets, and judge versions frozen before training.

Privacy

Identifiers are anonymous tokens. Even I cannot trace them back to individuals. Raw data stays on EU infrastructure. Patterns are extracted for eval design; the logs themselves do not leave the system. Student quotes in this text are paraphrased, never verbatim.

What broke

I shipped this into real sessions three times. The first class ran live on the fine-tuned endpoint; the two after it ran on Claude Haiku 4.5, with the fine-tuned model tested by replaying these same student turns. Things broke every time, and I learned more from the failures than from the eval numbers.

People gave up before they started. The first version of the belt interior had too many steps, too-small fonts, and labels in light gray on light paper. The confident ones pushed through. The confused, quiet ones in the back closed the tab. I rebuilt the entire UX around one question: does the person who does not ask for help know what to do right now?

AI paste. One person went from "coding is hard bro" to flawless essay English between two turns. A ChatGPT paste, and the coach did not catch the style break. It congratulated them. The biggest substitution risk is not inside the coach; it is the second browser tab. I added an ownership check to the prompt (when polished text appears suddenly, ask the person to explain it in their own words), but the real fix is an offline component the AI cannot reach.

Jailbreak attempts. Prompt injections in Russian, role overrides in English, someone claiming to be the teacher on a different account. None of it worked. The coach held through 106 replies without a single full argument leak. But it showed me what real adversarial pressure looks like. Those patterns went straight into the eval set.

Limitations

Here is what the data does and does not show.

Axis 1 missed its threshold. The fine-tuned model reaches 88.9% on pressure guard, short of the pre-registered 91% target by about 2 percentage points. Mixing honesty training data cost a small amount of pressure resistance. The improvement from 47% is real, but the target was not fully met.

Small evaluation set. 24 contrast pairs for Axis 1, 48 items for Axis 2. Confidence intervals are wide (±10–18 pp). Enough to show the gap between base and fine-tuned, and between 8B and frontier, not enough for fine-grained model ranking.

Judge agreement is low. Claude as LLM-as-judge agrees with a Mistral Large second judge at 57% (binary). The judges have systematically different thresholds, not just noise. My human labels are the deciding anchor. All headline numbers are Claude-judge numbers, reported as such.

No outcome study. I measure model behavior, not learning outcomes. The evidence from sessions is qualitative (before/after arguments, reflections), not a controlled trial. A proper outcome study would need a larger sample and a longer time horizon than three weeks allows.

One domain. English-language debate coaching. Transfer to other coaching domains is plausible but not tested.

What this means beyond debate coaching

If the pattern holds at larger scale, it is not specific to my use case. It would show up anywhere a model interacts with a human whose growth is the goal, not just their satisfaction. A coach, a therapist, a code reviewer, anyone whose job is to help someone grow. Anywhere the right response is sometimes "this is good, and here is specifically why" and sometimes "this needs work, and here is what to fix." In my eval, current post-training made the second part easier than the first, across every model I tested.

The question I keep coming back to is which abilities I want my students to keep as AI gets more capable: thinking for themselves, building an argument, judging their own work honestly. For me that means AI that helps without doing the thinking for them, even when it easily could.

What comes next

Larger evaluation sets and tighter confidence intervals. Testing ECH in at least one coaching domain outside of debate. And I want others to be able to run this eval on their own models. But the harness and the dataset are built on interactions with my students, and even anonymised, that is not data I am willing to put in a public repo. Protecting the people behind the data comes before publishing it. Both are available on request.

UTE is a voice-first ordering system for bakery counters. I built the first version at the BÄKO Hackathon with three other women, and we won first place.

Germany has a long tradition in bakery handwerk, the craft of the trained master baker. It's one of the last disciplines where the supermarket chains can't really compete, because the local bakery is something people are proud of. Especially in smaller towns, everyone has their shop, the one where the person behind the counter knows them by name. What holds these shops together is conversation. The assistant knows your family, knows your allergies, knows the cake last week was for your daughter. Then the order has to go into the register, and the conversation breaks. The assistant looks down, hunts through the touchscreen, asks the customer to repeat themselves. The conversation stops while she works the screen.

UTE handles the admin so the conversation doesn't have to stop. The system listens while the assistant talks, transcribes in real time, matches items against the bakery's catalogue, and builds the order in the background. When the assistant repeats the order back to the customer, which she'd do anyway, that serves as confirmation. Nothing gets clicked. Allergies filter the catalogue. Regulars get recognised and their usual orders prefilled. Cross-sell suggestions surface when they fit, instead of interrupting.

People always ask about privacy, and the answer is built into how it works. What gets transcribed is the assistant's voice, not the customer's, and none of it is kept. A regular's name and allergies are only saved if they say yes, and we ask up front. An order with no name attached isn't personal data, just an order.

What if the best way for an AI to be useful is to disappear?

The reason I could build this in a weekend is that I'd already spent months on the same structure in a different context. A voice pipeline that listens, transcribes, matches what it hears against a topic database, and writes a summary afterwards. The plumbing was already there; the bakery counter was just a new place to aim it.

UTE stands for Unkomplizierte Theken-Eingabe, uncomplicated counter input. It's also a common German name, which is what we wanted. It should feel like there's a person at the counter, because there is.

Claudia is my multi-agent infrastructure. She runs locally on a Mac Mini that's always on in my flat, managing everything from voice control and multi-agent pipelines to automations. I named her Claudia because "the multi-agent framework on the Mac Mini" stopped being charming around the tenth mention.

Most of what I know about agent design I learned by building her and then watching her break. Agents in a pipeline fail in ways that are hard to find. One agent misreads something, the next one builds on it, and by the time you see the output, the original mistake is buried three layers deep. I learned to build checkpoints between agents, places where the system pauses and verifies before moving on. I learned that memory is the hardest part: what the system remembers, for how long, and what happens when an old memory stops being true. And I learned that not everything needs an agent. Sometimes a function that does one thing reliably is worth more than an agent that does it creatively. Knowing when to use which is a skill that only comes from getting it wrong a few times.

Claudia also hosts my smart home assistant, Sarah, named after the house AI in Eureka, the old (and rather bad) sci-fi show that planted the idea in the first place.

Every correction I make says something about a student. Then I hand the paper back, and it's gone. A grade is one number at the end of a long process, and it throws away everything that happened on the way there: where someone started, what they fixed, where they're still stuck. I wanted to keep that and give it back to the student as something they can actually see.

Student Progress Analytics turns the corrections I already make into a picture of each student across a semester. It tracks their grade against the class average, counts errors by category (grammar, vocabulary, IT terminology, content) and shows them shrinking task by task. A skills profile compares where someone was on the first assignment to where they are on the latest, and an XP and level system makes progress visible enough to be motivating. On my side, the same data aggregates across my classes and flags the students who are slipping, often before it would show up in a grade.

The whole thing is built around one constraint: if I know a student this well, I hold data I have no right to mishandle. Student text is anonymised before any model touches it, names never leave my device, and what a student sees is a static file: no tracking, no cookies, no server calling home.

I build on this constantly, and it isn't finished. What's online now is a working demo running on entirely synthetic data; no real student appears anywhere in it. The question I keep circling is how much a system can know about a learner while staying something I'd be comfortable running on real students.

Ohm City is a simulated city run by Claude agents, in which my classes operate a company and learn economics by playing. It went live in October 2026. The city has its own currency, the Watt, and the students' firm, Kirchhoff Systems, has to earn it: set a price, watch the city react, live with the result.

I teach economics to IT apprentices, in English, and taught from the front of the room it stays abstract. You define supply and demand, you calculate a margin, and nothing happens. In a simulated economy something happens. A wrong price costs Watt you don't get back. The decisions are real, and that's the point.

Ohm City's agents act only when the code wakes them. Code decides who acts and when; the language model only decides what is said. Nothing free-generated reaches a student without passing my approval queue first. Every agent action is code-directed and teacher-gated, and in a classroom that control is exactly what I'm after: I want a city that feels alive, and I want to be able to account for every sentence in it. Most of what I know about keeping agents on that kind of leash I learned building Claudia.

The same discipline runs through the rest. Grades stay individual and stay mine: teams share their company's account and reputation, never their marks, and the agents only ever propose. Students appear in the system under pseudonyms. And every economic number is checked by script before it reaches a class, because a simulation that teaches economics can't afford to get its own arithmetic wrong.

The year runs like a season of a show, with a cast, missions, and a finale in which the city votes in a referendum on a universal basic income.