System One Models Will Revolutionise AI Software: Laya & Jev

September 15th was a seismic shift in the AI & tech world, that went largely undetected by the general public. As usual software engineers were on the forefront of the advancement that passed everyone else by as TypeSafe AI released the first commercial System One model, 'Jev.'

Three weeks later and I'd trained my own System One model for one of our upcoming products. This is the debrief on that experience.

Everyone watching AI right now knows the feeling. Someone messages you with a link to a new AI product on the market and you roll your eyes. "System One models?" you ask. Who cares? I've got Claude Opus and a whole lot of work to do. I don't have time for another Manus or Rabbit r1.

That's exactly what went through my head when I first heard about Jev. But people were asking my opinion, so I decided to look into it. Right upfront, TypeSafe AI's website does a very poor job of explaining the concept. You'd be forgiven for reading their release blog post and thinking that it was supposed to be a competitor to the frontier AI models from OpenAI and Anthropic (especially given the charts showing as much). They throw around words like "Reinforcement Learning for Calibrated Decisions" (RLCD), which is practically meaningless to consumers.

It took this far better article from LangChain to explain what Jev actually does. If you want the full breakdown, read that article, it's absolutely worth it. Below is an infographic from the article that blew my mind and articulated the use case better than anything TypeSafe put out.

As soon as I saw that example, the world of possibilities opened to me and my mind was flooded with solutions it provided to my problems I was facing. I recognised that it solved problems with two of our upcoming projects, Infinilang and kcalc.ai (website of the latter unfinished as of this post).

Both projects rely on LLMs to power their capabilities. I'm not the kind of person to shove AI into just anything to make a point, and that's not the kind of company I want to run. I conceptualised the ideas for these two projects independently while AI tools developed over the last 4 years, and they offer genuinely innovative solutions to problems grappled with by humans for years.

I'll give a quick introduction to these two projects so you have context, but we'll mainly be focusing on Infinilang in this article so skip the kcalc.ai part if you're short on time.

Ever tried to lose or gain weight? I have. I was in the gym for three years before I figured out that strength training was only a third of the body building picture. The other two were sleep and calorie intake. Unfortunately for me, weight gain is not something I do well, and eating more than I need to maintain my weight was too difficult.

I tried one of those calorie calculators you find on the app store. You know the kind. Track your food throughout the day and try to hit a target. The concept is great, but the execution is practically always terrible. You spend more time recording food and correcting macros nutrient balance and calories than you do actually eating. They usually offer a camera function too, which works about 20% of the time, and never gets the macros/portion size/calories right. Practically useless for homemade and non-standard food too.

Where kcalc.ai does things differently is that it's intended to reduce the friction of recording dramatically. When you snap a photo or perform a LiDAR scan, a complex 15 stage evidence pipeline processes the information. It uses a mix of LLMs, a vector database, a huge food and recipe database, vision models, 3D LiDAR scan reconstruction, multi-step reasoning agents, and a lot more.

In all practicality, the AI component of this app has been an enormous undertaking. It's no surprise that few have attempted this, because it requires an immense amount of software engineering experience and advanced systems thinking that few possess. It simply wouldn't be possible without the assistance an AI coding agent, or 10 million dollars and a year of 5 engineers time.

kcalc.ai faces a number of serious problems that needed to be dealt with to become commercially viable. The two big ones were accuracy evaluations and speed.

Let's look at a scenario.

The user uploads a picture of "Belgian Fries With Mayonnaise."

The evidence pipeline processes the image, and the combination of stages combined with multiple steps of LLM reasoning come to the conclusion that it is, indeed, Belgian fries, likely with mayonnaise. The interesting thing about kcalc.ai is, it doesn't necessarily hand back a pre-configured record to the user. It generates a record dynamically based on context and real size measurements with volume calculations. That's all fine and dandy. It works, though it takes about 2 minutes to work its way through the pipeline.

The app includes a 'World Atlas' feature that lets users collect national cuisines or "classics", and fill out their world map. As such, there's a list of cuisines that need to be checked at the end of the food identification. How do we do this? Traditionally we might've used some sort of regular expression (Regex) or similarity algorithm to search the list and find likely matches. This is the first thing I tried, and it led to consistent mistakes, the system awarding atlas entries by mixing up similarly named foods. So the next obvious choice was to pull the likely matches and have an LLM pick the most appropriate. The problem is that it took up too much compute resources and I couldn't justify blocking the calorie estimate feature for a minor extra feature.

A System One model in conjunction with a vector database provides the solution.

With a System One model, it takes milliseconds to get an accurate selection, and the resource costs are incredibly cheap in comparison to full blown LLMs. It's a lifesaver in scenarios where there are many possibly similar options.

The true beauty of the System One model interface is that it's incredibly easy to integrate. You could find hundreds of places to slot it into your codebase and it's so cheap and fast that it hardly warrants a second thought. This is just one example of the many places where I believe kcalc.ai will benefit from this new technology.

As a lifelong Japanese learner with severely limited time and (to be honest) a lack of talent with languages, I've picked up a somewhat passable grasp of the language. Enough to get around in Japan and even strike up some funny conversations in bars, but not quite enough to avoid the, "nihongo jyouzu's."

One of the trickiest parts about learning Japanese is 'kanji,' the third and most numerous set of characters that make up the Japanese writing system. Here's an example of the three character sets:

• Hiragana: ありがとうございます (romanised arigatou gozaimasu "Thank you very much")
• Katakana: コールドブリューアイスコーヒー (romanised korudo buru aisu kohi "Cold Brew Iced Coffee")
• Kanji: 人工知能 (romanised jinkou chinou "Artificial Intelligence")

Kanji is derived from Chinese characters, and have some of the same meanings, though usually completely different pronunciations. Where it gets complicated is that there are 2136 common kanji needed for functional literacy, and around 4000 kanji to approach native fluency. Making it worse is that kanji often have multiple readings, which means they're pronounced differently depending on where they're used. Pure nightmare to learn.

When OpenAI's GPT-3 first dropped, I was one of the early adopters and saw the near infinite possibilities of the technology early on. The big one for me was a tailored language learning experience, a way to learn a language more effectively than ever before.

I conceptualised three novel applications of LLMs at the time, and ruminated on the ideas for years. During that time, I developed two iterations of prototypes for the technology and proved that it does work.

That deliciously academic title was the name of a thesis I was working on for a masters degree based around the Infinilang idea. I'd originally intended to contribute these tools as open source research, but working a full-time job and resource limitations proved to be too problematic for that. So instead, I intend to bring Infinilang to market as a closed-source mobile app first. Later, I intend to release it as an open source desktop application with BYO AI capabilities.

So what does Infinilang bring to the table? For a Japanese learner, memorising kanji is only one aspect of learning to read effectively. Memorisation is currently a slow and tedious process involving spaced repetition systems (SRS). The famous ones are Anki and WaniKani, the latter being my personal favourite. SRS involves showing users flash cards repeatedly, while extending the time between each exposure progressively. The system is designed to show you a character you've learned just before you forget it each time. The more times you're exposed to the flash card, the longer you retain the information, and that retaining is more effective if you see it less and less often. SRS is provably the best way to memorise kanji, but the biggest weakness is teaching you how to effectively read sentences with kanji in them.

Just read NHK Easy articles, or children's books.

The traditional wisdom has always been to find easy reading material and try your best to work through the unknown kanji using furigana 東(とう)京(きょう) the little hiragana pronunciation markers that sit above kanji in some mediums. The problem is obvious. When you don't know at least one kanji, it trips you up completely and often makes the sentence unreadable.

So here's the thought. SRS apps offer an excellent tracking system of which kanji you've learned and what your proficiency is with it. What if you could find reading material that is made up of only kanji you've learned? That's what LLMs are capable of, and that's where Infinilang comes in.

The goal of Infinilang (phase 1) is to produce reading material tailored exactly to the user's skill level. We provide an LLM with your learned kanji, which is pulled directly from your SRS app, and then generate sentences within the constrained character set. It's not quite as simple as feeding the character set to an LLM and producing great reading material. Wrangling AI to perform the task well is a lot more challenging than you'd expect. But it does work.

The challenges faced in producing high quality reading material fall into a number of families, essentially questions that need to be asked to ensure that the LLM didn't make mistakes, and if it did, can we repair them without generating a completely new response.

• How natural a sentence is?
• Does the sentence read as filtered or damaged?
• Is the passage coherent?
• Is the passage on topic?
• Is the passage appropriate for minors?
• Does it contain real people, brands or places?
• Can an N+1 insertion be inferred from the context?
• Is the passage worth reusing in the future?
• Is the sentence grammatical?

That's not an exhaustive list, but those are the main questions that need to be asked about the generated content.

One thing that became blatantly clear early on is that utilising a third-party LLM API would tank the project from the beginning. API credits are simply too expensive, and would eat up most of the revenue from subscriptions. Or worse, it would actually put us in the negative.

There was only one solution, and that was to build our own AI infrastructure. Not an easy task for someone building a startup with limited capital available. The good news is that the smaller LLMs are becoming more and more capable every few months. They're certainly not on the level of the frontier models, I could only dream of having an Opus class LLM to work with. But with a narrowly scoped and well supported software architecture (i.e. a lot of wrangling), these smaller LLMs can produce high quality output.

That doesn't solve the fact that the infrastructure is small scale starting out, and inference resources will be limited with thousands of users generating content daily. We also need to generate content at a reasonable speed as well. Nobody wants to sit there for 2 minutes waiting for a paragraph of Japanese to appear on their screen, no matter how good it is.

So the inference tasks the AI would be required to perform per generation might look like this:

1. Generate a sentence, paragraph or short story within the constraints. Probably requires some reasoning, and the longer the content, the more time it takes.
2. Evaluate the material produced based on the families of questions listed above.
3. Pass sentences through focused surgical repair passes where fixes couldn't be applied programmatically.
4. Repeat step 2 once more.
5. Potentially repeat step 3 and step 2 one more time before sending the content off to the user.

Even with an enterprise-grade GPU and prefix caching, this process can take a minute or two, and we only have 8 slots for parallel inference, which means jobs have to be queued when more than 8 people are trying to generate at once. Not practical in any sense.

That's where System One models come into the picture.

System One models have a few key advantages over traditional LLMs that make them incredibly applicable in use-cases like mine.

1. They provide blazing fast responses. I'm based in Australia and Jev is run out of Silicon Valley, and a call still comes back in about 258ms, ping included. That's hugely advantageous versus a traditional LLM.
2. They answer many questions in parallel. Unlike an LLM, which needs to process each question in sequential order, System One models can take a bunch of questions and provide answers to them in roughly the same amount of time as a single question. That means if it took 60 seconds to effectively answer nine questions with an LLM, it only takes 258ms to answer those same nine with a System One model. The more questions you need to ask, the more advantage you get out of a tool like Jev.

3. They're unbelievably cheap. During this project, I only spent $0.025 on Jev per 1000 calls. One of the lessons is that Jev is incredibly economical to run. I would not feel uncomfortable integrating it everywhere, it's practically free.

One other understated advantage is the reliability. Some LLMs will go rogue in their responses and provide unstructured responses that can't be processed programmatically. Jev is 'typesafe' in that it always provides a valid response that can be parsed.

That brings me to Laya, the subject of this experiment and likely the reason you'd be reading this post at all. Within a couple of days of Jev's public release, a slew of open source attempts to recreate the technology.

https://github.com/NandhaKishorM/laya

Racking up a whopping 35,000 stars on GitHub at the time of writing, Laya came onto the scene fast and hard, with over 1000 commits within a couple weeks, and near 3000 at the time of writing. Quite an impressive launch, and it showed just how much interest the software engineering community has for System One models.

Laya came with three base models, laya (421M), laya-multilingual (322M) and laya-typed-decisions (421M). The short summary is that laya is the all-rounder model, laya-multilingual is better for non-English languages but weaker in accuracy, and laya-typed-decisions has higher accuracy at the cost of less situational flexibility and poor multilingual capability.

The most obvious advantage of Laya is that it's open source, and you control the infra. There's no concern about Laya suddenly going bankrupt and shutting down its servers, causing a mad rush to find a suitable replacement or face your software suffering catastrophic failure. That's an attractive incentive for companies like mine that are looking to scale big, and want to minimise the risk.

The other perk of Laya is the small model size. By default, it requires very little VRAM to run, and it even has the capability to run on CPU according to the massive AI generated README.md. I believe my experimentation had the base model running on 2GB of VRAM, easily fitting into my server infrastructure.

When something seems too good to be true, it usually is. The stats and comparisons to Jev that are found on the GitHub page make claims like these:

Looks pretty convincing to me. So what's the catch? Take a look at the themes that Laya's models were tested against.

'Email spam', 'phishing', 'RAG passage relevance' and 'support triage'. These are highly specific and highly relevant to the disadvantages. You'll notice that 'LLM guardrails', 'moderation' and 'model routing' are marked "held out," and that's an important distinction.

To put it simply, Laya is not a one-shot model. You can't just pick it up and put it into any use-case. Out of the box, it has very specific capabilities and it's practically useless at everything else. That's the first place that Jev really shows its capabilities, 'it just works.' My first tests with Infinilang and the laya-multilingual model were utter failures, it wasn't able to perform any of the required tasks to a sufficient degree.

After the initial failure, I didn't give up immediately. Since kcalc.ai and Infinilang are still a while away from production, I have some serious server infrastructure sitting there practically unused. That means Qwen 3.8 27B is sitting there waiting to do some work, and I can shut it down for training whenever I need a powerful GPU. It was time for some experiments.

In a very real sense, there are serious legal hurdles to training AI models. As a legal entity, Hex Pro has to follow these laws closely. So I went to great pains to ensure that I followed all licensing requirements.

The first experiment was relatively simple. The goal was to find out if a trained model could meet a pre-specified bar.

1. Qwen 3.8 generated 1,995 examples of Japanese in the same way that the Infinilang prototype would require.

2. Claude Opus evaluated each output with panels of 3 for six families, and that gave us our 'gold' to measure against.

3. Qwen 3.8 as the teacher labelled the training data with its own evaluations of the same six families.

The six families in the initial experiment were:

• S1: How natural a sentence is - Score out of 5 (Metric: Spearman)
• S2: Sentence reads as filtered or damaged - Yes or No (Metric: pAUROC)
• S3: The English is a faithful translation - Yes or No (Metric: pAUROC)
• P1: Best of three repair candidates, or none - Choice of 4 (Metric: Gold Concordance)
• P2: Are any of the repair candidates acceptable? - Yes or No (Metric: pAUROC)
• G1: Sentence is grammatical - Yes or No (Metric: pAUROC)

The outcome was disappointing. The real stand out was S1 (naturalness) which needed to meet a bar of Spearman 0.60 but fell short at 0.452. My immediate instinct was to can the Laya training concept right there. I'd spent about a day and a half on it at this point, and those results just weren't good enough.

I killed the experiment there, not worth the time. Jev is so cheap to run, training a model would cost magnitudes more money than Jev would for 5 years of heavy use. Later that day, on a whim, I decided to test Jev against the same experiment and I found something very interesting.

WOH! Hol' up. What are these numbers? The teacher was level with Jev on all metrics, and actually had a substantial win on picking between sentence repair candidates. The number that convinced me the most though was the 0.446 on S1. Jev didn't meet the 0.6 bar either.

Those results reinvigorated the project, and I immediately sought to build upon them. With a cursory attempt, I'd matched Jev for those Infinilang tasks. There was one spot where Jev really became unbeatable throughout, and that was calibration. While the actual outcomes were solid with Laya, the 'confidence' score Jev provides is vastly superior in every respect. Jev's scoring is multi-dimensional, and it knows when it's sure of something and when it isn't. For a one-shot model that can be slotted into any task, that's very important. For a model that's designed for a very specific set of tasks, that calibration number is less necessary.

At this point, experiment 1 was closed out and the benchmarks were frozen. Money was about to get spent, and things were going to get serious. The goal now was to achieve a substantial gain on Jev's capabilities for Infinilang. There were three ways I could see to improve on experiment 1's scores.

1. More training data
2. Better LLM models
3. A more diverse rubric

At this point, the decision was made to only produce Jev benchmarks after I'd gathered all of the results. As soon as I popped the Jev benchmark, I'd have to move on to a new dataset to keep Jev quarantined.

The first step was screening alternate teachers. What I was looking for was models that were either much larger, or that might perform better for the Infinilang Japanese constraints. The candidates were:

• DeepSeek V4.1 - Huge, capable reasoning model with reasonably cheap inference. Strong Japanese scores.
• Gemma 4 31B - Tested for the Infinilang prototype, and stood out for avoiding kanji leaks. Also small enough to run on my server infrastructure, so it's basically free to test.
• Qwen 3.8 Flash Next - Basically Qwen 4 preview. It's a mixture-of-experts (MOE) model, which makes it much larger and more knowledgeable than Qwen 3.8 27B.
• Hy3 & Inkling-Small - Two lesser known models that I thought might have potentially good Japanese capabilities.

Other models were rejected, usually because their licensing doesn't allow them to be used for training.

Qwen 3.8 Flash Next was dropped first. The big issue was self-preference risk since it comes from the same family as the generator (Qwen 3.8 27B). It's worth taking a look at Extreme Self-Preference in Language Models (Steven A. Lehr, Mary Cipperman & Mahzarin R. Banaji, 2026), a study which demonstrated that LLMs love their own output and tend to be critical of others' output.

"Self-preference is a fundamental feature of biological organisms. Since large language models (LLMs) lack sentience, they might be expected to avoid such distortions. Yet, across 72 experiments and ~41,000 queries, we discovered massive self-preferences in eight widely used LLMs."

The risk that self-preferencing would affect the results was too high, and the tests showed that its results aligned closely with Qwen 3.8 27B's anyway. It didn't make sense to invest real money in a model that posed that kind of risk when I could just use the 27B version for free and get a similar outcome.

Hy3 & Inkling-Small were out next. Not much to say here. They were expensive and didn't achieve better results than any other model.

DeepSeek V4.1 was the clear standout across the board. The results were way beyond what I'd expected. I always got the impression that DeepSeek was just a rip off low quality Chinese model that stole everything from OpenAI and Anthropic. While the latter is probably true, it's no longer low quality. It was clearly the best open-weight model that also allowed training on its data. God bless the Chinese.

Gemma 4 was the last model. On most families, DeepSeek dominated. Gemma 4 met their standard in most respects, but it ended up doing well on S3 (faithful translation). It also performed well on P2 (any repairs acceptable) and B2 (on topic). Since it was free to test, I decided it was worth proceeding with.

So all of future tests would be run on Qwen 3.8 27B, Gemma 4 31B and DeepSeek V4.1.

The initial experiment had a limited set of families, but there are a lot more ways this model could be used within Infinilang, and I wanted to make sure it was trained on every task it would need to fulfil.

The new rubric consisted of 14 families. The 6 mentioned above (S1, S2, S3, P1, P2, G1) had their wording refined and tightened up along with the new additions:

• B1: Passage is coherent - Yes or No (Metric: pAUROC)
• B2: Passage is on topic - Yes or No (Metric: pAUROC)
• B3: Passage is appropriate for minors - Yes or No (Metric: pAUROC)
• B4: No real brands or places - Yes or No (Metric: pAUROC)
• B5: The N+1 (new kanji) word can be inferred from context - Yes or No (Metric: pAUROC)
• B6: The passage is worth reusing - Score out of 5 (Metric: Harrell's C)
• B7: The passage's Japanese is natural - Score out of 5 (Metric: Harrell's C)
• PS2: Passage's Japanese reads as damaged - Yes or No (Metric: pAUROC)

This time, the training data wouldn't just be on a line of Japanese (1 or 2 sentences). It would be a mix of sentences, paragraphs and short stories. The rubric was expanded to cover this necessity. You'll also note that there were a few other features I had interest in, like caching for reuse for other users, and identifying minor appropriate content for age restrictions.

My GPU spent 10 hours whirring away generating new topics and training material with Qwen 3.8 27B. I opted to keep generating the training data with Qwen here, because that's the model that would be used in Infinilang generations going forward anyway. Using a more powerful model here would be counter productive.

I ended up with 215,000 rows, and Claude gold panel judgements on each of them. Jev was also going to be getting the expanded and enhanced rubric, which enhanced its results against the experiment 1 benchmark significantly.

Note that the following results utilise the new rules developed with the enhanced rubric. So they outperform experiment 1 significantly. Also for some families, I tested averaging DeepSeek + Gemma results and weighting them for agreement with each other. And another caveat is that in certain cases, for certain models, more reasoning actually harmed the outcome. So these results are the best I could get after tuning for different combinations of reasoning and non-reasoning, rules and no rules, for each model and family.

*K8: eight sampled short answers with thinking off, as a smoothed histogram

These numbers gave me a pretty good idea of what I had to work with. As noted previously, I also tested Jev on these new standards and found out it performed much better this time around. Jev stood out on longer text, a theme that should be considered in the context of Laya, which does not have as long a context window and handles longer text worse.

Now that I had the teachers for each family decided, it was time to actually train the System One model. Remember, experiment 2 was about determining which teacher would be best for each family. That does not necessarily mean that the student model will perform as well as the teacher, there will inevitably be a trade off when all of these tasks are combined into a single model. A consideration for prospective Laya users should be how many tasks you need a single model to perform. For example, if I were to train a separate model for each individual family and optimise it to the max, then placed 14 models strategically throughout my program, I'd get the best results all around. But the problem then is that you need to be running 14 separate models, which is a significantly greater allocation of GPU resources. So it's a question of scale. If you're a mega corporation with billions to spend on expensive servers, it's a no brainer. A single one of these models can serve an astronomical number of users for minimal resource allocation. For anyone else, it's likely not the best use of your precious GPU compute.

At this point, it was time to get serious. The GPU was fired up once more, and this time 35 hours was spent generating and labelling more training data. Then the Claude gold panels ran again on the new data, which cost upwards of 200 million tokens under an Ultra (formerly Max 20x) plan.

DeepSeek was the costly one, since it couldn't be run locally, it needed cloud inference. It was selected for 9 families and cost A$233.56, taking 18 hours to complete. It all looks so innocent in USD, but the bank transactions in AUD look so much worse. Gemma 4 inference was run on our own infrastructure, so it was free and took about 60 hours to label the few families it was allocated.

In the end, I trained four candidates with different tuning and concepts. prod-q38, prod-ds41, prod-mix1, and prod-mix2. Each student was trained with three arms and two seeds per arm, to ensure that we weren't caught by an outlier. The finalists were trained on three more seeds.

• Arm A: laya-multilingual (mmBERT) with its pretrained head.
• Arm B: modernbert-ja-310m with a fresh head.
• Arm E: sokudan-ja-310m transplanted into Arm B's architecture.

There was also an arm D which was a 130M parameter latency test, it trailed in performance.

The first and cheapest to train, it used purely Qwen in differing configurations (reasoning/non-reasoning and rules/no rules) depending on what scored best. I knew it wouldn't perform as well as the others, but it was the extension of experiment 1 and gave a baseline for how far the other models could improve.

DeepSeek was the clear best performer in the previous experiment, but Qwen had an advantage or was fairly close to DeepSeek in a few families. Where it matched or exceeded DeepSeek, I used Qwen to save some money.

This one took the best of each family and attempted to create the ultimate combo platter. Also testing in swapping of Qwen for DeepSeek in places where the prod-ds41 student appeared to perform worse than expected.

Similar concept to prod-mix1, but P1 and B3 were reverted to match prod-ds41 to see if it helped improve the results.

The training on different arms was a wise idea, because it revealed some very important data. All arms trained off of laya-multilingual (Arm A) were inferior to Arm B and Arm E. This was an important finding, because it showed that the multilingual version, at least in my case where I only needed Japanese, was trash.

I highly recommend that anyone that attempts to train a System One model, discard the out-of-the-box Laya models, and train off of a clean head of ModernBERT. Or at least run the multi-arm test as we did so that you don't get caught out by an inferior checkpoint.

These are the final numbers for each student versus Jev's final numbers with the enhanced rubric on the new training data.

The results are fascinating because they aren't always intuitive or expected. The critical thing to recognise is that the training for each family's values affects all other families to a small but compounding degree. So the numbers are all over the place. In some cases, the student scores better than the teacher, in many it scores slightly worse.

It's also very challenging to decide on an actual winner because the value of each family is not equal. For example, S1 (naturalness of Japanese) is an important family, and Jev ranks substantially better than the others here. Conversely, B7 (naturalness of Japanese for the whole passage) is also very important, and Jev trails behind all other models here.

In the same vein, just because a model 'wins' a lot of families, doesn't make it automatically better. prod-ds41, prod-mix1 and prod-mix2 are all very close to each other in most cases. The difference between prod-mix1's S2 at 0.890 and prod-ds41's at 0.889 is practically none. That becomes clearer if we cut out the noise and just show the advantage or disadvantage against Jev but only for those that win the family or tie for the win.

My opinion is that prod-ds41 is the desirable all rounder. prod-mix1 has some impressive results in a few families (notably P1, P2 & B6), but it loses ground heavily against Jev with others where prod-ds41 loses less (S3, S1, B1 & B3). prod-mix2 has some serious weaknesses when compared to the other two, notably weaker on several (S3, P1, P2 & B5), and not particularly strong on the rest. Across the board though, the three generally hold an advantage over Jev, with a few exceptions where Jev really excels (S1, S3 & B3).

So given the choice between prod-ds41 and prod-mix1, prod-ds41 comes out ahead purely by being more consistently strong across all families.

prod-ds41 ended up taking the crown and earning the label of infini-jp-v1. The numbers very clearly demonstrate that it has the capability to fill most of the roles, with the disappointing exception of S1, sentence naturalness, which I found another solution for anyway (more on that later).

So as to the main question, is Laya better than Jev. Maybe? But probably not. While it does, in this case, outperform Jev for this particular set of tasks, let's take stock of what it cost to develop.

All up, I spent US$270 (A$389) on this project in total, $268.23 on cloud inference, and $0.21 on Jev. Cheap by AI training standards, if the hundreds of billions in debt that Anthropic and OpenAI are taking on is to be believed. Not cheap by the standards of a startup with limited capital and minimal revenue (for now). But it was valuable R&D because it answered some important questions about whether I should be using this for my other projects in the future. Furthermore, I now have a training pipeline project with over 1000 commits and 14,522 unit tests which means that developing another System One model in the future is now just a matter of refining the rubric and performing the model bakeoffs.

This was not cheap in terms of hands on time. Thank Dario for Claude Code, because much of the busy work (watching for runs to end, queueing up the next parts and so on) was taken care of. It meant minimal downtime, the work kept on going overnight. Even still, it took 24 days to finish this project start to end. It would've taken six months without AI assistance, there's no doubt in my mind.

Regardless of AI saving a huge amount of time and allowing me to work on other things in parallel, my own hands on time invested would've been worth about A$5000 in labour at my old rate. That's a cost that needs to be considered when training a System One model.

I'm somewhat lucky in that I have access to a fast and powerful enterprise-tier GPU. But this project required over 40 hours of exclusive GPU time, and I dread to think about how much the energy bill was. I don't even want to find out.

The other ongoing cost here is that it takes up approximately 2GB of VRAM in my server.

This is an important question that's hard to quantify. These are the questions I'd want to answer if I was considering attempting this:

• Is Jev capable of performing the task well?
• How much call volume do you expect to have?
• How many and how diverse are the tasks you need it to do?
• How long will it take for you to make your investment back?
• How important is the peace of mind of knowing your software will function after TypeSafe disappears?
• How much do you care about accurate calibration/confidence?

In my case, the capability improvement may help Infinilang perform better. If Infinilang performs better as a result, and that allows me to build up a larger userbase, then it will inevitably pay itself off. That's the only way I can justify using Laya over Jev.

• Latency: From Australia, Jev takes about 258ms for a round trip. Laya averages about 19ms, it's at least 13x faster than Jev, and 2000x faster than DeepSeek.
• GPU Opportunity: It takes up 2GB of VRAM. That's not nothing, but it's not massive either. Here's the real advantage though. Those 14 families of questions that infini-jp-v1 is answering are questions that don't have to be answered (poorly) by Qwen 3.8 27B, freeing up the GPU to answer other, more important and appropriate requests.
• Imitating A Superior Model: You don't need to run DeepSeek V4.1 to get answers pretty close to what it would give. infini-jp-v1 performs far better than Qwen 3.8 27B for these tasks and you don't need to pull out a 614GB VRAM DeepSeek model to do it.
• Business Security: As mentioned a few times, I don't have to worry about Jev disappearing on me when I use my own model. No unexpected outages, no mad dash for an alternative. If my software is live, then I know my System One model is too.
• Narrow Task Superiority?: Whether it performs better is still yet to be seen in a production sense. We'll see. As you saw from the scores, theoretically it's 6% better in Infinilang, but that's across all families, not necessarily the load bearing ones.

For the uninitiated, a LoRA (Low-Rank Adaptation) is kinda like a plug and play persona for an LLM. You may have heard people refer to it instead as 'fine-tuning'. Unlike a System One model which is its own separate service, a LoRA is applied to the LLM directly upon inference to shift its weights and make it behave differently.

As such, during later development of Infinilang, I've trained a LoRA to help Qwen 3.8 27B produce better quality output and follow stricter rules. What this means is that infini-jp-v1 will be classifying higher quality material. It won't have to work so hard and should have more reliable impact, especially on users with smaller kanji pools.

I'll endeavour to improve Infinilang even more as development continues.