The Reference Problem

The easiest way to make your own result look stronger is to change what counts as a result.

Sunday Robotics has done exactly that. The company published a blog introducing what it calls a "Solve": every robotics claim should include three things. Performance (how well the robot did), scope (the conditions under which it did it), and adaptation cost (what had to change before it worked somewhere new), in plain English, how much custom setup each new home required. It's a strategic proposal, not a neutral contribution. Sunday is arguing that without all three, a success rate tells you almost nothing. A claimed 99.1% on one carefully prepared garment in one familiar room is not the same statement as a claimed 99.1% across 785 autonomous attempts, 9 garment types, in homes the company says the robot had never seen before, with zero per-home adaptation. The industry has spent years publishing the first number while quietly encouraging readers to imagine the second.

What gives Sunday's argument weight is that it comes with receipts. The company says it fixed its grading rubric before the evaluation began rather than adjusting it afterwards. It says every run was annotated twice, with disagreements reviewed. It says the homes used for evaluation were excluded from post-training, so the robot was not tuned on the places where it would later be tested. Whether every claim survives scrutiny is almost beside the point. This is a level of methodological detail that I rarely see in home robotics, where polished videos and isolated percentages have often stood in for evidence. If Sunday's standard became the expectation, a number of robotics demos would stop qualifying as meaningful proof of anything.

That is not an act of generosity. It is a competitive move. Sunday appears to have proposed the standard that best highlights the strengths of the result it has already produced. Companies do this all the time. The interesting part is not that Sunday has skin in the game. The interesting part is that its proposal forces everyone else to show theirs. Once you ask for performance, scope and adaptation cost together, the gaps become difficult to hide.

But none of this escapes the oldest problem in technology. Sunday owns the robot. Sunday selected the homes. Sunday wrote the rubric. Sunday’s annotators graded the outcome. Sunday reported the outcome. It may be the most methodologically honest result published in home robotics so far, and it is still a company marking its own homework. If scope and adaptation cost are what make a benchmark worth trusting, who gets to decide when a Solve is actually solved?

A public school district in western New York was putting a humanoid robot in front of students. So I looked up who builds it. Realbotix owns the business that makes RealDoll sex dolls. It bought that company in 2024, the same year it renamed itself from a crypto firm called Tokens.com. Realbotix says the robots and the dolls are separate technologies built by separate teams, and that no adult-products business is involved in the Salamanca project.

Salamanca City Central School District sits entirely within the Seneca Nation reservation. Around 79 percent of its students are economically disadvantaged. About 32 percent identify as American Indian or Alaska Native. The district serves roughly 1,300 students. The deal does not begin with a competitive procurement process. It begins, New York Focus reported, because the superintendent's former colleague has dinner with a Realbotix investor. One introduction leads to another. The district approves the purchase. The robot cannot identify its own students. It relies on each student to announce a numeric code at the start of every interaction so the system knows who it is talking to. The program is currently on hold. The district paused it on Friday after the New York State education commissioner wrote to the superintendent raising concerns about the robot's potential to mimic a teacher's role.

Realbotix lists the M-Series robot and avatar package at about $95,000. Salamanca agrees to pay $57,590. That leaves more than $37,000 between the sticker price and the invoice. The district does not receive charity. Realbotix receives something harder to buy. It receives a public school willing to become an early customer for an education business that does not yet exist as a separate company. Future districts do not see a prototype in a showroom. They see a robot standing in front of real students with a school district attached to its name. That reference carries weight that marketing material cannot manufacture. Salamanca gets access to hardware that would otherwise sit well beyond its budget. Both sides walk away with something they cannot easily purchase at full price. The question left hanging is why arrangements like this so often seem to find communities with the least room to absorb a mistake before they find communities with the most choices.

This issue kept returning to the same question from two different angles. Sunday Robotics published what may be the most methodologically honest result in home robotics — and still graded its own homework. A school district on a Native American reservation got a discount on a robot from a company that needed a reference customer. In both cases, the people with the most to gain from the outcome were the ones who decided what counted as success. That is not unique to robotics. It is just more visible here.