Expectation: build an AI wallpaper try-on in 2 days. Reality: I had to fine-tune my own model on Stable Diffusion

At that time, generative models (such as "Nano Banana") were all the hype on the market. At first glance, it seemed that the problem could be solved in two clicks. The plan was as reliable as a Swiss watch: get an API key, send a photo of the interior and wallpaper texture to the endpoint, accompany this with the right prompt (specifying to account for perspective, lighting, and scale), and get the finished result.

But in practice, it turned out that the task is far from trivial. It revealed a whole host of problems.

The illusion of ready-made solutions: why "out of the box" doesn't work

Our first candidate — the real and powerful Google Nano Banana Pro model — immediately delivered an unpleasant surprise. After the first tests and long manipulations with prompts, disappointment awaited us. It turned out that Nano Banana Pro in some cases behaves too "creatively". Instead of carefully replacing the wallpaper, it could drastically change the apartment layout to the one it finds more convenient, and most textures were stretched like photo wallpapers or a huge panel, completely ignoring the real size of the pattern (repeat).

For production (and even for a pilot launch) such hallucinations are unacceptable. We realized that the basic solution doesn't work, and started testing other available alternatives: from current versions of Flux Pro to fresh generative tools like ChatGPT Image 2, as well as specialized B2B services like Wizart.

Here is a brief summary of our observations.

Flux

Flux showed mixed results. In terms of pattern size, it produced a more or less acceptable result in most cases, but consistently distorted the texture itself. In some places, the wallpaper turned into outright "mush", and sometimes even on perfectly even surfaces.

• Consistently covers all walls, works well with natural lighting and shadows.

• Systematically adds extra switches, sockets, and fittings. Often distorts the pattern (especially breaks complex textures: damask, strict geometry). Periodically changes the texture color on its own, making it darker than the original.

OpenAI (ChatGPT Image 2.0)

The OpenAI model appealed to us more than the others. It seemed to handle the task conceptually, but we decided to conduct a stress test on complex patterns.

• Excellent at maintaining room geometry, does not suffer from hallucinations (does not add extra elements out of thin air), provides stable lighting.

• More often than others works by the method of "approximate rendering". The pattern remains recognizable, but is distorted. The main pain point is that the scale floats. The neural network is simply "too lazy" to deal with a small pattern on a distant wall (where the pattern should be smaller due to perspective), so it artificially increases the texture size. A paradox arises: the pattern itself is preserved perfectly, but a giant panel hangs on the wall. In one of the cases, the model even cropped part of the frame.

Wizart

We tested the last one as a specialized solution. But for production, it was completely unsuitable in terms of realism. It's not entirely clear why wallpaper factories install it on their websites (probably because of the convenient infrastructure, ready-made search, and integrations).

• Very gentle with the texture itself (the pattern is preserved almost always), does not litter the walls with unnecessary objects.

• Incomplete wall coverage with "torn" masks in all our tests. Unnatural lighting and shadows. Strong perspective distortions at the corners.

It became clear that ready-made generative models are blind to strict geometry of real space. They do not take into account the scale of the wallpaper repeat, "drift" in the perspective of a particular wall.

Under the hood of RoomRUM

I will not disclose the full architecture of our engine and exactly how we process photos in real time — this is the secret of the RoomRUM project. But I will talk about the foundation on which this engine operates.

The foundation of any ML model is data. Instead of cobbling together complex crutches around ready-made APIs, we went the route of creating a reference dataset. We categorically refused academic samples and assembled our own commercially clean dataset. As a base, we used photos from stock platforms and open datasets with a Public Domain license. We filtered the interiors we needed and proceeded to markup

Data preparation pipeline

To assemble the reference dataset on which we planned to fine-tune the model, we automated everything that could be automated, and handled the rest manually, breaking the process into clear stages:

• We did basic wall markup using a combination of Grounded-SAM (where Grounding DINO finds objects by the text prompt "wall", and SAM cuts them out) to immediately get masks with smooth edges.

• To preserve realism, we used image decomposition methods. We programmatically extracted an approximate mask of shadows and highlights from the original source and applied it over the wallpaper using basic blending modes.

• Automation often makes mistakes at junctions with furniture, around windows, and in corners. So we assigned one person to manually clean up the results. They meticulously corrected masks, adjusted perspective distortions of the pattern on complex corners, and erased shadow artifacts.

The result of this routine — we collected and verified around 1700 reference pairs of the type: Original interior ➔ Wallpaper repeat ➔ High-quality overlay with visually accurate scale and perspective.

Fine-tuning

Having this clean reference dataset in hand, we proceeded to fine-tune (in our case, training custom ControlNet and Inpainting modules on top of the Stable Diffusion base architecture). We split the sample into train and validation sets and started feeding these pairs to our pipeline. The main task was to make the algorithm identify, based on these 1700 examples, how the room size and pattern scale are visually related. It is impossible to calculate the absolute wall size in centimeters from a 2D photograph, so we taught the algorithm not precise geometry down to the millimeter, but the ability to build plausible perspective logic and texture scale. Training was conducted on rented clusters with A100 GPUs. We ran about 100 epochs until the validation loss metric reached a stable plateau. After that, we moved on to visual tests.

As a result, our algorithm indeed learned to automatically maintain the texture scale and neatly circumvent obstacles. (Although, in all fairness, it was still far from perfect: even after such training, annoying artifacts surfaced, which had to be addressed separately).

Comparative Benchmark (In-house): ChatGPT Image 2.0 vs RoomRUM 1.0

For an academic paper, comparing our model to a universal generative model would be incorrect. But for a business looking for a ready-made out-of-the-box solution, this is a real choice. We decided to pit our model against the best proprietary API — ChatGPT Image 2.0 (using standard generation parameters quality: "auto", while trying to get the most out of OpenAI through precise prompt engineering).

Benchmark Conditions (May 2026):

• Task: Replacing wallpaper on walls in photographs of residential premises.

• Test Set: 50 interior photographs with a reference pair "original room / target wallpaper". Scenes cover a wide range: empty and cluttered rooms, different lighting, columns, niches, mirrors. Textures range from plain linen to complex damasks.

• Evaluation Type: Manual expert evaluation (single-shot) based on 4 criteria.

• Scale: 0–2 (2 — no remarks, 1 — partial compliance, 0 — critical deviation). Maximum score per photo: 8. Maximum for the model: 400.

Grading Criteria

Note: generation time and inference cost were not evaluated in this test. We also deliberately did not make lighting realism a separate metric: modern generative models handle shadows and relighting perfectly out of the box on their own. We focused on what breaks most often — maintaining scale and geometry.

Summary results

On the filtered subset of photos that our algorithm deemed suitable for processing, RoomRUM showed a better score. In 31 out of 50 scenes (62%), our model was more accurate, in 11 (22%) there was a tie, and only in 8 scenes (16%) did the OpenAI model win.

But the devil is in the details. Let's break down each category separately.

Category analysis: where do neural networks break down?

1. Color (Parity)
Both models confidently work with conveying shades. There were no critical misses (score 0) for either. RoomRUM's deviations are associated with slight brightening or oversaturation of the frame, while ChatGPT's are associated with rare darkening. Both networks are suitable for the task of "guessing the color of the walls."

2. Texture (Different strategies)
ChatGPT won by 5 points, but the distribution of scores revealed a difference in architectural approach:

• ChatGPT often tries to recreate the texture perfectly (many "2" scores), but if it fails, it hallucinates and replaces the pattern with a visually similar one (five failures with "0").

• RoomRUM prefers safety. We have almost no zeros, but many ones: the model gives an "imitation" - retains the overall character of the pattern, but loses fine detail on distant plans with complex patterns.

3. Size
This revealed the main weakness of basic models in interior tasks. The gap was 48 points. For ChatGPT, the scale was correct in only 18% of cases. In 82% of scenes, the model enlarged the pattern: either moderately to compensate for detail or turning wallpaper into a giant fresco. The neural network simply doesn't understand physical dimensions.

Our engine, thanks to its trained spatial awareness, delivered 88% accurate hits without a single critical deviation. For real renovation work, this is a key factor: a person needs to understand whether their wall will end up with a wavy ripple of tiny flower patterns or two large flower buds.

4. Geometry
ChatGPT had 4 critical violations recorded (score of "0"): the model papered over window frames, covered doors, and altered room proportions, visually expanding the space. RoomRUM has no critical violations at all, but there are 15 scenes with minor defects (score of "1"): we might miss a strip behind a radiator, fail to apply texture to a mirror reflection, or do a sloppy job at a column joint.

ChatGPT Image 2.0 acts like a creative designer ready to draw in a window or alter proportions for a pretty picture. RoomRUM 1.0 works like a strict painter: it will never tear down your wall, but might forget to apply wallpaper behind a pipe.

The bitter truth: RoomRUM limitations

It would be dishonest to boast about metrics and stay silent about our model's main limitation. This problem directly impacts the sales funnel, and we deliberately left it out of the benchmark scope.

RoomRUM 1.0 accepts far from all photos for processing. Recently we launched a test advertising campaign to test the funnel with real user photos. The statistics are harsh: our preprocessing rejects around ~60% of uploaded frames, only successfully passing 40%-45% on to generation.

We deliberately made the preprocessing stage strictly mandatory. Practice has shown that if an "inconvenient" frame is passed further, the model either refuses to apply texture altogether, or applies it completely incorrectly. Rejections occur at the preprocessing stage if the photo:

• A space with overly complex architecture (multi-level niches, arches, an abundance of columns, wall niches).

• A room that is too cluttered with furniture or has numerous passageways (for example, corridors with a large number of doors).

• Poor lighting or a low-quality source (resolution below 800 pixels). Unusual angles.

This means that in real-world traffic for an online store, some users with complex or poorly taken photos simply won’t be able to immediately use the virtual try-on feature. To avoid returning defective results from "jack-of-all-trades" neural networks to customers, in such scenarios the system will simply ask the user to take a better shot from another angle following the on-screen instructions. This allows the quality bar to be maintained and prevents customers from being shown a clearly distorted image.

Limitations of the study itself

We intentionally introduced sample bias. Only the 50 scenes that successfully passed our preprocessing were included in the benchmark. We did not test the models on "rejected" frames, because for our product, generation failure is a controlled business decision (it is better to honestly ask the user to retake the room photo than to show them a distorted rendering and erode their trust). Therefore, we only compared quality in scenarios where RoomRUM gives the green light. It is telling that even in these comfortable, basic scenarios, ChatGPT still lost the battle for physical scale. In addition, for the sake of objectivity, it is worth acknowledging that the benchmark was conducted by two experts (without calculating inter-rater reliability), using a rough 0-2 scale and without accounting for variance (repeated generations). In the future, we plan to incorporate SSIM/LPIPS metrics and ΔE color distance evaluation.

What’s next?

On the current test set of 50 interior scenes, RoomRUM 1.0 achieves a higher aggregate score (84.0% vs 71.5%) thanks to pattern scale accuracy and preservation of scene geometry. For the applied task of renovation visualization — "exactly how the wallpaper will look in my room" — these parameters are critically important.

However, this conclusion should be adjusted given that the algorithm does not yet support certain angles and overly complex rooms. In such cases, the automated system fails, and asks the user to upload a photo from another angle.

The current version of RoomRUM is a working, but still first step. We are already actively working to eliminate the main drawbacks. Currently in development are:

1. Increasing success rate and improving geometry: We fine-tune the model on non-standard angles (without floor/ceiling) and complex spaces to reduce the frame rejection rate during preprocessing.

2. Floor and ceiling generation: So that users can preview laminate or tile paired with new wallpaper.

3. Application of combined wallpapers: We train the algorithm to correctly zone the space and apply companion wallpapers (for example, stretch an accent texture on one wall, and a background one on the rest).

4. Inference optimization: We are working on increasing generation speed and reducing computing costs to make the feature's economics even more attractive for businesses.

The path from "we'll write a prompt for a ready-made API" to our own ML pipeline trained on A100s took us far more effort and time than we initially planned. The laws of physics turned out to be more stubborn than basic generative networks, but a high-quality dataset and manual annotation can deliver results.