"Room of Dirt" RTC-Service: how we scraped and repaired expensive network equipment

Hello, tekkix! Under the cut, read about how such a responsible area as the repair shop is arranged with us, as well as a story about cleaning the really dirtiest hardware you can imagine!

We at RTK-Service have a large laboratory for working with telecommunications equipment. Part of the laboratory is allocated to the repair area, where engineers and system technicians work. Without undue modesty, it can be said that these guys are worth their weight in gold! Which is not surprising: network equipment is expensive, and pros who really know how to fix it can be counted on the fingers.

This post will be a small virtual tour of our sanctum sanctorum.

Since our clients' equipment park - telecom operators - has everything you can imagine, we are engaged in repairing any network equipment! And this does not mean that we do not have a specialization. It's just that it is so extensive that it covers, perhaps, everything except the ancient PBXs from the 90s that are living out their lives. If they are still found somewhere, they are already preferred not to be repaired, but to be thrown away and replaced with something modern.

Various transport and IP equipment, GPON switches, digital PBXs, power supplies, and so on are regularly sent for repair. And since today there are a wide variety of models on the networks, we deal with hardware from all leading manufacturers, both foreign and domestic. Nevertheless, it can be said that our engineers specialize in complex industrial equipment, which is installed only on the networks of communication operators. No other industries use it. And it is also insanely expensive. Therefore, specialists in such hardware are hard to find.

How to repair heavy telecom hardware?

The main difficulty in maintaining such hardware is not only in the breadth of its nomenclature. To do something with a conditional Juniper, you need a laboratory, appropriate test benches, otherwise we will not be able to even diagnose, reproduce the problem and make sure that you have eliminated it.

First of all, it is necessary to connect and power it somewhere, for this you need chassis. You also need at least a minimum of cards so that the device just starts without traffic. Further, for full-fledged checks, traffic generators are needed, because most of the problems that we need to catch arise when the load approaches the maximum bandwidth of each of the devices.

That is why we maintain a whole machine room with stands in our laboratory. Now we have 43 racks that are filled with multi-vendor hardware (Huawei, Nokia, ECI, NEC and others) installed on the networks of our customers. The rows of racks are divided by technology - a row of DWDM, a row of IP, and so on. All this beauty helps to accurately emulate the telecom infrastructure of our customers.

There are different cases, of course. For example, once we had one board in diagnostics (IP from Juniper) for a whole month, that is, under traffic. As a result, we returned it back without repair, without finding any malfunctions. It's just that if the cause of the "breakdown" is the specifics of the configuration, software bug, external operating conditions, we certainly can't do anything.

SPA for hardware

A separate type of masochism of our repair engineers is a special "dirt room" where you can clean everything that can be cleaned in principle. The same traces of corrosion, which are very often found on old equipment.

An ultrasonic bath is installed here, which helps to clean flooded and dirty boards.

A sandblasting chamber connected to a compressor that creates a stream of compressed air.

The sealed chamber is actually needed not for the servers themselves, but for us. So as not to breathe in what is blown and washed off from dirty hardware. And the equipment can be in very poor condition. Dusty switches are, in principle, normal. Unfortunately, maintenance does not always keep track of cleanliness, especially when it comes to fans and filters. What is installed directly on the streets can have leaves, bugs, spiders, and other delights. These can be both radio relay stations and WiMAX components (which are rarely inspected), as well as any other switches - anything that can be placed in a street cabinet. Some devices are located in very hard-to-reach areas - at least along all our oil and gas pipelines there is a data transmission network.

As a result, some things generally need to be washed at a car wash. And although we do not have powerful Karcher pressure washers, we do have a large sink. And a watering can! Here we wash off the dirtiest dirt that can be.

Dirty champion!

Once, the dirtiest equipment imaginable ended up in our repair shop. Last year, Juniper devices from the ZIP warehouse in Sochi arrived. Spare parts were lying around, waiting for their time, but then a mudslide occurred (which is normal for Sochi) and flooded the warehouse. So it was not just water, but sludge with dirt and other filth.

The damage to the routers was discovered only after 10 days. Then for some time people thought about what to do with this horror - that is, a composite mixture of IT equipment and clay stood for some time. Then it traveled from Sochi to Moscow. Do you think anyone washed or at least shook it off? No!

So we washed it in our big sink. But we can't throw it away - the hardware is worth three million! Life did not prepare us for this, but we coped! In total, we received 11 devices in terrible condition. Of these, we managed to fully restore 3 routers. And several individual boards were also restored.

Our dirty "guests" went through several iterations in an ultrasonic bath, then were dried. Each unit of equipment was disassembled, and then with scrapers and brushes, manually, like archaeologists, we reached the microchips. Then we cleaned them with compressed air. In total, it took us a month to "disassemble to the screw and wash". We removed several buckets of dirt in the process. After that, the equipment was tested and checked for a long time.

As a result, we got quite decent hardware. Yes, its appearance left much to be desired, but 3 out of 11 Junipers continued to work. Not like new, of course, but it can be confidently used as a reserve, ZIP, test stands, in non-critical network areas. These tanks are not afraid of dirt, but here it is still sensitive electronics. By the way, the equipment in this form was, of course, outside the SLA, but we miraculously managed to fit into 90 days turnkey.

Fortunately, we don't get such beauty every day! Would you throw away such equipment, or would you be eager to fix it?