From "everything broke" to "you saved me": how a duty administrator's day goes in a cloud provider

08:00 – with the first rays of the sun, I happily get out of bed and go to warm up. A 20-minute run and a cool breeze gave me strength for the coming day.
08:30 – I return from the run and start having breakfast. The menu includes sweet tea, avocado, trout, and a newspaper. It seems it's time to develop oil fields, oil is getting more expensive.
09:00 – 09:40 – I calmly ride the subway and even manage to catch my bus.
09:40 – 10:00 – colleagues calmly hand over the shift to me.
10:00 – 19:00 – with a smart look, I solve complex problems.
20:00 – I send the evening report to the manager.
03:00 – I finish my work and…

Drin-drin-drin ….. the alarm clock rings and at this moment I realize that it was a good dream. In fact, everything is somewhat different. More precisely, everything is completely different.

08:30 – the first rays of the sun pass tangentially past me.

08:45 – 09:00 – the warm-up is reduced to a sprint to the metro station with a backpack full of food prepared for the day. The motivation to let the tired colleague off the shift on time and not be late for work brings the speed of my legs closer to the speed of light. By the way, Usain Bolt quit sports after seeing me (not from old age at all).

09:00 – 09:30 – on the way, I check my work email from my phone, naturally forgetting my password in my sleep and remembering it on the fifth try. I look at the tasks that have already come to the duty shift in advance.

09:40 – if the bus does not come to you, then you go to the bus. In my case, the second option.

09:45 – 10:00 – I arrive at the office, greet the AHO employee who is cleaning the leaves at the entrance, the Asset Protection Service employees who are so intently looking at the monitors that they have already greeted you on the approach.
I fly into the duty shift room. Here, the tasks left over from the previous day are being sorted out.

10:05 – dodging colleagues – duty engineers already running on customer requests, brewing a cup of tea, drinking a cup of tea. Everything must be in order.

10:06 – sitting down at the workplace. 2 eyes – 8 monitors, 2 hands – keyboard + mouse + radio + 3 phones. Checking that everything matches and you can work.

10:30 – 12:00 – new requests start coming in. It feels like you have known the words "migration", "network", "Nat", "FW" since birth, and cloud services in all forms have been with you since you learned to walk.

By lunchtime, the main front of work is formed, someone needs help to create a VM, deploy a service in test, see why the rule on the firewall is not working correctly, or help to log in with the issued account.

All possible questions flow to the support email, and if I have difficulty answering a question, there is always a friendly team ready to help with advice and offer a solution to the problem.

12:00 – 14:00 – I complete about 5-7 priority requests. Someone could not log in to the issued test. It's good that I can quickly look at the logs and identify incorrect login/password input. Someone incorrectly configured the network interface on the VM. Then I check the correctness of the mail rules, create a couple of accounts and mailboxes for internal employees.

14:05 – I see a yellow alarm on the monitoring system on the video wall. It's good that we have as many as 8 monitors on the entire wall, and even a division into three main groups. I don't believe what I see, and I move 50 active work tabs in the browser to look at the monitoring from my PC. Now I believe.

A large number of letters and color indicate that the problem needs attention, promptly deciphering the language of the ancient Incas.

I inform my colleagues, duty engineers, and duty operations about a possible malfunction at the power supply node. The chief power engineer is also not left out. In 4 minutes, the staff makes a round and inspects the client equipment, electrical panels, and air conditioners. No problems were found. Information comes that the work is planned.
It turns out that I just missed the notification about the start of work. Well, at least I additionally worked out the procedure for inspecting equipment and interacting with duty services.

14:30 – 15:00 - lunch time. I hand over the monitoring to my colleagues, leave the duty room, and from afar I smell the subtle notes of dough, cheese, and ham. A clear sign that someone has become a mom or dad, someone has a birthday or another holiday. It is customary in the team to share happiness with colleagues, and we do this through pizza.

I'm standing in the kitchen, congratulating, eating, and realizing that I forgot about the backpack with food. Everything is according to the instructions.

By the way, speaking of instructions, all standard tasks for employees are described in the corporate wiki, and detailed step-by-step instructions for clients are provided on the website.

15:00 – 15:15 – daytime walk outside. I go out on the porch, breathe in the fresh cool air, and look at the clouds. Not all the time to watch them on the monitor. True, from the services, the real ones only bring shadow and water... thanks, it started snowing, I'll go back to working with the digital ones.

15:30 – 17:00 – serious post-lunch requests are already starting. More and more often you can see constructions like "need consultation", "preliminary call required before starting work", "what is peering and how to set it up", "help with migration and backup", "help set up IPSec VPN". Here, in addition to my comments, I already involve colleagues-gurus from virtualization and related departments of the technical directorate. In particularly interesting cases, our service managers, as the client does not always know what they want. Together we offer solutions to problems and provide answers to questions.

17:00 – 17:05 – another cosmic quick tea, the number of tea bags in the mug is growing exponentially.

17:00 – 19:00 – finishing the remaining requests, trying to get as much feedback as possible from clients and employees on the requests before the end of the working day.

20:00 – summary of the day's work. I send an interim report to the manager.

20:00 – 03:00 – I carry out night work on expanding disk spaces and updates, prepare UZ for new employees, as well as test spaces for requests received at the end of the working day. I help duty engineers with requests, accompany the setup of IPMI, crimp cables, mount equipment in racks, and pull cross-connections.

Yes, yes, we can also work with our hands.

03:00 – 04:00 – I fill out the templates of the daily shift handover report.

04:00 – 08:00 – rest and a calm rise. Here it is already without warm-ups and exercises. In rare cases, if someone's log file in the database is clogged or a client needs help, the rise is unscheduled.

That's why I'm on duty, ready to help and respond to an emergency at any time.

08:00 – 09:30 – I check the meeting rooms before work and finally compile the daily report.

Sometimes it happens that during the cleaning of the room, the microphone off button is touched and at the beginning of the conference a new episode of "Who will find the problem faster or why we speak and are not heard" begins.

09:30 – 10:00 – the ritual of shift handover takes place. We exchange "understood, accepted", shake hands bravely, and I go home.

And also the duty administrator throughout the working day:

1. Monitors three main groups: energy, cooling, network, and cloud services.
2. Monitors IPS and user authorization attempts in various services. 3. Reacts and notifies responsible persons about the results of vulnerability scanning in networks and services.
4. Warns clients about abnormal loads on VMs located in the cloud.

The duty officer is the person who, when leaving, says goodbye to colleagues with the words: "Good afternoon! I am attaching the shift report"

This is how my day went yesterday. The day of the duty administrator. I will rest for three days, and then I will run through this schedule again.