How we accelerated the solution of our machine learning task by participating in the Big Mathematical Workshop

Given: a "future" task that has been postponed all the time

We are constantly improving the methods of software security analysis in our company. Not so long ago, we were attracted by a new technology for detecting vulnerabilities in the program code based on large language models (Large Language Models). Specifically, we were interested in the capabilities of large language models not only to identify vulnerabilities in the code but also to give the developer hints on how to change a particular section of the code and make it safe. This methodology exists and is presented in a number of scientific papers. But we needed to test it and draw conclusions about how applicable this methodology is in practice for solving our problems.

This task was the very one that falls into the category of "not urgent", "for the future" (and, as you know, "future" tasks tend to be postponed until this very "future" comes).

We have long known about the existence of the Big Mathematical Workshop (BMM) and had an idea of the level of tasks that the guys solve there. The event has been held since 2020. It is a kind of mathematical offline hackathon, where mathematicians — mostly students and young specialists, but not only them — gather to solve mathematical problems, problems from other fields of science, and the real sector. In 2024, the Workshop was held in four cities: Novosibirsk, Tomsk, Omsk, and Maykop.

And then we had an idea: “the future” has arrived our case may fit well into the BMM format. Of course, we could conduct all the necessary experiments ourselves and draw conclusions entirely on our own. But, on the other hand, we could share with the organizers of the Workshop a really interesting task from our work practice, which, moreover, does not require a long entry into the specifics of the work. And although this task was far from conformal mappings and cohomologies, in content it fit quite well into the event. So we decided to submit our case to the Big Mathematical Workshop.

Let's move on to practice

You can't just take and propose a task.
To have the proposed task accepted for solution within the framework of the BMM, you need to pass a rather strict selection. Firstly, the solution must imply the use of higher mathematics methods, otherwise the task may be rejected already at the application stage. For example, the task “Program a website for an online store for us” is not suitable. But the tasks “Forecast the demand in our online store” or “Optimize our warehouse stocks” are quite acceptable. Here, apparently, it played into our hands that methods of discrete mathematics, in particular, graph theory, are traditionally used to work with source code, and machine learning is also largely based on mathematical analysis, mathematical statistics and numerical methods.

Secondly, the task must be attractive enough for the participants, since they themselves choose which projects they would like to participate in. If no one chooses the project, it will not start. Apparently, our topic interested mathematics students - they considered that they could include it in their portfolio. Also, in the announcement, we promised that we would be ready to invite the participants of our project to our team for an internship - this could also play a role. Other projects related to large language models were also presented at the Workshop.

Two months before the start of the in-person stage, during the preparation phase for the Workshop, the BMM organizers assigned us a curator. The curator is the project manager from the Workshop: he is responsible for organizing the process of solving the assigned task and it is with him that we interacted on all processes. The curator carefully studied our task and prepared introductory materials for the participants. In addition, the curator was responsible for selecting participants for the project.

You can find out more about our task here.

Hour X (3 weeks): in-person stage

The main stage of the Workshop lasts three weeks - all practical work takes place during it. Participants come to the city of the event (in our case, it was Novosibirsk) and work on solving problems. In our team, besides the curator, there were four people: three senior students of mathematical fields from leading Russian universities - Novosibirsk State University, MAI, and ITMO - and a candidate of physical and mathematical sciences. Our curator was a master's student at Novosibirsk State University, studying in the field of "Data Science and Artificial Intelligence".

Our team was able to immerse themselves in the topic literally in one day, largely thanks to the curator's thorough preparation of the introductory materials. Already on the second day of the in-person stage, the work on solving the problem was in full swing: participants studied scientific publications, tried to reproduce the results, experimentally determined the possibilities and limitations of various methods, tested models on "their" data. The organizers also took care of the communication between participants from different projects: regular general reports and "roundabouts" were arranged at the event - when one team visits another. Thus, our team had the opportunity to closely communicate with teams from other projects related to both large language models and code analysis. This inevitably gave a certain synergistic effect.

When the guys finished working on our task, the result intrigued us and overall we were satisfied. Firstly, the team provided us with a detailed document containing a review of scientific articles, methods, and datasets on automatic vulnerability correction. In addition to the review, which is valuable in itself, the document included "test results" of many methods listed in the plan, and an analysis of their applicability in our conditions. For example, the team figured out which models actually work under the conditions of prohibiting the transfer of source code to third parties, and what the various training datasets presented in the publications are - how relevant they are to our tasks. In addition, the report included an analysis of the effectiveness of various prompts under different conditions. Naturally, we also received code snippets from the team that were used to conduct the experiments. I would like to note separately that following the Workshop, one of the participants of the BMM joined our team. With us, he continued to work on automatic vulnerability correction in the program code.

Could we have achieved this on our own within Astra, without involving the Big Mathematical Workshop? Of course, we could. But working together with the Workshop, on the one hand, gave us an additional impetus to start solving a promising, but not "urgent" task right now. On the other hand, it gave us additional resources during the difficult start-up period. It was also valuable for us to get a fresh look at the task of identifying vulnerabilities using machine learning in general, to check how much it corresponds to the established "consensus" that exists in our team at Astra. And of course, thanks to the Workshop, we got a new friend and a competent colleague.

Jamal, Junior Research Programmer:

I study applied mathematics at MAI and I wanted to gain experience working on an IT project with colleagues from the industry.

The most important thing is that the work took place in a certain environment: several teams and many people united by ideas were together. We constantly reflected and received feedback from experts and participants. Involvement in BMM not only provided useful experience and new acquaintances but also the development of important communication skills.

Would we recommend other organizations to participate in the Big Mathematical Workshop? Yes. But it is necessary to be aware of the limitations. Firstly, the task you bring to the Workshop should have mathematical content or imply a solution by mathematical methods. However, mathematics is such a pervasive science that many problems can be presented as mathematical. Secondly, the task should not be very urgent: the time from the application deadline (February 1) to the end of the face-to-face stage (end of July) is a whole six months. It seems that for these purposes, tasks from the conditional backlog, which the team left "to think about in the future," are best suited. It should be remembered that the task should be solved using means and data that are publicly available. And of course, you must be prepared for the fact that the method of solving the problem will also be published outside.