- Network
- A
librats: a new C++ library for distributed P2P applications
Hello everyone! I am the creator of the distributed search engine rats-search based on DHT. Its principle is quite simple: the search engine collects torrents from all network participants and creates a large distributed database for search, including metadata.
Hello everyone! I am the creator of the distributed search engine rats-search based on DHT ( GitHub ). Its operating principle is quite simple: the search engine collects torrents from all network participants and forms a large distributed database for searching, including metadata (such as descriptions and other information).
In this article, I want to talk about my new library for building distributed (p2p) applications, where knowing IP addresses of participants is not necessary, and searching is done through various protocols — DHT, mDNS, peer exchange, and others. I think, given the constant troubles happening around, this could turn out to be useful ;).
A bit of history
rats-search is quite an old project; I wrote it many years ago in Electron. Now there’s a need to rewrite the p2p component in C++ to improve connection quality and performance.
At first, just out of curiosity, I tried moving everything to libp2p and was unpleasantly surprised by its performance. Literally, with ~100 peers, the processor (Ryzen 5700) was loaded to 100% during DHT network steps, and memory consumption reached 500–600 MB. Of course, I realize that libp2p uses trendy coroutine libraries like it- and drags along 300–400 MB of dependencies in node_modules, but that’s just not right.
Yes, it’s JS, and direct comparison with C++ is not quite fair, but even JS applications usually don’t produce such poor results. Moreover, libp2p doesn’t support all languages: some implementations are stripped down, some are missing entirely. For example, in the C++ version, the mDNS protocol is simply not implemented. As a result, only Go and JS implementations are considered benchmark, and they differ noticeably from each other.
Why I decided to write my own library
I decided to abandon this “combine-implementation” and write my own. As a base, I chose an efficient low-level language, and then bindings to other languages can be built. This allows for a universal and truly efficient single implementation that can be used in other projects as well.
This is how librats appeared — a new C++ library that will become the future core of Rats, and can also be used in any other distributed applications.
What has already been implemented
Participant search via DHT
Communication between participants
Data / file / directory exchange
mDNS protocol for discovering participants in local networks
Historical peers — database of participants for reconnections
gossipsub protocol — for building message exchange in large distributed networks
Peer exchange
Support for custom protocols
ICE / STUN protocols
Encryption based on the noise protocol (Curve25519 + ChaCha20-Poly1305)
Supported platforms and languages
Build for Windows, macOS, Linux, Android
Compilers: GCC, Clang, MSVC
Usage: directly in C++, also in C and Java via JNI; in Android — via NDK
Examples
Now let's look at some technical examples and solutions to certain tasks.
Basic example: creating a client
#include "librats.h"
#include
#include
#include
int main() {
// Creating a simple P2P client
librats::RatsClient client(8080);
// Setting up a connection callback
client.set_connection_callback([](socket_t socket, const std::string& peer_id) {
std::cout << "✅ A new peer connected: " << peer_id << std::endl;
});
// Setting up a message callback
client.set_string_data_callback([](socket_t socket, const std::string& peer_id, const std::string& message) {
std::cout << "💬 Message from " << peer_id << ": " << message << std::endl;
});
// Starting the client
if (!client.start()) {
std::cerr << "Failed to start the client" << std::endl;
return 1;
}
std::cout << "🐀 librats client started on port 8080" << std::endl;
// Connecting to another peer (optional)
// client.connect_to_peer("127.0.0.1", 8081);
// Sending a message to all connected peers
client.broadcast_string_to_peers("Hello from librats!");
// Letting the program run
std::this_thread::sleep_for(std::chrono::minutes(1));
return 0;
} The core of the work involves transmitting three types of data:
binary,
textual,
JSON data.
You can choose listeners and configure the data transmission type depending on the task.
Defining your own protocol
You can set your own protocol. This allows you to automatically find only those participants who are working with your protocol or application.
#include "librats.h"
#include
int main() {
librats::RatsClient client(8080);
// Setting up a custom protocol for your application
client.set_protocol_name("my_app");
client.set_protocol_version("1.0");
std::cout << "Protocol: " << client.get_protocol_name()
<< " v" << client.get_protocol_version() << std::endl;
std::cout << "Discovery hash: " << client.get_discovery_hash() << std::endl;
client.start();
// Starting DHT discovery with a custom protocol
if (client.start_dht_discovery()) {
// Announcing our presence
client.announce_for_hash(client.get_discovery_hash());
// Searching for other peers using the same protocol
client.find_peers_by_hash(client.get_discovery_hash(),
[](const std::vector& peers) {
std::cout << "Found " << peers.size() << " peers" << std::endl;
});
}
return 0;
} Building a simple chat
A fairly typical task is organizing a chat. To start, you can use simple functions without applying a mesh network.
#include "librats.h"
#include
#include
int main() {
librats::RatsClient client(8080);
// Setting up message handlers using modern API
client.on("chat", [](const std::string& peer_id, const nlohmann::json& data) {
std::cout << "[CHAT] " << peer_id << ": " << data["message"].get() << std::endl;
});
client.on("user_join", [](const std::string& peer_id, const nlohmann::json& data) {
std::cout << "[JOIN] " << data["username"].get() << " joined" << std::endl;
});
// Callback on connection
client.set_connection_callback([&](socket_t socket, const std::string& peer_id) {
std::cout << "✅ Peer connected: " << peer_id << std::endl;
// Sending a welcome message
nlohmann::json welcome;
welcome["username"] = "User_" + client.get_our_peer_id().substr(0, 8);
client.send("user_join", welcome);
});
client.start();
// Sending a chat message
nlohmann::json chat_msg;
chat_msg["message"] = "Hello, P2P chat!";
chat_msg["timestamp"] = std::time(nullptr);
client.send("chat", chat_msg);
return 0;
} Working with Mesh Networks
If a larger network is needed (for example, to exchange messages between a large number of nodes), the gossipsub protocol can be used.
#include "librats.h"
#include
int main() {
librats::RatsClient client(8080);
// Setting up message handlers for topics
client.on_topic_message("news", [](const std::string& peer_id, const std::string& topic, const std::string& message) {
std::cout << "📰 [" << topic << "] " << peer_id << ": " << message << std::endl;
});
client.on_topic_json_message("events", [](const std::string& peer_id, const std::string& topic, const nlohmann::json& data) {
std::cout << "🎉 [" << topic << "] Event: " << data["type"].get() << std::endl;
});
// Notifications of peer joining/leaving
client.on_topic_peer_joined("news", [](const std::string& peer_id, const std::string& topic) {
std::cout << "➕ " << peer_id << " joined " << topic << std::endl;
});
client.start();
client.start_dht_discovery();
// Subscribing to topics
client.subscribe_to_topic("news");
client.subscribe_to_topic("events");
// Publishing messages
client.publish_to_topic("news", "Breaking: librats is awesome!");
nlohmann::json event;
event["type"] = "celebration";
event["reason"] = "successful_connection";
client.publish_json_to_topic("events", event);
std::cout << "📊 Peers in 'news': " << client.get_topic_peers("news").size() << std::endl;
return 0;
} Data Exchange
The library supports data exchange, including files and entire directories.
#include "librats.h"
#include
int main() {
librats::RatsClient client(8080);
// Setting up file transfer callbacks
client.on_file_transfer_progress([](const librats::FileTransferProgress& progress) {
std::cout << "📁 Transfer " << progress.transfer_id.substr(0, 8)
<< ": " << progress.get_completion_percentage() << "% complete"
<< " (" << (progress.transfer_rate_bps / 1024) << " KB/s)" << std::endl;
});
client.on_file_transfer_completed([](const std::string& transfer_id, bool success, const std::string& error) {
if (success) {
std::cout << "✅ Transfer completed: " << transfer_id.substr(0, 8) << std::endl;
} else {
std::cout << "❌ Transfer error: " << error << std::endl;
}
});
// Automatic acceptance of incoming file transfers
client.on_file_transfer_request([](const std::string& peer_id,
const librats::FileMetadata& metadata,
const std::string& transfer_id) {
std::cout << "📥 Incoming file: " << metadata.filename
<< " (" << metadata.file_size << " bytes) from " << peer_id.substr(0, 8) << std::endl;
return true; // Auto-accept
});
// Allow file requests from the "shared" directory
client.on_file_request([](const std::string& peer_id, const std::string& file_path, const std::string& transfer_id) {
std::cout << "📤 Request: " << file_path << " from " << peer_id.substr(0, 8) << std::endl;
return file_path.find("../") == std::string::npos; // Path traversal protection
});
client.start();
// File transfer settings
librats::FileTransferConfig config;
config.chunk_size = 64 * 1024; // chunks of 64KB
config.max_concurrent_chunks = 4; // 4 parallel chunks
config.verify_checksums = true; // integrity check
client.set_file_transfer_config(config);
// Transfer examples (replace "peer_id" with a real peer ID)
// std::string file_transfer = client.send_file("peer_id", "my_file.txt");
// std::string dir_transfer = client.send_directory("peer_id", "./my_folder");
// std::string file_request = client.request_file("peer_id", "remote_file.txt", "./downloaded_file.txt");
std::cout << "File transfer is ready. Connect peers and exchange files!" << std::endl;
return 0;
} Configuration setup
For example, you can set configuration for logging:
#include "librats.h"
#include
int main() {
librats::RatsClient client(8080);
// Enable and configure logging
client.set_logging_enabled(true);
client.set_log_file_path("librats_app.log");
client.set_log_level("INFO"); // DEBUG, INFO, WARN, ERROR
client.set_log_colors_enabled(true);
client.set_log_timestamps_enabled(true);
// Configure log file rotation
client.set_log_rotation_size(5 * 1024 * 1024); // max file size 5MB
client.set_log_retention_count(3); // keep 3 old log files
std::cout << "📝 Logging to: " << client.get_log_file_path() << std::endl;
std::cout << "📊 Log level: " << static_cast(client.get_log_level()) << std::endl;
std::cout << "🎨 Color logging: " << (client.is_log_colors_enabled() ? "Yes" : "No") << std::endl;
client.start();
// All librats operations will now be logged
client.broadcast_string_to_peers("This action will be recorded in the logs!");
// Clear log file if needed (uncomment to use)
// client.clear_log_file();
return 0;
} And you can work with data saving configuration:
#include "librats.h"
#include
int main() {
librats::RatsClient client(8080);
// Set custom data directory for config files
client.set_data_directory("./my_app_data");
// Load saved configuration (if exists)
if (client.load_configuration()) {
std::cout << "📄 Loaded existing configuration" << std::endl;
} else {
std::cout << "📄 Using default configuration" << std::endl;
}
// Get our persistent peer ID
std::cout << "🆔 Our peer ID: " << client.get_our_peer_id() << std::endl;
client.start();
// Try to reconnect to previously connected peers
int reconnect_attempts = client.load_and_reconnect_peers();
std::cout << "🔄 Attempted to reconnect to " << reconnect_attempts << " previous peers" << std::endl;
// Configuration is automatically saved when client stops
// Files created: config.json, peers.rats, peers_ever.rats
// Manual save if needed
client.save_configuration();
client.save_historical_peers();
std::cout << "💾 Configuration will be saved to: " << client.get_data_directory() << std::endl;
return 0;
} The code provides not only configuration saving but also the history of connections. This is useful for future launches: if you need to quickly restore connections, you can use the saved data.
Imagine the situation: a "bird" flew into the servers of a local law-abiding provider, and some routes or connections suddenly got blocked. In this case, having a database of known peers allows quickly restoring the network operation without the need to search for all participants again.
And lastly, since other languages are mentioned, here's an example in Java for Android. I decided to present the full Activity example, focusing on the setupRatsClient() method:
public class MainActivity extends AppCompatActivity {
private static final String TAG = "LibRatsExample";
private static final int PERMISSION_REQUEST_CODE = 1;
private RatsClient ratsClient;
private TextView statusText;
private TextView messagesText;
private EditText hostInput;
private EditText portInput;
private EditText messageInput;
private Button startButton;
private Button connectButton;
private Button sendButton;
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
initViews();
checkPermissions();
setupRatsClient();
}
private void initViews() {
statusText = findViewById(R.id.statusText);
messagesText = findViewById(R.id.messagesText);
hostInput = findViewById(R.id.hostInput);
portInput = findViewById(R.id.portInput);
messageInput = findViewById(R.id.messageInput);
startButton = findViewById(R.id.startButton);
connectButton = findViewById(R.id.connectButton);
sendButton = findViewById(R.id.sendButton);
startButton.setOnClickListener(this::onStartClicked);
connectButton.setOnClickListener(this::onConnectClicked);
sendButton.setOnClickListener(this::onSendClicked);
// Set default values
hostInput.setText("192.168.1.100");
portInput.setText("8080");
messageInput.setText("Hello from Android!");
updateUI();
}
private void checkPermissions() {
String[] permissions = {
Manifest.permission.INTERNET,
Manifest.permission.ACCESS_NETWORK_STATE,
Manifest.permission.ACCESS_WIFI_STATE,
Manifest.permission.CHANGE_WIFI_MULTICAST_STATE
};
boolean allGranted = true;
for (String permission : permissions) {
if (ContextCompat.checkSelfPermission(this, permission) != PackageManager.PERMISSION_GRANTED) {
allGranted = false;
break;
}
}
if (!allGranted) {
ActivityCompat.requestPermissions(this, permissions, PERMISSION_REQUEST_CODE);
}
}
private void setupRatsClient() {
try {
// Enable logging
RatsClient.setLoggingEnabled(true);
RatsClient.setLogLevel("INFO");
// Create client on port 8080
ratsClient = new RatsClient(8080);
// Set up callbacks
ratsClient.setConnectionCallback(new ConnectionCallback() {
@Override
public void onConnection(String peerId) {
runOnUiThread(() -> {
appendMessage("Connected to peer: " + peerId);
updateUI();
});
}
});
ratsClient.setStringCallback(new StringMessageCallback() {
@Override
public void onStringMessage(String peerId, String message) {
runOnUiThread(() -> {
appendMessage("Message from " + peerId + ": " + message);
});
}
});
ratsClient.setDisconnectCallback(new DisconnectCallback() {
@Override
public void onDisconnect(String peerId) {
runOnUiThread(() -> {
appendMessage("Disconnected from peer: " + peerId);
updateUI();
});
}
});
appendMessage("LibRats client successfully created");
appendMessage("Version: " + RatsClient.getVersionString());
} catch (Exception e) {
Log.e(TAG, "Failed to create RatsClient", e);
appendMessage("Error: " + e.getMessage());
}
}
private void onStartClicked(View view) {
if (ratsClient == null) return;
try {
int result = ratsClient.start();
if (result == RatsClient.SUCCESS) {
appendMessage("Client started successfully");
appendMessage("Our Peer ID: " + ratsClient.getOurPeerId());
updateUI();
} else {
appendMessage("Failed to start client: " + result);
}
} catch (Exception e) {
Log.e(TAG, "Error starting client", e);
appendMessage("Error starting client: " + e.getMessage());
}
}
private void onConnectClicked(View view) {
if (ratsClient == null) return;
String host = hostInput.getText().toString().trim();
String portStr = portInput.getText().toString().trim();
if (host.isEmpty() || portStr.isEmpty()) {
Toast.makeText(this, "Enter host and port", Toast.LENGTH_SHORT).show();
return;
}
try {
int port = Integer.parseInt(portStr);
int result = ratsClient.connectWithStrategy(host, port, RatsClient.STRATEGY_AUTO_ADAPTIVE);
if (result == RatsClient.SUCCESS) {
appendMessage("Connecting to " + host + ":" + port);
} else {
appendMessage("Failed to connect: " + result);
}
} catch (NumberFormatException e) {
Toast.makeText(this, "Invalid port number", Toast.LENGTH_SHORT).show();
} catch (Exception e) {
Log.e(TAG, "Connection error", e);
appendMessage("Connection error: " + e.getMessage());
}
}
private void onSendClicked(View view) {
if (ratsClient == null) return;
String message = messageInput.getText().toString().trim();
if (message.isEmpty()) {
Toast.makeText(this, "Enter a message", Toast.LENGTH_SHORT).show();
return;
}
try {
// Get connected peers
String[] peerIds = ratsClient.getPeerIds();
if (peerIds.length == 0) {
Toast.makeText(this, "No connected peers", Toast.LENGTH_SHORT).show();
return;
}
// Send to the first connected peer
int result = ratsClient.sendString(peerIds[0], message);
if (result == RatsClient.SUCCESS) {
appendMessage("Sent: " + message);
messageInput.setText("");
} else {
appendMessage("Failed to send message: " + result);
}
} catch (Exception e) {
Log.e(TAG, "Error sending message", e);
appendMessage("Error sending message: " + e.getMessage());
}
}
private void appendMessage(String message) {
Log.d(TAG, message);
messagesText.append(message + "\n");
// Scroll down
messagesText.post(() -> {
int scrollAmount = messagesText.getLayout().getLineTop(messagesText.getLineCount())
- messagesText.getHeight();
if (scrollAmount > 0) {
messagesText.scrollTo(0, scrollAmount);
} else {
messagesText.scrollTo(0, 0);
}
});
}
private void updateUI() {
if (ratsClient == null) {
statusText.setText("Status: Not initialized");
startButton.setEnabled(false);
connectButton.setEnabled(false);
sendButton.setEnabled(false);
return;
}
try {
int peerCount = ratsClient.getPeerCount();
statusText.setText("Status: " + peerCount + " peers connected");
// Enable/disable buttons depending on state
startButton.setEnabled(true);
connectButton.setEnabled(true);
sendButton.setEnabled(peerCount > 0);
} catch (Exception e) {
statusText.setText("Status: Error");
Log.e(TAG, "Error updating UI", e);
}
}
@Override
protected void onDestroy() {
super.onDestroy();
if (ratsClient != null) {
try {
ratsClient.stop();
ratsClient.destroy();
} catch (Exception e) {
Log.e(TAG, "Error destroying client", e);
}
}
}
@Override
public void onRequestPermissionsResult(int requestCode, String[] permissions, int[] grantResults) {
super.onRequestPermissionsResult(requestCode, permissions, grantResults);
if (requestCode == PERMISSION_REQUEST_CODE) {
boolean allGranted = true;
for (int result : grantResults) {
if (result != PackageManager.PERMISSION_GRANTED) {
allGranted = false;
break;
}
}
if (!allGranted) {
Toast.makeText(this, "Network permissions are required for LibRats", Toast.LENGTH_LONG).show();
}
}
}
}Conclusion
And finally — about performance. Remember those 500–600 MB of memory when running p2p via libp2p? Let's see what happens with librats:
…isn't the result clearly better?
I hope this library proves useful. Thank you for your attention! I may expand the article as bindings for other languages are added and new functions are implemented.
Write comment