In this blog post, we’ll introduce our open-source eBPF Agent solution, focusing on its ability to intercept and monitor SSL/TLS encrypted traffic. This feature enables our costumers to detect traffic patterns that could be overloading the system and impacting performance. We will start with a theoretical overview and end with a practical code example that you can try out yourself.
The Challenge of SSL/TLS on Kubernetes
In our recent post, we explored the observability of the PostgreSQL (Layer 7) protocol. This involved attaching our eBPF programs to the Kernel tracepoints that capture application data sent and received over the socket. You can find the post at the following link.
However, secure communication channels pose a unique challenge. By the time data reaches the Linux send() and recv() syscalls for SSL/TLS traffic, it is already encrypted and indecipherable.
How can we effectively monitor this encrypted traffic without compromising security?
This is where eBPF technology comes into play. eBPF allows us to attach custom programs just before data is encrypted and sent, and after it is received and decrypted. While it’s possible to achieve this by hard-coding changes into your application, eBPF offers several significant advantages:
- No code changes required: eBPF operates independently of your application.
- Dynamic implementation: It can be added without restarting your app.
- Efficient resource usage: Often executed directly in the kernel, eBPF requires fewer CPU and memory resources compared to user-space telemetry collectors.
- Enhanced context extraction: It can gather much more application context than user-space provided attributes.
By leveraging eBPF, we can observe and monitor applications of all types, from monoliths to microservices, whether they are deployed as FaaS, on Kubernetes, or any other architecture you can imagine — even for SSL/TLS encrypted traffic. But the key question still remains: How does it work?
eBPF uprobes and Kubernetes Monitoring
In 2004, Linux introduced support for kernel probes (“kprobes”) with the 2.6 Linux Kernel, marking the beginning of the era of kernel function tracing. Kprobes allowed developers to attach small programs, or probes, to the kernel without modifying its source code. This innovation paved the way for numerous modern applications, ranging from performance monitoring and profiling to intrusion detection. Furthermore, with the release of the 3.5 Kernel, support for user probes (“uprobes”) was added, extending the capabilities of dynamically attached probes from the kernel into user space. While the primary distinction between kprobes and uprobes lies in their attachment points — kernel space and user space, respectively — there are also two other related types of probes:
- Kprobes/Uprobes: Gather information by intercepting functions as they are called.
- Kretprobes/Uretprobes: Executed when a function returns.
Nowadays, both types of probes can dynamically be attached to mostly all possible functions, but they differ in the data they capture. Kprobes and uprobes intercept input parameters to functions, while kretprobes and uretprobes capture returned values. These probes often work together, rather than as separate entities, to provide comprehensive tracing.
💡 If you’re searching for a specific kernel source function name, you can use
cat /proc/kallsyms. This command accesses the kernel's symbol table, which contains both function names and variable names.
As SSL/TLS traffic encryption is processed by user space libraries, kernel probes are of no help to us here. Thankfully, user probes can help us intercept unencrypted application data, ensuring we can effectively monitor and observe encrypted traffic — be it on bare metal servers or Kubernetes cluster.
We, at Anteon have developed an eBPF agent that does just that. To be more specific, we provide support for observing traffic encrypted using OpenSSL library including its legacy versions. In order to achieve that we hook onto:
SSL_write(): function used in the OpenSSL library to write data to an SSL/TLS connection. We primarily utilize it to intercept the data written by the client — not yet encrypted.
SEC("uprobe/SSL_write_v3")
void BPF_UPROBE(ssl_write_v3, void * ssl, void* buffer, int num) {
ssl_uprobe_write_v_3(ctx, ssl, buffer, num, 0);
}SSL_read(): function used in the OpenSSL library to read data from an SSL/TLS connection. It allows us to intercept the decrypted data received by the server as well as parsing the return code e.g. HTTP status code like 200 (Successful GET request).
SEC("uprobe/SSL_read_v3")
void BPF_UPROBE(ssl_read_enter_v3, void* ssl, void* buffer, int num) {
__u64 pid_tgid = bpf_get_current_pid_tgid();
__u32 pid = pid_tgid >> 32;
__u64 id = pid_tgid | TLS_MASK;
ssl_uprobe_read_enter_v3(ctx, id, pid, ssl, buffer, num, 0);
}
SEC("uretprobe/SSL_read")
void BPF_URETPROBE(ssl_ret_read_v3) {
__u64 pid_tgid = bpf_get_current_pid_tgid();
__u64 pid = pid_tgid >> 32;
__u64 id = pid_tgid | TLS_MASK;
int returnValue = PT_REGS_RC(ctx);
process_exit_of_syscalls_read_recvfrom(ctx, id, pid, returnValue, 1);
}💡 The interesting part is that we don’t even need any TLS/SSL certificates for decryption or encryption!
This allows us to observe the application protocol both before encryption and after decryption, and render the data onto our platform without compromising security. Here’s an example for HTTP/S traffic:
💡 Anteon platform demo is available on demo.getanteon.com. Give it a look.
The code snippets follow patterns of our Alaz eBPF Agent. The complete source code is available in our GitHub repository.
To be honest, there’s quite a bit of code surrounding the described functionality, primarily focused on extracting the buffer and conducting other protocol-related checks. Presenting Alaz in its entirety might be slightly complex for now. However, to provide you with a tangible example, we’ve prepared a focused demo code that incorporates only the features relevant to TLS/SSL and HTTP traffic. You can access it at the following link.
Performance Evaluation
One of the first concerns when deploying a program that intercepts and parses every message sent over your network is how much latency does it infer as well how much load is that on the host running the program. So we ran a couple of tests for you. The tests involved measuring the average latency over 1000 requests for each HTTP method type.
Our results indicate that the eBPF program adds a constant eBPF overhead of approximately 0.2µs on average — yes, remarkably low. Additionally, the average CPU load introduced by each hook is as follows: 0.1% for uprobe/SSL_write*, 0.007% for uprobe/SSL_read*, and 0.3% for uretprobe/SSL_read.
You can find the load testing programs in the
/perfdirectory of the repository referenced below.
These findings address the trade-off between the added latency and CPU load due to eBPF instrumentation and the benefits of TLS/SSL encrypted traffic observation and analysis.
Conclusion
By leveraging eBPF, our Alaz eBPF Agent can dynamically intercept and observe encrypted traffic, providing deep insights into application performance and traffic patterns. The remarkably low overhead and minimal CPU load make this approach ideal for high-confidentiality applications running in complex environments like Kubernetes. We invite you to explore our demo and GitHub repository to see the full potential of eBPF in action.
