Hacker's Handbook

Unleashing the Power of Erlang's BEAM

A case study with Delta Exchange

Today I will share some insight from our collaboration with Delta Exchange (aka Delta), a leading player in the cryptocurrency trading space. At HappiHacking, our team of experts had the privilege of working closely with Delta Exchange to address their architectural challenges, scale their system, and provide comprehensive education on the BEAM virtual machine.

This case study highlights some of the things we do best at HappiHacking. We help our customers to help themselves. With our experience in software development and specifically from high-performance BEAM systems, we can quickly pinpoint problems and suggest solutions. It also shows our ability to share our experience and educate our customers.

We call this service Rubberducking as a Service™ (RDaaS™).

Part 1: Architectural Review and Scalability Challenges

Delta Exchange has been at the forefront, providing a robust platform for matching trades between buyers and sellers. However, every successful system faces its share of challenges, and Delta Exchange is no exception. In the world of high-volume cryptocurrency trading, scalability, and efficient processing are paramount. Delta Exchange approached us with a specific need - to optimize their application's performance, enhance distribution across nodes, and identify any architectural bottlenecks.

We meticulously reviewed their existing architecture, considering the nuances of the BEAM scheduler, and identified areas for improvement. From optimizing process distribution and supervisor structures to investigating run queue size fluctuations, we left no stone unturned in our pursuit of architectural excellence.

The RabbitMQ Conundrum:

During our work, Delta encountered a problem with RabbitMQ overloading Mnesia (Erlang's built-in database). This raised concerns and prompted the need to understand the issue in depth to prevent it from happening again. To investigate the problem, HappiHacking engaged in a detailed analysis of the RabbitMQ architecture. It was revealed that the size of certain RabbitMQ files indicated considerable activity in the corresponding tables. While the Mnesia warning was deemed harmless, the sheer number of queues raised eyebrows and pointed to potential architectural considerations.

Optimizations and Learnings:

The team at Delta Exchange delved into optimizing their system to address the challenges they encountered. They identified areas for improvement, such as reducing redundant data in inter-process communication and migrating away from Mnesia to alternative data storage solutions like ETS and Redis. These optimizations not only enhanced system efficiency but also paved the way for a more streamlined and scalable architecture.

Unraveling the GenServer Timeouts

We also discovered some problems with GenServer call timeouts. With GenServer processes timing out during handle calls, it became apparent that congestion was a likely culprit. Tobias recommended examining network throughput and monitoring bursts in Erlang processes and message queues to identify the point of origin. Additionally, increasing the distribution buffer busy limit and investigating network-level congestion were suggested as potential solutions.

Tools for Monitoring and Postmortem Analysis:

Delta also sought advice on monitoring tools to aid in tracking bursts in Erlang processes and message queues. Tobias highlighted the importance of having systems in place to report such events and examine them postmortem. While interactive monitoring through tools like the Observer CLI and sorting processes based on reductions and message queues can be helpful, it's advantageous to have a comprehensive monitoring infrastructure to gain deeper insights into system behavior.

We have continued to develop this course, you can read more about it in the post Unlocking Performance: Introducing HappiHacking's New Course on BEAM.

Part 2: Harnessing the Power of BEAM VM

To empower Delta Exchange's team with in-depth knowledge, we designed a series of comprehensive BEAM VM courses. Our experienced instructors educated Delta Exchange personnel on the inner workings of the BEAM virtual machine, covering critical topics such as processes, memory subsystems, networking, debugging, and more. Through engaging lectures, hands-on exercises, and ongoing support, we equipped Delta Exchange's team with the tools and insights needed to harness the full potential of BEAM.

Outcomes and Achievements:

Delta Exchange's experience with scaling challenges and production insights demonstrates their commitment to providing a high-performance trading platform.

Thanks to our collaborative efforts, Delta Exchange experienced significant improvements in its system's performance, scalability, and overall architecture. The architectural review led to the implementation of optimized process distribution, enhanced supervisor structures, and proactive measures to mitigate run queue size issues. Additionally, the BEAM VM education equipped Delta Exchange's personnel with a deep understanding of the virtual machine, empowering them to make informed decisions and drive continuous improvement.

One notable optimization was the reduction of redundant data in inter-process communication. By carefully optimizing payloads and leveraging technologies like 'protobuf', Delta Exchange significantly reduced payload sizes, resulting in enhanced efficiency and improved overall system performance.

Additionally, Delta Exchange made a strategic decision to migrate away from Mnesia and embrace alternative data storage solutions like ETS and Redis. This shift not only provided them with more flexibility but also allowed them to leverage the unique capabilities of these technologies for their specific use cases.

Their commitment to continuous improvement and willingness to explore new technologies sets them apart in the fast-paced world of cryptocurrency trading.

As Delta Exchange continues to push the boundaries of cryptocurrency trading, they stand ready to face future challenges armed with knowledge and experience. The lessons learned from their RabbitMQ adventures serve as a reminder that continuous optimization and monitoring are essential components of building and maintaining a robust and scalable system.

We're proud to have played a pivotal role in helping Delta Exchange tackle its scalability challenges and unlock the true power of the BEAM virtual machine. Our collaboration stands as a testament to the transformative impact of architectural review, scalability solutions, and comprehensive education.

Are you ready to unleash the full potential of your software system? Contact HappiHacking today and enjoy a world of optimized performance, scalable architecture, and unrivaled expertise.