Andrea Leopardi's Blog, page 2

RPC over RabbitMQ (with Elixir)

At Community we use RabbitMQ, a lot. It���s the infrastructure backbone that allows our services (over fourty at this point) to communicate with each other. That mostly happens through events (since we have an event-sourced system), but in some cases what we need is a request-response interaction between two services. This is the best tool in a few use cases, like retrieving data on the fly or asking a service to do something and return a response. An industry standard for such interactions is HTTP, but we are not big fans of that. Instead, since RabbitMQ is so ubiquitous in our system, we settled on using it for request-response interactions as well in the form of Remote Procedure Calls (RPCs). In this post, I���ll go over the architecture of such interactions, the RabbitMQ topologies we use to make them work, the benefits around reliability and the compromises around performance, and finally how this all implemented to be as fault-tolerant as possible with Elixir.

Process pools with Elixir's Registry

<p>When you have a limited number of resources that you have to share for your all application, like database connections or worker processes, what you need is a <em>pool</em>. In this post, we're going to take a look at one possible pooling strategy that highly leverages Elixir's built-in <a href="https://hexdocs.pm/elixir/Registry.ht... resulting in fast, reliable, and cleanly designed pools.</p>

Process pools with Elixir's Registry

When you have a limited number of resources that you have to share for your all application, like database connections or worker processes, what you need is a pool. In this post, we���re going to take a look at one possible pooling strategy that highly leverages Elixir���s built-in Registry resulting in fast, reliable, and cleanly designed pools.

Sharing Protobuf schemas across services

The system that we���re building at Community.com is made of a few services (around fifteen at the time of writing) that interact with each other through a basic version of event sourcing. All events are exchanged (published and consumed) through RabbitMQ and are serialized with Protobuf. With several services already and many more coming in the future, managing the Protobuf schemas becomes a painful part of evolving and maintaining the system. Do we copy the schemas in all services? Do we keep them somewhere and use something akin to Git submodules to keep them in sync in all of our projects? What do we do?! In this post, I���ll go through the tooling that we came up with in order to sanely manage our Protobuf schemas throughout our services and technology stack.

Persistent connections with gen_statem

Our applications often interact with external systems. In many cases, we need a persistent connection to one or more of these external services. For example, if your application makes continuous use of a database, you���ll likely want to stay connected to such database so that you can avoid spending time and resources connecting and disconnecting each time you perform a request. With Erlang and Elixir, the natural abstraction to maintain a persistent connection is a process. In this post, we���ll have a look at how we can take advantage of the gen_statem behaviour to write state machine processes that act as persistent connections to external systems.

A story of regret and retiring a library from Hex

Now the story of an Elixir library and the one author who had no choice but to take it away.

The guts of a property testing library

Property-based testing is a common tool to improve testing by testing properties of a piece of software over many values drawn at random from a large space of valid values. This methodology was first introduced in the paper QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs, which describes the basic idea and shows a possible implementation in Haskell. Since then, many tools to aid in property based testing appeared for many programming languages: as of the time of writing, there���s libraries for Haskell, Erlang, Clojure, Python, Scala, and many others. A few days ago I released the first version of StreamData, a property testing (and data generation) library for Elixir (that is a candidate for inclusion in Elixir itself in the future). This post is not an introduction to property-based testing nor a tutorial on how to use StreamData: what I want to do is dig into the mechanics of how StreamData works, its design, and how it compares to some of the other property-based testing libraries mentioned above.

Using C from Elixir with NIFs

Erlang supports a way to implement functions in C and use them transparently from Erlang. These functions are called NIFs (native implemented functions). There are two scenarios where NIFs can turn out to be the perfect solution: when you need raw computing speed and when you need to interface to existing C bindings from Erlang. In this article, we���re going to take a look at both use cases.

Compile-time work with Elixir macros

Macros are a very common way to do metaprogramming in Elixir. There are manyresources that explain what macros are and how to use them (much better than Icould): there���s the Macro chapter from the���Getting Started��� guide on Elixir���s website, an awesomeseries of articles by Sa��a Juri��, and even abook (Metaprogramming Elixir) by Chris McCord. In thisarticle, I���ll assume you are familiar with macros and how they work and I���lltalk about another use case of macros that is rarely examined: doingcompile-time things in macros.

Handling TCP connections in Elixir

Elixir is frequently used in network-aware applications because of the core design of Erlang and the Erlang VM. In this context, there���s often the need to connect to external services through the network: for example, a classic web application could connect to a relational database and a key-value store, while an application that runs on embedded systems could connect to other nodes on the network.