How to create and load modules dynamically at runtime in Elixir, or Erlang

dynamicelixirerlangmodule

The basic scenario is this: I need to load text from a database, and then turn that text into an Elixir module (or an Erlang module) and then make calls into it. The text is effectively the same as a module file. So this is a form of hot code loading. I want to compile the "file" and then load the resulting module, then make calls into it. Later I'll unload it. The only difference is the code exists in a database rather than a file on the disk. (and it doesn't exist at the time that I'm writing the code that will be loading it.)

I know Erlang supports hot code loading, but seems focused on compiling files on disk and then loading the beams. I wish to do this as a more dynamic process, and I won't be replacing running code, but loading the code, then running it, then unloading it.

There are several facilities in Elixir for evaluating code at runtime. I'm trying to figure out how to do this with them, and the documentation is a bit sparse.

Code.compile_string(string, "nofile")

"returns a list of tuples where the first element is the module name and the second one is its binary". So, now I have the modules names and their binaries, but I do not know of a way to then load the binaries into the runtime and call into them. How would I do that? (There's no function for that in the Code library that I can see.)

Possibly I could then use the Erlang function:

:code.load_binary(Module, Filename, Binary)  ->
           {module, Module} | {error, What}

Ok, so this returns a tuple with the atom "module" and then the Module. If the string loaded from the database defined a module called "Paris", how in my code would I then execute

paris.handler([parameters])

since I don't know in advance that there will be a module called paris? I could know, by having the string "paris" also stored in the database that this is the name, but is there any way of calling into a module, using a string as the name of the module you're calling?

There is also:

eval(string, binding // [], opts // [])

Which evaluates the contents of the string. Can this string be the entire definition of a module? It appears not. I'd like to be able to write this code that's being evaluated in such a way that it has multiple functions that call each other–e.g. a complete little program, with a pre-defined entry point (Which could be a main, such as "DynamicModule.handle([parameter, list])"

Then there's the EEx module, which has:

compile_string(source, options // [])

Which is great for doing templates. But ultimately it only seems to work for the use case where there's a string and you've got Elixir code embedded in it. It evaluates the string in the context of the options and produces a string. I'm seeking to compile the string into one or more functions that I can then call. (If I can only make one function that's fine, that function can pattern match or switch into doing the other things that are needed….)

I know this is unconventional, but I have my reasons for doing it this way and they are good ones. I'm looking for advice about how to do this, but don't need to be told "don't do that". It seems like it should be possible, Erlang supports hot code loading and Elixir is pretty dynamic, but I just don't know the syntax, or the right functions. I'll be monitoring this question closely. Thanks in advance!

EDITS based on the first answers:

Thanks for the answers, this is good progress. As Yuri showed, eval can define a module, and as José points out, I can just use code eval for small parts of code with bindings.

The code being evaluated (whether turned into a module, or not) is going to be fairly complex. And its development would be best involving breaking it down into functions and calling those functions.

To help, let me provide some context. Assume I'm building a web framework. The code loaded in from the database is handlers for specific URIs. So, when an HTTP call comes in, I might load the code for example.com/blog/ This page might involve several different things, such as comments, a list of recent posts, etc.

Since many people are hitting the page at the same time, I'm spawning a process to handle each page view. Thus there are many times when this code may be evaluated simultaneously, for different requests.

The the module solution allows one to break the code up into functions for different parts of the page (eg: the list of posts, comments, etc. ) And I would load the module once, at startup, and let many processes spawn that call into it. The module is global, correct?

What happens if there's a module already defined? EG: When the module changes, and there are processes already calling that module.

In iex, I am able to redefine a module that's already been defined:

iex(20)> Code.eval "defmodule A do\ndef a do\n5\nend\nend"
nofile:1: redefining module A

What happens if I redefine the module at runtime, to all the processes currently calling into that module? Also, will this redefinition work outside of iex, in normal operation?

Assuming that redefining the module would be problematic, and that modules being global might run into problems with namespace collisions, I looked into using eval to define a function.

If I can merely have the code from the database define functions, then those functions are within the scope of whatever process, and we don't have the possibility of global collisions.

However, this doesn't seem to work:

iex(31)> q = "f = function do
...(31)> x, y when x > 0 -> x+y
...(31)> x, y -> x* y
...(31)> end"
"f = function do\nx, y when x > 0 -> x+y\nx, y -> x* y\nend"
iex(32)> Code.eval q
{#Fun<erl_eval.12.82930912>,[f: #Fun<erl_eval.12.82930912>]}
iex(33)> f
** (UndefinedFunctionError) undefined function: IEx.Helpers.f/0
    IEx.Helpers.f()
    erl_eval.erl:572: :erl_eval.do_apply/6
    src/elixir.erl:110: :elixir.eval_forms/3
    /Users/jose/Work/github/elixir/lib/iex/lib/iex/loop.ex:18: IEx.Loop.do_loop/1

iex(33)> f.(1,3)
** (UndefinedFunctionError) undefined function: IEx.Helpers.f/0
    IEx.Helpers.f()
    erl_eval.erl:572: :erl_eval.do_apply/6
    erl_eval.erl:355: :erl_eval.expr/5
    src/elixir.erl:110: :elixir.eval_forms/3
    /Users/jose/Work/github/elixir/lib/iex/lib/iex/loop.ex:18: IEx.Loop.do_loop/1

I also tried:

    iex(17)> y = Code.eval "fn(a,b) -> a + b end"
{#Fun<erl_eval.12.82930912>,[]}
iex(18)> y.(1,2)
** (BadFunctionError) bad function: {#Fun<erl_eval.12.82930912>,[]}
    erl_eval.erl:559: :erl_eval.do_apply/5
    src/elixir.erl:110: :elixir.eval_forms/3
    /Users/jose/Work/github/elixir/lib/iex/lib/iex/loop.ex:18: IEx.Loop.do_loop/1

So, in summary:

Can modules be redefined using Code.eval when there are processes calling into them?
Is it possible to use Code.eval to make functions whose scope is bound to the process in which Code.eval was called?
If you understand what it is I'm trying to do, can you suggest a better way to go about it?

Also, if there is a better forum where I should be asking this, feel free to let me know. And if there are docs or relevant examples I should read, please feel free to point me to them. I'm not trying to get you to do all the work, I just am unable to figure it out myself.

I'm learning Elixir specifically for the ability to dynamically evlauate code, but my Elixir knowledge is minimal now- I just started- and my erlang is rusty too.

Much thanks!

Best Answer

As you described, there are many different approaches you could take by ultimately they boil down to two different categories: 1) code compilation and 2) code evaluation. The example you described above requires compilation, which will define a module and then you would have to invoke it. However, as you found out, it requires defining a module name and potentially purging and discarding those modules when the database changes. Also, note that, defining modules may exhaust the atom table, as an atom is created for every module. I would only use this approach if you need to compile at maximum a dozen modules.

(A small note, Code.compile_string already defines the module, so you don't need to call load_binary).

Maybe a simpler approach is to call Code.eval passing the code from the database, thus code evaluation. It works fine if all you want to is to evaluate some custom rules. The Code.eval accepts a binding, which would allow you to pass parameters to the code. Let's suppose you have "a + b" stored in the database, where a and b are the parameters, you could evaluate it as:

Code.eval "a + b", [a: 1, b: 1]

EDIT BASED ON QUESTION'S EDITS

If you are evaluating code, keep in mind that Code.eval returns the result of evaluating the code and the new binding, so your example above would be better written as:

q = "f = function do
x, y when x > 0 -> x+y
x, y -> x* y
end"

{ _, binding } = Code.eval q
binding[:f].(1, 2)

However, given your use case, I wouldn't consider using eval but I would indeed compile a module. Since the information is coming from the database, you can use this fact to generate unique modules per record for you. For example, you can assume developers will be adding the contents of a module to the database, something like:

def foo, do: 1
def bar, do: 1

After you get this information from the database, you can compile it using:

record   = get_record_from_database
id       = record.id
contents = Code.string_to_quoted!(record.body)
module   = Module.concat(FromDB, "Data#{record.id}")
Module.create module, contents, Macro.Env.location(__ENV__)
module

Where record is anything that you get back from the database. Assuming the record id is 1, it would define a module FromDB.Data1 which then you would be able to invoke foo and bar.

Regarding code reloading, both defmodule and Module.create use :code.load_binary to load the module. Which means that if you update the module, the old version is still kept around until another new version is loaded.

One of the things you should add as well is cache expiration, so you don't need to compile a module on every request. This can be done if you have a lock_version in the database that you increment every time you change the record contents. The final code would be something like:

record  = get_record_from_database
module  = Module.concat(FromDB, "Data#{record.id}")
compile = :code.is_loaded(module) == false or
            record.lock_version > module.lock_version

if compile do
  id       = record.id
  contents = Code.string_to_quoted!(record.body)
  contents = quote do
    def lock_version, do: unquote(record.lock_version)
    unquote(contents)
  end
  Module.create module, contents, Macro.Env.location(__ENV__)
end

module

Related Solutions

Node.js – Declare multiple module.exports in Node.js

You can do something like:

module.exports = {
    method: function() {},
    otherMethod: function() {},
};

Or just:

exports.method = function() {};
exports.otherMethod = function() {};

Then in the calling script:

const myModule = require('./myModule.js');
const method = myModule.method;
const otherMethod = myModule.otherMethod;
// OR:
const {method, otherMethod} = require('./myModule.js');

Why are there two kinds of functions in Elixir

Just to clarify the naming, they are both functions. One is a named function and the other is an anonymous one. But you are right, they work somewhat differently and I am going to illustrate why they work like that.

Let's start with the second, fn. fn is a closure, similar to a lambda in Ruby. We can create it as follows:

x = 1
fun = fn y -> x + y end
fun.(2) #=> 3

A function can have multiple clauses too:

x = 1
fun = fn
  y when y < 0 -> x - y
  y -> x + y
end
fun.(2) #=> 3
fun.(-2) #=> 3

Now, let's try something different. Let's try to define different clauses expecting a different number of arguments:

fn
  x, y -> x + y
  x -> x
end
** (SyntaxError) cannot mix clauses with different arities in function definition

Oh no! We get an error! We cannot mix clauses that expect a different number of arguments. A function always has a fixed arity.

Now, let's talk about the named functions:

def hello(x, y) do
  x + y
end

As expected, they have a name and they can also receive some arguments. However, they are not closures:

x = 1
def hello(y) do
  x + y
end

This code will fail to compile because every time you see a def, you get an empty variable scope. That is an important difference between them. I particularly like the fact that each named function starts with a clean slate and you don't get the variables of different scopes all mixed up together. You have a clear boundary.

We could retrieve the named hello function above as an anonymous function. You mentioned it yourself:

other_function(&hello(&1))

And then you asked, why I cannot simply pass it as hello as in other languages? That's because functions in Elixir are identified by name and arity. So a function that expects two arguments is a different function than one that expects three, even if they had the same name. So if we simply passed hello, we would have no idea which hello you actually meant. The one with two, three or four arguments? This is exactly the same reason why we can't create an anonymous function with clauses with different arities.

Since Elixir v0.10.1, we have a syntax to capture named functions:

&hello/1

That will capture the local named function hello with arity 1. Throughout the language and its documentation, it is very common to identify functions in this hello/1 syntax.

This is also why Elixir uses a dot for calling anonymous functions. Since you can't simply pass hello around as a function, instead you need to explicitly capture it, there is a natural distinction between named and anonymous functions and a distinct syntax for calling each makes everything a bit more explicit (Lispers would be familiar with this due to the Lisp 1 vs. Lisp 2 discussion).

Overall, those are the reasons why we have two functions and why they behave differently.

Best Answer

Related Solutions

Node.js – Declare multiple module.exports in Node.js

Why are there two kinds of functions in Elixir

Related Topic