How do you create and load modules dynamically at runtime in Elixir, or Erlang?

Question

The basic scenario is this: I need to load text from a database, and then turn that text into an Elixir module (or an Erlang module) and then make calls into it. The text is effectively the same as a module file. So this is a form of hot code loading. I want to compile the "file" and then load the resulting module, then make calls into it. Later I'll unload it. The only difference is the code exists in a database rather than a file on the disk. (and it doesn't exist at the time that I'm writing the code that will be loading it.)

I know Erlang supports hot code loading, but seems focused on compiling files on disk and then loading the beams. I wish to do this as a more dynamic process, and I won't be replacing running code, but loading the code, then running it, then unloading it.

There are several facilities in Elixir for evaluating code at runtime. I'm trying to figure out how to do this with them, and the documentation is a bit sparse.

Code.compile_string(string, "nofile")

"returns a list of tuples where the first element is the module name and the second one is its binary". So, now I have the modules names and their binaries, but I do not know of a way to then load the binaries into the runtime and call into them. How would I do that? (There's no function for that in the Code library that I can see.)

Possibly I could then use the Erlang function:

:code.load_binary(Module, Filename, Binary)  ->
           {module, Module} | {error, What}

Ok, so this returns a tuple with the atom "module" and then the Module. If the string loaded from the database defined a module called "Paris", how in my code would I then execute

paris.handler([parameters])

since I don't know in advance that there will be a module called paris? I could know, by having the string "paris" also stored in the database that this is the name, but is there any way of calling into a module, using a string as the name of the module you're calling?

There is also:

eval(string, binding // [], opts // [])

Which evaluates the contents of the string. Can this string be the entire definition of a module? It appears not. I'd like to be able to write this code that's being evaluated in such a way that it has multiple functions that call each other--e.g. a complete little program, with a pre-defined entry point (Which could be a main, such as "DynamicModule.handle([parameter, list])"

Then there's the EEx module, which has:

compile_string(source, options // [])

Which is great for doing templates. But ultimately it only seems to work for the use case where there's a string and you've got Elixir code embedded in it. It evaluates the string in the context of the options and produces a string. I'm seeking to compile the string into one or more functions that I can then call. (If I can only make one function that's fine, that function can pattern match or switch into doing the other things that are needed....)

I know this is unconventional, but I have my reasons for doing it this way and they are good ones. I'm looking for advice about how to do this, but don't need to be told "don't do that". It seems like it should be possible, Erlang supports hot code loading and Elixir is pretty dynamic, but I just don't know the syntax, or the right functions. I'll be monitoring this question closely. Thanks in advance!

---

EDITS based on the first answers:

Thanks for the answers, this is good progress. As Yuri showed, eval can define a module, and as José points out, I can just use code eval for small parts of code with bindings.

The code being evaluated (whether turned into a module, or not) is going to be fairly complex. And its development would be best involving breaking it down into functions and calling those functions.

To help, let me provide some context. Assume I'm building a web framework. The code loaded in from the database is handlers for specific URIs. So, when an HTTP call comes in, I might load the code for example.com/blog/ This page might involve several different things, such as comments, a list of recent posts, etc.

Since many people are hitting the page at the same time, I'm spawning a process to handle each page view. Thus there are many times when this code may be evaluated simultaneously, for different requests.

The the module solution allows one to break the code up into functions for different parts of the page (eg: the list of posts, comments, etc. ) And I would load the module once, at startup, and let many processes spawn that call into it. The module is global, correct?

What happens if there's a module already defined? EG: When the module changes, and there are processes already calling that module.

In iex, I am able to redefine a module that's already been defined:

iex(20)> Code.eval "defmodule A do\ndef a do\n5\nend\nend"
nofile:1: redefining module A

What happens if I redefine the module at runtime, to all the processes currently calling into that module? Also, will this redefinition work outside of iex, in normal operation?

Assuming that redefining the module would be problematic, and that modules being global might run into problems with namespace collisions, I looked into using eval to define a function.

If I can merely have the code from the database define functions, then those functions are within the scope of whatever process, and we don't have the possibility of global collisions.

However, this doesn't seem to work:

iex(31)> q = "f = function do
...(31)> x, y when x > 0 -> x+y
...(31)> x, y -> x* y
...(31)> end"
"f = function do\nx, y when x > 0 -> x+y\nx, y -> x* y\nend"
iex(32)> Code.eval q
{#Fun<erl_eval.12.82930912>,[f: #Fun<erl_eval.12.82930912>]}
iex(33)> f
** (UndefinedFunctionError) undefined function: IEx.Helpers.f/0
    IEx.Helpers.f()
    erl_eval.erl:572: :erl_eval.do_apply/6
    src/elixir.erl:110: :elixir.eval_forms/3
    /Users/jose/Work/github/elixir/lib/iex/lib/iex/loop.ex:18: IEx.Loop.do_loop/1

iex(33)> f.(1,3)
** (UndefinedFunctionError) undefined function: IEx.Helpers.f/0
    IEx.Helpers.f()
    erl_eval.erl:572: :erl_eval.do_apply/6
    erl_eval.erl:355: :erl_eval.expr/5
    src/elixir.erl:110: :elixir.eval_forms/3
    /Users/jose/Work/github/elixir/lib/iex/lib/iex/loop.ex:18: IEx.Loop.do_loop/1

I also tried:

    iex(17)> y = Code.eval "fn(a,b) -> a + b end"
{#Fun<erl_eval.12.82930912>,[]}
iex(18)> y.(1,2)
** (BadFunctionError) bad function: {#Fun<erl_eval.12.82930912>,[]}
    erl_eval.erl:559: :erl_eval.do_apply/5
    src/elixir.erl:110: :elixir.eval_forms/3
    /Users/jose/Work/github/elixir/lib/iex/lib/iex/loop.ex:18: IEx.Loop.do_loop/1

So, in summary:

1. Can modules be redefined using Code.eval when there are processes calling into them?

2. Is it possible to use Code.eval to make functions whose scope is bound to the process in which Code.eval was called?

3. If you understand what it is I'm trying to do, can you suggest a better way to go about it?

Also, if there is a better forum where I should be asking this, feel free to let me know. And if there are docs or relevant examples I should read, please feel free to point me to them. I'm not trying to get you to do all the work, I just am unable to figure it out myself.

I'm learning Elixir specifically for the ability to dynamically evlauate code, but my Elixir knowledge is minimal now- I just started- and my erlang is rusty too.

Much thanks!

Solution 1

As you described, there are many different approaches you could take by ultimately they boil down to two different categories: 1) code compilation and 2) code evaluation. The example you described above requires compilation, which will define a module and then you would have to invoke it. However, as you found out, it requires defining a module name and potentially purging and discarding those modules when the database changes. Also, note that, defining modules may exhaust the atom table, as an atom is created for every module. I would only use this approach if you need to compile at maximum a dozen modules.

(A small note, Code.compile_string already defines the module, so you don't need to call load_binary).

Maybe a simpler approach is to call Code.eval passing the code from the database, thus code evaluation. It works fine if all you want to is to evaluate some custom rules. The Code.eval accepts a binding, which would allow you to pass parameters to the code. Let's suppose you have "a + b" stored in the database, where a and b are the parameters, you could evaluate it as:

Code.eval "a + b", [a: 1, b: 1]

EDIT BASED ON QUESTION'S EDITS

If you are evaluating code, keep in mind that Code.eval returns the result of evaluating the code and the new binding, so your example above would be better written as:

q = "f = function do
x, y when x > 0 -> x+y
x, y -> x* y
end"

{ _, binding } = Code.eval q
binding[:f].(1, 2)

However, given your use case, I wouldn't consider using eval but I would indeed compile a module. Since the information is coming from the database, you can use this fact to generate unique modules per record for you. For example, you can assume developers will be adding the contents of a module to the database, something like:

def foo, do: 1
def bar, do: 1

After you get this information from the database, you can compile it using:

record   = get_record_from_database
id       = record.id
contents = Code.string_to_quoted!(record.body)
module   = Module.concat(FromDB, "Data#{record.id}")
Module.create module, contents, Macro.Env.location(__ENV__)
module

Where record is anything that you get back from the database. Assuming the record id is 1, it would define a module FromDB.Data1 which then you would be able to invoke foo and bar.

Regarding code reloading, both defmodule and Module.create use :code.load_binary to load the module. Which means that if you update the module, the old version is still kept around until another new version is loaded.

One of the things you should add as well is cache expiration, so you don't need to compile a module on every request. This can be done if you have a lock_version in the database that you increment every time you change the record contents. The final code would be something like:

record  = get_record_from_database
module  = Module.concat(FromDB, "Data#{record.id}")
compile = :code.is_loaded(module) == false or
            record.lock_version > module.lock_version

if compile do
  id       = record.id
  contents = Code.string_to_quoted!(record.body)
  contents = quote do
    def lock_version, do: unquote(record.lock_version)
    unquote(contents)
  end
  Module.create module, contents, Macro.Env.location(__ENV__)
end

module

Solution 2

Code.eval can be used to define a module:

iex(1)> Code.eval "defmodule A do\ndef a do\n1\nend\nend" 
{{:module,A,<<70,79,82,49,0,0,2,168,66,69,65,77,65,116,111,109,0,0,0,98,0,0,0,11,8,69,108,105,120,105,114,45,65,8,95,95,105,110,102,111,95,95,4,100,111,99,115,9,102,117,...>>,{:a,0}},[]}
iex(2)> A.a
1

Does this help?

Solution 3

Have you checked: Dynamic Compile Library by Jacob Vorreuter. See example below

1> String = "-module(add).\n -export([add/2]). \n add(A,B) -> A + B. \n".
"-module(add).\n -export([add/2]). \n add(A,B) -> A + B. \n"
2> dynamic_compile:load_from_string(String).
{module,add}
3> add:add(2,5).
7
4>

question

answer

Comments

[nirvana]

This helped a great deal, and led to a productive night of exploration. I voted your answer up, but unfortunately some jerk has downvoted it. I regret that there is only one upvote I can give. Since my question was a bit wordy and vague, I've extended it based on your resposne with the results of my experiments: You can redefine modules, and that's good, but I'm not sure of the consequences. And I seem unable to define simple functions (which would presumably have the scope of the process in which Code.eval was called.)

[nirvana]

That library is helpful and shows that what I need to do can be done in erlang, which will be very useful if I decide that I don't want to program in elixir. However, my concern about modules is their global nature. Consider the example you referred to- imagine the server is getting two requests to do that work. Maybe it checks to see if the module is already loaded and doesn't dynamically load it twice... but when its done, how can it know that it is safe to unload it ? That there's not some other process using that module? I think process scope would be ideal.

[nirvana]

This would impose a predetermined form on the code, such that it is all broken up into small strings to eval. I've edited my question to explain further, but if I could define functions within the code and then call them inside the string, and also from outside, that would possibly be ideal. But it didn't work when I tried it. I'd like the STRING to be able to define functions A, B & C, say, and outside code define D, E, and F. And from inside the string that is sent to Code.eval and outside, be able to call A, B, C, D, E & F. EG: all functions scoped to the process. Is this possible?

[Muzaaya Joshua]

i think in such a case, you would try to get module info to see if it already is loaded, then your application would act accordingly.

[José Valim]

I have just answered your new questions.

[nirvana]

Only one remaining bit of confusion. Looking at your code, the last line is simply "module" which is effectively "FromDB.Data1" in this example. When you have "module" on a line alone like that, is that the equivalent of calling FromDB.Data1.main() or some default entry point? Or put another way, your code creates a module whose name is based on database info, which is fine, just how do I call it from the code that is outside of it? Can I write "module.renderRequest(a, b, c)" and that will call the renderRequest function defined in the module loaded from the database?

[José Valim]

Yes, we return the module which in this case is going to be FromDB.Data1 (not a binary, but an atom representing the module). This means we can invoke functions in it. All you need to do is ensure that at least a function named main (or anything you prefer) is available in the database, and then invoke it as: module.main(a, b, c).

[yaka]

it appears Code.eval no longer exists in Elixir 1.10.2. Instead, you should use Code.eval_string