Most Magik developers have never heard of a Closure, yet Functional Programming languages frequently use it to do powerful and useful things. And Magik supports Closures.
However, practically all Magik developers are taught to use the Object Oriented Programming (OOP) paradigm when they first learn Magik. Then as they start working on existing code, it’s OOP pretty much all the time. And closures don’t often appear in OOP.
Using procedures in a functional programming manner is not something we tend to do in Magik. In fact, we use procedures sparingly and, generally, imperatively.
However closures can help in a number of areas (such as with data privacy, currying and partial applications), so it’s beneficial to understand what they are and how they work.
Closures define what’s in a procedure’s lexical scope (in other words, it’s the procedure bundled with references to its lexical environment), and there are some surprising results surrounding that — which we’ll look at a bit later.
But first, let’s discuss lexical scope.
Lexical Scope
In Magik, if you define a local variable at the top of a method, a block in that method will have access to that variable. That’s lexical scope (blocks that are textually enclosed within other blocks have access to their outer blocks’ variables).
Look at the following code.
_global lexical_scope <<
_proc @lexical_scope(p_first_name)
_block
_local l_last_name << "Beeblebrox"
_for i _over 1.upto(3)
_loop
write(p_first_name, " ", l_last_name)
_endloop
_endblock
_endproc
There are two blocks in the lexical_scope procedure (one beginning on line 3 and the other starting at the loop statement in line 6). The write statement (line 8) references the parameter p_first_name and the local variable l_last_name from outside its block’s scope.
Executing the code writes the values stored in p_first_name and l_last_name to the terminal, thereby demonstrating the inner-most block can access variables in its enclosing blocks.
Magik> lexical_scope("Zaphod")
Zaphod Beeblebrox
Zaphod Beeblebrox
Zaphod Beeblebrox
Magik>
This is an example of lexical scope. It’s called, “lexical scope,” because during the lexing phase, the lexer uses the location of the variable declaration, in the source, to determine where that variable is in scope.
However this example is not a closure because the inner blocks are still inside the scope of the enclosing block (i.e. the lexical_scope proc has not yet gone out of scope). This results in the variable (p_first_name) still being free — in other words the free variable is not yet bound to a final value in the lexical environment.
An Actual Closure
Now look at this code…
_global closure <<
_proc @closure(p_first_name)
_return _proc @innerProc(p_last_name)
_import p_first_name
_for i _over 1.upto(3)
_loop
write(p_first_name, " ", p_last_name)
_endloop
_endproc
_endproc
We’ve modified the code slightly to return a procedure (line 4). In line 6 we have to import the p_first_name parameter because Magik doesn’t do automatic lexical scoping for procedures like other languages do for functions.
At this point you might be thinking there’s no way this code will work, because the p_first_name parameter variable will only exist during the closure procedure’s life. So when closure finishes executing, p_first_name won’t exist any longer when innerProc is eventually invoked.
That’s a reasonable assumption, but look at what happens when we execute the procedure.
Magik> z << closure("Zaphod")
proc innerproc(p_last_name)
Magik> z("Beeblebrox")
Zaphod Beeblebrox
Zaphod Beeblebrox
Zaphod Beeblebrox
Magik> a << closure("Arthur")
proc innerproc(p_last_name)
Magik> a("Dent")
Arthur Dent
Arthur Dent
Arthur Dent
Magik>
In line 1, we invoked the closure procedure, setting the first name to Zaphod, and stored a reference to the procedure it returned (innerProc) in the variable z. The closure is now stored in the variable z.
In line 4 we then invoked the procedure (referenced by z) and provided the last name, Beeblebrox, to invoke the inner procedure. This wrote, “Zaphod Beeblebrox” to the terminal.
Wait a minute! Why is, “Zaphod,” still being written out when we invoke z()? How in the world does the inner procedure still have access to the outer procedure’s p_first_name variable after the outer procedure has finished executing and gone out of scope?
We did a similar thing for the variable a, in lines 9 and 12, and the same thing happened. a() remembers the value, “Arthur.”
What the heck?!?
I’ll get to the reason in a moment, but for now, look at this…
Magik> z("Dent")
Zaphod Dent
Zaphod Dent
Zaphod Dent
Magik> a("Beeblebrox")
Arthur Beeblebrox
Arthur Beeblebrox
Arthur Beeblebrox
Magik>
Whoa! It happened again! z() is remembering, “Zaphod,” and a() is remembering, “Arthur.” What’s happening here?
A closure is what’s happening.
When innerProc is returned, the closure procedure goes out of scope. This results in its free variable (p_first_name) now being bound (or closed in the lexical environment) to the specific value referenced by p_first_name when the closure procedure went out of scope. At this point we have a closure.
The procedure referenced by z still remembers the variable p_first_name as being set to Zaphod although the outer scope closure procedure has finished executing and returned the inner procedure. So when we invoke the inner procedure, it still has access to the outer procedure’s parameter and remembers its value as it was at the time the inner procedure was returned.
And that’s why the procedures referenced by z and a can write, “Zaphod” and “Arthur” respectively.
This is usually surprising to most Magik developers.
The reason for this behaviour is that procedures in Magik form closures — which is a combination of the procedure and the lexical environment (such as outer procedures’ variables and resources) attached to the procedure when it was returned.
And the reason the inner procedure can still access the p_first_name variable is because it maintains a reference to its lexical environment even after the outer procedure has gone out of scope.
An easy way to remember this is to think of a closure as a procedure wearing a backpack. The backpack contains the values of all the variables that were in its scope when the closure was created. If those variables belonged to outer blocks that went out of scope, the backpack stores their values.
This behaviour is common in many languages. Magik implements it in a slightly different way, but other than having to use the _import statement to access variables in the outer procedure’s scope, for all intents and purposes, closures and lexical scoping work the same way in Magik as they do in other languages (such as Scheme, JavaScript, LISP and C#).
Alright then… that’s pretty cool, but why are they useful?
To answer that, let’s look at a couple more examples.
Utilizing Closures
In this example we’ll see how closures can help us implement truly private data and behaviour.
We’ve all been told that encapsulating data (and behaviour) is necessary to create easily maintainable software… and that’s true, because modifying shared data results in a much higher probability of causing unintended side-effects that can lead to problems in other parts of the application.
Unfortunately when we encapsulate data inside a class, it’s not truly private. Subclasses and all sorts of methods have access to these data (even if they don’t require such access).
Closures come to the rescue. See if you can figure out what this code is doing…
_global counter <<
_proc @counter()
_local l_count << 0
_local l_add << _proc @add(p_value)
_import l_count
l_count +<< p_value
write("New Value: ", l_count)
_endproc
_return _proc @inc()
_import l_add
l_add(1)
_endproc
_endproc
Got it? Great. Let’s analyze it anyways.
In line 4 we’re setting a local variable, l_count, to 0. This will be our private variable.
In line 6, we’re defining a procedure. This will be our private procedure.
Finally, in line 15, we’re returning a procedure, inc(), our public procedure, that imports the l_add variable pointing to our add procedure. In line 18 we call add with an argument of 1 (that is, increment, by 1, whatever the value in l_count happens to be).
Now when we execute counter(), inc() will be returned and counter() will go out of scope. However inc() will still have access to its lexical environment (that includes add()).
Notice we didn’t import l_count, so inc() doesn’t have direct access to it, but because inc() has access to add() and add() has access to l_count, inc() has indirect access to l_count. Had we wanted direct access, we would have simply imported l_count in inc().
Now let’s invoke counter() and see what happens.
Magik> c << counter()
proc inc
Magik> c()
New Value: 1
Magik> c()
New Value: 2
Magik> c()
New Value: 3
Magik> c()
New Value: 4
Magik> c1 << counter()
proc inc
Magik> c1()
New Value: 1
Magik> c()
New Value: 5
Magik>
See how we create multiple counters (lines 1 and 16) and they independently maintain their correct counts. Changes to the private variable l_count in one closure does not affect the variable in the other closure. Each closure references its own instance of l_count.
Also note that since counter() has gone out of scope, l_count and add() cannot be referenced except through inc(). The data and procedure are truly private to inc().
What we’ve created is a factory procedure, counter(), that returns a public inc() procedure while hiding the private add() procedure and l_count variable. This pattern is very useful in many situations as you’ll see in other articles on this site.
We can also use this pattern to manage the global namespace and hide a module’s private data and procedures. By packaging code inside well-encapsulated modules, entire classes of errors are automatically eliminated.
Let’s see how we can do this.
_block
_global test_module
test_module <<
_proc @test_module()
_local l_first_name << "Zaphod"
_local l_last_name << "Beeblebrox"
_local l_ship << "The Heart of Gold"
_constant WRITE_PARAM << _proc @write_param(p_str)
write(p_str)
_endproc
_constant EXPORT << beeble_object.new()
EXPORT.first_name << _proc @first_name()
_import l_first_name
_import WRITE_PARAM
WRITE_PARAM(l_first_name)
_endproc
EXPORT.last_name << _proc @last_name()
_import l_last_name
_import WRITE_PARAM
WRITE_PARAM(l_last_name)
_endproc
EXPORT.ship << _proc @ship()
_import l_ship
_import WRITE_PARAM
WRITE_PARAM(l_ship)
_endproc
_return EXPORT
_endproc()
_endblock
As before, we’ve created private local variables in lines 6, 7 and 8 as well as a private procedure (WRITE_PARAM) in line 10.
We then create an export object in line 15 that will contain the public procedures we want to expose (lines 17, 24 and 31). Notice how we use the private variables and private procedure in each of the public procedures. We have to explicitly import them, they’re not available by default. So ship() does not have access to l_first_name for example.
Then, in line 38, we return the export object. This object is a hard object that does not contain any data, just methods (i.e. procedures).
Finally, in line 40, we immediately invoke the test_module procedure by appending ().
When this block is compiled, it creates the global variable test_module that references the hard object with our public methods. We can use the module like this…
Magik> test_module.ship()
The Heart of Gold
Magik>
Magik> test_module.first_name()
Zaphod
Magik>
Magik> test_module.last_name()
Beeblebrox
Magik>
Note how all public methods are hidden behind the test_module object so they don’t pollute the global namespace. Also notice that while the public methods we exported are available, the private variables (l_first_name, l_last_name and l_ship) and private procedure (WRITE_PARAM) are not because they remain locked away in the closure. In fact there is no way to access these private entities except through the public methods.
That last part is important because it provides truly private data and methods accessible only internally by the module.
Using Magik’s standard exemplar to create new classes (and then defining variables and methods on them) does not provide truly private data and behaviour because subclasses and other methods on the class can access these items by default (not to mention we can change private methods to be public, using the set_private(_false) method, at the Magik prompt and use sys!slot() and sys!perform() to break encapsulation).
So if you’re writing a library or module and want truly private data and methods, closures provide a nice way of getting that.
In Closing
There are also other uses for closures, especially in functional programming — where they are used for currying and partial application.
Closures can take some time to fully comprehend, so it’s a good idea to go through the examples carefully and play around with the code at the Magik prompt. Once you have a basic understanding, the next step is to explore more advanced ideas (such as the aforementioned currying and partial application). But hopefully this article gives you a good start.
